Nylon Pendulum Movement

    shoal

    0.2.3 • Public • Published

    Shoal

    Build Status

    Shoal allows you to group and run multiple processes. Shoal manages these processes and exposes controls and status information via a HTTP JSON API. Aside from the JSON interface there is a Web user interface and a CLI.

    Use-cases:

    • Centralize process/worker configuration using a single JSON config file (Avoid hundreds of upstart scripts)
    • Use a CLI and Web UI to monitor and control a large number of process/worker instances
    • Simplifying process/worker management for developers (e.g. local installs of ETL's)
    • Smart deployments: Automatically stop workers that have been removed from configuration, and start instances that were added
    • Centrally control logging, environment and restart behavior of processes

    Status & Security

    We provide two authentication mechanisms out of the box (see Authentication below), however these are not configured by default and have not been battle-tested in production environments.

    By default, the Shoal Manager (shoald) binds to 127.0.0.1 (loopback device) and allows execution of commands sent to it. This is something to be taken into consideration when running outside of sandboxed environments.

    Please be mindful of this when running this in a production environment.

    Install

    npm install shoal
    

    Usage

    In order to deploy and monitor processes with one of the user interfaces, you need to run the Shoal Manager. This is done by executing the shoald command that's included with the NPM module.

    By default, the Shoal Manager listens for commands on port 54047 and also binds the Admin UI on port 54047:

    $ shoald
    Shoal Manager running on http://127.0.0.1:54047
    Shoal Admin UI running on http://127.0.0.1:54048
    

    Here are some CLI options that can be given to shoald:

    $ shoald --help
    
      Usage: shoald [options]
    
      Options:
    
        -h, --help                     output usage information
        -V, --version                  output the version number
        -c, --config <path>            Run server with this configuration
        -pd, --deploy <path>           Run Manager with this process deployment
        -p, --port <port>              Port to bind on (Default: 54047)
        -b, --bind <host>              Host to bind on (Default: 127.0.0.1)
        -ap, --admin-port <port>       Port to bind Admin UI on (Default: 54048)
        -ab, --admin-bind <host>       Host to bind Admin UI on (Default: 127.0.0.1)
        -ah, --admin-html-file <path>  Path to HTML file to customize Admin UI
        --disable-ui                   Disable Admin UI
        --quiet                        Don't show Shoal info
        --verbose                      More verbose output
    
    

    Once the Shoal Manager is running process configuration can be deployed with the shoal deploy command:

    shoal deploy examples/simple/processes.json
    

    A status of current processes and number of instances can be retrieved using the shoal status command:

    shoal status
    

    Build Status

    Alternatively, you can use the Admin Web UI to view and control processes (Defaults to http://localhost:54048/):

    Build Status

    Process Configuration

    The Shoal Manager takes a JSON configuration that allows us to specify which processes should be run and how they should be executed. In Shoal, a process is a definition of a command that should be daemonized. One process can have many instances that all perform the same job.

    Here's an example configuration:

    {
      "env": {
        "NODE_ENV": "staging"
      },
      "killTimeoutMs": 30000,
      "logRoot": "/var/log",
      "processes": [
        {
          "name": "Cassandra Worker",
          "cmd": "node",
          "args": ["script/worker.js", "cassandra"],
          "env": {
            "NAME": "$processName",
          },
          "instances": 8,
          "autoRestart": true,
          "autoRestartTimeoutMs": 3000
        },
        {
          "name": "Some Daemon",
          "cmd": "somed",
          "args": ["-f"],
          "killTimeoutMs": 1000,
          "logRoot": "~/log",
          "outFile": "somed.log",
          "errFile": "somed.error",
          "instances": 0
        }
      ]
    }

    Each process in the processes array takes the following required parameters:

    Parameter Description
    name Name of the process/worker
    cmd Unix command to be executed without parameters (can be absolute path too)
    args Array of arguments to be given to the unix command
    instances The number of instances of this process that should be run. Can be 0 for inactive process definitions

    The following options can be given either in the top-level JSON or in the indvidual process definitions (overrides global settings):

    Option Defaults Description
    env {} Hash with environment variables that should be used. See below for available Macros
    killTimeoutMs Number of miliseconds after which a unresponsive process instance (not responding to SIGHUP) should be send a SIGKILL
    autoRestart false Automatically restart a process instance after exit
    autoRestartTimeoutMs 1000 Number of miliseconds to wait until restarting the process instance
    logRoot Specify output logging directory. When specified each process will have stdout and stderr send to respective process-name.out and process-name.err files
    outFile $processShortName.out Override default stdout output file. Requires logRoot to be set
    errFile $processShortName.err Override default stderr output file. Requires logRoot to be set
    quiet false Don't show any output from shoal manager server or admin UI

    Environment Macros

    When configuring custom environment variables for a process or configuration, you can use the following internal variables/macros:

    Variable Description
    $processName Configured name of the process
    $processShortName A dashed and lowercase conversion of the processName (e.g. My Worker => 'my-worker')
    $processId A unique ID of the process based on the name, cmd and env
    $shoalCwd The working directory path that the Shoal Manager is running in
    $shoalVersion Shoal version
    $logRoot The configured log root for the process

    Server/Client Configuration

    Using the --config flag you can supply a JSON file with configuration options to the shoald and shoal commands:

    Option Defaults Description
    host 127.0.0.1 Shoal Manager host to bind or connect to
    port 54047 Shoal Manager port to use for binding or connecting
    adminHost 127.0.0.1 Shoal Admin UI host to bind on
    adminPort 54048 Shoal Admin UI port to bind on
    auth Authentication options for Shoal Manager (see below)
    adminAuth Authentication options for Shoal Admin UI (see below)

    Authentication

    Currently we support two methods of authentication: basic and ssl.

    Basic Authentication

    Here's an example of a server-client configuration with basic authentication:

    {
      "auth": {
        "type": "basic",
        "key": "my scecret"
      },
      "disableAdminUi": true
    }

    Instead of specifying the key in the configuration you can instead use keyFile to point to a file.

    Basic auth can also be applied on the Admin UI as follows:

    {
      "auth": {
        "type": "basic",
        "key": "my scecret"
      },
      "adminAuth": {
        "type": "basic",
        "username": "admin",
        "key": "secret"
      },
    }

    SSL Authentication

    A more secure way of authenticating is to use SSL. We recommend this authentication strategy.

    This requires first setting up the certificates (CA, Server and Client). A convenience script is provided in tools:

    bash tools/create-ssl-keys.sh

    Once the certificates are made, configure your server:

    {
      "auth": {
        "type": "ssl",
        "keyFile": "cert/server.key",
        "certFile": "cert/server.crt",
        "caFile": "cert/ca.crt",
        "passphrase": "shoal"
      },
       "disableAdminUi": true
    }

    And your client:

    {
      "auth": {
        "type": "ssl",
        "keyFile": "cert/client.key",
        "certFile": "cert/client.crt",
        "caFile": "cert/ca.crt",
        "passphrase": "shoal"
      },
      "host": "localhost"
    }

    A full example of this can be found in examples/ssl-auth.

    Future

    • Improve default security features
    • Allow Shoal to dynamically control the number of instances per process definition based on the 'business' of the running instances

    License

    (The MIT License)

    Copyright (c) 2014 Dominiek ter Heide dominiek@bottlenose.com

    Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

    The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

    Install

    npm i shoal

    DownloadsWeekly Downloads

    5

    Version

    0.2.3

    License

    MIT

    Last publish

    Collaborators

    • dominiek