Clussh

Stream task execution, through ssh, to one or many host, one or many times, in parallel or in series.

Features:

• Distribute tasks upon many hosts
• Stream task from stdin
• Stream task outputs and progression stats to ldjson stream
• Support scaling, parallel execution and retry, network failure and host unavailability

Installation

npm install -g clussh

This will install three command-line cli:

• clussh - The main command line
• clussh-board - A streamable clussh progress board
• clussh-log - Pretty print clussh logs

Pre-requirement: NodeJS and its awesome universe

Usages & exemples

# Run "Hello $(hostname)" on the default worker ssh://yourusername@localhost 
clussh
 
# Run given command on default worker 
clussh --cmd 'echo "Hello world"'
clussh --cmd 'echo "Hello world"' --scale 10 # Teen times 
clussh --cmd 'echo "Hello world"' --scale 10 --concurrency 2 # Two in parallel 
 
# Run given bash script on default worker 
clussh ./install.sh
 
# Run given bash script on to agents (with a concurrency of 2 for agent-b) 
# (NB: for ssh cooking & baking reasons, the shell script must ends with an exit statement) 
clussh -w ssh://[e-mail] -w ssh://[e-mail]?concurrency=2 ./runme.sh
 
# Pipe task from line delimited json stream 
echo '
{ "id": "task-a", "cmd": "echo hello; sleep 1" }
{ "id": "task-b", "cmd": "echo hello; sleep 2", "scale": 10 }
{ "id": "task-c", "script": "./runme.sh", args: ["--option", "value"] }
{ "id": "task-d", "script": "./runme.sh", "worker": "ssh://[e-mail]" }
' | clussh -w ssh://user@agent-a -w ssh://user@agent-b
 
# You can pipe infinite line delemited json stream (eg. event logs for a file wacher...) 
infinite-stream | clussh
 
# Print human readable logs 
clussh | clussh-log
 
# Print nice-looking progress board 
clussh | clussh-board
 
# Print human readable logs 
clussh | clussh-log
 
# Save logs for further exploration (and still display the board) 
clussh | tee output.log | clussh-board
tail -f output.log | clussh-log   # Another console... 
tail -f output.log | clussh-board # Another console... 
tail -f output.log | ndjson-filter 'd.type === "fail"' | prettyldjson # Another console... 

Command line usage and configuration

clussh [options] [script filepath]
 
Options:
  --help             Show help                                         [boolean]
  --version          Show version number                               [boolean]
  --retry, -r        How many retry for each task          [number] [default: 0]
  --timeout, -t      Task timeout in millisecond or parseable ms time (eg. 1h,
                     2d or `3 days`)                   [string] [default: "10d"]
  --worker, -w       Repeatable worker uri list
                                   [array] [default: "ssh://jponchon@localhost"]
  --concurrency, -c  Concurrency per worker                [number] [default: 1]
  --scale, -s        Default scale                         [number] [default: 1]
  --cmd              Command to execute                                 [string]
  --script           Script to execute (please ensure proper exit)      [string]
  --identity, -i     List ssh identity files to use        [array] [default: []]

Clussh can be configured with a .clusshrc file (see https://www.npmjs.com/package/rc)

{
  "worker": [
    "ssh://[e-mail]",
    "ssh://[e-mail]",
    "ssh://[e-mail]?concurrency=2"
  ],
  "concurrency": 4,
  "retry": 10
}

Above, one worker uri overide the default clussh configuration using url query string (?concurrency=2 for host agent-b.local). This will result in two workers for agent-b.local: one with the default concurrency (4) and another with a concurrency of 2.

Log message format

Clussh output line delemited json stream consumable by clussh-board or clussh-log or any other line delemited json tools (see prettyldjson for pretty print and ndjson-cli for map, reduce, filtering, etc.)

TODO: Fields documentation & messages types

API

const clussh = require('clussh')
 
const clusshStream = clussh({
  cmd: 'hello',
  worker: [ 'ssh://[e-mail]', 'ssh://[e-mail]' ],
  retry: 3,
  concurrency: 2
})
 
clusshStream.pipe(process.stdout)

---

License: MIT - Novadiscovery