Huxtable: hierarchical remote command execution via SSH
Huxtable is a simple mechanism for executing commands on clusters of servers organized in an arbitrary hierarchical fashion. It was originally developed to simplify managing complex firewall setups on a cluster, but it can be used to run just about any kind of command.
Huxtable can work many servers at the same time, and allows you to group machines logically so that you can execute the same command against multiple hosts just by adding or removing them to the appropriate group.
Usage
Huxtable is designed to run from the command line; you can install it with npm
:
npm install -g huxtable
The executable can be invoked by name; you can pass --help
to discover the options it offers—they're fairly self-explanatory.
Specs
Huxtable is controlled by means of specfiles. A specfile (or spec for short) is a YAML file that describes your targets and the commands you want to run against them; since JSON is a subset of YAML, you can also write your specs in JSON, or even in “relaxed” JSON without quotation marks around property names.
A spec is divided in three sections:
default
defines properties that apply to every host.groups
defines properties that apply to specific named groups of hosts.hosts
defines properties that apply to a specific machine, and describes the membership of individual hosts into zero or more groups.
Common properties
Each of the three main sections contains four subsections:
before
defines a group of commands that are executed as soon as a connection is established with the remote hostcommands
defines a group of commands that are executed right afterbefore
after
defines a group of commands that are executed right aftercommands
error
defines a group of commands that are executed if any of the other commands returns a non-zero exit code
Why this arrangement? At runtime, Huxtable computes the properties that apply to each host using this algorithm:
- Take the default properties
- Add the properties that belong to each group of which the host is a member
- Add the properties for the host
By dividing commands in three phases, you can control the flow of execution and still break up your commands in a convenient way.
Connection properties
Huxtable supports three properties to provide connection details; they can be specified at either the default, group, or host level (with each overwriting the one before):
username
The username to be used when connecting.password
The password to be used when connecting.privateKey
The private key to be used for key-based authentication.passphrase
The passphrase require to decrypt the private key, if any.
In addition, every host must have a hostname
property that provides either its FQN or IP address.
Group membership
Each host can—but doesn't have to—be a member of one or more groups. Simply provide the name of each group in the groups
property. For example:
hosts:
web-1:
hostname: 192.168.0.2
groups: [ 'web' , 'mysql' ]
Note that group membership is not recursive; a group cannot be a member of another group, and groups
properties in a group will be ignored.
Issuing commands
Commands are specified as a list of either strings or hashes. In the case of a string, the command is executed as-is; if, instead, you pass a hash, Huxtable looks for a template
property, which provides a template for the command, and an at
property, that provides a template for substitutions of the @
character inside the template.
The at
property, in turn, should be a hash that can contain one of two properties: host
, with a value that references a specific host, and group
, with a value that references a group. At runtime, the template is compiled by either replacing the @
character with either the hostname of the host or by creating a separate copy of the command for each host that is part of a group.
For example:
groups:
web-workers:
mysql-hosts:
commands:
- template: iptables -I INPUT -s @ -j ACCEPT
at: { group : 'web-workers' }
hosts:
web-1:
hostname: 192.168.1.1
groups: [ 'web-workers' ]
web-2:
hostname: 192.168.1.2
groups: [ 'web-workers' ]
mysql-1:
hostname: 192.168.1.3
groups: [ 'mysql-hosts' ]
In output, this spec for the mysql-1 host will yield the following:
iptables -I INPUT -s 192.168.1.1 -j ACCEPT
iptables -I INPUT -s 192.168.1.2 -j ACCEPT
Example
As an example, consider setting up iptable rules for a cluster that contains the following:
- A cluster of five machines running Riak (
riak-1
throughriak-5
) - A web proxy (
web-proxy
) that load-balances a cluster of three web workers (web-1
throughweb-3
)
Generally speaking, we want:
- All machines to refuse connections by default unless otherwise specified or established/related
- All machines to allows SSH connections
- The web proxy to allow incoming http and https connections, and allow outgoing connections to the workers on port 8000
- The workers to allow incoming connections on port 8000 from the proxy, and outgoing connections to the Riak cluster on port 8198
- The Riak machines to be able to speak to each other, and allow incoming connections from the web workers on port 8198
- On success, use
iptables-save
to save the rules to/etc/iptables.conf
- On error, use
iptables-restore
to avoid leaving the firewall in an incorrect state
This could be done with a spec that looks like this:
default:
username: root
privateKey: /home/me/.ssh/id_rsa
before:
- iptables -P INPUT ACCEPT
- iptables -P OUTPUT ACCEPT
- iptables -P FORWARD ACCEPT
- iptables -F
commands:
- iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
- iptables -A INPUT -p tcp --dport ssh -j ACCEPT
- iptables -A OUTPUT -m conntrack --ctstate INVALID -j DROP
- iptables -A OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
after:
- iptables -P INPUT DROP
- iptables -P FORWARD DROP
- iptables -P OUTPUT DROP
- iptables-save > /etc/iptables.conf
error:
- iptables-restore /etc/iptables.conf
groups:
web-workers:
commands:
- template: iptables -A INPUT -p tcp --dport 8000 -s @ -j ACCEPT
at: { host : 'web-proxy' }
- template: iptables -A OUTPUT -p tcp --dport 8098 -d @ -j ACCEPT
at: { group : 'riak-nodex' }
riak-nodes:
commands:
- template : iptables -A INPUT -p tcp -s @ -j ACCEPT
at : { group : 'riak-nodes' }
- template : iptables -A INPUT -p tcp --dport 8098 -s @ -j ACCEPT
at : { group : 'web-workers' }
hosts:
web-proxy:
hostname: 192.168.1.1
commands:
- iptables -A INPUT -p tcp --dport http -j ACCEPT
- iptables -A INPUT -p tcp --dport https -j ACCEPT
- template : iptables -A OUTPUT -p tcp --dport 8000 -d @ -j ACCEPT
at : { group : webWorkers }
web-1:
hostname: 192.168.1.2
groups: [ 'webWorkers' ]
web-2:
hostname: 192.168.1.3
groups: [ 'webWorkers' ]
web-3:
hostname: 192.168.1.4
groups: [ 'webWorkers' ]
riak-1:
hostname: 192.168.1.5
groups: [ 'riakNodes' ]
riak-2:
hostname: 192.168.1.6
groups: [ 'riakNodes' ]
riak-3:
hostname: 192.168.1.7
groups: [ 'riakNodes' ]
riak-4:
hostname: 192.168.1.8
groups: [ 'riakNodes' ]
riak-5:
hostname: 192.168.1.9
groups: [ 'riakNodes' ]
With this arrangement, you can now easily add or remove machines from the various clusters, or change the way they are supposed to communicate without any duplication in your commands.