testarmada-crows-nest

Crows-Nest

A supervisor tool to launch and monitor multiple SauceLabs Sauce Connect tunnel in high availability mode.

The tool tracks the availability of each tunnel. If any of the tunnels is unresponsive, the supervisor will attempt to terminate the tunnel gracefully and start a new one. Additionally, the tool provides the ability to perform "rolling restarts" all the tunnels periodically to avoid known issues with long-running tunnel processes.

By launching sauce tunnels in high availability mode with crows-nest, you won't need to:

1. manually check the availability of each tunnel
2. manually launch up new tunnels to replace the dead ones
3. manually and periodically restart all tunnels

Media

• Building a Better Wormhole: A presentation about crows-nest at SauceCon 2017 (video, slides)

Usage

Prerequisites

1. crows-nest@1.6.0 and above requires node@6 and above.
2. npm install to install.

Configuration

./config.json
{
  "tunnel": {
    "username": "",
    "accessKey": "",
    "verbose": false,
    "proxy": null,
    "tunnelIdentifier": "",
    "noRemoveCollidingTunnels": true,
    "sharedTunnel": true
  },
  "supervisor": {
    "portStart": 4000,
    "portIndent": 5
  },
  "stats": {
    "statsType": "",
    "statsHost": "",
    "statsPort": null,
    "statsPrefix": "",
    "statsDatabase": ""
  },

  "restartCron": "0 */4 * * *"
}

./config.json file has the basic configurations that are required by Sauce Connect.

Please note: since saucelabs-connect@4.4.x, waitTunnelShutdown is removed from configuration.

Tunnel config

To launch your own tunnels, tunnel.username, tunnel.accessKey and tunnel.tunnelIdentifier are mandatory. More configurations please refer to this page sauce-connect-launcher.

In high availability mode all tunnels share the same tunnel.tunnelIdentifier. tunnel.tunnelIdentifier can be any string. One suggested convention is to use this ID to describe the geographical location where your tunnel terminates. For example, east or west.

The restartCron value is any valid cron schedule. For example, 0 */4 * * * would mean "every 4 hours". We recommend crontab.guru for help generating valid cron strings to match the desired schedule.

You can set tunnel.username and tunnel.accessKey using one of the following methods:

1. Specifying the values in ./config.json
2. Setting the environment variable SAUCE_USERNAME and SAUCE_ACCESS_KEY

Supervisor config

Since @1.5.0 crows-nest supports multi-tenant mode. If you want to run multiple crows-nest instances per box, supervisor.portStart and supervisor.portIndent are configs you want to change. supervisor.portStart allows you to identify the port of the first tunnel of current crows-nest instance will use, then every following tunnels will pick up the port supervisor.portStart + index * supervisor.portIndent.

Stats config

Since @1.3.0 crows-nest starts to support pushing stats to a statsd-like system. To utilize this data pushing function you need to explicitly add --stats in your command. There are only two adaptors supported for now, adaptor for telegraf and adaptor for pushing data into influxdb directly. You can extend lib/stats/base to add more adaptors.

For instance, in config.json, stats.statsType = influxdb will tell the adaptor factory to look for a mapping with key influxdb configured in factory.js. Adaptor factory will return an instance of influxdb adaptor which is defined in influxdb.js. If there is no adaptor found by adaptor factory, it will throw an error to prevent crows-nest from starting.

Stats only supports gauge for now.

Following data will be gathered and pushed

1. How many connection attempts a tunnel has been made before failing
2. How many connection attempts a tunnel has been made for now
3. How many attempts a tunnel has been made to successfully connect
4. When a tunnel successfully stopped
5. How long a tunnel run (unix timestamp)
6. How long a tunnel takes to successfully connect

Other configs

If the rolling restart feature is enabled, restartCron must be a valid cron value.

Help

./bin/supervise --help

Basic usage

Start one sauce tunnel in high availability mode:

./bin/supervise --tunnels 1

With rolling restart feature:

SAUCE_USERNAME=xxx SAUCE_ACCESS_KEY=yyy ./bin/supervise --tunnels 1 --rollingRestart

Advanced usage

Read sauce tunnel configuration from ./myproject/config.json and launch 20 sauce tunnels in high availability mode, with rolling restarted and stats data pushing feature enabled

./bin/supervise --tunnels 20 --config ./myproject/config.json --rollingRestart --stats

Daemon

We use pm2 to run the supervisor process as a daemon

To start:

./node_modules/.bin/pm2 start ./bin/supervise --kill-timeout 600000 --silent -- --tunnels 10 --rollingRestart

To stop:

./node_modules/.bin/pm2 stop supervise

To view the log:

./node_modules/.bin/pm2 log --lines 100

Multi tenants

To start:

./node_modules/.bin/pm2 start -n ${TENANT_NAME} ./bin/supervise --kill-timeout 600000 --silent -- --tunnels 10 --rollingRestart --config ${TENANT_NAME}_config.json

To stop:

./node_modules/.bin/pm2 stop -n ${TENANT_NAME}

Design

Architecture

Components

There are two major components in Crows-nest, Supervisor and Tunnel

Tunnel

Crows-nest Tunnel maintains the life cycle of a Saucelabs Sauce Connect Tunnel. It is a one to one mapping to Saucelabs Sauce Connect Tunnel instance. It does following things

1. Start a Sauce Connect Tunnel as child process
2. Monitor the status of current child process
3. Terminate current Sauce Connect Tunnel nicely if the connection drops and start a new one.
4. Restart current Sauce Connect Tunnel if scheduled by rollingRestart
5. Stop current Sauce Connect Tunnel

Supervisor

Crows-nest Supervisor keeps track of all Crows-nest Tunnels. It does following things

1. Initialize all Tunnels
2. Start all Tunnels by sending start signals to each Tunnel
3. Restart all tunnels by sending restart signals to each Tunnel according to schedule
4. Stop all Tunnels by sending stop signals to each Tunnel

Running From Docker

1. Setup your config.json
2. Build:

docker build -t testarmada/crows-nest .

1. Run:

docker run --rm testarmada/crows-nest [YOUR COMMAND HERE]

For example:

docker run --rm testarmada/crows-nest bin/supervise --tunnels 1

Randomness

To avoid network blast (in case all tunnels are scheduled at the same time), some randomness is introduced

1. Each tunnel takes random delay [0, 5000] ms to start
2. Each tunnel takes random delay [0, 5000] ms to stop

Licenses

All code not otherwise specified is Copyright Wal-Mart Stores, Inc. Released under the MIT License.