shrun (beta)
Test NodeJS CLI commands in isolated docker containers.
- Dependencies
- Quickstart
- Spec format
- Available CLI options
- Why shrun?
- Examples and resources
- Known issues and unfinished work
Dependencies
Docker is required to run shrun
Install Docker here if you need it
Quickstart
Add shrun to your project
npm install --save-dev shrun# or for yarn yarn add --dev shrun
Build the default shrun image
npx shrun build
depending on how Docker is installed, sudo
may be necessary
sudo -E npx shrun build
Note: build also takes an optional
--image
flag which accepts either "alpine" or "ubuntu" (default: ubuntu)
Create a simple spec
By default, shrun
looks for "specs" (CLI tests) in the <project-root>/specs
directory. Here is an example spec that tests the echo
command in bash:
specs/demo.yml
---- test: Test echo (good-path) steps: - in: echo "Hello world" out: |- Hello world
Replace or add to this spec to test your specific CLI command. As long as you built the default shrun
image, it should work out of the box.
For detailed information about writing shrun
specs, read below.
Run the demo.yml spec
npx shrun
sudo version
sudo -E npx shrun
Spec format
- test: Test init help output setup: - "curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash -" - "sudo apt install nodejs" steps: - in: npm init --help out: |- npm init [--force|-f|--yes|-y|--scope] npm init <@scope> (same as `npx <@scope>/create`) npm init [<@scope>/]<name> (same as `npx [<@scope>/]create-<name>`) aliases: create, innit
test: string
(required) - each spec test must have a test
stanza with a unique name. For those who are familiar with Jest/Ava/Mocha, this maps directly to the test("someName", () => {})
format used by those frameworks.
setup?: string[]
- the setup section allows you to run a series of shell commands before the test itself runs. This is convenient for tests that rely on a specific set of environment variables, need iptables configured etc. For those who are familiar with Jest/Ava/Mocha, this partially maps to the beforeEach
(more like beforeThis
since you specify it per test) construct.
steps: Step[]
(required) - steps are where the bulk of your test logic is defined and there is no limit to the number you can have per test. All steps must have an in
entry, this is what will actually be run against the containers internal shell. If a step is expected to succeed, it is a PassStep and must have an out
entry. in
and out
map to actual
and expected
in traditional testing frameworks. If a test is not expected to succeed (not 0 exit code), it must either have an err
or exit
entry. err
is similar to out
but is checked against stderr as opposed to stdout. exit
makes it possible to specify the expected exit code that resulted from running the tests in
statement.
There are also two other stanzas not used in the above spec:
cleanup?: string[]
- the exact same as setup
but runs after the test has finished. Useful for resource cleanup. Maps to the afterEach/afterThis
construct in traditional testing frameworks.
foreach: Map<string, string>[]
- allows a single test to be run multiple times with different input values.
Available CLI options
Note: we did our best to test all of the Jest options we are surfacing, but some may not behave as expected. Feel free to open an issue if you run into any issues.
--help Show help [boolean] --version, -v Print the version and exit [boolean] --volume, --vo Mount a volume into your docker container [array] --dockerImage Docker image to use when running Shrun tests [string] --dockerEnvVars Environment variables that should be passed into the Docker container [array] --bail, -b Exit the test suite immediately after `n` number of failing tests. [boolean] --cache Whether to use the transform cache. Disable the cache using --no-cache. [boolean] --ci Whether to run Jest in continuous integration mode. This option is on by default in most popular CI environments. It will prevent snapshots from being written unless explicitly requested. [boolean] [default: false] --clearCache Clears the configured Jest cache directory and then exits. Default directory can be found by calling jest --showConfig [boolean] --color Forces test results output color highlighting
Why shrun?
shrun
aims to bring consistency and convenience to the process of testing command line tools. Each individual spec is run in an isolated Docker container, leaving you free to break things without worrying about affecting other tests. The best part is that shrun
is built on Jest, so many of the standard Jest options work out of the box.
Benefits of shrun
- Enables you to run thousands of parallel tests, each in their own isolated sandbox.
- Makes it possible to test the end-to-end flow of your CLI, including installation and removal.
- Built on top of Jest and accepts many of the Jest CLI options.
- Simple test-spec format makes it a joy to write tests. Tests can even be used to drive the CLI product development process.
Examples and resources
A standalone example repo has been created to showcase shrun
. The repo contains a basic command line tool testshrun
and a few example specs:
https://github.com/rylandg/shrun-basic-demo
Here is the demo.yml
spec from that repo:
---- test: Simple passing case foreach: - PY_VERSION: python2 steps: - in: echo "Hello guys {PY_VERSION}" out: |- Hello guys {PY_VERSION}- test: Exit code failure (bad-path) steps: - in: Hello world err: |- bash: line 1: Hello: command not found exit: 127- test: Test help (good-path) steps: - in: testshrun help out: |- You will receive no help with testshrun- test: Test run (good-path) steps: - in: testshrun run out: |- Running sucks- test: Test no command (bad-path) steps: - in: testshrun err: |- Please provide a command to testshrun exit: 1- test: Test unknown command (bad-path) steps: - in: testshrun madeup err: |- madeup is not a recognized testshrun command exit: 1
Known issues and unfinished work
-
shrun
does not currently support a configuration file. This feature is planned and prioritized and the file will support many of the configuration options available to Jest. -
some CLI outputs still refer to Jest instead of
shrun
-
shrun
does not support test coverage as this is quite difficult to support. I would love to support this but realistically it might not happen.
Aknowledgements
shrun
is originally based off open source work I did for my previous employer Binaris. Here's a link to that original project (MIT).- My girlfriend (Jeanie) co-ported
shrun
with me
Contributors ✨
Thanks goes to these wonderful people (emoji key):
Ryland Goldstein 📆 ⚠️ 💻 |
jeanieholtz 💻 |
Heath Blandford 📖 |
This project follows the all-contributors specification. Contributions of any kind welcome!