api-bench-runner
This is a CLI tool which can be used to run api-benchmark tests. Its interface is inspired by mocha.
Usage
The idea behind this tool is to allow you to define tests in one or across multiple scripts using easy to use hooks much like mocha
.
Example of a Test file
'use strict';
suite('Status Route', () => {
service('my-service', 'http://localhost:8080');
suite('Multiple requests in Parallel', () => {
options({
runMode: 'parallel',
minSamples: 200,
maxTime: 20,
});
route('status', {
method: 'get',
route: 'status',
expectedStatusCode: 200,
maxMean: 0.2, // 200ms
});
});
suite('Multiple requests in Sequence', () => {
options({
runMode: 'sequence',
minSamples: 200,
maxTime: 20,
});
route('status', {
method: 'get',
route: 'status',
expectedStatusCode: 200,
maxMean: 0.2, // 200ms
});
});
});
In the above example we use the suite
hook to define a test suite. This suite will result in a single api-benchmark
test run. The suite
hook takes a name (just for the output) and a callback function that will be executed to configure the suite. The callback will include the other hooks which lets you define the service, routes, and options.
You can have multiple test files which api-bench-runner
will aggregate together allowing you to keep you test code clear and module.
By convention api-bench-runner
will look for tests under the bench
folder.
├── bench
│ ├── nested-test-example.js
│ └── simple-test-example.js
└── lib
└── server.js
Example of Command
> node_modules/.bin/api-bench-runner .
This command will look for all the tests under the bench
folder and run them displaying the results in the terminal.
You can redirect api-bench-runner
to a different location if you choose not to put your tests under a bench
folder.
> node_modules/.bin/api-bench-runner ./benchmark
You can use glob patterns to find test files as well
> node_modules/.bin/api-bench-runner ./test/**/*.bench.js
Output
The output of the tool will be a print out to the terminal showing the test results. When all tests pass (or you did not define restrictions which would cause failures) you will see each test (grouped by test suite) listing info such as operations per second and a summary saying everything passed.
If however you do have restrictions and the route did not meet them or something else went wrong you will see that test run listed with an error and a summary of all test failures printed out at the bottom.
Reporters
You can specify different reporters for api-bench-runner
to use depending on what kind of output you need. To pick a reporter use the --reporter
flag.
Terminal
By default it will print to the terminal but you can make this explicit by setting the --reporter
flag to stdterm
or default
> node_modules/.bin/api-bench-runner ./test/**/*.bench.js --reporter default
HTML
You can have api-bench-runner
generate an HTML report. The report generated is the report generated by api-benchmark. To pick the HTML reporter set the --reporter
flag to html
.
> node_modules/.bin/api-bench-runner ./test/**/*.bench.js --reporter html
By default the HTML report will be generated in the working directory that api-bench-runner
is in and will save it to the file benchmarks.js
. To specify a different file name you can use the --out
flag.
> node_modules/.bin/api-bench-runner ./test/**/*.bench.js --reporter html --out my-results.html
Note The --out
flag has no effect on stdterm
reporter and right now is only used by the html
reporter.
Multiple Reporters
If you would like to see the results in the terminal and get a HTML report generated you can specify both reporters via the --reporter
flag.
> node_modules/.bin/api-bench-runner ./test/**/*.bench.js --reporter stdterm,html
Hooks
Define a Suite
You can define a test suite via the suite
hook.
A suite equates to a single api-benchmark
test run. The suite
hook takes two parameters: a title (just for the output) and a callback function for configuring the suite. The callback will include all the other hooks which lets you configure this suite's services, routes, and options.
suite('My suite that tests the status route', () => {
// Configure the suite here
});
You can nest suites inside each other for cases where you have shared configurations.
suite('My suite that tests my-service', () => {
service('my-service', 'http://localhost:8080'); // All child suites use this same service.
suite('My suite that tests the status route', () => {
options(...);
route(...);
});
suite('My suite that tests the version route', () => {
options(...);
route(...);
});
});
Both the version and status suites since they don't define a service of their own will check there parent to find out what service to run the tests against. Nesting suites allows for sharable configurations.
Defining the Service
The tests defined in the suite need to run against a running HTTP server, you can tell the suite where to find the server via the service
hook.
suite('My suite that tests my-service', () => {
service('my-service', 'http://localhost:8080');
options(...);
route(...);
});
The service
hook takes two parameters: a name (must be a valid ECMAScript object property name) and the URL to the service to run the tests against. The URL is to be the base URL which we can extend with the route paths.
If your hitting a development lap or production environment or a local environment you've manually started up then you can just put in the URL as a string as seen above; however, if the site is dynamic and needs to be brought up when the tests starts and thus might have a dynamic URL (random port?) you can pass in a callback function which will be called by the suite when it starts executing.
function findService() {...}
suite('My suite that tests my-service', () => {
service('my-service', findService);
options(...);
route(...);
});
The function you pass in must return a string consisting of the service URL.
Define Suite Options
Each suite can be configured with options defining how the test should be run. This includes how many samples, if the tests should be parallel or sequential, and the maximum number of concurrent requests if tests are to be done in parallel.
You can define these options via the options
hook. This hook is optional however as each option has a default value.
suite('Multiple requests in Sequence', () => {
options({
runMode: 'sequence',
minSamples: 200,
maxTime: 20,
});
route(...);
});
These options can also be nested in a parent suite if they are common.
suite('My suite that tests my-service', () => {
// All child suites use this same service and test options.
service('my-service', 'http://localhost:8080');
options({
runMode: 'sequence',
minSamples: 200,
maxTime: 20,
});
suite('My suite that tests the status route', () => {
route(...);
});
suite('My suite that tests the version route', () => {
route(...);
});
});
These options are the api-benchmark
options, for a list of valid options see the api-benchmark
documentation
Defining a Route
To define a given test for the suite you need to define the route to test. This can be done via the route
hook.
The route
hook takes two parameters: A name (must be a valid ECMAScript object property name) and the relative route path.
suite('My suite that tests the status route', () => {
service('my-service', 'http://localhost:8080');
route('status', 'api/status');
});
This will tell api-benchmark
to run the tests per the suite options against the route api/status
attached to the service (i.e. http//localhost:8080/api/status
).
By default this will be a GET
and will always pass displaying the operation per second results; however, if you want to benchmark a different HTTP verb (i.e. POST
, PUT
, DELETE
) or want to define failure conditions (must be faster then Xms, must return with 200) you can pass in a route options object instead of the route string.
suite('My suite that tests the status route', () => {
route('status', {
method: 'get',
route: 'api/status',
expectedStatusCode: 200,
maxMean: 0.2, // 200ms
});
});
These options are the api-benchmark
route options, for a list of valid options see the api-benchmark
documentation
Setting up a Suite
When running the tests you might need to setup the environment first. You might need to insert entries into the database if testing a DELETE
route or create users to log in as, you might even just simply need to bring up the server if testing locally. Like mocha
these setup/tear down steps can be handled via before
and after
hooks.
suite('Multiple requests in Sequence', () => {
before(() => {
// do Setup
});
after(() => {
// do tear down
});
service('my-service', 'http://localhost:8080');
options(...);
route(...);
});
A suite can also have multiple before
/after
hooks which will be executed in the order that they are defined in the suite.
suite('Multiple requests in Sequence', () => {
before(() => {...}); // runs first
before(() => {...}); // runs second
before(() => {...}); // runs last
after(() => {...}); // runs first
after(() => {...}); // runs last
service('my-service', 'http://localhost:8080');
options(...);
route(...);
});
These before
and after
hooks can be on each suite when suits are nested. These hooks will only be run once and will be run in descending order (parent suite hooks are run first followed by children).
suite('Status Route', () => {
service('my-service', 'http://localhost:8080');
before(() => {...});
after(() => {...});
suite('Multiple requests in Parallel', () => {
before(() => {...});
after(() => {...});
options(...);
route(...);
});
suite('Multiple requests in Sequence', () => {
before(() => {...});
after(() => {...});
options(...);
route(...);
});
});
In the above when the parallel suite begins first it will trigger its parents before
hooks then its own followed by the test then its after
hooks. Next the sequential test will start, as its parents before
hook have already run they will be skipped so it then runs its before
hooks followed by the test and ending with its after
hooks. Since there are no more sibling suites the parent suite's after
hooks will now be triggered. This is not unlike mocha
.
Async Setup
The before
and after
hooks take only one parameter and that is the callback function to execute when the hook is triggered. This callback function is run synchronously; however, if it returns a Promise
the execution will wait until the promise is resolved. Alternatively the passed in function can accept a done
callback so that it can tell the before
/after
hook when its finished.
suite('Multiple requests in Sequence', () => {
let server;
before((done) => {
server = require(`server`);
server.once('listening', () => {
done();
});
});
after((done) => {
server.close((err) => {
if (err) {
console.log('Error shutting down server');
} else {
console.log('Server shutdown');
}
done();
});
});
service('my-service', () => server.url);
options(...);
route(...);
});
Global Setup
If you have tests across a number of files having each file bring up and tear down your service is going to be a pain. If you need to bring up your service once at the beginning and only tear it down at the very end of the test run you can define a global setup script.
By convention api-bench-runner
will look for a special script called setup.js
which contains a single suite
with before
and after
hooks that it will use as the grandparent suite for all the files. That means that any before
hooks defined in this file will be triggered once at the start of the entire test run and any after
hooks will be triggered right before the tool finishes allowing for a one time global setup and tear down steps.
The setup.js
file is expected to live under the bench
folder of the working directory which the api-bench-runner
tool is running.
├── bench
│ ├── nested-test-example.js
│ ├── setup.js
│ └── simple-test-example.js
└── lib
└── server.js
If you run api-bench-runner
in the project root it will looks for bench/setup.js
and add it as the root suite for all the defined tests. If it is not found then no global setup/tear down hooks will be setup.
You can however tell api-bench-runner
where to local the global setup script via the --setup
flag if you choose to have a different folder structure.
├── test
│ ├── nested-test-example.bench.js
│ └── simple-test-example.bench.js
├── setup.js
└── lib
└── server.js
Given the above structure you can execute the following command to find all the right scripts:
> node_modules/.bin/api-bench-runner ./test/*.bench.js --setup ./setup.js
Troubleshooting
If your tests are not running correctly and you suspect there is something wrong with the hooks (did the service come up?) or otherwise just want to see more info on what is going on there are two options:
Turn on API-Benchmark Debug mode
Each route
can be run in debug mode which simply displays more info to the terminal. If everything is working you just get pretty much the same output as what api-bench-runner
generates but on error you'll get more info.
suite('My suite that tests the status route', () => {
route('status', {
debug: true,
method: 'get',
route: 'api/status',
expectedStatusCode: 200,
maxMean: 0.2, // 200ms
});
});
Turn on logging
The api-bench-runner
tool has logging although it does not log anything by default. The log messages are printed to the terminal so by default we log nothing to keep the terminal output clean. To see additional logging you can set the log level to debug
or tace
via the --level
flag.
> node_modules/.bin/api-bench-runner --level debug .
This will print out information about each stage of the tool; that is when it starts executing hooks or starts the test runs.
Development
Running Tests
To run the tests for api-bench-runner
use the npm test
command.
Make sure to run npm install
first.
You can have the tests watch for changes and re-run automatically by using the command npm run watch
.
Style Guide
The Airbnb JavaScript Style Guide is used for this project so you must comply to that rule set. You can verify your changes are in compliance via the npm run lint
command.
This project does have a few alterations to the rules such as requiring the use of the use strict
pragma as we don't use babel
. The npm run lint
command enforces the alterations. To see all alterations look in the .eslintrc.json
files.
Contributing
Bug reports and pull requests are welcome. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the Contributor Covenant code of conduct.
License
The tool is available as open source under the terms of the MIT License.