Mutant
Note: this is very much at the proof-of-concept stage
What is it?
mutant
is a mutation testing framework designed to help evaluate the quality of test suites for JavaScript applications.
What's the point?
Code coverage tools are useful to give an idea of how much of your code runs when your tests run, giving an overview of areas that may have been overlooked.
However, they do not provide a great deal of insight into how closely the code behaviour matches that which is delineated in the tests. A test simply triggering a code path is not enough to be confident the test adequately defines the required behaviour to an acceptable level of detail.
Mutation testing, on the other hand, is designed to test how far the code can stray from the original while the tests still pass. It does this by making incremental modifications to the code under test, and then repeatedly running the test suite against the new versions produced. If the tests pass for a mutated version, we may gain some insight into where there might be gaps in how the tests are designed.
Install
You can install mutant
globally, but the preferred way is to install it as a local dev-dependency:
npm i -D mutant-test
You can then use it via a script in package.json
:
Or access the local binary directly using
./node_modules/.bin/mutant /path/to/test/file.js
Usage
You can use mutant
alongside any test framework that can output results in TAP format.
Some examples for popular frameworks:
mocha --reporter tapava --taptaptape
Currently mutant
only supports running against a single test file at a time, so your test framework will also need to support passing the path to run a subset of tests. Watch this space for the ability to run full test suites.
Config
mutant
expects a config file to exist at .mutant/config.js
(might make this a little more flexible in the future). This is where you tell mutant
how to run your tests via the tests.run
property. Note, currently mutant
only supports running against a single test file at a time, and the tests.run
config property must contain a $FILE
placeholder where the filename will be passed into your test runner, e.g. for Mocha, your tests.run
config value might look like this:
tests:run: 'mocha --reporter tap $FILE'
Then when mutant
comes to run your tests it can simply replace the $FILE
placeholder with the path to your test file.
Mutation plugins
Mutations are provided to mutant
via plugins and can be selectively enabled / disabled via the config file. Babel is used to parse and traverse your code, as well as to generate the mutants based on the mutation plugins you select.