Visual regression testing framework, comparing screenshots from multiple builds.
Table of Contents
- Getting Started
- Project Naming
- Third-party libraries
Install this module with the following command:
npm install kobold
Install this module globally with the following command:
npm install -g kobold
This will make sure that you don't have to enter the whole path when running Kobold.
Add the module to your
npm install --save kobold
Add the module to your
npm install --save-dev kobold
Kobold can be executed by running
kobold script in the
bin directory of the package. When you installed the module globally, then you can us only
kobold to run it.
Only one parameter is required to run the tests:
With this parameter, Kobold starts running tests on the
test/ui/regression directory. It analyzes what screens were already approved, what screens are available in the current build, and what screens it has to compare. Then, it creates the test-code and executes the image-comparison, creating the test results in the
Instead of implicitly using the last argument for the test-path, you can also be more specific and supply the
kobold --test-path test/ui/regression
By default, Kobold will look for four directories:
approved- All approved screens should be available in this folder. These are the "golden standard" images that will be used to compare them to build screens.
build- Screens that were just taken are saved in this folder. This is most likely done by some Selenium tests, taking screenshots and saving them in this directory. These images will be compared to the images in the
approvedfolder, creating comparison result images in
highlight- Image comparison results of the latest test-run are saved in this folder. These images will highlight the differences between the
approved-image and the
config- Custom configuration files (.js or .json) for image specific comparison options. Sometimes, it is helpful to tune the comparison of a specific image. These options overwrite globally set comparison options. They are also optional. See the Blink-Diff Project WebSite for more information on these options.
The folder names for
config can be changed by supplying the parameters
kobold --approved-folder "golden" --highlight-folder "differences" /test/ui/regression
This test-run will look for
differences for the
highlight folder respectively instead of using the default folder names. In this specific case,
build was not changed.
Sometimes, you want the tests to fail when previously approved screens are missing from the
build directory. This can be the case when for example the tests that created these screenshots didn't fully run. Use the
--fail-orphans flag to make these tests fail:
kobold --fail-orphans test/ui/regression
--fail-additions flag will fail tests for screens that are new, screens that were never approved before.
kobold --fail-orphans --fail-additions test/ui/regression
Kobold also supports all Mocha parameters which can be interspersed with the Kobold parameters:
kobold --slow 3000 --fail-orphans --fail-additions --reporter dot --test-path test/ui/regression
Since Kobold is built on top of Mocha, it supports all of its parameters in addition to the following:
approved-folder- Name of the approved folder (default: 'approved')
build-folder- Name of the build folder (default: 'build')
highlight-folder- Name of the highlight folder (default: 'highlight')
config-folder- Name of the config folder (default: 'config')
fail-orphans- Flag that determines that tests should fail when screens that were previously approved, being in the
approvefolder, are not found in the
fail-additions- Flag that determines that tests should fail when unapproved screens are found in the
buildfolder, meaning screens that are missing from the
test-path- Path to directory with all test related folders as mentioned above (required). Uses implicitly the last parameter element when none is given.
config- Path to a config file (.js or .json) that is used to overwrite comparison options. (default: none) See the Blink-Diff Project WebSite for more information on these options.
The global configuration file that can be selected by supplying
--config to the command-line describes default-values for the comparison.
Here is an example of the file (
config.js - can also be a JSON file):
moduleexports ="delta": 35 // Changing the distance of pixel-comparison;
This file then can be called as follows:
kobold --config config.js test/ui/regression
Configuration files for specific screens on the other hand describe only the comparison options for that screen.
Here is an example for a screen that is named "YDN_Missing"; the path is
config/YDN_Missing.json (could also be a
"outputBackgroundOpacity": 03"outputBackgroundGreen": 100
This example will make the background for the comparison image look greener, and lightens-up the whole comparison by reducing the opacity of the background-mask.
Kobold creates a regular test report as if the tests were manually written, and it also creates the following images, comparing the approved (left) with the build screenshot (right), pointing out the differences (middle).
These examples can be found in the
examples folder. The above results were produced by running the following command in the project root with a globally installed Kobold:
Generate the API-documentation with following command:
npm run docs
This will generate the source-code documentation in the
docs folder of the module root.
Run the tests with the following command:
npm run test
The code-coverage will be written to the
coverage folder in the module root.
A Kobold is a gnome that can make itself invisible and is often the source of nuisances - so, pretty much like visual regressions.
The following third-party libraries are used by this module:
- blink-diff: https://github.com/yahoo/blink-diff
- preceptor-core: https://github.com/yahoo/preceptor-core
- kobold-core: https://github.com/yahoo/kobold-core
- log4js: https://github.com/nomiddlename/log4js-node
- promise: https://github.com/then/promise
- mocha: https://github.com/visionmedia/mocha
- chai: http://chaijs.com
- istanbul: https://github.com/gotwarlost/istanbul
- yuidocjs: https://github.com/yui/yuidoc
The MIT License
Copyright 2014 Yahoo Inc.