japp

JAPP - The Image Comparer

JAPP is a node module used for visual regression testing of FusionCharts. It can take a snapshot of a webpage using phantomJS, and can compare it to a reference image using Resemble.js. Additionally, it can also build the image collection from the spec files and even convert files containing chart data json into spec files which can be used for testing.

Getting Started

npm install japp

OR

add japp as a dependency in the package.json of your module and run npm install

You are now ready to use JAPP.

Usage

Building the reference images repository

Building the repository of reference images is a pre-requisite for comparing generated snapshots.

The command-line option --save should be used when the snapshots generated by the specs are to be saved to be used as reference images during subsequent regression testing.

The path to the folder where the images should be saved can be provided in the japp_config.json or the command line option --savePath [path/to/folder]

Additionally, while saving the snapshots, JAPP can also overwrite the existing reference image paths provided in the specs using the --overwriteRef command line option. The new reference paths that are injected into the specs are w.r.t the reference image folder provided in the japp_config.json (refImagesRoot) or as a command line option. The new set of images that are generated should be simply be moved into this reference images folder from the saved images folder and you are now ready to carry out the regression tests.

Comparing snapshots and reference images

Once the specs and the reference images have been properly linked, the execution of the specs for the purpose of regression testing can be carried out. The js source folder that contains the FusionCharts source (map files in a separate folder named maps inside the source folder) needs to be specified either in the japp_config.json or as a command line argument (--jsSourceRoot). Similarly the config option specSource is needed to indicate the folder where all the spec files are stored.

if config is not stored in the root of the module

japp --config [path/to/config]

if the details of the test execution is needed

japp --debugMode

Most of the configurations provided in the japp_config.json can be overriden via a homonymous command line argument.

if the threshold for image comparison needs to be overridden

japp --threshold

Storing the snapshots

Usually the generated snapshots are deleted after the comparison is done, but in order to save the snapshots for reviewing later create a folder called snapshots in the root of the project. The snapshots folder can be configured in the japp_config.json (snapshotPath). If JAPP finds that the snapshot folder exists, it saves all the snapshots in this folder and additionally creates a japp_mismatch.html & japp_match.html containing all the mismatched images and the matched images respectively.

Possible outcomes for each spec

1. Snapshot generated matches with the refernce image
2. Snapshot generated does not match with the reference image
3. Reference image is missing
4. Snapshot not generated

Exit codes

0 - All specs matched the reference images
1 - Spec(s) matching failed
2 - Reference image(s) either not provided or missing
3 - 1 & 2
4 - Spec(s) could not generate snapshots
5 - 4 & 1
6 - 4 & 2
7 - 4 & 2 & 1
8 - Path provided to save the images does not exist
9 - Path provided to source the specs does not exist
10 - Path provided to fetch the js source does not exist

The results for all the specs that get executed is stored in the JSON format and is written to the file japp_result.json once all the specs have been executed.

The file japp_log.txt also contains a log of the actions that take place as the execution of the image comparison takes place.

Configuration

• savePath: The directory in which to save the images generated from the specs. This option is to be used in combination with --save.
• specSource: The path from the project root, to the folder containing the specs to be executed.
• jsSourceRoot: The path from the project root, to the folder containing the FusionCharts javascript files.
• threshold: The mismatch threshold for resemble. If the mismatch is more than this threshold the test will fail.
• maxThreads: Maximum number of threads that can be spawned to execute the test cases.
• dataSourceFolder: The path from the project root, to the folder containing the data files to be converted to JAPP specs. This option is to used when --generateSpecs option is provided in the command line.

Spec format

{
    test_name: {
        description: "This is a test spec",
        options: {
            // object passed to the FusionCharts constructor
        },
        refimage: "path/to/the/reference_image.png"
    },
    test_name_2: {
        // ...
    }
}

Known Issues:

There seem to be differences detected in images if the number of threads involved while generating the images is different from the number of threads used while comparing.