qf2e

QF, Front to End!

This is a work in progress on a lightweight end to end cross browser regression testing tool running on nodejs and selenium.

qf2e will run a set of scenarios (tests) on two separate domains, take screenshots at the end of each scenarios and compare the results. Diff images for comparison will be written down to disk and qf2e will output some TAP formated test output.

This tool is highly inspired by BBC's wraith and other similar tools, trying to have everything run in node (so you can write your scenarios in javascript).

Installation

First of all, you will need selenium, this might sound dauting but is actually easy.

You can either download their jar distribution and run it standalone, or if you are using OS X, install it with brew:

> brew install selenium-server-standalone
> brew install chromedriver

The chromedriver is optional, but useful if you want to test on chrome as selenium doesn't come with it by default (if you only want to test on firefox, you are good without it).

Next, you want to get qf2e:

npm install qf2e

Defining Scenarios

Scenarios can be specified in a configuration file called testsfile.js and require the definition of a few parameters.

Here is a basic example:

 
module.exports = {
  domains: {
    com: "https://www.google.com",
    fr: "https://www.google.fr"
  },
  paths: {
    home: "/",
    query: "/search?q=quantifind",
    more: {
      path: "/",
      behaviour: function(browser) {
        return browser
          .elementByCss('input[type=text]')
          .type("nodejs")
          .elementByCss("button[name='btnG']")
          .click()
          .sleep(1000);
      }
    }
  },
  outpath: "/tmp/qf2egoogle",
  threshold: 0,
  browser: 'chrome',
  selenium: {
    hostname: '127.0.0.1',
    port: 4444
  }
};
 

• First, we define two domains that we want to compare. In the example we want to compare google.com and google.fr, but you will probably want to compare your prod server with your localhost, or maybe two localhost servers running on different branches.

domains is an object that maps identifiers to their root URL. The identifiers can be anything you want and that will make sense to you in the test outputs.

• Second, you need to define a number of paths (this is a misnamer and will probably change soon). Each path is a "scenario" to test and qf2e will compare screenshots of each path on each domain that you specify.

paths is a map from an identifier (again, anything you want) to either: - a relative path from the domain root - a relative path + a behaviour function.

The behaviour function receive a browser instance from wd which hands the control over the selenium browser to you. You can do whatever you want to the page you are on (even navigate away) as long as you return the chained browser object. Qf2e will take a screenshot at that point.

If paths is an object, you can also specify the following path configuration:

• a sleepDuration period in milliseconds to wait before taking the screenshot. This is useful to wait for all animations to finish before taking the screenshot.
• some extra configuration for the screenshot comparison tool by adding a blinkConfig propery:

more: {
     path: "/",
      sleepDuration: 3000,
      blinkConfig: {
      blockOut: [
         {x: 0, y: 0, width: 1700, height: 400}
       ]
      }
}

See yahoo/BlinkDiff Object Usage documentation for available options.

• outpath specifies where you want the screenshots and diff images to be stored on disk
• threshold decides of when a test fails on passes. It is in percent of pixel changes in the page (note that a big change in a page might translate to a small number of pixel chantes)
• browser is the browser on which you want to run the scenarios. You need to have a driver installed with Selenium for the specified browser. For Firefox, you get it by default; I believe that IE comes by default too. For Chrome, you will have to install chromedriver.
• selenium specifies the connection parameters for your selenium instance. The example settings should be fine if you are running the standalone server on your computer.
• defaultBehaviour specify a function f(browser) that returns a browser and is applied before all tests that have no behaviour defined.

Output

QF2E will output test results following the TAP format:

± % qf2e --tests ~/dev/qf2e/testsfile.js
#running tests from ~/qf2e/testsfile.js
# will output screenshots to /tmp/qf2egoogle
#done taking screenshots
1..3
not ok 1 com vs fr - home: 3.7% difference (/tmp/qf2egoogle/home-diff.png)
not ok 2 com vs fr - query: 23% difference (/tmp/qf2egoogle/query-diff.png)
not ok 3 com vs fr - more: 15% difference (/tmp/qf2egoogle/more-diff.png)

You might want to use a tap formater to have a nicer experience. Or if you are running this on a CI server, install a TAP plugin for that server.

QF2E will also output a set of screenshot and "diff" images to the specified outpath. For instance, for the example testsfile.js provided in this repo, you would get diff images like this one:

Contributing

QF2E is released under the MIT License and we welcome any contributions within this license. Any pull request is welcome and will be reviewed and merged as quickly as possible.

Because this open source project is released by Quantifind as a company, if you want to submit a pull request, you will have to sign the following simple contributors agreement:

• If you are an individual, please sign this contributors agreement and send it back to [e-mail]
• If you are contributing changes that you did as part of your work, please sign this contributors agreement and send it back to [e-mail]