perfjankie

Browser Performance regression suite

perfjankie

PerfJankie is a tool to monitor smoothness and responsiveness of websites and Cordova/Hybrid apps over time. It runs performance tests using browser-perf and saves the results in a CouchDB server. It also has a dashboard that displays graphs of the performance metrics collected over time that you help identify performance trends, or locate a single commit that can slow down a site.

After running the tests, navigate to the following url to see the results dashboard.

http://couchdb.serverl.url/databasename/_design/site/index.html

Here is a dashboard created from a sample project.

Why ?

Checking for performance regressions is hard. Though most modern browsers have excellent performance measurement tools, it is hard for a developer to check these tools for every commit. Just as unit tests check for regressions in functionality, perfjankie will help with checking regressions in browser rendering performance when integrated into systems like Travis or Jenkins.

The results dashboard

Setup

Perfjankie requires Selenium as the driver to run tests and CouchDB to store the results. Since this is based on browser-perf, look at setting up browser-perf for more information.

Usage

Perfjankie can be used as a node module, from the command line, or as a Grunt task and can be installed from npm using npm install perfjankie.

Node Module

The API call looks like the following

 
var perfjankie = require('perfjankie');
perfjankie({
  "url": "http://localhost:9000/testpage.html", // URL of the page that you would like to test. 
 
  /* The next set of values identify the test */
  name: "Component or Webpage Name", // A friendly name for the URL. This is shown as component name in the dashboard 
  suite: "optional suite name", // Displayed as the title in the dashboard. Only 1 suite name for all components 
  time: new Date().getTime(), // Used to sort the data when displaying graph. Can be the time when a commit was made 
  run: "commit#Hash", // A hash for the commit, displayed in the x-axis in the dashboard 
  repeat: 3, // Run the tests 3 times. Default is 1 time 
 
  /* Identifies where the data and the dashboard are saved */
  couch: {
    server: 'http://localhost:5984',
    database: 'performance',
    updateSite: !process.env.CI, // If true, updates the couchApp that shows the dashboard. Set to false in when running Continuous integration, run this the first time using command line.  
    onlyUpdateSite: false // No data to upload, just update the site. Recommended to do from dev box as couchDB instance may require special access to create views. 
  },
 
  callback: function(err, res) {
    // The callback function, err is falsy if all of the following happen 
    // 1. Browsers perf tests ran 
    // 2. Data has been saved in couchDB 
    // err is not falsy even if update site fails.  
  }
 
  /* OPTIONS PASSED TO BROWSER-PERF  */
  // Properties identifying the test environment */ 
  browsers: [{ // This can also be a ["chrome", "firefox"] or "chrome,firefox" 
    browserName: "chrome",
    version: 32,
    platform: "Windows 8.1"
  }], // See browser perf browser configuration for all options.  
 
  selenium: {
    hostname: "ondemand.saucelabs.com", // or localhost or hub.browserstack.com 
    port: 80,
  },
 
  BROWSERSTACK_USERNAME: process.env.BROWSERSTACK_USERNAME, // If using browserStack 
  BROWSERSTACK_KEY: process.env.BROWSERSTACK_KEY, // If using browserStack, this is automatically added to browsers object 
 
  SAUCE_USERNAME: process.env.SAUCE_USERNAME, // If using Saucelabs 
  SAUCE_ACCESSKEY: process.env.SAUCE_ACCESSKEY, // If using Saucelabs 
 
  /* A way to log the information - can be bunyan, or grunt logs. */
  log: { // Expects the following methods,   
    fatal: grunt.fail.fatal.bind(grunt.fail),
    error: grunt.fail.warn.bind(grunt.fail),
    warn: grunt.log.error.bind(grunt.log),
    info: grunt.log.ok.bind(grunt.log),
    debug: grunt.verbose.writeln.bind(grunt.verbose),
    trace: grunt.log.debug.bind(grunt.log)
  }
 
});
 

Note that most of these options are similar to the options passed to browser-perf. Refer to the browser-perf options for a mode detailed explanation.

Grunt Task

To run perfjankie as a Grunt task, simple load task using grunt.loadNpmTasks('perfjankie');, define a perfjankie task and pass in all the options from above as options to the Grunt task. Here is an example.

Command line

Run perfjankie --help to see a list of all the options. Quick Note - to only update site the first time, run the following from the command line.

$ perfjankie --config-file=local.config.json --only-update-site

The config file can contain server configuration and can look like this.

Hosting dashboard on a different server

You can also host the HTML/CSS/JS for displaying the results dashboard on not on CouchDB, but a different static server, possibly behind a CDN. In such cases,

1. Use the npm module and host the contents of the site folder.
2. Open index.html and insert the following snippet in the <head> section

<script type="text/javascript">window.DB_BASE="http://couchdb.server.url/databasename/_design";</script> 

This will ensure that all requests for data are made to the other CouchDB server. Also ensure that the CouchDB server has CORS turned on.

What does it measure?

Perfjankie measures page rendering times. It collects metrics like frame times, page load time, first paint time, scroll time, etc. It can be used on

• long, scrollable web pages (like a search result page, an article page, etc). The impact of changes to CSS, sticky headers and scrolling event handlers can be seen in the results.
• components (like bootstrap, jQuery UI components, ReactJS components, AngularJS components, etc). Component developers just have to place the component multiple times on a page and will know if they caused perf regressions as they continue developing the component. For more information, see the documentation for browser-perf