node package manager
Orgs are free. Discover, share, and reuse code in your team. Create a free org »

saucelabs-browsertime

Saucelabs Browsertime Build Status

Using Webdriver to collect Navigation Timing.

Travis job runs / Build #51 / JSON output

Usage: sauce-browsertime [options] [urls, ...]
 
    $ sauce-browsertime http://example.com -b firefox -n 3 --assert-pageload-max=5000 --assert-frontend-p90=2500
 
See https://saucelabs.com/platforms for the list of available OS / Browser / Version
 
Options:
 
  -b, --browser          - Saucelabs browser (default: chrome)
  -p, --platform         - Saucelabs platform (default: unspecified)
  -t, --type             - Saucelabs device type (default: unspecified)
  -o, --orientation      - Saucelabs device orientation (default: unspecified)
  -v, --version          - Saucelabs browser version (default: unspecified)
  -n, --runs             - Number of runs per URL (default: 1)
  -R, --reporter         - Mocha reporter (default: json)
  -H, --hostname         - Webdriver-grid hostname (default: ondemand.saucelabs.com)
  --port                 - Specify webdriver-grid port (default: 80)
  -h, --help
 
Asserts:
 
  --assert-name-stat=[value]   - Specify a test assertion on [name] metric
  --assert-name=value          - Specify a test assertion on [name] average value
 
Available stats: min, max, avg, media, mad, p60, p70, p80, p90

Every webdriver enabled browser on Saucelabs, implementing Navigation Timing API should be supported.

Install

$ npm install saucelabs-browsertime -g

# local
$ npm install saucelabs-browsertime
node_modules/.bin/sauce-browsertime -h

Asserts

--assert-* params sets up test assertion on a specific interval and stat.

Stats are based on https://github.com/tmcw/simple-statistics. Use the -n, --runs option to set the number of pageload per URL.

$ node bin/sauce-browsertime -h --assert-help
 
Asserts:
 
  --assert-name-stat=[value]   - Specify a test assertion on [name] metric
  --assert-name=value          - Specify a test assertion on [name] average valu
e
 
 
Metrics:
 
  --assert-domainLookup
  --assert-redirection
  --assert-serverConnection
  --assert-serverResponse
  --assert-pageDownload
  --assert-domInteractive
  --assert-domContentLoaded
  --assert-pageLoad
  --assert-frontEnd
  --assert-backEnd
 
Available stats: min, max, avg, media, mad, p60, p70, p80, p90

Example on sauceio.com, running 10 times, with assertion on pageLoad, backandTime and frontendTime.

$ sauce-browsertime sauceio.com -n 10 \
  --assert-pageLoadTime-max=5000 \
  --assert-pageLoadTime-avg=2000 \
  --assert-backendTime-max=800 \
  --assert-backendTime-p90=700 \
  --assert-frontend=100 \
  --reporter spec
 
Collecting Navigation Timings with chrome
  sauceio.com
    √ sauceio.com #1
    √ sauceio.com #2
    √ sauceio.com #3
    √ sauceio.com #4
    √ sauceio.com #5
    √ sauceio.com #6
    √ sauceio.com #7
    √ sauceio.com #8
    √ sauceio.com #9
    √ sauceio.com #10
 
Asserts - sauceio.com
  √ Assert pageload max <= 5000 (Value: 285)
  √ Assert pageload avg <= 2000 (Value: 96.8)
  √ Assert backend max <= 800 (Value: 29)
  √ Assert backend p90 <= 700 (Value: 16.5)
  √ Assert frontend avg <= 100 (Value: 83.2)
 
  15 passing (23s)

Reporters

Mocha reporters should be usable for the most part

$ mocha --reporters

    dot - dot matrix
    doc - html documentation
    spec - hierarchical spec list
    json - single json object
    progress - progress bar
    list - spec-style listing
    tap - test-anything-protocol
    landing - unicode landing strip
    xunit - xunit reporter
    html-cov - HTML test coverage
    json-cov - JSON test coverage
    min - minimal reporter (great with --watch)
    json-stream - newline delimited json events
    markdown - markdown documentation (github flavour)
    nyan - nyan cat!

few examples

JSON (default)

$ sauce-browsertime http://example.com -n 3 --reporter json
{
  "stats": {
    "suites": 2,
    "tests": 3,
    "passes": 3,
    "pending": 0,
    "failures": 0,
    "start": "2014-04-30T21:28:43.124Z",
    "timings": {
      "http://example.com": {
        "domainLookupTime": {
          "min": 0,
          "max": 0,
          "avg": 0,
          "median": 0,
          "mad": 0,
          "p60": 0,
          "p70": 0,
          "p80": 0,
          "p90": 0
        },
        "redirectionTime": {
          "min": 0,
          "max": 0,
          "avg": 0,
          "median": 0,
          "mad": 0,
          "p60": 0,
          "p70": 0,
          "p80": 0,
          "p90": 0
        },
        "serverConnectionTime": {
          "min": 0,
          "max": 0,
          "avg": 0,
          "median": 0,
          "mad": 0,
          "p60": 0,
          "p70": 0,
          "p80": 0,
          "p90": 0
        },
        "serverResponseTime": {
          "min": 2,
          "max": 6,
          "avg": 4,
          "median": 4,
          "mad": 2,
          "p60": 4,
          "p70": 6,
          "p80": 6,
          "p90": 6
        },
        "pageDownloadTime": {
          "min": 0,
          "max": 1,
          "avg": 0.77777777777777777,
          "median": 1,
          "mad": 0,
          "p60": 1,
          "p70": 1,
          "p80": 1,
          "p90": 1
        },
        "domInteractiveTime": {
          "min": 17,
          "max": 46,
          "avg": 29.333333333333332,
          "median": 25,
          "mad": 8,
          "p60": 25,
          "p70": 46,
          "p80": 46,
          "p90": 46
        },
        "domContentLoadedTime": {
          "min": 17,
          "max": 46,
          "avg": 29.333333333333332,
          "median": 25,
          "mad": 8,
          "p60": 25,
          "p70": 46,
          "p80": 46,
          "p90": 46
        },
        "pageLoadTime": {
          "min": 19,
          "max": 46,
          "avg": 30,
          "median": 25,
          "mad": 6,
          "p60": 25,
          "p70": 46,
          "p80": 46,
          "p90": 46
        },
        "frontEndTime": {
          "min": 11,
          "max": 20,
          "avg": 14.333333333333334,
          "median": 12,
          "mad": 1,
          "p60": 12,
          "p70": 20,
          "p80": 20,
          "p90": 20
        },
        "backEndTime": {
          "min": 4,
          "max": 33,
          "avg": 15,
          "median": 8,
          "mad": 4,
          "p60": 8,
          "p70": 33,
          "p80": 33,
          "p90": 33
        }
      }
    },
    "caps": {
      "rotatable": false,
      "browserConnectionEnabled": false,
      "acceptSslCerts": false,
      "cssSelectorsEnabled": true,
      "javascriptEnabled": true,
      "databaseEnabled": false,
      "chrome.chromedriverVersion": "26.0.1383.0",
      "locationContextEnabled": false,
      "takesScreenshot": true,
      "platform": "linux",
      "browserName": "chrome",
      "version": "28.0.1500.95",
      "hasMetadata": true,
      "nativeEvents": true,
      "applicationCacheEnabled": false,
      "webStorageEnabled": true,
      "handlesAlerts": true
    },
    "end": "2014-04-30T21:28:52.644Z",
    "duration": 9520
  },
  "tests": [ ... includes raw timings here as duration props ... ],
  "passes": [ ... same as above ... ]

Spec

$ sauce-browsertime http://example.com -n 3 --reporter spec

Collecting Navigation Timings with chrome
  http://example.com
    √ http://example.com #1
    √ http://example.com #2
    √ http://example.com #3


  3 passing (11s)

Nyan!

$ sauce-browsertime http://example.com http://example.com/page-one http://example.com/page-two -n 4 --reporter nyan

 12  -_-_-_-_-_-_-_,------,
 0   -_-_-_-_-_-_-_|   /\_/\
 0   -_-_-_-_-_-_-^|__( ^ .^)
     -_-_-_-_-_-_-  ""  ""

  12 passing (23s)

HTML templates

Not a a mocha reporter per say. Transforms the raw JSON data of a previous run into an HTML page, with visual representation of the Timing objects thanks to http://kaaes.github.io/timing/

Usage: sauce-browsertime-html [options] file.json

    $ sauce-browsertime results.json > index.html

Graphs generated thanks to http://kaaes.github.io/timing/
Browser logos thanks to https://github.com/alrra/browser-logos

Options:

  -b, --browser          - Browser under test (used to match browser logo)

Jenkins template

config.xml file is a Jenkins Job template that can be used to quickly setup a Job to run the perf test, at a fixed interval or on SCM change:

  • Includes Job parameters for:

    • PERF_URLS: List of URLs to test. Whitespace separated.
    • SAUCE_USERNAME: Saucelabs username
    • SAUCE_ACCESS_KEY: Saucelabs API key
    • SAUCE_BROWSERS: Comma separated list of browsers.
  • Shell scripts

    • Install npm packages on first run (npm install saucelabs-browsertime)
    • A run for each browser, stats & raw metrics available at ./results/BUILD_NUMBER/BROWSER/metrics.json, HTML report at ./results/BUILD_NUMBER/BROWSER/metrics.json.
  • Plugin configuration for htmlpublisher (optional to have it installed, jenkins will just ignore the conf)

See config.xml file for further detail.

# Job creation
$ curl -X POST -H 'Content-Type: application/xml' $JEKINS_URL/pluginManager/installNecessaryPlugins --data-binary @config.xml

# Check plugins
$ curl -X POST -H 'Content-Type: application/xml' $JEKINS_URL/pluginManager/prevalidateConfig --data-binary @config.xml

# Install plugins
$ curl -X POST -H 'Content-Type: application/xml' $JEKINS_URL/pluginManager/installNecessaryPlugins --data-binary @config.xml

TODOs

  • [] Statsd / Graphite reporter
  • Assert system (like phantomas)
  • HTML reporter with http://kaaes.github.io/timing/ widget
  • Option for the number of runs, and stats (avg, median, percentiles)