Saucelabs Browsertime 
Using Webdriver to collect Navigation Timing.
Travis job runs / Build #51 / JSON output
Usage: sauce-browsertime [options] [urls, ...]
$ sauce-browsertime http://example.com -b firefox -n 3 --assert-pageload-max=5000 --assert-frontend-p90=2500
See https://saucelabs.com/platforms for the list of available OS / Browser / Version
Options:
-b, --browser - Saucelabs browser (default: chrome)
-p, --platform - Saucelabs platform (default: unspecified)
-t, --type - Saucelabs device type (default: unspecified)
-o, --orientation - Saucelabs device orientation (default: unspecified)
-v, --version - Saucelabs browser version (default: unspecified)
-n, --runs - Number of runs per URL (default: 1)
-R, --reporter - Mocha reporter (default: json)
-H, --hostname - Webdriver-grid hostname (default: ondemand.saucelabs.com)
--port - Specify webdriver-grid port (default: 80)
-h, --help
Asserts:
--assert-name-stat=[value] - Specify a test assertion on [name] metric
--assert-name=value - Specify a test assertion on [name] average value
Available stats: min, max, avg, media, mad, p60, p70, p80, p90
Every webdriver enabled browser on Saucelabs, implementing Navigation Timing API should be supported.
Install
$ npm install saucelabs-browsertime -g
# local
$ npm install saucelabs-browsertime
node_modules/.bin/sauce-browsertime -h
Asserts
--assert-* params sets up test assertion on a specific interval and stat.
Stats are based on https://github.com/tmcw/simple-statistics. Use the -n, --runs option to set the number of pageload per URL.
$ node bin/sauce-browsertime -h --assert-help
Asserts:
--assert-name-stat=[value] - Specify a test assertion on [name] metric
--assert-name=value - Specify a test assertion on [name] average valu
e
Metrics:
--assert-domainLookup
--assert-redirection
--assert-serverConnection
--assert-serverResponse
--assert-pageDownload
--assert-domInteractive
--assert-domContentLoaded
--assert-pageLoad
--assert-frontEnd
--assert-backEnd
Available stats: min, max, avg, media, mad, p60, p70, p80, p90
Example on sauceio.com, running 10 times, with assertion on pageLoad, backandTime and frontendTime.
$ sauce-browsertime sauceio.com -n 10 \
--assert-pageLoadTime-max=5000 \
--assert-pageLoadTime-avg=2000 \
--assert-backendTime-max=800 \
--assert-backendTime-p90=700 \
--assert-frontend=100 \
--reporter spec
Collecting Navigation Timings with chrome
sauceio.com
√ sauceio.com #1
√ sauceio.com #2
√ sauceio.com #3
√ sauceio.com #4
√ sauceio.com #5
√ sauceio.com #6
√ sauceio.com #7
√ sauceio.com #8
√ sauceio.com #9
√ sauceio.com #10
Asserts - sauceio.com
√ Assert pageload max <= 5000 (Value: 285)
√ Assert pageload avg <= 2000 (Value: 96.8)
√ Assert backend max <= 800 (Value: 29)
√ Assert backend p90 <= 700 (Value: 16.5)
√ Assert frontend avg <= 100 (Value: 83.2)
15 passing (23s)
Reporters
Mocha reporters should be usable for the most part
$ mocha --reporters
dot - dot matrix
doc - html documentation
spec - hierarchical spec list
json - single json object
progress - progress bar
list - spec-style listing
tap - test-anything-protocol
landing - unicode landing strip
xunit - xunit reporter
html-cov - HTML test coverage
json-cov - JSON test coverage
min - minimal reporter (great with --watch)
json-stream - newline delimited json events
markdown - markdown documentation (github flavour)
nyan - nyan cat!
few examples
JSON (default)
$ sauce-browsertime http://example.com -n 3 --reporter json
{
"stats": {
"suites": 2,
"tests": 3,
"passes": 3,
"pending": 0,
"failures": 0,
"start": "2014-04-30T21:28:43.124Z",
"timings": {
"http://example.com": {
"domainLookupTime": {
"min": 0,
"max": 0,
"avg": 0,
"median": 0,
"mad": 0,
"p60": 0,
"p70": 0,
"p80": 0,
"p90": 0
},
"redirectionTime": {
"min": 0,
"max": 0,
"avg": 0,
"median": 0,
"mad": 0,
"p60": 0,
"p70": 0,
"p80": 0,
"p90": 0
},
"serverConnectionTime": {
"min": 0,
"max": 0,
"avg": 0,
"median": 0,
"mad": 0,
"p60": 0,
"p70": 0,
"p80": 0,
"p90": 0
},
"serverResponseTime": {
"min": 2,
"max": 6,
"avg": 4,
"median": 4,
"mad": 2,
"p60": 4,
"p70": 6,
"p80": 6,
"p90": 6
},
"pageDownloadTime": {
"min": 0,
"max": 1,
"avg": 0.77777777777777777,
"median": 1,
"mad": 0,
"p60": 1,
"p70": 1,
"p80": 1,
"p90": 1
},
"domInteractiveTime": {
"min": 17,
"max": 46,
"avg": 29.333333333333332,
"median": 25,
"mad": 8,
"p60": 25,
"p70": 46,
"p80": 46,
"p90": 46
},
"domContentLoadedTime": {
"min": 17,
"max": 46,
"avg": 29.333333333333332,
"median": 25,
"mad": 8,
"p60": 25,
"p70": 46,
"p80": 46,
"p90": 46
},
"pageLoadTime": {
"min": 19,
"max": 46,
"avg": 30,
"median": 25,
"mad": 6,
"p60": 25,
"p70": 46,
"p80": 46,
"p90": 46
},
"frontEndTime": {
"min": 11,
"max": 20,
"avg": 14.333333333333334,
"median": 12,
"mad": 1,
"p60": 12,
"p70": 20,
"p80": 20,
"p90": 20
},
"backEndTime": {
"min": 4,
"max": 33,
"avg": 15,
"median": 8,
"mad": 4,
"p60": 8,
"p70": 33,
"p80": 33,
"p90": 33
}
}
},
"caps": {
"rotatable": false,
"browserConnectionEnabled": false,
"acceptSslCerts": false,
"cssSelectorsEnabled": true,
"javascriptEnabled": true,
"databaseEnabled": false,
"chrome.chromedriverVersion": "26.0.1383.0",
"locationContextEnabled": false,
"takesScreenshot": true,
"platform": "linux",
"browserName": "chrome",
"version": "28.0.1500.95",
"hasMetadata": true,
"nativeEvents": true,
"applicationCacheEnabled": false,
"webStorageEnabled": true,
"handlesAlerts": true
},
"end": "2014-04-30T21:28:52.644Z",
"duration": 9520
},
"tests": [ ... includes raw timings here as duration props ... ],
"passes": [ ... same as above ... ]
Spec
$ sauce-browsertime http://example.com -n 3 --reporter spec
Collecting Navigation Timings with chrome
http://example.com
√ http://example.com #1
√ http://example.com #2
√ http://example.com #3
3 passing (11s)
Nyan!
$ sauce-browsertime http://example.com http://example.com/page-one http://example.com/page-two -n 4 --reporter nyan
12 -_-_-_-_-_-_-_,------,
0 -_-_-_-_-_-_-_| /\_/\
0 -_-_-_-_-_-_-^|__( ^ .^)
-_-_-_-_-_-_- "" ""
12 passing (23s)
HTML templates
Not a a mocha reporter per say. Transforms the raw JSON data of a previous run into an HTML page, with visual representation of the Timing objects thanks to http://kaaes.github.io/timing/
Usage: sauce-browsertime-html [options] file.json
$ sauce-browsertime results.json > index.html
Graphs generated thanks to http://kaaes.github.io/timing/
Browser logos thanks to https://github.com/alrra/browser-logos
Options:
-b, --browser - Browser under test (used to match browser logo)
Jenkins template
config.xml file is a Jenkins Job template that can be used to quickly
setup a Job to run the perf test, at a fixed interval or on SCM change:
-
Includes Job parameters for:
PERF_URLS: List of URLs to test. Whitespace separated.SAUCE_USERNAME: Saucelabs usernameSAUCE_ACCESS_KEY: Saucelabs API keySAUCE_BROWSERS: Comma separated list of browsers.
-
Shell scripts
- Install npm packages on first run (npm install saucelabs-browsertime)
- A run for each browser, stats & raw metrics available at
./results/BUILD_NUMBER/BROWSER/metrics.json, HTML report at./results/BUILD_NUMBER/BROWSER/metrics.json.
-
Plugin configuration for htmlpublisher (optional to have it installed, jenkins will just ignore the conf)
See config.xml file for further detail.
# Job creation
$ curl -X POST -H 'Content-Type: application/xml' $JEKINS_URL/pluginManager/installNecessaryPlugins --data-binary @config.xml
# Check plugins
$ curl -X POST -H 'Content-Type: application/xml' $JEKINS_URL/pluginManager/prevalidateConfig --data-binary @config.xml
# Install plugins
$ curl -X POST -H 'Content-Type: application/xml' $JEKINS_URL/pluginManager/installNecessaryPlugins --data-binary @config.xml
TODOs
- [] Statsd / Graphite reporter
- Assert system (like phantomas)
- HTML reporter with http://kaaes.github.io/timing/ widget
- Option for the number of runs, and stats (avg, median, percentiles)