RunSauce
Run many kinds of simple tests with any Sauce Labs options.
This is probably mostly useful for people who work at Sauce.
Requirements
RunSauce requires Node.js version 7.6 or greater, and NPM.
Install
npm install -g runsauceConfigure
RunSauce needs to know your Sauce credentials before it can do anything useful. There's a handy interactive setup that will pre-populate a config file from your environment variables if it can.
# configure with your username / access key runsauce --setupIf you run RunSauce with --setup after this initial setup, you will be given the
ability to add servers to your config file. This can be useful for targeting
dev environments or even local environments.
Basic Usage
# run a basic test with all the default options runsauce # check out all the options runsauce --helprunsauce --testsrunsauce --shortcuts # run a test with some custom options, like a guinea pig page test on chrome on # linux runsauce -t web_guinea -b c -p l # do something craaazy, like run a test against a self-signed https cert # on a custom dev server (assuming you have the config set) in Mac OS X 10.9 # with Safari 7, doing this 10 times with a concurrency of 5 tests at a time runsauce -c dev -t selfsigned -p m9 -b s -v 7 -r 10 -n 5Running sets of tests
You can also specify complex sets of tests and platforms to run. First, create a JSON file with your specification (notice the DSL you can use to restrict combinations of platforms):
{
"tests": [
{
"test": ["ios", "ios_hybrid", "ios_loc_serv"],
"device": ["ip", "ipa|t=ios"],
"version": ["6.1|a<=1.0", "7.0", "7.1", "8.0|a>=1.3.0", "8.1|a>=1.3.1"],
"backendVersion": ["1.0", "1.1", "1.2", "1.3.4"]
}, {
"test": ["android_hybrid|v>=4.4", "selendroid"],
"device": ["ae"],
"version": ["2.3", "4.0", "4.2", "4.3", "4.4", "5.0|a>=1.2.2"],
"backendVersion": ["1.0", "1.1", "1.2", "1.3.4"]
}
],
"config": "prod",
"build": "appium-smoke-%t",
"processes": 12
}
This will take each test set and run every possible combination of test type,
device name, platform version, and Appium version (but it will filter out
combinations deemed unsuitable by the restrictions---e.g., a platform version
of "6.1|a<=1.0" means that it will run on 6.1, but only in the case where
the Appium version is <= 1.0). It will run these tests against the prod
config, and give a build name of appium-smoke-%t (where %t is replaced by
a timestamp). Finally, it will run at a concurrency of 12! I can now run
this "build" with the following command:
./bin/runsauce -i my-build.json
(Note that you can also use the shortcut keys, e.g., "t" instead of "test"
and "d" instead of "device" and so on).
For a working example, see demo-build.json, included in this repo.
Sauce Connect-based tests
Some tests require you to start the Sauce Connect Proxy on your system in advance of running the test:
connectlocalname
Note that if running the localname test (which is a test of non-"localhost"
local name proxying), you'll need to pass the -l or --localname option to
runsauce with the 'interesting' local name (e.g., -l MyComputer.local)
Runsauce as a library
You can also build apps on top of runsauce. An example of this is in
lib/matrix (made available as ./bin/appium-matrix.js. This app reads
a "build" configuration (see above), then after running the set of Appium tests
puts together a 4-dimensional support matrix with stability annotations. It
outputs to the command-line or as HTML. For example, let's say I have the
desire to see how Appium fares across Appium versions and iOS versions with
respect to the safariIgnoreFraudWarning capability. Then I can build
a "build" configuration as follows in, say, "fraud-build.js":
module.exports = {
config: 'prod',
build: 'appium-fraud-matrix-%t',
processes: 20,
name: "Appium iOS fraud warning capability test"
tests: {
backendVersion: ['1.0.0', '1.1.0', '1.2.4', '1.3.6', '1.3.7-beta'],
version: ['6.1', '7.0', '7.1', '8.0|a>=1.3.6', '8.1|a>=1.3.6', '8.2|a>=1.3.7-beta'],
test: 'web_fraud',
browser: 's',
device: ['ip', 'ipa']
}
};
Now I can run the matrix app, specifying my input file and telling it to run each combination 5 times so we can generate meaningful reliability statistics:
./bin/appium-matrix.js -d -r 5 -i fraud-build.js
Ultimately the app will output something like:
┌───────────────────┬───────────────────────────────┬───────────────────────────────┬───────────────────────────────┬─────────┬───────────────────────────────┬─────────┐
│ │ iOS 6.1 │ iOS 7.0 │ iOS 7.1 │ iOS 8.0 │ iOS 8.1 │ iOS 8.2 │
├───────────────────┼───────────────────────────────┼───────────────────────────────┼───────────────────────────────┼─────────┼───────────────────────────────┼─────────┤
│ Appium 1.0.0 │ ┌───────────┬──────┬────────┐ │ ✗ │ ✗ │ — │ — │ — │
│ │ │ 0.92 │ iPad │ iPhone │ │ │ │ │ │ │
│ │ ├───────────┼──────┼────────┤ │ │ │ │ │ │
│ │ │ web_fraud │ ✓ │ 0.83 │ │ │ │ │ │ │
│ │ └───────────┴──────┴────────┘ │ │ │ │ │ │
├───────────────────┼───────────────────────────────┼───────────────────────────────┼───────────────────────────────┼─────────┼───────────────────────────────┼─────────┤
│ Appium 1.1.0 │ ✓ │ ✗ │ ✗ │ — │ — │ — │
├───────────────────┼───────────────────────────────┼───────────────────────────────┼───────────────────────────────┼─────────┼───────────────────────────────┼─────────┤
│ Appium 1.2.4 │ ✓ │ ✓ │ ┌───────────┬────────┬──────┐ │ — │ — │ — │
│ │ │ │ │ 0.92 │ iPhone │ iPad │ │ │ │ │
│ │ │ │ ├───────────┼────────┼──────┤ │ │ │ │
│ │ │ │ │ web_fraud │ ✓ │ 0.83 │ │ │ │ │
│ │ │ │ └───────────┴────────┴──────┘ │ │ │ │
├───────────────────┼───────────────────────────────┼───────────────────────────────┼───────────────────────────────┼─────────┼───────────────────────────────┼─────────┤
│ Appium 1.3.6 │ ✓ │ ┌───────────┬────────┬──────┐ │ ✓ │ ✗ │ ✓ │ — │
│ │ │ │ 0.83 │ iPhone │ iPad │ │ │ │ │ │
│ │ │ ├───────────┼────────┼──────┤ │ │ │ │ │
│ │ │ │ web_fraud │ ✓ │ 0.67 │ │ │ │ │ │
│ │ │ └───────────┴────────┴──────┘ │ │ │ │ │
├───────────────────┼───────────────────────────────┼───────────────────────────────┼───────────────────────────────┼─────────┼───────────────────────────────┼─────────┤
│ Appium 1.3.7-beta │ ┌───────────┬────────┬──────┐ │ ┌───────────┬────────┬──────┐ │ ✓ │ ✗ │ ┌───────────┬────────┬──────┐ │ ✓ │
│ │ │ 0.92 │ iPhone │ iPad │ │ │ 0.83 │ iPhone │ iPad │ │ │ │ │ 0.92 │ iPhone │ iPad │ │ │
│ │ ├───────────┼────────┼──────┤ │ ├───────────┼────────┼──────┤ │ │ │ ├───────────┼────────┼──────┤ │ │
│ │ │ web_fraud │ ✓ │ 0.83 │ │ │ web_fraud │ 0.83 │ 0.83 │ │ │ │ │ web_fraud │ 0.83 │ ✓ │ │ │
│ │ └───────────┴────────┴──────┘ │ └───────────┴────────┴──────┘ │ │ │ └───────────┴────────┴──────┘ │ │
└───────────────────┴───────────────────────────────┴───────────────────────────────┴───────────────────────────────┴─────────┴───────────────────────────────┴─────────┘
This depiction gives us a good idea of which platforms are supported and how
reliable they are! Run appium-matrix for the list of parameters:
Options:
-d, --detail Show detail view of failing tests [default: false]
-i, --infile JSON or JS file that exports a build definition [required] [default: null]
-f, --file File to store raw results [default: null]
-r, --runs Number of runs for each test [default: 3]
-s, --skip Skip actual test run and use results file [default: false]
-h, --html Output html table instead of CLI table [default: false]
Since matrix builds often take a long time to run, it can be useful to dump the
output into a format parseable by the tool, using the -f flag. Once this is
done, there are two further useful options:
appium-matrix -i build.js -f /my/output.json -s
The above will re-generate the CLI report from the output file rather than actually re-running the matrix.
appium-matrix -i build.js -f /my/output.json -s -h > /my/output.hml
The above will generate a beautiful little HTML page that is much more readable
especially for large matrices or ones with the -d option to show detailed
information.
(For a set of example matrix build definition files, check out
lib/matrix/builds)
Report results to Sumo Logic
If you have a SumoLogic collector endpoint set up, Runsauce will report detailed information about test passes and failures, including timing information, which can be used for later statistical purposes.
runsauce -i tests.json -j "https://endpoint1.collection.us2.sumologic.com/...."Building from source
git clone https://github.com/jlipps/runsauce.gitcd runsaucenpm install # run the (admittedly very few) tests npm test # configure runsauce, if you haven't already # ./bin/runsauce.js --setup # use runsauce to run a test on sauce! ./bin/runsauce.jsContainerized runsauce
- Build
docker build -t runsauce . --no-cache- Run
docker run -e SAUCE_USERNAME=<sauce_username> -e SAUCE_ACCESS_KEY=<sauce_access_key> --rm -it runsauce -t web -v 11.2 -a 1.8.1 -d "iPhone Simulator"