yanatra
Run E2E tests with JSON.
Built off of Protractor and Selenium.
Short for yanatramanav, the hindi word for robot.
Overview
Instead of tests like code:
browser;browserdriver;
Write tests like scripts:
"goto": "https://www.reddit.com/r/all" "click": "a[href=\"https://www.reddit.com/r/random/]"
yanatra runs "scripts": a linear sequence of actions. The kind of actions a Product Manager or a QA person would write down in excel spreadsheet and perform manually.
yanatra scripts standardize these informal tests in two important ways.
-
Readability A script is JSON array of objects. First attribute defines the action and the argument to be passed into into that action. Arguments can be strings, numbers, objects, depending on what the action accepts.
"sequence":"click": "#someButtonId""type":"keys": "one of the best.""css": "#allen" -
Simplicity Normal E2E tests are fully-expressive, and offer complex control flow. yanatra, on the other hand, limits the kinds of browser actions that can be be performed and how they are performed. For the most part your tests will do one action after another, after another, and fail if an action cannot be performed.
yanatra scripts compile into normal JS Protractor tests, which automatically includes workaround some of Selenium's edge cases.1
[1] yanatra aims to reduce barrier-to-entry for test creation and maintenance.
[2] Crashing if an element is not immediately clickable, for example.
Install
Install the latest Oracle Java JRE, if on Mac, or the latest OpenJDK, if on Linux.
Mac OS X users should use brew, e.g.:
$ brew update$ brew cask install java
Once you have the latest JRE, install yanatra globally:
$ npm install -g yanatra
Finally, get the latest Selenium jars and Chrome drivers:
$ yanatra update
You will now be able to use yanatra in any directory on your system.
To start using some sample tests, go to or make an empty directory. In that directory, issue the command:
$ yanatra init
You can then run those tests in your newly filled directory with the command:
$ yanatra run
Running scripts
Enter yanatra
into your console in any directory on your machine. By default, this command will list out all avaiable arguments, options and extensions if
a yanatra.js file is in the directory.
To run a specific json:
$ yanatra run relative/or/absolute/path/to/your/script.json
Or to run all valid json in a directory:
$ yanatra run relative/or/absolute/path/
If no yanatra file is specified, yanatra will use the command line arguments you pass in, followed by the default configuration.
By default, yanatra will ignore any dir that starts with an underscore (_
) when seeking scripts to run.
At the conclusion of the test, the results of each script will be written to a corresponding reports directory, and summarized in an index.html file.
Writing scripts
A script is a json test file.
A spec is a script run directly by yanatra.
A sequence is the steps inside of a script.
A step is an action at a particular moment of the script.
You can copy any one of the scripts example/
to start, or run yanatra init
to start them all.
yanatra works best when scripting "flows", multiple user actions across multiple pages or entry points. Unit tests are much better for testing components and modules in isolation.
Actions
An action is what the protractor does to the browser, like click
, type
, screenshot
and more.
Each action can accept an object, a string, or a numeric value as an argument. An object can support any kind of data, so you can think of non-object arguments as shorthand.
For example, the click action can be written simply as:
"click": "#jqueryCssSelector"
or explicitly as:
"click": "css": "#jqueryCssSelector"
You can view all the loaded actions, including your custom ones, by entering this command in console:
$ yanatra debug actions
For more on actions and their specific syntax, check out the actions api
Repeating Actions
Sometimes, one action will be repeated several times, using different arguments, e.g, filling out a form.
To reduce repetition, yanatra offers an optional array syntax for any action.
So instead of:
"type": "model": "ctrl.ngModel.data.title" "keys": "hello" "type": "model": "ctrl.ngModel.data.total" "keys": "mr robinson" "type": "model": "ctrl.ngModel.data.sub" "keys": "912.3"
Try:
"type": "model": "ctrl.ngModel.data.title" "keys": "hello" "model": "ctrl.ngModel.data.total" "keys": "mr robinson" "model": "ctrl.ngModel.data.sub" "keys": "912.3"
Note: this means an action cannot accept an array as its argument.
Expectations
This functionality is still in alpha
Expectations assert that certain variables, elements, and other things during your spec's runtime match the values written in your test.
You can access all expectations with the "expect" action.
For example, to test that a certain variable equals 10:
"expect": {
"equal": [
10,
{ "var": "window.products" }
]
}
The format for expecting variables is either to use a string or a number for a constant, or to use object that will be resolved into a value.
{"var": window.products} -> gets the value of window.products
{"model": view.model} -> gets the value of view.model in your angular.html
yanatra supports shorthands for more complex expectations, like if an element is visible, present, and more.
If the expectation fails to pass, the current spec will continue until its completion, and then terminate any other specs that have yet to run.
To run all specs disregarding expectation errors, add the --steamroll
flag in the terminal.
Context
Contexts are key-value mappings substituted into your yanatra scripts and subscripts at compile time.
To indicate that you want to substitute a value from the context into a string, add a #
right before and after the key, like so:
"context": "website": "http://allenhascheerfulresolve.com" "inputName": "a field_to_be_submitted" "inputValue": 10 "sequence": "goto": "#website#" "type": "name": "#inputName#" "keys": "#inputValue#" "submit" "goto": "#website#/a/path/that/will/be/added/onto/thesubstitionstring"
Note that a context substitution will not change while your tests are running, unless a custom action modifies runtime.context
Remembered variables
Remembered variables are key-value mappings substituted into your yanatra scripts at run time.
This means that you can do things like remembering the values on pages, and using them to change the arguments of future actions:
"remember": "var": "window.user_id" "@": "yanatraUserId" "goto": "/profile/@yanatraUserId@/"
In this example, if the user_id
on the page was 123456, the next action would go to the url /profile/123456/
To distinguish remembered variables from context variables in strings, you must place @
before and after your key in the string instead of #
.
Subscripts
A subscript is a script run inside of specs using the run
command. It should not be run as a spec, i.e., by itself.
Subscripts are useful when you find that your spec files are each getting very, very large, or many of its steps are being repeated several times.
You can go from this structure:
script_dir
|-- script.json
//script.json "click": "a[href='login']" "type": "css": "input[name=username]" "text": "NewUser1"
to this:
script_dir
|-- _subscript_dirname_here
| |-- go_to_login_and_type_something.json
|-- script.json
// script.json "run": "_subscript_dirname_here/go_to_login_and_type_something.json" //_subscript_dirname_here/go_to_login_and_type_something.json "click": "a[href='login']" "type": "css": "input[name=username]" "text": "NewUser1"
Note To clearly indicate a subscript is not a spec, place the subscript in a directory whose name starts with an underscore (_
).
Run Options
You can also specify three options to the run command to customize its behavior:
skip_on_check_fail
- iftrue
, yanatra will immediately stop executing the current subscript or any of its subscript if a single one of itscheck
actions fail.retry
- similar toskip_on_check_fail
, except yanatra will retry the subscript until either all of its checks passes, or it reaches the number of retries (the value of theretry
option).context
- a custom context for the given subscript, which will override both the current context and the subscript's context.
You can use these options by using an object instead of a string for the run action.
"run": "script": "_subscript_dirname_here/go_to_login_and_type_something.json" "retry": 10
Note: retry
includes skip_on_check_fail
by default, so you don't need to have both to do retries.
Checks
A "check" is an assertion that uses the same syntax as the "expect" action. The difference is that while an expectation will be written in the report, a check will only be logged as a runtime activity.
This means that if a check fails, the spec as a whole will not fail.
Checks are most useful while using the retry
or skip_on_check_fail
option, since failing those check commands
will trigger the retry
/skip_on_check_fail
check.
You can leverage checks to do simple conditional logic:
run this subscript if a check condition is true, or else skip the subscript. Or in the case of retries: repeat a subscript X
number of times until all checks have passed, or terminate the spec.
Extending yanatra
yanatra files are nodejs modules that alter and extend yanatra's default behavior.
For example, to set "./scripts/regression/"
as the default directory for running scripts,
and "../yanatra_reports"
as the default reports directory in ~/my_scratch_folder
, create
~/my_scratch_folder/yanatra.js
with the contents:
moduleexports = suites: "./scripts/regression/" reports: "../yanatra_reports"
You and others who have access to this dir will now get those two options automatically when issuing
$ yanatra run
To get started, go to an empty dir of your choosing and enter in console:
$ yanatra init
This will copy a boilerplate yanatra config, actions, scripts and reports directory into the current working directory.
You can specify a custom template to copy in example/
by issuing
$ yanatra init [template_name_directory_here]
Custom Actions
Your scripts will eventually require specific actions that yanatra does not provide out of the box. To add your own actions, specify the relative or absolute path to an actions module in your yanatra.js module:
moduleexports = suites: "./scripts/regression/" reports: "../yanatra_reports" actions: "./actions.js"
The object that is returned by your custom actions module will be merged with yanatra's base actions.
"Merged" means any one of your actions has the same name as a base yanatra action will override the original.
For example, the init
action is the first step of every spec and does nothing by default.
If you were to override it with action named init
in your custom actions module, any spec ran using that yanatra file will now start your test with the code of your custom init.
If we wanted every spec to (re)start at the root of our website:
moduleexports = { return browser }
Custom, new actions can be declared as additional key/values to the json object returned by
your actions module. The actions
variable supplied to the module gives you access to
default actions in your module, as long as they don't get overridden by your custom action.
var actions = yanatraactions;moduleexports = { return browser } { return yanatraactions; }
yanatra
object
the All the JS that is called, executed or referred to be yanatra during runtime will have access to a global variable
called yanatra
yanatra
contains five top-level properties:
- env - the configuration that yanatra starts each spec with.
- rem - the variables that have been remembered for the given spec.
- util - helper and utility functions to simpfy building actions.
- actions - the actions module that has been merged with your custom action module.
- baseActions - the original actions module that comes with yanatra.
Environment
An environment is a json object that includes:
- args - arguments specified in the command line
- config - settings in your yanatra file
Every environment for your current test run is available under the yanatra
global variable, as yanatra.env
{ return yanatraactions;}
You can attach additional objects, strings, numbers, and functions to yanatra.env
, although
you should do so using your own namespace on the global yanatra
object.
{ yanatramynamespace = loggedEnvCount: 0 ;} { console;}// will print out "envCount 0" on the first time// and then "envCount 1" on the second time, and so forth.
All of your modifications to the global yanatra object will persist only across subscripts, and not specs; this means that if you run a test in parallel, you won't be able to share runtime data between the processes.
Runtime
Runtime refers to yanatra's current state by the time of the step. It is supplied as the second argument of every action function.
For example, you can access all the context variables that are currently available for the current action
with runtime.context
. Note that changing the context here will only affect the script and its children, not the parent scripts.
{ return yanatraactions;}
Changing context can be helpful if you want to set a context/substitution programmatically at the start of the test.
{ runtimecontextrandomUserName = runtimecontexttime + "@gmail.com" return yanatraactions;}
And later:
"log": "#randomUserName#"
which will output the UNIX timestamp for when the test started, with the @gmail.com
at its end.
Rem
Rem, yanatra.rem
, is the object for all the key-values you stored with the remember
action. Unlike context which is overridden on a tree/lexical level at a compile time, Rem should only change during runtime.
Custom protractor config
Currently, yanatra does not properly support custom protractor configs. While the option is available in the terminal for the daring, using it will break many if not all of yanatra's behavior.
Custom Arguments
Thanks to yargs, you can add custom options to your yanantra command using your yanatra.js
file. Just add an options
object:
options: allowLogin: demand: false describe: "email/username for user" type: 'boolean'
You will then be able to issue a command like yanatra run --allowLogin
and see it as a possible option when the yanatra help prompt is shown for the directory your yanatra file is in.
In conjunction with the environment variable in your test, yanatra args can alter action behavior like so:
{ // did the test runner allow a login? (e.g. yanatra run --allowLogin) if !yanatraenvargsallowLogin return; // looks like it did, proceed with login return ;}
For more on the exact syntax of arguments you can add and leverage, please consult https://github.com/yargs/yargs.
Chrome Extension
Optional, very alpha, fork of daylight (https://github.com/segmentio/daydream).
Yanatra-Chrome records your browser actions as yanatra actions.
This will likely be placed into its own repository since there is no shared code between it and yanatra, the nodejs test runner.
Project
Known Bugs
- Missing unit tests for core yanatra functionality.
- Only Chrome is supported.
- Custom Protractor files are not easily integrated with yanatra.
- Certain actions will only work preoperly on AngularJS pages.
- Subscripts run with the
retry
orskip_all_on_check_fail
break if they also have run statements withretry
orskip_all_on_check_fail
.
FAQ
Who should use this?
The citizen developer who knows JSON, some CSS/HTML, and maybe a little JS. Or an engineer who wishes to do E2E testing a bit faster.
Why not just unit tests?
Test each part of the software, and then test the software as a whole.
How is this used at Kinnek?
We extend yanatra the exact same way as any other user would: a plethora of custom actions and pretty complex flows broken up into a bunch of little subscripts
Is there a roadmap?
New features, functionality, and customization will be stated in the TODO. An actual roadmap will be created depending on the popularity of yanatra.
I recommend posting a Github issue for any particular request you may have or bug you may encounter. Thanks!
Acknowledgements
People
Kinnek, for providing the development time to pursue the concept of script-driven testing.
Libraries
A small subset of yanatra's node dependencies that were particularly helpful.
- gulp-protractor
- jasmine-spec-reporter
- jasmine2-custom-message
- protractor
- protractor-fail-fast
- protractor-jasmine2-screenshot-reporter
- yargs
TODO
-
Standardizing on syntax of expectations and element/value resolving.
-
Unit and functional tests for the yanatra compiler
-
Documentation. Both the code commments and the markdown files: the Docs folder is autogenerated from jsdoc and right now the output is not as pretty as it should be.