Repeato CLI Test Runner

Introduction

Repeato CLI is a headless test runner that allows you to execute test batches designed with Repeato Studio.

We try to rely as much a possible on open standards, so all the workspace data is stored in either image formats or text based formats (mostly JSON). That makes it possible to generate tests not just with Repeato Studio, but also with tools of your choice. That's also true for the test run artefacts: All data is stored as image data (screenshots), XML (JUnit export, can be used for Jenkins) and JSON.

Repeato CLI is built using Node.js which makes it possible to run it on almost any system. However, it has dependencies to native modules, which makes it necessary to build parts of it for the target OS and platform.

We prebuilt native modules for NodeJs version 16, so we recommend using this version.

Installation:

Recommended way of using this package is via npx, which downloads and runs Repeato-CLI right away:

npx @repeato/cli-testrunner

If you run Repeato as part of your CI/CD, you might want to fix the version of the test runner, so you don't run into a situation where your setup breaks because of updates we ship. (Although those should usually not cause any problems!)

npx @repeato/cli-testrunner@1.2.24

The other possibility is to install it via npm:

npm -g @repeato/cli-testrunner

...and then run it like this:

cli-testrunner --help

Inputs:

The testrunner essentially needs two inputs:

The path to your workspace containing you batch
The batch ID of the batch you want to execute

cli-testrunner --help will print all the required and optional params:

Runs a selected batch of your Repeato workspace

Usage via npx:
  $ npx @repeato/cli-testrunner [options]

Usage when installed globally via npm:
  $ cli-testrunner [options]

Required params:
  --workspaceDir: Path to the workspace directory
  --deviceId: Optional. The device ID of the device you want to connect to
    List Android devices via 'adb devices' or iOS simulators via 'xcrun simctl list'

Optional params:
  --batchId: Which batch should be executed. 
    Just execute cli-testrunner --workspaceDir [workspace-dir] to see a list of contained batches
  --runMode: "AllTests" | "OnlyFailed" (default: "AllTests")
  --timeoutFactor: Since some CI servers sometimes don't provide as much performance as local machines, 
    it can be useful to increase the timeout when running on a server by a factor. E.g pass 1.5 to increase the timeouts by 50% (default: 1.0)
  --waitDurationBetweenSteps: Additional wait time between step executions can help during debugging or to increase test stability. 
    (default: 0, milliseconds)
  --outputDir: Override the default export path for batch run reports (default is: [workspaceDir]/batchRuns)
  --logLevel: "WARN" | "INFO" | "DEBUG" (default: "INFO")
  --force: Ignore errors (DB version mismatch) and execute anyway
  --licenseKey: License key to unlock full feature set of Repeato-CLI

Examples:
  $ npx @repeato/cli-testrunner --workspaceDir [path to your workspace]
  List all batches in your workspace
  
  $ npx @repeato/cli-testrunner --workspaceDir [path to your workspace] --batchId [ID of your batch]
  Executes all the tests in the given batch

Outputs:

A JUnit xml file is created in the root folder of Repeato CLI
For each test a testRun object is created in [workspaceDir]/[testId]/test.json
For each test step a screenshot is created in [workspaceDir]/[testId]/testRuns/[deviceId]/
An html batch report is created in the [workspaceDir]/batchRuns directory. The exported directory can be shared via a webserver to make the report accessible to your QA team.
A batchRun.json file containing all the batch run information in [workspaceDir]/batchReports/[your batch report] directory
Returns with exit code 0 if batch runner managed to run all tests, even if a test failed. Returns different exit code if some error occured during execution.

Requirements:

Supported Node.js version: 16 (v16.13.0 is tested). You can use nvm to switch to the right Node.js version: nvm use 16.

Supported operating systems: MacOS, Windows, Linux

For Apple Sillicon (M1 / ARM architecture), you need to use Node.js v16

Starting the testrunner

Simply execute npx @repeato/cli-testrunner [args]

Installing emulators or simulators

We have a couple of guides on how to install Android emulators or iOS simulators on your target system here: https://www.repeato.app/documentation/virtual-test-devices/

Install apps programmatically on your target emulator / simulator

On Android you can use ADB to push your APK to the virtual device.

Install ADB
adb push /path/to/myApp.apk

On iOS you can use IDB

Install IDB
idb install /path/to/myApp.app

Troubleshooting

Device does not connect on linux

There are two things that need to be set up correctly: each user that wants to use adb needs to be in the plugdev group, and the system needs to have udev rules installed that cover the device.

plugdev group: If you see an error message that says you're not in the plugdev group, you'll need to add yourself to the plugdev group:

sudo usermod -aG plugdev $LOGNAME

Note that groups only get updated on login, so you'll need to log out for this change to take effect. When you log back in, you can use the id command to check that you're now in the plugdev group.

udev rules: The android-sdk-platform-tools-common package contains a community-maintained default set of udev rules for Android devices. To install:

apt install android-sdk-platform-tools-common

npm