testable-utils
    TypeScript icon, indicating that this package has built-in type declarations

    0.5.11 • Public • Published

    Testable Script Utilities

    A set of utility APIs for use while running Testable scenarios (Webdriver.io or Node.js script).

    Installation

    Available on the public NPM registry as testable-utils.

    npm install testable-utils --save
    

    Local Testing

    When you run your script locally, any calls to Testable APIs will print to the console. During actual test execution via Testable test runners the API calls integrate as expected.

    If you are using Webdriver.io with our custom commands you will need to add the following in your wdio.conf.js file to ensure they get registered:

    ...
    before: function (capabilities, specs, browser) {
    	require('testable-utils').init();
    },
    ...
    

    APIs

    Capture Metrics

    Capture custom metrics during your test. Testable supports 3 types of metrics. See our custom metrics documentation for more details. Note that these utils do not support traces yet.

    Counter

    results([resource], [url]).counter(name, [increment], [units])
    results([resource], [url]).counter(options)
    

    Keep track of a counter across test execution. Namespace defaults User. Increment defaults to 1. Resource and url default to blank and are included with the "overall results".

    For example:

    var results = require('testable-utils').results;
    
    results().counter('slowRequests', 1, 'requests');
    results().counter({ namespace: 'User', name: 'fastRequests', val: 2, units: 'requests' });
    

    Timing

    results([resource], [url]).timing(name, timing, [units])
    results([resource], [url]).timing(options)
    

    Capture a timing. Namespace defaults to User. Units defaults to ms. Resource and url default to blank and are included with the "overall results". Testable will calculate various aggergations like min, max, average, standard deviation, and the percentiles defined in your test configuration.

    For example:

    var results = require('testable-utils').results;
    
    results('Google Homepage', 'https://www.google.com').timing('pageLoadMs', 1294);
    results().timing({ namespace: 'User', name: 'latencyMs', val: 196, units: 'ms' });
    

    Histogram

    results([resource], [url]).histogram(name, bucket, [increment])
    results([resource], [url]).histogram(options)
    

    Capture a histogram. Namespace defaults to User. Increment defaults to 1. Resource and url default to blank and are included with the "overall results".

    For example:

    var results = require('testable-utils').results;
    
    results().histogram('httpResponseCodes', 200);
    results().histogram({ namespace: 'User', name: 'bandwidthByType', key: 'text/html', val: 1928 });
    

    Metered

    results([resource], [url]).metered(name, bucket, val, [units])
    results([resource], [url]).metered(options)
    

    A metered metric is intended to capture live utilization of a resource in a particular bucket. Testable uses metered metrics to capture CPU utilization, memory utilization, active connections, and bandwidth per test runner.

    Testable will calculate the peak value across all buckets in each 10 second time interval during the test. For example, this can be used to observe the test runner with the highest CPU utilization.

    Namespace defaults to User. Resource and url default to blank and are included with the "overall results".

    For example:

    const results = require('testable-utils').results;
    const info = require('testable-utils').info;
    
    results().metered('Browser Heap Memory', `User ${info.globalClientIndex} - Chrome`, 1025781, 'bytes');
    

    Synchronizing Across Concurrent Users

    Testable provides several APIs to assist in coordinating across the concurrent users that are part of your test. These APIs are only available for browser-based tests and not protocol level Node.js scripts.

    Barriers

    Returns a promise that will not resolve until all concurrent users globally reach this barrier.

    results.barrier(name, [value]);
    

    For example:

    results.barrier('Login').then(function() {
    	// continue executing after all users reach the "Login" barrier.
    })

    Optionally accepts a value as the second parameter if you do not expect all concurrent users globally to reach this barrier.

    Get execution wide metric value

    results.get(name, [bucket])
    results.get(options)

    Read the current value of a metric aggregated across the entire test execution.

    For example to read the current value of the Slow Requests custom counter metric:

    results.get('Slow Requests').then(function(value) {
    	// use the value here
    });
    results.get({ namespace: 'User', name: 'Slow Requests' }).then(function(value) {
    	// use the value here
    });

    Wait for metric value

    results.waitForValue(options)
    results.incrementAndWaitForValue(options)

    Wait for the indicated metric to be greater than or equal to the specified value across the entire test execution or for the timeout to be reached. Returns a Promise object. When run locally or in a smoke test it will resolve successfully immediately.

    If you use the incrementAndWaitForValue variation it will first increment a counter by 1 with the given name and then wait for the global value to reach the specified threshold.

    The options object can include:

    • namespace: Namespace of the metric. Defaults to Testable for system metrics and User otherwise.
    • name: Metric name
    • value: The value the metric must be greater than or equal to before the Promise resolves successfully. Required.
    • timeout: Optional timeout (in milliseconds) after which to fail and stop waiting. Defaults to 0 (i.e. no timeout).
    • key: Bucket to get the metric for. Not applicable for counters but required for histogram and timing metrics

    For example to wait on the value of the counter Slow Requests to be >= 2:

    results().counter('Slow Requests', 1, 'requests');
    results.waitForValue({ namespace: 'User', name: 'Slow Requests', value: 2, timeout: 10000 }).then(function() {
    	// called when the metric reaches at least 2 aggregated across the test execution or immediately when run locally or in a smoke test
    });

    Stopwatch

    Utility API to time how long a piece of code takes to execute and capture as a Testable metric.

    For example:

    var stopwatch = require('testable-utils').stopwatch;
    
    stopwatch(function(done) {
      // some code we want to time here
      // call done() once it is done, can be async
      done();
    }, { namespace: 'User', name: 'myCustomTimer' }).then(function() {
    	// called after done() is invoked
    });

    Logging

    By default all Webdriver.io output to stdout and stderr is logged at the debug level during test execution. To log at other log levels use this API.

    Examples:

    var log = require('testable-utils').log;
    
    log.trace("This will only be captured during a smoke test or when run locally");
    log.debug("Some useful debug");
    log.info("Some useful info");
    log.error("Whoops something went wrong");
    log.fatal("Something went really wrong, lets abort the entire test");

    Execution Info

    Information about the test context in which this test is executing. See the Testable docs for more details.

    When executed locally the info is set to dummy values.

    Accessible at:

    // Information on chunk, concurrent client, execution, data stores, iteration, etc
    var info = require('testable-utils').info;
    
    // true when run locally, false when run on testable
    var isLocal = require('testable-utils').isLocal;

    CSV

    Read one or more rows from a CSV file. When run locally it looks for the CSV file on the local filesystem. When run on Testable, make sure you upload the CSV to the scenario.

    The API is as described in the Testable documentation.

    For the below examples we will use a data.csv file:

    Symbol,Price
    MSFT,100
    IBM,101
    GOOG,600
    

    See the Testable documentation for full details of the options.

    Get row by index

    Gets a row from the CSV file by index. Indices start at 1. get() return a Promise.

    var dataTable = require('testable-utils').dataTable;
    dataTable
    	.open('data.csv')
    	.get(1)
    	.then(function(result) {
    		// Example:
    		// { index: 1, data: { Symbol: 'MSFT', Price: '100' }, indexed: [ 'MSFT', '100' ] }
    		console.log('Symbol: ' + result.data['Symbol']);
    	});

    Get random row

    Gets a random row from the CSV file. random() return a Promise.

    var dataTable = require('testable-utils').dataTable;
    dataTable
    	.open('data.csv')
    	.random()
    	.then(function(result) {
    		// Example:
    		// { index: 1, data: { Symbol: 'MSFT', Price: '100' }, indexed: [ 'MSFT', '100' ] }
    		console.log('Symbol: ' + result.data['Symbol']);
    	});

    Iterate CSV

    Iterate over the CSV file, retrieving 1 or more rows. The iterator is global across the entire test execution.

    The next() function takes an optional options object that supports the following properties:

    • rows: The number of rows to return. Defaults to 1.
    var dataTable = require('testable-utils').dataTable;
    dataTable
    	.open('data.csv')
    	.next({ rows: 1 })
    	.then(function(results) {
    		// Example:
    		// [ { index: 1, data: { Symbol: 'MSFT', Price: '100' }, indexed: [ 'MSFT', '100' ] } ]
    		console.log('Symbol: ' + results[0].data['Symbol']);
    	});

    Async Code

    Node.js Only

    When running a Node.js script on Testable and use a 3rd party module that performs async actions you might need to tell Testable when the action is finished. Testable automatically instruments many modules so you don't need to do this including async, http, https, request, net, ws, socketio, engineio, tls, setTimeout, setInterval.

    For other async code use the below.

    var execute = require('testable-utils').execute;
    execute(function(finished) {
    	// my async code here, call finished() when done
    	// ...
    	finished();
    });

    Manual Live Event

    You can manually trigger an event while a test is running from the test results page (action menu => Send Live Event) or our API. Your script can listen for this event and perform an action in response. This is useful if you want to have all the virtual users perform an action at the exact same time for example. The event name/contents can be whatever you want.

    For local testing and smoke testing on Testable, you can also trigger the event in your script by checking the isLocal and isSmokeTest boolean variables.

    Example (Node.js):

    const request = require('request');
    const testableUtils = require('testable-utils');
    const fireNow = testableUtils.isLocal || testableUtils.isSmokeTest;
    const events = testableUtils.events;
    const execute = testableUtils.execute;
    
    execute(function(finished) {
      events.on('my-event', function(symbol) {
        request.get('http://sample.testable.io/stocks/' + symbol);
        events.finish()
        finished();
      });
    });
    
    if (fireNow) {
        // trigger the event when smoke testing or run locally for testing
        events.emit('my-event', 'MSFT');
        events.finish()
    }

    Example (Webdriver.io):

    const testableUtils = require('testable-utils');
    const fireNow = testableUtils.isLocal || testableUtils.isSmokeTest;
    
    describe('Load Url Requested in Event', function() {
      it('should load url', function() {
        browser.testableLogInfo('Waiting on load-url event');
        // no timeout (0), use Google url for local/smoke testing
        const url = browser.testableWaitForEvent('load-url', 0, 'https://google.com');
        browser.url(url);
        browser.testableScreenshot('Requested Url');
      });
    });

    Test Framework Syntax

    Structure your tests using the familiar Mocha.js test case syntax. When run on the Testable platform you will be able to see a test case report of all test steps including pass/fail status, duration, and error details on failure.

    For Example:

    const testableUtils = require('testable-utils');
    const describe = testableUtils.describe;
    const it = testableUtils.it;
    // await if there is any async code in your tests
    await describe('My example test suite', function() {
        it('First test step', async function() {
            await driver.get('http://www.google.com'); // Selenium javascript example code
        });
    });

    Full API:

    describe(suite, fn); // fn should be a function that contains a sequence of it() blocks
    it(test, fn); // fn executes a single test step
    beforeEach(fn); // runs before each test step (it block)
    afterEach(fn); // runs after each test step (it block)
    before(fn); // runs before each suite (describe block)
    after(fn); // runs after each suite (describe block)

    Wait For Finish

    Use this function to wait for the remainder of the test duration before the script finishes. Returns a Promise so that you can run some cleanup code before the script exits.

    For tests configured for a set number of iterations the promise resolved immediately. Otherwise the promise will not resolve until the remaining duration has passed.

    const testableUtils = require('testable-utils');
    testableUtils.waitForFinish().then(() => {
    	console.log('finished!');
    });
    

    Webdriver.io Commands

    All of the API calls above are registered as custom commands with Webdriver.io.

    Simply include var testableUtils = require('testable-utils'); in you test spec or in one of the configuration file hooks like onPrepare() and all the below custom commands will automatically get registered.

    Note that all the Webdriver.io commands can be used in a synchronous fashion.

    Screenshots

    One command that has no testable-utils equivalent is browser.testableScreenshot(name). This command takes a screenshot and puts it in the output directory to be collected as part of the test results. It also includes a prefix to make it easier to identify: [region]-[chunk]-[user]-[iteration]-[name].png. Tests are broken up into chunks, and within each chunk users and iterations are numbered starting at 0. So for example us-east-1-123-0-0-MyHomePage.png would be chunk id 123, first user, first iteration, image name MyHomePage.

    Command Mappings

    Webdriver.io testable-utils
    browser.testableInfo()
    info
    var result =
      browser.testableCsvGet(csvFile, index);
    dataTable
      .open(csvFile)
      .get(index)
      .then(function(result) { ... }
    var result =
      browser.testableCsvRandom(csvFile);
    dataTable
      .open(csvFile)
      .random()
      .then(function(result) { ... }
    var results =
      browser.testableCsvNext(csvFile[, options]);
    dataTable
      .open(csvFile)
      .random([options])
      .then(function(results) { ... }
    var result =
      browser.testableResult([resource], [url]);
    browser.testableCounter(
      result,
      name,
      [increment],
      [units]);
    results([resource], [url])
      .counter(name, [increment], [units]);
    var result =
      browser.testableResult([resource], [url]);
    browser.testableTiming(
      result,
      name,
      timing,
      [units]);
    results([resource], [url])
      .timing(name, timing, [units]);
    var result =
      browser.testableResult([resource], [url]);
    browser.testableHistogram(
      result,
      name,
      bucket,
      [increment]);
    results([resource], [url])
      .histogram(name, bucket, [increment]);
    browser.testableLogTrace(msg);
    log.trace(msg);
    browser.testableLogDebug(msg);
    log.debug(msg);
    browser.testableLogInfo(msg);
    log.info(msg);
    browser.testableLogError(msg);
    log.error(msg);
    browser.testableLogFatal(msg);
    log.fatal(msg);
    browser.testableWaitForEvent(eventName[, timeout[, defaultVal]]);
    events.on(eventName);
    browser.testableGetMetric(options);
    results.get(options);
    browser.testableWaitForValue(options);
    results.waitForValue(options);
    browser.testableWaitForFinish();
    testableUtils.waitForFinish().then(() => { });
    browser.testableBarrier(name, [value]);
    results.barrier(name, [value]);
    // blocks until done() is called
    browser.testableStopwatch(function(done) {
      // some code to time
      done();
    }, metricName, [resource]);
    // returns Promise immediately
    stopwatch(function(done) {
      // some code to time
      done();
    }, metricName, [resource]);

    Keywords

    Install

    npm i testable-utils

    DownloadsWeekly Downloads

    76,192

    Version

    0.5.11

    License

    MIT

    Unpacked Size

    58.5 kB

    Total Files

    12

    Last publish

    Collaborators

    • avistramer