Dynamic Test Automation

The usual model for testing software is different variations of the same concept: test only what you need to. There is a cost to developing tests and this cost eats into developing the product. The goal of Dynamic Test Automation (DTA) is to take an old feature model and easily build out every possible permutation of tests for a given feature, without writing those tests "by hand".

The Feature Model

For this kind of testing, we use a simple feature model defined by the following properties:

1. Initial State: A known, familiar state to the user.

2. Operations: A set of actions that affect the initial state.

3. Expected State: How the user expected the state to change for the given operations.

Parts of a Dynamic Test

DTA works by defining various dimensions, which are arrays of properties. It will build out the cartesian product of all of the defined dimensions to generate a test case, and tie a set of operations to that test case. You can define as many dimensions with as many properties in each as you like, but beware that each property added will double the amount of total tests!

dimensions.js

const dimensions = {
  bunType: [ "regular", "poppySeed", "none" ],
  cheeseType: [ "cheddar", "american", "pepperJack", "gouda", "noCheese" ],
  veggies: [ "lettuce", "tomato", "onion", "pickles" ],
  condiments: [ "ketchup", "mustard", "mayo" ],
  meatType: [ "beef", "pork", "tofu"]
}

These dimensions would result in 540 test cases. The test case generator builds cases bottom-up, so the first test case here would be:

bunType: "regular" - cheeseType: "cheddar" - veggies: "lettuce" - condiments: "ketchup" - meatType: "beef"

It is recommended that your dimensions are pulled from enums/constants from within the application so the generated tests update alongside your source.

These test cases are mapped to callbacks via the Operations class. This is where you can pull in aplication code, mocks, etc and tie them to a dimension. For example:

operations.js

const ops = new Operations(Object.keys(dimensions));
ops.mapOperation('bunType', property => ops.state.bunType = property);

When the class is parsing the test case above, it will run that callback when it encounters bunType. These callbacks always only take one argument, which is the currently-iterated property of that dimension. In the case of bunType, it's properties are: regular, poppySeed, tofu.

A dynamic test cannot work without a callback provided for every dimension. It will throw an error if a dimension is encountered that does not have a registered callback.

The dynamicTest helper

You can import Generators directly, and use it inside a regular test. A dynamic test has to be set up within a nested describe, within a loop:

const dimensions = require(`path-to-dimension.js`);
const ops = require(`path-to-operations-instance`));
const testCases = generators.testCases(dimensions);

describe(`Test E V E R Y T H I N G`, () => {
  for (const testCase of testCases) {
    describe(`${testCase.case}`, () => {
      beforeEach( async () => {
        const operations = ops.buildOperationSet(testCase.permutation);
        for (const operation of operations) {
          await operation();
        }
      });

      afterEach(() => {
        // Reset initial state after each test
        ops.state = {...initialStateObject};
      });

      // Each "test" will run against 540 different permutations of the initial state
      test(`changeMeat() testing`, () => {
        changeMeat(ops.state, 'tofu');
        expect(ops.state.meatType).toEqual('tofu');
      });
    });
  }
});

The dynamicTest takes care of test case construction and operation execution for you. It returns a describe definition using your test definition:

dynamicTest(`Test E V E R Y T H I N G`, () => {
  afterEach(() => {
    // Reset initial state after each test
    ops.state = {...initialStateObject};
  });

  // Each "test" will run against 540 different permutations of the initial state
  test(`changeMeat() testing`, () => {
    changeMeat(ops.state, 'tofu');
    expect(ops.state.meatType).toEqual('tofu');
  });
});

Using dynamicTest requires that your dimensions.js and operations.js exist in the same directory as your test definition (for now). It also means that you cannot define a new beforeEach(), as it will overwrite the pre-built one.