@muraldevkit/ds-manifest-builder

Test Manifest Builder

This package contains code for generating a manifest for running automated scans in Mural's design system. It also includes utility functions for running the automated scans in a Cypress environment.

Table of Contents

1. Getting started
   1. Available configurations
2. Sample script
3. How it works
4. Output
5. Usage
6. Usage with configuration file
7. Manifest object
8. Interactions
9. Scan scopes
10. Global scan utilities
    1. globalTest
    2. globalTestThemes
    3. aXeValidationTest
    4. IBMaValidationTest
    5. HTML validation
       1. installHtmlValidate
       2. HTMLValidationTest
11. Testing

Getting started

To use the Test Manifest Builder, you should install it in your package using the following approach to reduce having to manually update the package.json file:

lerna exec "npm pkg set devDependencies.@muraldevkit/ds-manifest-builder=*" --scope=@muraldevkit/<PACKAGE_NAME>

lerna exec "npm install --package-lock-only" --scope=@muraldevkit/<PACKAGE_NAME>

After adding this new dependency you should run npm ci from the root of the monorepo to bootstrap all required dependencies.

Once installed, you will need to create a script to run via Node in your package.json file. In this script, you will need to import the buildManifest function with the correct configurations. Check the Available configurations for more details.

Available configurations

Option	Required	Type	Default	Description	Suggestions
directory	required	string	null	relative path from current directory to the Storybook assets folder	use process.cwd() in the script file, path.join(process.cwd(), 'storybook-assets')
outputPath	optional	string	./tests/manifest.json	the output location relative to process.cwd()	Default creates a file at ./tests/manifest.json
exportType	optional	string	null	Name of the framework, if different from our default Web Component framework	Examples: React, Angular, Vue

Sample script

const { buildManifest } = require('@muraldevkit/ds-manifest-builder');

/**
 * Generate a test manifest to `./tests/manifest.json`
 */
buildManifest({
	directory: 'storybook-static'
});

How it works

The manifest builder works by pulling data from two different sources:

1. A Storybook API called STORYBOOK_CLIENT_API which holds information from all the rendered stories. Note: Storybook is loaded from the storybook-static folder, don't forget to build it first.
2. config.ts files that are present in all of the design system components.

When this data is pulled, the manifest builder will attempt to merge both sources. It will match the id of an object from the Storybook API with the title property of a named export in a config.ts file. We use an internal function to convert the title into a valid Storybook id.

// config.ts
export const storyData = {
	title: 'Components/Button'
	...
}

// object from Storybook API
{
	id: 'components-button--basic'
	...
}

IMPORTANT

• If we don't find a configuration object that matches with the Storybook id, we exclude that Storybook object from the manifest. This ensures that config.ts files are our source of truth to include or exclude items from the manifest and gives our developers control over which demos to include in our automated tests.

• If an object that is exported from a config.ts file lacks a title property, it will be excluded as well. We have this validation in place to support the presence of other kinds of named exports in config.ts files that are not relevant to the manifest builder.

• config.ts named exports with a title property can match more than one Storybook object. This behavior is intended to allow reuse of the same configuration objects across different demos.

Example:

// config.ts
export const storyData = {
	title: 'Components/Button'
	...
}

// both objects from Storybook API will get merged with 'Components/Button' giving two merged objects as result
{
	kind: 'Button',
	id: 'components-button--basic',
	...
}
{
	kind: 'Button',
	id: 'components-button--custom',
	...
}

Note: If you want to use a custom Story title, you must ensure that the exported Story name is the same.

Example:

// config.ts:
export const customStoryName = {
	...sharedStoryData,
	args: {
		...sharedStoryData.args
	},
	manifest: {
		exclude: {
			// These are all tested by the default demo
			default: [
				'A11Y',
				'HTML',
				'FUNC'
			]
		}
	},
	name: 'Custom Story Name',
	title: 'Components/Component Family'
};

// stories.tsx:
export const CustomStoryName = Template.bind({});

Output

The manifest.json file stores an object where each property corresponds to one of the global scan utilities used by the design system. The scan scopes are: - 'A11Y': axe and IBMa accessibility checks - 'FUNC': functional exported tests - 'HTML': HTML validation

Each scan scope contains an array of components, and each component has an array of scenarios.

{
	"A11Y": [
		{
			"kind": "Avatar",
			"name": "Basic",
			"scenarios": [
				{
					"name": "Default Scenario",
					"url": "iframe.html?id=components-avatar--basic&viewMode=story"
				}
			]
		}
	],
	"FUNC": [
		{
			"kind": "Avatar",
			"name": "Basic",
			"scenarios": [
				{
					"name": "Default Scenario",
					"url": "iframe.html?id=components-avatar--basic&viewMode=story"
				}
			]
		}
	],
	"HTML": [
		{
			"kind": "Avatar",
			"name": "Basic",
			"scenarios": [
				{
					"name": "Default Scenario",
					"url": "iframe.html?id=components-avatar--basic&viewMode=story"
				}
			]
		}
	]
}

In this example, each scan scope stores a single component, in this case the Avatar component. It has one scenario, called Default Scenario, which is the default scenario for all components that don't use the manifest object in their configuration.

Usage

The manifest builder is an automated script that will create test scenarios. That said, it is important to be aware that updating a scenario's configuration could generate new assets, such as baselines for our accessibility tests or multiple new snapshots taken for our visual regression tests.

For example, let's imagine you are working on the Button component and have a working demo for it in Storybook. Most likely, it should have:

• A config.ts file which exports an object that could be named storyData to configure the story file. If that object has a title prop, it is considered a valid named export for the manifest builder. Note: the title must be unique.
• A <COMPONENT_NAME>.stories.tsx where we import that object. This file will have a default export where we merge the contents of storyData. This object will be the template for one or many Storybook demos.

export const storyData = {
	args: {
		disabled: defaults.disabled
	},
	argTypes: {
		disabled: {
			control: {
				type: 'boolean'
			}
		}
	},
	title: 'Components/Button'
};

This configuration is used to create a demo that supports a boolean control for the disabled arg type. It also has a title property which will make it a valid named export for the manifest builder.

If we run the manifest builder it will produce a single scenario called Default Scenario. The resulting test manifest will be:

{
	"kind": "Basic",
	"name": "Button",
	"scenarios": [
		{
			"name": "Default Scenario",
			"url": "iframe.html?id=components-button--basic&viewMode=story"
		}
	]
}

By default all of our scan scopes: A11Y, HTML, and FUNC and will have at least one scenario with the name of default scenario. The manifest builder allows you to exclude it for a specific scan scope by adding a exclude property to the manifest object.

Usage with configuration file

Alternatively you can use the manifest builder with non-design system Storybook implementations via a global configuration file to define manifest data.

After installing the manifest builder package:

• create a global configuration file. We support different formats:
• json: manifest-builderrc.json
• yaml: manifest-builderrc.yaml
• javascript: manifest-builder.config.js

We suggest javascript as this format can leverage JSDoc types to support type checking.

This file needs to be placed at the same level or above the file that runs the buildManifest function. For example, if you are running node ./lib/manifest.js, the path to the manifest-builder.config.js file should be ./manifest-builder.config.js.

The global configuration file should follow this structure:

// using JavaScript extension: manifest-builder.config.js
/**
 * @type {{ components: import('@muraldevkit/ds-manifest-builder').PartialConfigData[] }}
 */
module.exports = {
	components: [
		{
			manifest: {
				// ... component manifest configuration
			},
			title: 'Components/Button'
		},
		// ... separate object for each demo in Storybook to test
	]
};

All the objects that are present in the components array will be matched against the components that are being pulled from Storybook. When working with the global manifest configuration, keep in mind a couple of key points:

• If a component from Storybook doesn't match any object from this array it will be ignored in manifest builder.
• If a configuration file exists, the manifest builder will omit fetching data from config.ts files. Both sources can't be used together.

Manifest object

The manifest builder supports a manifest property within a config.ts file's valid named exports. This property allows user configuration to be passed in and determines which scenarios are created for that component.

The type definition of the manifest property is:

type Manifest = {
	exclude?: {
		default: ScansScope[]; // ['A11Y' | 'HTML' | 'FUNC']
	}
	scans?: {
		[key: string]: boolean | ScansScope[];
	};
	interactions?: {
		on:
			| {
					[key: string]: string | boolean;
				}
			| string[]
			| string;
		testConfigs: {
			[key: string]: string;
		};
	}[];
};

The manifest builder supports excluding the default scenario for a specific scan scope. You need to create an exclude object with the default property inside of the manifest object. You can add an array with the specific scopes that should not have the default scenario.

This example shows how you can exclude the default scenario for A11Y and HTML scopes. Please consider that the default scenario is only created when there are no argTypes in place for a specific scan scope.

manifest: {
	exclude: {
		default: ['A11Y', 'HTML']
	}
}

The most basic configuration we can use creates scenarios that include a given Storybook argType:

export const storyData = {
	args: {
		disabled: defaults.disabled,
	},
	argTypes: {
		disabled: {
			control: {
				type: 'boolean'
			}
		},
	},
	manifest: {
		scans: {
			disabled: true
		}
	}
	title: 'Components/Button'
};

This configuration will create scenarios for all scan scopes including the disabled argType and will output the following result:

{
	"kind": "Button",
	"name": "Basic",
	"scenarios": [
		{
			"name": "Scenario with disabled: true",
			"url": "iframe.html?id=components-button--basic&viewMode=story&args=disabled:true"
		},
		{
			"name": "Scenario with disabled: false",
			"url": "iframe.html?id=components-button--basic&viewMode=story&args=disabled:false"
		}
	]
}

If we want to create scenarios with more than one argType, we simply need to add the name of the argType to the scans object:

{
	manifest: {
		scans: {
			disabled: true,
			kind: true,
			size: true,
			...
		}
	}
}

Interactions

The manifest object also supports configuration to create interactions:

{
	manifest: {
		scans: {
			kind: true
		},
		interactions: [
			{
				testConfigs: {
					hoverSelector: 'mrl-button'
				},
				on: {
					kind: 'primary'
				}
			}
		]
	}
}

In this example, we instruct the manifest builder to create scenarios that support hover interactions when the argType kind has a value of primary. The scenarios that include this argType value will be duplicated and extended with the testConfigs object. This will then be used by our global scans to support tests that require extra configuration. The testConfigs object is not strongly typed to allow users to pass different properties, but please check that the properties you provide are supported by the global scans.

Scan scopes

The manifest builder can create scenarios which only include certain argTypes for specific scans. This is useful because it avoid creating unnecessary test scenarios. You can configure the scan scopes inside the scans object, within the manifest object:

We support two different values for argTypes:

• true: creates scenarios for all scan scopes
• ['A11Y', 'HTML']: creates scenarios only for selected scopes

{
	manifest: {
		scans: {
			[k: string]: boolean | ScansScope[]
		}
	}
}

Example:

// Button component
manifest: {
	scans: {
		disabled: ['A11Y'],
		size: true
	}
}

This configuration will create scenarios that include the disabled argType only for the A11Y scope. If we use true instead of an array, the argType will run all scans. The array syntax can be used to define specific scans to run, and any other scans not defined in the array will be ignored.

{
	"A11Y": [
		{
			"kind": "Button",
			"name": "Basic",
			"scenarios": [
				{
					"name": "Scenario with disabled: true, size: large",
					"url": "iframe.html?id=components-button--basic&viewMode=story&args=disabled:true;size:large"
				},
				...
			]
		}
	],
	// 'HTML' will have the same output
	"FUNC": [
		{
			"kind": "Button",
			"name": "Basic",
			"scenarios": [
				{
					"name": "Scenario with size: large",
					"url": "iframe.html?id=components-button--basic&viewMode=story&args=size:large"
				},
				...
			]
		}
	]
}

If a component is configured in a way that all of the given argTypes are configured to only run against a specific scope, we will generate a default scenario for all other scan scopes to run against. In the following example, the manifest builder will generate scenarios against all size and disabled options for our A11Y scans in addition to a default scenario for HTML, and FUNC scans.

manifest: {
	scans: {
		disabled: ['A11Y'],
		size: ['A11Y']
	}
}

Global Scan Utilities

The following utility functions use the test manifest created by this package within a Cypress and Storybook environment.

Note: These files are meant for use within a testing environment which means they are transpiled differently than the core manifest builder. To access these files, use @muraldevkit/ds-manifest-builder/dist/scan-utils for your import.

globalTest

Runs any global scan against the test manifest.

Parameter	Type	Description
options.scenarioTest	function	specific test for each scenario, accepts a string of the testUrl and a string for the testName
options.testDescription	string	description of the test being run that is used as part of the output message
options.useThemeName	boolean	indicates if the theme name should be part of each test's name.
manifest	TestManifest	list of test configurations for every demo/scenario
scope	ScansScope	scan scope to test, could be 'A11Y', 'FUNC', or 'HTML'

Example:

import { globalTest } from '@muraldevkit/ds-manifest-builder/dist/scan-utils';
import manifest from '../manifest.json';

/**
 * Tests Meow of each scenario
 *
 * @param {string} testUrl - the URL of the component to test
 * @returns {void}
 */
const meowTest = (testUrl: string): void => {
	cy.visit(testUrl);
	// ...
};

context('Meow Tests', () => {
	globalTest(
		{
			scenarioTest: meowTest,
			testDescription: 'is valid according to Meow'
		},
		manifest,
		'A11Y'
	);
});

globalTestThemes

Extends the globalTest function to apply the test across multiple themes.

Parameter	Type	Description
options	GlobalTestOptions	sets up the test configuration
themes	Array<Theme>	list of theme metadata
manifest	Array<TestManifestData>	list of test configurations for every demo/scenario
scope	ScansScope	scan scope to test, could be 'A11Y', 'FUNC', or 'HTML'

Example:

import {
	globalTestThemes,
	VisualRegressionTest
} from '@muraldevkit/ds-manifest-builder/dist/scan-utils';
import manifest from '../manifest.json';

const themes = [
	{
		class: ['mrl-theme-light'],
		color: '#FFFFFF',
		id: 'light',
		name: 'Light Theme'
	},
	{
		class: ['mrl-theme-dark'],
		color: '#000000',
		id: 'dark',
		name: 'Dark Theme'
	}
];

context('Visual Regression Tests', () => {
	globalTestThemes(
		{
			scenarioTest: VisualRegressionTest,
			testDescription: 'should match design'
		},
		themes,
		manifest
	);
});

aXeValidationTest

Ensures components are accessible with axe.

Parameter	Type	Description
testUrl	string	the URL of the component to test
ignoredRules	RuleObject	axe rules to ignore based on application needs

Example:

const url = 'iframe.html?id=components-example--basic&viewMode=story';

const IgnoredAxeRules = {
	'foo': { bar: false }
	...
};

// using the default ignored axe rules (src/scan-utils/ignoredRules/aXe.ts)
aXeValidationTest(url);

// using custom ignored rules for your particular application
aXeValidationTest(url, IgnoredAxeRules);

IBMaValidationTest

Ensures components are accessible with IBMa.

Parameter	Type	Description
testUrl	string	the URL of the component to test
testName	string	the name of the test being run
ignoredRules	Array	IBMa rules to ignore based on application needs

Example:

const url = 'iframe.html?id=components-example--basic&viewMode=story';

const IgnoredACheckerRules = ['foo', 'bar'];

// using the default ignored IBMa rules (src/scan-utils/ignoredRules/ibma.ts)
aXeValidationTest(url);

// using custom ignored rules for your particular application
aXeValidationTest(url, IgnoredAxeRules);

HTML validation

HTML validation requires additional Cypress configuration in addition to its scan function:

installHtmlValidate

Sets up HTML Validator installation for Cypress.

Parameter	Type	Description
installFunc	Function	Cypress on function that HTML Validator calls on install
config	object	HTML Validator configuration object
options	object	HTML Validator options object

Usage in the Cypress plugins file:

module.exports = (on: Function) => {
	// config and options parameters have default values in the function declaration
	installHtmlValidate(on);
};

HTMLValidationTest

Ensures components have valid HTML using Cypress' html-validate plugin.

Parameter	Type	Description
testUrl	string	the URL to test

Example:

import {
	globalTest,
	HTMLValidationTest
} from '@muraldevkit/ds-manifest-builder/dist/scan-utils';
import manifest from '../manifest.json';

context('HTML Validation Tests', () => {
	globalTest(
		{
			scenarioTest: HTMLValidationTest,
			testDescription: 'should have valid HTML'
		},
		manifest,
		'HTML'
	);
});

Testing

Jest unit tests can be run within the package directory using:

npm test

Test files are located in the tests directory and new files should follow the naming convention <scriptToTest>.test.ts.