A set of SCXML tests, and an HTTP client test runner for testing against SCXML HTTP test servers.
The SCXML Test Framework project has two main purposes:
- To collect test cases to run against SCXML implementations.
- To implement a generic test runner client for a client-server, HTTP- and JSON-based SCXML testing protocol. By separating out the client test runner from the SCXML interpreter on the server, it should be possible to test various SCXML implementations in a language-agnostic way.
Each SCXML test case comprises a pair of documents:
- an SCXML document
- a JSON "test script", which defines events to send into the state machine, and the expected basic state machine configuration after processing each event
See test basic/basic1.json for an example of the JSON test script format:
This test script indicates that after SCXML document basic/basic1.scxml is loaded into the SCXML interpreter, the expected initial configuration of the state machine will be a single state with id "a". Next, an event with name "t" and no data will be dispatched on the state machine, and the resulting state machine configuration will be a single state with id "b".
Note that the "initialConfiguration" and "nextConfiguration" properties should only contain the ids of expected basic states, which is to say "initialConfiguration" and "nextConfiguration" specify a "basic configuration", or a configuration of basic states. As a "full configuration", or a configuration composed of both basic and non-basic states, can be derived from a basic configuration, specifying only basic configurations in the test script can be done without leading to a loss of safety or generality.
Also note that the format of the test script assumes that each event dispatched on the state machine will trigger a single macrostep which may update the state configuration. The state machine configuration at the end of the macrostep can then be compared to an expected configuration. The testing framework explicitly does not test the intermediate state changes resulting from individual microsteps. This implies that the SCXML implementation being tested must be able to report the state configuration at the end of a macrostep in order for this testing scheme to be applied.
The JSON test script may also specify that an event should be sent after a delay. For example, from delayedSend/send1.json:
This test says that after SCXML document delayedSend/send1.scxml is loaded, the initial configuration will contain only the state with id "a". After dispatching event "t1" on the loaded SCXML session, the next configuration will contain only state with id "b". The test runner will then wait 100 milliseconds, and subsequently dispatch event "t2" on the loaded SCXML session. The expected next configuration will contain only the state with id "d".
The JSON test scripts included in this project assume a particular Statecharts semantics will be implemented by the SCXML interpreter. These semantics are currently those of the SCION project, which are documented here, and are not the same as the semantics specified by the Step Algorithm in the SCXML specification. The reason for this is documented here. A desirable feature for this project would be to allow the test scripts to be parameterizable for different Statecharts semantics. For now the best approach to allow for alternative semantics is to fork this project and rewrite the JSON test scripts as needed.
The test runner client can run tests sequentially or in parallel. It will exit when all tests have completed, and its exit status will be the number of tests failed or errored; thus, if all tests pass, the return status will be 0.
The SCXML Test Framework can be installed through npm, which is bundled with node.js:
npm install scxml-test-framework
npm install git://github.com/jbeard4/scxml-test-framework.git
To run it:
node lib/test-client.js [--test-server-url url] [--parallel] [path/to/test1.scxml [path/to/test2.scxml ...]]
For example, to run the client on all tests included in this project in parallel against the SCXML test server running on localhost:9000 (in the bash shell):
node lib/test-client.js --test-server-url localhost:9000 --parallel test/*/*.scxml
Note that each SCXML document specified should have a JSON test script in the same directory, with the same basename and a ".json" extension. This is already done for the tests included with this project.
A test case involves the following sequence of events:
- The client selects a test case, and sends the server a "load" event and the associated SCXML document.
- The server receives request to load the SCXML document, and creates a new SCXML session from the document. The server also generates a token that can be used to map subsequent client requests back to the newly-created SCXML session. The server then returns initial configuration of the SCXML session, and the generated token, to the client on the HTTP response.
- The client receives server response, and compares the returned initial configuration to the expected initial configuration specified in the test script.
- For each event and expected configuration in JSON test script:
- The client sends event to server. Each event is sent along with SCXML session token.
- The server receives the event and token, and uses the token to retrieve SCXML session. The server then dispatches the received event on the SCXML session, and returns the new SCXML session configuration to the client on the HTTP response.
- The client receives the new configuration on the HTTP response and compares it to the expected configuration. If the configuration from the server matches the expected configuration, then the client will continue sending events; otherwise, the test fails.
The use of tokens is needed because the client may run tests in parallel, rather than sequentially, which would require multiple SCXML sessions to be loaded on the server simultaneously, and the token is thus needed to distinguish them.
Here is an example of the JSON sent over the wire when running test basic/basic1.scxml:
Client request to load statechart.
Server response with token and initial configuration
Client request to send event to statechart associated with token 1
Server response with next state configuration
An example of an SCXML test server implementation can be found in the SCION project here.