RDF-Test-Suite.js
This tool executes the RDF and SPARQL test suites with any given system.
It can either output human-readable test results to the console, or it can output machine-readable reports in the EARL vocabulary.
Currently, the following test suites are supported:
- SPARQL 1.0
- SPARQL 1.1
- RDF/XML
- N-Triples
- N-Quads
- Turtle
- TriG
- JSON-LD (1.0 and 1.1)
- RDFa
- Microdata to RDF
- Notation3
- RDF-star and SPARQL-star
- JSON-LD-Star
Not all RDF test suites are supported at the moment. However, this package is fully modular, so that support for different test cases can be implemented easily.
Planned support:
- SPARQL 1.1 (other specifications)
- RDF Schema and Semantics
- JSON-LD (other test suites)
Installation
Either install it globally:
$ yarn global add rdf-test-suiteOr locally (as a dev dependency):
$ yarn add --dev rdf-test-suiteUsage
After installing, the rdf-test-suite script will become available.
This script requires some kind of engine as first argument, and a test suite manifest URL as second argument.
The engine is a JavaScript file that can handle the test suite.
The interface of this engine depends on the manifest.
For example, the SPARQL 1.1 test suite requires an
IQueryEngine,
while the RDF/XML test suite requires a
IParser.
Basic execution
The following command executes the SPARQL 1.1 test suite
on the engine myengine.js:
$ rdf-test-suite myengine.js http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttlAn example of what this myengine.js looks like
is available for the Comunica SPARQL engine.
This command will output something like this:
...
✔ syn-bad-pname-04 (http://www.w3.org/2009/sparql/docs/tests/data-sparql11/syntax-query/manifest#test_pn_bad_04) 0.293588ms
✔ syn-bad-pname-05 (http://www.w3.org/2009/sparql/docs/tests/data-sparql11/syntax-query/manifest#test_pn_bad_05) 0.498646ms
✔ syn-bad-pname-06 (http://www.w3.org/2009/sparql/docs/tests/data-sparql11/syntax-query/manifest#test_pn_bad_06) 0.513092ms
✔ syn-bad-pname-07 (http://www.w3.org/2009/sparql/docs/tests/data-sparql11/syntax-query/manifest#test_pn_bad_07) 0.498646ms
✔ syn-bad-pname-08 (http://www.w3.org/2009/sparql/docs/tests/data-sparql11/syntax-query/manifest#test_pn_bad_08) 0.293588ms
✔ syn-bad-pname-09 (http://www.w3.org/2009/sparql/docs/tests/data-sparql11/syntax-query/manifest#test_pn_bad_09) 0.293588ms
✔ syn-bad-pname-10 (http://www.w3.org/2009/sparql/docs/tests/data-sparql11/syntax-query/manifest#test_pn_bad_10) 0.516346ms
✔ syn-bad-pname-11 (http://www.w3.org/2009/sparql/docs/tests/data-sparql11/syntax-query/manifest#test_pn_bad_11) 0.498646ms
✔ syn-bad-pname-12 (http://www.w3.org/2009/sparql/docs/tests/data-sparql11/syntax-query/manifest#test_pn_bad_12) 0.516346ms
✔ syn-bad-pname-13 (http://www.w3.org/2009/sparql/docs/tests/data-sparql11/syntax-query/manifest#test_pn_bad_13) 0.498646ms
✔ syn-pp-in-collection (http://www.w3.org/2009/sparql/docs/tests/data-sparql11/syntax-query/manifest#test_pp_coll) 0.516346ms
✖ 105 / 268 tests succeeded!
Test filtering
With the -t option, you can filter tests that should be executed
based on a regex that will be matched with the test URI.
For example:
$ rdf-test-suite myengine.js http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl \
-t test_pn_bad_0Test skipping
With the --skip option you can filter tests that should be skipped
based on a regex that will be matched with the test URI.
For example:
$ rdf-test-suite myengine.js http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl \
--skip test_pn_bad_0$ rdf-test-suite myengine.js http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl \
--skip (test_pn_bad_0|test_pn_bad_1)Summarized output
By default, the printed test results are verbose, and print expanded views of failed tests.
You can change this to a more compact view using -o summary.
For example:
$ rdf-test-suite myengine.js http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl \
-o summaryEARL output
Test results are human readable by default. This can be changed to output machine-readable reports in the EARL vocabulary in the Turtle serialization. These reports can be published to report the compliance of engines to certain specifications, like on W3C’s RDF Test Curation Community Group.
As EARL reports require some metadata on your engine,
you will need to provide a properties file via the -p argument.
This properties file can look something like this:
{
"applicationBugsUrl": "https://github.com/comunica/comunica/issues",
"applicationDescription": "A Comunica engine for SPARQL query evaluation over heterogeneous interfaces",
"applicationHomepageUrl": "http://comunica.linkeddatafragments.org/",
"applicationNameFull": "Comunica SPARQL",
"applicationNameNpm": "@comunica/actor-init-sparql",
"applicationUri": "https://www.npmjs.com/package/@comunica/actor-init-sparql",
"authors": [
{
"homepage": "https://www.rubensworks.net/",
"name": "Ruben Taelman",
"uri": "https://www.rubensworks.net/#me"
}
],
"licenseUri": "http://opensource.org/licenses/MIT",
"reportUri": null,
"specificationUris": [
"http://www.w3.org/TR/sparql11-query/"
],
"version": "1.2.3"
}The properties file can point to a non-existing file,
in which case it will be auto-generated from the package.json
file in the current working directory.
For example:
$ rdf-test-suite myengine.js http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl \
-o earl -p earl-meta.jsonRestricting execution to certain specifications
Some test suites (like the one of SPARQL 1.1) contains
tests for multiple specifications.
If you only want to test your system for a single specification,
then you can define this with the -s parameter.
For example, the following command executes the SPARQL 1.1 test cases that apply to the http://www.w3.org/TR/sparql11-query/ specification.
$ rdf-test-suite myengine.js http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl \
-s http://www.w3.org/TR/sparql11-query/Enabling HTTP caching
By default, rdf-test-suite will look up all required manifest files.
If this takes too much time, or if you don't have internet connection,
then you can enable HTTP caching with the -c argument,
so that all files only have to be looked up once.
$ rdf-test-suite myengine.js http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl \
-c path/to/cache/If you don't provide a caching value after the -c,
then the directory will default to .rdf-test-suite-cache/.
Ignore exit code
When there are failing tests,
rdf-test-suite will exit with code 1 instead of 0.
This can be useful in Continuous Integration tools.
If you want to disable this behaviour,
you can add the -e flag to force an exit code 0.
$ rdf-test-suite myengine.js http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl \
-ePass custom options to the engine
With the optional -i option,
a JSON string with arguments can be passed to the engine.
These arguments will become available in
both IQueryEngine,
and IParser.
$ rdf-test-suite myengine.js http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl \
-i '{ "myProperty": "myValue" }'Test timeouts
By default, all tests are allowed to run 3000 ms.
Using the -d option, you can change this value.
$ rdf-test-suite myengine.js http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl \
-d 5000Map URLs to local files
In cases where you are developing your own test manifests, it may be useful to intercept certain URL lookups and return local files instead.
In order to achieve this, the -m option can be used,
with a mapping defined with the following pattern: URL~PATH.
$ rdf-test-suite myengine.js http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl \
-m http://w3c.github.io/rdf-tests/sparql11/data-sparql11/~/path/to/my/files/
Using rejected tests
In cases where you wish to execute a test suite that contains
rdft:Rejected test entries rdf-test-suite will skip those tests
by default. If you wish to execute those tests as well, you can
use the -r option.
$ rdf-test-suite myengine.js http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl \
-rOnly using explicitly approved tests
In cases where you wish to execute a test suite that contains
rdft:Approved test entries rdf-test-suite will, by default
execute those tests as well as all other tests that are not
explicitly rejected. If you wish to only execute those tests
that are explicitly approved, you can use the -a option.
$ rdf-test-suite myengine.js http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl \
-aSupported test suites
License
This software is written by Ruben Taelman.
This code is released under the MIT license.