ete

Eden Testing Framework

This is a small testing framework for programs that behave as pure functions, allowing you to define these tests in plain markdown format. As a result, your tests become highly accessible to the public.

FAQ

What do you mean with "pure functions"?

We mean that, like an algrebraic mathematical function, the program's output is fully defined by its input. In other words, if you provide your program with the input foo and it outputs baz, you can be assured that it will always output baz given foo.

What are examples of programs that behave as pure functions?

Good examples of programs that behave as pure functions are parsers and lexers. Yet other, more advanced examples include evaluators that evaluate mathematical functions. To accomodate testing these kinds of programs, this project was started.

How does it work?

The idea is to re-use as much of the already existing tools as possible. Therefore, test cases are just shell commands that recieve as input some data and output some data, which is then compared with an expected value. This way, the test commands are nothing more than string -> string-functions. It is up to you to write a small wrapper around whatever it is that you want to test that provides such a function.

Installation

You will need a recent version of NodeJS installed together with a version of npm or yarn. Then you will be able to install the application using something similar to:

npm i -g ete

If you $PATH is set up correctly, you can run your tests by simply running ete in the shell. It will automatically scan for a test directory and add all test files to it. The full list of flags and commands that are supported can be consulted with ete --help.

Examples

There are several example test suites that show you what this library is capable of. They can be found in the test/ directory.

Defining Your Tests

First note that you can put plain Markdown text as you would normally do, and it won't interfere with the tests. Use this to your advantage to create descriptive tests that clearly tell the user what it is that is being tested.

Specifying a Test Command

First of all, you need to explicitly tell ete how to run the tests. This is done by creating a header titled Command somewhere at the top of your file. It should contain a code block with shell specified as the language with the command that you'd like to run, like so:

## Command

This is the command that will be used to run tests in this test suite:

```shell
my-parser --read-from-stdin
```

If your command to run differs depending on the platform that the user is running ete from, then you can replace shell with the name of the specific shell that is available on that platform. Currently, you can choose from the following shells:

• dos
• bash

Adding Test Cases

You define a new test by placing some specific code block under a separate heading. You can nest headings however you like. ete is smart enough to only take the title of the header closest to the code block as name of the test. Next, you will want to define a subheader with the text Result. This tells ete to compare and diff against any code block that appears underneath the Result-header.

Here's an example of a test case that could be used by an s-expression parser wrapped with a program that dumps the AST as NAME [value..]-entries.

## Can parse some s-expressions

```scheme
foo
bar
baz
(one two three)
'a-symbol
'another-long-symbol
```

### Result

```
SYMBOL foo
SYMBOL bar
SYMBOL baz
PAIR
SYMBOL one
PAIR
SYMBOL two
PAIR
SYMBOL three
NULL
QUOTE
SYMBOL a-symbol
QUOTE
SYMBOL another-long-symbol
```

Roadmap

Currently, ete is more of a proof-of-concept than a rigorous testing framework. Much remains to be done. Here are some ideas:

• Supports other methods of comparison other than ordinary string equality
• Correct handling of multiple code blocks under the same header
• Grouping test cases under their common header (?)
• Programmatic API

License

Copyright 2017 Sam Vervaeck

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.