Eden Testing Framework
This is a small testing framework for programs that behave as pure functions, allowing you to define these tests in plain markdown format. As a result, your tests become highly accessible to the public.
FAQ
What do you mean with "pure functions"?
We mean that, like an algrebraic mathematical function, the program's output is
fully defined by its input. In other words, if you provide your program with the
input foo and it outputs baz, you can be assured that it will always output
baz given foo.
What are examples of programs that behave as pure functions?
Good examples of programs that behave as pure functions are parsers and lexers. Yet other, more advanced examples include evaluators that evaluate mathematical functions. To accomodate testing these kinds of programs, this project was started.
How does it work?
The idea is to re-use as much of the already existing tools as possible.
Therefore, test cases are just shell commands that recieve as input some data
and output some data, which is then compared with an expected value. This way,
the test commands are nothing more than string -> string-functions. It is up
to you to write a small wrapper around whatever it is that you want to test
that provides such a function.
Installation
You will need a recent version of NodeJS installed together with a version of npm or yarn. Then you will be able to install the application using something similar to:
npm i -g ete
If you $PATH is set up correctly, you can run your tests by simply running
ete in the shell. It will automatically scan for a test directory and add
all test files to it. The full list of flags and commands that are supported
can be consulted with ete --help.
Examples
There are several example test suites that show you what this library is capable of. They can be found in the test/ directory.
Defining Your Tests
First note that you can put plain Markdown text as you would normally do, and it won't interfere with the tests. Use this to your advantage to create descriptive tests that clearly tell the user what it is that is being tested.
Specifying a Test Command
First of all, you need to explicitly tell ete how to run the tests. This is
done by creating a header titled Command somewhere at the top of your file.
It should contain a code block with shell specified as the language with the
command that you'd like to run, like so:
## Command
This is the command that will be used to run tests in this test suite:
```shell
my-parser --read-from-stdin
```
If your command to run differs depending on the platform that the user is
running ete from, then you can replace shell with the name of the specific
shell that is available on that platform. Currently, you can choose from the
following shells:
- dos
- bash
Adding Test Cases
You define a new test by placing some specific code
block under a
separate heading. You can nest headings however you like. ete is smart enough
to only take the title of the header closest to the code block as name of the
test. Next, you will want to define a subheader with the text Result. This
tells ete to compare and diff against any code block that appears
underneath the Result-header.
Here's an example of a test case that could be used by an s-expression parser
wrapped with a program that dumps the AST as NAME [value..]-entries.
## Can parse some s-expressions
```scheme
foo
bar
baz
(one two three)
'a-symbol
'another-long-symbol
```
### Result
```
SYMBOL foo
SYMBOL bar
SYMBOL baz
PAIR
SYMBOL one
PAIR
SYMBOL two
PAIR
SYMBOL three
NULL
QUOTE
SYMBOL a-symbol
QUOTE
SYMBOL another-long-symbol
```
Roadmap
Currently, ete is more of a proof-of-concept than a rigorous testing
framework. Much remains to be done. Here are some ideas:
- Supports other methods of comparison other than ordinary string equality
- Correct handling of multiple code blocks under the same header
- Grouping test cases under their common header (?)
- Programmatic API
License
Copyright 2017 Sam Vervaeck
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.