Orca: Distributed load testing
Orca provides tools and a simple protocol for running distributed load tests of any kind of service. You supply an NPM-installable package that implements the load test, and Orca arranges for its performance across the desired number of test clients.
Orca also provides a REST API service for inspecting results and a web application for visualizing them.
Test packages
An Orca test package is simply a Node.js module which exports a test class. The test class must implement a run method, in which the load testing work is performed. Orca test clients install the package at runtime, using a reference provided by the test orchestrator. Any package reference supported by NPM will work.
Orca in action
Load testing is managed by two kinds of components
- Leader: orchestrates the distributed load test and stores the results.
- Nodes: perform the actual load testing.
Orca conducts a load test in three distinct phases:
Announce
The leader publishes a message announcing a load test, then waits for the required number of nodes to respond. If too few nodes respond within a reasonable amount of time, the leader aborts.
Prepare
The leader publishes a message describing the test package and options. Nodes install the package and instantiate the test class with the provided options. Nodes report success or failure of the preparation stage to the leader, which aborts the test run on any failures.
Run:
A full test consists of several upward steps in concurrency. For each level, the leader publishes a message instructing the nodes to generate load at the specified concurrency. The leader then waits until all nodes have completed the test and reported their results before moving on to the next step.
When all steps have been completed, the leader stores the combined results in MongoDB and prints to stdout a BSON query sufficient to locate the results.
System software dependencies
- node.js 0.8.x
- ruby 1.9.x: Rake is used for development, testing, and build tasks
- redis: message transport between components
- mongodb: test result storage
Usage
Run npm install in the project's top level directory to install the required dependencies.
Leader
bin/lead -c path/to/config.cson -t path/to/test_spec.cson
A command line tool which orchestrates the performance of the distributed load testing. The name field of the test specification determines which pubsub channel will be used to communicate with the clients.
Distributed Nodes
bin/node -c path/to/config.cson -n test_name
The agents that actually run the load tests. The test_name flag determines which pubsub channel will be used for communicating with the leader.
API Server
bin/api_server -c path/to/config.cson
Runs an HTTP service which dispatches requests as tasks in message queues. Workers take the tasks, process them, and return results to the API Server, which uses the results to craft HTTP responses.
API Worker
bin/tests_worker -c path/to/config.cson
Retrieves test results from MongoDB and formats them as needed. This may include aggregation and analysis.
Orca Web
A rich web application for displaying test results, consisting of HTML/CSS/JS only, and can be served by any web server (e.g. Apache, nginx). The application communicates with the Orca reporting API, which means it needs to know where you are running the API server. This is accomplished by providing a configuration file at build time.
Orca Web configuration file
In the root of the Orca project create a new file inside the orca/web/ directory called web/configuration.json
{
"service":{
"url":"http://localhost:8000"
}
}
Building the Orca Web files
In the root of the Orca project run rake build:web which will output the website files to orca/build/web
Configure your web server appropriately to serve the orca/build/web files
Configuration files
Orca uses configuration files in CSON format, which is like JSON, but with CoffeeScript syntax. No more brace or comma woes, and comments are allowed.
Environment configuration file config.cson:
redis:
host: "localhost"
port: 6379
mongo:
host: "localhost"
port: 27017
database: "orca"
api:
port: 8000
service_url: "http://localhost:8000"
Test specification configuration file test_spec.cson:
name: "example_test"
description: "A trivial HTTP request"
quorum: 2
repeat: 3
step: 16
timeout: 5000 # 5 seconds
package:
# a reference usable by `npm install`
reference: "http://github.com/dyoder/orca-test/tarball/master"
# the module name to require
name: "orca-test"
options:
# Options to provide to the test module's run method
# Here, it's merely the location of the service to be load tested.
service:
url: "http://localhost:1337"