orcastrike

Distributed load testing for node.js using pirate.

Orca: Distributed load testing

Orca provides tools and a simple protocol for running distributed load tests of any kind of service. You supply an NPM-installable package that implements the load test, and Orca arranges for its performance across the desired number of test clients.

Orca also provides a REST API service for inspecting results and a web application for visualizing them.

An Orca test package is simply a Node.js module which exports a test class. The test class must implement a run method, in which the load testing work is performed. Orca test clients install the package at runtime, using a reference provided by the test orchestrator. Any package reference supported by NPM will work.

Load testing is managed by two kinds of components

  • Leader: orchestrates the distributed load test and stores the results.
  • Nodes: perform the actual load testing.

Orca conducts a load test in three distinct phases:

The leader publishes a message announcing a load test, then waits for the required number of nodes to respond. If too few nodes respond within a reasonable amount of time, the leader aborts.

The leader publishes a message describing the test package and options. Nodes install the package and instantiate the test class with the provided options. Nodes report success or failure of the preparation stage to the leader, which aborts the test run on any failures.

A full test consists of several upward steps in concurrency. For each level, the leader publishes a message instructing the nodes to generate load at the specified concurrency. The leader then waits until all nodes have completed the test and reported their results before moving on to the next step.

When all steps have been completed, the leader stores the combined results in MongoDB and prints to stdout a BSON query sufficient to locate the results.

  • node.js 0.8.x
  • ruby 1.9.x: Rake is used for development, testing, and build tasks
  • redis: message transport between components
  • mongodb: test result storage

Run npm install in the project's top level directory to install the required dependencies.

bin/lead -c path/to/config.cson -t path/to/test_spec.cson

A command line tool which orchestrates the performance of the distributed load testing. The name field of the test specification determines which pubsub channel will be used to communicate with the clients.

bin/node -c path/to/config.cson -n test_name

The agents that actually run the load tests. The test_name flag determines which pubsub channel will be used for communicating with the leader.

bin/api_server -c path/to/config.cson

Runs an HTTP service which dispatches requests as tasks in message queues. Workers take the tasks, process them, and return results to the API Server, which uses the results to craft HTTP responses.

bin/tests_worker -c path/to/config.cson

Retrieves test results from MongoDB and formats them as needed. This may include aggregation and analysis.

A rich web application for displaying test results, consisting of HTML/CSS/JS only, and can be served by any web server (e.g. Apache, nginx). The application communicates with the Orca reporting API, which means it needs to know where you are running the API server. This is accomplished by providing a configuration file at build time.

In the root of the Orca project create a new file inside the orca/web/ directory called web/configuration.json

{
   "service":{
      "url":"http://localhost:8000"
   }
}

In the root of the Orca project run rake build:web which will output the website files to orca/build/web

Configure your web server appropriately to serve the orca/build/web files

Orca uses configuration files in CSON format, which is like JSON, but with CoffeeScript syntax. No more brace or comma woes, and comments are allowed.

redis:
  host: "localhost"
  port: 6379
mongo:
  host: "localhost"
  port: 27017
  database: "orca"
api:
  port: 8000
  service_url: "http://localhost:8000"
name: "example_test"
description: "A trivial HTTP request"
quorum: 2
repeat: 3
step: 16
timeout: 5000 # 5 seconds
package: 
  # a reference usable by `npm install`
  reference: "http://github.com/dyoder/orca-test/tarball/master"
  # the module name to require
  name: "orca-test"
options:
  # Options to provide to the test module's run method
  # Here, it's merely the location of the service to be load tested.
  service:
    url: "http://localhost:1337"