cockrel

Cockrel.js

Cockrel is a library for building durable asynchronous processing chains. It is plugin extensible and has a strong focus on durability via configurable queues.

Think of Cockrel as the glue between your wonderfully isolated intranet of things.

Builtin steps

• .do(fn || chain, args) Execute a sync/async function or sub-chain.
• .map(chain) Execute a sub-chain over an array.
• .as(key, chain) Saves the result of a sub-chain AS key on previous result.
• .pick(Object) Transform an object into a different object.
• .when(condition, chain) Execute a sub-chain when condition is met.
• .begin(data) Begin processing the chain, returns a promise.
• .end(bool) Terminate the chain at this point, as success or error.
• .error(chain) Handle an error from the previous step with a sub-chain.
• .debug() Pretty prints the previous value then returns it as is.

TODO steps

• .filter(Object) Filter an Array.
• .queue(config, chain) Like do but with configurable queue management and retry behaviour.

Other step plugins

Object validation

• cockrel-mog validate objects with mog

Webserver integration

• cockrel-middleware use cockrel chains as express middleware

Pub/sub integration

• TODO cockrel-sqs-pubsub Use cockrel chains as AWS SQS subscribers
• TODO cockrel-redis-pubsub Use cockrel chains as redis pub/sub subscribers

Network / RPC integration

• cockrel-request Make HTTP requests
• cockrel-ht-client Invoke HT APIs from your chains

Concepts

It's about chainable Promises

Cockrel implements chainable 'steps' which perform a function and then pass their output on to the next step in the chain. Cockrel works best when passing objects however any kind of data can be passed. Step methods are implemented as plugins and it's straight forward to add your own.

co.do(myFunc)
  .do(myOtherFunc) ...

Kicking off processing

All chains begin processing when their .begin step is invoked. This step is special in that it returns a Promise, no further steps can be chained.

co.do(myFunc)
  .do(myOtherFunc)
  .begin({ animal : 'cat'}); // -> Promise

'Picking' params

Many steps take params, and frequently these params will support 'picking' (see .pick for in-depth details). Picking is the process of providing a pick-specification in the form of a string or object, who's value or internal values will be 'picked' from the previous step's response.

co.do(myFunc, { type : '@animal'})
  .do(myOtherFunc)
  .begin({ animal : 'cat'}); 

Sub-chains

Some steps accept sub-chains, these are evaluated just like any other function and can provide branching behaviour etc.

let p = co
  .do(myFunc, { type : '@animal'})
  .as('colour', co.do(getRandomAnimalColor))  // .as takes a sub-chain
  .begin({ animal : 'cat'}); 
 
p.then(console.log);
// { animal : 'cat', colour : 'silver' } 

Tutorial

Here is a quick example of using cockrel chains to perform complex async actions, try this out in your node repl, substitute my name 'g5095' for your github account.

Firstly we'll use the request plugin to get some info about my github account:

const co = require('cockrel');
require('cockrel-request')(co);
 
co.request('https://api.github.com/users/g5095') 
  .debug() // insert a debug step so we can see what we got
  .begin() // start the chain processing

Great, you should see some data about my account on github. One of those is a URL to another endpoint, 'repos_url'. Lets query that as well and see what we get:

co.request('https://api.github.com/users/g5095') 
  .request('@repos_url') // using the repos link in the first data, load repo info
  .debug() // insert a debug step so we can see what we got
  .begin() // start the chain processing

Ok, now we've got a list of my public repositories on github, lets map over those and pick out a few useful bits of information:

co.request('https://api.github.com/users/g5095') 
  .request('@repos_url')
  .map(co.pick({
    name : '@name', 
    language : '@language', 
    watchers : '@watchers',
    url : '@html_url'}))
  .debug()
  .begin()

Nice! now you have a chain that makes some requests, and collates some data.

Lets turn that into a REST api using express and cockrel-middleware that will let anyone get this data for their own github account, we'll use a few more cockrel steps to present the data nicely:

const express = require('express');
const co = require('cockrel');
const mid = require('cockrel-middleware');
require('cockrel-request')(co);
mid(co);
 
const api = express();
 
api.get('/github/:user', mid.begin(
  co.request('https://api.github.com/users/{params.user}') 
    .as('repoData',
      co.request('@repos_url')
        .map(co.pick({name : '@name', language : '@language',
          watchers : '@watchers', url : '@html_url'})))
    .pick({name : '@name', avatar : '@avatar_url', projects : '@repoData'})
    .response('json') ))
 
api.listen(8080);

Now you can hit http://localhost:8080/github/ with an http client.

NOTE: the github APIs will start timing out requests unless you have an API key, so if your server stops working that's why :)

Documentation

Cockrel comes with several buil-in steps to perform common actions:

do

Execute a sync/async function or sub-chain.

The do step takes a callable:

• a vanilla sync function which can throw an error or return a result
• or a Promise
• or a Cockrel chain

Any arguments will be passed into the callable after being interpreted as pickable values.

If no arguments are passed, the previous step's return value will be passed as the first argument.

If the callable is a cockrel chain, only the first argument will be passed to that chain's begin function.

Examples:

call a vanilla JS function with arguments 1, 2, 3:

.do(func, 1, 2, 3)

call a vanilla JS function with the last chained value:

.do(func)

call a vanilla JS function with arguments picked from the last chaned value:

.do(func, '@firstname', '@details.phoneNumber')

or:

.do(func, { name : '@firstname', number : '@details.phoneNumber'})

call a Cockrel chain with the last chained value:

.do(subChain)

call a Cockrel chain with a different value:

.do(subChain, '@firstname')

map

.map takes an array and maps it to the provided chain. When the chains for each item are complete, .map will continue to the next step with an array containing the results of all mapped chains.

Example:

  // some step providing an array of objects ..
  .map(co.do(data => data.name))  ->  ['Jo', 'Leigh', 'Dave', ...]

as

Saves the result of a sub-chain AS key on previous result. .as takes a key and a chain, it then executes the provided chain and returns the previous step's output with the sub-chain's result added 'as' key. If the previous step's result was not an object, a new object is created with the previous result as 'previous'

  ...
  .as('names', co.do(someFunc))
  // result now has names : <result> added

pick

Transform an object into a different object. This is an explicit step which implements the same 'picking' logic available to params, as a transformation step in your chain.

Rules:

• A string value beginning with @ is considered a path and will be replaced with the value from the original object.

• A string consisting only of @ will be replaced with the entire value of the original object.

• A string containing {paths.in.handlebars} will be processed with the paths looked up from the original object.

• Anything else will be left as it is.

Examples:

Assuming the previous value passed was:

 { test : { foo : 1 }, test2 : 'hello'}

then .pick instructions:

  .pick({ greeting : '@test2', number : '@test.foo' })

The value is now:

  { greeting : 'hello', number : 1 }

When picking you can simply provide a string if you only want a single value, for instance '@foo.bar' will be replaced by the value foo.bar, '@' will match the entire previous value, 'hello world!' will remain 'hello world!', while 'hello {thing.to.greet}!' will be replaced with the value matching that path.

when

Execute a sub-chain when condition is met.

when takes an expression as a string, and a sub-chain, and evaluates the expression against the data provided by the previous step. If the expression evaluates truthfully the sub-chain is executed and it's value passed down to the next step, otherwise the previous value is passed on and the sub-chain is not executed.

Example:

  co.map(co.when('fruit.kind == "citrus"', co.debug())) // debug all citrus fruit 
  .begin([{kind : 'citrus', name : 'lemon'}, {kind : 'ficus', name : 'fig'}])

When uses angular-expressions to evaluate the expression part. Most obvious expressions available to javascript will evaluate as expected, and the expression is compiled into a function and cached for performance.

error

Handle an error from the previous step with a sub-chain.

.error provides a sub-chain that will be invoked to handle an error condition in the previous step. The provided error sub-chain must return a corrected value which will be passed on to the next step, or call .end() to indicate the entire chain should terminate with the last value as an error.

Example:

Here we have a step that rejects it's promise, the following .error sub-chain will handle the error, and continue to the next do step. If the first step had not rejected, this .error chain would be skipped over.

  .do(thingThatErrors)
  .error(co.do(somethingToCorrectTheError))
  .do(nextStep)

Here the .error chain is terminated with a call to .end() signaling that the error could not be resolved and the following steps should never be called.

  .do(thingThatErrors)
  .error(co.do(logTheError).end())
  .do(nextStepWillNotBeCalled)

end

Terminate the chain at this point, as success or error.

.end(bool) will terminate the chain and immediately fire the chain's original respond/reject handlers with the last step's value. respond if bool is true, reject if bool is false.

begin

Begin processing the chain, returns a promise.

Calling .begin with a value will start processing the chain with that value, for instance:

co.do(console.log).begin('hello world') // -> Promise

Is a rather laborious method of printing hello world, but demonstrates the concept.

Chain state (plugin authors only):

If you are implementing cockrel plugins, you may be wrapping a call to .begin if you are accepting a sub-chain, or wrapping cockrel in another library such as cockrel-middleware which will fire off chains in response to HTTP requests.

In this case, you may be passed some chainState which needs to be passed as the second argument to chain.begin. chainState is an internal bucket of resources which follows the chain, allowing plugins to access items from the beginning of the chain lifecycle.