snurra

Snurra

Snurra is a fresh adaptation of the message bus pattern by using streams as the interface instead of the more traditional callback pattern.

Streams: A tiny crash course

The evolution of the JavaScript culture has gone from callbacks to events to promises. Streams are the next level of goodness.

Read and output a file with callbacks;

fs.readFile('/data.txt', function (err, data) {
  process.stdout.write(data);
});

Same thing with streams:

fs.createReadStream('/data.txt').pipe(process.stdout);

When you program with streams, you are like a plumber connecting pipes together. In the above line of code, a read stream for the data.txt file is created, and then piped to the process.stdout stream.

Streams are historically used for reading and writing big files, often over the network, but recently many programmers have started making use of them as a general programming pattern.

Highland.js: An even tinier crash course

Highland.js (created by the author of the async library) is an amazing library that let's you do filter, map and other coolness on streams. Think of it like an underscore but with streams instead of arrays, or a library that does for streams what jQuery did for the DOM:

var _ = require('highland');
function isBlogPost(doc) {
  return doc.title && doc.body;
}
var docs = database.createReadStream();
var output = fs.createWriteStream('blogposts_only.txt');
_(docs)
  .filter(isBlogpost)
  .map(function(blogpost) {
    return JSON.stringify(blogpost)
  })
  .pipe(output);

The above code is similar to the earlier example, but in this case, we are reading from a database that happens to have a streaming interface, and filter out only blogposts, which we then write to a database.

You'll notice that most examples in this README uses Highland.js. Snurra deals only with vanilla node streams and doesn't require you to use highland.js whatsoever - they just play nice together and make these examples much more readable and demonstrative of the power of this way of programming.

For a more elaborate example of how to program with streams, check out Solving Coding Challenges with Streams and if you're still interested in learning more after that, go download the educational stream adventure. Streams are a big subject, and it's still in it's infancy in the JavaScript world. Welcome to the front, soldier!

What is a message bus and why should I have one?

Over time, software becomes more complex. In order to make complex software easier to reason about, we divide the software into smaller, individual parts. We want to make these parts as unaware of each other as possible, in order to allow us to reason about one problem at a time, and also to give us the ability to easily replace individual parts of the software.

One popular trick for achieving this decoupling is to use a message bus. A message bus is a thing that different parts of your program communicates through, by sending simple messages, instead of communicating with each other directly.

Snurra Syntax

Broadcast streams

The first and most fundamental Snurra concept that you need to know about are broadcast streams.

A broadcast stream works like a walkie-talkie; when you write a value to a broadcast stream, that value will be received by readers of all other broadcast streams on the same channel.

The code below broadcasts a user object on the channel 'user-change'. Then, the broadcasted object is received by the last part of the code, which uses it to update the the UI.

var _ = require('highland');
var snurra = require('snurra');
var bus = snurra();
 
bus.broadcast( 'user-change' ).write( { name : 'Dr. Wafflehat' });
 
_(bus.broadcast( 'user-change' )).each(function(change) {
  $( '#example1' ).html( 'Name: ' + change.name );
})

In the code above, the first thing we do is to create a bus:

var bus = snurra();

The bus is a space for channels to live in - think of it as a tiny telephone system, or a discussion forum for code. After declaring the bus, we create broadcast stream on the channel of user-change, and write to it:

bus.broadcast( 'user-change' ).write( { name : 'Dr. Wafflehat' });

We need to actually have something to receive this broadcast and do something with it, so we create another broadcast stream on the same channel and then render values on that stream:

_(bus.broadcast( 'user-change' )).each(function(change) {
  $( '#example1' ).html( 'Name: ' + change.name );
})

Notice that we can start listening after writing. This is a feature of Snurra - the bus will actually defer all writes until the next run loop so that we can write code like the above.

Channels are always strings, but the values written to them can be anything that can be serialized to JSON. Note that this means that you cannot send functions in any form through the bus; Snurra is opinionated and considers it to be naughty to send behavior as messages.

Log output

The killer feature of Snurra is that it logs everything that happens, so that you can inspect what values are being sent on what channels, giving you a much better sense of what is happening in your app, and in what order. This makes it a lot easier to see where things go wrong. The most basic example of this is outputting everything that happens as to your terminal:

_(bus.log()).map(console.log)

Example log entry:

{
  receiver: 'broadcast',
  channel: 'my-channel',
  value: {
    myProperty: 123
  }
}

The log is also very handy when writing unit tests, often completely removes the need of mocking.

Spy streams

It's really handy to have the log as a debugging tool, but it will only show you stuff sent through Snurra, and in some cases, the bug you're looking for is in a chain of non-snurra streams. In this situation, you can can create a spy stream that you pipe things through in the pipeline. It will simply log any values it receives and then pass it on:

_(bus.broadcast('some-channel'))
  .map(function(value) {
    // we suspect there might be some bug in this logic
  })
  .through(bus.spy('debug-mapper-output'))
  .filter(function(value) {
    // do some other stuff, will will get the return
    // value from the .map
  })
  .pipe(bus.broadcast('some-other-channel'))

Example log entry:

{
  receiver: 'spy',
  name: 'debug-mapper-output', // Optional, but handy
  value: /* result from map function will be here */
}

Request streams and responders

When you write a value to a broadcast stream, the code doing the writing has no idea what happens after it writes. This is a great feature of the message bus pattern, which decouples your app components from eachother, but in some cases, you actually do need the result. For these cases, you can use request streams instead broadcast streams.

In the code below, we send a command through the adder request stream, and then write the result to console. We then create the responder.

 
_([{
  left: 3,
  right: 5
}])
  .through(bus.responder('addition'))
  .each(function(result) {
    console.log("3+5 is:", result)
  })
 
bus.responder('addition', function(input) {
  _(input).map(function(command) {
    return command.left + command.right
  })
})

Example log entry:

{
  receiver: 'responder'
  channel: 'addition',
  input: {
    left: 3,
    right: 5
  },
  output: 8
}

This code creates a requester stream on the additon channel:

bus.requester('addition')

When you write a message to a requester stream, it will pipe that message through the responder registered on the same channel as the requester, and then pipe the result out the other end of the requester stream.

For this to work, we must also define a responder on the same channel:

bus.responder('addition', function(input) {
  _(input).map(function(command) {
    return command.left + command.right
  })
})

By passing a second argument, a function which accepts a stream, input, and then returns a new stream, we define what will happen to values sent to responders on the addition channel.

Unlike broadcast streams, responder streams can only have one responder per channel (it would be nonsensical to get multiple responses to a single request). Request streams also do not affect eachother like broadcast streams do - only the request stream that received the request will pipe out the response to that request. No other request streams will pipe out that particular response, even if they are on the same channel.

Advanced filtering

In case that you want to do some more complex subscribing than than just listening to a broadcast, you can snoop bus.log() to do things based on what happens in snurra.

_(bus.log()).filter(function(logEntry) {
  // Do stuff based on the channel name or other things.
})

Do note that filtering based on log entry will be coupled to Snurra, so take care when using it so that most of the streams in your main program stays unaware of Snurra.

Beyond this, Snurra doesn't offer any additional, built-in filtering functionality. The rationale for this is that the streaming interface allows the user of Snurra to use the high-level stream libraries like highland.js which are so capable that weighing down Snurra with filtering capabilities would not make sense:

var unhandledOrders = _(bus.broadcast('orders'))
  .filter(function(order) { return !order.shippingdate });
 
unhandledOrders.each(function(order) {
  // do stuff with order that is not handled
})