noflo-runtime-msgflo

NoFlo runtime, designed for use with msgflo. Loads a NoFlo graph and registers it as a msgflo participant. The exported ports of the NoFlo network then becomes accessible through message queues (AMQP or MQTT), and msgflo can coordinate how multiple such networks send data between eachother.

Will eventually allow a single entry-point for FBP runtime protocol clients (like Flowhub) to access NoFlo FBP networks that span multiple nodes.

Usage

Using the commandline tool

noflo-runtime-msgflo --name myworker --graph project/WorkerGraph --broker amqp://foo.cloudamqp.com/bar

Altenatively one can use the embedding API, see src/mount.coffee

runtime = require 'noflo-runtime-msgflo'
options =
  name: 'gssmeasure'
  graph: 'the-grid-api/PrecomputeGss',
  basedir: __dirname
  broker: config.amqp.url,
rt = new runtime.mount.Mounter options
rt.start (err, rt) ->
  # started

Debugging

noflo-runtime-msgflo supports flowtrace allows to trace & store the execution of the NoFlo network, so you can debug any issues that would occur. You can enable tracing using --trace commandline argument, or via the trace:start FBP runtime protocol message.

noflo-runtime-msgflo --graph project/MyMainGraph --trace

To trigger dumping a flowtrace locally, send the SIGUSR2 Unix signal

kill -SIGUSR2 $PID_OF_PROCESS
... Wrote flowtrace to: /tmp/1151020-12063-ami5vq.json

Or, to trigger a flowtrace remotely, send a trace:dump FBP protocol message to the queue of participant.

msgflo-send-message --queue .fbp.$participantId.receive --json '{ "protocol":"trace", "command":"dump", "payload":{ "graph":"default", "type":"flowtrace.json"} }'

Then, assuming .fbp.$participantId.send has been bound to queue my-flowtraces, one could download it

msgflo-dump-message --queue my-flowtraces --amount 1

You can now use various flowtrace tools to introspect the data. For instance, you can get a human readable log using flowtrace-show

flowtrace-show /tmp/1151020-12063-ami5vq.json

-> IN repeat CONN
-> IN repeat DATA hello world
-> IN stdout CONN
-> IN stdout DATA hello world
-> IN repeat DISC
-> IN stdout DISC

Optimizing startup time

Since NoFlo 0.7 FBP manifest is used, making it possible to cache components which turns process startup faster due to less disk IO. To cache graph's components, first install noflo and run noflo-cache-preheat to create the FBP manifest file fbp.json. Then run the graph passing the cache parameter:

noflo-runtime-msgflo --cache true --name myworker --graph project/WorkerGraph --broker amqp://foo.cloudamqp.com/bar

TODO

0.3

• Implement FBP protocol over msgflo transport