Dataflow programming


Dataflow programming

dflow is a minimal Dataflow programming engine.

Table Of Contents:

With npm do

$ npm install dflow

or install globally if you want dflow cli in your path

$ npm install dflow -g

With bower do

$ bower install dflow

and it will install only the engine, client side.

The following simple graph is executed client side by dflow engine.

See test/page.html for another working example of dflow in a browser context.

Node.js Stream Playground first example is

var fs = require("fs");
// Read File 
    // Write File 

It is ported to script stream.js which evaluates graph stream.json using few custom functions.

The main advantage of dflow design is that you do not need to write components or plugins to extend it. You can use one of the most powerful JavaScript features: functions. Write your functions or import them from other packages like JQuery or underscore and you are able to use them as tasks and connect them with pipes.

Also every dflow graph is a function itself, so why not packaging it and put it on npm!?

It is really easy: create your dflow graph and save it to a JSON file, index.json for instance; then launch npm init as usual and when prompted for the entry point write index.json.

Simple as that, see packagedGraph as an example.

Launch dflow from command line, and start editing your first graph using your favourite browser.

Usage: dflow [path/to/graph.json]

If no graph is given, an empty graph named graph.json will be created.

Open your browser and go to http://hostname-where-you-launched-dflow.example.org:3000.

Double click on the SVG canvas to open a text input where you can write the task name you want to create.

Click on a task to select it: addInput, addOutput and deleteTask buttons will appear.

Drag an output into an input to create a pipe.

Click on a pipe to delete it.

dflow exports a fun function ☺

It has the following signature.

 * Create a dflow function.
 * @param {Object} graph to be executed
 * @param {Object} [additionalFunctions] is a collection of functions
 * @returns {Function} dflowFun that executes the given graph

dflow.fun(graph) returns a dflowFun function which has also a graph property, to remember that it is a function generated by dflow.

Actually dflow exposes also the following functions, available only inside a dflow graph:

A graph is a collection of tasks and pipes that can be stored in JSON format.

Every task refers to a function which output can be piped as an argument to another task.

A graph has the following properties

  • task: collection of function names.
  • pipe: connections from the output of a task to an input of another task.
  • data: (optional) persistence object.
  • func: (optional) collection of subgraphs.
  • view: (ignored) object containing information used by flow-view to render a graphic representation of the graph.
  • info: (to be defined) meta data, like author, version and, above all, doc which is a dflow graph itself.

dflow provides few builtin functions and injects the following ones

  • return: a task that accepts one argument and behaves like a Return statement.
  • arguments: task that returns the arguments of dflowFun.
  • arguments[0] ... arguments[N]: tasks that return the arguments[i] of dflowFun.
  • this: refers the dflowFun function.
  • this.graph: contains the graph itself.
  • @foo: accessor to graph.data.foo.
  • &bar: returns bar function.
  • .quz, .quz(): returns a dot-operator-like function.
  • any taskName found in global context, walking throw dot operator, is available as a task. For example
    • isFunction
    • Math.cos
    • process.version: will resolve to a function that returns it

Note that optional collection of additionalFunctions, in order to avoid conflicts with injected functions, must contain function names validated by following the rules:

  • cannot be the name of an injected function: return, arguments, arguments[0] ... arguments[N], this and this.graph are reserved names.
  • cannot start with a dot, ampersand or at sign: names @foo, &bar, .quz and .quz() for an additional function are not allowed.

dflow is MIT licensed.

It is developed in my spare time and, as far as I know, by now I am my only user.

I wrote few times a dataflow engine, the first one was PNI (Perl Node Interface) and the design evolved until I could say confidently that dflow is here to stay.

Use cases I can think about dflow right now are many, but, the possibilities are I.M.H.O. outstanding: from client to server, from JavaScript to cross language, from mono-thread to graphs distributed on a network and, above all, from skilled programmer who implement functions … to artists, genetic engineers, data scientists, etc. that use those functions to create dflow graphs and get results nobody could even imagine.

If this is also your vision or you just want to use dflow, contact me.

My goal is to say to a dflow user:

Mamma mia! Did you achieve that with dflow?