Simple Storm-like distributed application, Work in Progress
Simple Storm-like distributed application, implementation, for Node.js. See
Via npm on Node:
npm install simplestorm
Reference in your program:
var simplestorm = require'simplestorm';
You have Spouts (message sources) and Bolts (message processor). An Spout should have a
// ...contextemitmsg; // you can emit a message many times// ....
An spout emit message via its controller, in any of its methods.
A Bolt has a
// Message process// and emit new message(s)contextemitnewmsg;
There is a topology builder:
// Objectsvar spout = ;var downloader = ;var resolver = ;var harvester = ;// Setting Buildervar builder = sscreateTopologyBuilder;buildersetSpout"spout" spout;buildersetBolt"downloader" downloadershuffleGrouping"resolver"shuffleGrouping"spout";buildersetBolt"resolver" resolvershuffleGrouping"harvester";buildersetBolt"harvester" harvestershuffleGrouping"downloader";var topology = buildercreateTopology;
Start a topology:
Stop a topology (WIP):
A topology worker can listen external messages, sent from other topology workers:
If they are many topology workers, they can share its address using a central server. In a process, create a topology server:
var tserver = sscreateTopologyServer;tserverlistenport;
In each topology worker, connect to the topology server:
When a topology worker connects to the central server, it sends its address. The server shares that address with the rest of the topology workers. See Distributed Web Crawler with Server/Workers for a running example.
git clone git://github.com/ajlopez/SimpleStorm.gitcd SimpleStormnpm installnpm test
- Web Crawler sample shows a local topology running a web crawler.
- Distributed Web Crawler is a central server using queues and many SimpleStorm topologies running in nodes.
- Distributed Web Crawler with Server/Workers A central topology server and many topologies running in worker processes.
- 0.0.1: Published.
- 0.0.2: Published. Using SimpleQueue 0.0.2 for Distributed Web Crawler.
- 0.0.3: Published. Major internal/external refactor. Spout.start, Bold.process. Topology.listen. Topology Server. Task message exchange between running worker processes (in different nodes). Workers sample.
- 0.0.4: Published. Using SimpleMessages 0.0.6
- Implement local ack of message tree
- Implement distributed ack
- Use a queue that can replay a fail message
- Adopt MultiNodes
If you submit a pull request, please be sure to add or update corresponding
test cases, and ensure that
npm test continues to pass.