Modular queue system for node.js
A modular queue server for node.js
- Built on ES6 Harmony generators
- Support for multiple queues
- Completely modular
- Pluggable data store
- Process jobs using distributed workers over http, tcp or ipc
npm install katar --save
katar and its subcomponents harness the full power of ES6 generators and as such require node 0.11.9 or higher, and must run node with the --harmony flag. If you don't like typing this, add an alias to your shell profile:
alias node='node --harmony'
Here's an example of a complex queue server that lets you create multiple queues and use HTTP polling workers
// use mongo for storing tasksvar katarDb = require'katar-mongodb''localhost/queue';// initialise a katar servervar katar = require'katar' db: katarDb ;// create a data queue that fetches data from a urlvar dataQueue = katarqueue'fetch-data';// create a youtube queue that downloads videos from youtube// this queue is persistent - this means once a task is completed, it is retained in the database for historical reasonsvar ytQueue = katarqueue'youtube' persistent: true ;// insert a data taskdataQueueinsertdata: '';// insert a data task with high priority// supported values - HIGH, MEDIUM, LOWdataQueueinsertdata: ''priority: katarconstantspriorityHIGH;// insert a youtube request with an object data type// all data types supported by the data adapter are supported as data is passed straight to the adapterytQueueinsertdata:url: ''res: '720p';// create a http server that workers can poll and fetch jobs// server is a koa-framework objectvar workerServer = require'katar-worker-http'katar: katarport: 3000;// add any custom routes as needed// see koa-framework npm module for usageworkerServerapiv1routerget'/hello'thisbody = hello: 'world' ;;
The queue system can be backed by any database. Simply create a data adapter that has the required methods
npm install katar-memorydb
npm install katar-mongodb
To simplify the task of creating data adapters, a test suite is published that can be used to test the functionality of the adapter meets the needs of katar.
npm install katar-db-test
Katar allows workers to subscribe for new jobs, poll the server or push new jobs to the server
Create a server that listens for worker requests. Workers can poll the server on a timely basis to fetch new jobs and report when jobs have been finished.
npm install katar-worker-http
- Initial release