Modular queue system for node.js
A modular queue server for node.js
npm install katar --save
katar and its subcomponents harness the full power of ES6 generators and as such require node 0.11.9 or higher, and must run node with the --harmony flag. If you don't like typing this, add an alias to your shell profile:
alias node='node --harmony'
Here's an example of a complex queue server that lets you create multiple queues and use HTTP polling workers
// use mongo for storing tasksvar katarDb = 'localhost/queue';// initialise a katar servervar katar = db: katarDb ;// create a data queue that fetches data from a urlvar dataQueue = katar;// create a youtube queue that downloads videos from youtube// this queue is persistent - this means once a task is completed, it is retained in the database for historical reasonsvar ytQueue = katar;// insert a data taskdataQueue;// insert a data task with high priority// supported values - HIGH, MEDIUM, LOWdataQueue;// insert a youtube request with an object data type// all data types supported by the data adapter are supported as data is passed straight to the adapterytQueue;// create a http server that workers can poll and fetch jobs// server is a koa-framework objectvar workerServer =katar: katarport: 3000;// add any custom routes as needed// see koa-framework npm module for usageworkerServerapiv1router;
The queue system can be backed by any database. Simply create a data adapter that has the required methods
npm install katar-memorydb
npm install katar-mongodb
To simplify the task of creating data adapters, a test suite is published that can be used to test the functionality of the adapter meets the needs of katar.
npm install katar-db-test
Katar allows workers to subscribe for new jobs, poll the server or push new jobs to the server
Create a server that listens for worker requests. Workers can poll the server on a timely basis to fetch new jobs and report when jobs have been finished.
npm install katar-worker-http