Manage master + workers with the help of Redis
assemblage is a super simple module to create distributed clusters of worker processes with one or more master.
Master process has an interface to add and remove jobs. Worker processes check in for new jobs and emit information about new jobs received or removed. If master goes down, no new jobs can be added but workers continue working with current jobs unaffected. When a worker goes down, master reassigns all the jobs of the worker to other workers.
All the synchronization between master and workers is done with the help of Redis.
Master shares jobs between clients.
Create master with
var assemblage = require"assemblage";var master = assemblagecreateMaster"my-cluster" host: "localhost";
To add a new job, simply call
masteraddJobkey: "value"console.log"Added new job with ID: %s" jobId;;
To remove previously registered job, simply call
masterdeleteJob"12345"console.log"Deleted job with ID: %s" jobId;;
Worker receives jobs queued by the master.
Create a worker with
var assemblage = require"assemblage";var worker = assemblagecreateWorker"my-cluster" host: "localhost";
Worker can listen for new jobs with
"add" event. The event has one parameter - job object.
workeron"add"console.log"Received new job %s" jobid;console.logjobpayload;;
Job removals with can be listened with
"delete" events on the job object. Whatever this job was doing when a
"delete" event is received, should be terminated.
"delete" is emitted only once, so no need to use
on() method to listen for it, use
jobonce"delete"console.log"Remove job %s" jobid;;
You can remove a job in the worker as well with using
job.deleteJob(callback) - this doesn't actually remove the job, but queues it for deletion.
A worker can be closed and all jobs released for reassignment to other workers with
For all jobs a
"delete" event is emitted as well when terminating
Terminating a worker (either by
terminate method or forced by the expired lock) also emits
workeron"close"console.log"The worker was closed, no new jobs for this one";;
On unexpected errors an
"error" event is thrown.
Install vows and test as usual
$ npm install vows $ npm test