redis-job-queue

Abstraction on top of the redis list data type for queueing data. There are two main classes, QueueConsumer and QueuePublisher. QueueConsumer, as the name suggests, consumer data from a queue, while QueuePublisher publishes data to a queue. Both also have helper methods for dealing specifically with job processing. That code is fairly well commented, so it's probably best to just read the docs in the actual file.

The QueuePublisher has the ability to create jobs and call a callback on them when the job is complete. This is done by using a little bit of hacker and redis pubsub.

Before a job is sent to the queue, a unique, random id is assigned to it. The queue publisher first subscribes to pubsub channel with the name id and the publishes the job to queue. When the queue consumer on the other end is finished processing that job, it publishes any data generated by that job to a pubsub channel with the name id. This data will be received by the queue publisher, which then stops listening on the channel named id and then calls the callback supplied for that job.

todo

• Currently, only the queue publisher has explicit code for handling jobs. The code that handles jobs on the queue consumer side is still setting in the docker_processor.js file and needs to be adapted and moved into redqueue.js.