Truly distributed and parallel parallel queue on top of redis and zookeeper without SPoF.
Reliable queue that duplicate items among redis servers and synchronize with zookeeper. Use if you don't want to loose your data and don't mind if some items can be processed more than 1 time if redis fails.
npm install zk-redis-queue
You may be interested in test/test.js for now. In this example you'll need 2 redis servers on 127.0.0.1:6379 and 127.0.0.1:6380. We'll use shared zk instance so performance will be poor, please start your own cluster for production use.
var Redis = require"redis"Queue = require"../index"ZK = require"zkjs"zk = hosts: "api.yongwo.de:2181" root: "/pew-pew" timeout: 2000redisOne = RediscreateClient6379 "127.0.0.1" retry_max_delay: 1000redisTwo = RediscreateClient6380 "127.0.0.1" retry_max_delay: 1000queue = redisOne redisTwo zk "woo"; // "woo" is a queue key// error event is triggered for recoverable errors,// if error is unrecoverable, you'll receive error in callbackqueueon"error"console.log"Got warning" error;;// we are ready to startqueueon"ready"// push item to queuequeuepush"pew"if error// push failed on every redis serverthrow error;;// let's try to get something for 5 times (second argument)queuepopif error// no redis servers are alive// or zookeeper is down// damn!throw error;if item// we got item and it's locked for us!console.log"Look what I've got:" item;// remove item after all work is doneremove// no error if everything is down,// "error" event will be emitter// and item can be processed by another workerconsole.log"Item removed!";;5;;
Queue consists of 2 elements: zookeeper pool (synchronization to make parallel processing possible) and redis pool (data storage). Redis servers are completely independent. We suggest you to have at least 3 redis servers on different physical servers.