filepile

file backed local work queues

filepile

file backed work queues

var filepile = require('filepile');
 
// create a new pile called 'emails' 
var pile = filepile('emails', function(detailsdone) {
    // details is whatever job data you provide 
    // call done() when done 
});
 
// add things to the pile 
// the will be processed by the function above 
pile({ to: 'foobar@example.com', body: 'hello' });

Each filepile will only allow for one consumer but multiple producers. Any named pile which already has a consumer on the same system will not allow another consumer to process jobs. If one of the consumers dies, then another consumer will start.

var cluster = require('cluster');
 
if (cluster.isMaster) {
    for (var i=0 ; i<4 ; ++i) {
        cluster.fork();
    }
    return;
}
 
var filepile = require('filepile');
 
// only one of the processes will invoke this function 
var pile = filepile('emails', function(detailsdone) {
    console.log(details);
    done();
});
 
// both processes will generate "work" 
setInterval(function() {
    pile({ foo: 'bar' });
}, 1000);

All producers write json files to a folder in /tmp. These files are read by the single consumer and processed. A lockfile is used to ensure that there is only one consumer.

The consumer listens using fs.watch for new files and processes them.

filepile is not meant to work across machines at this time and will only ensure one consumer with multiple processes on the same machine.

npm install filepile