node package manager
Stop writing boring code. Discover, share, and reuse within your team. Create a free org »


filepile Build Status

file backed work queues


var filepile = require('filepile');
// create a new pile called 'emails' 
var pile = filepile('emails', function(details, done) {
    // details is whatever job data you provide 
    // call done() when done 
// add things to the pile 
// the will be processed by the function above 
pile({ to: '', body: 'hello' });

multiple producers single consumer

Each filepile will only allow for one consumer but multiple producers. Any named pile which already has a consumer on the same system will not allow another consumer to process jobs. If one of the consumers dies, then another consumer will start.

var cluster = require('cluster');
if (cluster.isMaster) {
    for (var i=0 ; i<4 ; ++i) {
var filepile = require('filepile');
// only one of the processes will invoke this function 
var pile = filepile('emails', function(details, done) {
// both processes will generate "work" 
setInterval(function() {
    pile({ foo: 'bar' });
}, 1000);

how it works

All producers write json files to a folder in /tmp. These files are read by the single consumer and processed. A lockfile is used to ensure that there is only one consumer.

The consumer listens using for new files and processes them.


filepile is not meant to work across machines at this time and will only ensure one consumer with multiple processes on the same machine.


npm install filepile