divide and conquer mongoose collections
Allows you to divide and conquer an entire mongo collection using mongoose.
We can give divide a model, and it will find id ranges for the given batch size. These can be pushed onto a queue for processing.
var waitress = require'waitress'PikaQueue = require'pika-queue'daq = require'mongoose-divide-and-conquer'Model = require'./some/model'queue =;daqdividemodel: ModelbatchSize: 1000if err throw err;var done = waitressbatcheslengthif err throw err;console.log'all done!';processexit;;console.log'queueing %d batches' batcheslength;batchesforEachqueuequeueJob'divide-and-conquer' batchconsole.log'batch %s finished' i;doneerr;;;;
We can pop batches off the queue, and give them to conquer with two methods.
The first method is used to process one document. The second method is called when all documents have been processed. All documents from the batch are processed in parallel.
var waitress = require'waitress'PikaQueue = require'pika-queue'daq = require'mongoose-divide-and-conquer'Model = require'./some/model'queue =;queuemonitorJobQueue'divide-and-conquer'daqconquerbatch: batchmodel: Model// do something with the documentprocessstdoutwrite'.';done;if err throw err;console.log;console.log'processed %d docs' count;cbnull count;;;