Given a MongoDB query cursor, process the results in parallel, up to the specified limit.
var broadband = ;var cursor = mongoCollection;return;
We wanted to work with MongoDB queries the way we work with async.eachLimit, but without yanking everything into memory at once with
Specifically, we wanted to resize some images in parallel, rather than waiting to do them one at a time. We have a MongoDB collection with information about all of the images. But there are a lot of them, so we don't want to yank all of that information into memory up front.
broadband wraps MongoDB's
Cursor.nextObject with a queueing mechanism that allows several results to be processed at once, but only up to the limit you specify. You don't run out of memory due to too many image processes, you don't wait too long, and you don't have to load the entire array into memory at once. Everybody gets a medal.
What about errors?
If an error occurs,
- Stop starting new iterator callbacks.
- Wait for any outstanding iterator callbacks to finish.
- Invoke the final callback (its third argument) with the first error it received.
Using broadband without mongodb
You can pass any object with a
nextObject method as the "cursor." That method should invoke its callback with
(err, object). If there is no error,
object should be the next object retrieved from your data source. If there are no more objects, pass
About P'unk Avenue and Apostrophe
broadband was created at P'unk Avenue for use in many projects built with Apostrophe, an open-source content management system built on node.js. If you like
broadband you should definitely check out apostrophecms.com.
Feel free to open issues on github.
- Adds support for both MongoDB 2 and 3 via the cursor
- Adds JS linting to the tests.
- Declared 1.0.0 stable as this has long been a component of Apostrophe. Updated lodash dependency to satisfy
- Fixed a rare race condition which caused
broadbandto invoke its final callback more than once.
- Initial release. With shiny unit tests, of course.