async-iterable-limiter
Iterate over async iterable with concurrency limit.
Difference from many other similar modules is that it supports iterating over async iterables (like ReadableStream or database cursor).
Note: documentation and tests are work in progress, but it should already work :)
Usage
Simple:
const Limiter = require('async-iterable-limiter');
const cursor = db.collection().find(); // todo: better example
await Limiter.iterate(cursor, 5, async (item) => {
await doWork(item);
});
Complex (internal implementation of iterate
function):
const Limiter = require('async-iterable-limiter');
const limiter = new Limiter({ concurrency: 5 });
for await (const item of iterable) {
await limiter.ready();
limiter.add(taskReturningPromise(item));
}
await limiter.finished();
TODO:
Functionality:
- [ ] add
stopOnError
parameter (true by default in current implementation) - [ ] maybe add
Limiter.map
function that would produce an array with sequential results - [ ] maybe add
Limiter.mapIterable
function that would produce another async iterable - [ ] make
inProgress
andconcurrency
readonly properties, to prevent mess
Tests:
- [x] add test for error handling of async iterable
- [x] add test for error handling of task
- [ ] add test for invalid input of params
- [ ] add test for expected behavior of multiple calls of Limiter.ready/finished class
Documentation:
- [ ] readme
- [ ] api surface documentation
- [ ] typescript definitions