super-worker
Split a large number of concurrent tasks into execution
Build Setup
# install dependenciesnpm install super-worker --save# node testcd node_modules/super-workernpm run test
API
SuperWorker
- new SuperWorker(promiseFunc, max_job, max_retry, rest)
- addJob(...args)
- addImportantJob(...args)
- on('finished', () => {})
- on('retry', job => {console.log('【retry】', job.args, job.retry)})
Usage
【example】: What should I do if I want to grab a lot of pagination from a website? like this? error!
for var i = 1; i < 10000; i++
As above, you will cause a large number of requests in an instant, which may cause the other server to go wrong, so that most of the returned results are wrong.
All, below is a correct example.
const SuperWorker = const fetch = //request 100 pages at the same time. //retry 3 times after fetch error.//sleep 300ms before performing the next taskvar superWorker = fetch 100 3 300//rename addJobsuperWorkerfetch = superWorkeraddJob let startTime = superWorkersuperWorker for var i = 1; i < 10000; i++ superWorker