middleware based lightweight crawler framework.
- Customizable pre-request, pre-process middleware stacks (it enables to log, cache, normalize, etc...)
- Cancelable crawler
- Customizable caching strategy
- Parallel crawling
- Pause/Resume crawling (it enables to sleep crawler)
- Error handling (it enables to retry)
$ npm i -S stackable-crawler
import StackableCrawler, {
CancelRequest
} from 'stackable-crawler';
const crawler = new StackableCrawler({
prerequest: [
options => {
console.log('options:', options);
return options;
}
],
processor([response, body]) {
return new Promise((resolve, reject) => {
saveFileFunction(body, error => {
if (error) return reject(error);
resolve();
});
});
}
});
crawler.on('error', (error, url) => {
console.error(error, url);
});
crawler.add('https://github.com/');
prerequest
middlewares stack can have sideeffect about requesting options. options
is request module's request option. If prerequest middleware throw CancelRequest
error, to request to that url was canceled.
preprocess
middlewares stack can have sideeffect about response, body, requestOptions. They are also from request.
Use sleep function
const sleepCrawler = (crawler, sleepTime, interval) => {
setTimeout(() => {
crawler.stop();
crawler.once('stopped', () => {
setTimeout(() => crawler.start(), sleepTime);
});
}, interval);
}
Class of crawler. Extends EventEmitter3.
Take one argument, configure object.
{
concurrency: 1, // max # of parallel crawling
prerequest: [], // prerequest middleware functions
requestCache() {}, // cache strategy
preprocess: [], // preprocess middleware functions
processor() {}, // main callback to handle crawled Document
}
Function that process requestOption
s. Default argument is { uri: url }
. Function must return new (mutated) requestOptions or Promise[requestOptions].
Function that returns cached value or undefined
. Type of cached value is T or Promise[T]. If cached value returned, crawler pass through that values to processor function. If undefined returned, crawler fetch document as usual.
Function that process [response, body, requestOptions]
. response
and body
are fetching result of request module. It also return new argument or Promise.
Main function. You can do everything here. It can return promise.
Add url to crawling task queue.
Pause crawler. If crawler has one more running tasks, these are still running until finished, but no more run new task.
Resume crawler if paused.
These methods are inherit from EventEmitter3.
available event and args
- event:
error
, args:[error, url]
- event:
stopped
Error class. If crawler throws CancelRequest error throw new CancelRequest()
, crawler stop to request that url with no error.
Bundled simple middlewares.
Filter only valid url. If url is invalid, this middleware throw CancelRequest error.
Function that returns simple in memory cache middleware. Do urlCache()
when use. If requesting url was already cached, this middleware throw CancelRequest error.
Very simple console.log middleware
Replace body to cheerio object ($
). Next middlewares and processor call with [response, $, requestOptions]
MIT