MongoDB queue for Simplecrawler
This is a queue implementation for simplecrawler powered by MongoDB.
Installation
npm install --save simplecrawler-mongo-queue
Usage
First of all, create a new Simplecrawler instance as described in the documentation. Then create the queue instance and assign it to crawler.queue
property.
const Crawler = ;const MongoQueue = ; async { const crawler = 'http://example.com'; crawlerqueue = await MongoQueue; crawlerstart;};
The create
method returns MongoQueue
instance and has two arguments:
datastore
- the application should provide a MongoDB collection where the queue will be stored.name
(optional) - a name of the queue to distinguish the different crawlers. If the argument is omitted the constructor creates a random queue name.