simplecrawler-mongo-queue

1.0.1 • Public • Published

MongoDB queue for Simplecrawler

npm Travis (.org) Dependency Status devDependency Status Greenkeeper badge

This is a queue implementation for simplecrawler powered by MongoDB.

Installation

npm install --save simplecrawler-mongo-queue

Usage

First of all, create a new Simplecrawler instance as described in the documentation. Then create the queue instance and assign it to crawler.queue property.

const Crawler = require('simplecrawler');
const MongoQueue = require('simplecrawler-mongo-queue');
 
(async () => {
  const crawler = new Crawler('http://example.com');
  crawler.queue = await MongoQueue.create(datastore, name);
  crawler.start();
})();

The create method returns MongoQueue instance and has two arguments:

  • datastore - the application should provide a MongoDB collection where the queue will be stored.
  • name (optional) - a name of the queue to distinguish the different crawlers. If the argument is omitted the constructor creates a random queue name.

Resources

Package Sidebar

Install

npm i simplecrawler-mongo-queue

Weekly Downloads

0

Version

1.0.1

License

MIT

Unpacked Size

10.4 kB

Total Files

5

Last publish

Collaborators

  • kbychkov