mrspider-mongodb-persister

Middleware for mrspider typically used at the end of a middleware chain for saving the results of a crawl to a mongoDb database. mrspider-validator is typically used before this to mark the page as valid or not to avoid storing rubbish in the database.

Install

npm i -S mrspider-mongodb-persister

Usage

var mrspider = require('mrspider');
var mrspiderMongoPersister = require('mrspider-mongodb-persister');
var spider = mrspider();
var mongoPersister = mrspiderMongoPersister({
  url: 'mongodb://db1.example.net',
  collection: 'crawlResults'
});
 
spider.use(mongoPersister);