tarantula

nodejs crawler/spider which provides a simple interface for crawling the Web

node-tarantula

nodejs crawler/spider which provides a simple interface for crawling the Web. Its API has been inspired by crawler4j.

var brain = {
 
    legs: 8,
 
    shouldVisitfunction(uri) {
        return true;
    }
 
};
 
var tarantula = new Tarantula(brain);
 
tarantula.on('data', function (uri) {
      console.info('200', uri);
});
 
tarantula.on('done', function() { 
    console.log('done'); 
});
 
tarantula.start(["http://stackoverflow.com"]);

If you would like to use the included PhantomJS plugin, you'll need to install the PhantomJS app (it is not an npm module).

  1. You can download PhantomJS on their website.
  2. It's also on popular OS Package Managers: brew install phantomjs, apt-get install phantomjs