HCrawler

A hierachical crawler with concurrency control. Provide DOM facility for fetch data from web sites.

Quick Example

crawler.run(
 
  //href array
  href_array,
 
  // parse function for each level
  [
    parse_href,
    parse_info
  ],
  
  // callback function
  function (results) {
    save_csv('info.csv');
  },
 
  // breadth first strategy
  'breadth'
);

How to

Pls see vessel_crawler.js for detail.

Require

async, cheerio