hcrawler

a hierachical web crawler with concurrency control and server-side jQuery support

#HCrawler

A hierachical crawler with concurrency control. Provide DOM facility for fetch data from web sites.

crawler.run(
 
  //href array 
  href_array,
 
  // parse function for each level 
  [
    parse_href,
    parse_info
  ],
  
  // callback function 
  function (results) {
    save_csv('info.csv');
  },
 
  // breadth first strategy 
  'breadth'
);

Pls see vessel_crawler.js for detail.

async, cheerio