Share your code. npm Orgs help your team discover, share, and reuse code. Create a free org »

    hcrawlerpublic

    HCrawler

    A hierachical crawler with concurrency control. Provide DOM facility for fetch data from web sites.

    Quick Example

    crawler.run(
     
      //href array 
      href_array,
     
      // parse function for each level 
      [
        parse_href,
        parse_info
      ],
      
      // callback function 
      function (results) {
        save_csv('info.csv');
      },
     
      // breadth first strategy 
      'breadth'
    );

    How to

    Pls see vessel_crawler.js for detail.

    Require

    async, cheerio

    Keywords

    none

    install

    npm i hcrawler

    Downloadsweekly downloads

    1

    version

    0.0.5

    license

    none

    repository

    github.com

    last publish

    collaborators

    • avatar