410 packages found

    A specification compliant robots.txt parser with wildcard (*) matching support.

    published 3.0.0 9 months ago
    M
    Q
    P

    Crawler is a web spider written with Nodejs. It gives you the full power of jQuery on the server to parse a big number of pages as they are downloaded, asynchronously

    published 1.3.0 2 years ago
    M
    Q
    P

    A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.

    published 2.0.3 5 months ago
    M
    Q
    P

    A tiny node module to detect spiders/crawlers quickly and comes with optional middleware for ExpressJS

    published 2.0.0 3 years ago
    M
    Q
    P

    Crawler is a web spider written with Nodejs. It gives you the full power of jQuery on the server to parse a big number of pages as they are downloaded, asynchronously

    published 1.3.1 3 months ago
    M
    Q
    P

    Generic web crawler powered by Node.js

    published 1.4.1 6 years ago
    M
    Q
    P

    极简网络蜘蛛爬虫,适用任何网站,只需设置一条规则,就可以把你想要网站上的内容整理出来,非常方便,简单!

    published 5.0.10 3 years ago
    M
    Q
    P

    Crawler (spider) of site web pages by domain name

    published 1.2.3 a year ago
    M
    Q
    P

    This is an ES6 adaptation of the original PHP library CrawlerDetect, this library will help you detect bots/crawlers/spiders vie the useragent.

    published 3.3.0 9 months ago
    M
    Q
    P

    a node spider framework

    published 1.0.11 4 years ago
    M
    Q
    P

    Parses the wget spider output into an object

    published 2.0.0 7 years ago
    M
    Q
    P
    M
    Q
    P

    a simplified directed web crawler, easy to use for scraping pages and downloading resources of page.

    published 1.7.3 4 years ago
    M
    Q
    P

    Scrape/Crawl article from any site automatically. Make any web page readable, no matter Chinese or English.

    published 0.5.6 4 years ago
    M
    Q
    P

    Starter Template for testing PhantomJS ‘Applications’ with Jasmine, Grunt, and Istanbul

    published 1.0.0 9 years ago
    M
    Q
    P

    simple polite crawling of the web.

    published 5.1.2 7 years ago
    M
    Q
    P

    Fast and lightweight web crawler with built-in cheerio, xml and json parser.

    published 0.3.3 8 years ago
    M
    Q
    P

    Simple WAF to integrate with Node.js web systems

    published 1.0.3 3 years ago
    M
    Q
    P

    The spider's sharp ax.

    published 0.0.7 5 years ago
    M
    Q
    P

    Super configurable async web spider

    published 0.3.0 7 years ago
    M
    Q
    P