keywords:spider - npm search

robots-parser

A specification compliant robots.txt parser with wildcard (*) matching support.

published version 3.0.1, 2 years ago79 dependents licensed under $MIT

4,567,833

simplecrawler

Very straightforward, event driven web crawler. Features a flexible queue interface and a basic cache mechanism with extensible backend.

kbychkov

published version 1.1.9, 5 years ago77 dependents licensed under $BSD-2-Clause

141,124

shift-parser

ECMAScript parser that produces a Shift format AST

shapesecurity

published version 8.0.0, 3 years ago31 dependents licensed under $Apache-2.0

101,192

isbot-fast

JavaScript module detecting bots/crawlers/spiders via user-agent

mahovich

published version 1.2.0, 5 years ago11 dependents licensed under $MIT

72,320

es6-crawler-detect

This is an ES6 adaptation of the original PHP library CrawlerDetect, this library will help you detect bots/crawlers/spiders vie the useragent.

jefferyhus

published version 4.0.2, 6 days ago8 dependents licensed under $MIT

22,630

crawler

Crawler is a ready-to-use web spider that works with proxies, asynchrony, rate limit, configurable request pools, jQuery, and HTTP/2 support.

mike442144

published version 2.0.2, 9 months ago124 dependents licensed under $MIT

18,290

spider-detector

A tiny node module to detect spiders/crawlers quickly and comes with optional middleware for ExpressJS

michael.heuberger

published version 2.1.0, 8 months ago6 dependents licensed under $MIT

11,733

@zachleat/spider-pig

Get a list of local URL links from a root URL.

zachleat

published version 3.0.0, 3 years ago1 dependents licensed under $MIT

10,894

robots-txt-parser

A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.

cakroyd

published version 2.0.3, 3 years ago5 dependents licensed under $MIT

6,307

@lightningchart/lcjs

A high-performance charting library.

niilo.keinanen

published version 7.0.3, a month ago1 dependents licensed under $SEE LICENSE IN LICENSE

5,147

@spider-cloud/spider-client

Isomorphic Javascript SDK for Spider Cloud services

jeffmendez

published version 0.1.32, a month ago0 dependents licensed under $MIT

4,182

@nodebb/spider-detector

A tiny node module to detect spiders/crawlers quickly and comes with optional middleware for ExpressJS

baris

published version 2.0.3, 10 months ago0 dependents licensed under $MIT

3,594

node-html-crawler

Crawler (spider) of site web pages by domain name

safonovpro

published version 1.2.3, 4 years ago0 dependents licensed under $MIT

1,941

@a11ywatch/crawler

gRPC tokio based web crawler

jeffmendez

published version 0.9.9, 8 months ago1 dependents licensed under $MIT

874

roboto

A web crawler for Nodejs.

jculvey

published version 0.8.2, 11 years ago1 dependents licensed under $MIT

1,053

wget-parser

Parses the wget spider output into an object

muji

published version 2.0.0, 9 years ago0 dependents licensed under $MIT

831

An extremely lightweight HTTP request client for the command-line. Supports: http, https, proxy, redirects, cookies, content-encoding, multipart/form-data, multi-threading, recursive website crawling and mirroring.

warren-r-bank

published version 4.0.25, 13 days ago1 dependents licensed under $GPL-2.0

735

node-webcrawler

Crawler is a web spider written with Nodejs. It gives you the full power of jQuery on the server to parse a big number of pages as they are downloaded, asynchronously

mike442144

published version 0.8.0, 8 years ago6 dependents licensed under $ISC

763