npm

281 packages found

M
Q
P

Description

Crawler is a web spider written with Nodejs. It gives you the full power of jQuery on the server to parse a big number of pages as they are downloaded, asynchronously

Keywords

Publisher

published 1.2.12 months ago
M
Q
P

Description

ECMAScript parser that produces a Shift format AST

Keywords

Publisher

published 7.0.018 days ago
M
Q
P

Description

A tiny node module to detect spiders/crawlers quickly and comes with optional middleware for ExpressJS

Keywords

Publisher

published 1.0.195 months ago
M
Q
P

Description

Very straightforward, event driven web crawler. Features a flexible queue interface and a basic cache mechanism with extensible backend.

Keywords

Publisher

published 1.1.83 days ago
M
Q
P

Description

A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.

Keywords

Publisher

published 1.7.02 months ago
M
Q
P

Description

a reliable web crawling & scraping framework for Node.js.

Keywords

Publisher

published 1.6.516 days ago
M
Q
P

Description

Unofficial API for zhihu (https://www.zhihu.com)

Keywords

Publisher

published 2.4.32 years ago
M
Q
P

Description

A robots.txt reader, parser and matcher.

Keywords

Publisher

published 1.0.162 years ago
M
Q
P

Description

A crawler implemented using a headless browser (Chrome).

Keywords

Publisher

published 1.4.06 months ago
M
Q
P

Description

Cylon module for RollingSpider

Keywords

Publisher

published 1.2.23 years ago
M
Q
P

Description

A simple and fully customizable web crawler/spider for Node.js with server-side DOM. Comes with elegant and hell-simple APIs.

Keywords

Publisher

published 1.5.09 months ago
M
Q
P

Description

Very straightforward, event driven web crawler. Features a flexible queue interface and a basic cache mechanism with extensible backend.

Keywords

Publisher

published 1.2.63 months ago
M
Q
P

Description

Parses the wget spider output into an object

Keywords

Publisher

published 2.0.03 years ago
M
Q
P

Description

xvideos.com api implementation.

Keywords

Publisher

published 1.1.1a year ago
M
Q
P

Description

har download 从 chrome 生成的 har 文件中下载整个网站所有资源

Keywords

Publisher

published 1.0.42 years ago
M
Q
P

Description

极简网络蜘蛛爬虫,适用任何网站,只需设置一条规则,就可以把你想要网站上的内容整理出来,非常方便,简单!

Keywords

Publisher

published 5.0.67 months ago
M
Q
P

Description

Node.js module for crawling the web

Keywords

Publisher

published 1.0.53 years ago
M
Q
P

Description

convert ASTs between Shift and SpiderMonkey formats

Keywords

Publisher

published 2.0.13 years ago
M
Q
P

Description

Fetch website with all the resources/responses/requests to local files, using puppeteer.

Keywords

Publisher

published 1.7.2a year ago
M
Q
P

Description

simple polite crawling of the web.

Keywords

Publisher

published 5.1.23 years ago