Search results
445 packages found
基于Node.js的网络爬虫
A web crawler. Automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.
``` sh npm install -g puppet-spider ```
A simple web crawler that keeps to the domain
Crawl multiple domains using one or more entry URLs.
Web crawler configured by JSON configurations defining what data fields to scrape from the visited websites using regular expressions or DOM selectors and how to export them as JSON
DCrawler is a distribited web spider written in Nodejs and queued with Mongodb. It gives you the full power of jQuery to parse big pages as they are downloaded, asynchronously. Simplifying distributed crawler!
Walk a website checking for bad links
Easily build flexible, scalable, and distributed, web crawlers.
blocky spider creatures for your voxel.js game