npm

crawler

Crawler is a web spider written with Nodejs. It gives you the full power of jQuery on the server to parse a big number of pages as they are downloaded, asynchronously

node-webcrawler

Crawler is a web spider written with Nodejs. It gives you the full power of jQuery on the server to parse a big number of pages as they are downloaded, asynchronously

seenreq

A library to test if a url(request) is crawled, usually used in a web crawler. Compatible with `request` and `node-crawler`

gearman-node-bda

Node.js library for the Gearman distributed job system with support for multiple servers

bottleneckp

asynchronous rate limiter with priority

mof

mof is short for Middleware of Floodesh

mof-cheerio

cheerio wrapper for middleware of floodesh

floodesh

Floodesh is a distributed web spider/crawler written with Nodejs.

mof-charsetparser

the charset parser for middleware of floodesh

mof-iconv

Convert the response body to String from one encoding to another encoding

mof-uarotate

middleware of floodesh, which rotates User-Agent automatically

mof-seenreq

seenreq wrapper for middleware of floodesh

mof-validbody

the response body checker for middleware of floodesh

mof-recheck

the record checker for middleware of floodesh

floodesh-lib

core library for floodesh, which contains core request, response and context class, also used for flowesh

floodesh-cli

a cli tool for floodesh

floodesh-daemon

daemon for floodesh/flowesh process manager

bdautil

many of the utilities are useful for application

ffuncs

a functional library based on ramda

mof-request

A middleware wrapped request.js module for floodesh

mof-bottleneck

A middleware wrapped Bottleneckp module for floodesh

file-cookie-store-sync

Store cookies in Netscape's file

mof-whacko

A middleware wrapped whacko module for floodesh

mof-proxy

proxy middleware for floodesh

mof-statsd

A middleware wrapped statsd client module for floodesh