npm

crawler

Crawler is a web spider written with Nodejs. It gives you the full power of jQuery on the server to parse a big number of pages as they are downloaded, asynchronously

bottleneckp

asynchronous rate limiter with priority

mof-reqadapter

middleware of floodesh, which adapts queued options to request options

mof-normalizer

middleware of floodesh, which normalizes queue options

mof-uarotate

middleware of floodesh, which rotates User-Agent automatically

flowesh

Flowesh is the non-cluster version of floodesh. It's a middleware based web spider which is lightweight and easy to maintain

mof-statsdclient

middleware of floodesh, which provides a statsd client that allows you to send data to statsd server and automatically has some statistics sent that generally apply to all requests.

captcha-recognizer

extracts contents out of a png, jpg or tiff, based on tesseract, an open source c++ ocr engine

bda-util

commonly used tools for bda projects

sails-hook-hotcron

a hot deployment supporting crontab hook for sails

mof-genestamp

middleware of floodesh, prints gene and url of a task, along with # of new tasks and # of records