Crawler is a web spider written with Nodejs. It gives you the full power of jQuery on the server to parse a big number of pages as they are downloaded, asynchronously. Scraping should be simple and fun!
Have a look at alternatives modules:
node-crawler aims to be the best crawling/scraping package for Node.
- A clean, simple API
- server-side DOM & automatic jQuery insertion with Cheerio (default) or JSDOM
- Configurable pool size and retries
- Priority of requests
- forceUTF8 mode to let node-crawler deal for you with charset detection and conversion
- A local cache
- node 0.10 and 0.12 support
The argument for creating this package was made at ParisJS #2 in 2010 ( lightning talk slides )
Help & Forks welcomed!
$ npm install crawler
var Crawler = require"crawler";var url = require'url';var c =maxConnections : 10// This will be called for each crawled page// $ is Cheerio by default//a lean implementation of core jQuery designed specifically for the server$'a'eachvar toQueueUrl = $aattr'href';cqueuetoQueueUrl;;;// Queue just one URL, with default callbackcqueue'';// Queue a list of URLscqueue'''';// Queue URLs with custom callbacks & parameterscqueueuri: ''jQuery: false// The global callback won't be calledconsole.log'Grabbed' resultbodylength 'bytes';;// Queue using a functionvarreturn '' + search;;cqueueuri: googleSearch'cheese';// Queue some HTML code directly without grabbing (mostly for tests)cqueuehtml: '<p>This is a <strong>test</strong></p>';
For more examples, look at the tests.
You can pass these options to the Crawler() constructor if you want them to be global or as items in the queue() calls if you want them to be specific to that item (overwriting global options)
This options list is a strict superset of mikeal's request options and will be directly passed to the request() method.
Basic request options:
uri: String, the URL you want to crawl
timeout: Number, in milliseconds (Default 60000)
- All mikeal's requests options are accepted
callback(error, result, $): A request was completed
onDrain(): There is no more queued requests
maxConnections: Number, Size of the worker pool (Default 10),
priorityRange: Number, Range of acceptable priorities starting from 0 (Default 10),
priority: Number, Priority of this request (Default 5),
retries: Number of retries if the request fails (Default 3),
retryTimeout: Number of milliseconds to wait before retrying (Default 10000),
Server-side DOM options:
jQuery: true, false or ConfObject (Default true) see below Working with Cheerio or JSDOM
forceUTF8: Boolean, if true will try to detect the page charset and convert it to UTF8 if necessary. Never worry about encoding anymore! (Default false),
incomingEncoding: String, with forceUTF8: true to set encoding manually (Default null)
incomingEncoding : 'windows-1255'for example
cache: Boolean, if true stores requests in memory (Default false)
skipDuplicates: Boolean, if true skips URIs that were already crawled, without even calling callback() (Default false)
userAgent: String, defaults to "node-crawler/[version]"
referer: String, if truthy sets the HTTP referer header
rateLimits: Number of milliseconds to delay between each requests (Default 0) Note that this option will force crawler to use only one connection (for now)
Crawler by default use Cheerio instead of Jsdom. Jsdom is more robust but can be hard to install (espacially on windows) because of contextify.
Which is why, if you want to use jsdom you will have to build it, and
require('jsdom') in your own script before passing it to crawler. This is to avoid cheerio crawler user to build jsdom when installing crawler.
###Working with Cheerio
jQuery: true //(default)//ORjQuery: 'cheerio'//ORjQuery:name: 'cheerio'options:normalizeWhitespace: truexmlMode: true
These parsing options are taken directly from htmlparser2, therefore any options that can be used in
htmlparser2 are valid in cheerio as well. The default options are:
normalizeWhitespace: falsexmlMode: falsedecodeEntities: true
###Working with JSDOM
In order to work with JSDOM you will have to install it in your project folder
npm install jsdom, deal with compiling C++ and pass it to crawler.
var jsdom = require'jsdom';var Crawler = require'crawler';var c =jQuery: jsdom;
node-crawler use a local httpbin for testing purpose. You can install httpbin as a library from PyPI and run it as a WSGI app. For example, using Gunicorn:
$ pip install httpbin // launch httpbin as a daemon with 6 worker on localhost $ gunicorn httpbin:app -b 127.0.0.1:8000 -w 6 --daemon // Finally $ npm install && npm test
After installing Docker, you can run:
// Builds the local test environment $ docker build -t node-crawler . // Runs tests $ docker run node-crawler sh -c "gunicorn httpbin:app -b 127.0.0.1:8000 -w 6 --daemon && npm install && npm test" // You can also ssh into the container for easier debugging $ docker run -i -t node-crawler bash
- Refactoring the code to be more maintenable, it's spaghetti code in there !
- Have a look at the Cache feature and refactor it
- Same for the Pool
- Proxy feature
- This issue: https://github.com/sylvinus/node-crawler/issues/118
- Make Sizzle tests pass (jsdom bug? https://github.com/tmpvar/jsdom/issues#issue/81)
- More crawling tests
- Document the API more (+ the result object)
- Option to wait for callback to finish before freeing the pool resource (via another callback like next())