Noncollinear Perpendicular Microcrystalline
Share your code. npm Orgs help your team discover, share, and reuse code. Create a free org »

crawlstream

0.3.5 • Public • Published

crawlstream

A website crawler that gives a readable stream of request streams.

Development of this module has been sponsored by Knowit

Build Status

Installation

$ npm install crawlstream

Running the tests

$ npm test

Examples

Printing out the paths of all the pages found.

Streaming API

var crawlstream = require('crawlstream');
 
crawlstream('mysite.com', 10)
    .on('data', function(req) {
        console.log(req.uri.path);
    });

Callback API

var crawlstream = require('crawlstream');
 
crawlstream('mysite.com', 10, function(err, req) {
    console.log(req.uri.path);
});

Methods

var crawlstream = require('crawlstream')

crawlstream(baseUrl, concurrency, limit, [callback])

Crawl all pages under baseUrl.

Optionally supply a callback(err, req) which will receive the request stream(!) for all pages.

License

Copyright 2012 Knowit

MIT

Keywords

none

install

npm i crawlstream

Downloadsweekly downloads

0

version

0.3.5

license

none

homepage

github.com

repository

Gitgithub

last publish

collaborators

  • avatar
Report a vulnerability