node package manager
Share your code. npm Orgs help your team discover, share, and reuse code. Create a free org »



simple web crawler with node. experimenting with streams.

Here's a simple example

        'routes': {
                //Routes that get called and crawling does not progress beyond 
                'leaf': {
                        '.**': function(tr, link) {
                                tr.selectAll('p', function(div) {
                //Routes that get called and all links are enqueued 
                'tree': {
                        '.**': function(tr, link) {}
                //Routes not to follow 
                'ignore': [],
        'seed': [''],
        'concurrency': 50,
        'backend': new wubwub.Backends.Simple()


This example will start at, and follow any link and print the contents of every p element on every leaf page to stdout


There are three kinds of routes, leaf routes, tree routes, and ignored routes.

  • Ignored routes: a regex in the ignored route list won't be fetched. This overrides any previous route spec, such that if a leaf and ignored route are matched, the route is ignored.
  • Leaf routes: a leaf route is one that when matched, will call the callback specified but not enqueue any of the links on the page.
  • Tree route: a leaf route is one that when matched, will call the callback specified and enqueue all the links on the page (provided they aren't ignored and match one of the tree or leaf routes)


Callbacks in the leaf and tree routes are passed a trumpet instance for the page that was just fetched, as well as the url that was matched.


Specifies how many requests can be running at the same time


Specifies which URL to start crawling at


Controls how links are enqueued for crawling. Two are included, wubwub.Backends.Simple() and wubwub.Backends.Redis() which implement put() and get(onLink) methods for links, and keep track of what URLs have already been seen. You can implement your own backend as long as it implements those methods. get() is async and takes a callback to be executed once the link is fetched.