crawlify
Crawlify uses a pool of PhantomJS browsers to create a static copy of your single-page application. Rather than waiting an arbitrary amount of time, crawlify includes a client-side component that you use within your own code. This makes crawling blazing fast.
Installation
npm install crawlify
bower install crawlify
Example
First include crawlify.js
or crawlify.min.js
in your webpage. This will place a crawlify
object on window
that you can call start and stop on.
Client
$; // Let crawlify know there's something to wait forcrawlify;
Server
Here we're going to cache the page so that the server can reply instantly
var Crawlify = ;var crawl = workers: 4 // Defaults to 1, reset: '/'; var cache = {}; crawl; // At a later time, crawlify uses the same browser instance to apply pushState// at the new target page.crawl;
Development
- Clone the repo
npm install
bower install
- Make sure you have mocha installed
npm test