A declarative HATEOAS API crawler for node.js
hyperspider is great for folks that want to create clean, granular, and self-documenting hypertext APIs, but avoid the latency of remotely fetching hundreds of tiny HTTP resources.
Let's say you had a Twitter clone with a hypertext JSON API, including the following two sample endpoints:
To create a single
following-detailed resource so that API consumers don't have to make a separate HTTP call for each resource, use hyperspider like this:
var hyperspider = require"hyperspider"var options =host: "mytwitterclone.biz"path:"/users/jed""/users/jed/following""/users/*"hyperspideroptions// data is an array with the result entities of 6 endpoints:// 1. /users/jedschmidt// 2. /users/jedschmidt/following// 3. /users/janl// 4. /users/cramforce// 5. /users/hblank// 6. /users/theophani
See the tests for a working example.
Use npm to install hyperspider:
npm i hyperspider
options: An object containing the same options as for node's http.request method, with one exception:
path can be any of the following:
- a normal url path, such as
- a function that takes a path and returns a boolean
- a RegExp that matches a path
- a wildcard url path to turn into a RegExp, with
- an array containing any of the above
path must contain at least one normal url path to serve as the starting point for the crawl.
callback(err, data): Buffers the event stream into a single callback.
err is null if no errors occurred, and otherwise an array of errors. data is an array of string entities, one for each successful HTTP call. Omit this to listen for a stream of events.
This method takes a single argument, an HTTP response body string. Override this before adding listeners to customize how hyperspider should extract urls from each resource. By default, the response body is parsed into JSON, extracting the value of every nested
href property, but you could roll your own, such as parsing
Link headers, etc.