A module to analyse websites for SEO, validation and code-quality

Webcheck gives you an infrastructure to analyze web resources. It is build very generic for supporting a wide range of possible use cases. You are able to analyze a single page, a whole domain or even everything connected to one or more seed resources. You can analyze every content-type.

Version 1.0.0 is refactored completely. Webcheck uses streams. Because of the streams it uses plugins instead of middleware. There is a list of plugins in the github repository. Please send me the link to yours if you develop one. Otherwise you should look on npm for modules with prefix 'webcheck-'.

npm install webcheck
var Webcheck = require('webcheck');
var AnyPlugin = require('webcheck-any-plugin');
var webcheck = new Webcheck();
var anyPlugin = new AnyPlugin({your: 'options'});
    url: 'http://some.website/url'
}, function (err) {
    if (err) {
        console.error('There was an error while crawling', err);
    console.log('Crawling done...');

Since version 1.0.0 webcheck uses streams instead of callbacks. It is not compatible to older versions!

Webcheck is small featured. You should extend your functionality with plugins. Take a look at the list of plugins.

For further information about plugins in webcheck take a look at the plugin readme.

Add a plugin to webcheck.

Request a resource

  • url | {string} [mandatory]: URL to crawl
  • wait | {Number}: Time to wait till request (default: 0)
  • headers | {Object}: Default headers (default: {"User-Agent": "webcheck v1.0.0"})
  • request | {request}: The used request-module
  • immediately | {boolean}: Should the crawl push as next one to queue.
  • parameters | {Object}: A object to pass parameters to other plugins about this crawl

This is a instance of request. Webcheck use this as function to request a resource. If you want another request function, for example to request resources from TOR with torrequest, you are able to swap this property.

Array of used middleware.

All events emitted on the webcheck object.

var webcheck = new Webcheck();
webcheck.on(event, fn);

Webcheck emits the following events:

  • crawl (request-settings): Emitted directly after calling crawl method.
  • request (request-settings): Emitted before request is executed.
  • result ({url, request-settings, request, response}): Emitted after middleware are executed and document is fetched
  • drain: Emitted on draining of queue
  • queue (request-settings): Emitted before adding to queue
  • addPlugin (plugin): Emitted when a plugin is added
  • enablePlugin (plugin): Emitted when a added plugin gets enabled
  • disablePlugin (plugin): Emitted when a added plugin gets disabled