Sample website text content over time.
Data collection sometimes includes collecting the same data over time. This module, which is basically a wrapper around needle-schedule, enables you to collect e.g. the html content of websites over time and (optionally) store the data with the chronostore module, thus sampling sites. On top of storing the results as files, it also emits the results when collected, enabling you to further process the data, e.g. pushing the data to a database.
A heavy emphasis is put on stability and logging. Everything, even the logging provided by bunyan, is covered with tests. It also allows you to forward your logging to the console, loggly, logentries and slack.
Linted with ESLint, tested with tape and 100% coverage with covert.
npm install sitesampler
Logging is performed by bunyan and all logging is enabled with env vars. For instance, to run the example with logging turned on, run the command
SITESAMPLER_LOG=1 node example.js.
Logging can be posted to loggly by setting the following env vars:
SITESAMPLER_LOGGLY_SUBDOMAIN: your loggly subdomain.
SITESAMPLER_LOGGLY_TOKEN: your loggly token.
SITESAMPLER_LOGGLY_LEVEL: the level to log. Defaults to
Logging can be posted to logentries by setting the following env vars:
SITESAMPLER_LOGENTRIES_TOKEN: your logentries token.
Logging can be posted to slack by setting the following env vars:
SITESAMPLER_SLACK_WEBHOOKURL: your slack webhook url.
SITESAMPLER_SLACK_CHANNEL: your slack channel. Defaults to the channel of the webhook.
SITESAMPLER_SLACK_LEVEL: the level to log. Defaults to
SITESAMPLER_SLACK_USERNAME: the username to post with. Defaults to
When instantiating the sitesampler, it requires settings to work. Settings is thus either an object or a path to a json file containing the settings.
settings (object | string): object with settings or path to settings. Defaults to
targets(object) - an array of targets for the schedule-schedule module under the hood.
options(object) - default options that will be passed through to schedule-schedule. See example below or
chronostore(object) - default options that will be passed through to chronostore. If not defined, nothing will be written to disk. To use chronostore defaults, pass an empty object
rethrowErrors(object) - whether sitesampler should rethrow eerrors from goldwasher and chronostore. Defaults to
Starts the sitesampler.
Stops the sitesampler.
Sitesampler is an event emitter that will emit collected results or when an error is encountered. See the example below. Remember to handle errors or your program will crash if an error is encountered. This will, however, usually only happen if you point it at something that isn't HTML.
var sitesampler = ;var ss = ;ss;ss;ssstart;
"targets":"url": """rule": "second": 15 35 55"options":"rule": "second": 1"needleRetry":"retries": 10"chronostore":"root": "./chronostore"