overlog

Simple logs collection and retrieval.

overlog (node.js module)

An easy to use solution to store and fetch the logs.

  • Storer: push(entry) adds an entry.
  • Fetcher: fetch(time_interval) returns all entries from certain time interval.
  • Fetcher: follow() mimics tail -f.
  • Fetcher: fetch_and_follow() fetches entries from certain time range and goes into following mode afterwards.

Both storer and fetcher can be used via command line or via node.js bindings.

Shell tests provide a good idea on basic usage and functionality.

Directory to store the files has to be provided. Technically, two directories are required: one for intermediate files, append-only, and one for thefinalized files. They will be created as $DIR/intermediate/ and $DIR/destination/ as necessary.

In PubSub mode, an HTTP status endpoint is being exposed as well. All it takes to use PubSub is to specify PubSub channel name, default values for other parameters would work.

If not using PubSub for storer's locking mechanism, path to a lock file (via pidlock, directory and file) is required as well.

Log entries are JSON objects. They should have the "ms" field set to Date.now() at the moment log entry was created.

  • Append-only intermediate files.
  • Atomically rename them into final format.
  • If necessary, replay older files at startup.
  • Keep first and last timestamps in the filenames for easier access.

In PubSub mode, an HTTP status page is being exposed, check out /statusz.

The tool was designed for the fetch_and_follow() usecase, that is easiest to access with fetcher's command line flags --last_ms_then_follow and --since_ms_then_follow.

node.js code performing these actions can be used directly as well.

  • With HTTP status endpoint, PubSub and a command-line tool.
  • Process-level locked.
  • Unit-tested through.