@chcaa/simple-web-log-analytics

Simple web log analytics

This package provides basic website analytics based on Apache2 log files.

The package provides two kinds of functionality.

1. A stand-alone log reader, parser, and analyzer module to be used with npx in the command line.
2. A viewer class which provides basic functionality for loading processed log files from a directory.

This README.md is therefore structured in two parts.

About Apache2 log files

In its standard configuration, Apache2 saves log files to /var/log/apache2/ on an Ubuntu server. This directory and its contents are root-access only. Therefore, you should copy the log files to a different directory which can be read and written by the user running the script. We recommend one of two practices:

1. Manually sudo cp the files to a different directory (if only done once), or
2. Set up a cron job to call a script which first copies the files as root, then runs this parser as the desired user.

You would likely have set up log rotation for Apache2. Log rotation rotates logs continuously and compresses old logs as gzip. The script automatically detects and unzips gzipped logs in the process. This package assumes that log rotation is enabled, but it should work regardless.

Apache2 creates several types of logs https://httpd.apache.org/docs/2.4/logs.html. N is an integer starting from 1.

• access.log, access.log.N, access.log.N.gz - Logs for server administration. Disregarded by script.
• error.log, error.log.N, error.log.N.gz - Logs for error handling. Disregarded by script.
• [domain].log, [domain].log.N, [domain].log.N.gz - Logs for a particular web domain, e.g. my-site.com.

Apache2 on a VMWare virtual server

Some VMWare setups run an Avi load balancer process, which continuously makes HTTP requests to gauge the server status. Logs can therefore become long quickly. This package automatically removes all load balancing requests from the logs (these have the following user-agent signature: "avi/1.0").

Parser

The module should in most cases be called from the command line, likely in conjunction with a weekly cron job:

npx @chcaa/simple-web-log-analytics@latest -p "/tmp/log/apache2" -d "my-site.com" -o "/my-app-data/logs"

Be aware that, by default, Apache2 writes log files to /var/log/apache2/ (on Ubuntu) which is root-only. Make sure that you created a copy of the log files in a different directory that can be accessed by the user running the script. See the section for more.

CLI options

• -p, --path <fullDirPath> [required] - The full path to the Apache2 log files you wish to use.
• -d --domain-pattern <fileNamePattern> [required] - The domain pattern which match relevant log files (e.g., my-site.com). As Apache2 creates all logs in one directory, this pattern will be matched to only use the relevant domain logs. See the relevant section for more information.
• -o --output-path <outputPath> [optional] - If provided, will write the log results to the path. Must be an absolute directory path. If the directory does not exist, it will be created recursively. If not provided, the log results will be written to the log directory defined in --path.

Viewer

The Viewer module can be imported to your web project. It is quite simple and currently only exposes a single class with one method.

The Viewer module assumes that a directory exists which contains JSON log files produced by the Reader/Parser module. There must be one or more JSON files in the format requests-YYYYMMDD.json, e.g., requests-20250302.json.

Installation

npm install @chcaa/simple-web-log-analysis

Usage

The main part is the Viewer class. It exposes a constructor and one method: getLatestLog.

Constructor

The constructor takes one argument: An absolute path to a directory containing one or more parsed log file(s) in the format described above.

Throws an error if invalid, relative, or non-existing path.

GetSummary()

This method returns the contents of the latest log file as POJO (plain ol' JavaScript object). Returns null if no files JSON found within the directory which matches the filename pattern described above.

Example

import { Viewer } from '@chcaa/simple-web-log-analytics';

let viewer = new Viewer("/tmp/logs/"); // dir path
let log = viewer.getSummary(); // returns the content of the latest log file

The package also exposes the Reader and Parser classes for users who wish to write their own implementations:

import { Reader, Parser } from '@chcaa/simple-web-log-analytics';