Worker thread based async Console Transport for winston
Whats new in version 1.0.2
- Added
ConsoleTransportInWorker#flush
method for allowing manual termination of the worker thread from the parent context
The Problem
In NodeJS's V8 and similarly in other JS engines console.log
is just a wrapper around process.stdout.write
with formatting options and it is a Blocking IO operation. In case of large payloads (API responses, formatted buffers etc) writing to console can be a very resource and time consuming operation, which can cause performance degradation in APIs. Many corporations demand for capturing every event in and out in their web-based software systems, mainly for Audit and Compliance reasons.
In modern NodeJS backend applications the Winston logging library is the most commonly used approach to tackle logging tasks due to its flexibility on Log Entry formatting, and the numerous Transport options available, such as HTTP, Stream, File and Console, next to many community built extensions.
Most of PaaS Cloud Platforms and Dockerized hosting solutions, whether Managed or built on top of one of the available Open-Source Cloud Orchestration tools however demanding for Console based logging strategy, due to the simplicity of capturing the outputs of process models realtime, which have some challenges on its own, especially at scale.
The Solution
This extension is aimed to overcome two limitations of the NodeJS platform:
- throttling the event loop when logging operation is dispatched - eg blocking applications to process new incoming requests, or delaying the execution of business-critical logic
- delaying response delivery to clients if the payload of a log entry is large and the logging operation takes place before serving the requestor networked-component
To overcome these I have created an extension on top of winston.transports.Console
called ConsoleTransportInWorker
which works the following way:
- when the transport is configured for a
winston.Logger
instance, on initialization the Strategy is creating a separate Worker Thread usingworker_threads
internal module (node >=v16) - when a logging operation is dispatched, the log event object is forwarded to this Worker Thread
- the Worker Thread - using its own separate Event Loop - performs the Blocking
console.log
IO operation - note, that the dispatcher's thread is not affected at this point - when the write was successful, the Worker Thread dispatches an event back to the main thread, which then triggers the logging queue to continue processing the next entries
Usage
import * as winston from 'winston';
import {
ConsoleTransportInWorker
} from '@greeneyesai/winston-console-transport-in-worker';
...
export const logger: winston.Logger = winston.createLogger({
format: combine(timestamp(), myFormat),
level: Level.INFO,
transports: [new ConsoleTransportInWorker()],
});
Benchmark results
Logging pre-generated large strings, ellapsed time between logging call (transport.log(...)
) start and completion
License
GNU Lesser General Public License v3.0
© GreenEyes Artificial Intelligence Services, LLC.