@dynatech-corp/nestjs-fluentd-logger

Fluentd Logger

A Logger wrapper that allows to send your application logs to an instance of Fluentd.

The library supports NestJS context, log severity and hostname collection (for distributed systems).

Installation

Install @dynatech-corp/nestjs-fluentd-logger package

# npm installation
npm install @dynatech-corp/nestjs-fluentd-logger

# yarn installation
yarn add @dynatech-corp/nestjs-fluentd-logger

Usage

This section covers multiple use-cases for this logger.

There are two providers that implement functionality.

Provider FluentConnection handles communication with Fluentd instance. FluentConnection provider accepts the same connection parameters as the underlying @fluent-org/logger library. The connection is lazy-loaded, meaning it is initiated when the first logs appear, not on module initialization.

Provider FluentLogger is an interface that allows to log different severity and context messages to FluentConnection. FluentLogger is created with Transient scope, which ensures independent logger instance for each injection.

Logging to Fluentd

The basic use case is logging information directly to Fluentd.

Register fluent logger services in you AppModule

import { Module } from '@nestjs/common';
import {
  FluentLogger,
  FluentConnection,
} from '@dynatech-corp/nestjs-fluentd-logger';

@Module({
  providers: [
    FluentConnection,
    FluentLogger,
  ],
})
export class AppModule {}

Use logger in your bootstrap file

Option bufferLogs is used to buffer logs that are generated before logger is initialized.

Logger is initialized with await app.resolve, because it had a dependency on another service FluentConnection, that is initialized with Dependency Injection.

Calling app.flushLogs() right after we initialize logger passes the logs to the logger right away. This helps to prevent cases when there's an initialization error and no logs are displayed.

import { NestFactory } from '@nestjs/core';
import { AppModule } from './app.module';
import { FluentLogger } from '@dynatech-corp/nestjs-fluentd-logger';

async function bootstrap() {
  const app = await NestFactory.create(AppModule, {
    // buffer logs untill logger is initialized
    bufferLogs: true,
  });

  // use FluentLogger as the main logger
  app.useLogger(await app.resolve(FluentLogger));
  // flush logs after we setup the logger
  app.flushLogs();

  await app.listen(3000);
}
bootstrap();

Write logs from your controllers / services

Using Logger proxy-class seems to currently be the cleanest approach to initializing Logger with proper context.

Internally it uses the logger we provided during bootstrap with app.useLogger.

import { Controller, Get, Logger } from '@nestjs/common';

@Controller()
export class AppController {
  // create logger instance with proper context
  private readonly logger = new Logger(AppController.name);

  @Get()
  getHello(): string {
    // log your message
    this.logger.log('Log "Hello, world!" message');
    // ... do stuff
    return 'Hello, world!';
  }
}

Customize Fluentd connection

There are cases when you want to log data to a custom destination, e.g. Fluentd that's located at another IP address, listens to a non-default port, etc.

Custom configurations can be passed to FluentdConnection provider's constructor. The parameters use an interface from @fluent-org/logger.

import { Module } from '@nestjs/common';
import {
  FluentLogger,
  FluentConnection,
} from '@dynatech-corp/nestjs-fluentd-logger';

@Module({
  providers: [
    // fluentd connection service
    {
      provide: FluentConnection,
      useFactory: () => {
        return new FluentConnection({
          prefix: 'logs.my_app',
          connection: {
            socket: {
              host: '10.0.2.12',
              port: 20022,
              timeout: 10000,
            },
          },
        });
      },
    },
    // fluentd logger
    FluentLogger,
  ],
})
export class AppModule {}

Customize connection with Environment variables

For cases when you have different environments (dev/stage/prod) and want to configure your log delivery accordingly, you can use @nestjs/config library to provide dynamic configurations to the connection.

Install @nestjs/config

Get configurations module into your project

npm install @nestjs/config

Use configurations module inside factory

By injecting ConfigService into provide factory, you can pass custom configurations from your .env file. Note that you can additionally configure ConfigService to fetch configurations from different sources (like json or yaml files, etc.)

import { Module } from '@nestjs/common';
import { ConfigModule } from '@nestjs/config';
import {
  FluentLogger,
  FluentConnection,
} from '@dynatech-corp/nestjs-fluentd-logger';

@Module({
  imports: [
    // global config module
    ConfigModule.forRoot({ isGlobal: true }),
  ],
  providers: [
    // fluentd connection service
    {
      provide: FluentConnection,
      useFactory: (config: ConfigService) => {
        return new FluentConnection({
          prefix: config.get<string>('LOGS_PROJECT'),
          connection: {
            socket: {
              host: config.get<string>('LOGS_HOST'),
              port: config.get<number>('LOGS_PORT'),
              timeout: config.get<number>('LOGS_TIMEOUT'),
            },
          },
        });
      },
      inject: [ConfigService],
    },
    // fluentd logger
    FluentLogger,
  ],
})
export class AppModule {}

Create .env file with configuration parameters

Example .env file for FluentConnection provider configuration.

LOGS_PROJECT=logs
LOGS_HOST=127.0.0.1
LOGS_PORT=24224
LOGS_TIMEOUT=10000

Configure logs destination with environment variables

Despite the fact that centralized logging with fluentd is crucial for production systems, it's not convenient for development.

You would want to have logs displayed in console output during development, but have them in your centralized log analysis system in staging/production.

Luckily, this can be done with provider useFactory and @nestjs/config module.

Create logger with useFactory

Create factory, that determines what logger to use based on LOGS_OUTPUT env variable value.

Note that internally FluentConnection lazy-loads the connection. So in case when we use ConsoleLogger as our main logger, the connection isn't tried and there are no errors.

import { ConsoleLogger, Logger, Module } from '@nestjs/common';
import {
  FluentLogger,
  FluentConnection,
} from '@dynatech-corp/nestjs-fluentd-logger';
import { ConfigModule, ConfigService } from '@nestjs/config';

@Module({
  imports: [ConfigModule.forRoot({ isGlobal: true })],
  providers: [
    {
      provide: FluentConnection,
      useFactory: (config: ConfigService) => {
        return new FluentConnection({
          prefix: config.get<string>('LOGS_PROJECT'),
          connection: {
            socket: {
              host: config.get<string>('LOGS_HOST'),
              port: config.get<number>('LOGS_PORT'),
              timeout: config.get<number>('LOGS_TIMEOUT'),
            },
          },
        });
      },
      inject: [ConfigService],
    },
    {
      provide: Logger,
      useFactory: (config: ConfigService, fluent: FluentConnection) => {
        // get LOGS_OUTPUT variable value
        const output = config.get<string>('LOGS_OUTPUT');
        // create NestJS ConsoleLogger for development (console)
        if (output === 'console') {
          return new ConsoleLogger(undefined, { timestamp: true });
        }
        // create FluentLogger instance for staging / production
        if (output === 'fluent') {
          return new FluentLogger(fluent);
        }
        // throw error when the variable is not Configured
        throw new Error('LOGS_OUTPUT should be console|fluent');
      },
      // inject ConfigService - for configuration values
      // inject FluentConnection - for when FluentLogger is instantiated
      inject: [ConfigService, FluentConnection],
    },
  ],
})
export class AppModule {}

Change bootstrap function to use generic Logger

Because our factory provides generic NestJS Logger in provider factory provide: Logger, we need to change our bootstrap signature to use that logger.

Now we use app.useLogger(await app.resolve(Logger)) NestJS generic Logger, which is dynamically provided with Dependency Injection.

import { NestFactory } from '@nestjs/core';
import { AppModule } from './app.module';
import { Logger } from '@nestjs/common';

async function bootstrap() {
  const app = await NestFactory.create(AppModule, {
    bufferLogs: true,
  });

  // we use generic nestjs Logger
  // that would get resolved to what we configured
  app.useLogger(await app.resolve(Logger));
  app.flushLogs();

  await app.listen(3000);
}
bootstrap();

Add parameters to .env file

Defing which logger you want to use in this environment with LOGS_OUTPUT variable.

# would configure logger to write to flunet
LOGS_OUTPUT=fluent
# would configure logger to write to console output
#LOGS_OUTPUT=console

Development

The approach for development was taken from Publishing NestJS Packages with npm by NestJS.

Install dependencies

npm install

Start watching for library files

npm run start:dev

Start docker composer

docker compose up

Install dependencies for test-app

cd test-app

Install test-app dependencies

npm install

Start test-app

npm run start:dev

Stay in touch

• Author Bogdans Ozerkins
• Website https://dynatech.lv