Wondering what’s next for npm?Check out our public roadmap! »

    logstash-relay

    2.1.0 • Public • Published

    Logstash-Relay

    NPM

    Actual version published on npm Travis build status Total npm module downloads Codacy Badge Codacy Coverage Badge Dependencies badge

    A simple NodeJS service to relay JSNLogs or Log4Javascript/Log4JS(client) event messages to Logstash.

    Table of Contents

    1. Documentation
      1. Modes of Operation
        1. Stand-alone Service
          1. Installation
          2. Configuration
            1. Port Usage
            2. SSL/TLS Certs
          3. Running
        2. Embedded Service
          1. Installation
          2. Usage
          3. API
        3. AWS Lambda Function
          1. Installation, Packaging, and Deployment
          2. Usage
          3. Local Testing
        4. Docker Container
          1. Installation
          2. Configuration
          3. Running
      2. Configuration Object
      3. Logging Object
      4. Logstash
        1. Configuration
        2. Event Data
      5. REST API
        1. Endpoints
        2. Calling
    2. License

    Goals

    logstash-relay is designed to be a simple pass-through service that relays external logging events from JSNLogs or Log4JS/Log4Javascript to Logstash with the goal of easily facilitating the collection of external logging centrally into ElasticSearch. It can be embedded within another service, setup and configured as a stand-alone service, or setup and configured as an AWS Lambda function.

    Documentation

    Modes of Operation

    The logstash-relay service has four possible modes of operation:

    Stand-alone Service

    The stand-alone service option is available when it is desirable to use new/existing server architecture and more control over the environment is required.

    Installation

    To install, clone the git repository:

    $ git clone https://github.com/OutOfSyncStudios/logstash-relay.git
    cd logstash-relay

    Configuration

    To configure, run:

    npm run config

    This will ask a series of questions which provides the base configuration. Alternatively, it is possible to manually edit the <logstash-relay-home>/config/config.js file to make adjustments. The configuration file is an exported version of the Configuration Object.

    Port Usage

    By default, Logstash-Relay listens for HTTP request over port 8080 and HTTPS requests over port 8443 instead of ports 80 and 443 respectively. For security, on most Linux-based platforms the ports 80 and 443 are not available for services that do not run as the root user. To avoid any potential security issues or configuration hangups, it is recommended that Logstash-Relay is configured to use the default ports and that ports 80 and 443 are rerouted using iptables (or the system equivalent) to Logstash-Relay's default ports.

    Additionally, firewall rules should be opened to allow traffic to the appropriate ports.

    For Example:

    Add Rules

    $ iptables -t nat -A PREROUTING -i eth0 -p tcp --sport 80 --dport 8080 -j REDIRECT
    $ iptables -t nat -A PREROUTING -i eth0 -p tcp --sport 443 --dport 8443 -j REDIRECT
    $ service iptables save

    Delete Rules

    $ iptables -t nat -D PREROUTING -i eth0 -p tcp --sport 80 --dport 8080 -j REDIRECT
    $ iptables -t nat -D PREROUTING -i eth0 -p tcp --sport 443 --dport 8443 -j REDIRECT
    $ service iptables save
    SSL/TLS Certs

    If handling HTTPS requests are desired in stand-alone mode, then SSL/TLS certs must be included within the ./config/ssl folder of the project. Additionally, the file locations for the certificate files must be provided in the service configuration located in ./config/config.js. This file may be manually edited or configured using npm run config on the command-line. Both the Key and Cert files must be included and the files must have read permissions for the user that will be used to run the service.

    Running

    Before running, Logstash should be configured as outlined below. You should perform any of the operations listed below from the logstash-relay folder.

    Note: When using stand-alone mode, it is recommended that a process manager, such as PM2, be used. Regardless of how the service is run, proper startup scripts will be needed to ensure that Logstash-Relay restarts whenever the server is rebooted.

    With NodeJS
    $ node main.js
    With NodeJS and an external configuration

    It is possible to pass an external configuration file.

    $ node main.js -c <fullpath to config file>
    With PM2
    $ pm2 start main.js -n "Logstash-Relay" -i 1
    With PM2 and an external configuration

    It is possible to pass an external configuration file.

    $ pm2 start main.js -n "Logstash-Relay" -i 1 -- -c <fullpath to config file>

    Embedded Service

    The embedded option is available if including the service bundled as a part of another service is desired.

    Installation

    npm install logstash-relay

    Usage

    Within a library or application, add the following code:

    const LogstashRelay = require('logstash-relay');
     
    // Create the relay service
    let relay = new LogstashRelay(config, logger);
     
    // Start the relay service
    relay.init();
     
    // Close the relay service
    relay.close();

    Note: The same considerations for port usage and certificates apply to using the service in embedded mode.

    API

    constructor(config[, logger])

    Creates a new Logstash-Relay agent.

    const LogstashRelay = require('logstash-relay');
     
    let relay =  new LogstashRelay(config, logger);

    Where the config and logger parameters are as outlined below. The logger is optional, and if no logger is provided then all logging is sent to /dev/null.

    .init()

    Initializes and starts the Logstash-Relay agent.

    relay.init();
    .close()

    Shuts down the Logstash-Relay agent. Because the agent maintains an active thread, this operation must be performed to allow the application to gracefully shut down.

    relay.close();

    AWS Lambda Function

    The service is also available to be run in a completely serverless environment by utilizing it as an AWS Lambda Function connected through CloudFormation and API Gateway. To use this mode, the AWS credentials must have permissions to create and use CloudFormation, S3, API Gateway, CloudFront, and Lambda resources.

    Installation, Packaging, and Deployment

    These steps support MacOS and Linux. When using Windows environment locally, it is recommended that a staging AWS EC2 Linux instance with NVM(NodeJS) and Python is setup and used.

    1. If it is not already, install the AWS CLI on the staging environment (MacOS/Linux only).
    2. Clone the git repo:
    $ git clone https://github.com/OutOfSyncStudios/logstash-relay.git
    cd logstash-relay
    1. Run the base configurator:
    $ npm run config
    1. Run the AWS configurator:
    $ npm run aws-config
    1. Run the Setup:
    $ npm run setup

    This last step will create an S3 Bucket and a CloudFormation Stack. The CloudFormation Stack will in turn setup an AWS Lambda function and connect it to CloudFront and API GateWay. Additionally, routing through API Gateway and Route53 are possible to create a "pretty" URL that can connect to the API Gateway endpoint for the Lambda function. Please consult the AWS API Gateway documentation for additional details.

    Usage

    The Lambda function can be called by making the appropriate REST Endpoint request to the API Gateway prod endpoint. The correct prod Invoke URL can be retrieved throug the API Gateway Dashboard under APIs > (Your LogStashRelayAPI Name) > Stages.

    Local Testing

    To test as an AWS Lambda function, it is highly recommended to use the AWS SAM CLI Tool with Docker from your local environment and ensure that the Invoke URL returns data as expected with your configuration before deployment to AWS.

    Docker Container

    Installation

    To install, clone the git repository:

    $ git clone https://github.com/OutofSyncStudios/logstash-relay.git
    $ cd logstash-relay
    

    Configuration

    To configure, run:

    npm run config

    This will ask a series of questions which provides the base configuration. Alternatively, it is possible to manually edit the <logstash-relay-base>/config/config.js file to make adjustments. The configuration file is an exported version of the Configuration Object.

    Running

    Before running, ensure that a compatible statsd service is configured to listen for events on the configured port and the any firewall rules are open between the service and the statsd service.

    Building the Docker Image
    $ npm run docker-build
    Running the Built Docker Image
    $ npm run docker-run

    Configuration Object

    {
      server: {
        port: 8080,
        shutdownTime: 1000,
        timeout: 5,
        sslEnabled: false,
        sslPort: 8443,
        sslKey: `${__dirname}/ssl/localhost.key`,
        sslCert: `${__dirname}/ssl/localhost.pem`
      },
      logging: {
        // Logging Configuration
        logDir: './logs',
        options: { json: false, maxsize: '10000000', maxFiles: '10', level: 'silly' },
        logstashLogging: false
      },
      logstash: {
        logging: { host: 'logstash-server', port: 5000, appName: 'logstash-relay' },
        relay: { host: 'logstash-relay-server', port: 5050, appName: 'client-errors' }
      }
    }
    parameter type description
    server.port Integer The port to listen for HTTP requests
    server.shutdownTime Integer Time in millisecond to allow for graceful shutdown
    server.timeout Integer Time in seconds to wait to receive data from client before timeing out
    server.sslEnabled Boolean Enable handling of HTTPS requests (stand-alone and embedded modes only)
    server.sslPort Integer The port to listen for HTTPS request when sslEnabled === true
    server.sslKey String The full path to the Certificate .key file for HTTPS request when sslEnabled === true
    server.sslCert String The full path to the Certificate .pem/.crt file for HTTPS request when sslEnabled === true
    logging Object Required but only used in stand-alone mode. These options only pertain to logging events generated by the service, not events which are being relayed through the endpoints.
    logging.logDir Integer The full or relative path (from the Logstash-Relay base folder to store logs
    logging.options.json Boolean Store Logstash-Relay service events in JSON format
    logging.options.maxsize Integer String Max logfile size in bytes before logrotation
    logging.options.maxFiles Integer String Max number of rotated logfiles to keep for logrotation
    logging.options.level String The lowest log level to store in files (silly,debug,info,warn,error)
    logging.logstashLogging Boolean Send service log events to logstash
    logstash.logging Object Logstash information for Logstash-Relay log events when logging.logstashLogging === true
    logstash.relay Object Logstash information for relayed log events
    logstash.*.host String IP/Domain of the logstash server for this configuration
    logstash.*.port Integer UDP/TCP Port that handled the events
    logstash.*.appName String Unique identifying name that Logstash uses to classify events sent to ElasticSearch
    logstash.*.mode String Mode to send logstash messages. Expect the value 'udp' or 'tcp'. If no value is provided, then UDP is used

    Logging Object

    The Logging object is an instance of any logging library, such as Winston or Bunyan, which support the .error(...), .info(...), .debug(...), and .log(...) methods. When in stand-alone mode, the service will use the configuration values to create an instance of Winston.

    Logstash

    Configuration

    This package contains four files -- ./config/logstash-relay-tcp.conf, ./config/logstash-relay-udp.conf, ./config/logstash-logging-tcp.conf and ./config/logstash-logging-udp.conf -- that are necessary for proper configuration of the Logstash to listen for incoming events on the correct TCP/UDP ports. Only the associated one UDP or TCP file is needed depending on the configuration options.

    logstash-logging-*.conf is only used for the stand-alone mode and handles events created by Logstash-Relay itself when the option logging.logstashLogging is set to true in the service configuration.

    logstash-relay-*.conf handles all events that are passed to the relay and is necessary for proper functioning.

    To setup Logstash with these files:

    1. Copy the appropriate files to the Logstash server instance configuration folder. This is usually /etc/logstash/conf.d for most configurations.
    2. Edit the file(s) and update the ElasticSearch server host address as appropriate for the environment.
    3. Restart Logstash -- usually this is done with the command $ initctl restart logstash.

    Event Data

    Event data is passed to Logstash as a JSON object formatted like below:

    {
      type: 'client_error',
      name: 'The name of the log',
      requestID: 'af4b33d2ae870d',
      client_error: 'error message or JSON object',
      actualIP: 'client_IP_address',
      ip: 'forwarded_client_IP_address_from_proxy',
      callID: 'UID for the relay request',
      headers: 'An object containing all http headers sent to the service',
      clientTimestamp: timestamp
    }
    parameter type description
    type String Always 'client_error'
    name String The log name from JSNLogs events, not provided in Log4JS events
    requestID String A unique id used by JSNLogs to group events together, not provided in Log4JS events
    client_error String or Object The error message or object that was provided in the event
    actual_IP String The IPv4 or IPv6 of the request origin. If the request was proxied, this will be the proxy server address
    ip String The IPv4 or IPv6 of the request origin. If the request contains an x-forward header, then this will be the forwarded address
    callID String A UUIDv4 for the Logstash-Relay request, used for request tracking
    headers Object An object containing all the request headers from the origin. This usually contains the user-agent and other important details
    clientTimestamp Timestamp A timestamp of when the error was generated on the client. For JSNLogs, this will be the time the event was generated. For Log4JS, this will be the time the event was recieved by the relay.

    REST API

    Once the service is setup and running, then the REST API microservice will be available. The service provides two identical POST endpoints.

    Endpoints

    OPTIONS *

    This exists for CORS requests.

    POST /api/logger

    POST jsnlog.logger

    Takes standard JSNLogs or Log4JS events passed in the POST body, and sends them to the configured Logstash service.

    JSNLogs Events

    JSNLogs events are structured as follows:

    {
      r: 'requestID',
      lg: [
        {
          n: 'logName',
          l: 'logLevel',
          t: timestamp
          m: 'logMessage (may be a JSON object or string)'
        },
        ...
      ]
    }

    Log4JS Events

    {
      level: 'logLevel',
      message: 'logMessage (string only)'
    }

    Calling

    Setup JSNLogs or Log4JS to create AJAX requests that point to http(s)://<yourserviceURL>/api/logger or http(s)://<yourserviceURL>/jsnlog.logger. Alternatively, call these endpoints directly within an application using one of the log formats outlined above.

    License

    Copyright (c) 2017, 2018, 2019 Jay Reardon Copyright (c) 2019 Out of Sync Studios LLC -- Licensed under the MIT license.

    Install

    npm i logstash-relay

    DownloadsWeekly Downloads

    6

    Version

    2.1.0

    License

    MIT

    Unpacked Size

    94.7 kB

    Total Files

    34

    Last publish

    Collaborators

    • avatar