Responsible: #research-and-development

@rplan/express-middleware

This is a collection of reusable express middlewares for logging and error handling.

Introduction

All our NodeJs servers should use the serverLifecycle. It ensures a correct lifecycle in the context of K8s.

// example lifecycle
const app = new Express()
const port = 3000
const { shutdown, shutdownCompleted } = handleServerLifecycle(app, port, {
  onShutdown: async () => { disconnectFromDbs() },
})

Also each server should use our standard middleware:

import { createWebapiServer } from '@rplan/express-middleware'

function createApp() {
  const app = createWebapiServer({
    // list you readyness checks. No requests will be routed if not ready
    readinessChecks: [
      ['DB connection', () => throw new Error('DB not connected')]
    ],
    metrics: {
      // list the routes which contain IDs so they get aggregated in prometheus
      pathPatterns: ['/entities/:id'],
    },
    // Validates default reqHeaders (x-user-id, ...). Default: true
    validateRequestHeaders: true,
  })
}

Collection of Middlewares

These are usually used and already provided by the createWebapiMiddleware()

// example middleware
import {
  requestIdMiddleware, requestLogger, loggingHandler, createHealthRoutes, validateRequestHeaders,
} from '@rplan/express-middleware'
// container probes /live /ready /health
// (used by kubernetes live cycle. Make sure to configure the routes in your deployment)
app.use(createHealthRoutes([
  ['db connection', async () => db.ping()],
]))
// provides the requestId (part of the req header, set by the api gateway)
app.use(requestIdMiddleware())
// validates if required headers are sent (e.g. x-user-id, x-request-id, ...)
app.use(validateRequestHeaders())
// provides a request logger configured with request and reqId
app.use(requestLogger())
// does request logging (needed by our security guidelines)
app.use(loggingHandler(HANDLER_LOG_LEVEL.INFO))
router.use(requestContext)

// the actual service routes
app.use(someRoute)

createHealthRoutes()

This middleware sets up the required health routes of the service:

• /live: The service is started up and is not broken
• /ready: The service is ready to accept connections (e.g. DBs are connected)

These routes are used by K8s in order to check the state of the service. Both will automatically switch to not ready/live on shutdown of the service.

Normally you only want to configure the readynessChecks and let it check for working DB connections. A service may enter a "non-ready" state from time to time if DBs are not reachable. In this case K8s will not route new requests to it but the service will not be restarted.

In case your service can reach a broken state, just shutdown the service (see serverLifecycle()).

Only in rare occasions you want to configure the livenessChecks:

• the service can reach a broken state
• it could be that the service can repair itself

If a service is not "live" anymore for a longer time, K8s will kill and restart it.

Configuration

The buckets for metrics collection can be configured using an environment variable:

metricsBuckets='[10, 100, 1000]' node lib/svc

This makes it easier to optimize buckets for a specific service.

catchAsyncErrors

A middleware wrapper that catches errors of the underlying middleware and pass it to the next function

  app.get('/some-route', catchAsyncErrors(async (req, res) => {
    if (errorCondition) {
      throw new Error('foo') // error can now be handled in a error middleware
    }
    // ...
  }))

unexpectedErrorHandler

This middleware sends the http status code 500 to the client, if an unexpected error occurs. The error is logged with the module @rplan/logger.

The unexpectedErrorHandler should be added as the last middleware to the express app.

Example for adding the middleware:

  import express from 'express'
  import { unexpectedErrorHandler } from '@rplan/express-middleware'
  
  // ...
  
  const app = express()
  
  app.use(someRoute)
  
  app.use(unexpectedErrorHandler)

expectedErrorHandler

This middleware sends a http status code 4xx, if an expected error occurs. The error is logged with the module @rplan/logger. Place the middleware at the end, but before the unexpectedErrorHandler.

Example for adding the middleware:

  import express from 'express'
  import { expectedErrorHandler, unexpectedErrorHandler } from '@rplan/express-middleware'
  
  // ...
  
  const app = express()
  
  app.use(someRoute)
  
  app.use(expectedErrorHandler)
  app.use(unexpectedErrorHandler)

Predefined standard errors

There are the following standard errors defined:

• NotFoundError (sends 404)
• ConflictError (sends 409)
• BadRequestError (sends 400)
• ForbiddenError (sends 403)
• UnauthorizedError (sends 401)

  app.get('/some-route', catchAsyncErrors(async (req, res) => {
    if (notFoundCondition) {
      throw new NotFoundError('foo') 
      // middleware sends status 404 with the body { name: 'NotFoundError', message: 'foo' }
    }
    // ...
  }))

`

Custom errors

With the function registerError custom errors can be registered together with a http status code.

    // custom error declaration
    export class CustomError extends Error {}
    CustomError.prototype.name = CustomError.name

    // register the error
    import { registerError } from '@rplan/express-middleware'
    registerError(CustomError, 442)

    // throw the error
    app.get('/some-route', catchAsyncErrors(async (req, res) => {
      if (customFoundCondition) {
        throw new CustomError('custom') 
        // middleware sends status 442 with body { name: 'CustomError', message: 'custom' }
      }
      // ...
    }))

Make sure that the name property of the error is unique, errors with same name can't be registered twice.

The message of the error is exposed to the calling client. Make sure that the error message do not contain any security information, like session ids.

loggingHandler

Using this middleware can help to have a uniform standard logging of requests and to have unique indexes in elk/ kibana stack.

This middleware should be placed at the top and use @rplan/logger for logging.

  import express from 'express'
  import { loggingHandler, HANDLER_LOG_LEVEL } from '@rplan/express-middleware'
  
  // ...
  
  const app = express()
  
  app.use(loggingHandler(HANDLER_LOG_LEVEL.INFO))
  
  app.use(someRoute)

requestMetrics

Collect metrics of requests based on the request method, path and response status code. The metrics are collected with prom-client which should be made available as a peer dependency. The middleware only collects metrics but doesn't provide an endpoint for prometheus itself. A metrics endpoint has to be provided on its own using prom-client.

The following metrics are collected:

• http_requests_total - Counts all requests using labels for method, path and status
• http_request_duration_ms - Collects response times in a histogram using labels for method, path and status

Options:

• pathPatterns - array of express path patterns which are used to normalize paths. This is helpful for endpoints which contain path parameters and will collect all corresponding requests with the pattern as path label
• ignoredPaths - array of paths to ignore for metric collection. Also recognizes path patterns.
• requestDurationBuckets - the buckets to use for the request duration histogram

import express from 'express'
import { requestMetrics } from '@rplan/express-middleware'

const app = express()

app.use(requestMetrics({
  pathPatterns: [
    '/foo/:id',
    '/foo/:id/test',
  ],
  ignoredPaths: [
    '/metrics',
  ],
  requestDurationBuckets: [10, 100, 1000, 2000],
}))

app.get('/foo/:id', (req, res) => {
  // ....
})

detectAbortedRequests

Detects if the client side aborted/closed the request prematurely. In case the request is detected as aborted it will create a corresponding log entry. Additionally, this middleware provides an API for checking if the request has been aborted.

import express from 'express'
import { 
  detectAbortedRequests,
  isAbortedByClient,
  HANDLER_LOG_LEVEL,
} from '@rplan/express-middleware'

const app = express()

app.use(detectAbortedRequests({
  logLevel: HANDLER_LOG_LEVEL.INFO,
}))

app.get('/foo/:id', (req, res) => {
  if (isAbortedByClient(req)) {
    // do something if aborted
  } else {
    // ...
  }
})

requestIdMiddleware

Ensures that each request gets a unique id which can be used for correlation, e.g. to correlate all log entries which belong to a particular request. Takes either a client provided request id from the x-request-id header or generates a new request id.

import express from 'express'
import {
  requestIdMiddleware,
  getRequestId,
  requestLogger,
  loggingHandler,
} from '@rplan/express-middleware'

const app = express()
app.use(requestIdMiddleware())
app.use(requestLogger())
app.use(loggingHandler())

app.use('/foo', (req, res) => {
  const requestId = getRequestId(res)
  // ...
})

requestContext

The requestContext middleware provides a convenient API for request scoped properties. The base version provides access to the headers for futher service request, request id and request logger (requires the corresponding middlewares) but the context can be extended by the consumer as needed.

import express from 'express'
import {
  initializeRequestContext,
  RequestContextBase,
  requestIdMiddleware,
  requestLogger,
  loggingHandler,
} from '@rplan/express-middleware'
import { resourceService } from './resource-service'
const {
  requestContext,
  getRequestContext,
} = initializeRequestContext(req => new RequestContextBase())

const app = express()
app.use(requestIdMiddleware())
app.use(requestLogger())
app.use(loggingHandler())
app.use(requestContext)

app.get('/foo', (req, res) => {
  const ctx = getRequestContext(req)
  ctx.getLogger().info('logging via request context logger')

  const absences = resourceService.fetchAbsences({
    projectId: req.params.projectId
  }, {
    headers: ctx.getServiceRequestHeader()
  })
  // ...
})

Custom Headers

There are certain custom headers that are scoped to the request context(mentioned above) and serve a specific purpose under the context of Allex.

Header	Type	Description
x-organization-id	string	The ID of the organization. It is only used by M2M token authentication in the customer API.
x-user-id	string	The ID of the user. It is used to identify the user that made the request.
x-service-id	string	The ID of the service. It is used to identify the service within the cluster that made the request.
x-request-id	string	The ID of the request. It is used to assign a unique ID to each request which is propagated through the cluster to link requests which helps in troubleshooting.
x-skip-permission-checks	boolean	Boolean flag to specify if permissions should be skipped. It is used to bypass permissions when they become redundant as services talk to eachother and perform actions based on permissions.

These headers are mentioned in the context of this documentation wherever important.

handleServerLifecycle

Provides a convenient method for gracefully handling the whole HTTP server lifecycle from startup to shutdown. In particular callbacks for startup and shutdown can be provided to be notified when the HTTP server started listening and to run cleanup on server shutdown. Additionally, it provides a graceful shutdown period to normally complete still running requests without accepting new requests.

import express from 'express'
import {
  handleServerLifecycle,
} from '@rplan/express-middleware'

const app = express()

app.get('/foo', (req, res) => {
  // ...
})

async function main() {
  const { shutdown, shutdownCompleted } = await handleServerLifecycle(
    app,
    3000,
    {
      onStart() {
        console.log('server started')
      },
      onShutdown() {
        console.log('server shutting down')
        // do cleanup
      },
    }
  )
  
  // shut down server programmatically
  setTimeout(shutdown, 10000)
  await shutdownCompleted
}

validateRequestHeaders

Ensures that each request has the following required headers:

• 'x-user-id', to identify the user of the context
• 'x-request-id', a unique id to track the request and upstream requests made because of the request

If one of these headers is not present the middleware will answer the request with status code Bad Request (status 400).

Note: The validateRequestHeaders should be added after the /health, /live, /ready and /metrics routes. Otherwise these routes will not work anymore, because of the missing headers.

Elastic Application Performance Monitoring(APM)

The createWebapiServer function takes care of setting up and running the Elastic APM agent so if you are using it to set the service up then the Elastic APM agent is integrated into your service out of the box.

If you are not using the createWebapiServer function to set up your service, there is a startElasticApmAgent function that you can invoke to set up the Elastic apm agent. This function should be called right at the start of your service, before executing any code.

import { startElasticApmAgent } from '@rplan/express-middleware'

// Right at the start of your service
startElasticApmAgent()

The activation of the Elastic APM agent is dependent on the ELASTIC_APM_ACTIVE environment variable. It should be explicitly set to true for the agent to be started. If this environment variable is missing or set to any other value than true then calling both the createWebapiServer and/or startElasticApmAgent function will not start the agent.

If you are using the createWebapiServer function but still want to be in control of starting the APM agent then you can pass the active option as false and manually call startElasticApmAgent. Without setting active to false, the APM agent will start more than once and result in an error.

import { startElasticApmAgent, createWebapiServer } from '@rplan/express-middleware'

const app = createWebapiServer({
  elasticApmAgentConfigOptions: {
    active: false
  },
})

startElasticApmAgent()