cloudwatch-backend-for-statsd

This repository is a clone of https://github.com/camitz/aws-cloudwatch-statsd-backend

For maintainability purpose and improvements I decided to clone the repository.

StatsD backend for AWS CloudWatch

Overview

StatsD is a smart Node.js package that collects and aggregates statistics from differents apps sent over the UDP protocol. At a set time interval it forwards the aggregated data to a configured backend. It is pluggable with several backends available, the most popular being Graphite, a python/django monitoring tool.

With cloudwatch-backend-for-statsd you can replace Graphite in favour of AWS Cloudwatch for your monitoring purposes, appropriate for sites on the Amazon EC2 cloud.

Counters, timers, gauges and sets are all supported.

Installation

You need node.js installed on your system aswell as StatsD. Follow the instructions on their sites or see this blog post/tutorial on how to install these components on a Windows system.

The CloudWatch backend is an npm package that can be installed with the npm command which comes with your installation of node.js. Go to the npm site for more information.

npm install cloudwatch-backend-for-statsd

The package is based on aws-sdk.

Configuration

The StatsD and its backends are configured in a json object placed in a file supplied to StatsD at the command line. For example, start StatsD with the following.

node ./stats.js ./myConfig.js

The following demonstrates the minimum config for the CloudWatch backend.

{
    backends: [ "cloudwatch-backend-for-statsd" ],
    cloudwatch: 
    {
        accessKeyId: 'YOUR_ACCESS_KEY_ID', 
        secretAccessKey:'YOUR_SECRET_ACCESS_KEY', 
        region:"YOUR_REGION"
    }
}

The access keys can be you personal credentials to AWS but it is highly recommended to create an ad hoc user via Amazon's IAM service and use those credentials.

The region is for example eu-west-1 or us-east-1.

The above will create a metric with the default namespace, AwsCloudWatchStatsdBackend, and send an http request to CloudWatch via awssum.

See the CloudWatch documentation for more information on these concepts.

The metric name, unit and value depends on what you send StatsD with your UDP request. For example, given

gorets:1|c

the Unit will be Counter, the metric name gorets. The value will be the aggregated count as calculated by StatsD.

ms corresponds the unit Milliseconds. *s and g to None.

Warning Indescriminate use of CloudWatch metrics can quickly become costly. Amazon charges 50 cents for each combination of namepace, metric name and dimension per month. However, the 10 first per month are free.

Using AWS Roles to obtain credentials (highly recommended)

If you don't add the accessKeyId and secretAccessKey to the configuration file, aws-sdk will automatically get the credentials regarding the AWS roles on your instance (via the metadata).

{
    backends: [ "cloudwatch-backend-for-statsd" ],
    cloudwatch:
    {
        region: 'YOUR_REGION',
        whitelist: ['YOUR_FULL_METRIC_NAME']
    }
}

Additional configuration options

The cloudwatch backend provides ways to override the name and namespace by cofiguration. It can also capture these components from the bucket name.

The following overrides the default and any provided namespace or metric name with the specified.

{
    backends: [ "cloudwatch-backend-for-statsd" ],
    cloudwatch: 
    {
        accessKeyId: 'YOUR_ACCESS_KEY_ID', 
        secretAccessKey: 'YOUR_SECRET_ACCESS_KEY', 
        region: 'YOUR_REGION',
        namespace: 'App/Controller/Action', 
        metricName: 'Request'
    }
}

Using the option processKeyForNamespace (default is false) you can parse the bucket name for namespace in addition to metric name. The backend will use the last component of a bucket name comprised of slash (/), dot (.) or dash (-) separated parts as the metric name. The remaining leading parts will be used as namespace. Separators will be replaced with slashes (/).

{
    backends: [ "cloudwatch-backend-for-statsd" ],
    cloudwatch: 
    {
        accessKeyId: 'YOUR_ACCESS_KEY_ID', 
        secretAccessKey: 'YOUR_SECRET_ACCESS_KEY', 
        region: 'YOUR_REGION',
        processKeyForNamespace:true
    }
}

For example, sending StatsD the following

App.Controller.Action.Request:1|c

will produce the equivalent to the former configuration example. Note that both will be suppressed if overriden as in the former configuration example.

Whitelisting Metrics

Using cloudwatch will incur a cost for each metric sent. In order to control your costs, you can optionally whitelist those metrics sent to cloudwatch. For example:

{
    backends: [ "cloudwatch-backend-for-statsd" ],
    cloudwatch: 
    {
        accessKeyId: 'YOUR_ACCESS_KEY_ID', 
        secretAccessKey: 'YOUR_SECRET_ACCESS_KEY', 
        region: 'YOUR_REGION',
        whitelist: ['YOUR_METRIC_NAME']
    }
}

The above configuration would only sent the metric named 'YOUR_FULL_METRIC_NAME' to cloudwatch. As this is an array, you can specify multiple metrics.

The metric name can also be represented as a regexp (we automatically add '^' and '$' around the metric name). For example you can pass :

{
    backends: [ "cloudwatch-backend-for-statsd" ],
    cloudwatch: 
    {
        accessKeyId: 'YOUR_ACCESS_KEY_ID', 
        secretAccessKey: 'YOUR_SECRET_ACCESS_KEY', 
        region: 'YOUR_REGION',
        whitelist: ['metric.name.error', '.*name.database.*']
    }
}

The above configuration would sent the metric named metric.name.error or my.metricname.database.error or toto.name.database.mysql.error ...etc to cloudwatch.

You can also simply pass a string :

{
    backends: [ "cloudwatch-backend-for-statsd" ],
    cloudwatch: 
    {
        accessKeyId: 'YOUR_ACCESS_KEY_ID', 
        secretAccessKey: 'YOUR_SECRET_ACCESS_KEY', 
        region: 'YOUR_REGION',
        whitelist: 'instance[1-9].database.*'
    }
}

This is useful if you are using multiple backends e.g. mysql backend and want to send some metrics cloudwatch (due to the associated cost) and all the metrics together to another backend. It is also useful if you want to limit the metrics you use in cloudwatch to those that raise alarms as part of your wider AWS hosted system.

Multi-region support

If you wish to send cloudwatch metrics to multiple regions at once, instead of

{
    backends: [ "cloudwatch-backend-for-statsd" ],
    cloudwatch: 
    {
        accessKeyId: 'YOUR_ACCESS_KEY_ID', 
        secretAccessKey:'YOUR_SECRET_ACCESS_KEY', 
        region:"YOUR_REGION"
    }
}

you can use the instances key under cloudwatch to configure a list of configurations.

{
    backends: ["cloudwatch-backend-for-statsd"],
    cloudwatch: {
        instances: [{
            accessKeyId: 'YOUR_ACCESS_KEY_ID',
            secretAccessKey: 'YOUR_SECRET_ACCESS_KEY',
            region: "YOUR_REGION_1",
            whitelist: ['YOUR_FULL_METRIC_NAME1']
        }, {
            accessKeyId: 'YOUR_ACCESS_KEY_ID',
            secretAccessKey: 'YOUR_SECRET_ACCESS_KEY',
            region: "YOUR_REGION_2",
            whitelist: ['YOUR_FULL_METRIC_NAME2']
        }]
    }
}