sematext-agent-docker

1.29.35 • Public • Published

Sematext Docker Agent

Sematext Docker ETP partner for Logging Deploy to Docker Cloud build status

Sematext Docker Agent collects Metrics, Events and Logs from the Docker API for SPM Docker Monitoring & Logsene / Hosted ELK Log Management. Works with CoreOS, RancherOS, Docker Swarm, Kubernetes, Apache Mesos, Hashicorp Nomad, Amzon ECS, ... see installation.

How it works

Video: Log Management for Docker

Sematext container shipping metrics and logs

Gathered information:

  • Operating System Metrics of the Host machine (CPU / Mem / Swap/ ...)
  • Docker Container Metrics/Stats
    • CPU Usage
    • Memory Usage
    • Network Stats
    • Disk I/O Stats
  • Docker Events
    • Version Information on Startup:
      • server-info – created by spm-agent framework with node.js and OS version info on startup
      • docker-info – Docker Version, API Version, Kernel Version on startup
    • Docker Events:
      • Container Lifecycle Events like
        • create, exec_create, destroy, export
      • Container Runtime Events like
        • die, exec_start, kill, pause, restart, start, stop, unpause, ...
  • Docker Logs
    • default fields
      • hostname / IP address
      • container id
      • container name
      • image name
      • message
    • Log format detection and log parsers: - NGINX - Apache httpd, Kafka, Solr, HBase, Zookeeper, Cassandra - MySQL - MongoDB - Redis - Elasticsearch - Nsq.io - JSON, ...

Installation

  1. Get a free account at sematext.com/spm

  2. Create an SPM App of type "Docker" and copy the SPM Application Token

  3. Run the image

    docker pull sematext/sematext-agent-docker
    docker run -d --name sematext-agent-docker -e SPM_TOKEN=YOUR_SPM_TOKEN -e LOGSENE_TOKEN=YOUR_LOGSENE_TOKEN  -e HOSTNAME  -v /var/run/docker.sock:/var/run/docker.sock sematext/sematext-agent-docker
    # Alternative TCP: default TCP 2375 on 'localhost' (=> container gateway address), -v is not required
    docker run -d --name sematext-agent-docker -e SPM_TOKEN=YOUR_SPM_TOKEN -e LOGSENE_TOKEN=YOUR_LOGSENE_TOKEN  -e HOSTNAME  sematext/sematext-agent-docker
    

    Required Parameters:

    • -e SPM_TOKEN - SPM Application Token
    • -e HOSTNAME - name of the docker host e.g. '$HOSTNAME' for Amazon ECS see HOSTNAME_LOOKUP_URL
    • -v /var/run/docker.sock - path to the docker socket (optional, if dockerd provides TCP on 2375, see also DOCKER_PORT and DOCKER_HOST paramter)

    TCP and TLS connection (> image version 1.29.13)

    If the unix socket is not available Sematext Agent assumes the Container Gateway Address (autodetect) and port 2375 as default (no TLS) - this needs no configuration. In case the Docker Daemon TCP settings are different, you have to configure the TCP settings. The TCP settings can be modified with the following parameters:

  • -e DOCKER_HOST - e.g. tcp://ip-reachable-from-container:2375/ - if not set unix:///var/run/docker.sock or if this does not exists tcp://gateway:2375 will be used. In this case you don't need -v to mount /var/run/docker.sock
  • -e DOCKER_PORT - in case Docker TCP connection is used, the agent will use its gateway address (autodetect) with the given DOCKER_PORT
  • -e DOCKER_TLS_VERIFY - 0 or 1
  • -e DOCKER_CERT_PATH - path to your certificate files, mount the path to the countainer with "-v $DOCKER_CERT_PATH:$DOCKER_CERT_PATH"

Example using docker-machine with Docker Swarm:

docker-machine env --swarm swarm-master
export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://192.168.99.101:3376"
export DOCKER_CERT_PATH="/Users/stefan/.docker/machine/machines/swarm-master"
export DOCKER_MACHINE_NAME="swarm-master"
eval "$(docker-machine env swarm-master)"
docker run -d --name sematext-agent --restart=always -e SPM_TOKEN=MY_TOKEN -e HOSTNAME  -e DOCKER_TLS_VERIFY -e DOCKER_CERT_PATH -e DOCKER_HOST -v $DOCKER_CERT_PATH:$DOCKER_CERT_PATH sematext/sematext-agent-docker
**Optional Parameters:**
- --privileged  might be required for Security Enhanced Linux (the better way is to have the right policy ...)
- -e HOSTNAME_LOOKUP_URL - On Amazon ECS, a [metadata query](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html) must be used to get the instance hostname (e.g. "169.254.169.254/latest/meta-data/local-hostname")
- -e HTTPS_PROXY - URL for a proxy server (behind firewalls)
- -e LOGSENE_URL - URL for bulk inserts into Logsene. Required only for Logsene On-Premises only.
- -e SPM_URL - URL for bulk inserts into SPM. Required only for SPM On-Premises. 

    **Docker Logs Parameters:**
- -e LOGSENE_TOKEN - Logsene Application Token for logs 
- Whitelist containers for logging
  - -e MATCH_BY_NAME - regular expression to white list container names 
  - -e MATCH_BY_IMAGE - regular expression to white list image names 
- Blacklist containers
  - -e SKIP_BY_NAME - regular expression to black list container names 
  - -e SKIP_BY_IMAGE - regular expression to black list image names for logging 
  - -v /yourpatterns/patterns.yml:/etc/logagent/patterns.yml - to provide custom patterns for log parsing, see [logagent-js](https://github.com/sematext/logagent-js)
- -v /tmp:/logsene-log-buffer a directory to store logs, in case of a network or service outage. Docker Agent deletes this files after successful transmission.  
- -e KUBERNETES set this variable to "1" to parse container names into the fields kubernetes.pod_name, kubernetes.namespace and kubernetes.container_name


You’ll see your Docker metrics in SPM after about a minute.
  1. Watch metrics, use anomaly detection for alerts, create e-mail reports and much more ...

Docker Events:

Log Routing

Routing logs from different containers to separate Logsene Apps can be configured via docker labels. Simply tag a container with the label (or environment variable) LOGSENE_TOKEN. Sematext Docker Agent inspects the containers for this Label and ships the logs to the defined Logsene App.

Example: The following command will start nginx webserver and logs for this container will be shipped to the related Logsene App.

docker run --label LOGSENE_TOKEN=REPLACE_WITH_YOUR_LOGSENE_TOKEN -p 80:80 nginx
# or use environment variable on Kubernetes (no support for Docker labels)
# docker run -e LOGSENE_TOKEN=REPLACE_WITH_YOUR_LOGSENE_TOKEN -p 80:80 nginx

All other container logs will be shipped to the Logsene App specified in the docker run command for sematext/sematext-agent-docker with the environment variable LOGSENE_TOKEN.

Integrated Log Parser

SPM for Docker recognizes log formats - so your logs arrive in a structured format in Logsene! The format recognition, data extractions, date parsing etc. is provided by logagent-js and covers:

  • Format detection e.g. for
    • Mongo DB
      • Nginx
      • Apache httpd, Kafka, Cassandra, HBase, Solr, Zookeeper
      • MySQL
      • Redis
  • plain text log messages
  • line delimited JSON logs
  • GeoIP enrichment for webserver logs, or any other field defined in the pattern definitions

To use a custom pattern definition simply mount a volume to '/etc/logagent/patterns.yml':

-v /mydir/patterns.yml:/etc/logagent/patterns.yml

Feel free to contribute to logagent-js to enrich the default pattern set.

Installation on CoreOS Linux

Sematext Agent for Docker can monitor CoreOS clusters including metrics and logs from Docker and journald.
See: Setup Sematext Docker Agent on CoreOS

Installation on Kubernetes

Run Sematext Docker Agent as Kubernetes DaemonSet.

  1. Create sematext-agent.yml - and set your SPM and Logsene App Token in the section spec.env.
  2. Run the DaemonSet
kubectl create -f sematext-agent.yml --validate=false

Installation on Docker Swarm

Please read Docker Swarm: Collecting Metrics, Events & Logs

Installation on Nomad by Hashicorp

See and example of the job description for Nomad by Hashicorp

Installation on Mesos / Marathon

Please note that you have to specify the number of Mesos nodes (instances), SPM App Token and Logsene App Token. Example call to the Marathon API:

curl -XPOST -H "Content-type: application/json" http://your_marathon_server:8080/v2/apps  -d '
{
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "sematext/sematext-agent-docker"
    },
    "volumes": [
      {
        "containerPath": "/var/run/docker.sock",
        "hostPath": "/var/run/docker.sock",
        "mode": "RW"
      }
    ],
    "network": "BRIDGE"
  },
  "env": {
        "LOGSENE_TOKEN": "YOUR_LOGSENE_TOKEN",
        "SPM_TOKEN": "YOUR_SPM_TOKEN" 
  },
  "id": "sematext-agent",
  "instances": 1,
  "cpus": 0.1,
  "mem": 100,
  "constraints": [
    [
      "hostname",
      "UNIQUE"
    ]
  ]
}

Support

  1. Please check the SPM for Docker Wiki
  2. If you have questions about SPM for Docker, chat with us in the SPM user interface or drop an e-mail to support@sematext.com
  3. Open an issue here
  4. Contribution guide here

Package Sidebar

Install

npm i sematext-agent-docker

Weekly Downloads

1

Version

1.29.35

License

Apache-2.0

Last publish

Collaborators

  • megastef