Neoclassical Programming Multitude

    @5to3/influx-firehose-forwarder

    1.0.4 • Public • Published

    influx-firehose-forwarder

    Travis (.org) npm

    A simple forwarder that acts as an InfluxDB subscriber, forwarding each write to an AWS Firehose stream. This forwarder is useful if you want to do just in time backups of writes to an InfluxDB. It acts as a subscriber that is called by InfluxDB upon every single write. Since InfluxDB retries failed sends, this code doesn't include any queing or retry logic.

    At 5to3 we're running this on a Raspberry Pi in our lab. It collects data from multiple sensors and stores it in InfluxDB. To avoid data loss in the worst-case-scenario, we're directly shipping off each measurement to a Firehose stream with S3 backend.

    Installation

    The easiest way is to install this package globally through NPM:

    $ npm -i -g influx-firehose-forwarder

    Running

    $ FH_DELIVERY_STREAM=test influx-firehose-forwarder

    How to keep it running

    If you're running the forwarder on an OS with systemd, you can easily add it as a unit to keep it running and automatically restarting in case of an error. Simply use the following example unit as a starting point for your own configuration.

    [Unit]
    Description=InfluxDB Firehose forwarder
    
    [Service]
    ExecStart=/usr/local/bin/node /opt/firehose/app.js
    Restart=always
    RestartSec=10
    StandardOutput=syslog
    StandardError=syslog
    SyslogIdentifier=influx-firehose-forwarder
    User=someuser
    Group=somegroup
    Environment=NODE_ENV=production
    Environment=FH_HOST=localhost
    Environment=FH_PORT=3000
    Environment=FH_DELIVERY_STREAM=test
    
    [Install]
    WantedBy=multi-user.target
    

    Adding the forwarder as a subscriber to InfluxDB

    Use the Influx shell to add the forwarder as a new subscriber:

    $ influx
    > CREATE SUBSCRIPTION "sub0" ON "your_db"."autogen" DESTINATIONS ALL 'http://127.0.0.1:3000'
    > SHOW SUBSCRIPTIONS
    name: your_db
    retention_policy name mode destinations
    ---------------- ---- ---- ------------
    autogen          sub0 ALL  [http://127.0.0.1:3000]

    Install

    npm i @5to3/influx-firehose-forwarder

    DownloadsWeekly Downloads

    2

    Version

    1.0.4

    License

    MIT

    Unpacked Size

    7.53 kB

    Total Files

    7

    Last publish

    Collaborators

    • mzange