Nightly Perpetrated Mischief

    cloki

    2.0.31 • Public • Published

    Codefresh build status CodeQL

    cLoki

    like Loki, but for ClickHouse

    cLoki is a flexible Loki 1 compatible LogQL API built on top of ClickHouse

    • Built in Explore UI and LogQL CLI for querying and extracting data
    • Native Grafana 2 and LogQL APIs for querying, processing, ingesting, tracing and alerting 3
    • Powerful pipeline to dynamically search, filter and extract data from logs, events, traces and beyond
    • Ingestion and PUSH APIs transparently compatible with LogQL, PromQL, InfluxDB, Elastic and more
    • Ready to use with Agents such as Promtail, Grafana-Agent, Vector, Logstash, Telegraf and many others
    • Cloud native, stateless and compact design

    :octocat: Get started using the cLoki Wiki 💡

    ezgif com-optimize 15

    Project Background

    The Loki API and its Grafana native integration are brilliant, simple and appealing - but we just love ClickHouse.

    cLoki implements a complete LogQL API buffered by a fast bulking LRU sitting on top of ClickHouse tables and relying on its columnar search and insert performance alongside solid distribution and clustering capabilities for stored data. Just like Loki, cLoki does not parse or index incoming logs, but rather groups log streams using the same label system as Prometheus. 3

    🔥 LogQL: Supported Features

    cLoki implements a broad range of LogQL Queries to provide transparent compatibility with the Loki API
    The Grafana Loki datasource can be used to natively query logs and display extracted timeseries

    🎉 No plugins needed

    🔥 Follow our examples to get started


    Log Streams

    cLoki supports input via Push API using JSON or Protobuf and it is compatible with Promtail and any other Loki compatible agent. On top of that, cLoki also accepts and converts log and metric inserts using Influx, Elastic, Tempo and other common API formats.

    Our preferred companion for parsing and shipping log streams to cLoki is paStash with extensive interpolation capabilities to create tags and trim any log fat. Sending JSON formatted logs is suggested when dealing with metrics.


    🔥 CliQL: Experimental 2.0 Features

    cLoki implements custom query functions for ClickHouse timeseries extraction, allowing direct access to any existing table

    ezgif com-gif-maker

    Timeseries

    Convert columns to tagged timeseries using the emulated loki 2.0 query format

    <aggr-op> by (<labels,>) (<function>(<metric>[range_in_seconds])) from <database>.<table> where <optional condition>
    
    Examples
    avg by (source_ip) (rate(mos[60])) from my_database.my_table
    
    sum by (ruri_user, from_user) (rate(duration[300])) from my_database.my_table where duration > 10
    

    ClickHouse

    Convert columns to tagged timeseries using the experimental clickhouse function

    Example

    clickhouse({ 
      db="my_database", 
      table="my_table", 
      tag="source_ip", 
      metric="avg(mos)", 
      where="mos > 0", 
      interval="60" 
    })
    
    Query Options
    parameter description
    db clickhouse database name
    table clickhouse table name
    tag column(s) for tags, comma separated
    metric function for metric values
    where where condition (optional)
    interval interval in seconds (optional)
    timefield time/date field name (optional)

    🔥 Tempo: Supported Features

    cLoki Pulse offers experimental support for the Grafana Tempo API providing span ingestion and querying

    At database level, Tempo Spans/Traces are stored as tagged Logs and are accessible from both LogQL and Tempo APIs


    Setup

    Check out the Wiki for detailed instructions or choose a quick method:

    🚏 GIT (Manual)

    Clone this repository, install with npmand run using nodejs 14.x (or higher)

    npm install
    CLICKHOUSE_SERVER="my.clickhouse.server" CLICKHOUSE_AUTH="default:password" CLICKHOUSE_DB="cloki" node cloki.js
    🚏 NPM

    Install cloki as global package on your system using npm

    sudo npm install -g cloki
    cd $(dirname $(readlink -f `which cloki`)) \
      && CLICKHOUSE_SERVER="my.clickhouse.server" CLICKHOUSE_AUTH="default:password" CLICKHOUSE_DB="cloki" cloki
    🚏 PM2
    sudo npm install -g cloki pm2
    cd $(dirname $(readlink -f `which cloki`)) \
      && CLICKHOUSE_SERVER="my.clickhouse.server" CLICKHOUSE_AUTH="default:password" CLICKHOUSE_DB="cloki" pm2 start cloki
    pm2 save
    pm2 startup
    🚏 Docker

    For a fully working demo, check the docker-compose example


    Logging

    The project uses pino for logging and by default outputs JSON'ified log lines. If you want to see "pretty" log lines you can start cloki with npm run pretty

    Configuration

    The following ENV Variables can be used to control cLoki parameters and backend settings.

    ENV Default Usage
    CLICKHOUSE_SERVER localhost Clickhouse Server address
    CLICKHOUSE_PORT 8123 Clickhouse Server port
    CLICKHOUSE_DB cloki Clickhouse Database Name
    CLICKHOUSE_AUTH default: Clickhouse Authentication (user:password)
    CLICKHOUSE_PROTO http Clickhouse Protocol (http, https)
    CLICKHOUSE_TIMEFIELD record_datetime Clickhouse DateTime column for native queries
    BULK_MAXAGE 2000 Max Age for Bulk Inserts
    BULK_MAXSIZE 5000 Max Size for Bulk Inserts
    BULK_MAXCACHE 50000 Max Labels in Memory Cache
    LABELS_DAYS 7 Max Days before Label rotation
    SAMPLES_DAYS 7 Max Days before Timeseries rotation
    HOST 0.0.0.0 cLOKi API IP
    PORT 3100 cLOKi API PORT
    CLOKI_LOGIN undefined Basic HTTP Username
    CLOKI_PASSWORD undefined Basic HTTP Password
    READONLY false Readonly Mode, no DB Init
    FASTIFY_BODYLIMIT 5242880 API Maximum payload size in bytes
    FASTIFY_REQUESTTIMEOUT 0 API Maximum Request Timeout in ms
    FASTIFY_MAXREQUESTS 0 API Maximum Requests per socket
    TEMPO_SPAN 24 Default span for Tempo queries in hours
    TEMPO_TAGTRACE false Optional tagging of TraceID (expensive)
    DEBUG false Debug Mode (for backwards compatibility)
    LOG_LEVEL info Log Level
    HASH short-hash Hash function using for fingerprints. Currently supported short-hash and xxhash64 (xxhash64 function)

    Contributors

    Disclaimer

    ©️ QXIP BV, released under the GNU Affero General Public License v3.0. See LICENSE for details.

    1. cLoki is not affiliated or endorsed by Grafana Labs or ClickHouse Inc. All rights belong to their respective owners.

    2. Grafana®, Loki™ and Tempo® are a Trademark of Raintank, Grafana Labs. ClickHouse® is a trademark of ClickHouse Inc. Prometheus is a trademark of The Linux Foundation.

    3. cLoki is a 100% clear-room api implementation and does not fork, use or derivate from Grafana Loki code or concepts. 2

    Install

    npm i cloki

    DownloadsWeekly Downloads

    2

    Version

    2.0.31

    License

    AGPL-3.0

    Unpacked Size

    15 MB

    Total Files

    159

    Last publish

    Collaborators

    • lmangani