Navigator Prefabricating Marinates

    qryn

    2.1.12 • Public • Published

    Codefresh build status CodeQL

    qryn

    /ˈkwɪr..ɪŋ/

    LogQL for ClickHouse, and beyond

    qryn is a flexible LogQL API built on top of ClickHouse with native support for popular data ingestion formats

    • Built in Explore UI and LogQL CLI for querying and extracting data
    • Native Grafana 1 and LogQL APIs for querying, processing, ingesting, tracing and alerting 2
    • Powerful pipeline to dynamically search, filter and extract data from logs, events, traces and beyond
    • Ingestion and PUSH APIs transparently compatible with LogQL, PromQL, InfluxDB, Elastic and more
    • Ready to use with Agents such as Promtail, Grafana-Agent, Vector, Logstash, Telegraf and many others
    • Cloud native, stateless and compact design

    :octocat: Get started using our Wiki 💡

    ⚠️ Existing user and confused? The project has been renamed to qryn (/ˈkwɪr..ɪŋ/) 👍

    ezgif com-optimize 15

    Project Background

    qryn implements a complete LogQL API buffered by a fast bulking LRU sitting on top of ClickHouse tables and relying on its columnar search and insert performance alongside solid distribution and clustering capabilities for stored data. qryn does not parse or index incoming logs, but rather groups log streams using the same label system as Prometheus. 2

    🔥 LogQL: Supported Features

    qryn implements a broad range of LogQL Queries to provide transparent compatibility with the Loki API
    The Grafana Loki datasource can be used to natively query logs and display extracted timeseries

    🎉 No plugins needed

    🔥 Follow our examples to get started


    Log Streams

    qryn supports input via Push API using JSON or Protobuf and it is compatible with Promtail and any other LogQL compatible agent. On top of that, qryn also accepts and converts log and metric inserts using Influx, Elastic, Tempo and other common API formats.

    Our preferred companion for parsing and shipping log streams to qryn is paStash with extensive interpolation capabilities to create tags and trim any log fat. Sending JSON formatted logs is suggested when dealing with metrics.


    🔥 CliQL: Experimental 2.0 Features

    qryn implements custom query functions for ClickHouse timeseries extraction, allowing direct access to any existing table

    ezgif com-gif-maker

    Timeseries

    Convert columns to tagged timeseries using the emulated LogQL 2.0 query format

    <aggr-op> by (<labels,>) (<function>(<metric>[range_in_seconds])) from <database>.<table> where <optional condition>
    
    Examples
    avg by (source_ip) (rate(mos[60])) from my_database.my_table
    
    sum by (ruri_user, from_user) (rate(duration[300])) from my_database.my_table where duration > 10
    

    ClickHouse

    Convert columns to tagged timeseries using the experimental clickhouse function

    Example

    clickhouse({ 
      db="my_database", 
      table="my_table", 
      tag="source_ip", 
      metric="avg(mos)", 
      where="mos > 0", 
      interval="60" 
    })
    
    Query Options
    parameter description
    db clickhouse database name
    table clickhouse table name
    tag column(s) for tags, comma separated
    metric function for metric values
    where where condition (optional)
    interval interval in seconds (optional)
    timefield time/date field name (optional)

    Setup

    Check out the Wiki for detailed instructions or choose a quick method:

    🚏 GIT (Manual)

    Clone this repository, install with npmand run using nodejs 14.x (or higher)

    npm install
    CLICKHOUSE_SERVER="my.clickhouse.server" CLICKHOUSE_AUTH="default:password" CLICKHOUSE_DB="qryn" node qryn.js
    🚏 NPM

    Install qryn as global package on your system using npm

    sudo npm install -g qryn
    cd $(dirname $(readlink -f `which qryn`)) \
      && CLICKHOUSE_SERVER="my.clickhouse.server" CLICKHOUSE_AUTH="default:password" CLICKHOUSE_DB="qryn" qryn
    🚏 PM2
    sudo npm install -g qryn pm2
    cd $(dirname $(readlink -f `which qryn`)) \
      && CLICKHOUSE_SERVER="my.clickhouse.server" CLICKHOUSE_AUTH="default:password" CLICKHOUSE_DB="qryn" pm2 start qryn
    pm2 save
    pm2 startup
    🚏 Docker

    For a fully working demo, check the docker-compose example


    Logging

    The project uses pino for logging and by default outputs JSON'ified log lines. If you want to see "pretty" log lines you can start qryn with npm run pretty

    Configuration

    The following ENV Variables can be used to control qryn parameters and backend settings.

    ENV Default Usage
    CLICKHOUSE_SERVER localhost Clickhouse Server address
    CLICKHOUSE_PORT 8123 Clickhouse Server port
    CLICKHOUSE_DB qryn Clickhouse Database Name
    CLICKHOUSE_AUTH default: Clickhouse Authentication (user:password)
    CLICKHOUSE_PROTO http Clickhouse Protocol (http, https)
    CLICKHOUSE_TIMEFIELD record_datetime Clickhouse DateTime column for native queries
    BULK_MAXAGE 2000 Max Age for Bulk Inserts
    BULK_MAXSIZE 5000 Max Size for Bulk Inserts
    BULK_MAXCACHE 50000 Max Labels in Memory Cache
    LABELS_DAYS 7 Max Days before Label rotation
    SAMPLES_DAYS 7 Max Days before Timeseries rotation
    HOST 0.0.0.0 HTTP API IP
    PORT 3100 HTTP API PORT
    QRYN_LOGIN undefined Basic HTTP Username
    QRYN_PASSWORD undefined Basic HTTP Password
    READONLY false Readonly Mode, no DB Init
    FASTIFY_BODYLIMIT 5242880 API Maximum payload size in bytes
    FASTIFY_REQUESTTIMEOUT 0 API Maximum Request Timeout in ms
    FASTIFY_MAXREQUESTS 0 API Maximum Requests per socket
    FASTIFY_METRICS false API /metrics exporter
    TEMPO_SPAN 24 Default span for Tempo queries in hours
    TEMPO_TAGTRACE false Optional tagging of TraceID (expensive)
    DEBUG false Debug Mode (for backwards compatibility)
    LOG_LEVEL info Log Level
    HASH short-hash Hash function using for fingerprints. Currently supported short-hash and xxhash64 (xxhash64 function)

    Contributors

    Disclaimer

    ©️ QXIP BV, released under the GNU Affero General Public License v3.0. See LICENSE for details.

    1. Grafana®, Loki™ and Tempo® are a Trademark of Raintank, Grafana Labs. ClickHouse® is a trademark of ClickHouse Inc. Prometheus is a trademark of The Linux Foundation.

    2. qryn is a 100% clear-room api implementation and does not fork, use or derivate from Grafana Loki code or concepts. 2

    Install

    npm i qryn

    DownloadsWeekly Downloads

    118

    Version

    2.1.12

    License

    AGPL-3.0

    Unpacked Size

    15 MB

    Total Files

    161

    Last publish

    Collaborators

    • lmangani