kafka-streams
    TypeScript icon, indicating that this package has built-in type declarations

    5.0.0 • Public • Published

    node-kafka-streams

    Build Status npm version

    // suggested Node.js version: v12.16.1
    npm install --save kafka-streams
    
    const {KafkaStreams} = require("kafka-streams");
     
    const config = require("./config.json");
    const factory = new KafkaStreams(config);
     
    const kstream = factory.getKStream("input-topic");
    const ktable = factory.getKTable(/* .. */);
     
    kstream.merge(ktable).filter(/* .. */).map(/* .. */).reduce(/* .. */).to("output-topic");

    CHANGES: The latest version brings a lot of changes, please check here before updating.

    API Overview

    You might also like

    README Overview

    Prerequisites

    • Kafka broker should be version >= 0.11.x
    • Node.js should be version >= 8.x.x

    A note on native mode

    If you are using the native mode (config: { noptions: {} }). You will have to manually install node-rdkafka alongside kafka-streams. (This requires a Node.js version between 9 and 12 and will not work with Node.js >= 13, last tested with 12.16.1)

    On Mac OS High Sierra / Mojave: CPPFLAGS=-I/usr/local/opt/openssl/include LDFLAGS=-L/usr/local/opt/openssl/lib yarn add --frozen-lockfile node-rdkafka@2.7.4

    Otherwise: yarn add --frozen-lockfile node-rdkafka@2.7.4

    (Please also note: Doing this with npm does not work, it will remove your deps, npm i -g yarn)

    Aim of this Library

    • this is not a 1:1 port of the official JAVA kafka-streams
    • the goal of this project is to give at least the same options to a nodejs developer that kafka-streams provides for JVM developers
    • stream-state processing, table representation, joins, aggregate etc. I am aiming for the easiest api access possible checkout the word count example

    Description

    kafka-streams 🐙 equivalent for nodejs ✨🐢🚀✨ build on super fast 🔥 observables using most.js 🤘

    ships with sinek 🙏 for backpressure

    comes with js and native Kafka client, for more performance and SSL, SASL and Kerberos features

    the lib also comes with a few window operations that are more similar to Apache Flink, yet they still feel natural in this api :squirrel:

    overwriteable local-storage solution allows for any kind of datastore e.g. RocksDB, Redis, Postgres..

    async (Promises) and sync stream operators e.g. stream$.map() or stream$.asyncMap()

    super easy API :goberserk:

    the lib is based on sinek, which is based on kafka-node's ConsumerGroups

    Port Progress Overview

    • core structure
    • KStream base - stream as a changelog
    • KTable base - stream as a database
    • KStream & KTable cloning
    • complex stream join structure
    • advanced joins see
    • windows (for joins) see
    • flink like window operations
    • word-count example
    • more examples
    • local-storage for etl actions
    • local-storage factory (one per action)
    • KStorage example for any DB that supports atomic actions
    • backing-up local-storage via kafka
    • kafka client implementation
    • KTable replay to Kafka (produce)
    • stream for topic message production only
    • sinek implementation
    • backpressure mode for KafkaClient
    • auto-json payloads (read-map/write-map)
    • auto producer partition and keyed-message handling
    • documentation
    • API description
    • higher join & combine examples
    • embed native client librdkafka for more performance
    • SSL
    • SASL
    • Kerberos

    Operator Implementations

    • map
    • asyncMap
    • constant
    • scan
    • timestamp
    • tap
    • filter
    • skipRepeats
    • skipRepeatsWith
    • slice
    • take
    • skip
    • takeWhile
    • skipWhile
    • until
    • since
    • reduce
    • chainReduce
    • forEach (observe)
    • chainForEach
    • drain
    • _zip
    • _merge
    • _join
    • _combine
    • _sample
    • throttle
    • debounce
    • delay
    • multicast
    • A description of the operators can be found here
    • Missing an operator? Feel free to open an issue 👮

    Additional Operators

    • mapStringToArray
    • mapArrayToKV
    • mapStringToKV
    • mapParse
    • mapStringify
    • atThroughput
    • mapWrapKafkaPayload
    • mapToFormat
    • mapFromFormat
    • Want more? Feel free to open an issue 👮

    Stream Action Implementations

    • countByKey
    • sumByKey
    • min
    • max
    • Want more? Feel free to open an issue 👮

    Join Operations

    Operation description

    KStream Status

    • merge
    • outerJoin
    • innerJoin
    • leftJoin
    • branch

    KTable Status

    • merge
    • outerJoin
    • innerJoin
    • leftJoin

    KTable <-> KStream Status

    • merge
    • outerJoin
    • innerJoin
    • leftJoin

    Window Operations

    KStream

    • window
    • advanced window
    • rolling window

    More

    Can I use this library yet?

    Yes.

    Are we ready for production yet?

    Probably, yes. 😄

    Even More

    Forks or Stars give motivation :bowtie:

    Install

    npm i kafka-streams

    DownloadsWeekly Downloads

    184

    Version

    5.0.0

    License

    MIT

    Unpacked Size

    118 kB

    Total Files

    25

    Last publish

    Collaborators

    • krystianity