kafka-stream-map

Overview

Reads from a Kafka input stream and writes to an ouput stream. Assumes the use of the Confluent Kafka REST Proxy. Every input elements can produce zero or more output elements. Thus the transformation is analogous to a flatMap function. The transformation may have side effects.

The library contains an lib/env.sh helper that forces the user of the library to set the following environment variables which are automatically picked up:

LOG_NAME=
KAFKA_REST_ENDPOINT=
STREAM_INPUT_TOPIC=
STREAM_INPUT_CONSUMER_GROUP=
STREAM_INPUT_POLL_INTERVAL=
STREAM_OUTPUT_TOPIC=
STREAM_OUTPUT_SCHEMA_ID=

Usage

var StreamMapper = require('kafka-stream-map');
var mapper = StreamMapper.loadFromEnv();

mapper.start(function(instance){
  instance.on('read', function(record){
    instance.emit('write', {count: 1});
  });
});