kinesis_stream_fetcher

1.2.0 • Public • Published

Kinesis Stream Fetcher

Kinesis Stream Fetcher is a basic library to fetch messages from a Kinesis stream without using the multilang daemon. It works as a simple poller by launching child processes for each stream that needs to be monitored.

Version 1.0.0 supports multiple partitions and launches a process for each paritition. The partitions config is no longer relevant as the information is not directly fetched from Amazon API.

Install

npm install kinesis_stream_fetcher --save

Usage

Before starting development ensure that your AWS configuration parameters are present in your environment under the following variable.

export AWS_ACCESS_KEY_ID="ACCESS_KEY_ID"
export AWS_SECRET_ACCESS_KEY="SECRET_ACCESS_KEY"
export AWS_REGION="ap-southeast-1"
export KINESIS_PARTITION_KEY="sweetcoffee"

Update

Version 1.2.0 includes a backward compatible support for instanceIds. Each instanceId identifies a deployment uniquely, thus allowing you to run multiple instances of Kinesis Stream Fetcher.

This was a bug in the earlier versions where it assumed that the library would be used for a single instance

let streamConfig = {
    instanceId: 'instance-1',
    redisUrl: 'redis://localhost:6379',
    streams: [{
        name: 'development-transaction_events-1',
        partitions: 1
    },{
        name: 'test-transaction_events',
        partitions: 1
    }]
};
const KinesisStreamFetcher = require('kinesis_stream_fetcher');
let fetcher = new KinesisStreamFetcher(streamConfig);
fetcher.on('message', (data) => {
  console.log(data);
})

Description

The consumer expects a redis connection to store the last read state of a stream. The first attempt is always to begin from the start of the stream.

Versions

Current Tags

  • Version
    Downloads (Last 7 Days)
    • Tag
  • 1.2.0
    0
    • latest

Version History

Package Sidebar

Install

npm i kinesis_stream_fetcher

Weekly Downloads

0

Version

1.2.0

License

ISC

Last publish

Collaborators

  • supersid