AsyncKafka
This package is a promise based async client for Kafka that wraps the packages node-rdkafka and avro-schema-registry to create a high level, high preforming NPM Package for consuming and producing topics on Kafka.
The package aim to provide a solid user friendly Kafka client that can be used for common usage senarios of Kafka including shared state storage in compacted topics. The client try to encapsulate best practice and experience working with both Kafka and node-rdkafka and to protect users against common pittfalls and performance impacts by trying to balance these.
Documentation
Documentation is FINALLY found here!
Installation
npm install async-kafka
The package depends on the node.js wraper node-rdkafka for the very efficent C++ librdkafka library for Kafka. This package uses gyp to install. For spesific install condsiderations for node-rdkafka please visit node-rdkafka.
Other dependencies: async, avro-schema-registry, avsc, clone, uuid
Status
In early phase but elements of consumer and producer is derived from a previous long duration tested consumers and producers.
Bugs may exist -> Please report bugs in repository
Features
- All interaction with client is done with asyncfunctions (returns promise).
- Fast, low latency, low memory and cpu friendly
- Smart handling of Kafka, librdkafka and node-rdkafka quirks and oddities
- Internal queue and backpreassure handling
- Default values for all settings
- Configurable options for performance tuning
- Configurable stats and logging
- Decoding and encoding of avro, json and string messages
- Automatic lookup of avro schema in schema registry
- Gracefull shutdown of clients on external signals (SIGINT, SIGTERM) and on unhandled promise rejections and uncaught exeptions.
Consumer features
- Configurable auto commit interval
- Auto detection and decoding of AVRO and JSON messages
- Automatic lookup and retrival of AVRO schemas in schema register
- Automatic handling of reassignments and rebalancing
- Adding and removing topic subscriptions on the fly without loosing offsets on existing subscriptions
Per topic consumer features
- Set message decoder type or provide custom async decoder
- Consume from earliest, latest, next or last commited message, regardless of previous commits
- Configurable auto commit that will commit at consumer configured interval
- Ensure commit of last processed message before unassign of partition if auto commit is enabled.
- On graceful shutdown try to commit last prosessed offsets for topic partitions with previous commits in subscribe session
- Configurable resume from last offset on reassign regadless of commits
- Three stage processig with user defined async filter, map and process functions
- Detection of eof to notify that topics are read up to current offset
- Immediate detection of end of file on empty topics
Producer features
- Encoding of JSON, STRING and BUFFER (binary) messages (TO-DO: AVRO)
- Automatic lookup and retrival of AVRO schemas in schema register based on configuration
- Automatic queue size based on desired maximum messages in flight without delivery confirmation
- Backpressure handling trough promises
- Optional async callback on delivery acknowledgement with tracking token.
Admin features (TO-DO)
- Create, change and delete topics on broker
- Create, change and delete schemas in schema registry
Versioning
AsyncKafka uses a [major].[minor].[patch] versioning scheme with the following change definitions.
- Major: Release with majore rework or changes to functionality that can break backward compability.
- Minor: Release with new functionality but backward compatible with earlier versions of same major version.
- Patch: Release with bugfixes or optimization not changing the functionality of minor version.
TO-DO
- 0% Test coverage can't be good
- High level admin client
- More examples and usage documentation