node-kafka-client
A wrapper library of node-rdkafka with extended features
Install
Important: We are using
node-rdkafka 2.10.1
npm install @kafka/node-kafka-client
Usage
Consumer
const { Consumer } = require("@kafka/node-kafka-client");
const consumer = new Consumer({
// Required options
name: "test-consumer-name",
groupId: "test-group-id",
host: "localhost:9092",
topic: "test-topic",
// Optional options
connectTimeout: 5000,
mode: "non-flowing",
intervalFetchMessage: 10, // Only affect for non-flowing mode
numMsgFetchPerTime: 1, // Only affect for non-flowing mode
logger: null,
});
consumer.listen(async (data) => {
console.log("Listen data:", data);
});The consumer inherit all node-rdkafka configuration options for you to set or override. Using snippet config below and reference original node-rdkafka lib for more detail of configuration.
{
...
rdKafkaConfig: {
debug: "all",
},
rdKafkaTopicConfig: {
"auto.offset.reset": "latest",
},
}consumer.on("event.log", console.log);Flowing mode:
- Pros
- Concurrent handler processing message.
- Cons
- Might cause OOM (out-of-memory) when start the consumer with the lag offset too much
- Need to handler duplicated messages.
- Each consumer will block a thread of libuv.
- Suitable for:
- Small processing time handler.
- The result of processing messages is independent.
Non-flowing:
- Pros:
- Handle one-by-one messages. The next message just consumed after the previous committed
- It's not block the thread of libuv.
- Fully control the interval and number of message to fetch
- Cons:
- Non-concurrency consume messages, higher latency compare to flowing mode.
- Suitable for:
- High demand of consistency handling messages or the result of processing a message depends on the previous one
- Running multiple consumers in one application
- Long-time-taking handler
Producer
[will be updated]
Benchmark
[will be updated]