node-kafka-rest

Node Kafka REST API

DISCLAIMER: The goal of this project is to mimic confluent/kafka-rest-proxy to offer a faster interface to topic data for applications like kafka-topics-ui, as kafka-rest-proxy troubles with larger message payloads or a large amount of messages on the topic. If you are looking for a complete rest-proxy for kafka, this is not the tool you are looking for.

Quick start

• Run kafka-setup: yarn run kafka:start
• Open kafka topics ui: http://localhost:8000

Curling the API

Get topics

curl -X GET \
  http://localhost:8082/topics \
  -H 'cache-control: no-cache' \
  -H 'content-type: application/json' \
  -d '{
    "name": "peter",
    "format": "json",
    "auto.offset.reset": "earliest"
  }'

Get topic infos

curl -X GET \
  http://localhost:8082/topics/peters-topic \
  -H 'cache-control: no-cache' \
  -H 'content-type: application/json' \
  -d '{
    "name": "peter",
    "format": "json",
    "auto.offset.reset": "earliest"
  }'

Create a consumer

curl -X POST \
  http://localhost:8082/consumers/peter \
  -H 'cache-control: no-cache' \
  -H 'content-type: application/json' \
  -d '{
    "name": "peter",
    "format": "json",
    "auto.offset.reset": "earliest"
  }'

Use consumer to subscribe to topic

curl -X POST \
  http://localhost:8082/consumers/peter/instances/peter/subscription \
  -H 'cache-control: no-cache' \
  -H 'content-type: application/json' \
  -d '{
    "topics": ["peters-topic", "ks-fourth-4b63a506-4e46-4c78-a0b2-24922a1124db"]
  }'

Fetch data

curl -X GET \
  http://localhost:8082/consumers/peter/instances/peter/records \
  -H 'cache-control: no-cache' \
  -H 'content-type: application/json'

Close and remove consumer

curl -X DELETE \
  http://localhost:8082/consumers/peter/instances/peter \
  -H 'cache-control: no-cache' \
  -H 'content-type: application/json'

Using Docker

If you want to customize any Kafka REST parameters, simply add them as environment variables, e.g.:

• To set the http.port parameter set the environment to KAFKA_REST_HTTP_PORT: 8082.
• To turn on debug mode for consumer and producer set KAFKA_REST_DEBUG: "all".
• To set the group.id only for consumer set the environment to KAFKA_REST_CONSUMER_GROUP_ID: "kafka-rest-consumer-group".

The full list and descriptions of config parameters can be found here.

Check out our docker compose example:

---
version: "2"
services:
  zookeeper:
    image: wurstmeister/zookeeper:latest
    ports:
      - 2181:2181
 
  kafka:
    image: wurstmeister/kafka:0.10.2.1
    ports:
      - "9092:9092"
    links:
      - zookeeper
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_PORT: 9092
      KAFKA_ADVERTISED_HOST_NAME: "kafka"
      KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://kafka:9092"
      KAFKA_LISTENERS: "PLAINTEXT://:9092"
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_CREATE_TOPICS: "test:1:1"
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
 
  kafka-rest:
    image: nodefluent/kafka-rest
    ports:
      - 8082:8082
    links:
      - kafka
      - zookeeper
    depends_on:
      - kafka
      - zookeeper
    environment:
      KAFKA_REST_DEBUG: "all"
      KAFKA_REST_HTTP_PORT: 8082
      KAFKA_REST_CONSUMER_METADATA_BROKER_LIST: "kafka:9092"
      KAFKA_REST_PRODUCER_METADATA_BROKER_LIST: "kafka:9092"
 
  kafka-topics-ui:
    image: landoop/kafka-topics-ui
    ports:
      - 8000:8000
    links:
      - kafka-rest
    depends_on:
      - kafka-rest
    environment:
      KAFKA_REST_PROXY_URL: "http://kafka-rest:8082"
      PROXY: "true"