Narnia's Psychedelic Mushrooms

    node-kafka-rest

    0.15.0 • Public • Published

    Node Kafka REST API

    DISCLAIMER: The goal of this project is to mimic confluent/kafka-rest-proxy to offer a faster interface to topic data for applications like kafka-topics-ui, as kafka-rest-proxy troubles with larger message payloads or a large amount of messages on the topic. If you are looking for a complete rest-proxy for kafka, this is not the tool you are looking for.

    Quick start

    • Run kafka-setup: yarn run kafka:start
    • Open kafka topics ui: http://localhost:8000

    Curling the API

    Get topics

    curl -X GET \
      http://localhost:8082/topics \
      -H 'cache-control: no-cache' \
      -H 'content-type: application/json' \
      -d '{
        "name": "peter",
        "format": "json",
        "auto.offset.reset": "earliest"
      }'

    Get topic infos

    curl -X GET \
      http://localhost:8082/topics/peters-topic \
      -H 'cache-control: no-cache' \
      -H 'content-type: application/json' \
      -d '{
        "name": "peter",
        "format": "json",
        "auto.offset.reset": "earliest"
      }'

    Create a consumer

    curl -X POST \
      http://localhost:8082/consumers/peter \
      -H 'cache-control: no-cache' \
      -H 'content-type: application/json' \
      -d '{
        "name": "peter",
        "format": "json",
        "auto.offset.reset": "earliest"
      }'

    Use consumer to subscribe to topic

    curl -X POST \
      http://localhost:8082/consumers/peter/instances/peter/subscription \
      -H 'cache-control: no-cache' \
      -H 'content-type: application/json' \
      -d '{
        "topics": ["peters-topic", "ks-fourth-4b63a506-4e46-4c78-a0b2-24922a1124db"]
      }'

    Fetch data

    curl -X GET \
      http://localhost:8082/consumers/peter/instances/peter/records \
      -H 'cache-control: no-cache' \
      -H 'content-type: application/json'

    Close and remove consumer

    curl -X DELETE \
      http://localhost:8082/consumers/peter/instances/peter \
      -H 'cache-control: no-cache' \
      -H 'content-type: application/json'

    Using Docker

    If you want to customize any Kafka REST parameters, simply add them as environment variables, e.g.:

    • To set the http.port parameter set the environment to KAFKA_REST_HTTP_PORT: 8082.
    • To turn on debug mode for consumer and producer set KAFKA_REST_DEBUG: "all".
    • To set the group.id only for consumer set the environment to KAFKA_REST_CONSUMER_GROUP_ID: "kafka-rest-consumer-group".

    The full list and descriptions of config parameters can be found here.

    Check out our docker compose example:

    ---
    version: "2"
    services:
      zookeeper:
        image: wurstmeister/zookeeper:latest
        ports:
          - 2181:2181
     
      kafka:
        image: wurstmeister/kafka:0.10.2.1
        ports:
          - "9092:9092"
        links:
          - zookeeper
        depends_on:
          - zookeeper
        environment:
          KAFKA_BROKER_ID: 1
          KAFKA_PORT: 9092
          KAFKA_ADVERTISED_HOST_NAME: "kafka"
          KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://kafka:9092"
          KAFKA_LISTENERS: "PLAINTEXT://:9092"
          KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
          KAFKA_CREATE_TOPICS: "test:1:1"
          KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
     
      kafka-rest:
        image: nodefluent/kafka-rest
        ports:
          - 8082:8082
        links:
          - kafka
          - zookeeper
        depends_on:
          - kafka
          - zookeeper
        environment:
          KAFKA_REST_DEBUG: "all"
          KAFKA_REST_HTTP_PORT: 8082
          KAFKA_REST_CONSUMER_METADATA_BROKER_LIST: "kafka:9092"
          KAFKA_REST_PRODUCER_METADATA_BROKER_LIST: "kafka:9092"
     
      kafka-topics-ui:
        image: landoop/kafka-topics-ui
        ports:
          - 8000:8000
        links:
          - kafka-rest
        depends_on:
          - kafka-rest
        environment:
          KAFKA_REST_PROXY_URL: "http://kafka-rest:8082"
          PROXY: "true"

    Keywords

    none

    Install

    npm i node-kafka-rest

    DownloadsWeekly Downloads

    1

    Version

    0.15.0

    License

    MIT

    Unpacked Size

    140 kB

    Total Files

    31

    Last publish

    Collaborators

    • krystianity