ccloud-node-console-client

Example console consumer and console producer using Node.JS and Confluent Cloud

Features

• Up to date support for Confluent Cloud - Apache Kafka as a Service (see https://www.confluent.io/confluent-cloud/)
• Up to date feature set from use of node-rdkafka node.js client (see https://github.com/Blizzard/node-rdkafka)
• Tested on Ubuntu, CentOS, macOS, and Raspbian
• Supports dynamic topic, key, value, and partition inputs
• Supports 0.10 and 0.11 timestamps

Dependencies

Linux dependencies

• openssl
• libssl-dev
• libsasl2-dev
• libsasl2-modules
• C++ toolchain

macOS dependencies

• Apple Xcode command line tools (for the compiler)
• openssl installed via Brew (needed for root certs file in /usr/local/etc/openssl/cert.pem)
• Export CPPFLAGS=-I/usr/local/opt/openssl/include and LDFLAGS=-L/usr/local/opt/openssl/lib

Install

Install ccloud-node-console-client from npm on macOS

brew install openssl
export CPPFLAGS=-I/usr/local/opt/openssl/include
export LDFLAGS=-L/usr/local/opt/openssl/lib
sudo -E npm install -g ccloud-node-console-client

Install ccloud-node-console-client from npm on Ubuntu

sudo apt install nodejs-legacy
sudo apt install openssl libssl-dev libsasl2-dev libsasl2-modules
sudo npm install -g ccloud-node-console-client

Install ccloud-node-console-client from npm on CentOS

sudo yum install epel-release nodejs
sudo yum install openssl openssl-devel cyrus-sasl-devel
sudo npm install -g ccloud-node-console-client

Uninstall

sudo npm uninstall -g ccloud-node-console-client

Useage

ccloud-console-producer --help
Usage: ccloud-console-producer -e <endpoint> -k <apikey> -s <apisecret> -t <topic> -S <ssl ca location> -v

Options:
  -e, --endpoint   Confluent Cloud Endpoints (Broker List)  [required]
  -k, --apikey     Confluent Cloud API Key                  [required]
  -s, --apisecret  Confluent Cloud API Secret               [required]
  -t, --topic      Kafka Topic to consume from              [required]
  -v, --verbose    Verbose mode                             [boolean]
  -S, --sslcaloc   SSL CA Location                          [default: "/usr/local/etc/openssl/cert.pem"]
  -?, --help       Print usage information

Missing required arguments: e, k, s, t


ccloud-console-consumer --help
Usage: ccloud-console-consumer -e <endpoint> -k <apikey> -s <apisecret> -t <topic> -c <consumer group id> -S <ssl ca location> -v -b

Options:
  -e, --endpoint   Confluent Cloud Endpoints (Broker List)       [required]
  -b, --beginning  Consume messages from beginning               [boolean]
  -k, --apikey     Confluent Cloud API Key                       [required]
  -s, --apisecret  Confluent Cloud API Secret                    [required]
  -t, --topic      Kafka Topic to consume from                   [required]
  -c, --cgid       Consumer group.id (defaults to autogenerated
  -v, --verbose    Verbose mode                                  [boolean]
  -S, --sslcaloc   SSL CA Location                               [default: "/usr/local/etc/openssl/cert.pem"]
  -?, --help       Print usage information

Missing required arguments: e, k, s, t

Example Consumer

$ ccloud-console-consumer -e $CCLOUD_BROKERS -k $CCLOUD_SASL_USERNAME -s $CCLOUD_SASL_PASSWORD -t mytopic -v
Using consumer group.id = node-console-consumer594945
Setting auto.offset.reset = largest
Created consumer subscription on topic = mytopic
Got a message on topic "mytopic"
{ value: <Buffer 22 62 61 72 22>,
  size: 5,
  key: 'foo',
  topic: 'mytopic',
  offset: 0,
  partition: 0 }
calling commit
<Control-C>
Caught interrupt signal
consumer disconnected. {"connectionOpened":1498683661438}

Example Producer

$ ccloud-console-producer -e $CCLOUD_BROKERS -k $CCLOUD_SASL_USERNAME -s $CCLOUD_SASL_PASSWORD -t mytopic
Confluent Cloud connection is ready
prompt: key:  foo
prompt: value:  bar
Command-line input received:
  Message key: foo
  Message value: bar
Message published

Troubleshooting

If you see the following error when you run either console producer or consumer, it means you have not installed librdkafka correctly with the required SSL and SASL libraries. See install instructions for installing openssl and setting compiler flags.

Caught error: Error: Invalid value for configuration property "security.protocol"

If the ccloud-console-producer or ccloud-console-consumer immediately exits to the shell prompt you are likely missing the Root CA Certificates. See the location of these certs (below) and add the correct -S flag pointing the the location of the certs on your operating system.

The default SSL Certificate location is /usr/local/etc/openssl/cert.pem which works on macOS but every flavor of Linux puts root certificates in different places.

Ubuntu/Debian/Raspbian: /etc/ssl/certs
CentOS/RedHat: /etc/pki/tls/cert.pem
macOS: /usr/local/etc/openssl/cert.pem (from `brew install openssl`)

For Ubuntu add the -S /etc/ssl/certs flag to specify your certificate location:

ccloud-console-producer -e $CCLOUD_BROKERS -k $CCLOUD_SASL_USERNAME -s $CCLOUD_SASL_PASSWORD -t mytopic -v -S /etc/ssl/certs

For CentOS/RedHat add the -S /etc/pki/tls/cert.pem flag to specify your certificate location:

ccloud-console-producer -e $CCLOUD_BROKERS -k $CCLOUD_SASL_USERNAME -s $CCLOUD_SASL_PASSWORD -t mytopic -v -S /etc/pki/tls/cert.pem