sublog-http
A microservice to subscribe to a Redis pubsub channel, and serve messages via HTTP.
Example problem description
This service is intended for a personal requirement to subscribe to logging messages published via Redis. These are arrays published via pubsub.
redis-cli publish 'logger:mylogger' '["info", {"name": "evanx"}]'
where we might subscribe in the terminal as follows:
redis-cli psubscribe 'logger:*'
where we see the messages in the console as follows:
Reading messages... (press Ctrl-C to quit)
1) "psubscribe"
2) "logger:*"
3) (integer) 1
1) "pmessage"
2) "logger:*"
3) "logger:mylogger"
4) "[\"info\", {\"name\": \"evanx\"}]"
However we want to pipe to a command-line JSON formatter to enjoy a more readable rendering:
We found that redis-cli psubscribe
didn't suit that use case, e.g. piping to jq
or python -mjson.tool
to format the JSON.
Incidently see https://github.com/evanx/sub-push where we transfer messages to a list, brpop
and then pipe to jq
as an initial work-around.
Also see https://github.com/evanx/sub-write to subscribe and write to stdout
with optional JSON formatting.
However it seemed like a good idea to use a browser to render the logging messages, even for local viewing,
which prompted the development of this sublog-http
service.
Implementation
The essence of the implementation is as follows:
{ sub; sub; return ;}
where we keep a list of the last 10 messages in reverse order by splicing incoming messages into the head of the array.
We publish these messages
via HTTP using Koa:
{ api; app; app; stateserver = app;}
where we format the JSON for mobile browsers i.e. without JSON formatting extensions.
evans@eowyn:~$ curl -s -I localhost:8080
HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8
Note that config
is populated from environment variables as follows:
const config = 'subscribeChannel' 'httpPort' 'redisHost';
where we default redisHost
to localhost
Note that we check that an environment variable is not empty, for safety sake.
For example the following command line runs this service to subscribe to channel logger:mylogger
and serve the JSON messages via port 8888
subscribeChannel=logger:mylogger httpPort=8888 npm start
Incidently, some sample Node code for a client logger that publishes via Redis:
const createRedisLogger = 'debug' 'info' 'warn' 'error';
where the logger level
is spliced as the head of the arguments
array.
Note that logged errors are specially handled i.e. a slice of the stack
is logged.
Later we'll publish a more sophisticated client logger with rate limiting:
const minute = ; if metricminute !== minute if metricignored > 0 client; metricminute = minute; metriccount = 0; metricignored = 0; else metriccount++; if optionsminuteLimit && metriccount > optionsminuteLimit metricignored++; return;
Docker notes
This tested on Docker 1.12 (Ubuntu 16.04) and 1.11 (Amazon Linux 2016.09)
docker -v
Docker version 1.12.1, build 23cf638
Docker version 1.11.2, build b9f10c9/1.11.2
cat /etc/issue
Ubuntu 16.04.1 LTS
Amazon Linux AMI release 2016.09
Build application container
Let's build our application container:
docker build -t sublog-http:test https://github.com/evanx/sublog-http.git
where the image is named and tagged as sublog-http:test
Alternatively git clone
and npm install
and build from local dir e.g. if you wish to modify the Dockerfile
git clone https://github.com/evanx/sublog-http.git && cd sublog-http && npm install && docker build -t sublog-http:test .
where the default Dockerfile
is as follows:
FROM mhart/alpine-node
ADD package.json .
RUN npm install
ADD src .
ENV httpPort 8080
EXPOSE 8080
CMD ["node", "--harmony-async-await", "src/index.js"]
where we ADD package.json
and RUN npm install
first before ADD src
- so that if the source has changed but not package.json
then the cached intermediate image after npm install
is stil usable for a fast rebuild.
Run on host network
Using the latest Docker version or 1.12, we run on the host's network i.e. using the host's Redis instance:
docker run --network=host -e NODE_ENV=test \ -e subscribeChannel=logger:mylogger -e httpPort=8088 -d sublog-http:test
where we configure its port to 8088
to test, noting:
- although by default the port is
8080
and that is exposed via theDockerfile
- as the network is a
host
bridge, so the reconfiguredhttpPort
is accessible on the host
This container can be checked as follows:
docker ps
to see if actually started, otherwise omit-d
to debug.netstat -ntl
to see that a process is listening on port8088
http://localhost:8088
viacurl
or browser
Ensure that Redis is running on the host i.e. localhost
port 6379
Test message
We can publish a test logging message as follows:
redis-cli publish logger:mylogger '["info", "test message"]'
HTTP fetch:
curl -s http://localhost:8088 | python -mjson.tool
Sample output:
Bridge network
Alternatively for Docker 1.11 without --network=host
but configuring a redisHost
IP number:
docker run -e NODE_ENV=test -e subscribeChannel=logger:mylogger \ -e redisHost=$redisHost -d sublog-http:test
where redisHost
is the IP number of the Redis instance to which the container should connect.
Note that it cannot be localhost
as the context is the container which is running the HTTP service only.
Nor can it be omitted as localhost
is the default Redis host used by this service.
We publish a test message as follows:
redis-cli -h $redisHost publish logger:mylogger '["info", "test message"]'
where naturally we must specify the same redisHost
to which the service connects
i.e. not the default localhost
unless its external IP number was provided to the service,
and even then rather use that to test.
Get container ID, IP address, and curl:
sublogContainer=`docker ps -q -f ancestor=sublog-http:test | head -1`sublogHost=`docker inspect --format '{{ .NetworkSettings.Networks.bridge.IPAddress }}' $sublogContainer`echo $sublogHostcurl -s http://$sublogHost:8080 | python -mjson.tool
Note that in this case the port will be the 8080
default configured and exposed in the Dockerfile
Incidently we can kill all containers by our image name as follows:
ids=`docker ps -q -f ancestor=sublog-http:test`[ -n "$ids" ] && docker kill $ids
Altogether:
Isolated Redis container and network
In this example we create an isolated network:
docker network create --driver bridge redis
We can create a Redis container named redis-logger
as follows
docker run --network=redis --name redis-logger -d redis
We query its IP number and store in shell environment variable loggerHost
loggerHost=`docker inspect --format '{{ .NetworkSettings.Networks.redis.IPAddress }}' redis-logger`
which we can debug via
echo $loggerHost
to see that set e.g. to 172.18.0.2
Finally we run our service container:
docker run --network=redis --name sublog-http-mylogger \ -e NODE_ENV=test -e redisHost=$loggerHost -e subscribeChannel=logger:mylogger -d sublog-http:test
where we configure redisHost
for the redis-logger
container via environment variable.
Note that we:
- use the
redis
isolated network bridge for theredis-logger
container - configure
subscribeChannel
tologger:mylogger
via environment variable - name this container
sublog-http-mylogger
- use the previously built image
sublog-http:test
Get its IP address:
myloggerHttpServer=`
docker inspect --format '{{ .NetworkSettings.Networks.redis.IPAddress }}' sublog-http-mylogger
`
Print its URL:
echo "http://$myloggerHttpServer:8080"
Curl test:
curl -s $myloggerHttpServer:8080 | python -mjson.tool
Related projects
See
- https://github.com/evanx/sub-push - subscribe to Redis pubsub channel and transfer messages to a Redis list
- https://github.com/evanx/sub-write - subscribe to Redis pubsub channel and write to
stdout
with optional JSON formatting
We plan to publish microservices that similarly subscribe, but with purpose-built rendering for logging messages e.g. error messages coloured red.
Watch