cloudq

Cloud Message Queue Server (using CouchDb) NewRelic enabled

cloudQ

A http message/job queue that is easy to publish, consume and complete messages...

git clone https://github.com/twilson63/node-cloudq.git
npm install .
 
# configure env vars 
export COUCH=http://localhost:5984
export DB=cloudq
export TOKEN=foo
export SECRET=bar
export PORT=8000
export TIMEOUT=100000
 
npm run-script setup
npm start

First, you need to install couchdb, you can download couchdb at http://couchdb.apache.org/

npm install cloudq -g
export COUCH=http://localhost:5984
export DB=cloudq
export TOKEN=foo
export SECRET=bar
export PORT=8000
export TIMEOUT=100000
 
# run server 
 
cloudq

A job message queue server, allows your applications to push jobs to a queue, then worker applications can watch the queue and request for a job, when the worker receives the job, it does the work, then sends a complete message back to the server. Each job as a pre-defined schema that consists of two attributes:

  • klass
  • args

The klass attribute is a string represents the name of object that you wish to invoke. The args attribute is an array of parameters that you wish to provide to that objects perform method.

{ "job":
  {
    "klass": "Mailer",
    "args": [{"to": "foo@email.com", "subject": "hello"}]
  },
  "priority": 100
}

publishes the job to the queue named send_mail

curl -XPOST -d '{ "job": { "klass": "Mailer", "args": [{"to": "foo@email.com", "subject": "hello"}]}}' \
-H "Content-Type: application/json" \
http://localhost:8000/send_mail
 
#> { "ok": true, "id": "c3b9e16d1efc436b7e30543bcf00182a", "rev": "1-977964400b51b5a1673fa7ac76d33874" } } 

consumes the next highest job in the queue

curl http://localhost:8000/send_mail
#>{ "klass": "Mailer", "args": [{"to": "foo@email.com", "subject": "hello"}], "id": "1", "ok": true} 
curl -XDELETE http://cloudq.example.com/send_mail/1
#>{ "status": "success"} 

Authorization

Currently authorization is done by environment varables:

TOKEN and SECRET

Theses env variables should match with basic authentication, per request:

curl http://token:secret@localhost:3000/foo

Test Successful Authentication:

curl -XPOST -d '{ "job": { "klass": "Mailer", "args": [{"to": "foo@email.com", "subject": "hello"}]}}' http://token:secret@cloudq.example.com/send_mail

Logging

CloudQ uses bunyan as the logger and returns a stream of json, but if you want to put it into a more common format, then you can use the bunyan command to pipe the json into a readable format.

npm install bunyan -g
cloudq | bunyan
 

Produces:

2013-11-05T22:01:23.911Z]  INFO: cloudq/4187 on thing-4.local:
    0: {
      "ok": true,
      "id": "_design/dequeue",
      "rev": "17-d66392bf5441a2cae9bf4c52700cfeff"
    }
    --

for a shorter format

cloudq | bunyan -o short`

NewRelic

CloudQ is NewRelic Ready, simply supply an ENV Var for your New Relic key and you should be good to go.

# enable New Relic
export NEW_RELIC_LICENSE_KEY=xkkk
export NEW_RELIC_APP_NAME=cloudq 
 
cloudq | bunyan

Deploy

mkdir mycloudq
cd mycloudq
npm init
# edit package.json and set "node": "~0.6.x" 
npm install cloudq --save
echo 'require("cloudq/server");' >> server.js
jitsu databases create couch cloudq
jitsu env set COUCH http://xxxx263878962530.iriscouch.com:5984
jitsu env set DB cloudq
jitsu env set TOKEN foo
jitsu env set SECRET bar
 
jitsu deploy
# create an iriscouch account 
mkdir mycloudq
cd mycloudq
npm init
# edit package.json and set "node": "~0.6.x" 
npm install cloudq --save
echo 'web: ./node_modules/cloudq/bin/cloudq' >> Procfile
echo 'node_modules' >> .gitignore
git init
git add .
git commit -am "first commit"
heroku create
heroku config:add COUCH=http://mydb.iriscouch.com
heroku config:add DB=cloudq
heroku config:add TOKEN=foo
heroku config:add SECRET=bar
 
git push heroku master
npm test

see LICENSE

  1. ONLY THREE CORE API METHODS
  • POST /queue - PUBLISH a JOB on the QUEUE
  • GET /queue - CONSUME a JOB
  • DELETE /queue/id - Mark JOB as Completed
  • tokens authorization
  • create acl for queues, views, bulk updates

pull requests welcome