cloudq

2.9.1 • Public • Published

cloudQ

Build Status

A http message/job queue that is easy to publish, consume and complete messages...

Install from source

git clone https://github.com/twilson63/node-cloudq.git
npm install .
 
# configure env vars 
export COUCH=http://localhost:5984
export DB=cloudq
export TOKEN=foo
export SECRET=bar
export PORT=8000
export TIMEOUT=100000
 
npm run-script setup
npm start

Install and Run Locally

First, you need to install couchdb, you can download couchdb at http://couchdb.apache.org/

npm install cloudq -g
export COUCH=http://localhost:5984
export DB=cloudq
export TOKEN=foo
export SECRET=bar
export PORT=8000
export TIMEOUT=100000
 
# run server 
 
cloudq

Usage

A job message queue server, allows your applications to push jobs to a queue, then worker applications can watch the queue and request for a job, when the worker receives the job, it does the work, then sends a complete message back to the server. Each job as a pre-defined schema that consists of two attributes:

  • klass
  • args

The klass attribute is a string represents the name of object that you wish to invoke. The args attribute is an array of parameters that you wish to provide to that objects perform method.

job schema

{ "job":
  {
    "klass": "Mailer",
    "args": [{"to": "foo@email.com", "subject": "hello"}]
  },
  "priority": 100
}

publish

publishes the job to the queue named send_mail

curl -XPOST -d '{ "job": { "klass": "Mailer", "args": [{"to": "foo@email.com", "subject": "hello"}]}}' \
-H "Content-Type: application/json" \
http://localhost:8000/send_mail
 
#> { "ok": true, "id": "c3b9e16d1efc436b7e30543bcf00182a", "rev": "1-977964400b51b5a1673fa7ac76d33874" } } 

consume

consumes the next highest job in the queue

curl http://localhost:8000/send_mail
#>{ "klass": "Mailer", "args": [{"to": "foo@email.com", "subject": "hello"}], "id": "1", "ok": true} 

complete

curl -XDELETE http://cloudq.example.com/send_mail/1
#>{ "status": "success"} 

Authorization

Currently authorization is done by environment varables:

TOKEN and SECRET

Theses env variables should match with basic authentication, per request:

curl http://token:secret@localhost:3000/foo

Test Successful Authentication:

curl -XPOST -d '{ "job": { "klass": "Mailer", "args": [{"to": "foo@email.com", "subject": "hello"}]}}' http://token:secret@cloudq.example.com/send_mail

Logging

CloudQ uses bunyan as the logger and returns a stream of json, but if you want to put it into a more common format, then you can use the bunyan command to pipe the json into a readable format.

npm install bunyan -g
cloudq | bunyan

Produces:

2013-11-05T22:01:23.911Z]  INFO: cloudq/4187 on thing-4.local:
    0: {
      "ok": true,
      "id": "_design/dequeue",
      "rev": "17-d66392bf5441a2cae9bf4c52700cfeff"
    }
    --

for a shorter format

cloudq | bunyan -o short`

NewRelic

CloudQ is NewRelic Ready, simply supply an ENV Var for your New Relic key and you should be good to go.

# enable New Relic
export NEW_RELIC_LICENSE_KEY=xkkk
export NEW_RELIC_APP_NAME=cloudq 

cloudq | bunyan

Deploy

Deploy to nodejitsu

mkdir mycloudq
cd mycloudq
npm init
# edit package.json and set "node": "~0.6.x" 
npm install cloudq --save
echo 'require("cloudq/server");' >> server.js
jitsu databases create couch cloudq
jitsu env set COUCH http://xxxx263878962530.iriscouch.com:5984
jitsu env set DB cloudq
jitsu env set TOKEN foo
jitsu env set SECRET bar
 
jitsu deploy

Deploy to heroku

# create an iriscouch account 
mkdir mycloudq
cd mycloudq
npm init
# edit package.json and set "node": "~0.6.x" 
npm install cloudq --save
echo 'web: ./node_modules/cloudq/bin/cloudq' >> Procfile
echo 'node_modules' >> .gitignore
git init
git add .
git commit -am "first commit"
heroku create
heroku config:add COUCH=http://mydb.iriscouch.com
heroku config:add DB=cloudq
heroku config:add TOKEN=foo
heroku config:add SECRET=bar
 
git push heroku master

Tests

npm test

License

see LICENSE

Contributing

GOALS

  1. ONLY THREE CORE API METHODS
  • POST /queue - PUBLISH a JOB on the QUEUE
  • GET /queue - CONSUME a JOB
  • DELETE /queue/id - Mark JOB as Completed

TODO

  • tokens authorization
  • create acl for queues, views, bulk updates

pull requests welcome

Package Sidebar

Install

npm i cloudq

Weekly Downloads

25

Version

2.9.1

License

none

Unpacked Size

120 kB

Total Files

24

Last publish

Collaborators

  • jackhq
  • twilson63