Hique

hique is a job queue for NodeJS.

feel free to contribute / open issues / create pull requests / fork

Introduction

hique is heavily inspired by kue and bee-queue, and after using both frameworks pretty extensively I found that, though very well written, these frameworks do not fulfill two of my most desired aspects in:

• Stability
• Scalability

hique was designed with these in mind.

Stability

hique differs from most frameworks by sacrificing a bit of performance to gain a much more stable environment, even when scaled up on different machines.

Scalability

To scale hique to available cpus / machines, simply create a NodeJS process with a hique worker pointing to the same monitor object as every other worker and voila! scaling done easy.

Installation

NPM

npm install hique

GitHub

npm install git+https://github.com/patriceperez/hique.git

Getting Started

Here is a simple example on how to set up a basic worker and a few jobs for testing

var hq = require('../lib/hique');
 
var monitor = new hq.Monitor();
var worker = new hq.Worker();
 
monitor.start();
 
worker.process('testJob', 5, function (job, done) {
    console.log('executed job %s with data %s', job.id, JSON.stringify(job.data));
    job.reportProgress(1, 1);
    done(null, job.data.test);
});
 
for (var i = 0; i < 13; i++) {
    worker.createJob('testJob', {test: i}, function (job) {
        console.log('save new job %s and data %s', job.id, JSON.stringify(job.data));
    });
}
 
worker.start();

check out the examples folder for more use cases

API Reference

Table of Contents

• Worker
  • Configuration
  • Processing Jobs
  • Creating Jobs
  • Pause / Resume
  • Get Existing Job
  • Get Completed Job Result
  • Get System Status
• Job
  • Report Progress
  • Add Child
  • Wait For Child Jobs
• Monitoring
  • Data Store
  • Native - In Memory
  • Creating Your Own Data Store

Worker

Configuration

Default configuration for workers

{
    job: {
        ttl: 5 * 60 * 1000
    },
    cleanUp: {
        active: true,
        refreshRate: 5 * 60 * 1000
    },
    refreshRate: 1000,
    monitor: {
        host: 'http://127.0.0.1',
        port: '3001'
    }
}

Any value can be overridden by providing a new value via the worker constructor:

new Worker({refreshRate: 2000})

Field	Description
job.ttl	maximum time allowed (in milliseconds) for a job to stay active
cleanUp.active	should the cleanup process remove outdated data from the data store
cleanUp.refreshRate	interval (in milliseconds) between cleanup iterations
refreshRate	interval (in milliseconds) between job updates fetching in the data store
monitor.host	the host of the coordinating data store
monitor.port	the port the data store is listening on

Processing Jobs

Process a new job type

worker.process(type, concurrency, function(job, done){
    // job logic
    done(error, result);
});

param	Description
type	string literal representing the job type
concurrency (optional)	integer representing the amount of concurrent jobs the worker can handle simultaneously

Creating Jobs

Create a new job

worker.createJob(type, data, function(job){
    //job object contains the job id as well as other meta data
});

param	Description
type	string literal representing the job type
data	JSON object providing data to the job execution function

Pause

Pause the worker from handling any new work

worker.pause();

Resume

Resume the worker to handle any new work

worker.start();

Get Existing Job

Get an existing job from redis with its current state

worker.getJob(type, id, function(job){
    //job object contains the job status as well as other meta data
});

param	Description
type	string literal represnting the job type
id	integer representing the job id

Get Completed Job Result

Get a completed job's result

worker.getJobResult(type, id, function(result){
    // handle result of the job
});

param	Description
type	string literal representing the job type
id	integer representing the job id

Get System Stats

Get an overview of each job type and its status (active, pending, etc...)

worker.getStats(function(stats){
    // handle system wide stats
});

Job

Job functions are available within the processing function, and can be used freely inside a worker.process() function

Report Progress

Report and save the current progress of the job

job.reportProgress(step, total);

param	Description
step	integer representing the current step progress from the total
total	integer representing the total amount of progress steps in the job

example: job.reportProgress(5,10) will result in 50% progress for the job

Add Child

Add a child job to the current job

job.addChild(job);

param	Description
job	a live Job object, usually gathered from a worker.getJob() or worker.createJob() functions

Wait For Child Jobs

Gather data from child jobs, previously added via job.addChild()

job.waitForChildren(function(){
    // handle data from children
});

• usually when delegating to child jobs, one would want to keep the parent alive untill all the children are done, in which case a done() can be called inside the job.waitForChildren() callbac

Testing

After cloning the repo and resolving dependencies via npm install, run

npm test

Monitoring

Data Store

hique saves all data in-memory by default. (using the 'native' adapter) Data is stored inside the data store through an adapter object, which is highly extensible, allowing adapters be written for other data stores (mysql, mongo, redis, etc`) fairly easily

Native - In Memory

The native adapter saves all data in-memory in the javascript's heap. Since the heap is limited to about 1.2G by default (per process) it can be switched to any other adapter.

Creating Your Own Data Store

In order to create your own data store please follow these simple steps:

1. Add a javascript file under the lib/adapters directory
2. Copy the code from stub.js in order to get the interface of all adapters
3. Implement all functions (refer to the native.js for more details about how to invoke the correct data in callbacks)
4. Add any default configuration values to config/default.js under the adapter key to be passed at initialization, this will allow to pass a config object at runtime for specific hosts, ports, etc`

• If you have written an adapter, Don`t be shy! - share it with everyone here, programmers will get an additional way to use hique, and you will gain the power of the masses in discovering bugs and issues