tusker

Tusker - Redis Based Distributed Task Locking

Why? Let's look at following scenario

• We are transcoding ogg streams into mp3 realtime
• And oggs are coming as chunks (<15 secs)
• We have deployed several servers and routing is random and stateless

Lets Cordinate this with Tusker

When we are encoding chunks

var tusker = require('tusker');
var taskManager = tusker.initialize();
 
//when we are begin to encode a chunk
var lockname = "" + Math.random();
taskManager.lock('the-task', lockname);
 
//after we do the encoding
taskManager.unlock('the-task', lockname);

At the end of the ogg stream

var tusker = require('tusker');
var taskManager = tusker.initialize();
 
var info = { some_data: 10 };
var options = {}
taskManager.close('the-task', info, options);

Available options

attempts

• no of attempts before before mark the task as failed
• new attempt can be occurred if task completed with an error or timeout occurred

timeout

• timeout for the task completion in millis
• if this value exceeded new attempt will be occurred

Receive notification after the all related chunks for a stream has encoded

var tusker = require('tusker');
var taskManager = tusker.initialize();
 
taskManager.fetchReleased(function(err, taskName, info, completed) {
 
    //merge chunks and notify back
    var err = null; //if there is an error assign error object 
    completed(err);
});

Invoking a timeout for locks

We cannot assure every lock we create will be unlocked. If the server died before unlock, we have a lock which will never be closed. So we need a timeout machanism.

var tusker = require('tusker
var taskManager = tusker.initialize();
 
//one minute timeout
taskManager.timeoutLocks(60 * 1000, function(err) {
    
});

NOTE: We've to invoke this method via a cron like tool

Invoking a timeout for task processing

• We need to timeout long running tasks, and allow another attempt to process it
• default timeout is 30 seconds otherwise if it is not defined when task closing via .close()

var tusker = require('tusker');
var taskManager = tusker.initialize();
 
//one minute timeout
taskManager.timeoutProcessing(function(err) {
    
});

NOTE: We've to invoke this method via a cron like tool

Statistics

var tusker = require('tusker');
var taskManager = tusker.initialize();
 
//one minute timeout
taskManager.stats(function(err, stats) {
    
 
});

Stats object is formatted as shown below:

{ completed: 101, released: 1, locked: 2, failed: 2, processing: 2 }