ParallelQueuePartitioner
Round-Robin parallel task dispatcher that executes tasks in sequence if related. The relationship is defined through the "partitionId". The partitioner works on top of any queue/broker.
Be aware that the version 2 works only with Node.js v4+
Installation
$ npm install parallel-queue-partitioner
For node 0.X (no longer supported)
$ npm install parallel-queue-partitioner@1.0.1
Usage
In a new js file, first reference the partitioner:
var parallelQueuePartitioner = ;var Partitioner = parallelQueuePartitionerPartitioner;
then reference cluster
var cluster = ;
We need to register the jobs in each cluster. Each job must contain a parameter and return a promise!
ifclusterisWorker parallelQueuePartitioner parallelQueuePartitioner
We have to produce the messages only from the master. A job message must contain:
- An id that identifies the job
- A partitionId, jobs with the same partitionId will be executed by one process in sequence; jobs with different partitionId will be executed in parallel
- A type that defines the job to run, in our case 'sum' or 'delayedSum'
- Optional data
let { var partitioner = numberOfWorkers: 8 //number of process workers ; ; //Arbitrary delayer to wait all forks are completed} ifclusterisMaster ;
Configuration
The partitioner cab be configured as follow:
- numberOfWorkers, is the number of processes we want to run (default 1)
- cleanIdlePartitionsAfterMinutes, after the given minutes, a job will clean the partition that were not used during that interval (default 15 minutes)
- loggerLevel, 'debug', 'info', 'warn', or 'error' (default 'error')
- consoleLogger, true or false, enable or disable the console logger (default: true)
- fileLogger, true or false, enable or disable the file logger (default: true)
- fileLoggerPath, the path where the logs will be saved (default: "./logger")
const partitioner = numberOfWorkers: 4 cleanIdlePartitionsAfterMinutes: 30 loggerLevel: 'info' consoleLogger: false fileLogger: false fileLoggerPath: "./bin/partitionerLogger";
Logger
If enabled, the logger will create the file in the folder specified by the user or, if not specified, in "./logger". The master and each worker will create its own log. The filenames are:
- master-pid-{PIDID}-partitioner.log
- worker-pid-{PIDID}-partitioner.log
Exemple how to use the partitioner with redis
"use strict" const parallelQueuePartitioner = ;const cluster = const process = const kue = const queue = kue ifclusterisWorker parallelQueuePartitioner parallelQueuePartitioner let { var partitioner = numberOfWorkers: 4 loggerLevel: 'debug' consoleLogger: true fileLogger: true fileLoggerPath: "./bin/logger" } let id = 0 ifclusterisMaster console for let i = 0; i < 50; i++ queue for let i = 0; i < 50; i++ queue for let i = 0; i < 150; i++ queue