Queuem

The simpliest queue for Node.js you've ever seen

It is made for cases when you need to run 1,000,000 tasks (parsing files, API requests, etc.) in a queue.

Was originally made for Page Check module.

Keep in mind:

• No need for local storage or database.
• The queue exists within the current Node.js process.
• Based on Node.js EventEmitter and has a few subtleties of use.

API

let Queue = new Queuem([options])

Properties

parallel

Type: Number
Default: 1
Get/set the number of concurrent tasks. Takes effect immediately only when the number increases

emptyDelay

Type: Number
After the queue is empty, the 'empty' event will be called immediately. This option allows to call the 'empty' event with a certain delay

queue

Type: Array
Tasks list. Decreases as the tasks are completed

processing

Type: Number
Amount of currently running tasks

Methods

append(task[, data])

Type: Function
Adds a new task to the end of the queue

prepend(task[, data])

Type: Function
Adds a new task to the beginning of the queue

pause()

Type: Function
Pauses processing the queue. But you can still add tasks in the queue

resume()

Type: Function

flush()

Type: Function
Clears the queue. Already running tasks would not be killed. empty event would be called immediately if there are no processing tasks

Task

Task function should have three arguments

next

Type: Function
When you feel like task is done, call next() function to pass the slot to the next task in a queue

Queue.append((next, data) => {
   // data => { job_id: 549 }
   
   // Tell the task manager that it can take the next task
   next();
 
}, { job_id: 549 })

data

Type: Any
Some data that needs to be passed into the task

Queue.prepend(fn, { job_id: 549 })

task

Type: Function
Task function, just in case you want to run the task one more time

Events

added

Useless event, but let it be

done

Triggered each time a task is completed. If you passed something to the next() function, it will be available throught handler function argument

Queue.on('done', (result) => {
   // result => { job_id: 549 }
})

changed

Triggered by any changes

Queue.on('changed', (e) => {
   // e.action => 'added' or 'done'
})

empty

Fires when queue is empty and there are no running tasks

Usage

const Queuem = require('queuem');
 
let Queue = new Queuem({ parallel: 2, emptyDelay: 5000 });
 
Queue.on('done', (result) => {
   console.log('done:',  result);
})
 
Queue.on('empty', () => {
   console.log('--- COMPLETED ---');
})
 
 
 
for(var i = 0; i < 300; i++){
   Queue.append(Task, { job_id: i })
}
 
function Task(next, data, task){
   if(data.error){
      // Append failed task back to the queue
      data.retries = data.retries + 1 || 1;
 
      if(data.retries < 10){
         Queue.append(task, data);
      }
   }
 
   setTimeout(next, Math.random() * 2000, data)
}
 
// Let's increase the number of parallel tasks
setTimeout(() => Queue.parallel = 50, 3000);

Changelog

v3.2.0 (2019-10-06):

• Added task argument

v3.1.1 (2019-10-01):

• Added emptyDelay option

v3.1.0 (2019-09-29):

• data and next arguments have been reversed

v3.0.0 (2019-09-22):

• Completely rethought the concept of the module

v1.1.1 (2018-02-18):

• Changed API, now constructor has only one argument
• Added callback function for each finished task
• Added con setter to changes the amount of concurrent tasks 'on the fly'