npm

Ready to take your JavaScript development to the next level? Meet npm Enterprise - the ultimate in enterprise JavaScript. Learn more »

limited-request-queue

3.0.4 • Public • Published

limited-request-queue NPM Version Bower Version Build Status Dependency Status

Interactively manage concurrency for outbound requests.

Features:

  • Concurrency & rate limiting prevents overload on your server
  • Per-Host concurrency limiting prevents overload on everyone else's servers
  • Pause/Resume at any time
  • Works in the browser (~4.4KB not gzipped)
// Will work with any similar module, not just "request"
var request = require("request");
var RequestQueue = require("limited-request-queue");
 
var queue = new RequestQueue(null, {
    item: function(input, done) {
        request(input.url, function(error, response) {
            done();
        });
    },
    end: function() {
        console.log("Queue completed!");
    }
});
 
var urls = ["http://website.com/dir1/", "http://website.com/dir2/"];
urls.forEach(queue.enqueue, queue);
 
setTimeout(queue.pause, 500);
setTimeout(queue.resume, 5000);

Installation

Node.js >= 4 is required. To install, type this at the command line:

npm install limited-request-queue

Note: for use in a web browser, you will likely need Object.assign and URL polyfills for maximum coverage.

Constructor

new RequestQueue(options, handlers);

Methods

.dequeue(id)

Removes a queue item from the queue. Use of this function is likely not needed as items are auto-dequeued when their turn is reached. Returns true on success or an Error on failure.

.enqueue(input)

Adds a URL to the queue. input can either be a URL String or a configuration Object. Returns a queue ID on success or an Error on failure.

If input is an Object, it will accept the following keys:

  • url: a URL String, URL or Node URL-compatible Object.
  • data: additional data to be stored in the queue item.
  • id: a unique ID (String or Number). If not defined, one will be generated.

.length()

Returns the total number of items in the queue, active and inactive.

.numActive()

Returns the number of items whose requests are currently in progress.

.numQueued()

Returns the number of items that have not yet made requests.

.pause()

Pauses the queue, but will not pause any active requests.

.resume()

Resumes the queue.

Options

options.ignorePorts

Type: Boolean
Default value: true
Whether or not to treat identical hosts of different ports as a single concurrent group. Example: when true, http://mywebsite.com:80 and http://mywebsite.com:8080 may not have outgoing connections at the same time, but http://mywebsite.com:80 and http://yourwebsite.com:8080 will.

options.ignoreSchemes

Type: Boolean
Default value: true
Whether or not to treat identical hosts of different schemes/protocols as a single concurrent group. Example: when true, http://mywebsite.com and https://mywebsite.com may not have outgoing connections at the same time, but http://mywebsite.com and https://yourwebsite.com will.

options.ignoreSubdomains

Type: Boolean
Default value: true
Whether or not to treat identical hosts of different subdomains as a single concurrent group. Example: when true, http://mywebsite.com and http://www.mywebsite.com may not have outgoing connections at the same time, but http://mywebsite.com and http://www.yourwebsite.com will.

This option is not available in the browser version (due to extreme file size).

options.maxSockets

Type: Number
Default value: Infinity
The maximum number of connections allowed at any given time. A value of 0 will prevent anything from going out. A value of Infinity will provide no concurrency limiting.

options.maxSocketsPerHost

Type: Number
Default value: 1
The maximum number of connections per host allowed at any given time. A value of 0 will prevent anything from going out. A value of Infinity will provide no per-host concurrency limiting.

options.rateLimit

Type: Number
Default value: 0
The number of milliseconds to wait before each request. For a typical rate limiter, also set maxSockets to 1.

Handlers

handlers.end

Called when the last item in the queue has been completed.

handlers.item

Called when a queue item's turn has been reached. Arguments are: input, done.

install

npm i limited-request-queue

Downloadsweekly downloads

5,322

version

3.0.4

license

MIT

homepage

github.com

repository

Gitgithub

last publish

collaborators

  • avatar
Report a vulnerability