limited-request-queue

    5.1.0 • Public • Published

    limited-request-queue NPM Version File Size Build Status Coverage Status Dependency Monitor

    Interactively manage concurrency for outbound requests.

    • Concurrency & rate limiting prevents overload on your network.
    • Per-Host concurrency limiting prevents overload on your target network(s).
    • Pause/Resume at any time.
    const queue = new RequestQueue()
      .on(ITEM_EVENT, (url, data, done) => {
        yourRequestLib(url, () => done());
      })
      .on(END_EVENT, () => console.log('Queue completed!'));
     
    const urls = ['http://domain.com/dir1/', 'http://domain.com/dir2/'];
    urls.forEach(url => queue.enqueue(new URL(url)));
     
    setTimeout(queue.pause, 500);
    setTimeout(queue.resume, 5000);

    Installation

    Node.js >= 10 is required. To install, type this at the command line:

    npm install limited-request-queue

    Usage

    Import as an ES Module:

    import RequestQueue, {END_EVENT, ITEM_EVENT} from 'limited-request-queue';

    Import as a CommonJS Module:

    const {default:RequestQueue, END_EVENT, ITEM_EVENT} = require('limited-request-queue');

    Constructor:

    new RequestQueue(options);

    Methods & Properties

    All methods from EventEmitter are available.

    .dequeue(id)

    Removes a queue item from the queue. Returns true if a queue item was removed and false if not. Use of this function is likely not needed as items are auto-dequeued when their turn is reached.

    .enqueue(url[, data, options])

    Adds a URL to the queue. Returns a queue item ID on success.

    • url must a URL instance.
    • data is optional and can be of any type.
    • options is an optional Object that overrides any defined options in the constructor (except for maxSockets).

    .has(id)

    Returns true if the queue contains an active or queued item tagged with id and false if not.

    .isPaused

    Returns true if the queue is currently paused and false if not.

    .length

    Returns the total number of items in the queue, active and inactive.

    .numActive

    Returns the number of items whose requests are currently in progress.

    .numQueued

    Returns the number of items that have not yet made requests.

    .pause()

    Pauses the queue, but will not pause any active requests.

    .resume()

    Resumes the queue.

    Options

    options.ignorePorts

    Type: Boolean
    Default value: true
    Whether or not to treat identical hosts of different ports as a single concurrent group. Example: when true, http://mydomain.com:80 and http://mydomain.com:8080 may not have outgoing connections at the same time, but http://mydomain.com:80 and http://yourdomain.com:8080 will.

    options.ignoreProtocols

    Type: Boolean
    Default value: true
    Whether or not to treat identical hosts of different protocols as a single concurrent group. Example: when true, http://mydomain.com and https://mydomain.com may not have outgoing connections at the same time, but http://mydomain.com and https://yourdomain.com will.

    options.ignoreSubdomains

    Type: Boolean
    Default value: true
    Whether or not to treat identical domains of different subdomains as a single concurrent group. Example: when true, http://mydomain.com and http://www.mydomain.com may not have outgoing connections at the same time, but http://mydomain.com and http://www.yourdomain.com will.

    This option is not available in the browser version (due to extreme file size).

    options.maxSockets

    Type: Number
    Default value: Infinity
    The maximum number of connections allowed at any given time. A value of 0 will prevent anything from going out. A value of Infinity will provide no concurrency limiting.

    options.maxSocketsPerHost

    Type: Number
    Default value: 2
    The maximum number of connections per host allowed at any given time. A value of 0 will prevent anything from going out. A value of Infinity will provide no per-host concurrency limiting.

    options.rateLimit

    Type: Number
    Default value: 0
    The number of milliseconds to wait before each request. For a typical rate limiter, also set maxSockets to 1.

    Events

    END_EVENT, 'end'

    Called when the last item in the queue has been completed/dequeued.

    ITEM_EVENT, 'item'

    Called when a queue item's turn has been reached. Arguments are: url, data, done. Call the done function when your item's operations are complete.

    Install

    npm i limited-request-queue

    DownloadsWeekly Downloads

    11,838

    Version

    5.1.0

    License

    MIT

    Unpacked Size

    48.2 kB

    Total Files

    11

    Last publish

    Collaborators

    • stevenvachon