redis-smq
    TypeScript icon, indicating that this package has built-in type declarations

    3.3.0 • Public • Published

    RedisSMQ - Yet another simple Redis message queue

    Tests Coverage Status NPM version NPM downloads Code quality

    A simple high-performance Redis message queue for Node.js.

    For more details about RedisSMQ design see https://medium.com/@weyoss/building-a-simple-message-queue-using-redis-server-and-node-js-964eda240a2a

    Features

    • High-performance message processing
    • Scalable: A queue can be consumed by many concurrent consumers, running on the same or different hosts.
    • Persistent: No messages are lost in case of consumer failures.
    • Atomic: A message can be delivered only to one consumer at a time.
    • Message expiration: A message will expire and not be consumed if it has been in the queue for longer than the TTL (time-to-live).
    • Message consume timeout: The amount of time for a consumer to consume a message. If the timeout exceeded, message processing is cancelled and the message is re-queued again.
    • Delaying and scheduling message delivery: Messages can be configured to be delayed, delivered for N times with an optional period between deliveries, and to be scheduled using CRON expressions.
    • Reliable Priority Queues: Supports priority messaging.
    • HTTP API: an HTTP interface is provided to interact with the MQ.
    • Web UI: The MQ state (input/processing/acks/unacks messages rates, online consumers, queues, etc.) is provided and updated in real-time. The Web UI allows monitoring and managing the MQ.
    • JSON Logging: Supports JSON log format for troubleshooting and analytic purposes.
    • Highly optimized: No promises, no async/await, small memory footprint, no memory leaks. See callbacks vs promises vs async/await benchmarks.
    • Configurable: Many options and features can be configured.
    • Supports both redis & ioredis: RedisSMQ can be configured to use either redis or ioredis to connect to Redis server.

    Table of content

    1. What's new?
    2. Installation
    3. Configuration
    4. Usage
      1. Basic
        1. Message Class
        2. Producer Class
        3. Consumer Class
      2. Advanced Topics
        1. Message Scheduler
        2. Priority Queues
        3. HTTP API
        4. Web UI
        5. Logs
    5. Performance
    6. Contributing
    7. License

    What's new?

    2021.10.07

    • With the release of v3.3.0, reliable, persistent priority queues are now supported.

    2021.09.08

    • A major release v3 is out.
    • Upgrading your installation to the newest version should be straightforward as most APIs are compatible, with some exceptions.
    • The project's code base has been migrated to TypeScript to make use of strong typings.
    • JavaScript's users are always first class citizens.

    See CHANGELOG for more details.

    Installation

    npm install redis-smq --save
    

    Considerations:

    • RedisSMQ is targeted to be used in production environments. Therefore, only active LTS and maintenance LTS Node.js releases (v12, v14, and v16) are supported. The latest stable Node.js version is recommended.
    • Minimal Redis server version is 2.6.12.

    Configuration

    See Configuration for more details.

    Usage

    Basics

    RedisSMQ provides 3 classes: Message, Producer and Consumer in order to work with the message queue.

    Message Class

    Message class is the main component responsible for creating and handling messages. It encapsulates and provides all the required methods needed to construct and deal with messages.

    const { Message } = require('redis-smq');
    const message = new Message();
    message
        .setBody({hello: 'world'})
        .setTTL(3600000)
        .setScheduledDelay(10)
        .setScheduledRepeat(6)
        .setScheduledPeriod(60)
        .setScheduledCron('* 30 * * * *');
    
    let messageTTL = message.getTTL();

    See Message Reference for more details.

    Producer Class

    Producer class is in turn responsible for producing messages.

    Each producer instance has an associated message queue and provides produceMessage() method which handle the message and decides to either send it to the message queue scheduler or to immediately enqueue it for delivery.

    // filename: ./examples/javascript/ns1-test-queue-producer.js
    
    'use strict';
    const { Message, Producer } = require('redis-smq');
    
    const message = new Message();
    
    message
        .setBody({hello: 'world'})
        .setTTL(3600000);
    
    const producer = new Producer('test_queue');
    producer.produceMessage(message, (err) => {
       if (err) console.log(err);
       else console.log('Successfully produced')
    });

    See Producer Reference for more details.

    Consumer Class

    The Consumer class is the base class for all consumers. All consumers extends this base class and implements consume() method which got called once a message is received.

    Consumer classes are saved per files. Each consumer file represents a consumer class.

    A consumer class may look like:

    // filename: ./examples/javascript/ns1-test-queue-consumer.js
    'use strict';
    
    const { Consumer } = require('redis-smq');
    
    class TestQueueConsumer extends Consumer {
        consume(message, cb) {
            console.log('Got a message to consume:', message);
            cb();
        }
    }
    
    const consumer = new TestQueueConsumer('test_queue');
    consumer.run();

    To start consuming messages, a consumer needs first to be launched from CLI to connect to the Redis server and wait for messages:

    $ node ./examples/javascript/test-queue-consumer.js
    

    Once a message is received and processed the consumer should acknowledge the message by invoking the callback function without arguments.

    The message acknowledgment informs the MQ that the message has been successfully consumed.

    If an error occurs, the message should be unacknowledged and the error should be reported to the MQ by calling the callback function. Failed messages are re-queued and delivered again unless message retry threshold is exceeded. Then the messages are moved to dead-letter queue (DLQ). Each message queue has a system generated corresponding queue called dead-letter queue where all failed messages are moved to.

    See Consumer Reference for more details.

    Advanced Topics

    Performance

    See Performance for more details.

    Contributing

    So you are interested in contributing to this project? Please see CONTRIBUTING.md.

    License

    MIT

    Install

    npm i redis-smq

    DownloadsWeekly Downloads

    761

    Version

    3.3.0

    License

    MIT

    Unpacked Size

    405 kB

    Total Files

    136

    Last publish

    Collaborators

    • weyoss