persistent-bag

This is to bags like redis is to queues. A bag (or multiset) is filled over time and processed at once.

Table of Contents: Overview, Install, Use, Dependencies, Todos/ Contribution

Overview

This node module persistent-bag aims to be a persistent bag (also called multiset) for accumulating data over the lifetime of an application and then emitting the data after a fixed interval in one aggregated object. Goal is to work like redis but not process the entries (or jobs) individually, but aggregated by time intervals. Therefore the API is a bit designed like kue (but not as good).

Install

Business as usual: npm install persistent-bag --save.

The MySQL database (see below) needs already to be set up. The tables are auto-generated on first launch.

Use

Examples can also be found in /samples.

Init persistent-bag

const PersistentBag = require('../index');
const bag = new PersistentBag({
  host: 'localhost',
  port: '3306',
  user: 'root',
  password: '',
  database: 'test'
});

This initializes the persistent-bag and opens the connection to the MySQL database for persistence. If the required data structure (i.e. one table called _persistent-bag) does not exist yet, it will be created.

Add something to a bag

bag.add({ test: false }, function (err, dataId) {
  console.log('Entry id: ' + dataId);
});

There is only one large bag where items can be add()ed to. So right now, there's no way to have two logical types of bag at the same time (see todos).

Although in fact, each new time interval (fixed to 15 minutes) is it's own bag, which is emitted right after the time interval for this bag is over. By the way, we could also make the time interval dynamic (see todos).

Process a bag

bag.process(function worker(bag, done) {
 
  console.log(bag.data);
 
  setTimeout(function () {
    done();
  }, 1000);
 
});

Processing bags is done by subscribing to persistent-bag as listed above. This supplied worker function then gets the whole bag with the aggregated data that has been added in bag.data.

After the bag has been processed, the provided callback done() needs to be called. This tells persistent-bag to emit the next bag. If done(err) is called with an error, the bag will not be marked as successfully processed and emitted again.

Dependencies

The items in the bag are stored persistently using a MySQL database, so first and most obvious dependecy is MySQL.

Furthermore, several npm modules are added as dependencies in the package.json. Those are:

• async for control flow
• moment for working with the time intervals
• mysql to connect to the MySQL database
• node-schedule to schedule processing
• sha1 to create the hashes for the bags
• underscore as utility belt

TODOs and Contribution

If you want to contribute, please do. Some suggestions:

1. Enable different types of bag: Just like kue has different type of jobs, we want different type of bags.
2. Make time interval dynamic: Currently, a new bag is created and emitted every 15 minutes. It would be useful to make this dynamic somehow, while stilling enabling persistence.