Ready to take your JavaScript development to the next level? Meet npm Enterprise - the ultimate in enterprise JavaScript. Learn more »

bigdata-flow

0.0.2 • Public • Published

Bigdata Flow

npmpackage

Queue manager for data entry and editing with Redis and MongoDB for large data streams

Install

$ npm install bigdata-flow

Usage

Flow control initially inserts information into Redis that will be synchronized with MongoDB without forcing large amounts of data

const BigdataFlow = require("bigdata-flow"),
      Flow = new BigdataFlow({
          mongodb: { url: "mongodb://localhost:27017", dbName: "flow" },
          redis: { host: "127.0.0.1", port: 6379 },
          actionsPerSecond: 1000
      });
 
Flow.then((flow) => {
    flow.synchronize();
}).catch((err) => {
    console.log(err);
});

After the configuration for connection with Redis and MongoDB the synchronize function performs the reading of the data registered in the maximum limit I define in the configuration 'actionsPerSecond', if the parameter is not configured the default value will be 1000 actions per second

Insert

The insert function will synchronize with MongoDB using the 'insert' function there is also a function for update and insert and update (upsert)

const crypto = require("crypto");
 
for(let i = 0; i < 100000; i++){
    flow.insert("mycollection", {
        hash: crypto.createHash('sha256').update(new Buffer(i), 'utf8').digest("hex")
    });
}

In the example above will be inserted 100,000 hash sequentially generated to popular Redis and synchronize with MongoDB according to the configured limit

Update/Upsert

In the case of update and upsert the query parameter will be used to filter the records that must be changed

const crypto = require("crypto");
 
for(let i = 0; i < 100000; i++){
    flow.upsert("mycollection", {
        hash: crypto.createHash('sha256').update(new Buffer(i), 'utf8').digest("hex")
    }, { lastmodified: new Date().getTime() });
}

install

npm i bigdata-flow

Downloadsweekly downloads

2

version

0.0.2

license

MIT

homepage

github.com

repository

Gitgithub

last publish

collaborators

  • avatar
Report a vulnerability