Nutritious Pumpkin Mash

    through2-batch

    1.1.1 • Public • Published

    Build Status

    through2-batch

    A stream that transforms chunks to batches form the stream. NPM

    A way to use a Node.JS Transform stream that batches chunks into an array (default is 10). Objects/chunks will still come in the same order, just in batched arrays.

    Built using through2 and has the same API with the addition of a batchSize option.

    Non-objectMode streams are supported for completeness.

    Written by Nima Gardideh (halfmoon.ws) and used in production by Taplytics.

    Install

    npm install --save through2-batch

    Examples

    Process rows from a CSV in batches.

    var through2Batch = require('through2-batch');
     
    fs.createReadStream('data.csv')
      .pipe(csv2())
      .pipe(through2Batch.obj(
        {batchSize: 100},
        function (batch, enc, callback) {
          var self = this;
          console.log(batch.length); // 100
          someThingAsync(batch, function (newChunk) {
            self.push(newChunk);
          });
      }));

    Don't specify a transform fn, and let the batches be processed by another stream

    var through2Batch = require('through2-batch');
     
    fs.createReadStream('data.csv')
      .pipe(csv2())
      .pipe(through2Batch.obj())
      .pipe(getSomeOtherStreamProcessingBatches());

    Contributing

    Fixed or improved stuff? Great! Send me a pull request through GitHub or get in touch on Twitter @ngardideh.

    Install

    npm i through2-batch

    DownloadsWeekly Downloads

    4,939

    Version

    1.1.1

    License

    MIT

    Unpacked Size

    5.58 kB

    Total Files

    4

    Last publish

    Collaborators

    • nimsical