Nearsighted Prank Master

    stream-batches

    1.2.0 • Public • Published

    stream-batches

    This is an object-mode stream that collects chunks and flushes them as batches.

    Flushing happens either when the incoming stream is done, or when a limit is reached.

    There are three kinds of limits: bytes, items and time. More than one limit can be active, and a flush occurs when any of them are hit.

    Example

    Send batches of 10 items through - this is the default behavior.

    mySource
      .pipe(batches({ limit: { items: 10 } }))
      .pipe(myDestination)

    Collect (up to) 1 MB worth of records:

    mySource
      .pipe(batches({ limit: { bytes: 1 * 1024 * 1024 } }))
      .pipe(myDestination)

    Collect up to 100 records no larger than 256kb but don't wait longer than 500ms

    mySource
      .pipe(batches({ limit: { items: 100, bytes: 256 * 1024, time: 500 } }))
      .pipe(myDestination)

    Byte Count

    When using a byte limit, the default counter is Buffer.byteLength. This can be customized with a byteCount function passed in via options.

    This inflates the byte count by 2 for each record:

    mySource
      .pipe(batches({ limit: { bytes: 256 }, byteCount: r => Buffer.byteLength(r) + 2 }))
      .pipe(myDestination)

    This might be useful if you're sending a request that has a payload limit, and you want to factor in the overhead of JSON.stringify on the batched array.

    Install

    npm i stream-batches

    DownloadsWeekly Downloads

    0

    Version

    1.2.0

    License

    MIT

    Unpacked Size

    5.94 kB

    Total Files

    4

    Last publish

    Collaborators

    • dougmoscrop