@mangosteen/batch-utils

batch-utils

Take an iterable (such as an array) of values, and batch the values into multiple fixed-length arrays.

Both sync and async iterable generators are available.

What is it for?

Imagine needing to parse a huge CSV file and loading it into a database. The file is too big to fit into memory. You could just read the file line by line, but sending individual INSERTs to a SQL database is slow.

@mangosteen/batch-utils allows you to to batch the entries in preparation before executing a batch SQL insert. This provides significant performance speed-up.

Installation

With npm do:

$ npm install @mangosteen/batch-utils

Basic Usage

import { batchItemsAsync } from '@mangosteen/batch-utils';

(async () => {
    const values = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10];
    const batches = batchItemsAsync(values, 3);

    for await (const batch of batches) {
        console.log(batch);
        // console.log(typeof batch); // Array<string>
    }
})();

This code will output the following arrays:

[1, 2, 3]
[4, 5, 6]
[7, 8, 9]
[10]

Advanced Usage

import fs from 'fs';
import { lineByLine } from '@mangosteen/line-by-line';
import { batchItemsAsync } from '@mangosteen/batch-utils';

(async () => {
    const inputStream = fs.createReadStream('./shakespeare.txt');
    const lines = lineByLine(inputStream);
    const lineBatches = batchItemsAsync(lines, 100);

    for await (const batch of lineBatches) {
        // sql.batchInsert(batch);
        console.log(typeof batch); // Array<string>
    }
})();

API

batchItemsAsync(items, desiredLength)

Generates arrays of length desiredLength containing the values from items. The final array may be shorter than desiredLength if there are not enough values in items.

items: AsyncIterable<T> : Iterable of items to batch up into fixed-size arrays.

desiredLength: number : The desired length of the generated arrays.

---

forEachBatchAsync(items, desiredLength, onBatch)

Performs the specified action for each array generated from items iterable.

The generated arrays will be of length desiredLength, except for the last array which may be shorter.

items: AsyncIterable<T> : Iterable of items to batch up into fixed-size arrays.

desiredLength: number : The desired length of the generated arrays.

onBatch: (batch: T[], batchIndex: number) => Promise<void> : Action to perform for each array.

---

batchItemsSync(items, desiredLength)

Generates arrays of length desiredLength containing the values from items.

The final array may be shorter than desiredLength if there are not enough values in items.

items: Iterable<T> : Iterable of items to batch up into fixed-size arrays.

desiredLength: number : The desired length of the generated arrays.

---

forEachBatchSync(items, desiredLength, onBatch)

Performs the specified action for each array generated from items iterable.

The generated arrays will be of length desiredLength, except for the last array which may be shorter.

items: Iterable<T> : Iterable of items to batch up into fixed-size arrays.

desiredLength: number : The desired length of the generated arrays.

onBatch: (batch: T[], batchIndex: number) => void : Action to perform for each array.