Nearsighted Prank Master

    @mangosteen/batch-utils
    TypeScript icon, indicating that this package has built-in type declarations

    1.0.0 • Public • Published

    batch-utils

    Take an iterable (such as an array) of values, and batch the values into multiple fixed-length arrays.

    Both sync and async iterable generators are available.

    What is it for?

    Imagine needing to parse a huge CSV file and loading it into a database. The file is too big to fit into memory. You could just read the file line by line, but sending individual INSERTs to a SQL database is slow.

    @mangosteen/batch-utils allows you to to batch the entries in preparation before executing a batch SQL insert. This provides significant performance speed-up.

    Installation

    With npm do:

    $ npm install @mangosteen/batch-utils
    

    Basic Usage

    import { batchItemsAsync } from '@mangosteen/batch-utils';
    
    (async () => {
        const values = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10];
        const batches = batchItemsAsync(values, 3);
    
        for await (const batch of batches) {
            console.log(batch);
            // console.log(typeof batch); // Array<string>
        }
    })();

    This code will output the following arrays:

    [1, 2, 3]
    [4, 5, 6]
    [7, 8, 9]
    [10]

    Advanced Usage

    import fs from 'fs';
    import { lineByLine } from '@mangosteen/line-by-line';
    import { batchItemsAsync } from '@mangosteen/batch-utils';
    
    (async () => {
        const inputStream = fs.createReadStream('./shakespeare.txt');
        const lines = lineByLine(inputStream);
        const lineBatches = batchItemsAsync(lines, 100);
    
        for await (const batch of lineBatches) {
            // sql.batchInsert(batch);
            console.log(typeof batch); // Array<string>
        }
    })();

    API

    batchItemsAsync(items, desiredLength)

    Generates arrays of length desiredLength containing the values from items. The final array may be shorter than desiredLength if there are not enough values in items.

    items: AsyncIterable<T> : Iterable of items to batch up into fixed-size arrays.

    desiredLength: number : The desired length of the generated arrays.


    forEachBatchAsync(items, desiredLength, onBatch)

    Performs the specified action for each array generated from items iterable.

    The generated arrays will be of length desiredLength, except for the last array which may be shorter.

    items: AsyncIterable<T> : Iterable of items to batch up into fixed-size arrays.

    desiredLength: number : The desired length of the generated arrays.

    onBatch: (batch: T[], batchIndex: number) => Promise<void> : Action to perform for each array.


    batchItemsSync(items, desiredLength)

    Generates arrays of length desiredLength containing the values from items.

    The final array may be shorter than desiredLength if there are not enough values in items.

    items: Iterable<T> : Iterable of items to batch up into fixed-size arrays.

    desiredLength: number : The desired length of the generated arrays.


    forEachBatchSync(items, desiredLength, onBatch)

    Performs the specified action for each array generated from items iterable.

    The generated arrays will be of length desiredLength, except for the last array which may be shorter.

    items: Iterable<T> : Iterable of items to batch up into fixed-size arrays.

    desiredLength: number : The desired length of the generated arrays.

    onBatch: (batch: T[], batchIndex: number) => void : Action to perform for each array.

    Install

    npm i @mangosteen/batch-utils

    DownloadsWeekly Downloads

    0

    Version

    1.0.0

    License

    MIT

    Unpacked Size

    24.1 kB

    Total Files

    10

    Last publish

    Collaborators

    • mangosteen