Neatly Punctuated Musings

    @elastic.io/batching-library
    TypeScript icon, indicating that this package has built-in type declarations

    2.0.3 • Public • Published

    CircleCI

    batching-library

    Library for batching in elastic.io. Batching allows users to combine several calls to API in one big call.

    BatchClient

    BatchClient represents client that works with batches. It contains logic for creating, updating statuses and storing batches.

    Constructor

    new BatchClient(store, config)

    BatchClient constructor expects three parameters:

    1. store - implementation of IStore interface. For example MongoStore. This is where batches will be stored.
    2. config - instance of BatchConfig. It represent configuration and defines limitation for batches e.g: maxSize, maxItemsNumber.
    Example
    import { MongoStore, IStore } from '@elastic.io/batching-library';
    import { BatchClient } from '@elastic.io/batching-library';
    import { BatchConfig } from '@elastic.io/batching-library';
    import { ConnectionOptions } from 'mongoose';
    
    const maxSize = 1000;
    const maxItemsNumber = 10;
    const maxWaitTime = 1000;
    const maxEmitRate = 1;
    const maxRetry = 0;
    const uri: string = 'url_to_mongo';
    const collection = 'collection_name';
    const connectionOpt: ConnectionOptions = {
      user: 'user',
      pass: 'password', 
    };
    const mgStore: IStore = new MongoStore(collection, uri, connectionOpt);
    const config = new BatchConfig(maxSize, maxItemsNumber, maxWaitTime, maxEmitRate, maxRetry);
    const client = new BatchClient(mgStore, config);

    Methods

    1. saveItem(item)

    Save provided item to batch, returns batch with saved item (without another Batch items).

    BatchItem properties
    Parameter Type Required Description
    id string false If not specified - uuid v1 will be generated
    item string true Body of the Batch Item
    Example
    const batch: Batch = await client.saveItem({ id: 0, item: {}});

    2. getReadyBatches()

    Using provided emitter emit batches with status 'READY'

    Example
    const batches: Batch[] = await client.getReadyBatches();
      await Promise.all(batches.map(async (batch) => {
        try {
          
         // process batch implementation
          
          await client.updateBatchStatusById(batch.id, 'SUCCESS'); // batch was successfully processed
        } catch (e) {
          log.error('Error: %o', e);
          await client.updateBatchStatusById(batch.id, 'FAILED'); // batch processed with error
        }
      }));

    Implementing your own batch storage

    For reference implementation example take a look at MongoStore

    1. Extend AbstractStore and override each abstract method.
    2. Implementation must ensure that each operation is transactional

    Environment variables

    Name Mandatory Description Values
    LOG_OUTPUT_MODE Yes Log record view format short long simple json bunyan
    LOG_LEVEL Yes Log Level fatal error warn info debug trace
    MONGO_URL No URL of mongo db used during integration tests url to mongo db
    MONGO_USER No Username of mongo db user, used during integration tests username
    MONGO_PASSWORD No Password for connecting to mongo db, used during integration tests password
    MONGO_DB No Database of mongo db used during integration tests database name
    MONGO_DROP_DB No If 1 drop database each test integration test if 0 dont 1 0

    Environment template file

    Limitations

    MongoStore:

    1. Supported Mongo database versions: 4.0 and higher.

    2. Retries do not implemented.

    3. Emit rate not implemented.

    4. Library is not guarantee sequence processing of batch item

    MaesterStore:

    1. Emit rate not implemented.

    Keywords

    Install

    npm i @elastic.io/batching-library

    DownloadsWeekly Downloads

    6

    Version

    2.0.3

    License

    Apache-2.0

    Unpacked Size

    244 kB

    Total Files

    59

    Last publish

    Collaborators

    • kevin_mobbs
    • stas1313
    • legendmiha
    • annekov
    • garry_ua
    • olga.virolainen
    • shkarupanick
    • jhorbulyk
    • irisk
    • vi6x
    • juhrmacher
    • illya_98