Needlessly Promiscuous, Modularize!

    @shelf/dynamodb-parallel-scan
    TypeScript icon, indicating that this package has built-in type declarations

    3.0.2 • Public • Published

    dynamodb-parallel-scan CircleCI npm (scoped)

    Scan DynamoDB table concurrently (up to 1,000,000 segments), recursively read all items from every segment

    Install

    $ yarn add @shelf/dynamodb-parallel-scan
    

    This library has 2 peer dependencies:

    • @aws-sdk/client-dynamodb
    • @aws-sdk/lib-dynamodb

    Make sure to install them alongside this library.

    Why this is better than a regular scan

    Easily parallelize scan requests to fetch all items from a table at once. This is useful when you need to scan a large table to find a small number of items that will fit the node.js memory.

    Scan huge tables using async generator or stream. And yes, it supports streams backpressure! Useful when you need to process a large number of items while you scan them. It allows receiving chunks of scanned items, wait until you process them, and then resume scanning when you're ready.

    Usage

    Fetch everything at once

    const {parallelScan} = require('@shelf/dynamodb-parallel-scan');
    
    (async () => {
      const items = await parallelScan(
        {
          TableName: 'files',
          FilterExpression: 'attribute_exists(#fileSize)',
          ExpressionAttributeNames: {
            '#fileSize': 'fileSize',
          },
          ProjectionExpression: 'fileSize',
        },
        {concurrency: 1000}
      );
    
      console.log(items);
    })();

    Use as async generator (or streams)

    Note: highWaterMark determines items count threshold, so Parallel Scan can fetch concurrency * 1MB more data even after highWaterMark was reached.

    const {parallelScanAsStream} = require('@shelf/dynamodb-parallel-scan');
    
    (async () => {
      const stream = await parallelScanAsStream(
        {
          TableName: 'files',
          FilterExpression: 'attribute_exists(#fileSize)',
          ExpressionAttributeNames: {
            '#fileSize': 'fileSize',
          },
          ProjectionExpression: 'fileSize',
        },
        {concurrency: 1000, chunkSize: 10000, highWaterMark: 10000}
      );
    
      for await (const items of stream) {
        console.log(items); // 10k items here
      }
    })();

    Read

    Publish

    $ git checkout master
    $ yarn version
    $ yarn publish
    $ git push origin master --tags

    License

    MIT © Shelf

    Install

    npm i @shelf/dynamodb-parallel-scan

    DownloadsWeekly Downloads

    3,095

    Version

    3.0.2

    License

    MIT

    Unpacked Size

    17.3 kB

    Total Files

    21

    Last publish

    Collaborators

    • anastasiash
    • kchlon
    • dmytro.harazdovskiy
    • duch0416
    • i5adovyi
    • olesiamuller
    • mykola.khytra
    • yuliiakovalchuk
    • el_scrambone
    • bodyaflesh
    • slavammellnikov
    • andrii-nastenko
    • andriisermiahin
    • arkadii007
    • mpushkin
    • batovpavlo
    • vasylhoban
    • andrew214
    • domovoj
    • vozemer
    • oleksii.dymnich
    • set001
    • bogdan.kyba
    • dima-bond
    • maksym.hayovets
    • oles.zadorozhnyy
    • ss1l
    • gemshelf
    • d-mii-
    • hartzler
    • tarasdyksan
    • andrii_butsa
    • nazarko_suv
    • bogdan.kolesnyk
    • vitaliishcherban
    • vladgolubev
    • marianna-milovanova
    • hmelenok
    • knupman
    • maaraanas
    • terret
    • chapelskyi.slavik
    • pihorb
    • irynah
    • diana.kryskuv
    • andy.raven
    • rafler
    • mykola.chepkyi
    • sskalp88
    • demiansua
    • yuriil
    • ktv18
    • drews_abuse
    • rostyslav-horytskyi
    • whodeen