batch-request-js

    2.0.2 • Public • Published

    batch-request-js CircleCI

    Batch promise based requests to overcome network limitations or API restrictions

    Install

    • yarn add batch-request-js

    Tests

    • yarn test-unit to run unit tests
    • yarn test-e2e to run e2e tests
    • yarn test-ci to run both unit and e2e tests

    Usage

    Suppose we'd like to fetch thousands of customers from an API. To avoid network limitations or rate limiting issues, we can batch the requests:

    // node.js
    const batchRequest = require('batch-request-js')
     
     
    async function getCustomers () {
      // define array of input data e.g. customerIds
      const customerIds = ['100', '101', '102', ... ]
      // define the async request to perform against each data input
      const request = (customerId) => fetch(`${API_ENDPOINT}/${customerId}`).then(response => response.json())
     
      // fetch customers in batches of 100, delaying 200ms inbetween each batch request
      const { error, data } = await batchRequest(customerIds, request, { batchSize: 100, delay: 200 })
     
      // Data from successful requests
      console.log(data) // [{ customerId: '100', ... }, ...]
     
      // Failed requests
      console.log(error) // [{ record: "101", error: [Error: Customer not found] }, ...]
    }

    Example

    // node.js
    const batchRequest = require('batch-request-js')
     
    async function getData () {
      // setup a 100 test records
      const records = Array(100).fill(0).map((d, i) => ({ item: i }))
      // all requests will succeed and be timestamped
      const request = record => Promise.resolve({ ...record, timestamp: Date.now() })
      // batch requests 20 at a time, delaying half a second after each batch request
      const result = await batchRequest(records, request, { batchSize: 20, delay: 500 })
      console.log(result)
    //   { 
    //     error: [],
    //     data: [
    //        { record: 0, timestamp: 1533552890663 },
    //        { record: 1, timestamp: 1533552890663 },
    //        { record: 2, timestamp: 1533552890663 },
    //        { record: 3, timestamp: 1533552890663 },
    //         ...
    //     ]
    }
     
    getData()

    Handling failed batch requests

    Rerun batch request with a filtered set of inputRecords to just those that failed on the previous attempt.

    Future

    Retry logic may be implemented to handle automatically rerunning batch-request for failing requests.

    License

    MIT

    Keywords

    none

    Install

    npm i batch-request-js

    DownloadsWeekly Downloads

    59

    Version

    2.0.2

    License

    MIT

    Unpacked Size

    128 kB

    Total Files

    10

    Last publish

    Collaborators

    • avatar