a "batch and queue" HTTP client for making bulk POST requests to an API endpoint:
ak-fetch
is a powerful and flexible tool designed for making bulk POST
requests to an API endpoint. It provides a simple interface to batch and queue requests which send data over the network; it adds oft needed features like concurrency control, retries, fire-and-forget making it a robust solution for handling large data transfer operations with ease.
i built this module when creating hello-mixpanel as i needed a way to send large amounts of data to various SaaS analytics APIs in a reliable and efficient manner. it was useful enough for me, that i made it into a proper package.
To get started with ak-fetch
, install the module using npm:
npm install ak-fetch
you may also use npx
to run the CLI without installing the package:
npx ak-fetch --help
Use ak-fetch as a module directly in your node script:
const akFetch = require('ak-fetch');
const config = {
url: 'https://api.example.com',
data: [...],
batchSize: 10,
concurrency: 5,
delay: 1000,
// ... other configurations
};
try {
const responses = await akFetch(config);
console.log('API Responses:', responses);
} catch (error) {
console.error('Error:', error);
}
or via the command line:
npx ak-fetch --url https://api.example.com './payloads.json' --batchSize 10 --concurrency 5
Use --help
to see all options
The ak-fetch module can be configured with a variety of options to suit your needs:
Option | Type | Description |
---|---|---|
url |
string |
The URL of the API endpoint. |
data |
Object[] |
An array of data objects to be sent in the requests. |
batchSize |
number |
# records per batch; Use 0 for no batching. |
concurrency |
number |
The level of concurrency for the requests. |
delay |
number |
The delay between requests in milliseconds. |
searchParams |
Object |
The search parameters to be appended to the URL. |
bodyParams |
Object |
The body parameters to be sent in the request. |
headers |
Object |
The headers to be sent in the request. |
verbose |
boolean |
Whether to log progress of the requests. |
dryRun |
boolean |
If true, no actual requests are made. |
logFile |
string |
File path where responses will be saved. |
retries |
number |
Number of retries for failed requests; use null for fire-and-forget. |
retryDelay |
number |
Delay between retries in milliseconds. |
retryOn |
number[] |
HTTP status codes to retry on. |
(note that with the CLI you may use camelCase or snake_case for options)
Contributions to ak-fetch are welcome! Feel free to open issues or submit pull requests.
ak-fetch is ISC licensed; use it however you wish.