Utilities for working with AWS. These utilities can be used for interacting with live AWS services or Localstack. For ease of setup, testing, and credential management, code interacting with AWS services should use the helpers in this module.
npm install @cumulus/aws-client
To use these utilities with Localstack, make sure you have a running instance of Localstack and set this environment variable:
NODE_ENV=test
- DynamoDbSearchQueue
-
Class to efficiently search all of the items in a DynamoDB table, without loading them all into memory at once. Handles paging.
- S3ListObjectsV2Queue
-
Class to efficiently list all of the objects in an S3 bucket, without loading them all into memory at once. Handles paging of listS3ObjectsV2 requests.
- S3ObjectStore
-
Class to use when interacting with S3
-
createKey(params) ⇒
Promise.<Object>
-
Create a KMS key
See https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/KMS.html#createKey-property for allowed params and return value.
-
encrypt(KeyId, Plaintext) ⇒
Promise.<string>
-
Encrypt a string using KMS
-
decryptBase64String(ciphertext) ⇒
string
-
Decrypt a KMS-encrypted string, Base 64 encoded
-
CloudFormation
-
~describeCfStack(StackName) ⇒
Promise.<CloudFormation.Stack>
-
~describeCfStackResources(StackName) ⇒
Promise.<CloudFormation.StackResources>
-
~getCfStackParameterValues(stackName, parameterKeys) ⇒
Promise.<Object>
-
~describeCfStack(StackName) ⇒
Describes a given CloudFormation stack
Kind: inner method of CloudFormation
Returns: Promise.<CloudFormation.Stack>
- The resources belonging to the stack
Param | Type | Description |
---|---|---|
StackName | string |
The name of the CloudFormation stack to query |
Describes the resources belonging to a given CloudFormation stack
See CloudFormation.StackResources
Kind: inner method of CloudFormation
Returns: Promise.<CloudFormation.StackResources>
- The resources belonging to the stack
Param | Type | Description |
---|---|---|
StackName | string |
The name of the CloudFormation stack to query |
Get parameter values for the given CloudFormation stack
Kind: inner method of CloudFormation
Returns: Promise.<Object>
- Object keyed by parameter names
Param | Type | Description |
---|---|---|
stackName | string |
The name of the CloudFormation stack to query |
parameterKeys | Array.<string> |
Key names for the stack parameters that you want to return |
CloudwatchEvents~putEvent(name, schedule, state, [description], [role]) ⇒ Promise.<CloudWatchEvents.PutRuleResponse>
Create a CloudWatch Events rule
Kind: inner method of CloudwatchEvents
Param | Type | Description |
---|---|---|
name | string |
the rule name |
schedule | string |
a ScheduleExpression |
state | string |
the state of the rule |
[description] | string |
|
[role] | string |
a Role ARN |
-
DynamoDb
-
static
-
.scan ⇒
Promise.<Object>
-
.createAndWaitForDynamoDbTable(params) ⇒
Promise.<Object>
-
.deleteAndWaitForDynamoDbTableNotExists(params) ⇒
Promise
-
.scan ⇒
-
inner
-
~get(params) ⇒
Promise.<Object>
-
~parallelScan(params) ⇒
Promise
-
~get(params) ⇒
-
static
Call DynamoDb client scan
See DocumentClient.scan()
for descriptions of params
and the return data.
Kind: static property of DynamoDb
Param | Type |
---|---|
params | Object |
Create a DynamoDB table and then wait for the table to exist
Kind: static method of DynamoDb
Returns: Promise.<Object>
- the output of the createTable call
Param | Type | Description |
---|---|---|
params | Object |
the same params that you would pass to AWS.createTable See https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-dynamodb/classes/dynamodb.html#createtable |
Delete a DynamoDB table and then wait for the table to not exist
Kind: static method of DynamoDb
Param | Type | Description |
---|---|---|
params | Object |
the same params that you would pass to AWS.deleteTable See https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-dynamodb/classes/dynamodb.html#deletetable |
Call DynamoDb client get
See DocumentClient.get()
for descriptions of params
and the return data.
Kind: inner method of DynamoDb
Throws:
-
RecordDoesNotExist
if a record cannot be found
Param | Type | Description |
---|---|---|
params | Object |
|
params.tableName | string |
Table name to read |
params.item | GetCommandInput.Key |
Key identifying object to get |
params.client | DynamoDBDocument |
Instance of a DynamoDb DocumentClient |
params.getParams | Object |
Additional parameters for DocumentClient.get() |
Do a parallel scan of DynamoDB table using a document client.
See https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Scan.html#Scan.ParallelScan. See DocumentClient.scan().
Kind: inner method of DynamoDb
Param | Type | Description |
---|---|---|
params | Object |
|
params.totalSegments | number |
Total number of segments to divide table into for parallel scanning |
params.scanParams | ScanInput |
Params for the DynamoDB client scan operation See https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Scan.html |
params.processItemsFunc | function |
Function used to process returned items by scan |
[params.dynamoDbClient] | DynamoDBDocument |
Instance of Dynamo DB document client |
[params.retryOptions] | pRetry.Options |
Retry options for scan operations |
Describe a Kinesis stream.
Kind: inner method of Kinesis
Returns: Promise.<Object>
- The stream description response
Param | Type | Description |
---|---|---|
params | Object |
|
params.StreamName | string |
A Kinesis stream name |
retryOptions | Object |
Options passed to p-retry module |
Bare check for EventBridge shape
Kind: inner method of Lambda
Invoke a Lambda function
Kind: inner method of Lambda
-
S3
-
static
.getS3Object- .recursivelyDeleteS3Bucket ⇒
-
.listS3ObjectsV2(params) ⇒
Promise.<Array>
- .listS3ObjectsV2Batch(params)
-
inner
-
~s3Join(...args) ⇒
string
-
~parseS3Uri(uri) ⇒
Object
-
~buildS3Uri(bucket, key) ⇒
string
-
~s3TagSetToQueryString(tagset) ⇒
string
-
~deleteS3Object(bucket, key) ⇒
Promise
- ~headObject(Bucket, Key, retryOptions) ⇒
-
~s3ObjectExists(params) ⇒
Promise.<boolean>
- ~waitForObjectToExist()
- ~s3PutObject(params)
- ~putFile()
- ~s3CopyObject()
- ~promiseS3Upload()
- ~streamS3Upload()
- ~getObjectReadStream()
- ~downloadS3File()
- ~getObjectSize()
- ~s3GetObjectTagging()
- ~s3PutObjectTagging()
- ~getObject()
- ~waitForObject()
- ~getObjectStreamContents()
- ~getTextObject()
- ~getJsonS3Object()
- ~fileExists()
- ~deleteS3Files(s3Objs)
-
~uploadS3FileStream(fileStream, bucket, key, s3opts) ⇒
Promise
- ~listS3Objects()
-
~deleteS3Buckets(buckets) ⇒
Promise
- ~calculateObjectHash(params)
-
~validateS3ObjectChecksum(params) ⇒
Promise.<boolean>
-
~getFileBucketAndKey(pathParams) ⇒
Array.<string>
-
~createBucket(Bucket) ⇒
Promise
-
~createS3Buckets(buckets) ⇒
Promise
-
~multipartCopyObject(params) ⇒
Promise.<{etag: string}>
- ~copyObject()
-
~moveObject() ⇒
Promise.<undefined>
-
~s3Join(...args) ⇒
-
static
Deprecated
Gets an object from S3.
Kind: static property of S3
Delete a bucket and all of its objects from S3
Kind: static property of S3
Returns: the promised result of S3.deleteBucket
Param | Description |
---|---|
bucket | name of the bucket |
Fetch complete list of S3 objects
listObjectsV2 is limited to 1,000 results per call. This function continues listing objects until there are no more to be fetched.
The passed params must be compatible with the listObjectsV2 call.
https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#listObjectsV2-property
Kind: static method of S3
Returns: Promise.<Array>
- resolves to an array of objects corresponding to
the Contents property of the listObjectsV2 response
Param | Type | Description |
---|---|---|
params | Object |
params for the s3.listObjectsV2 call |
Fetch lazy list of S3 objects
listObjectsV2 is limited to 1,000 results per call. This function continues listing objects until there are no more to be fetched.
The passed params must be compatible with the listObjectsV2 call.
https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#listObjectsV2-property
Kind: static method of S3
Param | Description |
---|---|
params | params for the s3.listObjectsV2 call |
Join strings into an S3 key without a leading slash
Kind: inner method of S3
Returns: string
- the full S3 key
Param | Type | Description |
---|---|---|
...args |
string | Array.<string>
|
the strings to join |
parse an s3 uri to get the bucket and key
Kind: inner method of S3
Returns: Object
- Returns an object with Bucket
and Key
properties
Param | Type | Description |
---|---|---|
uri | string |
must be a uri with the s3:// protocol |
Given a bucket and key, return an S3 URI
Kind: inner method of S3
Returns: string
- an S3 URI
Param | Type | Description |
---|---|---|
bucket | string |
an S3 bucket name |
key | string |
an S3 key |
Convert S3 TagSet Object to query string e.g. [{ Key: 'tag', Value: 'value }] to 'tag=value'
Kind: inner method of S3
Returns: string
- tags query string
Param | Type | Description |
---|---|---|
tagset | Array.<Object> |
S3 TagSet array |
Delete an object from S3
Kind: inner method of S3
Returns: Promise
- promise of the object being deleted
Param | Type | Description |
---|---|---|
bucket | string |
bucket where the object exists |
key | string |
key of the object to be deleted |
Get an object header from S3
Kind: inner method of S3
Returns: returns response from S3.headObject
as a promise
Param | Description |
---|---|
Bucket | name of bucket |
Key | key for object (filepath + filename) |
retryOptions | options to control retry behavior when an object does not exist. See https://github.com/tim-kos/node-retry#retryoperationoptions By default, retries will not be performed |
Test if an object exists in S3
Kind: inner method of S3
Returns: Promise.<boolean>
- a Promise that will resolve to a boolean indicating
if the object exists
Param | Description |
---|---|
params | same params as https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#headObject-property |
Wait for an object to exist in S3
Kind: inner method of S3
Param | Default | Description |
---|---|---|
params.interval | 1000 |
interval before retries, in ms |
params.timeout | 30000 |
timeout, in ms |
Put an object on S3
Kind: inner method of S3
Param | Description |
---|---|
params | same params as https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#putObject-property |
Upload a file to S3
Kind: inner method of S3
Copy an object from one location on S3 to another
Kind: inner method of S3
Upload data to S3
see https://github.com/aws/aws-sdk-js-v3/tree/main/lib/lib-storage
Kind: inner method of S3
Upload data to S3 using a stream
Kind: inner method of S3
Get a readable stream for an S3 object
Kind: inner method of S3
Downloads the given s3Obj to the given filename in a streaming manner
Kind: inner method of S3
Get the size of an S3 object
Kind: inner method of S3
Get object Tagging from S3
Kind: inner method of S3
Puts object Tagging in S3 https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#putObjectTagging-property
Kind: inner method of S3
Gets an object from S3.
Kind: inner method of S3
Get an object from S3, waiting for it to exist and, if specified, have the correct ETag.
Kind: inner method of S3
Transform streaming response from S3 object to text content
Kind: inner method of S3
Fetch the contents of an S3 object
Kind: inner method of S3
Fetch JSON stored in an S3 object
Kind: inner method of S3
Check if a file exists in an S3 object
Kind: inner method of S3
Delete files from S3
Kind: inner method of S3
Param | Description |
---|---|
s3Objs | An array of objects containing keys 'Bucket' and 'Key' |
Upload the file associated with the given stream to an S3 bucket
Kind: inner method of S3
Returns: Promise
- A promise
Param | Type | Description |
---|---|---|
fileStream | ReadableStream |
The stream for the file's contents |
bucket | string |
The S3 bucket to which the file is to be uploaded |
key | string |
The key to the file in the bucket |
s3opts | Object |
Options to pass to the AWS sdk call (defaults to {} ) |
List the objects in an S3 bucket
Kind: inner method of S3
Delete a list of buckets and all of their objects from S3
Kind: inner method of S3
Returns: Promise
- the promised result of S3.deleteBucket
Param | Type | Description |
---|---|---|
buckets | Array |
list of bucket names |
Calculate the cryptographic hash of an S3 object
Kind: inner method of S3
Param | Type | Description |
---|---|---|
params | Object |
|
params.s3 | S3 |
an S3 instance |
params.algorithm | string |
cksum , or an algorithm listed in openssl list -digest-algorithms
|
params.bucket | string |
|
params.key | string |
Validate S3 object checksum against expected sum
Kind: inner method of S3
Returns: Promise.<boolean>
- returns true for success
Throws:
-
InvalidChecksum
- Throws error if validation fails
Param | Type | Description |
---|---|---|
params | Object |
params |
params.algorithm | string |
checksum algorithm |
params.bucket | string |
S3 bucket |
params.key | string |
S3 key |
params.expectedSum |
number | string
|
expected checksum |
[params.options] | Object |
crypto.createHash options |
Extract the S3 bucket and key from the URL path parameters
Kind: inner method of S3
Returns: Array.<string>
- [Bucket, Key]
Param | Type | Description |
---|---|---|
pathParams | string |
path parameters from the URL bucket/key in the form of |
Create an S3 bucket
Kind: inner method of S3
Param | Type | Description |
---|---|---|
Bucket | string |
the name of the S3 bucket to create |
Create multiple S3 buckets
Kind: inner method of S3
Param | Type | Description |
---|---|---|
buckets | Array.<string> |
the names of the S3 buckets to create |
Copy an S3 object to another location in S3 using a multipart copy
Kind: inner method of S3
Returns: Promise.<{etag: string}>
- object containing the ETag of the
destination object
note: this method may error if used with zero byte files. see CUMULUS-2557 and https://github.com/nasa/cumulus/pull/2117.
Param | Type | Default | Description |
---|---|---|---|
params | Object |
||
params.sourceBucket | string |
||
params.sourceKey | string |
||
params.destinationBucket | string |
||
params.destinationKey | string |
||
[params.sourceObject] | S3.HeadObjectOutput |
Output from https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#headObject-property | |
[params.ACL] | string |
an S3 Canned ACL | |
[params.copyTags] | boolean |
false |
|
[params.chunkSize] | number |
chunk size of the S3 multipart uploads |
Copy an S3 object to another location in S3
Kind: inner method of S3
Param | Default | Description |
---|---|---|
params.ACL | an S3 Canned ACL | |
params.copyTags | false |
|
params.chunkSize | chunk size of the S3 multipart uploads |
Move an S3 object to another location in S3
Kind: inner method of S3
Param | Type | Default | Description |
---|---|---|---|
[params.ACL] | string |
an S3 Canned ACL | |
[params.copyTags] | boolean |
false |
|
[params.chunkSize] | number |
chunk size of the S3 multipart uploads |
-
SNS
-
~publishSnsMessageWithRetry(snsTopicArn, message, retryOptions) ⇒
Promise.<undefined>
- ~createSnsTopic(snsTopicName) ⇒
-
~publishSnsMessageWithRetry(snsTopicArn, message, retryOptions) ⇒
Publish a message to an SNS topic. Does not catch errors, to allow more specific handling by the caller.
Kind: inner method of SNS
Param | Description |
---|---|
snsTopicArn | SNS topic ARN |
message | Message object |
retryOptions | options to control retry behavior when publishing a message fails. See https://github.com/tim-kos/node-retry#retryoperationoptions |
Create an SNS topic with a given name.
Kind: inner method of SNS
Returns: - ARN of the created SNS topic
Param | Description |
---|---|
snsTopicName | Name of the SNS topic |
Create an SQS Queue. Properly handles localstack queue URLs
Kind: inner method of SQS
Send a message to AWS SQS
Kind: inner method of SQS
Receives SQS messages from a given queue. The number of messages received can be set and the timeout is also adjustable.
Kind: inner method of SQS
Bare check for SQS message Shape
Kind: inner method of SQS
Extract SQS message body
Kind: inner method of SQS
Delete a given SQS message from a given queue.
Kind: inner method of SQS
Test if an SQS queue exists
Kind: inner method of SQS
-
StepFunctions
-
static
-
.describeExecution(params) ⇒
Promise.<DescribeExecutionOutput>
-
.describeStateMachine(params) ⇒
Promise.<DescribeStateMachineOutput>
-
.getExecutionHistory(params) ⇒
Promise.<GetExecutionHistoryOutput>
-
.listExecutions(params) ⇒
Promise.<Object>
-
.describeExecution(params) ⇒
-
inner
-
~executionExists(executionArn) ⇒
Promise.<boolean>
-
~executionExists(executionArn) ⇒
-
static
Call StepFunctions DescribeExecution
See StepFunctions.describeExecution()
for descriptions of params
and the return data.
If a ThrottlingException is received, this function will retry using an exponential backoff.
Kind: static method of StepFunctions
Param | Type |
---|---|
params | DescribeExecutionInput |
Call StepFunctions DescribeStateMachine
See StepFunctions.describeStateMachine()
for descriptions of params
and the return data.
If a ThrottlingException is received, this function will retry using an exponential backoff.
Kind: static method of StepFunctions
Param | Type |
---|---|
params | DescribeStateMachineInput |
Call StepFunctions GetExecutionHistory
See StepFunctions.getExecutionHistory()
for descriptions of params
and the return data.
If a ThrottlingException is received, this function will retry using an exponential backoff.
Kind: static method of StepFunctions
Param | Type |
---|---|
params | GetExecutionHistoryInput |
Call StepFunctions ListExecutions
See StepFunctions.listExecutions()
for descriptions of params
and the return data.
If a ThrottlingException is received, this function will retry using an exponential backoff.
Kind: static method of StepFunctions
Param | Type |
---|---|
params | Object |
Check if a Step Function Execution exists
If a ThrottlingException is received, this function will retry using an exponential backoff.
Kind: inner method of StepFunctions
Param | Type | Description |
---|---|---|
executionArn | string |
the ARN of the Step Function Execution to check for |
Class to efficiently search all of the items in a DynamoDB table, without loading them all into memory at once. Handles paging.
Kind: global class
Drain all values from the searchQueue, and return to the user. Warning: This can be very memory intensive.
Kind: instance method of DynamoDbSearchQueue
Returns: Promise.<Array>
- array of search results.
View the next item in the queue
This does not remove the object from the queue. When there are no more items in the queue, returns 'null'.
Kind: instance method of DynamoDbSearchQueue
Returns: Promise.<Object>
- an item from the DynamoDB table
Remove the next item from the queue
When there are no more items in the queue, returns 'null'.
Kind: instance method of DynamoDbSearchQueue
Returns: Promise.<Object>
- an item from the DynamoDB table
Class to efficiently list all of the objects in an S3 bucket, without loading them all into memory at once. Handles paging of listS3ObjectsV2 requests.
Kind: global class
View the next item in the queue
This does not remove the object from the queue. When there are no more items in the queue, returns 'null'.
Kind: instance method of S3ListObjectsV2Queue
Returns: Promise.<Object>
- an S3 object description
Remove the next item from the queue
When there are no more items in the queue, returns 'null'.
Kind: instance method of S3ListObjectsV2Queue
Returns: Promise.<Object>
- an S3 object description
Class to use when interacting with S3
Kind: global class
-
S3ObjectStore
-
.signGetObject(objectUrl, [options], [queryParams], presignOptions) ⇒
Promise.<string>
-
.signHeadObject(objectUrl, [options], [queryParams], presignOptions) ⇒
Promise.<string>
-
.signGetObject(objectUrl, [options], [queryParams], presignOptions) ⇒
Returns an HTTPS URL that can be used to perform a GET on the given object store URL
Kind: instance method of S3ObjectStore
Returns: Promise.<string>
- a signed URL
Throws:
- TypeError - if the URL is not a recognized protocol or cannot be parsed
Param | Type | Description |
---|---|---|
objectUrl | string |
the URL of the object to sign |
[options] | string |
options to pass to S3.getObject |
[queryParams] | string |
a mapping of parameter key/values to put in the URL |
presignOptions | RequestPresigningArguments |
presignOptions |
s3ObjectStore.signHeadObject(objectUrl, [options], [queryParams], presignOptions) ⇒ Promise.<string>
Returns an HTTPS URL that can be used to perform a HEAD on the given object store URL
Kind: instance method of S3ObjectStore
Returns: Promise.<string>
- a signed URL
Throws:
- TypeError - if the URL is not a recognized protocol or cannot be parsed
Param | Type | Description |
---|---|---|
objectUrl | string |
the URL of the object to sign |
[options] | string |
options to pass to S3.getObject |
[queryParams] | string |
a mapping of parameter key/values to put in the URL |
presignOptions | RequestPresigningArguments |
presignOptions |
Create a KMS key
See https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/KMS.html#createKey-property for allowed params and return value.
Kind: global function
Param | Type |
---|---|
params | Object |
Encrypt a string using KMS
Kind: global function
Returns: Promise.<string>
- the Base 64 encoding of the encrypted value
Param | Type | Description |
---|---|---|
KeyId | string |
the KMS key to use for encryption |
Plaintext | string |
the string to be encrypted |
Decrypt a KMS-encrypted string, Base 64 encoded
Kind: global function
Returns: string
- the plaintext
Param | Type | Description |
---|---|---|
ciphertext | string |
a KMS-encrypted value, Base 64 encoded |
Cumulus is a cloud-based data ingest, archive, distribution and management prototype for NASA's future Earth science data streams.
To make a contribution, please see our contributing guidelines.
Generated automatically using npm run build-docs