delaysqs
A simple node module to extend the SQS delay over the 15 minutes hard-limit imposed by Amazon. This is done by effectively
reading and relaying the message at 15-minute intervals until the remaining delay time is below the 15-minute limit.
In other words, if at 01:00:00 I want to send a message to the Queue with a delay of 1 hr 50 minutes, the delaysqs library
will first send the message with a delay of 900 seconds, and it will repeat at 01:15:00, 01:30:00 and so on until 02:45:00,
at that point it will send the message once more to the queue but with a DelaySeconds
attribute of 300 seconds, so that
the message will appear in queue at 02:50:00.
The relaying process uses a custom MessageAttribute
named deliveryTimestamp
which contains the UNIX timestamp at
which the message should finally be consumed. Specifying the explicitly the timestamp makes it such that even if the queue is
not polled for periods of time before the delivery time, the message still has a chance of being delivered in time.
Reliability
As SQS is an "At-least-once" deliver system, there is always a slight chance of the message being delivered to two message consumers. At each "relay" step therefore we have a possibility of the message being duplicated, with each of the copy of the message continuing its own independent existence. Therefore it is very important that the processors that will finally handle the message check whether the action indicated by the message has already been carried out or not.
Costs
Polling, sending and receiving messages to/from SQS has an associated cost. Delaysqs uses long polling to minimize the
roundtrips from the server in case of no traffic. The minimum costs for monitoring an SQS with long polling (prices are
region specific and subject to change, these one reported here are for region us-west-1
and are correct as of Apr 12th,
2016)
4320 request/days * $0.00000050 $/request = 0.0216 $/day (assuming you have already consumerd your first 1 milion requests) + data transfer fees
Due to the possibility of duplicated requests, it is advisable to use delayed messages just as triggers for specific actions, carrying only the id of the action (and not the entire action payload), making the data transfer costs a non issue.
Error handling
The aim of this library is to continue processing the messages in the queue as much as possible, despite of errors occuring. Errors
are forwarded verbatim to the error callback function. It is up to the client of this library to analyze the error and decide whether
it is the cast to invoke stopPolling
and interrupt the processing of messages.
Installation and usage
This package tries to be as little intrusive as possible. For this reason the AWS SDK dependency is declared as peer dependency,
and the methods expect to receive an instance of an AWS.SQS
object, thus leaving all the AWS configuration details to
the caller (chances are that if you want to use this module you are already using the AWS SDK and already have the credentials
set up).
To install do
npm install delaysqs
Then require the module in your code
var delaysqs = require('delaysqs');
The main method is the startPolling(sqs, queueUrl, messageCallback, errorCallback)
method which takes 4 parameters:
sqs
is a configured instance ofAWS.SQS
queueUrl
is aString
containing the URL of the SQS queue used to have delayed messages. The URL can be either found in the AWS Console, or queried from the AWS SDK usingAWS.SQS.getQueueUrl
methodmessageCallback
is a function that will be invoked with the body of a message when the deliveryTimestamp for that message has been reachederrorCallback
is a function that will receive errors generated by the library while accessing the SQS queue
Example
var AWS = ; var DelaySqs = ; sqs = region: "<my region>" accessKeyId: "<my access key id>" secretAccessKey: "<my secret access key>"; var queueUrl = 'https://sqs.region.amazonaws.com/account/queue_name' var delayer = ; delayer;