sqs-ecs-jobs
A worker that consumes jobs from SQS and runs them on predefined ECS tasks.
Has the following qualities:
- Jobs run at least once (quality of SQS)
- Each task runs within its own docker container
- Supports arbitrary data to be passed
- Simple JSON protocol that can be used anywhere SQS can be called from
It does not:
- Configure ECS task definitions for you
- Handle scheduling (please see Scheduling)
Running
You'll need to start two processes:
npm run consume
Which will consume jobs in the SQS queue in the format:
{
"name": "SomeECSTaskDefinitionName",
"payload": {
"arbitrary": 123,
"data": "here"
},
"maxRetries": 1
}
Where payload
are maxRetries
are optional (will retry infinitely by default). You'll also need to run:
npm run watch
This process will ensure that jobs are run successfully or, if they fail, will ensure they are scheduled to be rerun or moved in to a Dead Letter Queue of your choosing.
Docker
To run in docker:
docker run -i -t nearst/sqs-ecs-jobs npm run consume
docker run -i -t nearst/sqs-ecs-jobs npm run watch
You can pass either command line arguments or provide an environment to docker.
Config
Node-config is used for a flexible config:
{
"queues": {
"main": SQS main queue URL,
"processing": SQS processing queue URL,
"deadLetter": SQS Dead Letter Queue URL
},
"retryBase": Retry backoff multiplier (default 60),
"maxRetries": Total attempts to get job to run (default 6),
"clusterName": Cluster to run ECS tasks on
}
Can be configured using command line overrides or environmental variables:
{
"queues": {
"main": "QUEUE_URL",
"processing": "PROCESSING_QUEUE_URL",
"deadLetter": "DEAD_LETTER_QUEUE_URL"
},
"retryBase": "RETRY_BASE",
"maxRetries": "MAX_RETRIES",
"clusterName": "CLUSTER_NAME"
}
AWS config
AWS credentials need to be provided in any of their standard ways, but is probably easiest to provide using AWS environmental variables.
Scheduling
Scheduling is not supported natively, however you can utilise Cloudwatch scheduled events, SNS notifications and SQS SNS subscriptions to achieve this without having to run a separate cron daemon. This will have the guarantees of running that Cloudwatch scheduled events, SNS and SQS SNS subscriptions have.
Roadmap
- Support aggregation of similar tasks within a given timeframe (addition to job's JSON format)
- Better logging and error notifications
- Simple support interface to augment AWS console
Contributing
Feel free to contribute by forking and providing a Pull Request. Feel free to raise an issue if you don't know how to tackle a particular problem.