Failure injection for AWS Lambda - failure-lambda
Description
failure-lambda
is a small Node module for injecting failure into AWS Lambda (https://aws.amazon.com/lambda). It offers a simple failure injection wrapper for your Lambda handler where you then can choose to inject failure by setting the failureMode
to latency
, exception
, denylist
, diskspace
or statuscode
. You control your failure injection using SSM Parameter Store or AWS AppConfig.
How to install with parameter in SSM Parameter Store
- Install
failure-lambda
module using NPM.
npm install failure-lambda
- Add the module to your Lambda function code.
const failureLambda = require('failure-lambda')
- Wrap your handler.
exports.handler = failureLambda(async (event, context) => {
...
})
- Create a parameter in SSM Parameter Store.
{"isEnabled": false, "failureMode": "latency", "rate": 1, "minLatency": 100, "maxLatency": 400, "exceptionMsg": "Exception message!", "statusCode": 404, "diskSpace": 100, "denylist": ["s3.*.amazonaws.com", "dynamodb.*.amazonaws.com"]}
aws ssm put-parameter --region eu-west-1 --name failureLambdaConfig --type String --overwrite --value "{\"isEnabled\": false, \"failureMode\": \"latency\", \"rate\": 1, \"minLatency\": 100, \"maxLatency\": 400, \"exceptionMsg\": \"Exception message!\", \"statusCode\": 404, \"diskSpace\": 100, \"denylist\": [\"s3.*.amazonaws.com\", \"dynamodb.*.amazonaws.com\"]}"
- Add an environment variable to your Lambda function with the key FAILURE_INJECTION_PARAM and the value set to the name of your parameter in SSM Parameter Store.
- Add permissions to the parameter for your Lambda function.
- Try it out!
How to install with hosted configuration in AWS AppConfig
- Install
failure-lambda
module using NPM.
npm install failure-lambda
- Add the module to your Lambda function code.
const failureLambda = require('failure-lambda')
- Wrap your handler.
exports.handler = failureLambda(async (event, context) => {
...
})
- Create Application, Environment, Configuration Profile, and Hosted Configuration in AppConfig console.
- Deploy a version of the configuration.
- Add the AWS AppConfig layer for Lambda extensions to your Lambda function. See details.
- Add environment variables to your Lambda function.
FAILURE_APPCONFIG_APPLICATION: YOUR APPCONFIG APPLICATION
FAILURE_APPCONFIG_ENVIRONMENT: YOUR APPCONFIG ENVIRONMENT
FAILURE_APPCONFIG_CONFIGURATION: YOUR APPCONFIG CONFIGURATION PROFILE
- Add permissions to the AppConfig Application, Environment, and Configuration Profile for your Lambda function.
- Try it out!
Usage
Edit the values of your parameter in SSM Parameter Store or hosted configuration in AWS AppConfig to use the failure injection module.
-
isEnabled: true
means that failure is injected into your Lambda function. -
isEnabled: false
means that the failure injection module is disabled and no failure is injected. -
failureMode
selects which failure you want to inject. The options arelatency
,exception
,denylist
,diskspace
orstatuscode
as explained below. -
rate
controls the rate of failure. 1 means that failure is injected on all invocations and 0.5 that failure is injected on about half of all invocations. -
minLatency
andmaxLatency
is the span of latency in milliseconds injected into your function whenfailureMode
is set tolatency
. -
exceptionMsg
is the message thrown with the exception created whenfailureMode
is set toexception
. -
statusCode
is the status code returned by your function whenfailureMode
is set tostatuscode
. -
diskSpace
is size in MB of the file created in tmp whenfailureMode
is set todiskspace
. -
denylist
is an array of regular expressions, if a connection is made to a host matching one of the regular expressions it will be blocked.
Example
In the subfolder example
is a sample application which will install an AWS Lambda function, an Amazon DynamoDB table, and a parameter in SSM Parameter Store. You can install it using AWS SAM, AWS CDK, or Serverless Framework.
AWS SAM
cd example/sam
npm install
sam build
sam deploy --guided
AWS CDK
cd example/cdk
npm install
cdk deploy
Serverless Framework
cd example/sls
npm install
sls deploy
Notes
Inspired by Yan Cui's articles on latency injection for AWS Lambda (https://hackernoon.com/chaos-engineering-and-aws-lambda-latency-injection-ddeb4ff8d983) and Adrian Hornsby's chaos injection library for Python (https://github.com/adhorn/aws-lambda-chaos-injection/).
Changelog
2022-02-14 v0.4.4
- Switch to node-fetch@2.
2022-02-14 v0.4.3
- Updated dependencies.
2021-03-16 v0.4.2
- Puts the mitm object in the library global namespace so that it persists across function invocations.
- Syntax formatting.
2020-10-26 v0.4.1
- Made AppConfig Lambda extension port configurable using environment variable.
2020-10-25 v0.4.0
- Added optional support for AWS AppConfig, allowing to validate failure configuration, deploy configuration using gradual or non-gradual deploy strategy, monitor deployed configuration with automatical rollback if CloudWatch Alarms is configured, and caching of configuration.
- Hardcoded default configuration with
isEnabled: false
, to use if issues loading configuration from Parameter Store or AppConfig.
2020-10-21 v0.3.1
- Change mitm mode back to connect to fix issue with all connections being blocked.
2020-08-24 v0.3.0
- Changed mitm mode from connect to connection for quicker enable/disable of failure injection.
- Renamed block list failure injection to denylist (breaking change for that failure mode).
- Updated dependencies.
2020-02-17 v0.2.0
- Added block list failure.
- Updated example application to store file in S3 and item in DynamoDB.
2020-02-13 v0.1.1
- Fixed issue with exception injection not throwing the exception.
2019-12-30 v0.1.0
- Added disk space failure.
- Updated example application to store example file in tmp.
2019-12-23 v0.0.1
- Initial release
Contributors
Gunnar Grosch - GitHub | Twitter | LinkedIn
Jason Barto - GitHub | Twitter | LinkedIn
License
This code is made available under the MIT-0 license. See the LICENSE file.