serverless-offline-kinesis-streams
A Serverless Plugin that creates local Kinesis streams and then triggers your Serverless Lambda Functions upon receiving new records.
It works depends on serverless-offline
.
It works with:
serverless-webpack
serverless-typescript
serverless-parcel
Serverless Offline Kinesis Streams also works with multiple Kinesis streams in the same Serverless file. It respects batchWindow and batchSize. You can even listen to the same Kinesis Stream with multiple Lambda functions at once with different batchWindow and batchSize values.
Installation
Add the plugin to your project
yarn add -D serverless-offline-kinesis-streams
Then, add serverless-offline-kinesis-streams
to your plugins. Example:
plugins:
- serverless-layers
- serverless-plugin-typescript
- serverless-offline-kinesis-streams
- serverless-offline
Configuration
Simple setup
custom:
offlineKinesisStreams:
port: 4567 # Optional; 4567 is the default port. The plugin launches a local Kinesis instance on this port
region: local # Optional; local is the default
streams: # Required; Define all streams and their streamNames with number of shards.
- streamName: myFirstKinesisStream
shards: 1
functions:
myFirstKinesisStreamHandler:
handler: src/myFirstKinesisStreamHandler.handler
events:
- stream:
enabled: true
type: kinesis
arn: arn:aws:kinesis:${self:custom.region}:*:stream/myFirstKinesisStream # Same stream name from above.
Example with multiple stream:
custom:
offlineKinesisStreams:
port: 4567
region: local
streams:
- streamName: myFirstKinesisStream
shards: 1
- streamName: mySecondKinesisStream
shards: 1
functions:
myFirstKinesisStreamHandler:
handler: src/myFirstKinesisStreamHandler.handler
events:
- stream:
enabled: true
type: kinesis
arn: arn:aws:kinesis:${self:custom.region}:*:stream/myFirstKinesisStream
mySecondKinesisStreamHandler:
handler: src/mySecondKinesisStreamHandler.handler
events:
- stream:
enabled: true
type: kinesis
arn: arn:aws:kinesis:${self:custom.region}:*:stream/mySecondKinesisStream
Example with batchWindow and batchSize:
The example below will trigger both Lambda functions when the single Kinesis Stream has records. They are independently monitored and their batchWindow & batchSize events are individual respected.
So, when records are put into the Kinesis stream, the myFirstKinesisStreamHandler
will most likely pick them up first (10 records at a time). And then, every 60 seconds, alsoListensToFirstStream
will grab 1 record. (This example is contrived. You'd never want such a high batchWindow and low batchSize.)
custom:
offlineKinesisStreams:
port: 4567
region: local
streams:
- streamName: myFirstKinesisStream
shards: 1
functions:
myFirstKinesisStreamHandler:
handler: src/myFirstKinesisStreamHandler.handler
events:
- stream:
enabled: true
type: kinesis
arn: arn:aws:kinesis:${self:custom.region}:*:stream/myFirstKinesisStream
alsoListensToFirstStream:
handler: src/mySecondKinesisStreamHandler.handler
events:
- stream:
enabled: true
type: kinesis
batchWindow: 60 # Get records every 60 seconds
batchSize: 1 # Get one record at a time
arn: arn:aws:kinesis:${self:custom.region}:*:stream/myFirstKinesisStream
Thanks
Features were extended off of this implementation of a local Kinesis
Thanks to mhart for creating kinesalite