This is a highly configurable, abstract NodeJS framework for Amazon's Simple Workflow Service (SWF). It allows you to configure your decisions through a combination of Pipelines and Tasks (see below), the end result being complete separation between your decider, your activity poller (worker), and the actual activities.
Table of Contents
Preamble
Disclaimer
This project is still in active development. As such, the API is subject to change. We will stick to the typical major/minor/patch versioning system, in which any breaking API changes will bump the major version.
Installation
npm install swiffer-framework
Support
Please create an issue if you believe you have found a bug or are having trouble. If you're able, please create a failing test for the bug you find so we can easily address it.
Contributions
Contributions are welcome. Please follow the guideines in .jshintrc
and use JSBeautify before pushing. Also, make sure your code is tested with jasmine-node
How This Works In A Nutshell
- This package exports
decider
andworker
. - You will use these to create deciders and workers, both of which will be connected with AWS SWF.
- You will run your deciders and workers as NodeJS apps anywhere, for example as microservices on your own infrastructure.
- SWF, when given input, will call your deciders to decide what to do and then call your workers to actually do it.
Deciders
Deciders are configured via Pipelines and Tasks.
Basic Usage
var swf = AWS = ; var swfClient = region: 'us-east-1' accessKeyId: '{ACCESS-KEY}' secretAccessKey: '{SECRET-KEY}'; var decider = PIPELINE swfClient domain: 'SWF Domain' identity: 'Decider ID' taskList: name: 'Task List' ; decider;decider; deciderstart;
Pipelines
A pipeline is a collection of one or more tasks or child pipelines. Once all tasks in the main pipeline passed to the Decider have completed successfully, the workflow is marked as complete.
Pipeline Types
There are three types of pipeline:
Series Pipeline
This pipeline executes all of its tasks in sequential order.
The below example does the following:
- Schedule the
"My Cool Activity"
activity - Once it finishes successfully, start the
"My Timer"
timer - Timer fires after 10 seconds - schedule the
"Final Activity"
activity
var swf = ;var pipelines = swfdeciderPipelines;var Task = swfdeciderTask; var myPipe = type:'activity' name:'My Cool Activity' activityVersion:'0.1' type:'timer' name:'My Timer' delay:10 type:'activity' name:'Final Activity' activityVersion:'0.3' ;
Parallel Pipeline
This executes all of its tasks at the same time. It is most useful as a child in a Series pipeline.
The below example does the following:
- Schedule the
"My Cool Activity"
activity - Once it finishes successfully, start both activites in the child parallel pipeline
- Schedules the
"Final Activity"
activity after both activities in the child parallel pipeline finish successfully
var swf = ;var pipelines = swfdeciderPipelines;var Task = swfdeciderTask; var myPipe = type:'activity' name:'My Cool Activity' activityVersion:'0.1' type:'activity' name:'Parallel Task 1' activityVersion:'0.1' type:'activity' name:'Parallel Task 2' activityVersion:'0.1' type:'activity' name:'Final Activity' activityVersion:'0.3' ;
Continuous Pipeline
A Continuous pipeline is a Series pipeline that starts over if all of its tasks have completed successfully. It will keep running indefinitely unless you tell it to stop with a Signal.
The below example does the following:
- Schedule the
"My Cool Activity"
activity - Once it finishes, wait 60 seconds with the
"My Timer"
timer - Schedule the
"My Cool Activity"
activity once again (back to #1) - Receives a
"StopMyActivity"
signal and breaks its loop
var swf = ;var pipelines = swfdeciderPipelines;var Task = swfdeciderTask; var myPipe = type:'activity' name:'My Cool Activity' activityVersion:'0.1' type:'timer' name:'My Timer' delay:60 ;
Signaling Pipelines
All pipelines can react to a signal and start either a single task or a child pipeline.
For example, if you want a Series pipeline to wait an hour if it receives the "WaitOneHour"
signal, you would do the following:
var myPipe = type:'activity' name:'Activity1' activityVersion:'0.1' type:'activity' name:'Activity2' activityVersion:'0.1' type:'activity' name:'Activity3' activityVersion:'0.1' ;
Important Notes
- Different pipeline types respond to signals a bit differently. See below.
- If a workflow has already received a signal one or more times, and receives that signal again, and the task/pipeline triggered by the previous signal has not yet completed, then the most recent signal will be ignored.
Signaling Series Pipelines
When a Series pipeline receives a signal, it will not execute any normal task in the pipe until the signal has been handled. If it receives multiple signals at the same time, those signals will be handled in parallel, but the Series pipeline will not continue its normal execution until all signals have been handled.
Signaling Parallel Pipelines
When a Parallel pipeline receives a signal, it will both respond to the signal AND continue its normal execution.
Signaling Continuous Pipelines
Continuous pipelines react the same way as Series pipelines do to signals. However, you can also set a signal on which to break the continuous loop. See examples for Continuous pipeline configuration above.
Tasks
Tasks are the elements inside of a Pipeline and the next step(s) in the workflow.
Currently there are four task types supported: Activities, Timers, Lambda Functions, and Markers.
IMPORTANT: no two tasks in the same "main" pipeline (IE, the pipeline passed to the Decider
instance) can have the same name. The name is how swiffer determines the decisions for each decision task.
Activity Tasks
Activity tasks trigger the corresponding worker. They are made up of a name
, version
, and an optional Retry Strategy. See above examples for how to create an activity task.
Task Input
You can give either static or dynamic input (or no input) to your activity task. Simply define the input
property in the task configuration:
var task = type:'activity' name:'My Cool Activity' activityVersion:'0.1' input: foo:'bar' ;
Dynamic Task Input
From previous activity
To modify the input based on the results of the most recently completed "My Initial Activity" activity, do the following (the "$" is used to designate that it is a dynamic value):
var task = type:'activity' name:'My Cool Activity' input: foo:'$My Initial Activity.someProperty.myFoo' ;
Assuming the result of the "My Initial Activity" activity was something like:
...then the input passed to the "My Cool Activity" activity would be:
From workflow execution
To modify the input based on the initial input passed to the workflow, do the same as above, but substitute $$Workflow
for the key:
var task = type:'activity' name:'My Cool Activity' input: foo:'$$Workflow.someProperty.myFoo' ;
Timeout Configuration
SWF allows you to configure four different timeouts: scheduleToStartTimeout
, scheduleToCloseTimeout
, startToCloseTimeout
, and heartbeatTimeout
. You can provide these timeouts via your task configuration like so:
var task = type:'activity' name:'My Cool Activity' timeouts: scheduleToStart:30 scheduleToClose:300 startToClose:360 heartbeat:30 ;
They default to 60
, 360
, 300
, and 60
, respectively, if you do not set them.
Retry Strategies
Retry strategies are used to determine when and how to retry an activity that has failed or timed out.
Exponential Backoff
With an Exponential Backoff retry strategy, every failed execution of an activity will result in an exponentially greater timer before the next scheduled activity.
For example, the following task will be retried up to 5 times, with the backoff times being 2, 4, 8, and 16 seconds.
var task = type:'activity' name:'My Cool Activity' activityVersion:'0.1' retryStrategy:2 5;
Constant Backoff
Constant backoff strategies cause the decider to wait a constant number of seconds before retrying the activity.
For example, the following task will be retried up to 10 times before failing the workflow, with 30 seconds between each attempted execution:
var task = type:'activity' name:'My Cool Activity' activityVersion:'0.1' retryStrategy:30 10;
Immediate
Immediate retry strategies will retry the failed activity immediately.
The following task will be retried up to 5 times, with the retry happening immediately after the failed event:
var task = type:'activity' name:'My Cool Activity' activityVersion:'0.1' retryStrategy:5;
None
The "None" retry strategy will cause a fatal error after one activity execution failure. It is used by default so you should never have to access it directly.
Timer Tasks
Timer tasks tell SWF to wait a designated number of seconds before moving to the next task. See above examples.
Dynamic Timer Delays
The timer delay can be determined by the result of a previous activity. For example, to set the timer based on the results of the most recently completed "My Cool Activity" activity, do the following (the "$" is used to designate that it is a dynamic value):
var task = type:'timer' name:'My Timer' delay:'$My Cool Activity.someProperty.timerDelay';
Assuming the result of the "My Cool Activity" activity was something like:
...then the delay would be 45 seconds.
Lambda Function Tasks
Lambda function tasks trigger an AWS lambda function. They invoke the lambda function specified by functionName
, a distinct name
, and an optional Retry Strategy.
Lambda Task Input
You can give either static or dynamic input (or no input) to your lambda function task. Simply define the input
property in the task configuration, just like you would with the activity task input:
var task = type: 'lambda' name: 'Neato Lambda Function Activity' functionName: 'MyLambdaFunction' input: foo: 'bar' ;
Marker Tasks
Marker tasks record information in the workflow, which can be used for debugging purposes or for capturing the workflow state.
Marker Task Details
You can give either static or dynamic details (or no details) to your marker task. Simply define the details
property in the task configuration, just like you would with the activity task input:
var task = type: 'marker' name: 'Workflow Delay Time' details: '$My Cool Activity.someProperty.timerDelay';
Child Workflow Tasks
You can trigger a child workflow from within a parent workflow. For all intents and purposes, the child workflow is the same as an activity task (see above). It will be considered "done" when the entire child workflow has finished, and uses retry strategies in the same manner that activity tasks do. Note that in case you want to trigger the same child workflow in multiple parts of your pipeline, the name
property (which must be unique so swiffer can identify related tasks) is separate from the workflowName
property, which is the name of the workflow in SWF.
Example:
var task = type:'childWorkflow' name:'MyChildWorkflow' workflowName:'name of my child workflow' workflowVersion:'1.0';
Activity Workers
Activity workers are the opposite side of the equation from the Deciders. They perform the activities scheduled by their corresponding Decider.
Basic Usage
var swf = AWS = ; var swfClient = region: 'us-east-1' accessKeyId: '{ACCESS-KEY}' secretAccessKey: '{SECRET-KEY}'; var worker = swfClient domain: 'Testing' identity: 'test-worker' taskList: name: 'MyWorkflow' ; worker; worker; workerstart;
Worker Types
Inline
The inline worker is simply a Javascript function that gets bound to the Activity
object. Inline workers can call this.heartbeat()
to register an SWF heartbeat, this.error()
to signal an activity failure, and this.done()
to signal that the activity completed successfully. See above for an example.
Note that heartbeat()
, error()
, and done()
are wrapped in Futures
from laverdet/node-fibers. This allows you to call them and wait for an acknowledgement from SWF without needing messy asynchronous code.
AWS Lambda
This will allows you to defer the worker code to an AWS Lambda function. You must provide your own instance of AWS.Lambda
from the aws-sdk library. For example, to call a Lambda function called "MyLambdaFunction", that responds to the "LambdaActivity" activity, you can do the following:
var myLambdaClient = region: 'us-east-1' accessKeyId: '{ACCESS-KEY}' secretAccessKey: '{SECRET-KEY}'; worker;
Child Process
Coming soon. Will allow you to spawn an arbitrary shell process.