nxus-pipeliner
Pipeliner Module
A framework for creating and running data pipelines. Data pipelines have stages, which are made of an arbitrary number of tasks. Stages and tasks are run in serial: once a task completes, the next task in the pipeline is executed.
A task may complete synchronously, or asynchronously through use of a promise.
Pipelines take a data object as input, and each task operates on the object in some way.
For an example of the Pipeliner in action, checkout the nxus-static-site module.
Installation
> npm install nxus-pipeliner --save
Usage
Step 1: Define a pipeline
import {pipeliner} from 'nxus-pipeline'
pipeliner.pipeline('my-pipeline')
Step 2: Define tasks
A task is a javascript function that accepts any objects passed into the pipeline when it is run. Tasks are run serially in FIFO order.
let myTask = (word) => {
word.toUpperCase();
}
app.get('pipeliner').task('my-pipeline', myTask)
Step 3: Run a pipeline
Once all the tasks for a pipeline have been defined, the last step is to run the pipeline.
app.get('pipeliner').run('my-pipeline', someData)
API
Pipeliner
Extends NxusModule
Examples
let { dataword}let data = word: 'hello' let pipeliner = app pipelinerpipeliner
pipeline
Create a new pipeline configured with three stages: 'collect', 'process', and 'generate'. Does nothing if the named pipeline already exists.
Parameters
pipeline
string The name of the pipeline to create
task
Defines a task for a pipeline and a stage. Creates the pipeline if it does not already exist; adds the stage if it does not already exist. If multiple tasks are defined for a stage, they are run in the order defined.
Parameters
getPipelines
Returns all pipelines which have been defined.
Returns object A hash of the pipelines.
getPipeline
Returns a specific pipeline.
Parameters
pipeline
string The name of a pipeline to return.
Returns object The pipeline object.
run
Runs the specified pipeline, passing the arguments to each task.
Parameters
pipeline
string The name of the pipeline to runargs
...object Arguments to pass to the pipeline tasks
Returns Promise A promise that is fulfilled when the pipeline completes; it is rejected if any task in the pipeline fails (throws an error or returns a promise that is rejected)