Nostalgic Punchcard Missing

    nxus-pipeliner

    4.0.0 • Public • Published

    nxus-pipeliner

    Pipeliner Module

    Build Status

    A framework for creating and running data pipelines. Data pipelines have stages, which are made of an arbitrary number of tasks. Stages and tasks are run in serial: once a task completes, the next task in the pipeline is executed.
    A task may complete synchronously, or asynchronously through use of a promise.

    Pipelines take a data object as input, and each task operates on the object in some way.

    For an example of the Pipeliner in action, checkout the nxus-static-site module.

    Installation

    > npm install nxus-pipeliner --save
    

    Usage

    Step 1: Define a pipeline

    import {pipeliner} from 'nxus-pipeline'
    
    pipeliner.pipeline('my-pipeline')
    

    Step 2: Define tasks

    A task is a javascript function that accepts any objects passed into the pipeline when it is run. Tasks are run serially in FIFO order.

    let myTask = (word) => {
      word.toUpperCase();
    }
    
    app.get('pipeliner').task('my-pipeline', myTask)
    

    Step 3: Run a pipeline

    Once all the tasks for a pipeline have been defined, the last step is to run the pipeline.

    app.get('pipeliner').run('my-pipeline', someData)
    

    API


    Pipeliner

    Extends NxusModule

    Examples

    let myTask = (data) => {
      data.word.toUpperCase()
    }
    let data = {word: 'hello'}
     
    let pipeliner = app.get('pipeliner')
     
    pipeliner.pipeline('capitalize')
    pipeliner.run('capitalize', data).then(() => {
      console.log('data') // {word: 'HELLO'}
    })

    pipeline

    Create a new pipeline configured with three stages: 'collect', 'process', and 'generate'. Does nothing if the named pipeline already exists.

    Parameters

    • pipeline string The name of the pipeline to create

    task

    Defines a task for a pipeline and a stage. Creates the pipeline if it does not already exist; adds the stage if it does not already exist. If multiple tasks are defined for a stage, they are run in the order defined.

    Parameters

    • pipeline string The name of the pipeline
    • job function A function which accepts data

    getPipelines

    Returns all pipelines which have been defined.

    Returns object A hash of the pipelines.

    getPipeline

    Returns a specific pipeline.

    Parameters

    • pipeline string The name of a pipeline to return.

    Returns object The pipeline object.

    run

    Runs the specified pipeline, passing the arguments to each task.

    Parameters

    • pipeline string The name of the pipeline to run
    • args ...object Arguments to pass to the pipeline tasks

    Returns Promise A promise that is fulfilled when the pipeline completes; it is rejected if any task in the pipeline fails (throws an error or returns a promise that is rejected)

    Install

    npm i nxus-pipeliner

    DownloadsWeekly Downloads

    5

    Version

    4.0.0

    License

    MIT

    Last publish

    Collaborators

    • davidkellerman
    • loppear
    • mjreich
    • scottmaxson