Have ideas to improve npm?Join in the discussion! »

    dashflow

    1.4.0 • Public • Published

    Dashflow

    logo

    npm

    Netlify Status

    Automate local development tasks. See It In Action

    What can Dashflow do for you

    • Run many commands in background and collect their stdout/stderr
    • Automatically trigger new commands when output from those commands matches certain pattern
    • Serve a web dashboard page to visualize the status of those commands
    • Replace Makefile using a easy readable dashflow.yml YAML file

    run processes web dashboard

    Installation

    Make sure you have installed NodeJS.

    # install the package
    npm install -g dashflow
    

    Usage

    # print help
    $ dashflow --help
    
    # by default read from current folder dashflow.yml
    $ dashflow
    
    # support multi config files, config will be merged
    $ dashflow -c service1/dashflow.yml -c service2/dashflow.yml
    
    # custom http port
    $ dashflow -p 9528
    
    # use dashflow shell
    dashflow-shell~$ help
    
      Commands:
    
        help [command...]          Provides help for a given command.
        exit                       Exits application.
        events [pattern]           Dump events
        emit <event>               Emit a new event
        tail <streamOrWorkflowId>  Tail to a stream or workflow
        attach <streamID>          Attach to a stream
        restart <streamID>         Restart a stream
        env                        Print environment variables
    
    

    Concepts

    Dashflow is modeled around some key concepts.

    Please take some time to get familiar with those concepts so you can use the tool more effectively.

    Event

    An event is just a plain string, associated with its creation time. Usually events will have certain prefixes so that they can easily be distinguished from different sources of events.

    For example, here are some sample events:

    stream:watch-src:watch:add:src/new_file
    stream:watch-test:watch:addDir:src/test/
    command:lint:shell:yarn --silent lint:stdout:lint passed
    workflow:initial-lint:command:lint:shell:yarn --silent lint:stdout:lint passed
    

    In dashflow, all events store in memory (which means all events will be lost once the dashflow process exited).

    The default view of events are ordered by their creation time, newer to older.

    Command

    A command is an alias for a shell function, commands can be composed together, executing one by one or concurrently.

    The following is an example of command configurations:

    commands:
      // the following concise way of declaring a command is a shortcut for the formal one
      // which is demonstrated with "test" command
      lint: yarn --silent lint
      test:
        shell: { cmd: yarn --silent test --no-color --verbose }
      lint-then-test:
        // run those commands one by one
        serial:
          - lint
          - test
      lint-and-test:
        // run those commands concurrently
        parallel:
          - lint
          - test
      list-web-files:
        // by default we assume shell should executes from the same folder
        // where dashflow.yml file is located
        // but we can specify a sub folder by providing 'cwd' argument to shell
        shell: { cmd: ls, cwd: web }
    

    Stream

    A stream is something that will emit events, it can be either a long running process like a web server, or a one off command like running test.

    The following is an example of stream configurations:

    streams:
      watch-lib:
        // listen on current folder for file changes and report evens on matching files
        watch: { glob: "lib/**/*.js" }
      watch-test:
        // can specify a sub folder by providing 'cwd' argument to watch
        watch: { glob: "**/*.js", cwd: test }
      python:
        // can execute a shell command
        shell: { cmd: python -i }
      vim:
        // can specify a sub folder by providing 'cwd' argument to shell
        shell: { cmd: vim, cwd: test }
    

    Workflow

    A workflow represents a rule, typically some execution logic like "when A happens, then B then C then D".

    A concrete example is that when a file changed in "src" folder, we would like lint and test to be triggered automatically.

    A workflow also emit events, from this point of view it is also a special stream.

    In dashflow, workflows are always triggered while a new event matches given pattern.

    The following is an example of workflow configurations:

    workflows:
      initial-lint:
        // special event fires when dashflow starts
        match: SYSTEM:started
        command: lint
      initial-test:
        match: SYSTEM:started
        command: test
      lib-lint-then-test:
        match: watch-lib:.*
        // execute those commands serially
        serial:
          - command: lint
          - command: test
      test-lint-then-test:
        match: watch-test:.*
        // execute those commands parallelly
        parallel:
          - command: lint
          - command: test
          - restart: irb
      web-lint:
        match: watch-web:.*
        serial:
          - delay: 1000
          - command: lint
    

    Dashboard

    Dashflow starts an HTTP server when running in daemon mode, and what being served is an special page that visualizes all those events dashflow has collected. The page has many tabs, each tab is a dashboard. A dashboard has many widgets, each occupies a rectangular area.

    Different widget type behaves very differently:

    • a "log" widget shows a subset of all events by applying a filter
    • a "gauge" widget shows status texts base on calculations on event streams, like "passing/failed/unknown" etc
    • a "banner" widget displays static text

    For example:

    dashboards:
      all-in-one:
        - log:
            position: 0 0 45% 50%
            title: Lint
            filter: command:lint:.*
        - log:
            position: 45% 0 90% 50%
            title: Test
            filter: command:test:.*
        - banner:
            position: 90% 0 100% 16%
            content: This is Dashflow Dashboard
        - gauge:
            position: 90% 16% 100% 32%
            title: Lint Status
            filter: command:lint:state:.*
            scan:
              when:
                - pattern: started
                  text: Running
                  level: warning
                - pattern: exited with 0
                  text: Passed
                  level: success
                - pattern: exited with
                  text: Failed
                  level: error
              default:
                text: Unknown
    

    Configuration

    A dashflow.yml is required in order to utilize dashflow as a local development workflow orchestrator.

    Basically what we need to do is to model the workflow we already have into those concepts in dashflow.

    Here're some example configurations for your inspiration.

    dashflow.yml for a frontend project

    commands:
      lint: yarn --silent lint_min
      test: yarn --silent test_min
    
    streams:
      site: { shell: { cmd: yarn --silent site } }
      watch-src: { watch: { glob: "src/**/*.js*" } }
    
    workflows:
      initial-lint:
        match: SYSTEM:started
        command: lint
      initial-test:
        match: SYSTEM:started
        command: test
      on-src-update: |
        match: watch-src:.*
        parallel:
          - command: lint
          - command: test
    
    dashboards:
      spark-ui:
        - log:
            position: 0 0 90% 50%
            title: Webpack Logs
            filter: stream:site:.*
        - gauge:
            position: 90% 0 100% 25%
            title: Lint Status
            filter: command:lint:state:.*
            scan:
              when:
                - pattern: started
                  text: Running
                  level: warning
                - pattern: exited with 0
                  text: Passed
                  level: success
                - pattern: exited with
                  text: Failed
                  level: error
              default:
                text: Unknown
        - gauge:
            position: 90% 25% 100% 50%
            title: Test Status
            filter: command:test:state:.*
            scan:
              when:
                - pattern: started
                  text: Running
                  level: warning
                - pattern: exited with 0
                  text: Passed
                  level: success
                - pattern: exited with
                  text: Failed
                  level: error
              default:
                text: Unknown
        - log:
            position: quadrant/bottom-left
            title: Lint
            filter: command:lint:.*
        - log:
            position: quadrant/bottom-right
            title: Test
            filter: command:test:.*
    

    dashflow.yml for dashflow project itself

    Click here

    Reference

    commandID/streamID/workflowID is the key of a command/stream/workflow definition in the configuration file.

    Command

    # execute shell command
    commandID:
      shell:
        cmd: <shell command>
        cwd: "working folder, default to be where dashflow.yml is located"
    # produces events in following formats
    #   command:commandID:shell:<shell command>:stdout:<shell stdout>
    #   command:commandID:shell:<shell command>:state:started
    #   command:commandID:shell:<shell command>:state:exited with <shell exit code>
    

    Stream

    # listen on file changes
    streamID:
      watch: 
        glob: <glob pattern>
        cwd: "working folder, default to be where dashflow.yml is located"
        ignore: /regex pattern for files to ignore/
    # produces events in following formats
    #   stream:streamID:watch:add:<file path>
    #   stream:streamID:watch:addDir:<file path>
    #   stream:streamID:watch:change:<file path>
    #   stream:streamID:watch:unlink:<file path>
    #   stream:streamID:watch:unlinkDir:<file path>
    
    # execute shell command
    streamID:
      shell:
        cmd: <shell command>
        cwd: "working folder, default to be where dashflow.yml is located"
    # short form for the above if cwd is default
    streamID: <shell command>
    # produces events in following formats
    #   stream:streamID:shell:<shell command>:stdout:<shell stdout>
    #   stream:streamID:shell:<shell command>:state:started
    #   stream:streamID:shell:<shell command>:state:exited with <shell exit code>
    

    Workflow

    # execute command if there's a match
    workflowID:
      match: "event pattern that triggers this workflow"
      command: "command name to trigger"
    # produces events in following formats
    #   workflow:workflowID:<command event format here>
    
    # restart stream if there's a match
    workflowID:
      match: "event pattern that triggers this workflow"
      restart: <streamID>
    # produces events in following formats
    #   workflow:workflowID:restart:<streamID>
    
    # wait for a certain time if there's a match
    # usually used together with serial command
    workflowID:
      match: "event pattern that triggers this workflow"
      wait: <time in milliseconds>
    # produces no events
    
    # run workflow actions serially
    workflowID:
      match: "event pattern that triggers this workflow"
      serial: 
        - command: <cmd1>
        - wait: 1000
        - command: <cmd2>
    
    # run workflow actions parallelly
    workflowID:
      match: "event pattern that triggers this workflow"
      parallel: 
        - command: <cmd1>
        - command: <cmd2>
    

    Dashboard

    # show a subset of all event streams by applying a filter
    dashboardID:
      - log:
          position: <rectangular: x1 y1 x2 y2> OR <position-alias>
          title: <title string>
          filter: <filter pattern>
    
    # show a console log with a header gauge
    dashboardID:
      - log:
          position: <rectangular: x1 y1 x2 y2> OR <position-alias>
          title: <title string>
          filter: <log filter pattern>
          gauge:
            filter: <gauge filter pattern>
            scan:
              when:
                - pattern: started
                  text: Running
                  level: warning
                - pattern: exited with 0
                  text: Passed
                  level: success
                - pattern: exited with
                  text: Failed
                  level: error
              default:
                text: Unknown
    
    # show status text by running some calculations
    dashboardID:
      - gauge:
          position: <rectangular: x1 y1 x2 y2> OR <position-alias>
          title: <title string>
          filter: <filter pattern>
          scan:
            when:
              - pattern: <pattern 1>
                text: <status text 1>
                level: <status level: success|warning|error>
              - pattern: <pattern 2>
                text: <status text 2>
                level: <status level: success|warning|error>
            default:
              text: <default text>
              level: <default level, default: none>
    
    # display static content
    dashboardID:
      - banner:
          position: <rectangular: x1 y1 x2 y2> OR <position-alias>
          content: <static content string>
    

    Reference

    Position Aliases

    +----------------------------------------+
    |                                        |
    |                                        |
    |                                        |
    |                                        |
    |                                        |
    |                                        |
    |                                        |
    |                                        |
    |                                        |
    |                                        |
    |                                        |
    |                                        |
    |                                        |
    |                                        |
    |                                        |
    |                                        |
    +----------------------------------------+
    
                  fullscreen
    
    +----------------------+-----------------------+
    |                      |                       |
    |                      |                       |
    |                      |                       |
    | quadrant/top-left    |   quadrant/top-right  |
    |        OR            |          OR           |
    |     quadrant/2       |       quadrant/1      |
    |                      |                       |
    +----------------------------------------------+
    |                      |                       |
    |                      |                       |
    |                      |                       |
    | quadrant/bottom-left | quadrant/bottom-right |
    |        OR            |        OR             |
    |    quadrant/3        |      quadrant/4       |
    |                      |                       |
    |                      |                       |
    +----------------------+-----------------------+
    
                      quadrant
    
    +----------------------------------------+
    |                                        |
    |                                        |
    |                                        |
    |              quadrant/top              |
    |                                        |
    |                                        |
    |                                        |
    +----------------------------------------+
    |                                        |
    |                                        |
    |                                        |
    |                                        |
    |             quadrant/bottom            |
    |                                        |
    |                                        |
    |                                        |
    +----------------------------------------+
    
                      quadrant
    
    +-------------------+--------------------+
    |                   |                    |
    |                   |                    |
    |                   |                    |
    |                   |                    |
    |                   |                    |
    |                   |                    |
    |    quadrant/left  |    quadrant/right  |
    |                   |                    |
    |                   |                    |
    |                   |                    |
    |                   |                    |
    |                   |                    |
    |                   |                    |
    |                   |                    |
    +-------------------+--------------------+
    
                      quadrant
    

    Contributors

    Your contributions are highly welcomed! Please check out Contributor's Guide for more details.

    Acknowledgement

    Dashflow was initially built by engineers @ FreeWheel, a Comcast company, and donated to the open source community.

    Keywords

    none

    Install

    npm i dashflow

    DownloadsWeekly Downloads

    9

    Version

    1.4.0

    License

    Apache-2.0

    Unpacked Size

    399 kB

    Total Files

    46

    Last publish

    Collaborators

    • avatar