Maintainable automation.
🚧 Status: alpha🚧 Early development
Please expect breaking changes. Feel free to file an issue if you
- find a bug
- need a feature added or tweaked for your use case
- have any other suggestions
PipeDream is an open-source, general-purpose automation tool.
It is an alternative to shell scripts - just as powerful, but more portable, composable, testable and reliable.
How it works
Define a pipeline in a yaml file. Using a simple, but extensible syntax, you can call, chain, merge, and split pipelines, invoke shell commands, handle errors, write tests, create mocks, and much more.
Getting started
Have a look at the Quick Start Guide, the Documentation or just run the program. It will offer to create a sample pipeline file in your current folder.
Installation
From npm
npm i -g @l9/pipedream
Using homebrew (Mac OS/linux)
brew install layer9berlin/tap/pipedream
From source (requires Go installed)
git clone https://github.com/Layer9Berlin/PipeDream
cd PipeDream
go build
pipedream
then select Install when promted.
This will compile PipeDream and run it locally, allowing you to execute the install.pipe file (very meta), which in turn installs everything you need to run PipeDream anywhere on your machine. If you make changes to the code, just run PipeDream and execute the installation.pipe again to update your installed version. You can skip the installation step entirely if you only want to run PipeDream in the project folder and don't need any of the utility tools.
If you use PipeDream a lot, it might be convenient to set up an alias like p that is faster to type.
Uninstall
Installed via npm
npm uninstall -g @l9/pipedream
Installed via homebrew
brew uninstall pipedream
Compiled from source
Simply run
pipedream
in the repo source directory and select Uninstall.
Not sure
If you don't recall how you installed PipeDream, run
pipedream version
and check the via field.
Why wouldn't I just use a shell script?
Shell scripting is an art that takes years to master. Tools like shellcheck and bats can help you improve the code quality of your scripts, but they don't solve the core problems:
- There are many traps for beginners to fall into
- Each command has its own syntax (not to mention subtle differences between shells)
- Error handling is so difficult it's rarely done well
- Even simple tasks often require contorted solutions that break for edge cases
- A lack of tests and documentation means you generally don't want to adapt an existing script
If you have used a modern CI/CD, orchestration or containerization tool, you are probably already familiar with a different way to write a script: defining tasks in yaml format. PipeDream is the natural extension of these tools to your localhost. Automate anything, anywhere, using a simple syntax that follows the principle of least astonishment.
"It's like Ansible for localhost!"
Anonymous
We have created PipeDream to simplify maintenance and dependency management tasks that used to involve a large number of steps, combining many different tools running both locally and on remote servers.
To illustrate its potential, we have used PipeDream to define a universal dependency manager pipe. It offers a single, consistent interface pulling together results from different package managers.