pipeline
Heavily opinionated automated test, build and deploy pipelines.
Makes prod-faring development a joy.
Opinionated
- Monorepo
- Gitlab
- Gitlab CI/CD
- AWS
- AWS S3, EC2 (IAM, Scaling Groups, VPCs, ...)
- Terraform
- Docker
ToC
How to use
Prerequisites
- a Gitlab repo configured so that the CI/CD pipeline has access to the remote terraform state (define the
TF_VAR_ci_registry_password
andTF_VAR_ci_registry_user
environment variables inSettings > CI/CD > Variables
) see https://docs.gitlab.com/ee/user/infrastructure/#get-started-using-local-development. Other secrets will most likely be required by various modules.
Install the package at the root of a repo/monorepo
npm install @hauslo/pipeline
Generate a .pipeline.yml
configuration file
npx @hauslo/pipeline init
Add a resource to .pipeline.yml
version: x.x.x
namespace: xxxxxx
resources:
demo:
module: @hauslo/pipeline-website-static
src: website/demo
options: {}
Install the modules as required
npm install @hauslo/pipeline-website-static
Create a simple websites/demo/public/index.html
file
Now just commit and push to master
git add .
git commit -m "Test pipeline"
git push
And that's it, if your Gitlab CI was configured with AWS credentials the static website in websites/demo/public
will have been deployed to S3 from the ci/cd pipeline.
Modules
- @hauslo/pipeline-website-static
- website-node
- website-gatsby
- docker-app
- docker-node
- function-node
Infrastructure modules
Todo and ideas
-
- limit access to origin
-
proxy
- cloudflare reverse proxy for SSL, CDN and WAF
-
@hauslo/pipeline-website-static
-
proxy
option
-
Write a module
Modules are npm packages, they must be installed in the repository alongside @hauslo/pipeline
.
A module must export an async function that will receive a single object as argument, the signature of this object is
version: string
namespace: string
module: string
name: string
paths: object<string>
options: object<*>|undefined
id: string
domain: string
slug: string
Where
-
version
is the version of@hauslo/pipeline
, the argument object signature depends on this value. A module can ignore this argument if it exports.versionCompatibility
(ex:1.0.0
implies^1.0.0
,semver.satisfies()
is used) -
namespace
is a project namespace (ex:test/demo
, any character is acceptable but the namespace will be trimmed down as necessary when namespacing resources) -
module
is the module name given in.pipeline.yml
(ex:@hauslo/pipeline-website-static
) -
name
is the resource name, as defined in.pipeline.yml
-
paths
helps modules to navigate the project repository. These paths are expected to be used to build the pipeline. -
options
is the module options object found in.pipeline.yml
. It may contain values to configure the module's behavior. If the module exposes avalidate(options: *)
method then it is used to validate the user options -
id
is the resource unique identifier, it includes the project namespace, the pipeline module name, and the resource name. Thisid
, and the nextdomain
andslug
, should be used to name the built ci/cd pipeline and infrastructure (ex:app__hauslo_website_static__demo
) -
domain
is the same asid
but is more suited to name some infrastructure resources (ex:demo.hauslo-website-static.app
) -
slug
is the same asid
but is more suited to name some infrastructure resources (ex:demo-hauslo-website-static-app
)
An example of paths
:
{
"repo": ".",
"src": "front/demo",
"_src": "../..",
"build": ".pipeline",
"_build": "..",
"res": "front/demo",
"_res": "../.."
}
Where
-
repo
is the path to the root of the repository from the execution context (the execution context for thepipeline build
command, not for.gitlab-ci.yml
or terraform, for.gitlab-ci.yml
the execution context is the repo root, for terraform the execution context isbuild
so the repo path is_build
) -
src
is the path to the resource subrepo fromrepo
(ex:websites/demo/
) -
build
is the path to the pipeline build root fromrepo
(ex:.pipeline/
) -
res
is the path to the resource-specific build from the pipeline build root (ex:website-static/demo/
for.pipeline/website-static/demo
)
And
-
_<path>
is the reverse of<path>
forsrc
,build
andres
Modules can place shared build artifacts in build
(terraform configuration files for instance as this is the terraform execution context) and private build artifacts in build/res
The async function exported by the module can return an object
gitlabCiInclude: string|array<string>
Where
-
gitlabCiInclude
is the path to a.gitlab-ci.yml
file to include from the main.gitlab-ci.yml
Namespacing
For sanity, infrastructure resources should be named after id
, domain
or slug
.
Versionning
To ensure that a module is only ever used with a compatible version of pipeline
, it can export a versionCompatibility
property with a semver version number (ex: module.exports.versionCompatibility = "1.0.0";
)
Terraform
Modules can place shared build artifacts in build
(terraform configuration files for instance as this is the terraform execution context) and private build artifacts in build/res
.
Modules terraform configurations can declare the variables ci_registry
, ci_registry_user
and ci_registry_password
, these variables will be passed as environment variables from .gitlab-ci.yml.
Modules should document other secrets they require and users should define these secrets in Gitlab CI environment variables (ex: AWS_ACCESS_KEY_ID
and AWS_SECRET_ACCESS_KEY
).
variable "ci_registry" {
type = string
}
variable "ci_registry_user" {
type = string
}
variable "ci_registry_password" {
type = string
}