@holytiny/yprofile

yprofile

A tool to generate yaml file use 'in-block' profile.

• Motivation
  • Profile
  • Directly
• Quick Start
  • Install
  • Basic Usage
• Details
  • Profile Usage
  • Profiles
  • Path
  • Selector
    • Array Selector
  • Add
  • Replace
  • Remove
  • File Generate
    • Logic
    • Why
  • Directly Modify
    • Add
    • Remove
    • Replace
• NOTICE

Motivation

Profile

Sometimes, we would find that we need to modify the yaml file slightly under different conditions or in the different stages of the project. For example, we might use a yaml file to control the ci/cd system, for develop:

images:
  backend:
    image: john/devbackend
  backend-debugger:
    image: john/debugger
deployments:
  - name: backend
    helm:
      componentChart: true
      values:
        containers:
          - image: john/devbackend
          - image: john/debugger

For staging:

images:
  backend:
    image: john/stagingbackend
  backend-debugger:
    image: john/debugger
deployments:
  - name: backend
    helm:
      componentChart: true
      values:
        containers:
          - image: john/stagingbackend
          - image: john/debugger

And for production:

images:
  backend:
    image: john/prodbackend
deployments:
  - name: backend
    helm:
      componentChart: true
      values:
        containers:
          - image: john/devbackend
          - image: john/cache

Maintaining such yaml files is tendious and error prone. An elegant solution is to generate the yaml files from a template file, and save the modification actions in git repo. A devops tool devspace provides a method to support such solutions, however, saddly it can only be used to its own config file.

Motivated by devspace, yprofile provide nearly the same profile operations to any yaml files with profiles section, and use this template to generate templates as profile describing. These functions are supported by yprofile as generate command.

yprofile generate

Directly

We will find that sometimes we have to modify the yaml file directly in ci script, especially in gitops processes. For example, if we use helm as the manifest, during the ci process we need to modify the helm chart, at least the appVersion property, in another git repo for cd tools to deploy our containers to dock compose or k8s.

So it may be helpful the the yaml file can be directly modified from the command line. These functions are supported by yprofile as operate command.

yprofile add
yprofile remove
yprofile replace

Quick Start

Install

npm install @holytiny/yprofile -D

Basic usage

USAGE
  $ yprofile [COMMAND]

COMMANDS
  add       add a property and value pair in a yaml, mainly used for gitops ci
  generate  generate yaml file from yaml file template with in-block profiles, for manual use
  help      display help for yprofile
  remove    remove a property and its value from a yaml, mainly used for gitops ci
  replace   replace a property and value pair in a yaml, mainly used for gitops ci

If we have a template yaml file named test.template.yaml:

images:
  backend:
    image: john/devbackend
  backend-debugger:
    image: john/debugger
deployments:
  - name: backend
    helm:
      componentChart: true
      values:
        containers:
          - image: john/devbackend
          - image: john/debugger
profiles:
  - name: staging
    patches:
      - op: replace
        path: images.backend.image
        value: john/stagingbackend
      - op: remove
        path: deployments.name=backend.helm.values.containers[1]
  - name: production
    patches:
      - op: replace
        path: images.backend.image
        value: john/prodbackend
      - op: remove
        path: deployments.name=backend.helm.values.containers[1]
      - op: add
        path: deployments.name=backend.helm.values.containers
        value:
          image: john/cache

After we run the command:

npx yprofile generate test.template.yaml staging --output=test.yaml

We would get a yaml file named test.yaml:

images:
  backend:
    image: john/stagingbackend
  backend-debugger:
    image: john/debugger
deployments:
  - name: backend
    helm:
      componentChart: true
      values:
        containers:
          - image: john/devbackend

If a yaml file named Chart.yaml:

apiVersion: v2
name: first-chart
description: A Helm chart for Kubernetes
type: application
version: 0.1.0
appVersion: 1.16.0

After we run the command:

yprofile replace Chart.yaml appVersion 1.16.1

The Chart.yaml will become:

apiVersion: v2
name: first-chart
description: A Helm chart for Kubernetes
type: application
version: 0.1.0
appVersion: 1.16.1

This would be very useful in gitops ci script.

Details

Profile Usage

Simply speaking, yprofile uses generate or abbreviation form gen command to generate the yaml file from a template which has profiles section.

USAGE
  $ yprofile generate INPUT_FILE PROFILE

ARGUMENTS
  INPUT_FILE  input yaml file with profiles in block
  PROFILE     the profile used to generate the yaml file

OPTIONS
  -f, --force          generate the yaml file to output file regardless whether a file has already existed
  -h, --help           show CLI help
  -o, --output=output  the output file path. Default to the same path as input file and suffix with .out

ALIASES
  $ yprofile gen

Profiles

Simply speaking, profiles section are groups of operations. Profiles contains several profile, a profile should contains tow sections, name and patches.

The name of a profile represents the stage, env or process of the operations in it.The patches contains several patch in it. The patch is the real operation.

A patch contains op, path and an operational value. The op is the action to yaml template content, currently, remove, replace and add ops are supported.

profiles:
  - name: staging
    patches:
      - op: replace
        path: images.backend.image
        value: john/stagingbackend
      - op: remove
        path: deployments.name=backend.helm.values.containers[1]

Path

The path specifies which property or element of the template yaml to be modifed or added. The path contains selector which is seperated by the . dot sign. For example:

path: deployments.name=backend.helm.values.containers[1]

The path above contains 5 selector:

• deployments
• name=backend
• helm
• values
• containers[1]

The selector is used to trace down the yaml properties and specify the element to be operated.

Selector

There are tow type of selectors, map selector and array selector. The map selector is straight forward, it uses . sign to trace down the yaml element. For convenient and compatible with the devspace syntex, the array selector supports two styles: [] sign and = sign, which are equivalent in the result.

• deployments: map selector
• name=backend: array selector uses = sign
• helm: map selector
• values: map selector
• containers[1]: array selector uses [] sign

Array Selector

images:
  backend:
    image: john/devbackend
  backend-debugger:
    image: john/debugger
deployments:
  - name: backend
    helm:
      componentChart: true
      values:
        containers:
          - image: john/devbackend
          - image: john/debugger

To select a deployment named backend in the template above, these tow selectors are the same:

• deployments[0]
• deployments.name=backend

To select a container, which image is john/debugger, the two selectors below are the same:

• deployments.name=backend.helm.values.containers.image=john/debugger
• deployments[0].helm.values.containers[1]

Add

The add op can only add the value to the element DOES NOT EXIST, if the element has already existed, plase use replace.

If a yaml template named add-test.yaml:

images:
  backend:
    image: john/devbackend
  backend-debugger:
    image: john/debugger
deployments:
  - name: backend
    helm:
      componentChart: true
      values:
        containers:
          - image: john/devbackend
          - image: john/debugger
profiles:
  - name: production
    patches:
      - op: add
        path: images.frontend
        value:
          image: john/frontend
      - op: add
        path: deployments.name=backend.helm.values.containers
        value:
          image: john/cache
      - op: add
        path: deployments[0].helm.values.containers
        value:
          image: john/frontend

Run yprofile:

npx yprofile gen add-test.yaml production --output=add-test-res.yaml

The output file of add-test-res.yaml should be:

images:
  backend:
    image: john/devbackend
  backend-debugger:
    image: john/debugger
  frontend:
    image: john/frontend
deployments:
  - name: backend
    helm:
      componentChart: true
      values:
        containers:
          - image: john/devbackend
          - image: john/debugger
          - image: john/cache
          - image: john/frontend

Replace

If a yaml template named replace-test.yaml:

images:
  backend:
    image: john/devbackend
  backend-debugger:
    image: john/debugger
deployments:
  - name: backend
    helm:
      componentChart: true
      values:
        containers:
          - image: john/devbackend
          - image: john/devfrontend
          - image: john/debugger
profiles:
  - name: staging
    patches:
      - op: replace
        path: images.backend.image
        value: john/stagingbackend
      - op: replace
        path: deployments[0].helm.values.containers[0]
        value:
          image: john/backend
      - op: replace
        path: deployments.name=backend.helm.values.containers.image=john/devfrontend
        value:
          image: john/frontend
      - op: replace
        path: deployments.name=backend.helm.values.containers[2]
        value:
          image: john/deploy

Run yprofile:

npx yprofile gen replace-test.yaml staging --output=replace-test-res.yaml

The output file of replace-test-res.yaml should be:

images:
  backend:
    image: john/stagingbackend
  backend-debugger:
    image: john/debugger
deployments:
  - name: backend
    helm:
      componentChart: true
      values:
        containers:
          - image: john/backend
          - image: john/frontend
          - image: john/deploy

Remove

If a yaml template named remove-test.yaml:

images:
  backend:
    image: john/devbackend
  backend-debugger:
    image: john/debugger
deployments:
  - name: backend
    helm:
      componentChart: true
      values:
        containers:
          - image: john/devbackend
          - image: john/debugger
          - image: john/cache
          - image: john/frontend
profiles:
  - name: staging
    patches:
      - op: remove
        path: images.backend-debugger
      - op: remove
        path: deployments.name=backend.helm.values.containers[3]
      - op: remove
        path: deployments.name=backend.helm.values.containers.image=john/cache

Run yprofile:

npx yprofile gen replace-test.yaml staging --output=replace-test-res.yaml

The output file of replace-test-res.yaml should be:

images:
  backend:
    image: john/stagingbackend
  backend-debugger:
    image: john/debugger
deployments:
  - name: backend
    helm:
      componentChart: true
      values:
        containers:
          - image: john/backend
          - image: john/frontend
          - image: john/deploy

File Generate

Logic

If there is no file in the output path, the output file is always generated.

If there is already a file in the output path, yprofile will try to read the content of that file, if the content of that file is the same as the generated content, output process will be skipped.

If the content of that file is different from the content generated, if the -f or --force flag is not set, yprofile will issue an error and exit. The content of that file will not be changed.

If the -f or --force flag is set, yprofile will backup that file to a name associated with the current date time, and rewrite that file.

Why

This feature is for file debug.

Suppose that you generate a file named test.yaml using yprofile from the yaml template file test.template.yaml, and the test.yaml doesn't work, you can modify the content of test.yaml directly to test the function. After every things being done, you can use -f flag to regenerate the test.yaml file and you would have tow file, the test.yaml file which is buggy, and the backup file may named test-2020-10-11-13:45:18.yaml which is correct, then you can compare the two file to verify that the content of the template is correct.

Directly Modify

We can run yprofile to modify the yaml file directly rather than using profiles section.

The concepts used in such directly modify command is the same as profile methods.

Add

add a property and value pair in a yaml, mainly used for gitops ci

USAGE
  $ yprofile add INPUT_FILE PATH VALUE

ARGUMENTS
  INPUT_FILE  the file path of the yaml, such as /home/john/workspace/ci.yaml
  PATH        the path of the value to be added, such as ingress.enable
  VALUE       the value to be added, such as true or false

OPTIONS
  -h, --help  show CLI help

If we have a yaml file named add.yaml:

images:
  backend:
    image: john/devbackend
  backend-debugger:
    image: john/debugger
deployments:
  - name: backend
    helm:
      componentChart: true
      values:
        containers:
          - image: john/devbackend
          - image: john/debugger

Run yprofile add:

npx yprofile add add.yaml version 1.0.0

npx yprofile add add.yaml images.frontend image: john/frontend

npx yprofile add add.yaml deployments.name=backend.helm.values.containers image: john/cache

npx yprofile add add.yaml deployments[0].helm.values.containers image: john/frontend

The add.yaml should become:

images:
  backend:
    image: john/devbackend
  backend-debugger:
    image: john/debugger
  frontend:
    image: john/frontend
deployments:
  - name: backend
    helm:
      componentChart: true
      values:
        containers:
          - image: john/devbackend
          - image: john/debugger
          - image: john/cache
          - image: john/frontend
version: 1.0.0

Remove

remove a property and its value from a yaml, mainly used for gitops ci

USAGE
  $ yprofile remove INPUT_FILE PATH

ARGUMENTS
  INPUT_FILE  the file path of the yaml, such as /home/john/workspace/ci.yaml
  PATH        the path of the value to be removed, such as ingress.enable

OPTIONS
  -h, --help  show CLI help

If we have a yaml file named remove.yaml:

images:
  backend:
    image: john/devbackend
  backend-debugger:
    image: john/debugger
deployments:
  - name: backend
    helm:
      componentChart: true
      values:
        containers:
          - image: john/devbackend
          - image: john/debugger
          - image: john/cache
          - image: john/frontend

Run yprofile remove:

npx yprofile remove remove.yaml images.backend-debugger

npx yprofile remove remove.yaml deployments.name=backend.

npx yprofile remove remove.yaml deployments.name=backend.helm.values.containers.image=john/cache

The remove.yaml should become:

images:
  backend:
    image: john/devbackend
deployments:
  - name: backend
    helm:
      componentChart: true
      values:
        containers:
          - image: john/devbackend
          - image: john/debugger

Replace

replace a property and value pair in a yaml, mainly used for gitops ci

USAGE
  $ yprofile replace INPUT_FILE PATH VALUE

ARGUMENTS
  INPUT_FILE  the file path of the yaml, such as /home/john/workspace/ci.yaml
  PATH        the path of the value to be replaced, such as ingress.enable
  VALUE       the value to be replaced, such as true or false

OPTIONS
  -h, --help  show CLI help

If we have a file named replace.yaml:

images:
  backend:
    image: john/devbackend
  backend-debugger:
    image: john/debugger
deployments:
  - name: backend
    helm:
      componentChart: true
      values:
        containers:
          - image: john/devbackend
          - image: john/devfrontend
          - image: john/debugger

If we run yprofile replace:

npx yprofile replace replace.yaml images.backend.image john/stagingbackend

npx yprofile replace replace.yaml deployments[0].helm.values.containers[0] image: john/backend

npx yprofile replace replace.yaml deployments.name=backend.helm.values.containers.image=john/devfrontend image: john/frontend

npx yprofile replace replace.yaml deployments.name=backend.helm.values.containers[2] image: john/deploy

The replace.yaml should become:

images:
  backend:
    image: john/stagingbackend
  backend-debugger:
    image: john/debugger
deployments:
  - name: backend
    helm:
      componentChart: true
      values:
        containers:
          - image: john/backend
          - image: john/frontend
          - image: john/deploy

NOTICE

• Try to always use the = sign to select the arry
• Only use [] sign at the end of the path, if you want to use [] sign selector to select array element
• Try hard to avoid create patch rely on the results of other patches
• The content of output yaml file is sorted

Try to always use the = sign to select the arry

Only use [] sign at the end of the path, if you want to use [] sign selector to select array element

Because the content of the yaml template would change frequently and the index of the element in arry would change...

Try hard to avoid create patch rely on the results of other patches

For the sake of efficiency and speed, yprofile DO NOT reparse the output yaml after eache patch. The sequence of the yprofile is something like:

read the template -> read the profile -> do all patches to the template -> output

So the conent of the output DO NOT CHANGE actually after each patch before it is write to the output file.

This may conflict with your intuition，so please don't generate the file rely on the logic between each patch!!!

If you really need such functions, please consider using directly modify methods.

The content of output yaml file is sorted

To sematicly compare two yaml files, the yaml file is sorted immediatly after it is read, so the output of the yaml file is also sorted.

So the sequence of the output yaml file sections may be different from the input file.