Have ideas to improve npm?Join in the discussion! »

    @holytiny/yprofile
    TypeScript icon, indicating that this package has built-in type declarations

    2.0.1 • Public • Published

    yprofile

    A tool to generate yaml file use 'in-block' profile.

    oclif Version CI Downloads/week License

    Motivation

    Profile

    Sometimes, we would find that we need to modify the yaml file slightly under different conditions or in the different stages of the project. For example, we might use a yaml file to control the ci/cd system, for develop:

    images:
      backend:
        image: john/devbackend
      backend-debugger:
        image: john/debugger
    deployments:
      name: backend
        helm:
          componentChart: true
          values:
            containers:
              image: john/devbackend
              image: john/debugger

    For staging:

    images:
      backend:
        image: john/stagingbackend
      backend-debugger:
        image: john/debugger
    deployments:
      name: backend
        helm:
          componentChart: true
          values:
            containers:
              image: john/stagingbackend
              image: john/debugger

    And for production:

    images:
      backend:
        image: john/prodbackend
    deployments:
      name: backend
        helm:
          componentChart: true
          values:
            containers:
              image: john/devbackend
              image: john/cache

    Maintaining such yaml files is tendious and error prone. An elegant solution is to generate the yaml files from a template file, and save the modification actions in git repo. A devops tool devspace provides a method to support such solutions, however, saddly it can only be used to its own config file.

    Motivated by devspace, yprofile provide nearly the same profile operations to any yaml files with profiles section, and use this template to generate templates as profile describing. These functions are supported by yprofile as generate command.

    yprofile generate

    Directly

    We will find that sometimes we have to modify the yaml file directly in ci script, especially in gitops processes. For example, if we use helm as the manifest, during the ci process we need to modify the helm chart, at least the appVersion property, in another git repo for cd tools to deploy our containers to dock compose or k8s.

    So it may be helpful the the yaml file can be directly modified from the command line. These functions are supported by yprofile as operate command.

    yprofile add
    yprofile remove
    yprofile replace

    Quick Start

    Install

    npm install @holytiny/yprofile -D

    Basic usage

    USAGE
      $ yprofile [COMMAND]
     
    COMMANDS
      add       add a property and value pair in a yaml, mainly used for gitops ci
      generate  generate yaml file from yaml file template with in-block profiles, for manual use
      help      display help for yprofile
      remove    remove a property and its value from a yaml, mainly used for gitops ci
      replace   replace a property and value pair in a yaml, mainly used for gitops ci

    If we have a template yaml file named test.template.yaml:

    images:
      backend:
        image: john/devbackend
      backend-debugger:
        image: john/debugger
    deployments:
      name: backend
        helm:
          componentChart: true
          values:
            containers:
              image: john/devbackend
              image: john/debugger
    profiles:
      name: staging
        patches:
          op: replace
            path: images.backend.image
            value: john/stagingbackend
          op: remove
            path: deployments.name=backend.helm.values.containers[1]
      name: production
        patches:
          op: replace
            path: images.backend.image
            value: john/prodbackend
          op: remove
            path: deployments.name=backend.helm.values.containers[1]
          op: add
            path: deployments.name=backend.helm.values.containers
            value:
              image: john/cache

    After we run the command:

    npx yprofile generate test.template.yaml staging --output=test.yaml

    We would get a yaml file named test.yaml:

    images:
      backend:
        image: john/stagingbackend
      backend-debugger:
        image: john/debugger
    deployments:
      name: backend
        helm:
          componentChart: true
          values:
            containers:
              image: john/devbackend

    If a yaml file named Chart.yaml:

    apiVersion: v2
    name: first-chart
    description: A Helm chart for Kubernetes
    type: application
    version: 0.1.0
    appVersion: 1.16.0

    After we run the command:

    yprofile replace Chart.yaml appVersion 1.16.1
    

    The Chart.yaml will become:

    apiVersion: v2
    name: first-chart
    description: A Helm chart for Kubernetes
    type: application
    version: 0.1.0
    appVersion: 1.16.1

    This would be very useful in gitops ci script.

    Details

    Profile Usage

    Simply speaking, yprofile uses generate or abbreviation form gen command to generate the yaml file from a template which has profiles section.

    USAGE
      $ yprofile generate INPUT_FILE PROFILE
     
    ARGUMENTS
      INPUT_FILE  input yaml file with profiles in block
      PROFILE     the profile used to generate the yaml file
     
    OPTIONS
      -f, --force          generate the yaml file to output file regardless whether a file has already existed
      -h, --help           show CLI help
      -o, --output=output  the output file path. Default to the same path as input file and suffix with .out
     
    ALIASES
      $ yprofile gen

    Profiles

    Simply speaking, profiles section are groups of operations. Profiles contains several profile, a profile should contains tow sections, name and patches.

    The name of a profile represents the stage, env or process of the operations in it.The patches contains several patch in it. The patch is the real operation.

    A patch contains op, path and an operational value. The op is the action to yaml template content, currently, remove, replace and add ops are supported.

    profiles:
      name: staging
        patches:
          op: replace
            path: images.backend.image
            value: john/stagingbackend
          op: remove
            path: deployments.name=backend.helm.values.containers[1]

    Path

    The path specifies which property or element of the template yaml to be modifed or added. The path contains selector which is seperated by the . dot sign. For example:

    path: deployments.name=backend.helm.values.containers[1]

    The path above contains 5 selector:

    • deployments
    • name=backend
    • helm
    • values
    • containers[1]

    The selector is used to trace down the yaml properties and specify the element to be operated.

    Selector

    There are tow type of selectors, map selector and array selector. The map selector is straight forward, it uses . sign to trace down the yaml element. For convenient and compatible with the devspace syntex, the array selector supports two styles: [] sign and = sign, which are equivalent in the result.

    • deployments: map selector
    • name=backend: array selector uses = sign
    • helm: map selector
    • values: map selector
    • containers[1]: array selector uses [] sign

    Array Selector

    images:
      backend:
        image: john/devbackend
      backend-debugger:
        image: john/debugger
    deployments:
      name: backend
        helm:
          componentChart: true
          values:
            containers:
              image: john/devbackend
              image: john/debugger

    To select a deployment named backend in the template above, these tow selectors are the same:

    • deployments[0]
    • deployments.name=backend

    To select a container, which image is john/debugger, the two selectors below are the same:

    • deployments.name=backend.helm.values.containers.image=john/debugger
    • deployments[0].helm.values.containers[1]

    Add

    The add op can only add the value to the element DOES NOT EXIST, if the element has already existed, plase use replace.

    If a yaml template named add-test.yaml:

    images:
      backend:
        image: john/devbackend
      backend-debugger:
        image: john/debugger
    deployments:
      name: backend
        helm:
          componentChart: true
          values:
            containers:
              image: john/devbackend
              image: john/debugger
    profiles:
      name: production
        patches:
          op: add
            path: images.frontend
            value:
              image: john/frontend
          op: add
            path: deployments.name=backend.helm.values.containers
            value:
              image: john/cache
          op: add
            path: deployments[0].helm.values.containers
            value:
              image: john/frontend

    Run yprofile:

    npx yprofile gen add-test.yaml production --output=add-test-res.yaml

    The output file of add-test-res.yaml should be:

    images:
      backend:
        image: john/devbackend
      backend-debugger:
        image: john/debugger
      frontend:
        image: john/frontend
    deployments:
      name: backend
        helm:
          componentChart: true
          values:
            containers:
              image: john/devbackend
              image: john/debugger
              image: john/cache
              image: john/frontend

    Replace

    If a yaml template named replace-test.yaml:

    images:
      backend:
        image: john/devbackend
      backend-debugger:
        image: john/debugger
    deployments:
      name: backend
        helm:
          componentChart: true
          values:
            containers:
              image: john/devbackend
              image: john/devfrontend
              image: john/debugger
    profiles:
      name: staging
        patches:
          op: replace
            path: images.backend.image
            value: john/stagingbackend
          op: replace
            path: deployments[0].helm.values.containers[0]
            value:
              image: john/backend
          op: replace
            path: deployments.name=backend.helm.values.containers.image=john/devfrontend
            value:
              image: john/frontend
          op: replace
            path: deployments.name=backend.helm.values.containers[2]
            value:
              image: john/deploy

    Run yprofile:

    npx yprofile gen replace-test.yaml staging --output=replace-test-res.yaml

    The output file of replace-test-res.yaml should be:

    images:
      backend:
        image: john/stagingbackend
      backend-debugger:
        image: john/debugger
    deployments:
      name: backend
        helm:
          componentChart: true
          values:
            containers:
              image: john/backend
              image: john/frontend
              image: john/deploy

    Remove

    If a yaml template named remove-test.yaml:

    images:
      backend:
        image: john/devbackend
      backend-debugger:
        image: john/debugger
    deployments:
      name: backend
        helm:
          componentChart: true
          values:
            containers:
              image: john/devbackend
              image: john/debugger
              image: john/cache
              image: john/frontend
    profiles:
      name: staging
        patches:
          op: remove
            path: images.backend-debugger
          op: remove
            path: deployments.name=backend.helm.values.containers[3]
          op: remove
            path: deployments.name=backend.helm.values.containers.image=john/cache

    Run yprofile:

    npx yprofile gen replace-test.yaml staging --output=replace-test-res.yaml

    The output file of replace-test-res.yaml should be:

    images:
      backend:
        image: john/stagingbackend
      backend-debugger:
        image: john/debugger
    deployments:
      name: backend
        helm:
          componentChart: true
          values:
            containers:
              image: john/backend
              image: john/frontend
              image: john/deploy

    File Generate

    Logic

    If there is no file in the output path, the output file is always generated.

    If there is already a file in the output path, yprofile will try to read the content of that file, if the content of that file is the same as the generated content, output process will be skipped.

    If the content of that file is different from the content generated, if the -f or --force flag is not set, yprofile will issue an error and exit. The content of that file will not be changed.

    If the -f or --force flag is set, yprofile will backup that file to a name associated with the current date time, and rewrite that file.

    Why

    This feature is for file debug.

    Suppose that you generate a file named test.yaml using yprofile from the yaml template file test.template.yaml, and the test.yaml doesn't work, you can modify the content of test.yaml directly to test the function. After every things being done, you can use -f flag to regenerate the test.yaml file and you would have tow file, the test.yaml file which is buggy, and the backup file may named test-2020-10-11-13:45:18.yaml which is correct, then you can compare the two file to verify that the content of the template is correct.

    Directly Modify

    We can run yprofile to modify the yaml file directly rather than using profiles section.

    The concepts used in such directly modify command is the same as profile methods.

    Add

    add a property and value pair in a yaml, mainly used for gitops ci
     
    USAGE
      $ yprofile add INPUT_FILE PATH VALUE
     
    ARGUMENTS
      INPUT_FILE  the file path of the yaml, such as /home/john/workspace/ci.yaml
      PATH        the path of the value to be added, such as ingress.enable
      VALUE       the value to be added, such as true or false
     
    OPTIONS
      -h, --help  show CLI help

    If we have a yaml file named add.yaml:

    images:
      backend:
        image: john/devbackend
      backend-debugger:
        image: john/debugger
    deployments:
      name: backend
        helm:
          componentChart: true
          values:
            containers:
              image: john/devbackend
              image: john/debugger

    Run yprofile add:

    npx yprofile add add.yaml version 1.0.0
     
    npx yprofile add add.yaml images.frontend image: john/frontend
     
    npx yprofile add add.yaml deployments.name=backend.helm.values.containers image: john/cache
     
    npx yprofile add add.yaml deployments[0].helm.values.containers image: john/frontend

    The add.yaml should become:

    images:
      backend:
        image: john/devbackend
      backend-debugger:
        image: john/debugger
      frontend:
        image: john/frontend
    deployments:
      name: backend
        helm:
          componentChart: true
          values:
            containers:
              image: john/devbackend
              image: john/debugger
              image: john/cache
              image: john/frontend
    version: 1.0.0

    Remove

    remove a property and its value from a yaml, mainly used for gitops ci
     
    USAGE
      $ yprofile remove INPUT_FILE PATH
     
    ARGUMENTS
      INPUT_FILE  the file path of the yaml, such as /home/john/workspace/ci.yaml
      PATH        the path of the value to be removed, such as ingress.enable
     
    OPTIONS
      -h, --help  show CLI help

    If we have a yaml file named remove.yaml:

    images:
      backend:
        image: john/devbackend
      backend-debugger:
        image: john/debugger
    deployments:
      name: backend
        helm:
          componentChart: true
          values:
            containers:
              image: john/devbackend
              image: john/debugger
              image: john/cache
              image: john/frontend

    Run yprofile remove:

    npx yprofile remove remove.yaml images.backend-debugger
     
    npx yprofile remove remove.yaml deployments.name=backend.
     
    npx yprofile remove remove.yaml deployments.name=backend.helm.values.containers.image=john/cache

    The remove.yaml should become:

    images:
      backend:
        image: john/devbackend
    deployments:
      name: backend
        helm:
          componentChart: true
          values:
            containers:
              image: john/devbackend
              image: john/debugger

    Replace

    replace a property and value pair in a yaml, mainly used for gitops ci
     
    USAGE
      $ yprofile replace INPUT_FILE PATH VALUE
     
    ARGUMENTS
      INPUT_FILE  the file path of the yaml, such as /home/john/workspace/ci.yaml
      PATH        the path of the value to be replaced, such as ingress.enable
      VALUE       the value to be replaced, such as true or false
     
    OPTIONS
      -h, --help  show CLI help

    If we have a file named replace.yaml:

    images:
      backend:
        image: john/devbackend
      backend-debugger:
        image: john/debugger
    deployments:
      name: backend
        helm:
          componentChart: true
          values:
            containers:
              image: john/devbackend
              image: john/devfrontend
              image: john/debugger

    If we run yprofile replace:

    npx yprofile replace replace.yaml images.backend.image john/stagingbackend
     
    npx yprofile replace replace.yaml deployments[0].helm.values.containers[0] image: john/backend
     
    npx yprofile replace replace.yaml deployments.name=backend.helm.values.containers.image=john/devfrontend image: john/frontend
     
    npx yprofile replace replace.yaml deployments.name=backend.helm.values.containers[2] image: john/deploy

    The replace.yaml should become:

    images:
      backend:
        image: john/stagingbackend
      backend-debugger:
        image: john/debugger
    deployments:
      name: backend
        helm:
          componentChart: true
          values:
            containers:
              image: john/backend
              image: john/frontend
              image: john/deploy

    NOTICE

    • Try to always use the = sign to select the arry
    • Only use [] sign at the end of the path, if you want to use [] sign selector to select array element
    • Try hard to avoid create patch rely on the results of other patches
    • The content of output yaml file is sorted

    Try to always use the = sign to select the arry

    Only use [] sign at the end of the path, if you want to use [] sign selector to select array element

    Because the content of the yaml template would change frequently and the index of the element in arry would change...

    Try hard to avoid create patch rely on the results of other patches

    For the sake of efficiency and speed, yprofile DO NOT reparse the output yaml after eache patch. The sequence of the yprofile is something like:

    read the template -> read the profile -> do all patches to the template -> output
    

    So the conent of the output DO NOT CHANGE actually after each patch before it is write to the output file.

    This may conflict with your intuition,so please don't generate the file rely on the logic between each patch!!!

    If you really need such functions, please consider using directly modify methods.

    The content of output yaml file is sorted

    To sematicly compare two yaml files, the yaml file is sorted immediatly after it is read, so the output of the yaml file is also sorted.

    So the sequence of the output yaml file sections may be different from the input file.

    Install

    npm i @holytiny/yprofile

    DownloadsWeekly Downloads

    1

    Version

    2.0.1

    License

    MIT

    Unpacked Size

    43.7 kB

    Total Files

    23

    Last publish

    Collaborators

    • avatar