@snyk/vervet

vervet

Vervet is an HTTP API version lifecycle management tool, allowing APIs to be designed, developed, versioned and released from resources independently and concurrently.

In a large organization, there might be many teams involved in delivering a large API -- such as at Snyk where Vervet was developed.

Within a single small team, there is still often a need to simultaneously try new things in parts of an API while maintaining stability.

While Vervet was developed in the context of a RESTful API, Vervet can be used with any HTTP API expressed in OpenAPI 3 -- even if it does not adhere to strict REST principles.

API Versioning

To summarize the API versioning supported by Vervet:

What is versioned?

Resource versions are defined in OpenAPI 3, as if each resource were a standalone service.

How are resource version specs organized?

Resources are organized in a standard directory structure by release date, using OpenAPI extensions to define lifecycle concepts like stability.

How does versioning work?

• Resources are versioned independently by date and stability, with a well-defined deprecation and sunsetting policy.
• Additive, non-breaking changes can be made to released versions. Breaking changes trigger a new version.
• New versions deprecate and sunset prior versions, on a timeline determined by the stability level.

Read more about API versioning.

Features

A brief tour of Vervet's features.

Building a service OpenAPI from resources

Vervet collects the OpenAPI specification of each resource version and merges them into a series of OpenAPI specifications that describe the entire application, at each distinct release version in its underlying parts.

Given a directory structure of resource versions, each defined by an OpenAPI specification as if it were an independent service:

tree resources

resources
├── _examples
│   └── hello-world
│       ├── 2021-06-01
│       │   └── spec.yaml
│       ├── 2021-06-07
│       │   └── spec.yaml
│       └── 2021-06-13
│           └── spec.yaml
└── projects
    └── 2021-06-04
        └── spec.yaml

and a Vervet project configuration that instructs how to put them together:

cat .vervet.yaml

apis:
  my-api:
    resources:
      - path: "resources"
    output:
      path: "versions"

vervet build aggregates these resources' individual OpenAPI specifications to describe the entire service API at each distinct version date and stability level from its component parts.

tree versions

versions/
├── 2021-06-01
│   ├── spec.json
│   └── spec.yaml
├── 2021-06-01~beta
│   ├── spec.json
│   └── spec.yaml
├── 2021-06-01~experimental
│   ├── spec.json
│   └── spec.yaml
├── 2021-06-04
│   ├── spec.json
│   └── spec.yaml
├── 2021-06-04~beta
│   ├── spec.json
│   └── spec.yaml
├── 2021-06-04~experimental
│   ├── spec.json
│   └── spec.yaml
├── 2021-06-07
│   ├── spec.json
│   └── spec.yaml
├── 2021-06-07~beta
│   ├── spec.json
│   └── spec.yaml
├── 2021-06-07~experimental
│   ├── spec.json
│   └── spec.yaml
├── 2021-06-13
│   ├── spec.json
│   └── spec.yaml
├── 2021-06-13~beta
│   ├── spec.json
│   └── spec.yaml
└── 2021-06-13~experimental
    ├── spec.json
    └── spec.yaml

Code generation

Since Vervet models the composition, construction and versioning of an API, it is well positioned to coordinate code and artifact generation through the use of templates.

Generators may be defined in a YAML file, such as generators.yaml:

generators:
  version-readme:
    scope: version
    filename: "{{ .Path }}/README"
    template: "{{ .Here }}/templates/README.tmpl" # Located relative to the location of generators.yaml

The context of README.tmpl has full access to the resource version metadata and OpenAPI document object model.

Generated by vervet. DO NOT EDIT!

# My API

Files in this directory were generated by `@snyk/vervet`
for resource `{{ .ResourceVersion.Name }}` at version `{{ .ResourceVersion.Version.String }}`.

In a project with a .vervet.yaml configuration, execute the generators with

vervet generate -g generators.yaml

The simple generator above produces a README in each resource version directory.

tree resources

resources
└── thing
    └── 2021-10-21
        ├── README
        └── spec.yaml

Generators are defined using Go templates.

Template syntax may also be used to express a directory structure of many files. A more advanced example, an Express controller generated from each operation in a resource version OpenAPI spec:

generators:
  version-controller:
    scope: version
    # `files:` generates a collection of files -- which itself is expressed as a
    # YAML template.  Keys in this YAML are the paths of the files to generate,
    # whose values are the file contents.
    files: |-
      {{- $path := .Path -}}
      {{- $resource := .ResourceVersion -}}
      {{- $version := .ResourceVersion.Version -}}
      {{- range $path, $pathItem := .ResourceVersion.Document.Paths -}}
      {{- range $method, $operation := $pathItem -}}
      {{- $operationId := $operation.operationId -}}
      {{/* Construct a context object using the 'map' function */}}
      {{- $ctx := map "Context" . "OperationId" $operationId }}
      {{ $path }}/{{ $operationId }}.ts: |-
        {{/*
             Evaluate the template by including it with the necessary context.
             The generator's template (controller.ts.tmpl) is included as
             "contents" from within the `files:` template.
           */}}
        {{ include "contents" $ctx | indent 2 }}
      {{ end }}
      {{- end -}}
    template: "{{ .Here }}/templates/controller.ts.tmpl"

In this case, a template is being applied per operationId in the spec.yaml generated in the prior step. version-controller produces a collection of files, a controller module per resource, per version, per operation.

Finally, a note on scoping. Generators can be scoped to either a version or a resource.

scope: version generator templates execute with VersionScope. This maps 1:1 with a single resource version OpenAPI specification.

scope: resource generator templates execute with ResourceScope. This is a collection of resource versions, useful for building resource routers.

Installation

NPM

Within a project:

npm install @snyk/vervet

Or installed globally:

npm install -g @snyk/vervet

NPM packaging adapted from https://github.com/manifoldco/torus-cli.

Go

Go >= 1.16 required.

go install github.com/snyk/vervet/v6/cmd/vervet@latest

Building from source locally:

go build ./cmd/vervet

or

make build

Development

Vervet uses a reference set of OpenAPI documents in testdata/resources in tests. CLI tests compare runtime compiled output with pre-compiled, expected output in testdata/output to detect regressions.

When introducing changes that intentionally change the content of compiled output:

• Run go generate ./testdata to update the contents of testdata/output
• Verify that the compiled output is correct
• Commit the changes to testdata/output in your proposed branch

Releasing a new version

A new version of vervet will automatically be generated for Github and npm when new features are introduced, i.e. when commits are merged that are marked with feat:.

Deprecating a version

After removing the endpoint version code and specs, you may see this issue:

ENOENT: no such file or directory, open '.../spec.yaml'

To solve this:

1. Temporarily ignore the endpoint version code in .vervet.yaml
2. Remove the endpoint versions from catalog-info.yaml
3. Remove the old OpenAPI specs.

Example PR