Noodles Practicing Medicine

    @stackql/openapi-doc-util

    1.2.0 • Public • Published

    openapi-doc-util

    Command line utility to split OpenAPI documents into smaller, self contained, service oriented documents and prepare StackQL provider interfaces Command line utility to help developers prepare and submit StackQL provider specs, see StackQL Readme

    Installation

    NPM

    npm i @stackql/openapi-doc-util
    YARN

    yarn add @stackql/openapi-doc-util

    Setup

    from the package directory, run:

    npm link

    Background

    Read this section for background on the StackQL product

    The StackQL utility provides a SQL interface to cloud and SaaS providers, mapping a provider to an ORM, transpiling input SQL to provider API requests, and bringing back response data as a SQL based result set. StackQL is capable of DML operations such as `INSERT` and `DELETE` which can be used to provision or de-provision cloud resources, query operations using `SELECT` to collect, analyze, report on asset or configuration data, and lifecycle operations such as starting a VM instance using the `EXEC` command in StackQL.

    The StackQL ORM provides a logical model to work with cloud resources similar to the way databases are organized into schemas. This object/resource hierarchy is summarized below:

    provider/
    ├─ service/
    │  ├─ resource/
    │  │  ├─ fields
    │  │  ├─ methods
    

    an example would be:

    google/
    ├─ compute/
    │  ├─ instances/
    │  │  ├─ id, name, status, ...
    │  │  ├─ select, insert, delete, start, stop, ...
    

    Enabling StackQL to interact with the google provider using SQL semantics, for example:

    Provider discovery operations such as..

    SHOW RESOURCES IN google.compute;
    DESCRIBE google.compute.instances;

    Query operations such as..

    SELECT status, COUNT(*) as num_instances 
    FROM google.compute.instances
    WHERE project = 'myproject' and zone = 'us-west-1a'
    GROUP BY status;

    Provisioning operations such as creating a Compute Engine instance using an INSERT statement or deprovisioning an instance using a DELETE statement.

    Read this section for background on the StackQL Provider Registry

    StackQL provider interfaces (such as GCP, Okta, GitHub, AWS, Azure, etc) are defined using annotations to the provider API (OpenAPI) specification, these annotations or extensions allow StackQL to map the providers resource to the desired ORM and define routes for SQL verbs such as `SELECT`, `INSERT`, `DELETE`, and `EXEC`.

    Usage

    openapi-doc-util <command> [<arguments(s)>] [<flag(s)>]

    Commands

    validate

    Validates an OpenAPI specification document.

    split

    Splits an Open API document into smaller, self contained, service scoped documents based upon a specified service discriminator (JSONPath expression relative to each operation). Takes an OpenAPI document as input and outputs the following structure:

    {output_dir}/
    ├─ {provider_name}/
    │  ├─ {provider_version}/
    │  │  ├─ services/
    │  │  │  ├─ {service_name}/
    │  │  │  │  ├─ {service_name}.yaml
    │  │  │  ├─ .../
    

    The {service_name}.yaml file is a self contained, OpenAPI specification document, containing only operations and components for the specified service.

    provider-dev

    Iterates through a directory of services (created using the split function), generates an additional document for each service {service_name}-resources.yaml and an index document for all services named provider.yaml, this is useful for developing new StackQL providers. The output structure is as follows:

    {api_doc_dir}/
    ├─ {provider_name}/
    │  ├─ {provider_version}/
    │  │  ├─ services/
    │  │  │  ├─ {service_name}/
    │  │  │  │  ├─ {service_name}.yaml
    │  │  │  │  ├─ {service_name}-resources.yaml
    │  │  │  ├─ .../
    │  │  ├─ provider.yaml
    

    The original OpenAPI documents are not modified.

    The provider-dev function will infer SQL routes for operations under each resources under the sqlVerbs key, this should be reviewed by the developer, removing any duplicate or erroneous routes.

    provider-build

    Operates on the dev directory structure generated using the provider-dev command, assembles a complete Open API specification per service including the StackQL resources for each service in each document under components/x-stackQL-resources, and outputs the following structure to a new directory:

    {output_dir}/
    ├─ {provider_name}/
    │  ├─ {provider_version}/
    │  │  ├─ services/
    │  │  │  ├─ {service_name}/
    │  │  │  │  ├─ {service_name}.yaml
    │  │  │  ├─ .../
    │  │  ├─ provider.yaml
    

    The output from the provider-build command can be used to test locally (see Test Using a Local Registry). Once you have tested locally you can raise a pull request to stackql/stackql-provider-registry.

    Options

    --svcDiscriminator, -s JSONPath expression OR svcName:servicename

    The service discriminator option is used to determine how to split a large provider document into smaller, service scoped documents. The option is required for the split command and ignored otherwise. If you do not wish to spilt the provider API spec, specify svcName:<servicename> for this option which will define one service in the StackQL provider with the name provided in servicename.

    Example: -s "$['x-github'].category" would split the given provider API spec into service documents by matching the x-github.category value in each unique operation (combination of a path and an HTTP method) in API doc.

    --resDiscriminator, -r JSONPath expression OR path_tokens

    The resource discriminator option is used to determine how to identify StackQL resources in a given provider API spec. This option is used for the provider-dev command and ignored otherwise.

    Example: -r "$['x-github'].subcategory" would identify resources in the given provider API spec by matching the x-github.subcategory value in each unique operation (combination of a path and an HTTP method) in API doc.

    if path_tokens is specified for the --resDiscriminator option, the resource name will be derived by joining the 'meaningful' path tokens (not equivalent to the service name) with an '_'. For instance a path of /sites/{site_id}/service-instances would result in a resource name of sites_service_instances assuming the service name was not sites.

    --methodkey, -m JSONPath expression

    The method key option determines uniquely named operations which are mapped to SQL methods in the StackQL ORM. These methods are then mapped to default routes (SQL query and DML methods) in StackQL, the developer can override or update these mappings in the development docs which are outputted from the provider-dev command. This option is used for the provider-dev command and ignored otherwise.

    If supplied it must be a JSONPath expression relative to the operation (http path and verb), if not supplied it will default to operationId in the OpenAPI specification for each operation.

    --output, -o directory

    The output directory option specifies the root directory to output files from the split, or the root directory to output assembled provider specs from the provider-build command. The default is the current working directory. If the directory exists in either case the program will exit with an error unless the --overwrite option is set.

    If not supplied it will default to the current working directory

    --overwrite

    The overwrite option specifies whether to overwrite files in the output directory. The default is false.

    --debug, -d

    debug flag which can be set for additional print statements.

    Example

    This example demonstrates creating a StackQL provider for github and testing this provider locally using stackql

    Example Project Structure

    The following directory structure represents the target after an API dev workspace is set up and the StackQL provider for github is built.

    local-registry/
    ├─ ref/
    │  ├─ github/
    │  │  ├─ api.github.com.yaml
    ├─ dev/
    │  ├─ github/
    │  │  ├─ v0.1.0/
    │  │  │  ├─ provider.yaml
    │  │  │  ├─ services/
    │  │  │  │  ├─ repos/
    │  │  │  │  │  ├─ repos.yaml
    │  │  │  │  │  ├─ repos-resources.yaml
    │  │  │  │  ├─ .../
    ├─ src/
    │  ├─ github/
    │  │  ├─ v0.1.0/
    │  │  │  ├─ provider.json
    │  │  │  ├─ services/
    │  │  │  │  ├─ repos/
    │  │  │  │  │  ├─ repos-v0.1.0.json
    

    local-registry/ref/github/api.github.com.yaml is the source OpenAPI 3 specification for the github api, this was sourced from GitHub.

    The dev directory contains the output of the dev docs generated by the openapi-doc-util split command, which will then serve as input to the openapi-doc-util provider-dev command.

    The src directory contains the output of the StackQL provider interface generated from the dev docs using openapi-doc-util provider-build.

    1. Split large spec into service specs

    PowerShell:

    cd local-registry
    openapi-doc-util split `
    -n github `
    -v v0.1.0 `
    -s "$['x-github'].category" `
    -o .\dev `
    ./ref/github/api.github.com.yaml

    Bash:

    cd local-registry
    openapi-doc-util split \
    -n github \
    -v v0.1.0 \
    -s '$["x-github"].category' \
    -o ./dev \
    ref/github/api.github.com.yaml

    2. Create dev provider docs

    PowerShell:

    cd local-registry
    openapi-doc-util provider-dev `
    -n github `
    -v v0.1.0 `
    -r "$['x-github'].subcategory" `
    .\dev

    Bash:

    cd local-registry
    openapi-doc-util provider-dev \
    -n github \
    -v v0.1.0 \
    -r '$["x-github"].subcategory' \
    ./dev

    Assemble final provider docs

    PowerShell:

    cd local-registry
    openapi-doc-util provider-build `
    -n github `
    -v v0.1.0 `
    -o .\src `
    .\dev

    Bash:

    cd local-registry
    openapi-doc-util provider-build \
    -n github \
    -v v0.1.0 \
    -o ./src \
    ./dev

    Test Using a Local Registry

    cd local-registry
    PROVIDER_REGISTRY_ROOT_DIR="$(pwd)"
    REG_STR='{"url": "file://'${PROVIDER_REGISTRY_ROOT_DIR}'", "localDocRoot": "'${PROVIDER_REGISTRY_ROOT_DIR}'", "verifyConfig": {"nopVerify": true}}'
    AUTH_STR='{"github": { "type": "null_auth" }}'
    ./stackql shell --auth="${AUTH_STR}" --registry="${REG_STR}"

    Install

    npm i @stackql/openapi-doc-util

    DownloadsWeekly Downloads

    47

    Version

    1.2.0

    License

    MIT

    Unpacked Size

    60.5 kB

    Total Files

    15

    Last publish

    Collaborators

    • jeffreyaven