pshregistry-parser

Platform.sh Registry Parser

This is a simple library for parsing the Platform.sh registry. It provides access to a subset of Registry properties for each image, and it can be used to generate valid and up-to-date configuration YAML files for Platform.sh written in Node.js. It accesses data in a registry.json to generate those files.

Note: At this stage, the library requires a local copy of registry.json to be defined in order to work. Ultimately, there will be no local registry file it reads from, instead reading from a remote, regularly-updating source in GitLab.

Usage Example

This is an early stage of the registry project.

At this point, common actions like adding a newly supported version to an existing image requires you to manually add that new version to the <image>.versions.supported attribute in the registry.json file.

You can find the current up-to-date version of this file in the deployed public documentation.

Generating example configuration files

The library creates example YAML files that fall into three main categories:

• commented: Full configuration file used in a project, including heavy commenting. For example, for Elasticsearch,

  # The name given to the Elasticsearch service (lowercase alphanumeric only).
  mysearch:
    # The type of your service (elasticsearch), which uses the format
    # 'type:version'. Be sure to consult the Elasticsearch documentation
    # (https://docs.platform.sh/configuration/services/elasticsearch.html#supported-versions)
    # when choosing a version. If you specify a version number which is not available,
    # the CLI will return an error.
    type: elasticsearch:7.2
    # The disk attribute is the size of the persistent disk (in MB) allocated to the service.
    disk: 256

• full: Full configuration file used in a project. For the Elasticsearch .platform.app.yaml,

  relationships:
    elasticsearch: "mysearch:elasticsearch"

• snippet: Partial configuration file. Single-line for that image so that it could be used to append to another file if we ever want to have tools that generate complete configuration files for the user. For the Elasticsearch .platform.app.yaml snippet,

    elasticsearch: "mysearch:elasticsearch"

Each of these files for every image in a Registry object is generated from the write() method:

// updateConfigs.js
const psh = require("pshregistry-parser");

var registryLocation = "src/registry/images/registry.json";

var registry = new RegistryParser(registryLocation);
registry.write();

but you can also write files for individual images (i.e. Elasticsearch) with

cg.write("elasticsearch");

Filenames use the convention <image_name>.<config_file>.yaml. Therefore, the command cg.write("elasticsearch") would generate

saveDir/
    examples/
        commented/
            elasticsearch.app.yaml
            elasticsearch.services.yaml
        full/
            elasticsearch.app.yaml
            elasticsearch.services.yaml
        snippet/
            elasticsearch.app.yaml
            elasticsearch.services.yaml

content.json

Depending on your use case, you may not want to install this library as a dependency for your project, and instead point to another repository that serves the generated example configuration YAMLs as a resource. In order to facilitate your ability to do so, each examples subdirectory contains a file called content.json which lists all of the example files in that subdirectory. For example, examples/commented/content.json could include the object:

{
  "files": [
    "elasticsearch.app.yaml",
    "elasticsearch.services.yaml",
    "golang.app.yaml",
    "kafka.app.yaml",
    "kafka.services.yaml",
    "redis.app.yaml",
    "redis.services.yaml",
    "mariadb.app.yaml",
    "mariadb.services.yaml",
    "network-storage.app.yaml",
    "network-storage.services.yaml",
    "varnish.app.yaml",
    "varnish.routes.yaml",
    "varnish.services.yaml",
    "mysql.app.yaml",
    "mysql.services.yaml",
    "redis-persistent.app.yaml",
    "redis-persistent.services.yaml"
  ]
}

content.json files are not created when single images are passed to write(), only when files are written for every image in the Registry.

Save Location

You can specify the save location for the generated YAML files with a second parameter to RegistryParser()

const psh = require("pshregistry-parser");

var registryLocation = "src/registry/images/registry.json";
var saveLocation = "myfiles/examples/"

var registry = new RegistryParser(registryLocation, saveLocation);
registry.write();

If a directory is not specfied, the library will assume that all files should be generated in the same directory as registry.json. If examples or any of its subdirectories do not exist, they will be created.

Accessing images

Accessing all images

Each Registry image is an instance of a child of an Image object, and is accessible directly from the config generator instance through the images property. For example, registry.images["php"] returns

Runtime {
  indent: '    ',
  description: 'PHP service for Platform.sh.',
  repo_name: 'php',
  disk: false,
  docs:
   { relationship_name: null,
     service_name: null,
     url: '/languages/php.html' },
  endpoint: null,
  min_disk_size: null,
  name: 'PHP',
  runtime: true,
  type: 'php',
  supported: [ '7.1', '7.2', '7.3' ],
  deprecated: [ '5.4', '5.5', '5.6', '7.0' ],
  recommended: '7.3',
  supportedHTML: '<ul><li>7.1</li><li>7.2</li><li>7.3</li></ul>',
  supportedString: '7.1, 7.2, 7.3',
  deprecatedHTML: '<ul><li>5.4</li><li>5.5</li><li>5.6</li><li>7.0</li></ul>',
  deprecatedString: '5.4, 5.5, 5.6, 7.0',
  config:
   { app:
      { commented:
         '# The runtime the application uses. The \'type\' key defines the base container\n# image that will be used to run the application. There is a separate base\n# container image for each primary language for the application,\n# in multiple versions. Check the PHP documentation\n# (https://docs.platform.sh/languages/php.html#supported-versions)\n# to find the supported versions for the \'php\' type.\ntype: php:7.3',
        full: 'type: php:7.3',
        snippet: 'type: php:7.3' },
     routes: { commented: '', full: '', snippet: '' },
     services: { commented: '', full: '', snippet: '' } } }

PHP is a Runtime instance, whereas MariaDB is a Service instance. They each contain the same properties, some of which are built by pshregistry-parser and others will be a public subset of properties in the generated Registry. They only differ in the way they construct the template strings that are placed in their example configuration files.

Note: In the above example, registry.images.php is also valid, but not recommended generally. The keys of image objects use their type by default, and for certain images (i.e, "chrome-headless", "network-storage", "oracle-mysql", "persistent-redis") their type key includes a hyphen, which is not allowed for directly accessing object properties with registry.images.<image_type>.

The generated Registry will likely continue to use 'type' for image keys, so no workaround has been included to convert these examples into snakeCase. Feel free to use the other form if your use is restricted to a few images that you know will not run into this problem, but if you are going to be using every image that will include the examples above, stick to registry.images["image_type"].property.

There are certain special cases where configuration files for a service are very different than others, and in those cases the property for that image is an instance of its own class (e.g., Varnish, NetworkStorage).

Recommended versions

For each example configuration YAML file, a "recommended" supported version is chosen and used in each file where a version must be specified. In this case, the newest supported release is considered the recommended version, and is accessible with registry.images["elasticsearch"].recommended.

Accessing subsets of images

It is also possible to access a subset of the Registry's images, namely by accessing objects that include only those that are service or runtime images, with the properties registry.services and registry.runtimes. Individual runtime and service images can be accessed identically to images (e.g. registry.services["elasticsearch"]), as well as that individual image's properties (e.g. registry.services["elasticsearch"].recommended).

Future work

Use in the documentation

pshregistry-parser generates up-to-date example configuration YAML files that are currently pulled into the public documentation. Since 1.1.0, it generates additional markdown files that contain "Supported Version" tables for both service and runtime images that can also be included, that forego the necessity to create them during the doc's build process (aside from calling write()).

Currently, each image has the associated supportedHTML and deprecatedHTML properties meant to be pulled into the documentation at some point, specifically on each image's primary documentation page (e.g. Elasticsearch's supported and deprecated versions).

The adoption of pre-generated tables in 1.1.0 may influence how these supported/deprecated sections are handled moving forward. Namely, that another subdirectory is added (i.e. <saveDir>/versions or <saveDir>/lists) to the output of write() that writes files that contain these strings for each image (e.g. elasticsearch_supported.md), and that they are similarly included into the documentation.

They could include section headers (so that reference to an image's deprecated version's that does not have any will not generate that header despite the shortcode being included in the documentation) as well as descriptive text (I'm thinking specifically about the block of text commonly included in the "Deprecated" regarding upstreams). That way, each image's documentation page would look very similar:

# Elasticsearch

< Short description of Elasticsearch >

{% include "../../registry/images/lists/elasticsearch_supported.md" %}

{% include "../../registry/images/lists/elasticsearch_deprecated.md" %}

## Relationship

The format exposed in the `$PLATFORM_RELATIONSHIPS` [environment variable](/development/variables.md#platformsh-provided-variables):

{% codesnippet "https://examples.docs.platform.sh/relationships/elasticsearch", language="json" %}{% endcodesnippet %}

## Usage example

In your `.platform/services.yaml`:

{% codesnippet "/registry/images/examples/full/elasticsearch.services.yaml", language="yaml" %}{% endcodesnippet %}

In your `.platform.app.yaml`:

{% codesnippet "/registry/images/examples/full/elasticsearch.app.yaml", language="yaml" %}{% endcodesnippet %}

You can then use the service in a configuration file of your application with something like:

{% codetabs name="Java", type="java", url="https://examples.docs.platform.sh/java/elasticsearch" -%}

{%- language name="Node.js", type="js", url="https://examples.docs.platform.sh/nodejs/elasticsearch" -%}

{%- language name="PHP", type="php", url="https://examples.docs.platform.sh/php/elasticsearch" -%}

{%- language name="Python", type="py", url="https://examples.docs.platform.sh/python/elasticsearch" -%}

{%- endcodetabs %}

Generating the registry

Right now we will have to edit the registry.json by hand, but ideally this library will point at a "ground truth" repo somewhere else (that updates regularly) to get its data. When that happens, the registry.json file will be eliminated from docs altogether, and calling write() will also call an update() command to the ground truth registry.json in its final repo, sync up, and write the files using that data rather than a local source.

The decision that remains - that will influence how the final Registry is designed - is whether a project using pshregistry-parser places a call to:

• The registry.json within that repository (GitLab API call, requires a token) to make the request.
• A registry.json that is served by an application built by that repository connected to a Platform.sh project. In that case, keeping the Registry private (as it goes on to include more sensitive information about images/ repository locations) would require some other kind of authentication for that request.

Consequences

Currently, pshregistry-parser requires a local copy of the Registry and for that to be provided to the RegistryParser instance. The location of that file is used to determine the save locations of the generated files, unless an alternative saveDir is also passed to it.

Removing the ability to define a local registrySource at all is a logical option once this change is made, but then this save directory logic will need to be reworked.

At this stage, I'm thinking the following:

• registrySource as a parameter is disabled entirely, and instead the request to the generated source elsewhere is added to the constructor to define images.
• saveDir becomes the only parameter to RegistryParser, with a default value of null. From there, it wouldn't be necessary for projects that are interested in just accessing images.
• If generating files is the purpose of that project (i.e. the public docs), saveDir can be either passed to the constructor or with a new set method.