warehouse.ai

    7.2.0 • Public • Published

    Version npm License npm Downloads Build Status Dependencies

    warehouse.ai

    Warehouse is a scalable object ledger and CDN management system for powering dynamic web applications.

    The system exposes two set of APIs: a generic Object API and a CDN management API separately. The user can use the CDN API to upload the desired assets while using the Object API to store the metadata of those assets (e.g., file urls, etc.).

    The Object API acts as generic ledger and it can be used to store any type of data. In particular, the Object API is good at storing versioned JSON data that may have one or more "head" version(s) for one or more "environment(s)"; for example, "production" and "staging".

    Warehouse can be used for different use cases in different ways. At GoDaddy we mainly use it for storing and serving our web applications' compiled browser-side assets. Our applications' workflow can be summarized as follows:

    1. Engineers create a new service or package version and release it (e.g., npm publish, git push <tag>)
    2. A CICD workflow is triggered and runs the builds (e.g., webpack)
    3. When all builds are completed, the CICD workflow uploads the assets to Warehouse using the CDN API
    4. Metadata for those assets on the CDN are stored in the Object API to allow application servers to fetch the URL(s) for the latest assets for a given environment.

    Get Started

    Warehouse is a Node.js web service running on Fastify. It stores data in Amazon's DynamoDB.

    Quick Start

    git clone git@github.com:godaddy/warehouse.ai.git
    cd warehouse.ai && npm install
    npm start

    Programmatic Usage

    If you need further customization of Warehouse such as authentication and/or custom logging, you may wrap it using a fastify application.

    const createFastify = require('fastify');
    const warehouse = require('warehouse.ai');
    
    const fastify = createFastify({
      logger: {
        level: 'info'
      }
    });
    
    // Optional authentication on Warehouse protected routes
    fastify.decorate('verifyAuthentication', function (req, res, done) {
      // Implement auth strategy here
      done();
    });
    
    fastify.register(warehouse);
    
    fastify.ready(function () {
      fastify.listen(fastify.config.port, function (err) {
        if (err) throw err;
      });
    })

    API

    Object API

    Get object

    GET /objects/:name

    Optional query parameters:

    {
      "accept_variants": "<variants>", // if not specified it returns all variants
      "version": "<version>", // if not specified it returns the latest version
      "env": "<env>" // if not specified it defaults to `production`
    }

    Create object

    POST /objects
    
    {
      "name": "<name>",
      "env": "<env>",
      "version": "<version>",
      "variant": "<variant>", // optional, default to _default
      "expiration": "<expiration>", // optional, default to never
      "data": "<data>"
    }

    Delete object

    DELETE /objects/:name/:env

    Delete object version

    DELETE /objects/:name/:env/:version

    Optional query parameters:

    {
      "variant": "<variant>", // if not specified deletes all variants
    }

    Get object head

    GET /head/:name/:env

    Update object head

    PUT /objects/:name/:env
    
    {
      "head": "<version>"
    }

    CDN API

    Upload new asset

    POST /assets
    
    <binary-data>

    Optional query parameters:

    {
      "expiration": "<expiration>", // if not specified, default to never
    }

    Binary data can be a specific file, or a tarball containing multiple files.

    To attach metadata to a file (or files) upload a tarball that contains both the file (or files) and a _metadata.json.

    For example, a tarball containing the following files:

    main.js
    main.css
    _metadata.json // { "main.js": {"foo":"bar"}, "main.css": {"beep":"boop"}}

    will make the API returning the following response:

    {
      "fingerprints": [
        "0c0383549608cf3b4b01e8f3d15f7ce8.gz",
        "a013ff249608cf3b4b01e7rta6388817.gz"
      ],
      "recommended": [
        "0c0383549608cf3b4b01e8f3d15f7ce8/main.js",
        "a013ff249608cf3b4b01e7rta6388817/main.css"
      ],
      "files": [
        {
          "url": "https://cdn.com/0c0383549608cf3b4b01e8f3d15f7ce8/main.js",
          "metadata": {
            "foo": "bar"
          }
        },
        {
          "url": "https://cdn.com/a013ff249608cf3b4b01e7rta6388817/main.css",
          "metadata": {
            "beep": "boop"
          }
        }
      ]
    }

    Delete asset

    DELETE /assets/:sha

    CLI

    Warehouse comes along with a CLI that can be used to interact with the API and perform all the various system operations: from uploading a file to the CDN, to create a new object version into a brand new environment.

    Upload new asset and create object in the ledger

    Option 1:

    wrhs upload /dev/myFilesFolder @org/service --env development --version 1.21.5 --variant en-US --expiration 365d

    Option 2:

    wrhs cdn upload /dev/myFilesFolder --expiration 365d | wrhs object create @org/service --env development --version 1.21.5 --variant en-US --expiration 365d

    Upload new asset

    To upload multiple files within a folder:

    wrhs cdn upload /dev/myFilesFolder --expiration 365d

    To upload a specific file:

    wrhs cdn upload /dev/myFile.ext --expiration 365d --metadata '{ "beep": "boop" }'

    Create object

    With data argument:

    wrhs object create @org/service --env development --version 1.21.5 --variant en-US --expiration 365d --data '{ "foo": "bar" }'

    With data pipe:

    echo -n '{ "foo": "bar" }' | wrhs object create @org/service --env development --version 1.21.5 --variant en-US --expiration 365d

    Get object

    All variants:

    wrhs object get @org/service --env development --version 1.21.5

    Specific variant:

    wrhs object get @org/service --env development --version 1.21.5 --accept-variants pt-BR,pt,en-US

    Using --accept-variants flag, it returns the highest priority variant found. If no accept-variants are found, a 404 is returned.

    Update object head

    Promotion simply moves the object version head creating a record in the object history. Promotion happens for all the object variants at once.

    wrhs object set-head @org/service --env development --version 1.21.6

    Rollback object head

    wrhs object rollback-head @org/service --env development --version 1.21.4

    If --version argument is omitted, Warehouse will rollback to the previous version.

    Development

    Warehouse uses Localstack and Docker for local development.

    To properly run the application locally, open one terminal session and run:

    npm run localstack
    

    Then, open another terminal session and run:

    npm run init-localstack
    npm run dev
    

    Install

    npm i warehouse.ai

    DownloadsWeekly Downloads

    43

    Version

    7.2.0

    License

    MIT

    Unpacked Size

    398 kB

    Total Files

    40

    Last publish

    Collaborators

    • jgowdy
    • ibrandao
    • decompil3d
    • jpage
    • kinetifex
    • rxmarbles
    • kquerna
    • 3rdeden
    • msluther
    • jwilhelm
    • jcrugzz
    • dcousineau
    • swaagie
    • indexzero
    • sivanmehta
    • ywaaang
    • fritzmonkey
    • wjarvis
    • cfellin