Nondeterministic Polynomial Munchies

    @style.tools/ngx-keyval

    1.2.3 • Public • Published

    Build Status Version npm version

    ngx-keyval - Nginx key/value store

    A simple high performance and scalable key/value store with TTL based on Nginx proxy_cache with a Node.js client that ads an optional extra in-memory cache layer. The solution provides an option to use Google Cloud Storage as backup.

    The Nginx key/value store can be used via a simple HTTP request. Nginx allows advanced security and authentication that could enable public usage in a web application.

    The store enables to set HTTP headers and a content-type so that keys can be accessed as regular file URLs with meta-data stored in headers, e.g. your-key-value-store.local/filename.json with header X-Metadata. A HEAD request enables to query just the meta-data of a key.

    The data can be compressed using gzip with the ability to let Nginx handle decompression. Alternatively, the accept-encoding request header enables to handle decompression in the client.

    Nginx proxy_cache supports gigabytes of data per key and millions of keys with optimal performance. It is possible to access data in keys using a byte-range request to return a small part of a gigabyte size key, with high performance (managed by Nginx).

    # get data 
    curl -D - http://your-keyvalue-store.local/key
     
    # store data with a 1 hour TTL, gzip compression and a custom content-type 
    curl -D - \
      -H "Content-Type: application/json" \
      -H "X-gzip: 1" \
      -X POST -d '{"value": "data", "ttl": 3600, "content-type": "text/html", "headers": { "X-Meta": "meta-data" } }' http://your-keyvalue-store.local/key
     
    # delete key 
    curl -D - -H "X-DELETE:1" http://your-keyvalue-store.local/key

    A Node.js client provides an easy API.

    const ngxKeyVal = require('@style.tools/ngx-keyval');
     
    // initiate key/value store
    const store = new ngxKeyVal.client({
        "server": "http://your-keyvalue-store.local/",
        "gzip": true // compress data by default
    });
     
    // get data
    let data = await store.get('key');
     
    // set data with an 1 hour expire time
    await store.put('key', 'data', 60 * 60);
     
    // delete key
    await store.del('key');
     
    // set meta data
    await store.put('key', 'data', 60 * 60, {
      headers: {
        "x-metadata": "data 123",
        "data-x": "other-data"
      }
    });
     
    // get meta data
    await store.meta('key', [
      'x-metadata',
      'data-x',
      'date',
      'content-type'
    ]);
     
    /** options example **/
     
    // get data with custom miss-ttl and HTTP headers
    let data = await store.get('key', {
       "miss-ttl": 3600, // cache non-existent key requests in Nginx for 1 hour
       "headers": {
          "X-Authenticate-Me": "secret"
       },
       "gzip": "raw" // return raw (uncompressed) gzip data
    });
     
     
    /** in-memory examples **/
     
    // set data with a custom content-type and a 10 seconds in-memory cache
    await store.put('key', 'data', 60 * 60, {
       "content-type": "application/json",
       "gzip": true, // store data with gzip compression
       "content-type": "text/html",
       "headers": {
          "X-Meta": "meta-data"
       }
    }, 10);
     
    // get data from memory (memory is not used by default)
    let data = await store.get('key', null, true);
     
    // get data from memory with a result verification function
    let data = await store.get('key', null, function(data) {
       
       // data retrieved from memory
     
       // modify result
       data.verified = 'OK';
       return data;
     
       // do not use cached result
       // return false;
    });
     
     
    /** Google Cloud Storage backup examples **/
     
    // set data with persistent storage in Google Cloud Storage
    await store.put('key', 'data', 60 * 60, false, true);
     
    // set data with persistent storage in a custom Google Cloud Storage bucket
    await store.put('key', 'data', 60 * 60, false, 'bucket-name');
     
    // get data with persistent storage fallback
    let data = await store.get('key', null, false, true);
     
    // delete data from persistent storage
    await store.del('key', null, false, true);

    Installation

    npm install @style.tools/ngx-keyval --save

    Step 1: setup the Nginx key/value store server

    The server configuration in server.conf provides an example. You may need to tune the settings of the proxy_cache, the server name and the security settings (IP restriction).

    Step 2: setup the Node.js key/value store management server

    The key/value store uses a Node.js server as a management controller that is used by Nginx as an upstream.

    const ngxKeyVal = require('@style.tools/ngx-keyval');
     
    // initiate key/value server
    const server = new ngxKeyVal.server({
        "port": 14451,
        "verbose": true,
        "default_ttl": false,
        "miss_ttl": 1,
        "max_size": "50mb",
        "default_content_type": "plain/text",
        "persist": {
            "type": "@google-cloud/storage",
            "auth": {
                "projectId": "optimization",
                "keyFilename": "service-account-key.json"
            },
            "bucket": "ngx-keyval-test",
            "upload_options": {
                "gzip": true
            },
            "enabled": "header"
        }
    });
     
    // start server
    server.start();

    You can start the server using forever.

    forever start --uid "ngx-keyval" -a /home/path/to/server.js

    Update the Nginx server configuration with the correct IP and port of the Node.js server.

    # key/val management server
    upstream ngx-keyval-server {
      server  127.0.0.1:14451;
    }

    Backup via Google Cloud Storage

    Nginx proxy_cache has a hard cache size limit and automatically removes least accessed entries when the cache limit is reached. The solution provides the option to use a Google Cloud Storage bucket as backup.

    To use the Google Cloud Storage bucket you need to configure persist parameter in the Node.js server configuration (see above).

    Retrieving/storing persistent data

    The Google Cloud Storage bucket can be enabled by default or based on a HTTP header.

    The parameter persist#enabled accepts three values:

    • false
    • always
    • header

    When set to header, it is required to set the x-persist:1 header in the GET or POST request to enable the Google Cloud Storage bucket. It is possible to define the bucket using a JSON value: x-persist:{"bucket": "bucket-name"}.

    Description

    The Nginx key/value server + Node.js client provides three cache layers for optimal performance and reliability.

    • in-memory cache (memory-cache) with an independent TTL
    • Nginx key/value server
    • Google Cloud Storage backup

    Nginx TTL management is fast and efficient and the server supports gigabytes of data with optimal performance.

    Bottleneck

    The Node.js management server is used for non-existent key requests and PUT requests. It is possible to define a TTL for non existent (404) keys, both on request level (x-miss-ttl header) and on server level, so that Nginx will handle the load of any GET request related traffic. For PUT request related traffic or when the server is to receive lots of traffic for non-existent keys with unique names, then the Node.js server can become a bottle neck.

    To overcome the Node.js bottleneck, it is possible to use a Google Cloud Function or a server pool as Node.js upstream. A Cloud Function can handle any traffic but introduces a latency (for non-existent keys and PUT requests only) and costs.

    # key/val management server
    upstream ngx-keyval-server {
      server us-central1-ngx-keyval-12345.cloudfunctions.net:443;
     
      # server pool
      # server server-2;
      # server server-3;
      # server server-4;
    }

    The functionality of the Node.js management server is very simple. It will merely return the data that is sent by Nginx and modify the cache headers. It is therefor easy to scale the server.

    Install

    npm i @style.tools/ngx-keyval

    Homepage

    style.tools/

    DownloadsWeekly Downloads

    1

    Version

    1.2.3

    License

    MIT

    Unpacked Size

    42.4 kB

    Total Files

    8

    Last publish

    Collaborators

    • optimization