squared-functions

4.11.0 • Public • Published

squared-functions 4.11

  • NodeJS - 10 (Minimum)
  • Chrome - 12 (Recommended)

Examples use squared 3/4 although the concepts can be used similarly with any NodeJS application. Using squared or squared-express is not required.

NOTE: Feature lock is in effect for squared-functions other than extension compatibility. Any further development will be under E-mc.

Protocols

  • HTTP/2
  • FTP/SFTP/BitTorrent/Metalink - aria2 (apt/brew install aria2)
  • Unix domain socket - { socketPath: "/tmp/static0.sock", uri: "file:///path/filename" }

Document

Inline bundling options are available with these HTML tag names.

  • saveAs: html + script + link + raw assets
  • exportAs: script + style
  • exclude: script + link + style

Files with the same path and filename will automatically create a bundle assuming there are no conflicts in call ordering.

More advanced configurations are possible using a JSON/YAML external configuration file. Inline commands are usually sufficient for simple web pages.

NOTE: MIME (mimeType) is required when bundling torrents.

// HTML configuration using YAML is supported

{
  "selector": "head > script:nth-of-type(2), head > script:nth-of-type(3)",
  "type": "js",
  "saveAs": "js/modules2.js"
}

JS and CSS files can be bundled together with the "saveAs" or "exportAs" action. Multiple transformations per bundle can be chained using the "+" symbol.

+ saveAs: location | ~ // Same
+ exportAs: location

- ::
- format (chain "+")

These are the available option modifiers:

* preserve (true)
  - prevent unused styles from being deleted
    + html
    + css
* inline (true)
    + js: Rendered inline with <script>
    + css: Rendered inline with <style>
    + image: Rendered as base64 from file
* module (true | false - config | no-module - inline)
    + js: ESM
    + css: SASS
* extract (true)
    + css: @import rules are inlined into parent file (same origin)
* blob (true)
    + image: HTML and CSS from base64
    + font: CSS from base64
* dynamic (true)
    + image: srcset uses query parameters with an image resizer
    + element (non-void): mixed content which uses a view engine template (e.g. ejs)
* compress (array)
    + image: imagemin
    + font: woff + woff2
    + png: TinyPNG service (jpeg + webp)
    + gz: Gzip
    + br: Brotli
* static: (true)
    + all: Remove query string from URL
* crossorigin (true)
    + all: Same as preserveCrossOrigin [download: false]
* download (true - explicit | false)
    + all: Source files that are not usually downloaded (e.g. link[rel=alternate])
* hash
  - filename + content hash (productionRelease=true)
  - value can be limited to the starting prefix (e.g. md5[8] - Minimum is 4)
    + js
    + css
    + map
    + img
    + font
    + audio
    + video
* remove
    + all: Remove element during finalization
- charset (string)
    + utf-8 | utf-16 | utf-16le | latin1 | utf-16be (unsupported) [default: utf-8]

NOTE: Whitespace can be used between anything for readability.

<link rel="stylesheet" href="css/dev.css" data-chrome-file="saveAs:css/prod.css::beautify" data-chrome-options="preserve|md5|compress[gz]|utf-16">
<style data-chrome-file='{ "exportAs": "css/prod.css", "process": ["minify", "beautify"] }' data-chrome-options='{ "preserve": true, "hash": "md5", "compress": [{ "format": "gz" }], "encoding": "utf-16" }'>
  body {
    font: 1em/1.4 Helvetica, Arial, sans-serif;
    background-color: #fafafa;
  }
</style>
<script src="/dist/squared.js" data-chrome-file="saveAs:js/bundle1.js::minify" data-chrome-metadata='{ "custom-rollup": { "import-maps": {} }, "custom-terser": { "config": {} } }'></script>
<script src="/dist/squared.base.js" data-chrome-file="saveAs:js/bundle1.js"></script>
<script src="/dist/chrome.framework.js" data-chrome-file='{ "saveAs": "js/bundle2.js", "process": ["lint"] }'></script>

Built-In plugins

JS and CSS files can be optimized further using these settings:

  • beautify
  • lint
  • minify
  • es5 (Babel)
  • es5-minify (UglifyJS)
  • minify-svg (svgo)
  • compile (sass)
  • custom name

You can define or undefine your own optimizations in squared.json:

html

css

js

These listed plugins (npm i package) can be configured using a plain object (chrome.settings.transform). Other non-builtin transpilers can similarly be applied and chained by defining a custom function.

  • NPM package
  • Local file using module.exports (e.g. ".cjs")
  • Local plain file with single function (e.g. ".js")
  • Inline function

More advanced plugins can be written and installed through NPM. The only difference is the context parameter is set to the Document module.

Examples can be found in the "@squared-functions/document/packages" folder.

// squared.json

{
  "document": {
    "chrome": {
      "handler": "@squared-functions/document/chrome",
      "eval": {
        "function": true, // Enable inline functions
        "absolute": false, // Enable absolute paths to local files
        "template": false, // Enable external template functions
        "userconfig": false // Enable functions inside local files from user queries
      },
      "settings": {
        "directory": {
          "package": "../packages" // Override built-in plugins (base directory + users/username? + rollup.js)
        },
        "transform": {
          "html": {
            "posthtml": { // Built-in transformer
              "transform": {
                "plugins": [
                  ["posthtml-doctype", { "doctype": "HTML 5" }], // Plugins have to be pre-installed from NPM
                  ["posthtml-include", { "root": "./", "encoding": "utf-8" }]
                ]
              },
              "transform-output": {
                "directives": [
                  { "name": "?php", "start": "<", "end": ">" }
                ]
              }
            },
            "prettier": {
              "beautify": {
                "parser": "html",
                "printWidth": 120,
                "tabWidth": 4
              }
            }
          },
          "js": {
            /* Inline asynchronous function */
            "terser": { // npm i terser
              "minify-example": "async function (terser, value, options, require) { return await terser.minify(value, options.outputConfig).code; }", // this = NodeJS.process
              "minify-example-output": {
                "keep_classnames": true // "minify-example-output" creates variable "options.outputConfig"
              }
            },
            "@babel/core": {
              "npm-example": "npm:babel-custom", // function(@babel/core, value, options) (npm i babel-custom)
              "npm-example-output": "npm:babel-custom-output", // Configuration object (npm i babel-custom-output)
              /* OR */
              "npm-example-output": {
                "presets": ["@babel/preset-env"]
              },
              "es5-example": "./es5.js" // Local file - startsWith("./ | ../")
            }
          },
          "css": {
            /* Built-in transformer */
            "postcss": {
              "transform": {
                "plugins": [
                  "autoprefixer",
                  ["cssnano", { preset: ["cssnano-preset-default", { "discardComments": false }] }]
                ]
              }
            },
            /* Inline synchronous function + Promise */
            "sass": { // npm i sass
              "sass-example": "function (sass, value, options, resolve, require) { resolve(sass.renderSync({ ...options.outputConfig, data: value }).css); }",
              "sass-example-output": {
                "outputStyle": "compressed",
                "sourceMap": true,
                "sourceMapContents": true
              }
            },
            "sass": { // function (sass, value, options)
              "transform": "./sass-local.cjs", // Uses module.exports (debuggable)
              "transform-output": { // options.outputConfig
                "sourceMap": true
              }
            },
            /* npm i sass-custom (public) */
            "sass-custom": { // function (context /* Document */, value, options)
              "transform": { // options.baseConfig
                "sourceMap": true
              }
            }
          }
        }
      }
    }
  }
}

NOTE: Script and settings files with ".cjs" extension (configurable) will be loaded with "require" when using squared-express.

interface TransformOutput {
    pathname?: string;
    filename?: string;
    mimeType?: string;
    sourceFile?: string;
    sourcesRelativeTo?: string;
    sourceMap?: SourceMapInput;
    metadata?: unknown;
    external?: PlainObject;
}

interface ITransformSeries extends Module, TransformOutput {
    type: "html" | "css" | "js";
    baseConfig: PlainObject;
    outputConfig: PlainObject; // Same as baseConfig when using an inline transformer
    sourceMap: SourceMap; // Primary sourceMap
    code: string;
    metadata: PlainObject; // Custom request values and modifiable per transformer
    productionRelease: boolean;
    supplementChunks: ChunkData[];
    imported: boolean; // ESM detected
    createSourceMap(code: string): SourceMap; // Use "nextMap" method for sourceMap (additional sourceMaps)

    /* ESM */
    getMainFile?(code?: string, imports?: StringMap): Undef<SourceInput<string>>;
    getSourceFiles?(imports?: StringMap): Undef<SourceInput<[string, string?, string?][]>>;

    /* Return values */
    out: {
        sourceFiles?: string[]; // ESM
        ignoreCache?: boolean;
        messageAppend?: string;
        logAppend?: LogStatus[];
        logQueued?: LogStatus[];
    };

    /* Deprecated */
    outSourceFiles?: string[];
    outIgnoreCache?: boolean;
    outMessageAppend?: string;
    outLogAppend?: LogStatus[];
}
// es5.js
function (context, value, options, resolve, require) {
  context.transform(value, options.outputConfig, function (err, result) {
    resolve(!err && result ? result.code : "");
  });
}

// es5.cjs
const path = require('path');

module.exports = async function (context, value, options) {
  return await context.transform(value, options.outputConfig).code;
}

The same concept can be used inline anywhere using a <script> tag with the type attribute set to "text/template". The script template will be completely removed from the final output.

// "es5-example" is a custom identifier (chrome -> eval_template: true)

<script type="text/template" data-chrome-template="js::@babel/core::es5-example">
async function (context, value, options, require) {
  const options = { ...options.outputConfig, presets: ["@babel/preset-env"], sourceMaps: true }; // https://babeljs.io/docs/en/options
  const result = await context.transform(value, options);
  if (result) {
    if (result.map) {
      options.sourceMap.nextMap("babel", result.code, result.map);
    }
    return result.code;
  }
}
</script>

External configuration

JSON (json/js) configuration is provided for those who prefer to separate the bundling and transformations from the HTML. Any assets inside the configuration file will override any settings either inline or from JavaScript.

interface OutputModifiers {
    inline?: boolean; // type: js | css | base64: image | font
    blob?: boolean; // type: image | font (base64)
    preserve?: boolean; // type: html | css | cross-origin: append/js | append/css
    extract?: boolean; // type: css
    module?: boolean; // type: js | css (ESM)
    static?: boolean; // Removes URL search params
    dynamic?: boolean; // type: image (srcset) | element (non-void)
    remove?: boolean; // Removes element from HTML page
    ignore?: boolean;
    exclude?: boolean; // type: js | css (remove from HTML)
}

interface AssetCommand extends OutputModifiers {
    selector: string;
    mergeType?: "none" | "over" | "under"; // Use when different selectors target same element

    saveAs?: string; // js | css | image
    exportAs?: string; //js | css
    saveTo?: string; // image | video | audio (transforms create multiple files and are given generated filename)
    hash?: "md5" | "sha1" | "sha224" | "sha256" | "sha384" | "sha512"; // md5[8] will shorten hash to the first 8 characters

    pathname?: string; // alias for "saveTo"
    filename?: string; // pathname + filename = "saveAs"
    process?: string[]; // js | css | svg
    commands?: string[]; // image
    download?: boolean; // Same as preserveCrossOrigin (default is "true")
    cloudStorage?: CloudService[];
    tasks?: string[];
    watch?: boolean | { interval?: number, expires?: string }; // type: js | css | image (expires: 1h 1m 1s)
    attributes?: ObjectMap<Optional<string>>;
    template?: {
        module: string;
        identifier?: string;
        value?: string;
    };

    dynamic?: boolean; // Will ignore misplaced child elements prerendered in the browser

    incremental: false | "none" | "etag" | "exists"; // Will override batch request.incremental
    incremental: true; // Will override request.incremental = false

    metadata?: PlainObject; // Custom values passed to transformers
    metadata: { __sourcemap__: "inline" }; // System post processing command

    type: "html" | "js" | "css" | "data"; // Script templates (optional)
    type: "append/js" | "append/css" | "append/[tagName]"; // Includes "prepend"

    type: "text" | "attribute" | "display"; // dynamic is valid only with "text"
    dataSource?: CloudDatabase; // "cloud" (source)
    dataSource?: {
        source: "mongodb" | "redis" | "mysql" | "postgres" | "uri" | "local" | "export";
        postQuery?: string;
        preRender?: string;
        whenEmpty?: string;
    };

    type: "replace";
    textContent: string; // Replace element.innerHTML

    document?: string | string[]; // Usually "chrome" (optional)
}

Only one command per element is supported (except data sources) with the last selector taking precedence except when mergeType is defined.

// sqd.config

/* Glob index.html* - ordinal 1 */
[
  {
    "selector": "img#picture1",
    "commands": ["png@"]
  },
  {
    "selector": "img#picture2",
    "commands": ["webp%"]
  }
]

/* Glob index.html?output=prod - ordinal 2 */
{
  "selector": "img",
  "mergeType": "under",
  "hash": "sha256", // Merged
  "commands": ["jpeg@"] // No effect
}
/* OR */
{
  "selector": "img",
  "mergeType": "over",
  "hash": "sha256", // Merged
  "commands": ["jpeg@"] // All images will be JPEG
}
/* OR */
{
  "selector": "img",
  "mergeType": "none",
  "hash": "sha256", // No effect when "dev" is present
  "commands": ["jpeg@"] // Same
}
squared.saveAs("bundle.zip", { config: { uri: "http://localhost:3000/chrome/bundle.yml", mimeType: "text/yaml" } }); // "mimeType" (optional)
squared.saveAs("bundle.zip", { config: "http://localhost:3000/chrome/bundle.yml" });

// http://hostname/example.html -> http://hostname/example.html.json
squared.saveAs("example.zip", { config: { mimeType: "json" } });
squared.saveAs("example.zip", { config: "json" }); // json | yml | yaml

Here is the equivalent page using only inline commands with "data-chrome-file" and "data-chrome-tasks".

Cloud storage

Manual installation of the SDK is required including an account with at least one of these cloud storage provider.

* Amazon
  - https://aws.amazon.com/free (5GB - 12 months)

  + npm i aws-sdk (aws)
    <!-- OR -->
  + npm i @aws-sdk/client-s3 (aws-v3)

* Microsoft
  - https://azure.microsoft.com/en-us/free (5GB - 12 months)

  + npm i @azure/storage-blob (azure/az)

* Google
  - https://cloud.google.com/free (5GB - US)
  - https://firebase.google.com/products/storage (Cannot create new buckets - Use CLI)

  + npm i @google-cloud/storage (gcp/gcloud)
    <!-- OR -->
  + npm i firebase

* IBM
  - https://www.ibm.com/cloud/free (25GB)
  - Uses S3 compatibility API (v2)

  + npm i ibm-cos-sdk (ibm/ibm-v1)

* Oracle
  - https://www.oracle.com/cloud/free (10GB standard + up to 30GB)
  - Uses S3 compatibility API (v2)
  - Cannot create new public buckets

  + npm i aws-sdk (oci)

* MinIO
  - https://min.io/download (Local)
  - S3 compatible object storage

  + npm i minio (minio)

Environment variables can also be used for authorization.

* AWS
  - AWS_ACCESS_KEY_ID
  - AWS_SECRET_ACCESS_KEY
  - AWS_SESSION_TOKEN

* Azure
 - AZURE_TENANT_ID
 - AZURE_CLIENT_ID
 - AZURE_CLIENT_SECRET
 - npm i @azure/identity

* Azure (Unofficial)
  - AZURE_STORAGE_ACCOUNT
  - AZURE_STORAGE_KEY
  - AZURE_STORAGE_CONNECTION_STRING
  - AZURE_STORAGE_SAS_TOKEN
  - process.env.apply (squared.json)

* GCP
  - GOOGLE_APPLICATION_CREDENTIALS
  - npm i firebase-admin

* Minio
  - MINIO_ACCESS_KEY
  - MINIO_SECRET_KEY
  - MINIO_SESSION_TOKEN
  - MINIO_ENDPOINT
  - process.env.apply (squared.json)
// Optional fields are supported by all services

{
  "selector": "#picture1",
  "commands": [
    "png(100x200){90,180,270}" // Uploaded with UUID filename
  ],
  "cloudStorage": [
    {
      "service": "aws",
      "bucket": "squared-001",
      "credential": {
        "accessKeyId": "**********",
        "secretAccessKey": "**********",
        "region": "us-west-2",
        "sessionToken": "**********" // Optional
      },
      "credential": "main", // OR: Load host configuration from settings at instantiation
      /* Optional */
      "admin": {
        "publicRead": true, // Bucket + uploaded objects
        /* OR */
        "publicRead": 1, // Bucket only (s3::ListBucket*)
        /* OR */
        "acl": "public-read" | "public-read-write" | "authenticated-read" | "private", // Will modify existing bucket (equivalent policy) + uploaded files (canned ACL)

        "configBucket": {
          /* Newly created (during upload) */
          "create": {
            "ACL": "public-read", // CreateBucketRequest (https://docs.aws.amazon.com/AmazonS3/latest/userguide/acl-overview.html#canned-acl)
            "CreateBucketConfiguration": {
              "LocationConstraint": "us-west-1"
            }
          },
          /* Modify bucket policy (after upload) */
          "policy": {
            "Policy": "", // PutBucketPolicy (except IBM + OCI)
            /* OR */
            "ACL": "authenticated-read", // PutBucketAcl (except OCI)
            /* OR */
            "PublicAccessBlockConfiguration": { // PutPublicAccessBlock
              "BlockPublicAcls": false,
              "BlockPublicPolicy": false,
              "IgnorePublicAcls": false,
              "RestrictPublicBuckets": false
            }
          }
        }
      },
      /* Optional */
      "upload": {
        "active": false, // Separate from build (background) + Will not overwrite ACL
        "active": true, // Delays build until upload is complete
        "active": true, // Rewrites "src" to cloud storage location + Will overwrite ACL (public-read)
        "localStorage": false, // Remove current file from archive or local disk
        "filename": "picture1.webp", // Choose a different bucket filename
        "all": false, // Include transforms
        "overwrite": false, // Always use current filename
        "contentType": "application/octet-stream", // options.ContentType with primary has highest precedence

        "publicRead": true, // Will overwrite primary options.ACL (except OCI + Firebase)
        /* OR */
        "acl": "authenticated-read" | "aws-exec-read" | "bucket-owner-full-control" | "bucket-owner-read" | "private" | "public-read" | "public-read-write" // ObjectCannedlACL (optional)

        /* Optional - PutObjectRequest */
        "options": {
          "ContentType": "text/html", // Primary object only
          "ACL": "private", // All objects
          "Metadata": {} // All objects (primary + source maps + transforms)
        },
        "metadata": {
          "Content-Type": "text/html; charset=UTF-8", // Full replacement with primary only
          "Content-Encoding": "gzip",
          "Expires": "Wed, 21 Oct 2015 07:28:00 GMT"
        }
      },
      /* Optional */
      "download": {
        "filename": "picture2.png", // Required
        "versionId": "12345", // Retrieve a previous file snapshot
        "pathname": "download/images", // File adjacent or base directory when omitted (overrides "preservePath")
        "active": false, // Separate from build (background)
        "active": true, // Always write file or rename to main file when same extension
        "waitStatus": false, // Delay transaction until files are completely downloaded
        "overwrite": false, // Always write file
        "deleteObject": false, // Remove after successful download
        "deleteObject": {} // DeleteObjectRequest - All
      }
    },
    {
      "service": "azure", // OR: az
      "bucket": "squared-002",
      "credential": {
        "accountName": "**********", // +1 password option (required)
        "accountKey": "**********",
        "connectionString": "**********",
        "sharedAccessSignature": "**********",
        "computeHMACSHA256": "function (stringToSign) { *** require not available *** }" // Inline functions from a POST request
      },
      "admin": {
        "publicRead": true,
        /* OR */
        "acl": "blob" | "container",

        "configBucket": {
          "create": {
            "metadata": {} // ContainerCreateOptions
          }
        }
      },
      "upload": {
        "publicRead": false, // Not supported
        "acl": "", // Not supported

        "pathname": "a/b/c/", // Virtual directory in bucket (overrides "preservePath")
        "endpoint": "http://squared.azureedge.net/squared-002", // e.g. CDN

        /* Optional - BlockBlobUploadOptions */
        "options": {
          "blobHTTPHeaders": {
            "blobContentType": "text/html"
          },
          "metadata": {}
        }
      }
    },
    {
      "service": "gcp", // OR: gcloud
      "bucket": "squared-003", // UUID generated when omitted (optional)
      "credential": {
        "keyFilename": "./gcp.json", // Path to JSON credentials (Google Cloud)
        /* OR */
        "projectId": "squared", // Firebase
        "apiKey": "**********",
        "authDomain": "<project-id>.firebaseapp.com",
        "product": "firebase" // Required with using GOOGLE_APPLICATION_CREDENTIALS
      },
      "admin": {
        "publicRead": true, // New buckets (except OCI + Firebase)
        /* OR */
        "acl": "private" | "projectPrivate" | "authenticatedRead" | "publicRead" | "publicReadWrite" | "bucketAccessUniform" | "bucketAccessACL", // Will call setBucketPolicy after createBucket

        "emptyBucket": false, // More convenient than using "overwrite"
        "configBucket": {
          "create" {
            "location": "", // CreateBucketRequest (using credential for configuration is deprecated)
            "storageClass": ""
          },

          /* MakeBucketPrivateOptions - https://cloud.google.com/nodejs/docs/reference/storage/latest/storage/acl */
          "policy": {
            "acl": "private", // makePrivate + includeFiles + projectPrivate
            "acl": "projectPrivate", // makePrivate + allUsers (delete) + allAuthenticatedUsers (delete)
            "acl": "authenticatedRead", // projectPrivate + allAuthenticatedUsers:READER
            "acl": "publicRead", // makePublic + includeFiles
            "acl": "publicReadWrite", // publicRead + allUsers:WRITER
            "acl": [{ "entity": "allUsers", "role": "READER" } /* add */, { "entity": "allAuthenticatedUsers" } /* delete */] // Custom
            /* Unofficial aliases */
            "acl": "bucketAccessUniform", // Enable uniform bucket-level access
            "acl": "bucketAccessACL", // Revert uniform bucket-level access (within 90 days)
          },
          "website": {
            "indexPage": "index.html", // Optional (chrome: "true" for HTML page only)

            "errorPage": "404.html",
            /* OR: azure */
            "indexPath": "home.html", // Bucket name is included
            "errorPath": "errors/404.html"
          }
        },
        "preservePath": false // Use current pathname as base directory
      },
      "upload": {
        "publicRead": true, // Will not clobber existing ACLs
        "publicRead": 0, // Remove ACL without affecting other ACLs (GCP only)
        /* OR */
        "acl": "authenticatedRead" | "bucketOwnerFullControl" | "bucketOwnerRead" | "private" | "projectPrivate" | "publicRead", // PredefinedAcl (optional)

        "active": false, // Implicity "publicRead: true" except when explicitly "publicRead: false"

        /* Optional - UploadOptions */
        "options": {
          "contentType": "text/html",
          "predefinedAcl": "publicRead" // Supplementary are public

          "metadata": {}, // UploadMetadata
          /* OR */
          "customMetadata": {} // Firebase
        }
      }
    },
    {
      "service": "ibm",
      "bucket": "squared-004",
      "credential": {
        "apiKeyId": "**********",
        "serviceInstanceId": "**********",
        "region": "us-south",
        "endpoint": "https://s3.us-south.cloud-object-storage.appdomain.cloud", // Same as region (optional)
      },
      "admin": {
        "configBucket": { /* Same as AWS */ }
      },
      "upload": { /* Same as AWS */ }
    },
    {
      "service": "oci",
      "bucket": "squared-005", // New buckets are private when using S3 API
      "credential": {
        "region": "us-phoenix-1",
        "namespace": "abcdefghijkl",
        "accessKeyId": "**********",
        "secretAccessKey": "**********",
      },
      "admin": {
        "configBucket": { /* Same as AWS */ }
      },
      "upload": { /* Same as AWS */ }
    },
    {
      "service": "minio", // https://docs.min.io/minio/baremetal
      "bucket": "squared-006",
      "credential": {
        "accessKey": "**********",
        "secretKey": "**********",
        "endPoint": "127.0.0.1",
        "port": 9000, // Required
        "useSSL": false, // Required with "http"
        "region": "us-west-1" // Default is "us-east-1"
      },
      "admin": {
        "publicRead": true, // Will create a compatible S3 "public-read" policy
        /* OR */
        "acl": "readonly" | "writeonly" | "readwrite", //  // Will create a compatible MinIO policy

        "emptyBucket": true,
        "recursive": false, // Used with emptyBucket (default is "true")

        "configBucket": {
          "policy": "readonly" | "writeonly" | "readwrite", // Canned ACL (https://min.io/docs/minio/linux/administration/identity-access-management/policy-based-access-control.html)
          "policy": "public-read", // Same as AWS
          "policy": {
            "Version": "2012-10-17",
            "Statement": [{
              "Sid": "PublicRead",
              "Effect": "Allow",
              "Principal": {
                "AWS": ["*"]
              },
              "Action": ["s3:GetObject"],
              "Resource": ['arn:aws:s3:::bucketName/*']
            }
          }
        }
      },
      "upload": {
        "publicRead": true, // S3 request header "x-amz-acl" to "public-read"
        /* OR */
        "acl": "authenticated-read" | "aws-exec-read" | "bucket-owner-full-control" | "bucket-owner-read" | "private" | "public-read" | "public-read-write", // S3 Object Canned ACL

        "endpoint": "http://squared.min.io/squared-006", // Required when different from credential

        /* Optional - ItemBucketMetadata */
        "options": {
          "Content-Type": "image/webp" // Supplementary objects
        },
        "metadata": {
          "Content-Type": "image/png" // Primary object
        }
      }
    }
  ]
}

Creating a NPM scoped package with action function handlers can be used to perform cloud transactions.

// ICloudServiceClient

@oci-scoped/client
@oci-scoped/upload (optional)
@oci-scoped/download (optional)

{
  "cloudStorage": [
    {
      "service": "@oci-scoped"

      /* Same as above */
    }
  }
}

Samples can be found in the @squared-functions/cloud directory.

squared.saveAs("index.zip", {
  config: "http://localhost:3000/chrome/bundle.yml",
  saveAs: {
    html: {
      cloudStorage: [{ // Create static website
        service: "aws-v3",
        bucket: "squared-001",
        credential: {
          credentials: { // Preferred
            accessKeyId: "**********", // Only access key logins are supported with v3
            secretAccessKey: "**********",
            sessionToken: "" // Optional
          },
          /* ...accessKeyId = Will be copied into "credentials" */
          region: "us-west-2"
        },
        upload: {
          active: true,
          endpoint: "https://squared-001.s3.us-west-2.amazonaws.com", // Optional
          overwrite: true
        }
      }]
    },
    image: { // Non-element images using url() method
      cloudStorage: [{
        service: "aws",
        bucket: "squared-001",
        settings: "main",
        upload: {
          active: true
        }
      }]
    }
  }
});

Data Source

Static content can be generated using an AssetCommand with the "dataSource" property to perform basic text and attribute replacement.

Cloud

Each DocumentDB provider has a different query syntax. Consulting their documentation is recommended if you are writing advanced queries.

* Amazon DynamoDB
  - https://aws.amazon.com/dynamodb (25GB + 25 RCU/WCU)

  + npm i aws-sdk (aws)
    <!-- OR -->
  + npm i @aws-sdk/client-dynamodb (aws-v3)
  + npm i @aws-sdk/lib-dynamodb

* Microsoft Cosmos DB
  - https://azure.microsoft.com/en-us/services/cosmos-db (5GB + 400RU/s)

  + npm i @azure/cosmos (azure/az)

* Google Firestore / Datastore / BigQuery / Realtime Database / Bigtable / Spanner
  - https://cloud.google.com/firestore (1GB + 50K/20K r/w@day)
  - https://cloud.google.com/bigquery (10GB + 1TB queries/month)
  - https://firebase.google.com/products/realtime-database (1GB + 10GB)
  - https://cloud.google.com/bigtable (Paid)
  - https://cloud.google.com/spanner (Paid)

  + npm i @google-cloud/firestore (gcp/gcloud)
  + npm i @google-cloud/datastore
  + npm i @google-cloud/bigquery
  + npm i @google-cloud/bigtable
  + npm i @google-cloud/spanner
  + npm i firebase
  + npm i firebase-admin (optional)

* IBM Cloudant
  - IBM: https://www.ibm.com/cloud/cloudant (1GB + 20/10 r/w@sec)

  + npm i @cloudant/cloudant (ibm) [Deprecated]
    <!-- OR -->
  + npm i @ibm-cloud/cloudant (ibm-v1)

* Oracle Autonomous DB
  - https://www.oracle.com/autonomous-database (20GB)
  - https://www.oracle.com/autonomous-database/autonomous-json-database (Paid - 1TB)

  + npm i oracledb (oci)

* MongoDB Atlas
  - https://www.mongodb.com/atlas/database
  - URI authentication only when using MongoDB v3

  + npm i mongodb (atlas)

Environment variables can also be used for authorization.

* AWS
  - AWS_ACCESS_KEY_ID
  - AWS_SECRET_ACCESS_KEY
  - AWS_SESSION_TOKEN
  - AWS_REGION

* Azure (Unofficial)
  - AZURE_COSMOS_ENDPOINT
  - AZURE_COSMOS_KEY
  - process.env.apply (squared.json)

* GCP
  - GOOGLE_APPLICATION_CREDENTIALS
  - npm i firebase-admin

* IBM
  - CLOUDANT_URL
  - CLOUDANT_APIKEY
  - CLOUDANT_USERNAME
  - CLOUDANT_PASSWORD
  - serviceName = CLOUDANT
interface CloudDatabase {
    source: "cloud";
    name?: string;
    table?: string; // Required except with BigQuery + Realtime Database
    id?: string;
    query?: string | PlainObject | any[];
    value?: string | ObjectMap<string | string[]>; // Uses innerHTML for replacement when undefined
    index?: number;
    limit?: number;

    ignoreEmpty?: boolean; // Do not interpret empty results

    template?: string; // chrome.settings.directory.template (base directory + users/username?)
    encoding?: BufferEncoding; // utf-8 (default)

    params?: unknown[]; // Client params
    options?: PlainObject; // Client config (Coercible: Date | RegExp | URL | function)

    partitionKey?: string; // AWS + Azure + IBM + OCI

    // Function callback
    postQuery?: string | Function; // (ResultArray[], dbObject, require)
    preRender?: string | Function; // (string, dbObject, require)
    whenEmpty?: string | Function; // (EmptyArray[], dbObject, require)

    viewEngine?: {
        name: string; // NPM package name
        singleRow?: boolean; // Template data is sent in one pass using an Array[]
        outputEmpty?: boolean; // Pass empty results to template engine
        options?: {
            compile?: PlainObject; // template = engine.compile(value, options)
            output?: PlainObject; // template({ ...options, ...result[index] })
        };
    };

    // Caching - user@server/database
    credential: {
        uuidKey?: string; // Faster and better caching (UUID v4)
    };
    ignoreCache?: boolean | 0 | 1; // 0 - reset cache expiration | 1 - purge current + store latest
}

You can also use named callbacks for "postQuery" and "preRender" anywhere inside the HTML. It is more readable than inside a configuration file and can be reused for queries with the same mapping or formatting.

Instructions for coercing string values into native objects can be found in the MongoDB section.

NOTE: Using "template" (external) is the same as "value" (inline) except the reusable content is stored inside a local file inside a predefined template directory.

interface CloudDatabase {
    /* AWS */
    key?: Key;
    update?: UpdateItemInput | UpdateCommandInput /* v3 */; // Uses Key object or partitionKey/id (precedes GetItemInput during read operation)

    /* Azure */
    update?: PatchRequestBody; // partitionKey + id (JSON Patch: http://jsonpatch.com/)

    /* GCP */
    update?: StandardMap | string; // Firestore (id + single) | Firebase (query + multiple) | Spanner (DML)

    /* IBM-v1 */
    update?: PostDocumentParams; // Uses "postDocument" (id)

    /* OCI */
    update?: StandardMap; // Uses "replaceOne" (id)

    /* Atlas */
    update?: UpdateFilter<Document>; // Update document during a read operation
}

Update occurs during a get operation and will return the item with the latest changes only if it exists.

// "postQuery-example" is a custom identifier (chrome -> eval_template: true)

<script type="text/template" data-chrome-template="data::postQuery-example">
async function (items, dbObject) { // items - PlainObject[]
  if (items.length) {
    return await fetch("/db/url", { method: "POST", body: JSON.stringify(items) }).then(result => result.map(item => ({ name: item.key, value: item.value })));
  }
  return null; // "items" will display unmodified when not an array
}
</script>

<script type="text/template" data-chrome-template="data::preRender-example">
function (value, dbObject) { // value - string
  return value.replaceAll("<", "&lt;");
}
</script>

<script type="text/template" data-chrome-template="data::whenEmpty-example">
function (result, dbObject) { // result - PlainObject[]
  result[0] = { value: "Empty" }; // Array.length is 0
}
</script>

View engines with a "compile" template string to function (e.g. EJS) can be used instead for "text" and "attribute". Results from any data source are treated as an array with multiple rows being concatenated into one string.

/* AWS: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GettingStarted.NodeJs.html */
{
  "selector": "main p",
  "type": "text",
  "dataSource": {
    "source": "cloud",
    "service": "aws",
    "credential": { // Coercible: Date | RegExp | function
      "accessKeyId": "**********",
      "secretAccessKey": "**********",
      "region": "us-east-1", // Endpoint specified (optional)
      "endpoint": "https://dynamodb.us-east-1.amazonaws.com" // Local development (required)
    },
    "table": "demo",
    "query": {
      "KeyConditionExpression": "#name = :value",
      "ExpressionAttributeNames": { "#name": "id" },
      "ExpressionAttributeValues": { ":value": "1" }
    },
    "limit": 1, // Optional
    "value": "<b>${title}</b>: ${description}", // Only one field per template literal (optional)
    /* OR */
    "value": "`<b>${this.title}</b>: ${this.description} (${this.index * 2})`" // Function template literal - (chrome.settings.eval.function)
  }
}

/* Azure: https://docs.microsoft.com/en-us/azure/cosmos-db/sql-query-getting-started */
{
  "selector": "main p",
  "type": "text",
  "dataSource": {
    "source": "cloud",
    "service": "azure",
    "credential": {
      "endpoint": "https://squared-001.documents.azure.com:443",
      "key": "**********"
    },
    "name": "squared", // Database name (required)
    "table": "demo",
    "partitionKey": "Pictures", // Optional
    "query": "SELECT * FROM c WHERE c.id = '1'", // OR: storedProcedureId + partitionKey? + params?
    "query": {
      "query": "SELECT * FROM c WHERE c.lastName = @lastName AND c.address.state = @addressState", // SqlQuerySpec
      "parameters": [
        { "name": "@lastName", "value": "Wakefield" },
        { "name": "@addressState", "value": "CA" }
      ]
    },
    "value": "<b>${__index__}. ${title}</b>: ${description}" // "__index__": Result row index value
  }
}

/* GCP: https://firebase.google.com/docs/firestore/query-data/queries */
{
  "selector": "main p",
  "type": "text",
  "dataSource": {
    "source": "cloud",
    "service": "gcp",
    "product": "firestore", // Only GCP (recommended)
    "credential": {
      "keyFilename": "./gcp.json",
      "projectId": "squared" // Optional
    },
    "table": "demo",

    "query": [
      ["group", "==", "Firestore"], ["id", "==", "1"] // API: where (required) (deprecated)
    ],
    /* endAt + endBefore + limit + limitToLast + offset + orderBy + select + startAfter + startAt + where + withConverter */
    "query": [
      ["where", "group", "==", "Firestore"], // Method name + arguments (required)
      ["where", "id", "==", "1"],
      ["limitToLast", 2],
      ["orderBy", "title", "asc"]
    ],

    "orderBy": [
      ["title", "asc"] // Optional
    ],

    "value": "{{if !expired}}<b>${title}</b>: ${description}{{else}}Expired{{end}}", // Non-nested single conditional truthy property checks

    /* golang - html/template comparison syntax */
    "value": "{{if not expired}}<b>${title}</b>: ${description}{{else}}Expired{{end}}", // Case sensitive
    "value": "{{if and (user.total) (ge user.total postMin) (lt user.total postMax)}}<b>${title}</b>: ${description}{{else if (eq user.total 0)}}Expired{{end}}" // Parenthesis and spaces between conditions are required (and/or/not)
  }
}

/* Datastore: https://googleapis.dev/nodejs/datastore/latest */
{
  "selector": "main p",
  "type": "text",
  "dataSource": {
    "source": "cloud",
    "service": "gcp",
    "product": "datastore",
    "credential": {
      "keyFilename": "./gcp.json",
      "projectId": "squared" // Optional
    },

    "keys": [["kind", "name"]], // PathType[] | KeyOptions | string (required)
    "limit": 5, // Optional
    "removeEmpty": false, // Optional

    "kind": "Task", // Optional
    /* end + filter + groupBy + hasAncestor + limit + offset + order + select + start */
    "query": [
      ["filter", "done", "=", false], // Method name + arguments (required)
      ["filter", "priority", ">=", 4],
      ["order", "priority", { "descending": true }]
    ],

    "options": {}, // RunQueryOptions (optional)

    "viewEngine": "ejs", // npm i ejs (https://ejs.co/#docs)
    "template": "sub_dir/ejs/content.ejs", // chrome.settings.directory.template (base directory + users/username?)
    "encoding": "utf-8" // Optional
  }
}

/* BigQuery: https://googleapis.dev/nodejs/bigquery/latest */
{
  "selector": "main p",
  "type": "text",
  "dataSource": {
    "source": "cloud",
    "service": "gcp",
    "product": "bigquery",
    "credential": {
      "keyFilename": "./gcp.json",
      "projectId": "squared" // Optional
    },
    "query": "SELECT name, count FROM `demo.names_2014` WHERE gender = 'M' ORDER BY count DESC LIMIT 10", // Required
    "value": "<b>${title}</b>: ${description}"
  }
}

/* Realtime Database: https://firebase.google.com/docs/database/web/lists-of-data#filtering_data */
{
  "selector": "main p",
  "type": "text",
  "dataSource": {
    "source": "cloud",
    "service": "gcp",
    "product": "firebase", // Recommended
    "credential": {
      "apiKey": "**********",
      "authDomain": "<project-id>.firebaseapp.com",
      "databaseURL": "https://<database-name>.firebaseio.com",
      "product": "firebase" // Required with using GOOGLE_APPLICATION_CREDENTIALS
    },

    "query": "path/to/ref",  // Required
    /* endBefore + endAt + equalTo + limitToFirst + limitToLast + orderByChild + orderByKey + orderByPriority + orderByValue + startAt + startAfter */
    "orderBy": [
      ["orderByChild", "path/to/child"], // Optional
      ["startAfter", 5, "name"],
      ["limitToFirst", 1 /* Leading path is removed */]
    ],

    "viewEngine": "ejs" // Using a view engine is recommended due to object structure
  }
}

/* Bigtable: https://googleapis.dev/nodejs/bigtable/latest */
{
  "selector": "main p",
  "type": "text",
  "dataSource": {
    "source": "cloud",
    "service": "gcp",
    "product": "bigtable",
    "credential": {
      "keyFilename": "./gcp.json",
      "projectId": "squared", // Optional
      "apiEndpoint": "localhost:8086" // Optional
    },

    "name": "squared", // Instance (required)
    "table": "demo", // Required

    "query": {}, // Overrides "filter" options attribute (optional)

    "id": "rowKey1", // Uses "get" [single] (required)
    "columns": ["column1", "column2"], // Optional
    "options": { filter: {} }, // GetRowOptions (optional)
    /* OR */
    "options": { filter: {} }, // Uses "getRows" [multiple] + GetRowOptions (optional)

    "value": "<b>${title}</b>: ${description}"
  }
}

/* Spanner: https://googleapis.dev/nodejs/spanner/latest */
{
  "selector": "main p",
  "type": "text",
  "dataSource": {
    "source": "cloud",
    "service": "gcp",
    "product": "spanner", // Recommended
    "credential": {
      "keyFilename": "./gcp.json",
      "projectId": "squared" // Optional
    },

    "name": "squared", // Instance (required)
    "database": "sample", // Required

    "table": "demo",
    "query": {
      "columns": [], // transaction.ReadRequest (required)
      "keys": []
    },
    "update": {}, // object | object[] (optional)
    "updateType": 0 | 1 | 2, // 0 - update | 1 - insert | 2 - replace
    /* OR */
    "query": "SELECT 1" | { "sql": "SELECT 1", "params": {} }, // ExecuteSqlRequest (required)
    "update": {}, // DML statement (Same as "query")

    "options": {
      "databasePool": {}, // session-pool.SessionPoolOptions
      "databaseQuery": {}, // protos.IQueryOptions
      "tableRead": {}, // transaction.TimestampBounds
      "tableUpdate": {} // table.UpdateRowsOptions
    },

    "value": "<b>${title}</b>: ${description}"
  }
}

/* IBM-v1: https://github.com/IBM/cloudant-node-sdk#readme */
{
  "selector": "main p",
  "type": "text",
  "dataSource": {
    "source": "cloud",
    "service": "ibm-v1",
    "credential": {
      "serviceName": "squared", // Cloudant instance (required)

      "username": "**********", // Basic auth
      "password": "**********",
      /* OR */
      "apikey": "**********", // IAM tokem

      "url": "https://<username>.cloudantnosqldb.appdomain.cloud" // External endpoint (required)
    },
    "table": "demo", // "db" property during transaction

    /* Find */
    "query": {
      "selector": {
        "id": { "$eq": "1" } // Required
      }
    },
    /* Search */
    "query": {
      "ddoc": "demo-doc", // PostSearchParams
      "index": "demo-index", // https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-cloudant-search#index-functions
      "partitionKey": "", // Optional
      "query": "id:'1' AND title:'Bristol'" // Lucene syntax
    },
    /* View */
    "query": {
      "ddoc": "demo-doc", // PostViewParams
      "view": "demo-view", // https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-using-views
      "partitionKey": "" // Optional
    },
    /* Partition */
    "partitionKey": "Partition1", // When "query" undefined (PostPartitionAllDocsParams)

    "value": "<b>${title}</b>: ${description}"
  }
}

/* IBM (deprecated): https://github.com/cloudant/nodejs-cloudant#readme */
{
  "selector": "main p",
  "type": "text",
  "dataSource": {
    "source": "cloud",
    "service": "ibm",
    "credential": {
      "username": "**********", // Legacy credentials (alias for "account")
      "password": "**********",
      "url": "https://<username>.cloudantnosqldb.appdomain.cloud",
      /* OR */
      "url": "https://<username>:<password>@<username>.cloudantnosqldb.appdomain.cloud", // Basic auth
      /* OR */
      "iamApiKey": "**********", // IAM tokem
      "url": "https://<username>.cloudantnosqldb.appdomain.cloud"
    },
    "table": "demo",

    "query": { "selector": { "id": { "$eq": "1" } } }, // Find
    /* OR */
    "partitionKey": "Partition2", // All documents in partition

    "limit": 5,
    "value": "<b>${title}</b>: ${description}"
  }
}

/* OCI: https://docs.oracle.com/en/database/oracle/simple-oracle-document-access/adsdi/oracle-database-introduction-simple-oracle-document-access-soda.pdf */
{
  "selector": "main p",
  "type": "text",
  "dataSource": {
    "source": "cloud",
    "service": "oci",
    "credential": {
      "username": "**********", // alias for "user"
      "password": "**********",
      "connectionString": "tcps://adb.us-phoenix-1.oraclecloud.com:1522/abcdefghijklmno_squared_high.adb.oraclecloud.com?wallet_location=/Users/Oracle/wallet"
    },
    "table": "demo",

    "query": "SELECT * from demo WHERE id = '1'", // Column names might be UPPERCASED
    "query": "SELECT d.* from demo NESTED json_document COLUMNS(id, title, description) d WHERE d.id = '1'", // SODA
    /* OR */
    "query": { "id": { "$eq": "1" } }, // SODA

    "options": { "resultSet": true } // Optional

    "viewEngine": "ejs", // npm i ejs
    "preRender": "./pre-render.cjs", // Local files are supported (chrome -> eval_userconfig: true)
    "postQuery": "postQuery-example",
    "whenEmpty": "whenEmpty-example"
  }
}

/* Atlas: https://docs.mongodb.com/compass/master/query/filter */
{
  "selector": "main p",
  "type": "attribute",
  "dataSource": {
    "source": "cloud",
    "service": "atlas",
    "uri": "mongodb+srv://<username>:<password>@cluster0.abcde.mongodb.net/<database name>", // Required
    "name": "squared", // Database name (optional)
    "table": "demo", // Required

    /* MongoClientOptions */
    "credential": {
      "username": "**********", // Optional
      "password": "**********",
      /* OR */
      "auth": {
        "username": "**********", // node-mongodb v4
        "password": "**********"
      }
    },

    "query": {
      "id": {
        "$eq": "{{id}}"
      },
      "start_date": {
        "$gt": "new Date('2021-01-01')" // new Date("2021-01-01")
      },
      "$in": ["new RegExp(^mongodb, i)"], // Quotes are optional [/^mongodb/i]
      "$where": "function() { return this.name == 'mongodb.com'; }" // async is supported
    },
    "value": {
      "html_attribute": "db_column" // Required
    }
  }
}
// Retrieval using ID is supported by all providers

{
  "selector": "main img",
  "type": "attribute",
  "dataSource": {
    "source": "cloud",
    "service": "azure",
    "credential": "db-main",
    "name": "squared", // Azure (required)
    "table": "demo",

    "partitionKey": "Pictures", // Azure and IBM (optional)
    "id": "2", // OCI (server assigned)

    /* AWS */
    "key": { // Alias for partitionKey
      "id": "2", // Same
      "type": "image" // Additional fields are supported
    }

    /* Result: { src: "", other: {} } */
    "value": {
      "src": "src", // Use direct property access
      "alt": "{{if not expired}}other.alt{{else}}:text(Expired){{end}}", // Only one conditional per attribute
      "style": [":join(; )" /* " " (optional) */, "other.style[0]", "other.style[1]", ":text(display: none)"] // Same as: [":join(; )", "other.style", ":text(display: none)"]
    }
  }
}

Some queries use an optional parameters array (params) or configuration object (options) which is sent with the query when applicable. If you require this advanced usage then further instructions can be found in the database provider documentation.

When in development mode you can save read units by setting a timeout value for the DB cache.

// squared.json

"cloud": {
  "cache": {
    "aws": 0, // No cache per reload
    "azure": 60, // 1 minute
    "gcp": {
      "timeout": "1d", // 1 day
      "when_empty": true
    }
  }
}

Results are cached using the supplied credentials and queries will individually be cleared when the amount of time has expired.

Reusing configuration templates is possible using URL query parameters. Output values can be replaced using the {{param}} syntax.

// http://localhost:3000/project/index.html?table=demo&id=1

{
  "service": "azure",
  "credential": "db-main",
  "name": "squared",
  "table": "{{table}}",
  "partitionKey": "Pictures",
  "query": "SELECT * FROM c WHERE c.id = '{{id}}'",
  "value": "<b>${title}</b>: ${description}" // Not parsed
}

MongoDB

Local development may be faster using MongoDB instead of a cloud DocumentDB. It is completely free to use and includes a GUI data explorer.

* MongoDB Community Server
  - https://www.mongodb.com/try/download/community
  - https://docs.mongodb.com/drivers/node/current/fundamentals/authentication/mechanisms
  - https://docs.mongodb.com/compass/master/query/filter (query)

  - npm i mongodb

MongoDB Atlas installations also use the "mongodb" source format. All MongoDB authentication mechanisms are supported.

interface MongoDBDataSource {
    source: "mongodb";

    /* Required */
    uri?: string; // Connection string
    /* OR */
    credential?: string | MongoDBCredential;

    /* Optional */
    query?: Filter<Document> | { value: DocumentFilter, options: CommandOperationOptions }; // "value" is required when using "options"
    id?: string; // Uses ObjectId
    sort?: Sort | string | { value: Sort, direction: SortDirection };

    value?: string | ObjectMap<string | string[]>;

    options?: MongoClientOptions; // Overriden by credential
    client?: {
        db?: DbOptions; // Used as options with "name"
        collection?: CollectionOptions; // Used as options with "table"
    };
    execute?: {
        insert?: BulkWriteOptions; // Used as options with "update - insert"
        update?: UpdateOptions;
    };
    aggregate?: Document[] | { pipeline: Document[], options: AggregateOptions };

    cascade?: string; // UriDataSource
    fallback?: unknown; // object

    update?: UpdateFilter<Document> | OptionalUnlessRequiredId<unknown>[];
    updateType?: 0 | 1 | 2 | 3; // 0 - update | 1 - insert | 2 - replace | 3 - delete

    /* DB shared */
    usePool?: boolean | UUID; // UUIDv1-5 + db.settings.user_key (username@server/database)
    parallel?: false; // Used for batched queries (implicit: true)
    streamRow?: boolean | ((row: unknown) => Undef<Error | false>)>; // Not cached + postgres not supported
    willAbort?: boolean; // Module abort is called bypassing settings

    /* Same as CloudDatabase */
}

NOTE: Enabling usePool with a UUID key will also copy the value into credential.uuidKey. (without override)

// http://localhost:3000/project/index.html?id=1

{
  "selector": "main img",
  "type": "attribute",
  "dataSource": {
    "source": "mongodb",

    "uri": "mongodb://<username>@<password>:localhost:27017", // Recommended
    /* OR */
    "credential": { // Same as cloud settings "db-main"
      "server": "localhost:27017", // OR: 0.0.0.0
      /* OR */
      "hostname": "cluster0.abcdef.mongodb.net", // Required
      "port": 8080, // Default is "27017"

      "username": "**********", // Passed as URI query parameter (node-mongodb v3)
      "password": "**********",
      /* OR */
      "auth": {
        "username": "**********", // Passed in as options.auth (node-mongodb v4)
        "password": "**********"
      },

      /* Optional */
      "protocol": "mongodb+srv:", // "mongodb:" (default)

      "authMechanism": "MONGODB-X509",
      "authMechanismProperties": {}, // AuthMechanismProperties | string (SERVICE_NAME)
      "authSource": "",

      "tlsCertificateFile": "/path/tsl/x509/cert.pem", // node-mongodb v4
      "tlsCertificateKeyFile": "/path/tsl/x509/key.pem",
      "tlsCAFile": "",
      "tlsCertificateKeyFilePassword": "",
      "tlsAllowInvalidHostnames": false,
      "tlsAllowInvalidCertificates": false,
      "tlsInsecure": false,
      /* OR */
      "sslKey": "/path/ssl/x509/key.pem",
      "sslCert": "/path/ssl/x509/cert.pem",
      "sslCA": "",
      "sslPass": "",
      "sslValidate": true
    },

    "query": {
      "id": {
        "$eq": "{{id}}"
      },
      "name": {
        "$regex": "mongodb.*\\.com", // $regex: /mongodb.*\.com/si
        "$options": "si"
      },
      "start_date": {
        "$gt": "new Date('2021-01-01')" // new Date("2021-01-01")
      },
      "$in": ["new RegExp(^mongodb, i)"], // Quotes are optional [/^mongodb/i]
      "$where": "function() { return this.name == 'mongodb.com'; }" // "async" is supported
    },
    "value": {
      "src": "column_src", // Required
      "alt": "column_alt"
    },

    "usePool": true, // Optional
    "options": {
      "minPoolSize": 0,
      "maxPoolSize": 10
    }
  }
}

// IF conditional to completely remove an element (outerHTML)
{
  "selector": "main div",
  "type": "display",
  "dataSource": {
    "source": "mongodb",
    "uri": "mongodb://localhost:27017",
    "removeEmpty": true, // Includes invalid conditions (optional)
    /* Required */
    "value": "attr1", // Remove when: null or undefined
    "value": "-attr2", // Remove when: attr2=falsey
    "value": "+attr3", // Remove when: attr3=truthy
    "value": ["attr1" /* AND */, ":is(OR)", "-attr2" /* OR */, "-attr3" /* OR */, ":is(AND)", "+attr4" /* AND */] // Remove when: attr1=null + attr2|attr3=falsey + attr4=truthy
  }
}

To remove an element all AND conditions have to be TRUE and one OR per group is TRUE. Using a view engine is recommended if you require a more advanced conditional statement.

Returning an empty result or a blank string (view engine) is FALSE.

Redis

Using a key-value store is sufficient for generating simple static web pages.

  - https://redis.io/download (Linux)
  - https://redis.com/try-free (1 database + 30MB)

  - npm i redis
interface RedisDataSource {
    source: "redis";

    uri: string; // redis://localhost:6379 | redis://<username>:<password>@hostname:6380
    username?: string;
    password?: string;
    /* OR */
    credential?: string | ServerAuth;

    key?: string | string[];
    format?: "HASH" | "JSON"; // Default is "HASH"
    query?: string; // jsonpath + jmespath

    cascade?: string; // UriDataSource
    fallback?: unknown; // object

    field?: string; // GET - hash
    path?: string; // MGET - json

    /* https://github.com/redis/node-redis/tree/master/packages/search */
    search?: {
        index: string; // Preexisting schema
        /* OR */
        schema: CreateSchema; // Temporary schema
        schema: string; // chrome.settings.directory.schema (base directory + users/username?)
        /* OR */
        schema: CreateSchema; // Create schema as "index" (Not recommended - use CLI)
        index: string;

        query: string;
        options?: { // CreateOptions
            ON?: "HASH" | "JSON";
            PREFIX?: string | Array<string>;
        };
    };
    aggregate?: { /* Same as search */ };

    /* Advanced configuration (optional) */
    format?: "HKEYS" | "HVALS";
    update?: RedisSetValue | RedisSetValue[] | RedisJSONValue | RedisJSONValue[]; // @squared-functions/types/lib/db

    options?: {
        client?: RedisClientOptions;
        command?: RedisCommandOptions;
        get?: PlainObject;
        search?: SearchOptions;
        aggregate?: IAggregateOptions;
    };

    /* Same as MongoDBDataSource */
}
// http://localhost:3000/project/index.html?file=demo

{
  "selector": "main img",
  "type": "attribute",
  "dataSource": {
    "source": "redis",

    "uri": "redis://localhost:6379",
    "username": "**********",
    "password": "**********",
    "database": 1,
    /* OR */
    "uri": "redis://<username>:<password>@localhost:6379/<database>",
    /* OR */
    "credential": "main";
    /* OR */
    "credential": {
      "protocol": "", // Optional
      "server": "localhost:6379",
      "username": "**********",
      "password": "**********",
      "database": 1
    }

    "key": "demo:1",
    "query": "$.name", // jsonpath + jmespath (optional)
    /* OR */
    "search": {
      "schema": {
        "name": {
          "type": "TEXT", // SchemaFieldTypes.TEXT
          "sortable": true
        },
        "state": "TAG", // SchemaFieldTypes.TAG
        "age": "NUMERIC" // SchemaFieldTypes.NUMERIC
      },
      /* OR */
      "schema": "sub_dir/{{file}}.json", // yaml + json5 + toml + xml + cjs (parent: "schema")

      "query": "@state:{CA}",
      "options": {
        "ON": "HASH", // JSON
        "PREFIX": "noderedis:demo" // Optional
      }
    },
    "value": {
      "src": "column_src" // Required
    },

    "usePool": true, // Optional
    "options": {
      "client": {
        "isolationPoolOptions": {
          "min": 0,
          "max": 10
        }
      }
    }
  }
}

/* Auth - chrome.db */
{
  "redis": {
    "main": {
      "protocol": "", // Default is "redis:"
      "hostname": "", // Default is "localhost"
      "port": "", // Default is "6379"
      "username": "",
      "password": "",
      "database": 0 // SELECT index (number > 0)
    }
  },
  "sql": {
    "main": {
      "server": "localhost:3306", // Alias for hostname + port
      "username": "**********",
      "password": "**********",
      "database": "demo"
    }
  }
}

NOTE: Redis search will only return the "value" object with the id field appended as "__id__";

MySQL + PostgreSQL + Oracle + MSSQL

Querying SQL databases can be achieved using a simple SQL statement and optional parameters. These providers were designed to perform parallel SELECT statements and are not transactional.

  - MySQL Community Server
  - https://dev.mysql.com/downloads/mysql
  - https://www.npmjs.com/package/mysql#connection-options (Auth)

  - npm i mysql2

  - PostgreSQL
  - https://www.postgresql.org/download
  - https://node-postgres.com/features/connecting (Auth)

  - npm i pg

  - Oracle Database XE
  - https://www.oracle.com/database/technologies/xe-downloads.html
  - http://oracle.github.io/node-oracledb/doc/api.html#-16-connection-handling (Auth)

  - npm i oracledb

  - SQL Server
  - https://www.microsoft.com/en-us/sql-server/sql-server-downloads
  - https://tediousjs.github.io/tedious/api-connection.html (Auth)

  - npm i tedious
  - npm i tedious-connection-pool2 (optional)
interface SQLDataSource {
    source: "mysql" | "postgres" | "oracle" | "mssql";

    uri: "mysql://<username>:<password>@localhost:3306/database"; // uri (MySQL: Auth)
    uri: "postgresql://<username>:<password>@localhost:5432/database"; // connectionString (Postgres: Auth)
    uri: "<username>:<password>@localhost:1521/database"; // connectString | poolAlias (Oracle: Auth)
    uri: "<username>:<password>@localhost:1433/database"; // authentication.options + server + authentication (MSSQL: Auth)
    /* OR */
    credential: "main", // chrome.db[source]
    /* OR - Universal */
    credential: {
        hostname: "localhost", // Required
        port: 3306,
        username: "**********", // Required
        password: "**********",
        database: "example"
    };
    /* OR - Auth */
    credential: {
        host: "localhost", // mysql + postgres
        port: 5432,
        user: "**********",
        password: "**********",
        database: "example"
    };
    credential: {
        connectString: "localhost:1521/example", // oracle
        user: "**********",
        password: "**********"
    };
    credential: {
        server: "localhost", // mssql
        options: {
            port: 1433,
            database: "example",
            encrypt: true, // Azure
            trustServerCertificate: true // Local development
        },
        authentication: {
            type: "default",
            options: {
                userName: "**********",
                password: "**********"
            }
        }
    };

    query: "SELECT * FROM `table` WHERE `id` = ? AND `value` = ?";
    query: "./sub_dir/statement.sql"; // Extension ".sql" + chrome.settings.directory.sql (base directory + users/username?)

    params?: [1, "escaped"];

    /* mssql - http://tediousjs.github.io/tedious/parameters.html */
    params?: { "a": { value: "1", type: "VarChar", options: { length: 50 } }, "b": 2 /* Implicit: Int */ }; // MSSQLRequestParameters
    params?: [{ name: "c", type: "Decimal", value: 12.345, options: { precision: 10, scale: 2 } }];
    params?: [{ name: "d", type: "TVP", value: { columns: MSSQLRequestParameters[]; rows: unknown[][] }];
    params?: {
      input: Null<MSSQLRequestParameters>; // Two keys only for object detection (required)
      output: Null<MSSQLRequestParameters>; // Last row in result (data["__returnvalue__"] = true)
    };
    params_output?: MSSQLRequestParameters; // Deprecated
    storedProc?: boolean; // Call stored procedure
    usePool?: boolean | UUID | PoolConfig; // https://github.com/tediousjs/tedious-connection-pool + tedious 15 (auth)

    /* Same as MongoDBDataSource */
}

NPM packages can also be used to perform SQL statements.

// IDbSourceClient

{
  "dataSource": {
    "source": "sqlite-wrapper", // npm i sqlite-wrapper
    /* OR */
    "source": "@sqlite-scoped", // "client" appended + npm i @sqlite-scoped/client
    /* OR */
    "source": "@sqlite-scoped/query" // Settings uses full package name

    /* Same as above */
  }
}

Samples can be found in the @squared-functions/db directory.

Data Interchange

Using the same concept from databases you can also read from JSON/YAML/XML/TOML files.

  • JSON5: npm i json5
  • XML: npm i fast-xml-parser
  • TOML: npm i toml
interface UriDataSource {
    source: "uri";
    format: string; // json | yaml | json5 | xml | toml
    uri: string;

    query?: string; // if startsWith("$") Uses JSONPath <https://github.com/dchester/jsonpath> (npm i jsonpath)
    query?: string; // else Uses JMESPath <https://jmespath.org> (npm i jmespath)

    cascade?: string; // root.row1 + root.row1.row2[index] + root.row1[index].row2 = object (called before "query")
    /*
      When array [{ a: 1 }, { b: 2 }] then "0.a == 1" OR "1.b == 2" (leading only)
      When multi-dimensional array { a: [[1], [2,3]] } then "a[0].0 == 1" OR "a.1.1 == 3" (brackets are optional)
      When key has spaces { " a ": { "b ": { c: 1 } } } then " a "["b "].c OR " a "."b "[c] == 1 (quotes are optional)
      When key has dot operator  { a: { "b.c": 1 } } then a[b\\.c] == 1
      Invalid usage a[0][1][2] instead use a.0.1.2 or a[0].1[2] (max is 1 consequetive bracket)
    */
    fallback?: unknown; // Used when there are missing fields (object)

    options?: PlainObject; // Parser options (yaml + xml)

    /* Same as CloudDatabase (except no "id") */
}

interface LocalDataSource {
    source: "local";
    format: string; // json | yaml | json5 | xml | toml
    pathname: string; // Local file (permission) | chrome.settings.directory.data (base directory + users/username?)

    /* Same as UriDataSource */
}
// http://localhost:3000/project/index.html?file=demo

/* URI */
{
  "selector": "main img",
  "type": "attribute",
  "dataSource": {
    "source": "uri",
    "format": "json",
    "uri": "http://localhost:3000/project/{{file}}.json", // Local files require read permissions
    "query": "$[1]", // Row #2 in result array (optional)
    /* Result: { src: "image.png", other: { alt: "description" } } */
    "value": {
      "src": "src",
      "alt": "other.alt"
    }
  }
}

/* Local */
{
  "selector": "main img",
  "type": "attribute",
  "dataSource": {
    "source": "local",
    "format": "xml",
    "pathname": "sub_dir/{{file}}.xml", // yaml + toml (parent: "data")
    "query": "$.root.row[1]", // Second item in "row" array (JSONPath)
    "value": {
      "src": "src"
    }
  }
}

View engines can also be used to format the element "value" or innerHTML with any data source.

Export

Custom functions or packages can be used to return any kind of dataset from any source.

interface ExportDataSource {
    source: "export";
    params: unknown; // Passed into custom function (required)

    pathname?: string; // Module file (.cjs) | Local file | inline function | NPM hosted package
    /* OR */
    settings?: string;
    /* OR */
    execute?: Function;

    query?: string; // jsonpath + jmespath

    cascade?: string; // UriDataSource
    fallback?: unknown; // object

    persist?: boolean; // Default is "true"
    ignoreCache?: boolean | 1; // Same as Cloud
}
// squared.json

{
  "document": {
    "chrome": {
      "handler": "@squared-functions/document/chrome",
      "eval": {
        "function": true, // Enable inline functions
        "absolute": false // Enable absolute paths to local files
      },
      "settings": {
        "export": {
          "data-example": "function (params, resolve, require) { const fs = require('fs'); resolve(JSON.parse(fs.readFileSync(params.uri))); }",
          "async-example": "async function (params, require) { const fs = require('fs'); const result = await fs.promises.readFile(params.uri); return JSON.parse(result); }"
        }
      }
    }
  }
}
// postgres.cjs | NPM package

const pg = require("pg");

const config = {
  host: "localhost",
  user: "**********",
  password: "**********",
  database: "squared",
  port: 5432,
  ssl: true
};

module.exports = async function (params) {
  const client = new pg.Client();
  await client.connect();
  const { rows } = await client.query("SELECT * FROM demo WHERE id = $1", [params.id]);
  await client.end();
  return rows;
};

// mysql.js | inline (settings) | this (process)

function (params, resolve, require) {
  const mysql = require("mysql");
  const conn = new mysql.createConnection({
    host: "localhost",
    user: "**********",
    password: "**********",
    database: "squared",
    port: 3306,
    ssl: true
  });
  conn.connect();
  conn.query("SELECT * FROM demo WHERE id = ?", [params.id], (err, result) => {
    if (!err) {
      resolve(result);
    }
    else {
      console.log(err);
      resolve(null);
    }
  });
  conn.end();
}

async function (params, require) { /* Same */ }
{
  "selector": "main p",
  "type": "text",
  "dataSource": {
    "source": "export",

    "pathname": "postgres.cjs", // chrome.settings.directory.export (base directory + users/username?)
    "pathname": "/path/postgres.cjs", // Use "./" for relative paths (permission)
    "pathname": "npm:custom-package",
    /* OR */
    "settings": "data-example", // chrome.settings.export

    "value": "<b>${title}</b>: ${description}" // Same for all data sources
  }
}

Image

Image conversion can be achieved using the "commands" array property in a FileAsset object. The supported formats are:

* png - r/w
* jpeg - r/w
* webp - r/w
* bmp - r/w
* gif - r/w (w - ImageV3/jimp)
* tiff - r

* dwebp - r // npm i dwebp-bin
* cwebp - w // npm i cwebp-bin

* gifwrap - w  // Bundled with jimp
* gif2webp - w // npm i gif2webp-bin

NOTE: ImageV3/jimp supports using libwebp locally for all WebP transforms through squared settings.

// All commands are optional except "format". Outer groupings and inner brackets are required.

+ <format>

- @|%
- ~size(n)(w|x) // chrome only
- ( minSize(n,0) , maxSize(n,*)? )
- ( width(n|auto) x height(n|auto) [bilinear|bicubic|hermite|bezier]? ^(cover|contain|scale)?[left|center|right|top|middle|bottom]? #background-color? )
- ( left(+|-n) , top(+|-n) | cropWidth(n) x cropHeight(n) )
- { ...rotate(n|-n) #background-color? }
- | opacity(0.0-1.0) OR jpeg_quality(0-100) OR webp_quality(0-100?[photo|picture|drawing|icon|text]?[0-100]?) | // cwebp: -preset -near_lossless
- !method // No arguments (dither565|greyscale|invert|normalize|opaque|sepia)
- !method(1, "string_arg2", [1, 2], true, { "a": 1, "b": "\\}" } /* valid JSON */, 1 + 2 /* invalid */, ...args?) // No functions or native objects

@ - replace
% - smaller

Placing an @ symbol (png@) after the format will replace the original file inside the package. Using the % symbol (png%) will choose the smaller of the two files.

NOTE: You can use these commands with the setting "convertImages" in the Android framework.

<!-- https://github.com/oliver-moran/jimp/tree/master/packages/jimp#methods (jimp aliases) -->

* contain (ct) <!-- mode: left - 1 | center - 2 | right - 4 | top - 8 | middle - 16 | bottom - 32 -->
* cover (cv) <!-- Same -->
* resize (re)
* scale (sc)
* scaleToFit (sf)
* autocrop (au)
* crop (cr)
* blit (bt)
* composite (cp) <!-- mode: srcOver | dstOver | multiply | add | screen | overlay | darken | lighten | hardLight | difference | exclusion -->
* ma (mask)
* convolute (cl)
* flip (fl)
* mirror (mi)
* rotate (ro)
* brightness (br)
* contrast (cn)
* dither565 (dt)
* greyscale (gr)
* invert (in)
* normalize (no)
* fade (fa)
* opacity (op)
* opaque (oq)
* background (bg)
* gaussian (ga)
* blur (bl)
* posterize (po)
* sepia (se)
* pixelate (px)
* displace (dp)
* color (co)

Methods use simple bracket matching and does not fully check inside quoted strings. Unescaped "\" with unpaired ("{}" or "[]") will fail to parse.

// Multiple transformations use the "::" as the separator (data-chrome-commands)

webp(50000)(800x600[bezier]^contain[right|bottom]#FFFFFF)(-50,50|200x200){45,-45,215,315#FFFFFF}|0.5||100[photo][75]|!sepia

webp!opacity(0.5) // OR
webp!op(0.5)

webp~800w(800x600) // "srcset" (chrome)
webp~2x(1024x768)

NOTE: jimp rotation is counter-clockwise. Use {-45} to achieve a 45-degree clockwise rotation.

Tinify is used for image compression and supports PNG and JPEG. The first 500 images are free each month with a developer API key.

// squared.json
{
  "compress": {
    "tinify_api_key": "**********" // Default API key (optional)
  }
}
{
  "selector": "main img",
  "compress": [
    {
      "format": "png", // png | jpeg | webp
      "plugin": "tinify", // Optional
      "options": {
        "apiKey": "**********" // Overrides settings (optional)
      }
    }
  ]
}

Other formats can be compressed similarly using imagemin.

{
  "selector": "main img",
  "compress": [
    {
      "format": "png",
      "plugin": "imagemin-pngquant", // npm i imagemin-pngquant
      "options": {
        "quality": [
          0.6,
          0.8
        ]
      }
    }
  ]
}

If no exact match is found with format then all plugins will be applied to the unknown image. Multiple plugins of the same format will be processed in a series.

<!-- img | video | audio | source | track | object | embed | iframe -->

<img src="https://s3-us-west-2.amazonaws.com/s.cdpn.io/12005/harbour1.jpg"
     data-chrome-file="saveAs:images/harbour.webp"
     data-chrome-options="inline"> <!-- jpg to webp (images only) -->

You can use images commands with saveTo (directory) on any element where the image is the primary display output.

<!-- img | object | embed | iframe -->

<img src="https://s3-us-west-2.amazonaws.com/s.cdpn.io/12005/harbour1.jpg"
     data-chrome-file="saveTo:../images/harbour"
     data-chrome-commands="png(10000,75000)(800x600[bezier]^contain[right|bottom])::webp|0.5|"> <!-- separator "::" -->

Transformations are given a UUID filename except when "@" or "%" are used. Leaving "file" empty will save the transformations to the current image directory.

Tasks

Tasks can be performed preceding archiving or copying after file content has been downloaded and also transformed.

// squared.json

{
  "task": {
    "gulp": {
      "handler": "@squared-functions/task/gulp",
      "settings": {
        "minify": "./gulpfile.js", // Will execute "minify" task
        "compress": "./gulpfile.android.js",
        "users": {
          "username": {
            /* npm i -g gulp && cd ./username && npm link gulp */
            "minify": "./username/gulpfile.js" // Will override parent "minify"
          }
        },
        "css": {
          "path": "./gulpfile.js", // Required
          "tasks": ["minify", "beautify"],
          "opts": ["--series", "--continue"] // https://github.com/gulpjs/gulp-cli#flags
        }
      }
    }
  }
}

// chrome
{
  "selector": "head > script:nth-of-type(1)",
  "type": "js",
  "tasks": [
    { "handler": "gulp", "task": "minify" },
    { "handler": "gulp", "task": "beautify", "preceding": "true" }, // Execute tasks before transformations
    {
      "handler": "gulp",
      "task": {
        "path": "../gulp/transform.js", // User defined uses permission
        "tasks": ["minify"],
        "opts": ["--verify"]
      }
    }
  ]
}

// android
squared.saveAs("index.zip", {
  assets: [{
    pathname: "images",
    filename: "pencil.png",
    mimeType: "image/png",
    commands: ["jpeg", "bmp@(50000,100000)"],
    tasks: [{ handler: "gulp", task: "compress" }],
    uri: "http://localhost:3000/common/images/pencil.png"
  }]
});
<!-- chrome -->
<script src="/common/util.js" data-chrome-tasks="gulp:minify+gulp:beautify:true"></script>

<!-- android -->
<img src="https://s3-us-west-2.amazonaws.com/s.cdpn.io/12005/harbour1.jpg" data-android-tasks="gulp:compress">

<!-- json -->
<script src="/common/util.js" data-chrome-tasks='[{ handler: "gulp", task: "minify" }, { handler: "gulp", task: "beautify", preceding: "true" }]'></script>

NOTE: SRC (temp) and DEST (original) always read and write to the current directory.

// gulpfile.js

const gulp = require("gulp");
const uglify = require("gulp-uglify");

gulp.task("minify", () => {
  return gulp.src("*")
    .pipe(uglify())
    .pipe(gulp.dest("./"));
});

gulp.task("default", gulp.series("minify"));

Renaming files with Gulp is not recommended. It is better to use the "saveAs" or "filename" attributes when the asset is part of the HTML page.

Modifying content attributes

There are possible scenarios when a transformation may cause an asset type to change into another format.

<!-- before -->
<link id="sass-example" rel="alternate" type="text/plain" href="css/dev.sass">
{
  "selector": "#sass-example",
  "type": "css",
  "filename": "prod.css",
  "download": true, // Required when rel="alternate"
  "hash": "md5", // prod-8ba5cde2c04d6628dd6b0d62b0a3cac9.css (optional)
  "attributes": {
    "id": undefined,
    "rel": "stylesheet",
    "type": "text/css",
    "title": "",
    "disabled": null,

    /* SRI (optional) */
    "integrity": "sha512", // CORS required (sha256 | sha384 | sha512)
    "crossorigin": "anonymous" // Optional
  },
  "process": ["compile"]
}

Or you can use inline commands.

<!-- strict JSON -->
<link rel="alternate" type="text/plain" href="css/dev.sass" data-chrome-file='{ "saveAs": "css/prod.css", "process": ["compile"] }' data-chrome-attributes='{ "rel": "stylesheet", "type": "text/css", "title": "", "disabled": null, "integrity": "sha512", "crossorigin": "anonymous" }' data-chrome-options="download">
<!-- after -->
<link rel="stylesheet" type="text/css" title="" disabled href="css/prod.css" integrity="sha512-365M9279XHmH5iUcnWAVLXbLCqmntex51g5yp+jrzci79PhUOZyK/ebMIzoR46y7okDgZYZhTImDt7/06IhxQw==" crossorigin="anonymous">

You can also use the workspace feature in squared-express to precompile text assets and build the production release in one routine.

NOTE: Tags that are not well-formed may fail to be replaced. (e.g. CSS properties with embedded "}")

Appending external JS/CSS

You can append or prepend a sibling element (not child) which will be processed similarly to a typical "script" or "link" element.

<html>
<head>
  <title></title>
  <!-- Google Analytics -->
  <script>
  window.ga=window.ga||function(){(ga.q=ga.q||[]).push(arguments)};ga.l=+new Date;
  ga('create', 'UA-XXXXX-Y', 'auto');
  ga('send', 'pageview');
  </script>
  <script async src='https://www.google-analytics.com/analytics.js'></script>
  <!-- End Google Analytics -->
</head>
<body>
</body>
</html>
// All commands including "prepend" in relation to the base type are supported

[
  {
    "selector": "title",
    "type": "append/script", // All tags except "html"
    "textContent": "\\nwindow.ga=window.ga||function(){(ga.q=ga.q||[]).push(arguments)};ga.l=+new Date;\\nga('create', 'UA-XXXXX-Y', 'auto');\\nga('send', 'pageview');\\n" // YAML "|" operator preserves indentation (optional)
  },
  {
    "selector": "title",
    "type": "append/js", // prepend/css
    "download": false, // Explicit "false"
    "attributes": {
      "src": "https://www.google-analytics.com/analytics.js", // CSS: href (required)
      "async": null
    }
  }
]

[useOriginalHtmlPage=false]: The current state of the DOM is sent to the server which includes any updates made with JavaScript.

[useOriginalHtmlPage=true]: Will attempt to parse the page as it stored on file. Dynamic content (e.g. php) that create additional elements are not supported. Scripts which insert elements as the page is loading should similarly be performed after parsing the document. Errors might also occur with <p> tags when they contain tables and other block-level elements. If you are having problems using this option then make sure your page is a valid HTML5 document or you can try adding an element "id" instead as a last resort.

Watch

File watching is available with "copy" methods and uses HTTP HEAD requests to determine modifications. You can watch any file served with HTTP including files from another server.

// html | js | css | image

{
  "selector": "link",
  "type": "css",
  "watch": {
    /* Optional */
    "interval": 100, // Time between HTTP requests
    "expires": "1w 1d 1h 1m 1s", // Empty is never
    "main": false // Will rebuild the entire bundle
    "id": "watch1", // Resets an existing watch when it is the first active socket
    "reload": { // true
      "socketId": "111-111-111", // Use same id to reload multiple pages
      "port": 80,
      "secure": false, // Requires TLS/SSL key and cert
      "module": false, // "img" and "link" only
      "always": false // Ignore errors
    }
  },
  "process": [
    "bundle",
    "lint",
    "minify"
  ]
}

/* OR */

squared.copyTo("/local/user/www", {
  watch: true,
  saveAs: {
    script: { pathname: "../js", process: ["es5", "es5-minify"], watch: true },
    link: { pathname: "css", filename: "bundle.css", watch: { interval: 500 } }
  }
});
{
  "selector": "link",
  "type": "css",
  "watch": true // Uses defaults with optional authenticated settings
}

// squared.json
{
  "watch": {
    "settings": {
      "users": {
        "username": {
          "http://localhost:3000/index.html": { "interval": 100 }, // Order of precedence
          "http://localhost:3000/**/*": {/* Same */},
          "/path/to/project/file": {/* Same */}, // Local
          "/path/to/project/**/*": {/* Same */},
          "image/png": {/* Same */}, // MIME
          "image/*": {/* Same */}
        }
      }
    }
  }
}
<!-- chrome -->
<link rel="stylesheet" href="/common/util.css" data-chrome-watch="1000::1h 30m::111-111-111:8080[module|secure|always]"> <!-- "~" can be used for default value -->

<!-- android -->
<img src="images/harbour1.jpg" data-android-watch="true"> <!-- img | video | audio -->

<!-- json -->
<script src="/common/util.js" data-chrome-watch='{ "interval": 100, "expires": "1w 1d 1h 1m 1s", "reload": { "port": 443, "secure": true } }'></script>

Hot module replacement is only available for LINK[href] and IMG[src] elements. It is disabled by default due to possible conflicts with preloaded JavaScript.

[useOriginalHtmlPage=true]: HTML has partial watch support since most modifications will alter the original element index position.

  • script + link -> Inside source file
  • script + style (inline) -> Unbundled
  • elements (configured + data source) -> Uneditable

NOTE: If you configure an <img> tag (or any tag group) then you cannot add or remove more <img> elements without adding an element "id" to every element in the group.

Options: Development / Production

The entire page can similarly be transformed as a group using the "saveAs" attribute in options. Cloud storage can be used for all assets (except HTML) using the same configuration as element selectors.

squared.saveAs("index.zip", {
  productionRelease: false | "/absolute/path/wwwroot/", // Ignore local url rewriting and use absolute paths
  productionIncremental: false, // Use cached data when processing transformed files
  preserveCrossOrigin: false, // Ignore downloading a local copy of assets hosted on other domains
  useOriginalHtmlPage: false | "textarea|code", // Ignore tags causing parsing errors
  useUnsafeReplace: "html", // Use when there are no element tags inside comments and <script>
  useUnsafeReplace: "css", // Use when there are no comments or block delimiters inside property values (e.g. "{" "}")
  useUnsafeReplace: true, // html + css
  useUnsafeReplace: ["html", "css"], // Same
  stripCommentsAndCDATA: false | "script|picture", // Remove unused code from HTML + non-void tags

  /* Use when there are parsing errors */
  normalizeHtmlOutput: false | "?%", // Escape illegal tagNames and remove unnecessary trailing spaces (ignore: tagNames with RegExp negated characters <?php ?> | <%= ejs %>)
  escapeReservedCharacters: false, // Escape reserved characters inside attributes (e.g. "<")
  ignoreServerCodeBlocks: ["<%", "%>", "<?php", ";", "<?php", "?>"], // May produce better results when there is dynamic content (<% %> | <?php ; | <?php ?>) (useOriginalHtmlPage=true)

  incremental: false, // No cache is used + default is "true" (implicit)
  incremental: "none", // Will rebuild all assets and use available cache
  /* Use when paths are stable and only the content (html/js/css/db) is updated */
  incremental: "etag", // Will use current file if the ETag is still the same
  incremental: "exists", // Will always use current file if found

  removeInlineStyles: false, // Strip style="" attribute from all elements (useOriginalHtmlPage=false)
  removeUnusedClasses: false, // Selectors without :pseudo-class
  removeUnusedPseudoClasses: false, // Selectors with :pseudo-class (not recommend with forms :valid and active states :hover) (lowercase only)
  removeUnusedVariables: false, // --custom-variables
  removeUnusedFontFace: false, // @font-face
  removeUnusedKeyframes: false, // @keyframes
  removeUnusedMedia: false, // @media
  removeUnusedContainer: false, // @container (not recommended)
  removeUnusedSupports: false, // @supports

  /* Styles that should be kept which are still being used */
  retainUsedStyles: {
    "selectors": [], // Same without separators
    "variables": [],
    "pseudoClass": false, // Will keep "li a:hover" + "li a:visited" when "li a" is a used selector
    "@font-face": [],
    "@keyframes": [],
    "@media": [],
    "@container": [],
    "@supports": []
  },
  /* OR */
  retainUsedStyles: [
    /* CSS selectors (string | RegExp) */,
    /* CSS variables (string prefixed with '--') */,
    /* CSS @font-face (string enclosed within '|font-face:Times New Roman|') */,
    /* CSS @keyframes (string enclosed within '|keyframes:animationName|') */,
    /* CSS @media (string enclosed within '|media:only screen and (min-width: 800px)|') */,
    /* CSS @container (string enclosed within '|container:(min-width: 800px)|') */,
    /* CSS @supports (string enclosed within '|supports:(display: grid)|') */
  ],

  /* All attributes are optional */
  saveAs: {
    html: { filename: "index.html", process: ["beautify"], attributes: { lang: "en" } },
    script: {
      pathname: "../js",
      filename: "bundle.js", // Only for first bundle
      process: ["es5", "es5-minify"],
      ignoring: ["text/template"], // MIME types
      inlineAction: false // script + link + image
    },
    link: {
      pathname: "css"
      filename: "bundle.css",
      inline: true,
      preserve: true,
      attributes: { rel: "stylesheet", media: "all" },

      /* data-chrome-options + data-chrome-attributes + data-chrome-metadata + data-chrome-watch + data-chrome-tasks */
      inlineAction: true, // merge + override
      inlineAction: "merge", // data-chrome-attributes='{ media: "screen" }' -> { "rel": "stylesheet", media: "screen" }
      inlineAction: "override" // data-chrome-attributes='{ "rel": "alternate" }' -> { "rel": "alternate" }
    },
    image: {
      attributes: { loading: "lazy", width: "detect", height: "detect" }, // npm i probe-image-size (optional)
      commands: ["webp(480x300)"], // Only valid for binary images
      process: ["minify-svg"] // Only valid for text based images (e.g. SVG)
    },
    font: {
      pathname: "fonts",
      compress: "woff2", // truetype | opentype | woff | woff2
      blob: false, // Save base64 as file
      blob: "fonts/blob", // Overrides pathname for blob only
      inline: false, // Write file as data:font/ttf;base64
      customize: (uri, mimeType, command) => { // script | link | image | font
        if (mimeType === "font/ttf") {
          command.blob = true; // Does not alter font object
          return "filename.ttf";
        }
        return ""; // Do not alter filename
        /* OR */
        return null; // Ignore file
      }
    }
  }
});

Asset exclusion

You can exclude unnecessary processing files using the dataset attribute in <script|link|style> tags. Other elements can only be excluded when using a configuration file.

<script src="/dist/squared.js" data-chrome-file="exclude"></script>
<script src="/dist/squared.base.js" data-chrome-file="exclude"></script>
<script src="/dist/chrome.framework.js" data-chrome-file="exclude"></script>
<script data-chrome-file="exclude">
  squared.setFramework(chrome);
  squared.save();
</script>

There is a simpler way to exclude the same tags by using an HTML comment block only when inside the HEAD element.

NOTE: Cannot be used with option [stripCommentsAndCDATA=true].

<head>
  <title></title>
  <!-- EXCLUDE: start -->
  <meta charset="utf-8"> <!-- not excluded -->
  <script src="/dist/squared.js"></script>
  <script src="/dist/squared.base.js"></script>
  <script src="/dist/chrome.framework.js"></script>
  <script>
    squared.setFramework(chrome);
    squared.save();
  </script>
  <style>
    body {
      max-height: 800px;
    }
  </style>
  <!-- EXCLUDE: end -->
  <link rel="stylesheet" href="/common/util.css">
</head>

You can similarly prevent any element from being downloaded or transformed using the "ignore" command.

<iframe src="https://www.google.com/maps" data-chrome-file="ignore"></iframe>

LICENSE

MIT

Package Sidebar

Install

npm i squared-functions

Weekly Downloads

4

Version

4.11.0

License

MIT

Unpacked Size

100 kB

Total Files

3

Last publish

Collaborators

  • anpham6