assetflow

Asset deployment for node

assetflow

Assetflow is an asset deployment tool. It supports md5 hash comparison with S3 and enables you to create powerful asset flows easily and fast. It is a Grunt task and applies solid cache-busting techniques transparently.

If you are not familiar with Grunt check out the Grunt's Getting Started guide.

A typical deployment flow using assetflow:

  • Scan assets and generate MD5 hashes.
  • Create the manifest.json file.
  • Copy all assets and rename them with their md5 hash to a temporary location.
  • Perform HEAD operations with S3 and compare hashes using S3's ETAG.
  • Upload all assets that did not have a matching ETAG.

Optionally there are two more tasks you can perform:

  • Search & replace any set of assets based on a custom keyword, i.e. __ASSET(img/logo.jpg).
  • Create the clientManifest.js file, a client optimized subset of the manifest.
npm install assetflow --save-dev

The assets task performs these operations:

  • Scans all the defined assets.
  • Generates md5 hashes for each asset.
  • Create the manifest.json file.
    • If a manifest file exists, it will compare the hashes.
  • Copy all new or updated assets and rename them with their md5 hash to a temporary location.

When this task finishes all your assets have been copied to a new temporary folder that you defined. This folder will contain your assets renamed with their own hash, like so:

app.js --> app-h522md41d.js

The manifest.json file generated by this task keeps a reference to all your assets so their names can be properly resolved in all environments.

Type: string Default: manifest.json

Define the location of the manifest file.

Type: string Default: none

Add the url of your CDN to prepend it to all assets.

Type: string Default: none

The rel option will perform directory subtraction on the source to calculate the relative path to the asset. Consider this case:

Your folder of static assets is under assets/, so the path to your logo would be assets/img/logo.png which would be accessed by the browser as /img/logo.png.

Declaring the assets folder as a rel path will make sure that all assets have the proper url.

Example

assets: {
  options: {
    rel: 'assets/'
  },
  all: {
    src: 'assets/**',
    dest: 'temp/assets'
  }
}

Type: number Default: none

The md5 hash is 32 bytes long, you don't need all of it, use this option to truncate the hash down to n chars.

Type: string Default: none

This option will prepend a value to the asset's key. It is mostly used to prepend a slash and make the asset key absolute, for example:

By default, the assets task will create records in the manifest.json file as relative web paths: img/logo.png. If you need the key to be an absolute path then you have to use prepend.

options: {
  prepend: '/'
}

Type: number Default: 100

The maximum number of concurrent operations, in this case the operations are file copying.

Set this to 0 to disable throttling. Set it to 1 to make all operations serial.

Type: boolean Default: false

A fancy progress bar.

Type: boolean Default: false

Print extra debugging information.

Here's an annotated configuration for the assets task:

assets: {
  // global task options 
  options: {
    // don't output debug information 
    debug: false,
    // trancate the hash length to 8 chars. 
    truncateHash: 8,
    // define the location of the manifest file. 
    manifest: 'temp/manifest.json',
    // define the location of the cdn. 
    cdnurl: 'http://d3s3z9buwru1xx.cloudfront.net/assets/',
    // Set maximum number of file copying concurent operations. 
    maxOperations: 100,
    // Show a fancy progress indicator 
    progress: true,
    // Set path to substract so the relative path for the assets can be calculated. 
    rel: 'lib/'
  },
 
  all: {
    // local task options 
    options: {
      // change the rel path to test/case 
      rel: 'assets/'
    },
    src: [
      // all files under folder assets 
      'assets/**',
      // except all files in less folder 
      '!assets/less/**',
      // except all files in handlebars folder 
      '!assets/handlebars/**'
    ],
    dest: 'temp/assets'
  }
}

↑ Back to TOC

The assetsReplace task will search and replace the contents of your assets. It is useful for cases where you don't have the ability of a 'helper' to resolve your assets.

LESS files are a typical example, use a custom keyword to include your assets and run the assetsReplace task to populate the asset urls in your .less files. For example if the custom keyword is __ASSET():

@bg-dot-light: url(__ASSET(img/pdf-icon-cv.png)) repeat 0 0 #2a2a2a;

After the assetsReplace task is executed the same line will look like this:

@bg-dot-light: url(http://d3s3z9buwru1xx.cloudfront.net/assets/img/pdf-icon-cv-fk44j2s.png) repeat 0 0 #2a2a2a;

The assetsReplace task is based on grunt-string-replace by @erickrdch

Type: string Default: manifest.json

Define the location of the manifest file.

Type: string Default: none

Define the keyword that will be searched for replacement. Use the % char as a placeholder for the asset name. e.g.:

options: {
  key: '__ASSET(%)'
}

Type: string Default: none

Use this to define a regex as a keyword. The type must be string, but the string will be evaluated as a regex. The % char is a placeholder for the asset name and the reason why this option needs to be a string instead of a regex type.

So take case to double escape what you need escaped, e.g.

// this regex 
var reg = /match[\s]space/;
 
// is represented like that as a string: 
var strReg = 'match[\\\s]space';

Type: string Default: none

This option will prepend a value to the asset's key. It is mostly used to prepend a slash and make the asset key absolute.

Type: boolean Default: false

Print extra debugging information.

Here's an annotated configuration for the assetsReplace task:

assetsReplace: {
  // global task options 
  options: {
    // don't output debug information 
    debug: false,
    // define the location of the manifest file. 
    manifest: 'temp/manifest.json',
  },
  // the less target 
  less: {
    options: {
      // define the keyword for this target 
      key: '__ASSET(%)'
    },
    files: {
      // Search & replace all .less files under the assets/less folder 
      // and output the result in the temp/less folder 
      'temp/less/': ['assets/less/**/*.less']
    }
  },
  // the handlebars target 
  handlebars: {
    options: {
      // a regex for lax matching {{asset "%"}} allowing for spaces in between 
      // and single quotes. 
      keyRegex: '\\\{\\\{[\\\s]*asset[\\\s]+[\\\'\\\"]{1}%[\\\'\\\"]{1}[\\\s]*\\\}\\\}',
      // prepend the slash on every asset query 
      prepend: '/'
    },
    // all the files from the assets/handlebars folder 
    src: 'assets/handlebars/**/*.hbs',
    // output to temp/handlebars folder 
    dest: 'temp/handlebars/'
  }
}

Files that are produced with the assetsReplace task should be in your codebase and tracked.

For example, here is a typical setup for less files:

assets-replace.less
assets.less   <--- this is auto-generated by Grunt
main.less
restStyles.less
@import "assets"
 
@import "restStyles"
// .... 
// 
// ONLY ADD IMAGES TO assets-replace.less 
// IMAGES ADDED TO assets.less WILL BE OVERWRITTEN 
// 
 
@asset_img_logo: "__ASSET(/assets/logo.png)";
@asset_img_cover: "__ASSET(/assets/cover.jpg)";

This file is generated by the assetsReplace Grunt task.

// 
// ONLY ADD IMAGES TO assets-replace.less 
// IMAGES ADDED TO assets.less WILL BE OVERWRITTEN 
// 
 
@asset_img_logo: "/assets/logo.png";
@asset_img_cover: "/assets/cover.jpg";
#header{
  .logo {
    position: relative;
    margin: 0;
    background: url(@asset_img_logo) bottom center no-repeat;
  }
 
  .cover {
    background: url(@asset_img_cover) no-repeat #577d9c;
    cursor: default;
    padding-left: 35px;
    background-position: 9px 9px;
  }
}
 

↑ Back to TOC

The assetsBundle task will create a compact version of the manifest file, optimized for transferring to the client as a javascript file. The manifest.json file is pretty large and not suited for getting transfered. Furthermore it might be the case where you have several hundreds, if not thousands of assets but you only need a handful to be consumed by your front-end app. This is the task that enables you to do this.

Type: string Default: manifest.json

Define the location of the manifest file.

Type: boolean Default: false

export the assets as an AMD module.

Type: boolean Default: false

export the assets using commonjs pattern. e.g: module.exports=..

Type: string Default: ASSETS

Define a namespace to export using the global window Object. You can use dot notation.

Type: Array Default: []

An array of strings where you define the asset filename keys you want included.

Type: boolean Default: false

Print extra debugging information.

Here's an annotated configuration for the assetsBundle task:

assetsBundle: {
  // global task options 
  options: {
    // define the location of the manifest file. 
    manifest: 'temp/manifest.json',
    // export as AMD 
    amd: true
  },
 
  // Export as amd 
  amd: {
    dest: 'temp/bundles/clientManifest.amd.js'
  },
 
  // Export using namespaces 
  ns: {
    options: {
      // Export assets on this global namespace: 
      ns: 'app.assets'
    },
    dest: 'temp/bundles/clientManifest.ns.js'
  }
}

The javascript file generated by the assetsBundle task will export an Object that contains key value pairs to the assets. If you are using AMD to export the assets this is how you'd access them:

var assets    = require('assets');
 
var asset = assets['/img/logo.png'];
// --> https://d3s3z9buwru1xx.cloudfront.net/assets/img/logo-fh422j4f.png 

This is a weak part in the API and will surely see changes in the future.

↑ Back to TOC

The assetsS3 task will read the manifest.json file and upload all the assets to S3. Although it's optional, it is highly advised to use the checkS3Head option which enables md5 hash checking between S3 and your local files.

The assetsS3 task is based on grunt-S3 by @pifantastic. All options from that task are available in this one too.

These are the options that are only available in the assetsS3 task of Assetflow.

Type: string Default: manifest.json

Define the location of the manifest file.

Type: boolean Default: false

Perform md5 hash comparisons between the local assets and the ones in S3. It is highly advisable that you enable this option so you only upload files that have been updated.

Type: string Default: none

The rel option will perform directory subtraction on the source to calculate the relative path to the asset.

Type: number Default: 100

The maximum number of concurrent operations, in this case the operations are network uploads.

Set this to 0 to disable throttling. Set it to 1 to make all operations serial.

Type: boolean Default: false

A fancy progress bar.

Type: boolean Default: false

Print extra debugging information.

These options are available by Grunt-S3, sic:

The grunt-s3 task is now a multi-task; meaning you can specify different targets for this task to run as.

The following are the default options available to each target.

  • key - (string) An Amazon S3 credentials key
  • secret - (string) An Amazon S3 credentials secret
  • bucket - (string) An Amazon S3 bucket
  • region - (string) An Amazon AWS region (see http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region)
  • maxOperations - (number) max number of concurrent transfers - if not specified or set to 0, will be unlimited.
  • encodePaths - (boolean) if set to true, will encode the uris of destinations to prevent 505 errors. Default: false
  • headers - (object) An object containing any headers you would like to send along with the transfers i.e. { 'X-Awesomeness': 'Out-Of-This-World', 'X-Stuff': 'And Things!' }
  • access - (string) A specific Amazon S3 ACL. Available values: private, public-read, public-read-write, authenticated-read, bucket-owner-read, bucket-owner-full-control
  • gzip - (boolean) If true, uploads will be gzip-encoded.
  • gzipExclude - (array) Define extensions of files you don't want to run gzip on, an array of strings ie: ['.jpg', '.jpeg', '.png'].
  • upload - (array) An array of objects, each object representing a file upload and containing a src and a dest. Any of the above values may also be overriden.
  • download - (array) An array of objects, each object representing a file download and containing a src and a dest. Any of the above values may also be overriden.
  • del - (array) An array of objects, each object containing a src to delete from s3. Any of the above values may also be overriden.
  • debug - (boolean) If true, no transfers with S3 will occur, will print all actions for review by user

Here's an annotated configuration for the assetsS3 task:

assetsS3: {
  options: {
    // no debug info 
    debug: false,
    // enable checking md5 hashes by performing S3 HEAD requests. 
    checkS3Head: true,
    // the manifest file 
    manifest: 'temp/manifest.json',
 
    // aws credentials 
    key: 'AWS-KEY',
    secret: 'AWS-SECRET',
    bucket: 'S3-BUCKET',
 
    // Enable public access 
    access: 'public-read',
 
    // don't show the fancy progress indicator. 
    progress: false
  },
  all: {
    // These options override the defaults 
    options: {
      // limit concurent uploads to 100 
      maxOperations: 100
    },
    // the 'upload' option key is required 
    upload: {
      // all files from temp/assets 
      src: 'temp/assets/**',
      // a prefix folder on S3 
      dest:  'assets/',
      // the rel option 
      rel: 'temp/assets/',
      // gzip uploaded files 
      gzip: true,
      // excluded these extensions 
      gzipExclude: ['.jpeg', '.jpg', '.png', '.gif', '.less', '.mp3',
          '.mp4', '.mkv', '.webm', '.gz'],
      // upload assets with a looong expire Cache-Control header. 
      headers: {'Cache-Control': 'max-age=31536000, public'}
    }
  }
}

↑ Back to TOC

You can use the Assetflow library on node:

// mind the () in the end! 
var assets = require('assetflow')();
 
assets.config({
  manifest: __dirname + '/assetManifest.json'
});
 
var assetUrl = assets.asset('/img/logo.png');

Like the client API, node's API is weak and may change in the future.

↑ Back to TOC

  • v0.2.8, 29 Jul 2015
    • Allow trailing slash if no CDN is present.
  • v0.2.7, 22 May 2015
    • The saga continues with grunt-s3.
  • v0.2.6, 21 May 2015
    • Lol, found the required dep it's grunt-s3 0.2.0-alpha.3.
  • v0.2.5, 21 May 2015
    • Downgrade grunt-s3 t0 0.1.0.
  • v0.2.4, 21 May 2015
    • Downgrade grunt-s3 to 0.2.0-alpha.2.
  • v0.2.3, 21 May 2015
    • Now reports on S3 upload error
    • Points to a fork of grunt-s3 package with updated packages for node 0.12.
  • v0.2.2, 05 Feb 2015
    • Properly require Collections package.
  • v0.2.1, 18 Nov 2014
    • Moved Grunt to peerDependencies.
    • Upgraded all dependent packages to latest (except "when").
  • v0.1.6, 26 Jul 2013
    • Bug fix for file created with -undefined hash, thanks Younes.
  • v0.1.5, 21 Jul 2013
    • Updated dependencies versions (Knox specifically had issues that are now fixed)
  • v0.1.4, 08 May 2013
    • Updated Knox so Assetflow will work for node 0.10.x
    • Fixed bug when no rel path is used.
  • v0.1.2, 17 April 2013
    • S3 paths now get normalized using path.normalize().
  • v0.1.1, 19 March 2013
    • Added support for commonjs for browsers.
  • v0.1.0, Mid March 2013
    • Big Bang

Copyright 2012 Verbling (Fluency Forums Corporation)

Licensed under the MIT License