Disk Build helper
An NDCODE project.
Overview
The disk_build
package exports a function
disk_build(pathname, build_func, diag)
,
which given a pathname to an existing source file, will generate a pathname for
a corresponding on-disk output file, check whether this exists and is newer
than the original, and if not, rebuild it via a caller-provided function.
Calling API
The interface for the disk_build
-provided helper function disk_build()
is:
await disk_build(pathname, build_func, diag)
— checks if the source
file given in pathname
exists, generates the corresponding output pathname
and dependency-file pathname, reads the dependency file if possible, checks the
output is up-to-date with respect to all dependencies, and if not, generates
a temporary output pathname temp_pathname
, calls the user-provided callback
function build_func(temp_pathname)
to build the output into the temporary,
possibly writes a new dependency file, then renames everything into place.
Finally, it returns an dictionary {deps: ..., pathname: ...}
where deps
is
a list of dependencies not including the original, and pathname
is the output
file; these might come from the existing disk files or newly written ones.
The interface for the user-provided callback function build_func()
is:
await build_func(temp_pathname)
— the user must create the file
temp_pathname
and then return, or alternatively an exception can be thrown.
The return value can be either undefined
if no dependency file should be
written, or a list of pathnames referred to (not including the original).
About asynchronicity
There is nothing too complicated here, both disk_build()
and build_func()
are asynchronous and thus return a Promise
, and when the inner Promise
resolves, the outer Promise
resolves in turn. Either level throws exceptions,
with various causes such as disk errors, file not found, syntax error, etc.
There is no protection against multiple clients trying to access the same file
at once while the asychronous building operation proceeds. This is because the
accesses to disk_build()
are supposed to be wrapped in a BuildCache
, which
will exclude multiple access. There was no point making disk_build()
keep a
dictionary of operations in progress when BuildCache
already does this, since
disk_build()
does not otherwise need any state kept in RAM (it has the disk).
Usage example
Simple usage, showing use of the clean-css
CSS minimizer with disk_build()
,
in such a way as to return a string containing the minified CSS from the given
input file described by pathname
, without re-converting if already converted:
let CleanCSS = require('clean-css')
let disk_build = require('@ndcode/disk_build')
let fs = require('fs')
let util = require('util')
let clean_css = new CleanCSS({returnPromise: true})
let fs_readFile = util.promisify(fs.readFile)
let fs_writeFile = util.promisify(fs.writeFile)
let get_css_min = async pathname => {
let render = await disk_build(
pathname,
async temp_pathname => {
let render = await clean_css.minify(
await fs_readFile(pathname, {encoding: 'utf-8'})
)
return /*await*/ fs_writeFile(
temp_pathname,
render.styles,
{encoding: 'utf-8'}
)
},
true // diagnostics on
)
return /*await*/ fs_readFile(render.pathname)
}
Since it is rather common that various minimizers and template renderers return
an dictionary containing the result of processing plus some other information,
and so does disk_cache()
, we've adopted the convention of putting the result
of such calls in a variable called render
, despite multiple use of the name.
A more complicated example using a BuildCache
to exclude multiple callers:
let BuildCache = require('@ndcode/build_cache')
let CleanCSS = require('clean-css')
let disk_build = require('@ndcode/disk_build')
let fs = require('fs')
let util = require('util')
let build_cache = new BuildCache(true) // diagnostics on
let clean_css = new CleanCSS({returnPromise: true})
let fs_readFile = util.promisify(fs.readFile)
let fs_writeFile = util.promisify(fs.writeFile)
let get_css_min = pathname => /*await*/ build_cache.get(
pathname,
async result => {
let render = await disk_build(
pathname,
async temp_pathname => {
let render = await clean_css.minify(
await fs_readFile(pathname, {encoding: 'utf-8'})
)
return /*await*/ fs_writeFile(
temp_pathname,
render.styles,
{encoding: 'utf-8'}
)
},
true // diagnostics on
)
result.value = /*await*/ fs_readFile(render.pathname)
}
}
In the above there was no dependency tracking needed or wanted, since CSS files
cannot include further CSS source files. Less
files can, handled as follows:
let BuildCache = require('@ndcode/build_cache')
let disk_build = require('@ndcode/disk_build')
let fs = require('fs')
let util = require('util')
let less = require('less/lib/less-node')
let path = require('path')
let build_cache = new BuildCache(true) // diagnostics on
let fs_readFile = util.promisify(fs.readFile)
let fs_writeFile = util.promisify(fs.writeFile)
let get_css_less = pathname => /*await*/ build_cache.get(
pathname,
async result => {
let render = await disk_build(
pathname,
async temp_pathname => {
let render = await less.render(
await fs_readFile(pathname, {encoding: 'utf-8'}),
{
compress: true,
filename: pathname,
pathnames: [path.posix.dirname(pathname)]
}
)
await fs_writeFile(
temp_pathname,
render.css,
{encoding: 'utf-8'}
)
return render.imports
},
true // diagnostics on
)
result.value = /*await*/ fs_readFile(render.pathname)
}
}
We are relying on fs_readFile()
or less.render()
to throw exceptions if the
original stylesheet or any included stylesheet is not found or contains errors.
Also, note how much simplified the handling of asynchronicity is when using the
ES7 async
/await
syntax. We recommend to do this for all new code, and to do
it consistently, even if the use of Promise
directly might give shorter code.
Code which is not already promisified, can be promisified as shown, or else we
can add specific conversions in places by code like: await new Promise(...)
.
Note the comment /* await */
where a Promise
is passed through from a lower
level routine, an explicit await
would be consistent here but less efficient.
About diagnostics
If diag
is passed as true
, then a diagnostic will be printed indicating
whether an output file is being rebuilt, or an existing output is being reused.
GIT repository
The development version can be cloned, downloaded, or browsed with gitweb
at:
https://git.ndcode.org/public/disk_build.git
License
All of our NPM packages are MIT licensed, please see LICENSE in the repository.
Contributions
We would greatly welcome your feedback and contributions. The disk_build
is
under active development (and is part of a larger project that is also under
development) and thus the API is considered tentative and subject to change. If
this is undesirable, you could possibly pin the version in your package.json
.
Contact: Nick Downing nick@ndcode.org