A package-based parallel build system
This is a package-based parallel build system. A task is described as a package, and Incubator builds requested packages in parallel according to the resolved the dependencies.
npm install incubatorincubate --package-path=path pkg1 pkg2 ...
--package-path=PATH: mandatory, path to lookup packages, and specify multiple times for multiple paths
--build-dir=PATH: optional, base directory for generating intermediate build outputs, default is
_buildunder current directory
--release-dir=PATH: optional, base directory for final build outputs, default is
--package-cache=PATH: optional, directory for caching downloaded files, default is
--parallel=N: optional, explicitly specify the number of tasks to run in parallel
--parallel-max=N: optional, limit the number of parallel run tasks, default is number of CPUs on the system
--clean: optional, when specified, all packages are re-built, without checking changes of depended files/packages
--save-space: optional, delete intermediate files when a package is successfully built to use as less disk space as possible
--script: optional, display in log format, regardless whether current output is a TTY.
A full qualified package name includes the package name and suffixed with a version.
incubator-0.0.7. The version part conforms to semver.
When a package is referenced, either a package name or the full qualified package name is used. If only a package name is used, Incubator will match it with the latest version.
Packages are defined under the directory which is referenced as a
Incubator accepts multiple package paths by option
--package-path=<path> and looks up
packages in all these paths.
Under a certain package path, a package is defined as a directory named with package name.
Under this package directory, there are one or more files named as
So multiple versions of the package can be defined at the same place.
This directory also contains package specific files which will be used when building this package.
version.yml file consists of three sections: package information, source files, and build steps.
Here's an example:
---package:name: my-sample-packagedescription: A Sample Packageversion: 0.1.3dependencies:- my-sample-tools- my-sample-box: ">= 0.5.0"sources:- file: my-sample-source.tar.gzdigest: sha1:<SHA1SUM-OF-MY-SAMPLE-SOURCE>origins:- http://my-sample-site.org/downloads/my-sample-source.tar.gzbuild:- commands:- tar zxf $_SRCDIR/my-sample-source.tar.gz- workdir: my-sample-sourcepaths:- dep: my-sample-toolsdir: bincommands:- my-sample-make all SAMPLE_BOX=$_RELBASE/$_DEP_MY_SAMPLE_BOX- my-sample-make install DESTDIR=$_RELDIR
name: mandatory, defines the package name, must be consistent with package directory name;
description: optional, detailed description of the package;
version: mandatory, version of the package, must be consistent with this YAML file;
dependencies: optional, specifies all packages this one depends on.
dependencies, it is a list.
The item can be a single string representing a simple package name and the latest version will be matched,
or a key/value pair matches a package name with a version request.
Refer to semver for how to request a version.
sources is a list of source files.
A source file can be defined in two ways:
- A single file which can be downloaded with a list of URLs;
- A packaged file which already exists in the package folder;
- A source tree which must be checked out from a version control system.
For a single file, the following keys are defined:
file: mandatory, specifies the name of the file after download;
digest: recommended, specifies the digest (only SHA1 is supported at present) of the file, if not specified, the file will be downloaded every time;
origins: mandatory, a list of URLs (http/https/ftp are supported at present) for downloading the file, they are tried one-by-one.
For a packaged file, things are simpler:
file: mandatory, specifies the name of the file in package folder;
packaged: must be
trueto indicate the file is in package folder.
For a source tree, things are different:
scm: mandatory, specifies which version control system is used, only
gitis supported at present;
Other keys are version control system specific, for
repo: mandatory, the URL of the repository;
dir: optional, local directory for checking out the code, if not specified, this is derived from
checkout: optional, the full commit Id to checkout, if not specified, HEAD is checked out;
master: optional, name of remote master branch, default is
gitcmd: optional, the external
gitprogram, default is
build is a list of steps each managed by an engine specified by key
All other keys are engine-specific configurations.
The only engine currently supported is
shell which is also default (
engine is not specified).
It defines following keys:
workdir: enter the directory before executing the commands;
commands: a list of shell commands to be executed;
env: a hash of environment variables, the key is the variable name, variable expansion like
$VARNAMEis not supported;
paths: add sub-directory under release directory of depended packages are added to environment
dep: the depended package name
dir: sub-directory under the release directory
envPrefix: prefix for all generated environment variable names, not including those specified in
shell engine prepares a number of environment variables whose names can be prefixed by
envPrefix if specified:
_BLDSLOT: id of parallel execution slot running the command, the number is zero-based;
_BLDBASE: base directory for intermediate output files from all packages;
_BLDDIR: directory for intermediate outputs from current package, it is
_RELBASE: base directory for final output files from all packages;
_RELDIR: directory for final outputs from current package, it is
_SRCDIR: cache directory,
$_SRCDIR/filenamecan be used to locate a downloaded file;
_PKGDIR: directory containing package definition file;
_PKGFILE: path pointing to package definition file;
_PKGDEPS: a list of full qualified names of depended packages, one per line.
envPrefix is specified on this step, the environment variables above will be prefixed with the value.
SAMPLE_PKGDIR should be used instead of
For each of depended packages, engine defines an environment variable like:
PACKAGENAME as part of the variable name is derived from package name without version
by capitalizing all alphabetic characters and replacing
This is quite useful when some build commands reference the output of a depended package, like:
---package:...dependencies:- my-depbuild:- commands:- tar zxf $_RELBASE/$_DEP_MY_DEP/output.tar.gz...
Refer to Evo Cloud stemcell which uses Incubator to build Linux OS from scratch.
Incubator can be directly used as a grunt plugin:
gruntinitConfigincubator:path: __dirname + '/packages' // mandatory, array is also okoptions: // all optional'build-dir': __dirname + '/_build''package-cache': __dirname + '/_cache''release-dir': __dirname + '/_rels''clean': true'save-space': true'script': true;gruntloadNpmTasks'incubator';gruntregisterTask'default' 'incubate:my-pkg1:my-pkg2';;