gulp-source-scanner

Overview

gulp-source-scanner is a plugin for the gulp task runner which tries (without making guarantees, due to the complexity of the task) to detect some critically important file types which are due to be uploaded to source control (e.g. github).

Why should I use this plugin?

The aim is simple, prevent commonly found, sensitive file types from ending up in your source control repository and thus reduce the risk of leakage.

Which types of files can be detected?

Currently, gulp-source-scanner will, by default, try to find:

• SSL/TLS X.509 certificates (admittedly, not critical in most cases)
• SSL/TLS private keys (RSA)
• AWS API access tokens
• AWS API secrets
• DSA private keys
• CSRs

How much will this slow down my build?

In some (admittedly random tests) of ~10 genuine codebases pulled from github, the test took less than 2 seconds per codebase, even for 200k+ files.

How should I use this plugin?

The short answer is "any way you like", the longer answer is:

My guess/intention is that you'll insert it into your regular gulp-based build-chain (likely close to the last step) and probably either output a "clean" set of files to commit to source control or by setting the failOnDetection option and failing the build if any suspect files are detected.

Semver

This project aims to maintain the semver version numbering scheme.

Changelog

See the changelog file

Security

See the security file

Requirements

• Node runtime (I believe v0.12+ will work but haven't tested)
• NPM
• gulp (obviously!)

NPM/Node package dependencies

Production

• through2 - A gulp plugin simplifying library
• gulp-util - a gulp plugin helper library

development

• mocha - Test runner
• coveralls - Code coverage analysis reporter
• istanbul - Code coverage
• mocha-lcov-reporter - Code coverage

Installation

Installation is super simple, in your command line terminal, just cd to your project root and run:

npm install gulp-source-scanner --save-dev

Usage

Runtime usage of gulp-source-scanner is very simple, it's used as a regular gulp-plugin in your gulp task(s) and accepts a single options argument:

var gulp=require("gulp");
var gss=require("gulp-source-scanner");
 
var opts=
{
  logLevel:"warn", // "error" || "warn" || "info" || "debug"
  ignoreFilesListLocation:".gitignore", // location of a file which lists files which won't be committed to SC, set to false to disable
  ignoreFilesLargerThanMB:9, // Don't even scan files > this size in MB
  failOnDetection:false, // Fail on detecting a file which matches the defined scanTypes
  removeFilesFromOutput:"matches", // "matches" || "non-matches" || "none"
  scanTypes:
  {
    CSR:".csr", // certificate signing request
    TLSCert:[".pem",".crt", ".cert"], // RSA public keys (SSL/TLS cert)
    RSAPrivate:".key", // RSA private keys (SSL/TLS key)
    DSAPrivate:".key", // DSA private keys (SSL/TLS key)
    AWSAccessToken:true, // AWS API token: access token
    AWSSecretToken:true // AWS API token: secret token
  }
};
 
gulp.task("scan", function()
{
  gulp.src(['./**'])
  .pipe(gss(opts))
  .on('error', function(error)
  {
    gutil.log(gutil.colors.red('[SECURITY] ' + error));
    this.emit('end');
  })
  .pipe(gulp.dest('./out-dir'));
});

Controlling the files which will be scanned

You can use the gulp.src() function to determine which files will be scanned. This is standard gulp behaviour which can accept anything from single, absolute paths to arrays of many glob-able paths.

Ignoring files

gulp-source-scanner supports the .gitignore file or anything which is compatible with it to provide a list of files to ignore whilst scanning. If your file is not (relative to where you will run your gulp task from) /.gitignore, you'll need to use the ignoreFilesListLocation option to set its filename and path. Comments (line starting with #) in the file will be ignored.

Options

gulp-source-scanner has a number of options, supplied as an object, by which you can configure the plugin:

var opts=
{
  logLevel:"warn", // "error" || "warn" || "info" || "debug"
  ignoreFilesListLocation:".gitignore", // location of a file which lists files which won't be committed to SC, set to false to disable
  ignoreFilesLargerThanMB:9, // Don't even scan files > this size in MB
  failOnDetection:false, // Fail on detecting a file which matches the defined scanTypes
  removeFilesFromOutput:"matches", // "matches" || "non-matches" || "none"
  scanTypes:
  {
    CSR:".csr", // certificate signing request
    TLSCert:[".pem",".crt", ".cert"], // RSA public keys (SSL/TLS cert)
    RSAPrivate:".key", // RSA private keys (SSL/TLS key)
    DSAPrivate:".key", // DSA private keys (SSL/TLS key)
    AWSAccessToken:true, // AWS API token: access token
    AWSSecretToken:true // AWS API token: secret token
  }
};

NOTE: currently, you must provide all options or none, there is no option-by-option defaulting

Most of the above is completely self-explanatory though scanTypes warrants a little further explanation. scanTypes is an object which contains key-value pairs where the key is a string which must be one of the defined detectable types and the value is one of:

• String - a single filename extension (including the leading .) of files to scan
• Array of Strings - one or more file extensions (including the leading .) of files to scan
• true - scan all files for
• false - disable scanning for this type

Thus, for each detectable type, you can determine whether to scan some, all or none of the files defined by gulp.src().

Currently, you cannot filter the files to be scanned for by anything other than their filename extension.

Controlling the output files

You can use the standard gulp.dest function to determine where the scanned files should be output. gulp-source-scanner can filter the output via the option removeFilesFromOutput - options for this are:

• "matches" - remove files from the output which match a detectable type (i.e. files which contain one or more of the detectable types)
• "non-matches" - remove files which do not match a detectable type (i.e. files which do not contain any of the detectable types)
• "none" - do not remove any files from the output

These options (respectively) allow you to output a "clean" set of files, output a "dirty" set of files or output all files.

Failing a build on detection

If you want the build (well, strictly speaking, the gulp task) to fail (i.e. throw an error) on detecting a suspect file, use the failOnDetection (Boolean) option. Sadly, this option is not too pretty as it results in a stacktrace from node though it will (on *nix systems, not sure about Windows) at least result in a return/exit code of 1 (thus you can detect it programmatically).

Known issues / to-do

• The code is super messy and needs refactoring into a more testable/reliable format
• Currently, you have to pass in all options - an object merge would be much better
• The AWS secret detection filter sometimes triggers on RSA private keys
• A small number of functions are synchronous - these should be make async for better performance
• Occasionally, an error is thrown - there is no stacktrace and it appears to be a bug in node but I haven't been able to pin it down yet
• Need to add more detectable types (suggestion very welcome via issues!)
• It would be good to support more ignore types (currently it's just github)
• Add and improve tests (they are awful!)

Tests

There are some very, very basic built-in tests which use Mocha, you can run these (after installation) via:

npm test

All test should pass - we use Travis CI to verify this with each push to the GitHub master branch.

Bugs

If you find a bug, please let me know via an issue.

Contributing

If you have ideas for improvements or want to contribute a bug fix, please create an issue first so we can discuss and make sure we don't duplicate efforts and that the idea is in the right direction for the plugin.

License

MIT