gulp-i18n-preprocess

    1.2.3 • Public • Published

    Build Status Coverage Status npm

    gulp-i18n-preprocess

    Preprocess Polymer templates and extract UI strings to JSON for build-time I18N with i18n-behavior

    Build-Time I18N

    Features

    • Preprocess Polymer templates to replace hard-coded UI strings with {{annotated}} variables
    • Extract hard-coded UI strings to JSON
    • Embed the JSON to the target preprocessed templates as default strings
    • Export the JSON as files
    • Scan custom element HTMLs for constructing repository for I18N target attributes
    • Build-time functionalities in gulp-i18n-preprocess are in sync with those in i18n-behavior at run-time

    Install

        npm install --save-dev gulp-i18n-preprocess
    

    Quick Tour with polymer-starter-kit-i18n

    Workflow

    Build tasks from source to dist:

    1. Scan task

    • Scan source HTMLs for custom elements
    • Construct localizable attributes repository

    2. Preprocess task

    • Preprocess source HTMLs
    • Extract UI texts to JSON as default
    • Replace them with {{annotations}}
    • Embed default texts in HTMLs as JSON
    • Externalize default texts to JSON files
    • Put them in dist

    3. (Optional) Import XLIFF task with xliff-conv

    4. Leverage task with gulp-i18n-leverage

    • Update localized JSON files by merging differences in default JSON from the previous build
    • Put them in dist
    • Merge all the UI texts into bundles object

    5. Bundles task with fs.writeFileSync()

    • Generate default bundled JSON file bundle.json from the bundles object
    • Generate per-locale bundled JSON files bundle.*.json from the bundles object
    • Put them in dist

    6. (Optional) Export XLIFF task with xliff-conv

    7. Feedback task

    • Update default and localized JSON files in source to commit them later by a developer or a build system

    Usage

    Default options

    Sample to show default options:

        var gulp = require('gulp');
        var i18nPreprocess = require('gulp-i18n-preprocess');
     
        gulp.task('preprocess', function () {
          // default options values
          var options = {
            replacingText: false, // does not replace strings with {{annotations}}
            jsonSpace: 2, // JSON stringification parameter for formatting
            srcPath: 'app', // base source path
            force: false, // does not force preprocessing when i18n-behavior.html is not imported
            dropHtml: false, // does not drop the preprocessed HTML for output
            dropJson: false, // does not drop the extracted JSON files for output
            constructAttributesRepository: false, // does not construct localizable attributes repository
            attributesRepositoryPath: null // does not specify the path to i18n-attr-repo.html
          };
     
          return gulp.src([ 'app/elements/**/*.html' ])
            .pipe(i18nPreprocess(options))
            .pipe(gulp.dest('dist/elements'));
        });

    Scan task

    Note: Target HTMLs must import i18n-behavior.html directly.

    Input:

    • Custom element HTMLs in source

    Output:

    • attributesRepository object in gulpfile.js
        var gulp = require('gulp');
        var i18nPreprocess = require('gulp-i18n-preprocess');
     
        // Global object to store localizable attributes repository
        var attributesRepository = {};
     
        // Scan HTMLs and construct localizable attributes repository
        gulp.task('scan', function () {
          return gulp.src([ 'app/elements/**/*.html' ]) // input custom element HTMLs
            .pipe(i18nPreprocess({
              constructAttributesRepository: true, // construct attributes repository
              attributesRepository: attributesRepository, // output object
              srcPath: 'app', // path to source root
              attributesRepositoryPath: 
                'bower_components/i18n-behavior/i18n-attr-repo.html', // path to i18n-attr-repo.html
              dropHtml: true // drop HTMLs
            })) 
            .pipe(gulp.dest('dist/elements')); // no outputs; dummy output path
        });

    Preprocess task

    Note: Target custom element HTMLs must import i18n-behavior.html directly.

    Input:

    • Custom element HTMLs
    • Non-custom-element HTMLs in source

    Output:

    • Preprocessed HTMLs and default JSON files in dist
        var gulp = require('gulp');
        var merge = require('merge-stream');
        var i18nPreprocess = require('gulp-i18n-preprocess');
     
        // Global object to store localizable attributes repository
        var attributesRepository; // constructed attributes repository
     
        // Other standard pipes such as crisper / minification / uglification are omitted for explanation
        gulp.task('preprocess', function () {
          var elements = gulp.src([ 'app/elements/**/*.html' ]) // input custom element HTMLs
            .pipe(i18nPreprocess({
              replacingText: true, // replace UI texts with {{annotations}}
              jsonSpace: 2, // JSON format with 2 spaces
              srcPath: 'app', // path to source root
              attributesRepository: attributesRepository // input attributes repository
            }))
            .pipe(gulp.dest('dist/elements')); // output preprocessed HTMLs and default JSON files to dist
     
          var html = gulp.src([ 'app/**/*.html', '!app/{elements,test}/**/*.html' ]) // non-custom-element HTMLs
            .pipe(i18nPreprocess({
              replacingText: true, // replace UI texts with {{annotations}}
              jsonSpace: 2, // JSON format with 2 spaces
              srcPath: 'app', // path to source root
              force: true, // force processing even without direct i18n-behavior.html import
              attributesRepository: attributesRepository // input attributes repository
             }))
            .pipe(gulp.dest('dist'));
     
          return merge(elements, html);
        });

    Leverage task with gulp-i18n-leverage

    Input:

    • Current localized JSON files in source
    • Current default JSON files in source
    • Next default JSON files in dist

    Output:

    • Next localized JSON files in dist
    • Bundles object in gulpfile.js
        var gulp = require('gulp');
        var i18nLeverage = require('gulp-i18n-leverage');
     
        var bundles = {};
     
        gulp.task('leverage', function () {
          return gulp.src([ 'app/**/locales/*.json' ]) // input localized JSON files in source
            .pipe(i18nLeverage({
              jsonSpace: 2, // JSON format with 2 spaces
              srcPath: 'app', // path to source root
              distPath: 'dist', // path to dist root to fetch next default JSON files
              bundles: bundles // output bundles object
            }))
            .pipe(gulp.dest('dist')); // path to output next localized JSON files
        });

    Bundles task

    Input:

    • Bundles object in gulpfile.js

    Output:

    • Bundles JSON files in dist
        var gulp = require('gulp');
        var fs = require('fs');
        var JSONstringify = require('json-stringify-safe');
     
        var bundles; // constructed bundles
     
        gulp.task('bundles', function (callback) {
          var DEST_DIR = 'dist';
          var localesPath = DEST_DIR + '/locales';
     
          try {
            fs.mkdirSync(localesPath);
          }
          catch (e) {}
          for (var lang in bundles) {
            bundles[lang].bundle = true;
            if (lang) {
              fs.writeFileSync(localesPath + '/bundle.' + lang + '.json', 
                                JSONstringify(bundles[lang], null, 2));
            }
            else {
              fs.writeFileSync(DEST_DIR + '/bundle.json', 
                                JSONstringify(bundles[lang], null, 2));
            }
          }
          callback();
        });

    Feedback task

    Note: Target custom element HTMLs must import i18n-behavior.html directly.

    Input:

    • Next localized JSON files in dist
    • Custom element HTMLs
    • Non-custom-element HTMLs

    Output:

    • Overwritten localized JSON files in source
    • Overwritten default JSON files in source

    Outputs are ready to commit in the repository

        var gulp = require('gulp');
        var merge = require('merge-stream');
        var i18nPreprocess = require('gulp-i18n-preprocess');
     
        // Only applicable to development builds; Skip it in production builds
        gulp.task('feedback', function () {
          // Copy from dist
          var locales = gulp.src([ 'dist/**/locales/*.json', '!dist/locales/bundle.*.json'])
            .pipe(gulp.dest('app'));
     
          // Regenerate default JSON files
          var elementDefault = gulp.src([ 'app/elements/**/*.html' ])
            .pipe(i18nPreprocess({
              replacingText: false,
              jsonSpace: 2,
              srcPath: 'app',
              dropHtml: true,
              attributesRepository: attributesRepository
            }))
            .pipe(gulp.dest('app/elements'));
     
          // Regenerate default JSON files for non-custom-element HTMLs, i.e., i18n-dom-bind
          var appDefault = gulp.src([ 'app/**/*.html', '!app/{elements,test}/**/*.html' ])
            .pipe(i18nPreprocess({
              replacingText: false,
              jsonSpace: 2,
              srcPath: 'app',
              force: true,
              dropHtml: true,
              attributesRepository: attributesRepository
            }))
            .pipe(gulp.dest('app'));
     
          return merge(locales, elementDefault, appDefault);
        });

    Integrate with Polymer CLI project templates by polymer-build library (highly experimental)

    Notes:

    • As of polymer-build 0.4.0, polymer-build library is pre-release and subject to change.
    • As of Polymer CLI 0.13.0, the private API userTransformers is deprecated and no longer available.

    Set up package.json and the dependent packages of the following gulpfile.js

        npm init # if package.json is missing 
        npm install --save-dev gulp gulp-debug gulp-grep-contents \
          gulp-i18n-add-locales gulp-i18n-leverage gulp-i18n-preprocess \
          gulp-if gulp-ignore gulp-match gulp-merge gulp-size gulp-sort gulp-util \
          json-stringify-safe strip-bom through2 xliff-conv polymer-build plylog merge-stream

    Gulp Filters:

    • scan - Scan HTMLs and construct localizable attributes repository
    • basenameSort - Sort source files according to their base names; Bundle files come first.
    • dropDefaultJSON - Drop default JSON files to avoid overwriting new ones
    • preprocess - Preprocess Polymer templates for I18N
    • tmpJSON - Store extracted JSON in the temporary folder .tmp
    • importXliff - Import XLIFF into JSON
    • leverage - Merge changes in default JSON into localized JSON
    • exportXliff - Generate bundles and export XLIFF
    • feedback - Update JSON and XLIFF in sources
    • debug - Show the list of processed files including untouched ones
    • size - Show the total size of the processed files

    Gulp Tasks:

    • gulp locales --targets="{space separated list of target locales}"
    • gulp default - Build with polymer-build library for gulp

    gulpfile.js: Put it in the root folder of the project

        'use strict';
     
        var gulp = require('gulp');
        var gutil = require('gulp-util');
        var debug = require('gulp-debug');
        var gulpif = require('gulp-if');
        var gulpignore = require('gulp-ignore');
        var gulpmatch = require('gulp-match');
        var sort = require('gulp-sort');
        var grepContents = require('gulp-grep-contents');
        var size = require('gulp-size');
        var merge = require('gulp-merge');
        var through = require('through2');
        var path = require('path');
        var stripBom = require('strip-bom');
        var JSONstringify = require('json-stringify-safe');
        var i18nPreprocess = require('gulp-i18n-preprocess');
        var i18nLeverage = require('gulp-i18n-leverage');
        var XliffConv = require('xliff-conv');
        var i18nAddLocales = require('gulp-i18n-add-locales');
     
        const logging = require('plylog');
        const mergeStream = require('merge-stream');
     
        const isPolymerCLI = global._babelPolyfill;
     
        // Global object to store localizable attributes repository
        var attributesRepository = {};
     
        // Bundles object
        var prevBundles = {};
        var bundles = {};
     
        var title = 'I18N transform';
        var tmpDir = '.tmp';
     
        var xliffOptions = {};
     
        // Scan HTMLs and construct localizable attributes repository
        var scan = gulpif('*.html', i18nPreprocess({
          constructAttributesRepository: true, // construct attributes repository
          attributesRepository: attributesRepository, // output object
          srcPath: '.', // path to source root
          attributesRepositoryPath: 
            'bower_components/i18n-behavior/i18n-attr-repo.html', // path to i18n-attr-repo.html
          dropHtml: false // do not drop HTMLs
        }));
     
        var basenameSort = sort({
          comparator: function(file1, file2) {
            var base1 = path.basename(file1.path).replace(/^bundle[.]/, ' bundle.');
            var base2 = path.basename(file2.path).replace(/^bundle[.]/, ' bundle.');
            return base1.localeCompare(base2);
          }
        });
     
        var dropDefaultJSON = gulpignore([ 'src/**/*.json', '!**/locales/*.json' ]);
     
        var preprocess = gulpif('*.html', i18nPreprocess({
          replacingText: true, // replace UI texts with {{annotations}}
          jsonSpace: 2, // JSON format with 2 spaces
          srcPath: '.', // path to source root
          attributesRepository: attributesRepository // input attributes repository
        }));
     
        var tmpJSON = gulpif([ 'src/**/*.json', '!src/**/locales/*' ], gulp.dest(tmpDir));
     
        var unbundleFiles = [];
        var importXliff = through.obj(function (file, enc, callback) {
          // bundle files must come earlier
          unbundleFiles.push(file);
          callback();
        }, function (callback) {
          var match;
          var file;
          var bundleFileMap = {};
          var xliffConv = new XliffConv(xliffOptions);
          while (unbundleFiles.length > 0) {
            file = unbundleFiles.shift();
            if (path.basename(file.path).match(/^bundle[.]json$/)) {
              prevBundles[''] = JSON.parse(stripBom(String(file.contents)));
              bundleFileMap[''] = file;
            }
            else if (match = path.basename(file.path).match(/^bundle[.]([^.\/]*)[.]json$/)) {
              prevBundles[match[1]] = JSON.parse(stripBom(String(file.contents)));
              bundleFileMap[match[1]] = file;
            }
            else if (match = path.basename(file.path).match(/^bundle[.]([^.\/]*)[.]xlf$/)) {
              xliffConv.parseXliff(String(file.contents), { bundle: prevBundles[match[1]] }, function (output) {
                if (bundleFileMap[match[1]]) {
                  bundleFileMap[match[1]].contents = new Buffer(JSONstringify(output, null, 2));
                }
              });
            }
            else if (gulpmatch(file, '**/locales/*.json') &&
                     (match = path.basename(file.path, '.json').match(/^([^.]*)[.]([^.]*)/))) {
              if (prevBundles[match[2]] && prevBundles[match[2]][match[1]]) {
                file.contents = new Buffer(JSONstringify(prevBundles[match[2]][match[1]], null, 2));
              }
            }
            this.push(file);
          }
          callback();
        });
     
        var leverage = gulpif([ 'src/**/locales/*.json', '!**/locales/bundle.*.json' ], i18nLeverage({
          jsonSpace: 2, // JSON format with 2 spaces
          srcPath: '', // path to source root
          distPath: '/' + tmpDir, // path to dist root to fetch next default JSON files
          bundles: bundles // output bundles object
        }));
     
        var bundleFiles = [];
        var exportXliff = through.obj(function (file, enc, callback) {
          bundleFiles.push(file);
          callback();
        }, function (callback) {
          var file;
          var cwd = bundleFiles[0].cwd;
          var base = bundleFiles[0].base;
          var xliffConv = new XliffConv(xliffOptions);
          var srcLanguage = 'en';
          var promises = [];
          var self = this;
          var lang;
          while (bundleFiles.length > 0) {
            file = bundleFiles.shift();
            if (!gulpmatch(file, [ '**/bundle.json', '**/locales/bundle.*.json', '**/xliff/bundle.*.xlf' ])) {
              this.push(file);
            }
          }
          for (lang in bundles) {
            bundles[lang].bundle = true;
            this.push(new gutil.File({
              cwd: cwd,
              base: base,
              path: lang ? path.join(cwd, 'locales', 'bundle.' + lang + '.json')
                         : path.join(cwd, 'bundle.json'),
              contents: new Buffer(JSONstringify(bundles[lang], null, 2))
            }));
          }
          for (lang in bundles) {
            if (lang) {
              (function (destLanguage) {
                promises.push(new Promise(function (resolve, reject) {
                  xliffConv.parseJSON(bundles, {
                    srcLanguage: srcLanguage,
                    destLanguage: destLanguage
                  }, function (output) {
                    self.push(new gutil.File({
                      cwd: cwd,
                      base: base,
                      path: path.join(cwd, 'xliff', 'bundle.' + destLanguage + '.xlf'),
                      contents: new Buffer(output)
                    }));
                    resolve();
                  });
                }));
              })(lang);
            }
          }
          Promise.all(promises).then(function (outputs) {
            callback();
          });
        });
     
        var feedback = gulpif([ '**/bundle.json', '**/locales/*.json', '**/src/**/*.json', '**/xliff/bundle.*.xlf' ], gulp.dest('.'));
     
        var config = {
          // list of target locales to add
          locales: gutil.env.targets ? gutil.env.targets.split(/ /) : []
        }
     
        // Gulp task to add locales to I18N-ready elements and pages
        // Usage: gulp locales --targets="{space separated list of target locales}"
        gulp.task('locales', function() {
          var elements = gulp.src([ 'src/**/*.html' ], { base: '.' })
            .pipe(grepContents(/i18n-behavior.html/))
            .pipe(grepContents(/<dom-module /));
     
          var pages = gulp.src([ 'index.html' ], { base: '.' })
            .pipe(grepContents(/is=['"]i18n-dom-bind['"]/));
     
          return merge(elements, pages)
            .pipe(i18nAddLocales(config.locales))
            .pipe(gulp.dest('.'))
            .pipe(debug({ title: 'Add locales:'}))
        });
     
        if (isPolymerCLI) {
          module.exports = {
            transformers: [
              scan,
              basenameSort,
              dropDefaultJSON,
              preprocess,
              tmpJSON,
              importXliff,
              leverage,
              exportXliff,
              feedback,
              debug({ title: title }),
              size({ title: title })
            ]
          };
        }
        else {
          const polymer = require('polymer-build');
          //const optimize = require('polymer-build/lib/optimize').optimize;
          //const precache = require('polymer-build/lib/sw-precache');
          const PolymerProject = polymer.PolymerProject;
          const fork = polymer.forkStream;
          const polymerConfig = require('./polymer.json');
     
          //logging.setVerbose();
     
          let project = new PolymerProject({
            root: process.cwd(),
            entrypoint: polymerConfig.entrypoint,
            shell: polymerConfig.shell
          });
     
          gulp.task('default', () => {
            // process source files in the project
            let sources = project.sources()
              .pipe(project.splitHtml())
              // I18N processes
              .pipe(scan)
              .pipe(basenameSort)
              .pipe(dropDefaultJSON)
              .pipe(preprocess)
              .pipe(tmpJSON)
              .pipe(importXliff)
              .pipe(leverage)
              .pipe(exportXliff)
              .pipe(feedback)
              .pipe(debug({ title: title }))
              .pipe(size({ title: title }))
              // add compilers or optimizers here!
              .pipe(project.rejoinHtml());
     
            // process dependencies
            let dependencies = project.dependencies()
              .pipe(project.splitHtml())
              // add compilers or optimizers here!
              .pipe(project.rejoinHtml());
     
            // merge the source and dependencies streams to we can analyze the project
            let allFiles = mergeStream(sources, dependencies)
              .pipe(project.analyze);
     
            // fork the stream in case downstream transformers mutate the files
            // this fork will vulcanize the project
            let bundled = fork(allFiles)
              .pipe(project.bundle)
              // write to the bundled folder
              .pipe(gulp.dest('build/bundled'));
     
            let unbundled = fork(allFiles)
              // write to the unbundled folder
              .pipe(gulp.dest('build/unbundled'));
     
            return mergeStream(bundled, unbundled);
          });
        }

    API

    i18nPreprocess(options)

    options object

    • replacingText: Boolean, default: false - If true, UI texts are replaced with {{annotations}}
    • jsonSpace: Number, default: 2 - JSON stringification parameter for formatting
    • srcPath: String, default: 'app' - Path to source root
    • force: Boolean, default: false - Force preprocessing even if i18n-behavior.html or i18n-element.html is not imported
    • dropHtml: Boolean, default: false - If true, drop the preprocessed HTML for output
    • dropJson: Boolean, default: false - If true, drop the extracted JSON files for output
    • constructAttributesRepository: Boolean, default: false - If true, construct localizable attributes repository
    • attributesRepository: Object, default: {} - Input/Output - attributes respository object
    • attributesRepositoryPath: String, default: null - Path to bower_components/i18n-behavior/i18n-attr-repo.html
    • targetVersion: Number, default: 0 - Values: 1 for Polymer 1.x, 2 for Polymer 2.x, 0 for automatic detection by i18n-behavior.html or i18n-element.html import

    Quick Tour

    Quick deployment of polymer-starter-kit-i18n

        npm install -g polymer-cli
        npm install -g generator-polymer-init-i18n-starter-kit
        mkdir i18n-starter-kit
        cd i18n-starter-kit
        polymer init i18n-starter-kit
        # Add Locales
        npm run build locales -- --targets="de es fr ja zh-Hans"
        # Build
        npm run build
        # Translate XLIFF ./xliff/bundle.*.xlf
        # Build and Merge Translation
        npm run build
        # App with Run-time I18N on http://localhost:8080
        polymer serve
        # App with Build-time I18N on http://localhost:8080
        polymer serve build/bundled
    

    Change language

    1. Press F12 to open debugger console on the browser
    2. Navigate to the elements or DOM tab in the debugger
    3. Change lang attribute of html element from "en" to other locales such as "ja"
        <html lang="ja">

    Update UI strings

    1. Change any UI strings in the following HTMLs
        i18n-starter-kit/src/*.html
    
    2. Merge changes into JSON files
        cd i18n-starter-kit
        npm run build
    
    3. Check diffs
        git diff
    

    License

    BSD-2-Clause

    Install

    npm i gulp-i18n-preprocess

    DownloadsWeekly Downloads

    69

    Version

    1.2.3

    License

    BSD-2-Clause

    Last publish

    Collaborators

    • t2ym