@oliasoft-open-source/node-json-migrator

Node JSON Migrator

A simple JSON change management tool for Node.js

• Lightweight JSON change management tool for Node.js applications
• Written in JavaScript
• Pre-transpiled (no extra build steps)
• Not tied to any framework
• Open source (MIT License)

Features

• Versioning of JSON payloads
• Write change scripts (migrations) in JavaScript
• Apply changes to JSON payloads using a convenient migrate() function
• Works great in a team setting (multiple developers can track and handle changes)
• Supports automated change deployment across multiple environments
• Automatically tracks and applies only new (pending) change scripts
• Automatically detects and warns when previous scripts have been altered
• Optionally track version history in a PostgreSQL database (pg-promise interface)

How to use

Installation and setup

The database (if used) must already exist and have permissions granted.

Install the tool directly from Gitlab (published NPM packages may be available in future):

npm install --save git+https://gitlab.com/oliasoft-open-source/node-json-migrator.git

Defining migrations (change scripts)

Use the createMigration() function to generate a skeleton migration script, for example:

import {createMigration} from 'node-json-migrator';

const directory = 'json-migrations';
const description = 'add-panda';
await createMigration(directory, description);

This will create and populate skeleton files:

• add-panda/add-panda.js is the skeleton change script (alter this to describe the desired payload change)
• plan.json is the execution plan (defines the execution order / sequence for migration scripts)

Migration scripts export default functions that receive a payload and return the next payload. They must use immutable update patterns, and Immer is recommended for this. For example, after editing, the scripts might look like this:

json-migrations/add-panda/add-panda.js (adds a property to the payload)

import produce from 'immer';

export default (payload) = produce(state, (draft) => {
   draft.panda = 'Elliot';
});

json-migrations/plan.json (defines the migration execution plan)

[
  {
    "fileHash": "70fd80fd19828435b8656b8d01f99b840381050ded4aec2dd126e2f0c9864c88",
    "fileName": "add-panda.js",
    "sequence": "1"
  }
]

Note: the fileHash property is the SHA256 hash of the migration file content. You don't have to manually maintain that, the tool will sync it automatically for you.

Running migrations

In your application source code, import and call the migrate() function. The simplest usage (runs all migrations) is:

import {migrate} from 'node-json-migrator';

const {nextPayload} = await migrate({
  payload: {},
  config: {
    directory: 'json-migrations'
  }
});

Normally you want to track payload versions, and only execute pending migrations. To do this, you need to connect to a PostgreSQL database and pass a connected pg-promise database object. Then it looks like this:

import pgPromise from 'pg-promise';
import {migrate} from 'node-json-migrator';

//connect to the database somewhere in the database layer
const pgp = pgPromise({});
const pgpHelpers = pgp.helpers;
const database = pgp(/*connection string*/);
database.connect();

//if we know the current payload version, we can run only the newer, pending migration scripts
const currentPayloadVersion = undefined;

const {nextPayload, nextVersion} = await migrate({
  payload: {},
  config: {
    directory: 'json-migrations',
    database, //connected database object
    pgpHelpers, //pg-promise helpers: https://vitaly-t.github.io/pg-promise/helpers.html
    version: currentPayloadVersion
  }
});

Other config options

config = {
  dry: true, //execute without writing to database tables or files
  force: true, //execute bypassing some validation warnings
}

Usage Rules

• files in the plan must exist
• filenames must be unique and follow formatting rules
• sequences must be unique
• not allowed to change sequence numbers after releasing (merging) a migration
• not allowed to remove released files from the plan
• should release only one migration script at a time
• not allowed to alter/modify released migration files (instead, write a new migration to replace it)
• migration scripts should be repeatable (not fail if they run twice on the same payload)

Backfitting bugfixes

If you release a bad migration script, there is an escape-hatch for backfitting bugfixes:

• rename the faulty script to .skip.js (or .skip2.js if it happened more than once)
• add a new migration file with the original filename, to replace it (fix bug in new file)
• the replacement script will not re-execute on payloads that already had the original script applied

Or, you can backfit a script earlier in the sequence

• add a new migration file
• set its sequence to for example 1.1 (subsequence number)
• the backfitted script will execute one time on all payloads

How to contribute

Contribution is welcome via issue tickets and merge requests.

• coding style (ESLint and prettier) is mandatory (applied via pre-commit hook)
• to test: npm run test
  • database is mocked with pg-mem (in-memory database)
  • filesystem is mocked mock-fs
  • coverage reports are in test/coverage (not committed)
• to build: npm run buld (this transpile a production build to the dist directory)
  • the build dist directory should be committed