@gspace/mongotools
TypeScript icon, indicating that this package has built-in type declarations

2.2.9 • Public • Published

node-mongotools

NPM

This project provides 2 wrappers for :

  • mongodump,
  • mongorestore.

This project also include dropbox integration feature to

  • dump and upload onto dropbox,
  • restore from a dropbox hosted mongo backup.

There is an autonomous feature called rotation that provide a backup file rotation mechanism

  • remove N oldest deprecated backups.

This readme contains some samples. Other samples are under examples/.

You may notice that mongodump and mongorestore binaries are part of database-tools (follow standard installation for common OS). These binaries are mandatory for node-mongotools nodejs project as we are talking here of "wrapper" : a more convenient way to launch command + arguments from NodeJS code).

From a GitHub Actions context, you can install them via action-mongo-tools. There is an example in this project in main workflow that execute tests.

Command line usage

You should decide to install it globally without getting source or by cloning repository.

Install globally (beta)

npm install -g node-mongotools@latest

When using a global install node-mongotools command must be directly available (ie. you can replace node mt by node-mongotools in examples).

You could revert thins by using

npm uninstall -g node-mongotools

Install by getting source

git clone https://github.com/boly38/node-mongotools.git
# install dependencies
npm install

Environment setup - first time only

#~ setup environment variables
cp env/initEnv.template.sh env/initEnv.dontpush.sh
# you must update env/initEnv.dontpush.sh

Set your preferences

source your options

. ./env/initEnv.dontpush.sh

Basic feature

# create a mongo dump
node mt dump
# or if you rely on a global install
# node-mongotools dump
# apply the same logic for next examples 

# create a encrypted mongo dump
node mt dumpz

# list backups
node mt list

# restore a mongo local dump
# please note that mt restore use following options : dropBeforeRestore: true, deleteDumpAfterRestore: true
node mt restore backup/myDatabase__2020-11-08_150102.gz

# rotate backup files
node mt rotation

# Helper : show current options values
node mt options

Add in-line extra options

You could play with env default options plus in-line command extra options

# create dump of a given 'shippingprices' collection, provide a target backup name as '2023_01_shippingPrices.gz', and show mongodump command
MT_COLLECTION=shippingprices MT_FILENAME=2023_01_shippingPrices.gz MT_SHOW_COMMAND=true node mt dump

# show backup in list
node mt list

# using mongo: drop a given collection
mongo myDb --eval "db.shippingprices.drop()"

# restore collection
MSYS_NO_PATHCONV=1 MT_COLLECTION=shippingprices MT_SHOW_COMMAND=true node mt restore /backup/2023_01_shippingprices.gz
# Note that collection option will produce wildcard in nsInclude arg '--nsInclude myDb.*'

Dropbox feature

You have a dropbox access token in your preferences, (cf. "Mongo tools options")

# create a mongo dump is the same command
node mt dump
# restore a mongo dropbox dump 
node mt restore /backup/myDatabase__2020-11-08_150102.gz

# git bash for windows users having absolute path issue could use the following command
unalias node
MSYS_NO_PATHCONV=1 node mt restore /backup/myDatabase__2020-11-08_150102.gz

# rotate local and dropbox backup files
node mt rotation

Library use

Install dependency

You have to import as dependency

npm install node-mongotools

Define the requirements, example:

import { MongoTools, MTOptions, MTCommand } from "node-mongotools";

var mongoTools = new MongoTools();
const mtOptions = {
        db: 'myDb',
        port: 17017,
        path: '/opt/myapp/backups',
        dropboxToken: process.env.MYAPP_DROPBOX_SECRET_TOKEN
      };

This project is compatible with ES Module projects that rely on import. For now it's not compatible with CommonJS (require). But contribution are welcome.

List dumps

var promiseResult = mongoTools.list(mtOptions);

Dump

var promiseResult = mongoTools.mongodump(mtOptions);

Restore

var promiseResult = mongoTools.mongorestore(mtOptions);

Rotation

var promiseResult = mongoTools.rotation(mtOptions);

Mongo tools options

Each mongotools feature rely on Mongo tools options (aka. MTOption).

Options precedence is the following:

  • take options attribute if set,
  • else take related environment variable if any,
  • else take default value if any,
  • else if it's a mandatory option, throw an error.

TIP: you could also show current options by doing:

console.log(new MTOptions());

shared options

These options are used by dump and restore.

Either uri or host/port/db:

option env required default value description
uri MT_MONGO_URI true (none) mongodump uri, example mongodb+srv://granted-user:MySecretHere@cluster0.xzryx.mongodb.net/tMyDatababse. You could omit database name to dump all databases.

or

option env required default value description
db MT_MONGO_DB true (none) mongo database name. For dump only, you could set it to * to dump all databases
host MT_MONGO_HOST false 127.0.0.1 mongo database hostname
port MT_MONGO_PORT false 27017 mongo database port
username MT_MONGO_USER false (none) mongo database username
password MT_MONGO_PWD false (none) mongo database password
authDb MT_MONGO_AUTH_DB false admin mongo auth database

ssl options

Optional ssl related options

option env required default value description
ssl MT_MONGO_SSL false (none) if "1" then add --ssl option
sslCAFile MT_MONGO_SSL_CA_FILE false (none) .pem file containing the root certificate chain
sslPEMKeyFile MT_MONGO_SSL_PEM_KEY_FILE false (none) .pem file containing the certificate and key
sslPEMKeyPassword MT_MONGO_SSL_PEM_KEY_PASSWORD false (none) password to decrypt the sslPEMKeyFile, if necessary
sslCRLFile MT_MONGO_SSL_CRL_FILE false (none) pem file containing the certificate revocation list
sslFIPSMode MT_MONGO_SSL_FIPS false (none) if "1" then use FIPS mode of the installed openssl library
tlsInsecure MT_MONGO_TLS_INSECURE false (none) if "1" then bypass the validation for server's certificate chain

mongodump options

option env required default value description
path MT_PATH false backup dump target directory, created if it doesn't exist
dumpCmd MT_MONGODUMP false mongodump mongodump binary
fileName MT_FILENAME false <dbName_date_time.gz> dump target filename
encrypt false false encrypt the dump using secret
secret MT_SECRET false null secret to use if encrypt is enabled (aes-256-ctr require 32 byte length key)
encryptSuffix false .enc encrypt file suffix
includeCollections false (none) Deprecated - please use collection
collection MT_COLLECTION false (none) Collection to include, if not specified all collections are included
excludeCollections MT_EXCLUDE_COLLECTIONS false (none) Collections to exclude, if not specified all collections are included
numParallelCollections false 4 Number of collections mongodump should export in parallel.
viewsAsCollections false false When specified, mongodump exports read-only views as collections.

Simple example:

import { MongoTools, MTOptions, MTCommand } from "node-mongotools";
var mongoTools = new MongoTools();

mongoTools.mongodump({ 
   db:'myDatabase',
   path:'backup',
   username:'root', password:'mypass', authDb:'admin'
})
.then((success) => console.info("success", success) )
.catch((err) => console.error("error", err) );

mongorestore options

option env required default value description
dbFrom MT_MONGO_DB_FROM false (none) name of the source db (if db is not specified)
dbTo MT_MONGO_DB_TO false (none) name of the target db (if db is not specified)
dumpFile MT_DUMP_FILE true (none) dump file to restore
restoreCmd MT_MONGORESTORE false mongorestore mongorestore binary
dropBeforeRestore false false set it to true to append --drop option
deleteDumpAfterRestore false false set it to true to remove restored backup file
decrypt false false decrypt the dump using secret. Activated if suffix is detected
secret MT_SECRET false null secret to use if decrypt is enabled

Simple example:

import { MongoTools, MTOptions, MTCommand } from "node-mongotools";
var mongoTools = new MongoTools();

mongoTools.mongorestore({ 
   dumpFile:'backup/myDatabase__2020-11-8_160011.gz',
   username:'root', password:'mypass', authDb:'admin'
})
.then((success) => {
  console.info("success", success.message);
  if (success.stderr) {
    console.info("stderr:\n", success.stderr);// mongorestore binary write details on stderr
  }
})
.catch((err) => console.error("error", err) );

Dropbox options

You could create a dropbox app to get a token : cf. https://www.dropbox.com/developers/apps/ "Generated access token"

option env required default value description
dropboxLocalPath MT_DROPBOX_LOCAL_PATH false "dropbox" local directory to receive dropbox dump
dropboxAppKey MT_DROPBOX_APP_KEY false (none) (refresh token based) dropbox feature application key. (*1)
dropboxAppSecret MT_DROPBOX_APP_SECRET false (none) (refresh token based) dropbox feature application secret. (*1)
dropboxRefreshToken MT_DROPBOX_REFRESH_TOKEN false (none) (refresh token based) dropbox feature application refreshToken. (*1)
dropboxToken (*2) MT_DROPBOX_TOKEN false (none) DEPRECATED - activate dropbox feature if present. (*2)

(*1) dropboxAppKey, dropboxAppSecret and dropboxRefreshToken (long-lived offline refresh token) are required to activate optional dropbox feature. You have some help about how to get them in dropbox-refresh-token dedicated helper repository. (*2) dropboxToken option is DEPRECATED : This was legacy old-long-lived access-token - this kind of token are no more available from dropbox developers portal (src). Please switch to dropboxAppKey, dropboxAppSecret, and dropboxRefreshToken instead. For now this is still supported.

When a token is set,

  • the list feature will list the / + path dropbox directory
  • the mongodump feature will upload the dump onto / + path dropbox directory (in addition to spawn it locally),
  • the mongorestore feature will use dumpFile as dropbox dump location and retrieve it into dropboxLocalPath before doing the mongorestore action.

Rotation options

A safe time windows is defined by :

  • now - rotationWindowsDays day(s) ===> now
    where backups can't be removed.

Backup out of safe time windows are called deprecated backup.

  • rotationMinCount: minimum deprecated backups to keep,
  • rotationCleanCount: number of (oldest) deprecated backups to delete.
option env required default value description
rotationDryMode MT_ROTATION_DRY_MODE false false dont do delete actions, just print it
rotationWindowsDays MT_ROTATION_WINDOWS_DAYS true 15 safe time windows in days since now
rotationMinCount MT_ROTATION_MIN_COUNT true 2 minimum deprecated backups to keep.
rotationCleanCount MT_ROTATION_CLEAN_COUNT true 10 number of (oldest first) deprecated backups to delete.

Simple example:

MT_ROTATION_CLEAN_COUNT=2 \
MT_ROTATION_DRY_MODE=true \
MT_ROTATION_WINDOWS_DAYS=3 \ node mt rotation

Example details: if there is a backup that is more than 3 days old, keep 2 newer ones and delete the 10 oldest.

Dropbox limits:

  • rotation feature will not apply if dropbox backup target directory content contains more than 2000 files.

How to contribute

You're not a dev ? just submit an issue (bug, improvements, questions). Or else:

  • Clone
  • Install deps
  • Then mocha tests
git clone https://github.com/boly38/node-mongotools.git
npm install
npm run test
  • you could also fork, feature branch, then submit a pull request.

Services or activated bots

badge name description
CI/CD Github actions Continuous tests.
Audit Github actions Continuous vulnerability audit.
Reviewed by Hound Houndci JavaScript automated review (configured by .hound.yml)

Contributions

Repobeats

provided by Repobeats

Package Sidebar

Install

npm i @gspace/mongotools

Weekly Downloads

30

Version

2.2.9

License

MIT

Unpacked Size

222 kB

Total Files

38

Last publish

Collaborators

  • gspace