moplog
A generic processor of a MongoDB oplog, routing registered transactions to the specified consumer. It handles operations of type insert, update, delete, and command.
Usage
var Moplog = ; // Initialize with the location of the nconf config file to use and the // directory to the consumers.var moplog = './config.json' './consumers'; // Assuming that your config has valid MongoDB connection details as well as// valid consumers configured, this will kick off the processing.moplog;
Configuration
Configure moplog through a configuration file which you pass in as the first argument. Here is an example:
"source": "host": "mongodb://localhost:27017" "db": "local" "collection": "oplog.$main" "user": "" "pass": "" "collections": "myDatabase.myCollection": "myConsumer" "period": 5000 "lastTs": 1429559350001
source
: MongoDB server with oplog to processhost
: server hostname with mongodb:// prepended protocol (default:"mongodb://localhost:27017"
)db
: database name containing the oplog (default:"local"
)collection
: oplog collection name (default:"oplog.$main"
)user
: specify when server uses authentication, user must have access to oplog (default:""
, i.e., no authentication)pass
: specify when server uses authentication, user must have access to oplog (default:""
, i.e., no authentication)
collections
: map of database.collection to consumer to route operations to (default:{}
, i.e., no consumers)period
: how frequently to query for new oplog entries (defaults:5000
, i.e., 5 seconds)lastTs
: timestamp of last processed oplog document with ms resolution. This is updated after each additional document is processed. (default:0
, i.e., from the earliest operation available)
New consumers should be added to the app/consumers
subdirectory. Refer to test/consumers/testConsumer
as a reference of how to implement.
API
Moplog(configFile, consumerDir)
Constructor which takes two arguments to specify the config file and the base directory of the consumers. Return a moplog instance.
Moplog.connect()
Connects to the MongoDB referenced in the config, obtains a handle on the oplog collection, and kicks off processing.
Moplog.getConfig()
Returns the active configuration. Example response:
"source": "host": "mongodb://localhost:27017" "db": "local" "collection": "oplog.$main" "user": "" "pass": "" "collections": "source.content": "defaultConsumer" "period": 5000 "lastTs": 1430017964001
Moplog.getLag()
Returns the estimated lag in minutes of processed records based on timestamp of last processed operation. Example response:
"lagInMinutes": 1230
Logging
Moplog logs to a rolling log stored in the process' root directory as moplog.dat. It also supports Loggly by setting the following environment variables:
- LOGGLY_SUBDOMAIN
- LOGGLY_AUTH
- LOGGLY_INPUT_NAME
- LOGGLY_INPUT_TOKEN
Testing
All linting are configured in Gruntfile.js using both jshint and jscs.
grunt lint
All mocha unit tests are included in the test/ directory. These are run through the test grunt task:
grunt unit
To determine unit test coverage, run the cov grunt task. A terminal friendly coverage output will give percentages and highlight lines not covered by any test:
grunt cov