taara
A PostgreSQL table snapshot tool for node.js.
Installation
Install it using npm:
$ npm install taara
Or get it directly from: http://github.com/macobo/taara
Usage
To use the library, you need to configure to use storage engine. At this time, the library supports on-disk backups as well as S3 Backups.
Using S3 storage
var taara = ; var storageEngine = 'my-bucket' s3AuthParams;taara;
s3AuthParams
- See AWS SDK documentation for available options which are passed to new AWS.S3(): http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Config.html#constructor-property
Using on-disk storage
var storageEngine = '/path/to/backup/location';taara;
Storing a snapshot of a table.
var pgAuth = host: "localhost" port: 5432 user: "postgres" database: "postgres" password: "my-password"; // Stored along with the snapshot, can be later used during the restore processvar userMetadata = my: "value" ; taara;
The snapshot will be stored at snapshot/events_20160112-235323.snapshot
(depending
on date) and snapshot at metadata/events_20160112-235323.metadata
.
Listing snapshots and fetching stored metadata.
taara // Print out list of identifiers // And fetch metadata for one of the backups. ;
Restoring a snapshot
taara ;
API Documentation
Motivation
At Heap Analytics we use sharding, PostgreSQL and sharding extensively to store with our user data. As each shard is replicated, even if we lose a single database instance, we won't lose any data. However, if more databases than the replication factor should die, we'd need to deal with full-blown database restores simply to get at a few shards out of tens of thousands - something which is costly in terms of storage as well as time.
This library is an attempt to deal better with that problem - instead of making full-blown database backups, we store snapshots of tables on S3 with some Kafka metadata. Should shards "disappear" we can use the snapshot to restore the shards to a good state and replay all new events that that happened since using Kafka.
Contributing
To setup, you need to install a few dependencies:
npm install -g typescript gulp
npm install
To compile and to do a test run:
docker-compose up -d
- starts a database and fake_s3 instance. [1]gulp test
[1]: More on Docker Compose here.