s3-tune

Backup and restore AWS S3.

This tool can achieve synchronizations as:

• directory → bucket
• bucket → directory

Here, directory is located in local file system and made up of files and sub directories, while bucket is vessel in S3 where objects saved.

Table of Contents

• Links
• Get Started
• AWS Config
• CLI
• API
  • Parameter options
  • Get Into Sync Progress
  • Progress Events
    • Event: 'created'
    • Event: 'end'
    • Event: 'error'
    • Event: 'ignored'
    • Event: 'moveon'
    • Event: 'no-utf8-filename'
    • Event: 'skipped'
    • Event: 'warning'

Links

• CHANGE LOG
• Homepage

Get Started

In CLI:

# Command "s3-tune" will be generated.
npm install -g s3-tune

# Show help info.
s3-tune -h

# Backup objects into local directory.
s3-tune backup --aws-config <config.json> --bucket <bucket-name> --directory <path-name>

# Restore objects to AWS S3 bucket.
s3-tune restore --aws-config <config.json> --bucket <bucket-name> --directory <path-name>

As API:

const backupS3 = require('s3-sync/backup');

const progress = backupS3(options);

progress.on('error', (err) => {
	// ...
});

progress.on('end', (meta) => {
	// Sychronization successfully finished.
});

AWS Config

For CLI usage, the config.json is a JSON file which contains necessary properties required to access AWS resources. See Class: AWS.Config for details.

Here is a dummy example:

{
	// 20-character Id
	"accessKeyId": "1234567890ABCDEFGHIJKLMNOPQ", 
	
	// 40-character key
	"secretAccessKey": "1234567890abcdefghijklmnopqrstuvwxyz1234" 
}

In API mode, an instance of AWS.S3 should be passed through with name options.s3.

CLI

When installed globally, s3-tune will create a homonymous global command. Run s3-tune -h in terminal to print the man page.

s3-tune will occupy a hidden directory named .s3-tune in home directory of current user.

API

s3-tune offers functions to achieve two different tasks:

• jinang/Progress backup(object options)
• jinang/Progress restore(object options)

1. The functions accept similar options argument, see section Parameter options for details.
2. The functions are all asynchronous and will return an instance of jinang/Progress. Via the returned value, we may learn about and control the sync progress. See section Get Into Sync Progress for details.

Each function may be required solely:

const s3tune    = require('s3-tune');
const s3backup  = require('s3-tune/backup');
const s3restore = require('s3-tune/restore');

// E.g., next two functions are equivalent.
s3tune.backup
s3backup

For convenience, triditional invoking styles are also supported:

• void backup(object options, Function callback)
• void restore(object options, Function callback)
• Promise backup.promise(object options)
• Promise restore.promise(object options)

In traditional styles, the task is regarded as resolved only when all objects successfully uploaded or downloaded. Even if only one failure (something ignored) happens, it fails!

Parameter options

• AWS.S3 options.s3
  Instance of AWS.S3.

• string options.bucket
  S3 bucket name.

• string options.directory
  Absolute pathname in local file system.

• string options.marker
  Position indicating where to start.
  The marker should be an object name.

• string[] options.names
  Object names to be stored into or restored from local file system.

• Function options.mapper Object name mapper.

• Function options.filter Object name filter.
  For s3tune.restore() only.

• Function options.dualMetaFilter
  Filter with paramenter (stat, meta).
  For s3tune.restore() only.

• number options.maxCreated
  Maximum creation allowed (then the progress will be terminated).

• number options.maxCreating
  Maximum cocurrent creating operation allowed.

• number options.maxQueueing
  Maximum queue length allowed.
  When the queue length reaches this number, the progress will pause finding more objects / files until some in queue finished.

• number options.maxErrors
  Maximum exceptions allowed (then the progress will be terminated).

• number options.retry
  Maximum retry times on exception for each object or object list.

Get Into Sync Progress

Via the returned instance of jinang/Progress, we may learn about what happened and then control the sync progress.

• progress.on(string eventName, Function listener)
  See section Events During Sync Progress for aviable events and their accompanied arguments.

• progress.abort()
  Terminate the progress as soon as possible.

• progress.quit()
  Quit the progress gracefully.

Progress Events

Function s3tune.restore() or s3tune.backup() will return an instance of jinang/Progress. And the returned progress is also an instance of events.EventEmitter and may emit following events:

• created
• end
• error
• ignored
• moveon
• no-utf8-filename
• skipped
• warning

Following the conventions in Node.js Documentation, the avaiable event data will be written in list which follows the section title. E.g.

progress.on('fallInLove', function(boy, girl /* , ... */) {
	// ...
});

Aavailable event data will be listed under the event title. Looks like:
Events: 'fallInLove'

• Person boy
  • string name
  • number age
• Person girl
  • string name
  • number age

Not all events are emitted with data. And some or sometimes will be emitted with more than one parameter.

Event: 'created'

• Object meta
  • string name
    Name of object, also the relative pathname of file.

Emitted each time an object put to S3 bucket (in restore mode), or a file written in local file system (in backup mode).

Event: 'end'

• Object stat
  • number errors
    Total errors caught.
  • number created
    Total objects or files created.
  • number ignored
    Total objects or files ignored because of exceptions.

Emitted when the progress accomplished or terminated on too many errors. See options.maxErrors in Parameter options.

Event: 'error'

• Error error
• Object stat
  See Event: 'end' for details of stat.

An error event will be emitted only when:

1. some exception happens when putting object to S3 or writing file in local file system, and
2. it can not be overcome by retrying in limited times.

If there are retry chances left, an Event: 'warning' will be emitted instead.

See options.retry in Parameter options.

Event: 'ignored'

See Event: 'created' for associated event data.

Emitted each time failed to put an object to S3 bucket, or to write a file to local file system.

Event: 'moveon'

• string marker
  Actually, the mark is just the name of object to which the cursor points.

In functions s3tune.backup() and s3tune.restore(), multiple create operations will be made concurrently. The create operations begin in sequence but not always end in the same order. Event moveon will be emitted when something finished ('created' or 'ignored'), and all those ranked ahead have also been finished.

If the progress terminated by exceptions, you can restart it from the position marked by marker.

See options.marker in Parameter options.

Event: 'no-utf8-filename'

• Object fileInfo
  • string dirname
  • Buffer filenameBuffer

Only emitted in s3tune.restore(), each time on meeting with file or directory whose name is not utf-8 encoded. Such file or directory will not be restored to S3 bucket and no 'ignored' or 'skipped' will be emitted.

Event: 'skipped'

See Event: 'created' for associated event data.

Emmited each time when something skipped by the filter.

See options.filter and options.dualMetaFilter__ in Parameter options.

Event: 'warning'

• Error error
• Object stat
  See Event: 'end' for details of stat.

A warning event will be emitted only when:

1. some exception happens when putting object to S3 or writing file in local file system, and
2. there are still chances left to retry.

When all retry chances are exhausted, an Event: 'error' will be emitted instead.

See options.retry in Parameter options.