alamo

Overview

alamo is a wrapper around knox that provides an higher level abstraction for s3 with handling of response status codes and automatic parsing of XML error bodies. It also provide a consistent full (writing and reading) streaming interface, including multipart upload for large artifacts. Alamo implements automatic retries on error with exponential back-off.

Why alamo?

1. knox is quite low-level with regards to response and error handling which lead to code duplication to parse response status codes and errors for each request as pointed out by @domenic himself in https://github.com/Automattic/knox/issues/114
2. the aws-sdk allow uploading streams (in an awkward way) but does not allow retrieving streams
3. the aws-sdk allow multipart upload but is low-level and let a lot to implement by the caller
4. knox-mpu allows multipart upload but buffers everything in memory which is only viable when uploading from your desktop without concurrency, but not viable from a server
5. neither knox or knox-mpu implements retries which is problematic when uploading large artifacts to s3
6. Fort Knox - Fort Alamo

Usage

npm install --save alamo

API

alamo.createClient(options)

Returns an s3 client. It accepts the same options as knox.createClient plus the following ones that set how retries work (see retry):

• retries: max number of retries (default 10)
• factor: factor for retry (default 2)
• minTimeout: minimum time for first retry (default 1000)
• maxTimeout: maximum time for all retries (default 60000)

var client = require('alamo').createClient({
    key: process.env.AWS_ACCESS_KEY_ID,
    secret: process.env.AWS_SECRET_ACCESS_KEY,
    bucket: process.env.AWS_BUCKET,
    region: process.env.AWS_REGION
});

Client.prototype.client

Access to the lower level knox client

Client.prototype.createReadStream(filename, headers)

Returns a readable stream.

var fs = require('fs');
 
client.createReadStream('/somekey').pipe(fs.createWriteStream('somefile'));

• filename: the s3 file name to retrieve
• headers: optional headers
• options: retry options

Alias: readStream

Client.prototype.createWriteStream(filename, headers, content)

Returns a writable upload stream. You can optionally pass a buffer to upload instead of piping to it.

var fs = require('fs');
var ws = client.createWriteStream('/somekey');
 
fs.createReadStream('somefile')
    .pipe(ws)
    .on('error', console.error)
    .on('finish', console.log.bind(console, 'Upload complete'));

• filename: the s3 file name to upload to
• headers: optional headers
• content: optional content to upload. If content is passed, it is passed to the underlying request.end
• options: retry options

Alias: writeStream

Client.prototype.stream(method, filename, headers, content)

Generic stream implementation that accepts the method as 1st argument as 2nd argument

var fs = require('fs');
var ws = client.stream('PUT', '/somekey');
 
fs.createReadStream('somefile')
    .pipe(ws)
    .on('error', console.error)
    .on('finish', console.log.bind(console, 'Upload complete'));

• method: the http method e.g. GET, PUT
• filename: the s3 file name to upload to
• headers: optional headers
• content: optional content to upload. If content is passed, it is passed to the underlying request.end
• options: retry options

Client.prototype.get(filename, headers, cb)

Get an object and retrieve the response with the body

client.get('/somekey', function (err, res) {
  if (err) console.error(err);
  else console.log(res.statusCode, res.body.toString());
});

• filename: the s3 file name to retrieve
• headers: optional headers
• options: retry options
• cb: callback that returns an error or null as 1st argument, and the response with the body if no error as 2nd argument

Client.prototype.del(filename, headers, cb)

Delete an object from s3

client.del('/somekey', function (err, res) {
  if (err) console.error(err);
  else console.log('object deleted %d', res.statusCode);
});

• filename: the s3 file name to delete
• headers: optional headers
• options: retry options
• cb: callback that returns an error or null as 1st argument, and the response if no error as 2nd argument

Client.prototype.put(filename, content, headers, cb)

Put an object

client.put('/somekey', 'somedata', function (err, res) {
  if (err) console.error(err);
  else console.log('object uploaded %d', res.statusCode);
});

• filename: the s3 file name to upload to
• content: content to upload
• headers: optional headers
• options: retry options
• cb: callback that returns an error or null as 1st argument, and the response if no error as 2nd argument

Client.prototype.post(filename, content, headers, cb)

Post an object

client.post('/somekey', 'somedata', function (err, res) {
  if (err) console.error(err);
  else console.log('object uploaded %d with etag %s', res.statusCode, res.headers.etag);
});

• filename: the s3 file name to post to
• content: content to post
• headers: optional headers
• options: retry options
• cb: callback that returns an error or null as 1st argument, and the response if no error

Client.prototype.request(method, filename, content, headers, cb)

Generic non streaming interface

client.request('PUT', '/somekey', 'somedata', function (err, res) {
  if (err) console.error(err);
  else console.log('object uploaded %d with etag %s', res.statusCode, res.headers.etag);
});

• method: the http method e.g. GET, PUT, DELETE, POST
• filename: the s3 file name
• content: content to post
• headers: optional headers
• cb: callback that returns an error or null as 1st argument, and the response with the body if no error

Client.prototype.signedUrl(filename, expiration, options)

Returns a signed url

var url = client.signedUrl('/somekey', 1000 * 60 * 15);
console.log(url);

• filename: the s3 file name to retrieve
• expiration: number of milliseconds that the signed url is valid for
• options: signed url options passed to knox, take verb, contentType, and qs object

Client.prototype.multipart(filename, headers)

Returns a writable stream to upload using the s3 multipart API. The stream is uploaded by chunks of 5mb in parallel with max concurrent uploads and automatic retries

var fs = require('fs');
var ws = client.multipart('/somekey');
 
fs.createReadStream('somefile')
    .pipe(ws)
    .on('error', console.error)
    .on('finish', console.log.bind(console, 'Upload complete'));

• filename: the s3 file name to upload to
• headers: optional headers
• options: retry options

Comparison with knox

Retrieve a stream with knox with full error handling

var fs = require('fs');
var XML = require('simple-xml');
var req = client.get('/filename');
req.on('response', function (res) {
  if (res.statusCode !== 200) {
  	var body = '';
  	res.on('data', function (chunk) {
  		body += chunk.toString();
  	});
    res.on('end', function () {
        try {
            body = XML.parse(body);
            cb(new Error(body.message);
        } catch (err) {
            cb(new Error(body);
        }
        cb(null, res);
    });
  	res.on('error', cb);
  	return;
  }
  res.pipe(fs.createWriteStream('filename').on('error', cb).on('finish', cb.bind(null, null));
});
req.on('error', cb);

Roadmap

• Handle redirects and other 30x status codes: http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html
• Implement global max concurrent uploads
• Implement basic progress for multipart upload
• Accept string value for expiration that can be parsed by ms
• Add higher level functions for "file" upload / download
• Maybe use multipart upload automatically if content-length is unknown?
• Maybe allow automatic handling (parsing, marshalling) of json?

Contributions

Please open issues for bugs and suggestions in github. Pull requests with tests are welcome.

Author

Jerome Touffe-Blin, @jtblin, About me

License

alamo is copyright 2015 Jerome Touffe-Blin and contributors. It is licensed under the BSD license. See the include LICENSE file for details.