knox-mpu

Provide multi part upload functionality to Amazon S3 using the knox library

A Node.js client designed to make large file uploads to Amazon S3 via the MultiPartUpload API simple and easy. It's built on top of the excellent Knox library from the guys over at LearnBoost.

  • Simple and easy to use
  • Pipe either a file, or a stream directly to S3 (No need to know the content length first!)
  • Automatically separates a file/stream into appropriate sized segments for upload
  • Asynchronous uploading of segments
  • Handy events to track your upload progress

Planned

  • Better error handling (reuploading failed parts, etc)

Installation is done via NPM, by running npm install knox-mpu

To upload a stream, simply pass the stream when constructing the MultiPartUpload. The upload will then listen to the stream, and create parts from incoming data stream. When a part reaches the minimum part size, it will attempt to upload it to S3.

 
// Create a Knox client first 
var client = knox.createClient({ ... }),
    upload = null;
    
 
upload = new MultiPartUpload(
            {
                client: client,
                objectName: 'destination.txt', // Amazon S3 object name 
                stream: stream
            },
            // Callback handler 
            function(errres) {
                // If successful, will return a JSON object containing Location, Bucket, Key and ETag of the object 
            }
        );

To upload a file, pass the path to the file in the constructor. Knox-mpu will split the file into parts and upload them.

 
// Create a Knox client first 
var client = knox.createClient({ ... }),
    upload = null;
    
 
upload = new MultiPartUpload(
            {
                client: client,
                objectName: 'destination.txt', // Amazon S3 object name 
                file: ... // path to the file 
            },
            // Callback handler 
            function(errres) {
                // If successful, will return a JSON object containing Location, Bucket, Key and ETag of the object 
            }
        );

The following options can be passed to the MultiPartUpload constructor -

  • client Required The knox client to use for this upload request
  • objectName Required The destination object name/path on S3 for this upload
  • stream The stream to upload (required if file is not being supplied)
  • file The path to the file (required if stream is not being supplied)
  • headers Any additional headers to include on the requests
  • partSize The minimum size of the parts to upload (default to 5MB).

The MultiPartUpload will emit a number of events -

  • initiated Emitted when the multi part upload has been initiated, and received an upload ID. Passes the upload id through as the first argument to the event
  • uploading Emitted each time a part starts uploading. The part id is passed as the first argument.
  • uploaded Emitted each time a part finishes uploading. Passes through an object containing the part id and Amazon ETag for the uploaded part.
  • error Emitted each time a part upload fails. Passes an object containing the part id and error message
  • completed Emitted when the upload has completed successfully. Contains the object information from Amazon S3 (location, bucket, key and ETag)