@tylercapx/s3-zip

s3-zip

Forked from s3-zip. Download selected files from an Amazon S3 bucket as a zip file. Added ability to rename s3 files if given an object that maps s3 file names to its new name.

Install

npm install @tylercapx/s3-zip

AWS Configuration

Refer to the AWS SDK for authenticating to AWS prior to using this plugin.

Usage

Zip specific files

const fs = require('fs');
const join = require('path').join;
const s3Zip = require('@tylercapx/s3-zip');

const region = 'bucket-region';
const bucket = 'name-of-s3-bucket';
const folder = 'name-of-bucket-folder/';
const file1 = 'Image A.png';
const file2 = 'Image B.png';
const file3 = 'Image C.png';
const file4 = 'Image D.png';
const fileNameMap = {
  [file1]: 'New Image Name A.png';
  [file2]: 'New Image Name B.png';
  [file3]: 'New Image Name C.png';
  [file4]: 'New Image Name D.png';
}


const output = fs.createWriteStream(join(__dirname, 'use-s3-zip.zip'));

s3Zip.archive({ region: region, bucket: bucket }, folder, [file1, file2, file3, file4], null, fileNameMap).pipe(output);

You can also pass a custom S3 client. For example if you want to zip files from a S3 compatible storage:

const aws = require('aws-sdk');

const s3Client = new aws.S3({
    signatureVersion: 'v4',
    s3ForcePathStyle: 'true',
    endpoint: 'http://localhost:9000',
});

s3Zip.archive({ s3: s3Client, bucket: bucket }, folder, [file1, file2], null, fileNameMap).pipe(output);

Zip files with AWS Lambda

Example of s3-zip in combination with AWS Lambda.

Zip a whole bucket folder

const fs = require('fs');
const join = require('path').join;
const AWS = require('aws-sdk');
const s3Zip = require('@tylercapx/s3-zip');
const XmlStream = require('xml-stream');

const region = 'bucket-region';
const bucket = 'name-of-s3-bucket';
const folder = 'name-of-bucket-folder/';
const s3 = new AWS.S3({ region: region });
const params = {
    Bucket: bucket,
    Prefix: folder,
};

const filesArray = [];
const files = s3.listObjects(params).createReadStream();
const xml = new XmlStream(files);
xml.collect('Key');
xml.on('endElement: Key', function (item) {
    filesArray.push(item['$text'].substr(folder.length));
});

xml.on('end', function () {
    zip(filesArray);
});

function zip(files) {
    console.log(files);
    const output = fs.createWriteStream(join(__dirname, 'use-s3-zip.zip'));
    s3Zip.archive({ region: region, bucket: bucket, preserveFolderStructure: true }, folder, files).pipe(output);
}

Tar format support

s3Zip.setFormat('tar').archive({ region: region, bucket: bucket }, folder, [file1, file2]).pipe(output);

Zip a file with protected password

s3Zip
    .setRegisterFormatOptions('zip-encrypted', require('archiver-zip-encrypted'))
    .setFormat('zip-encryptable')
    .setArchiverOptions({ zlib: { level: 8 }, encryptionMethod: 'aes256', password: '123' })
    .archive({ region: region, bucket: bucket }, folder, [file1, file2])
    .pipe(output);

Archiver options

We use archiver to create archives. To pass your options to it, use setArchiverOptions method:

s3Zip
    .setFormat('tar')
    .setArchiverOptions({ gzip: true })
    .archive({ region: region, bucket: bucket }, folder, [file1, file2]);

Organize your archive with custom paths and permissions

You can pass an array of objects with type EntryData to organize your archive.

const files = ['flower.jpg', 'road.jpg'];
const archiveFiles = [
    { name: 'newFolder/flower.jpg' },

    /* _rw_______ */
    { name: 'road.jpg', mode: parseInt('0600', 8) },
];
s3Zip.archive({ region: region, bucket: bucket }, folder, files, archiveFiles);

Using with ExpressJS

s3-zip works with any framework which leverages on NodeJS Streams including ExpressJS.

const s3Zip = require('s3-zip');

app.get('/download', (req, res) => {
    s3Zip.archive({ region: region, bucket: bucket }, '', 'abc.jpg').pipe(res);
});

Above should stream out the file in the response of the request.

Debug mode

Enable debug mode to see the logs:

s3Zip.archive({ region: region, bucket: bucket, debug: true }, folder, files);

Testing

Tests are written in Node Tap, run them like this:

npm t

If you would like a more fancy coverage report:

npm run coverage