gulp-download-2
Heavily derived from gulp-download-stream and gulp-download
A tiny hyperquest gulp wrapper to download files over HTTP/HTTPS/FTP/FTPS + following redirects.
Features
- Progress bar
- Concurrent downloads without busy-waiting
- Redirect support (up to 10 hops)
-
ftp(s)://
support
Here's a nice example:
Downloading http://ipv4.download.thinkbroadband.com/512MB.zip...
downloading [====================] 205491/bps 100% 0.0s
Done
downloading [====================] 358297/bps 100% 0.0s
Done
downloading [====================] 2664869/bps 100% 0.0s
Done
downloading [=======-------------] 2126399/bps 33% 65.9s
Done
Problem
Other gulp download plugins buffer file contents in full before flushing to disk. gulp-download2
bypasses extra buffering by directly writing chunks to disk.
Solution
gulp-download2
avoids unnecessary and connection pooling.
gulp-download
vs. gulp-download2
Benchmarks: In gulp-download2
we saw an average increase of CPU utilization by 31% whereas gulp-download
writes the file content to a buffer and writes to the disk. This process is not as labor intensive as system calls:
gulp-download2 |
gulp-download |
---|---|
Looking at the memory consumption in gulp-download2
shows a max memory consumption of 262 MB whereas gulp-download
buffers the content into memory leading to a steady increase:
gulp-download2 |
gulp-download |
---|---|
Note: Profiling done with Syrupy.py and v8-profile
.
Installation
npm install gulp-download2 --save-dev # or to use yarn...
yarn add gulp-download2 --dev
Basic Usage
const gulp = require('gulp');
const download = require('gulp-download2');
gulp.task('download', () => download('http://example.com/file.jpg').pipe(gulp.dest('build')));
Download Multiple Files
To download multiple files, pass an array of strings to download
.
gulp.task('download', function () {
return download(['http://example.com/file.a', 'https://example.com/file.b']).pipe(
gulp.dest('build')
);
});
The files are downloaded concurrently into stream of Vinyl files, and so are suitable to be piped into other gulp plugins. Each Vinyl file is also itself a stream, and so any downstream plugins must also support stream-based Vinyl files.
Specify Local File Name
You can specify the local file names of files downloaded. You can do this for one file:
gulp.task('download', function () {
return download({
url: 'http://example.com/file.txt',
file: 'foo.txt',
}).pipe(gulp.dest('build'));
});
or for multiple files:
gulp.task('download', function () {
const files = [
{
url: 'http://example.com/file.txt',
file: 'foo.txt',
},
{
url: 'http://example.com/file2.csv',
file: 'data.csv',
},
];
return download(files).pipe(gulp.dest('build'));
});
Handling Errors
There are two different kinds of errors that can arise when we attempt to download from a remote resource:
- Hyperquest encounters an error with the stream
- Sends event object as a callback parameter
- Hyperquest returns an error status code (i.e. 404)
-
res.statusCode
is passed as a callback parameter
-
In either case, we can handle these by providing an error callback in our gulp task:
gulp.task('download', function () {
return download('http://foo.com/sample.txt', {
errorCallback: function (code) {
if (code === 404) {
console.error('Un oh, something bad happened!');
doSomethingElse();
} else if (code === 500) {
console.error('Fatal exception :(');
process.exit(1);
}
},
});
});
Pass Options to Hyperquest
You can pass options to request as the second argument. For example, you can request using HTTP authentication:
gulp.task('download', function () {
const config = {
auth: {
user: 'john_doe',
pass: '123_secret',
},
};
return download(
{
url: 'http://example.com/file.txt',
file: 'foo.txt',
},
config
).pipe(gulp.dest('build'));
});
See hyperquest options for more details.
Options
Option | Type | Required | Description |
---|---|---|---|
ci |
boolean |
No | Override default detection and suppress progress bars in CI mode |
errorCallback |
(code: number) => void |
No | Customize errors during download failure |