desktop-uploader module lets you easily write a desktop uploader for a remote service such as Dropbox, S3, Google Storage, or your own company using Node.js. You define directories to watch and a function that uploads a file entry, and
desktop-uploader handles the rest!
- Recursively watch folders and files for changes
- Uses native events (fsevents, inotify, ReadDirectoryChangesW)
- Whitelist file extensions you care about
- Persistent custom configuration values
- Persistent per-folder custom configuration
- Determine when a file is no longer being modified
- Upload files using custom business logic
- Concurrently upload many files
- Automatically handle retries of failures
- Keep a cache of info on already-uploaded files
- Throttle aggregate uploads to a set bandwidth (e.g. 100Kbytes/sec)
- Works with atom-shell and node-webkit for a cross-platform user interface
- Custom cache and ignore strategies (e.g. file hash instead of last modified time)
- Upload bandwidth auto-detection for throttling
- Allow manually adding items to the queue
Install like any other Node.js package, with NPM:
$ npm install --save desktop-uploader
Create a new desktop uploader instance. It takes an optional options object where you can set initial paths to watch and a few options, like the number of concurrent uploads and how many times to retry failures. Note: the uploader is created in a paused state.
var DesktopUploader = DesktopUploader;var request = ;var uploader =name: 'my-cool-app'paths: '/home/daniel/Pictures'concurrency: 3retries: 2;
Now we need to tell the uploader how to actually upload a file when that file is no longer being modified, since this is specific to your service and API. Here we are assuming that we are going to do an HTTP POST to
api.myservice.com to the
items collection using an OAuth bearer token for authentication. We'll be using the request library to make this easier.
Notice that you are piping
entry.stream into the request rather than reading it all into memory first. All that's left is to start the uploader:
At this point, the uploader is running. It is recursively watching all paths that you have configured and uploading new files.
You can dynamically add or remove paths, as well as path-specific custom configuration.
// Add a new path to watch, with a custom configuration which sets// an owner. Your `upload` method can use this configuration via// the `entry.config` attribute.uploader;// Edit an existing watched pathvar config = uploader;configowner = 'Daniel';// Remove a watched path and its configurationuploader;
You can then access the custom config during the upload process:
It's possible to automatically throttle uploads, or set throttling to a specific value. If you use the
entry.stream to pipe data to an HTTP request then all concurrent reads will be throttled to the aggregate global throttle value. For example, if three concurrent uploads are being performed, then the combined bandwidth they consume is the throttle limit.
// Throttle to 100 kbytes per seconduploaderthrottle = 100 * 1024;// Disable throttlinguploaderthrottle = false;
You can find an advanced, real-world example that uploads files to S3 in examples/example.litcoffee.
DesktopUploader class is an
EventEmitter and has the following events, properties, and methods, as well as those inherited from EventEmitter.
Emitted when the last item in the queue has finished uploading (or failed). At this point, the queue is empty and no items are being processed.
Emitted when an error occurs. The second argument, if present, is the filename which was being processed when the error occured.
Emitted when a file has been ignored (e.g. incorrect extension, no longer being watched, etc).
Log a debug message from the uploader.
Emitted when the uploader has been paused. The
type argument will be either
'watcher' depending on which was paused.
Emitted after an entry is finished uploading (including retries) and is going to be removed from the queue. Parameters are the enty and whether the upload was successful.
Emitted when an item is added to the queue. This event is fired after the item has been added or changed on disk and after a reasonable effort has been made to ensure it is no longer being written.
Emitted when the uploader has resumed uploading after being created or paused.
Emitted when a file is ready to be uploaded. This is where you implement custom logic to asyncronously upload the file. The
entry argument has the following fields:
|config||Custom configuration set on
|path||The full path to the file||
|root||The watched directory path||
|size||Approximate stream length in bytes||
|stream||Read stream to pipe into an HTTP request||
If throttling is enabled, then
stream will produce data to keep within your bandwidth limit. This event may be fired multiple times before the first upload has finished.
You must call the
done function to let the uploader know that it can process the next item in the queue.
Emitted when a folder is unwatched.
Emitted when a folder is watched.
concurrency = 2
This value determines the number of concurrent uploads. If throttling is enabled, then all uploads are throttled to the aggregate bandwidth limit. Setting a concurrency limit of
1 means only one upload at a time.
uploaderconcurrency = 5;
modifyInterval = 5000
This value determines how often in milliseconds a file is checked to see if it has been modified. If a file has not been modified between checks, then it is eligible to be uploaded and an
upload event will be fired. Defaults to 5 seconds.
retries = 0
This value determines the automatic retry count. Anytime the
done function is called with an error during the
upload event handler it is considered for a retry. The
upload event will be emitted again up to the number of retries. Set to zero to disable retry logic.
# Retry up to twouploaderretries = 2;# Disable retriesuploaderretries = 1;
A read-only array of tasks in the queue. Each task has the following fields:
|path||The full path to the file||
|root||The watched directory path||
throttle = false
This value determines the bandwidth throttling limit in bytes per second. Setting to
false, or no options will disable bandwidth throttling.
// Throttle to 10 Kbytes per seconduploaderthrottle = 10240;// Disable throttlinguploaderthrottle = false;
Create a new
DesktopUploader instance in a paused state. Takes the following optional parameters:
|concurrency||Number of concurrent uploads||
|configPath||Directory to store configuration||
|extensions||File extensions to watch||
|modifyInterval||Duration in ms to check file writes||
|name||Unique name used for configuration||
|paths||List of paths to watch||
|retries||Number of retries for failures||
|saveInterval||Duration in ms to save configuration||
|throttle||Limit bandwidth in bytes per second||
Note: extensions are not case-sensitive. You should always supply them in lowercase.
var uploader =name: 'my-cool-uploader'configPath: processenvHOMEpaths: '/some/path' '/another/path'extensions: 'jpg' 'png'throttle: 250 * 1024retries: 1;
Get the configuration for a particular watched directory by its path. You may modify the returned object. If not path is passed, then it returns an object where the keys are paths and the values are configs.
var config = uploader;configfoo = 3;var paths = uploader;console; // Prints out 3
Temporarily stop the uploader from firing
upload events. Existing in-flight items will complete, but no new items will be processed until
resume has been called. File system events will continue to add items on to the queue.
Temporarily ignore all file system events. No new or changed items will be added to the queue until
resume has been called. The
upload event will continue to be called for existing items in the queue. See the
pause method to prevent items already in the queue from being processed.
Start or resume the uploader and watcher. Since the uploader and watcher are created in a paused state, you must call this method to begin watching and uploading. Until this method is called, no items will be added to the queue and no
upload events are fired.
Give a hint that the uploader should save its configuration to disk in the near future. If
true, then save to disk right now. If
false, then at most
saveInterval milliseconds (see the constructor method) will pass before the file is saved. When your app is about to exit, you must remember to force an immediate save, otherwise data may be lost.
// Save in the near future, when convenientuploader;// Useful to call on app exituploader;
Remove a directory from being watched.
Add a new directory to recursively watch, with an optional config. The config will be saved between runs and is accessible during the
upload event via
This project uses Gulp and is written using CoffeeScript. That means that you do not edit the
.js files in the
lib folder - those are generated by the build system. Instead, you work on the
src folder. You can get started like so:
$ sudo npm install -g gulp$ git clone https://github.com/danielgtaylor/node-desktop-uploader$ cd node-desktop-uploader$ npm install
You can edit and then compile the source via:
$ gulp compile...
You can test in a Node shell via:
> var DesktopUploader = DesktopUploader;> uploader = ;> ...
You can run the unit tests via:
$ gulp test...
Pull requests are welcome, so please fork the project and submit one! Please keep in mind that any new features should include unit tests coverage, or they may be rejected.