Naming Prosecution Mitigator

    remote-sync

    1.0.3 • Public • Published

    Remote Sync

    A cross-platform LFTP wrapper for automating file synchronization.

    NPM

    Documentation | Repository

    Features

    • Leverage the power of LFTP to transfer files to/from remote host(s).
    • Logically organize LFTP operations into a workflow that is easily revisable.
    • For speed, open a persistent connection to a designated host and execute your commands in series.
    • Define separate settings (host|user|password) for independent operations that can be executed (a)synchronously.
    • Create a conditional chain of operations where you create the condition for further execution.
    • Use RemoteSync as a general purpose LFTP client.

    Dependency

    You'll need to have LFTP installed on your machine as its the child process RemoteSync wraps. I chose LFTP as my FTP(S) client because its easy to work with, comes preinstalled on most Unix-based OS distributions and has a nice port to Windows. If you don't already have LFTP installed (Windows) use a package management tool to get the latest version:

    Windows (Chocolatey)

    C:\> choco install lftp

    OSX (Homebrew)

    sudo brew install lftp

    Linux

    sudo apt-get install lftp
    sudo yum install lftp

    Configuration

    Constructor Object

    {
      operations : [{},{},..,{}],   // Array of operation objects. See below. 
      lftp_settings : {             // OPTIONAL: LFTP settings. See LFTP man page.
        'setting':'value'
      },
      user : 'user',                // User name for host. Default: ''
      pw : 'password',              // Password for host. Default: ''
      protocol : 'ftp',             // Protocol (Everything LFTP supports) Default: ftp
      host : 'some.host.com',       // Host name. Default: ''
      port : '21',                  // Port number. Default: ''
      persistent : false,           // Persistent connection. Default: false
      sync : true,                  // Synchronously execute operations. Default: true
      exit : false,                 // If persistent, close connection after operations finish. Default: false
      debug : true,                 // Pass debug flag to LFTP for verbose logging. Default: false
      stdio : stdio                 // OPTIONAL: Override stdio configuration of child process. This is overridden
                                    // by operation object's stdio property.
    }

    Operation Object

    {
        operation : 'download',     // Operation name. Something that relates to the command.
        command : 'lftp command',   // The LFTP command to run. Refer to man page.
        user : 'user',              // OPTIONAL: User name. If !persistent, overrides constructor value.
        pw : 'password',            // OPTIONAL: Password. If !persistent, overrides constructor value.
        protocol : 'ftp',           // OPTIONAL: Protocol. If !persistent, overrides constructor value.
        host : 'some.host.com',     // OPTIONAL: Host name. If !persistent, overrides constructor value.
        port : '21',                // OPTIONAL: Port number. If !persistent, overrides constructor value.
        settings : {                // OPTIONAL: Do stuff based on configuration.
            sync : fn(child),       // OPTIONAL: If !persistent and sync = true, call fn(child) on finish.
            stdio : stdio           // OPTIONAL: Override stdio configuration of child process. This overrides
                                    // constructor object's stdio property.
        }
    }

    stdio Object

    // https://nodejs.org/api/child_process.html#child_process_options_stdio
    {
        stdio_config : {stdio:[]},  // See above link for custom configuration.
        stdout : data => fn(data),  // Custom behavior for child's stdout event.
        stderr : data => fn(data),  // Custom behavior for child's stderr event.
        close : code => fn(code),   // Custom behavior for child's close event.
        error : error => fn(error)  // Custom behavior for child's error event.
    }

    Examples

    Fancy level 0: The following example will use a minimal configuration to demonstrate basic functionality. Open a non-persistent connection to ftp.host.com, execute operation 1 and exit.

    // client.js
    const RemoteSync = require('remote-sync');
    const config = {
        operations : [
            {
                operation : 'List',
                command : 'nlist files'
            }
        ],
        user : 'kurt',
        pw : 'foobar',
        host : 'ftp.host.com'
    };
    const client = new RemoteSync(config);
    client.perform(); // Returns a remote listing of files at ftp.host.com/files/

    Fancy level 1: Add lftp_settings to the constructor object to customize the session. Open a non-persistent connection to ftp.host.com using FTPES (Explicit FTP over TLS), set parallel transfer count to 5, execute operation 1 and exit. (Read about the flags passed to mirror at LFTP.)

    // client.js
    const RemoteSync = require('remote-sync');
    const command = 'mirror -c --only-missing <source> <dest>';
    const config = {
        operations : [
            {
                operation : 'mirror directory',
                command : command
            }
        ],
        user : 'kurt',
        pw : 'foobar',
        host : 'ftp.host.com',
        lftp_settings : {
            'ftp:ssl-force':'true',
            'ftp:ssl-protect-data':'true',
            'ssl:verify-certificate':'false',
            'net:max-retries':'2',
            'net:timeout':'10',
            'net:connection-limit':'5',
            'net:reconnect-interval-base':'5',
            'net:reconnect-interval-multiplier':'1',
            'mirror:parallel-transfer-count':'5'
        }
    };
    const client = new RemoteSync(config);
    client.perform(); // Mirror only missing files from remote source to local disk.

    Fancy level 2: Create a conditional chain of operations where you create the condition for further execution. Open a non-persistent connection to ftp.host.com and execute operation 1. If operation 1's status is not 0 (success) exit the parent process, halting any further execution. If operation 1 is successful, operations 2 & 3 will be executed.

    // client.js
    const RemoteSync = require('remote-sync');
    const mirror = 'mirror -c --only-missing <source> <dest>';
    const remove = 'rm -r <source>';
    const list = 'nlist files';
    const config = {
        operations : [
            {
                operation : 'mirror directory',
                command : mirror,
                settings : {
                    sync : child => {
                        if (child.status != 0) {
                            process.exit(1);
                        }
                    }
                }
            },
            {  
                operation : 'delete directory',
                command : remove
            },
            {
                operation : 'list directory',
                command : list,
                user : 'username',
                pw : 'password',
                host : 'other.host.com'
            }
        ],
        user : 'kurt',
        pw : 'foobar',
        host : 'ftp.host.com',
        lftp_settings : settings_obj, // omitted for brevity
        sync : true
    };
    const client = new RemoteSync(config);
    client.perform();   // Mirror only missing files from remote source to local disk.
                        // If successful, delete the remote source and get remote listing
                        // from other.host.com/files/

    Fancy level 3: Open a persistent connection (persistent: true) to the designated host and perform operations in series. In this configuration a connection is established to ftp.host.com, each operation's command is executed in series followed by severing the connection (exit: true).

    // client.js
    const RemoteSync = require('remote-sync');
    const mirror = 'mirror -c --only-missing <source> <dest>';
    const upload = 'mirror -R -c --only-newer --overwrite --exclude .git/ <local> <remote>'; // reverse mirror -R
    const config = {
        operations : [
            {
                operation : 'mirror directory',
                command : mirror
            },
            {  
                operation : 'upload directory',
                command : upload
            }
        ],
        user : 'kurt',
        pw : 'foobar',
        host : 'ftp.host.com',
        lftp_settings : settings_obj, // omitted for brevity
        persistent : true,
        exit : true
    };
    const client = new RemoteSync(config);
    client.perform();   // Connection stays open until `mirror` and `upload` complete

    General Purpose LFTP Client

    'use strict';
    const RemoteSync = require('remote-sync');
    const rs = new RemoteSync({
      user : 'kurt',
      pw : 'foobar',
      host : 'ftp.host.com'
    });
    // Define an iterator and execute commands in series
    rs[Symbol.iterator] = function* () {
        yield rs.commands('nlist files').execute();
        yield rs.commands('nlist files/completed').execute();
        yield rs.commands('nlist files').execute();
    };
    [...rs];

    Common Commands

    Mirror directory from source to dest

    const mirror = 'mirror -c --only-missing <source> <dest>';

    Upload directory from local to remote

    const upload = 'mirror -R -c --only-newer --overwrite --exclude .git/ <local> <remote>';

    Delete remote source

    const remove = 'rm -r <source>';

    Get remote listing source

    const list = 'nlist <source>';

    Cronjob

    # Run client.js every 15 minutes 
    0/15 * * * * path/to/node path/to/client.js

    Install

    npm i remote-sync

    DownloadsWeekly Downloads

    13

    Version

    1.0.3

    License

    MIT

    Unpacked Size

    21.7 kB

    Total Files

    4

    Last publish

    Collaborators

    • heavymedl