Remote Sync
A cross-platform LFTP wrapper for automating file synchronization.
Features
- Leverage the power of LFTP to transfer files to/from remote host(s).
- Logically organize LFTP operations into a workflow that is easily revisable.
- For speed, open a persistent connection to a designated host and execute your commands in series.
- Define separate settings (host|user|password) for independent operations that can be executed (a)synchronously.
- Create a conditional chain of operations where you create the condition for further execution.
- Use RemoteSync as a general purpose LFTP client.
Dependency
You'll need to have LFTP installed on your machine as its the child process RemoteSync wraps. I chose LFTP as my FTP(S) client because its easy to work with, comes preinstalled on most Unix-based OS distributions and has a nice port to Windows. If you don't already have LFTP installed (Windows) use a package management tool to get the latest version:
Windows (Chocolatey)
C:\> choco install lftp
OSX (Homebrew)
sudo brew install lftp
Linux
sudo apt-get install lftpsudo yum install lftp
Configuration
Constructor Object
operations : {}{}{} // Array of operation objects. See below. lftp_settings : // OPTIONAL: LFTP settings. See LFTP man page. 'setting':'value' user : 'user' // User name for host. Default: '' pw : 'password' // Password for host. Default: '' protocol : 'ftp' // Protocol (Everything LFTP supports) Default: ftp host : 'some.host.com' // Host name. Default: '' port : '21' // Port number. Default: '' persistent : false // Persistent connection. Default: false sync : true // Synchronously execute operations. Default: true exit : false // If persistent, close connection after operations finish. Default: false debug : true // Pass debug flag to LFTP for verbose logging. Default: false stdio : stdio // OPTIONAL: Override stdio configuration of child process. This is overridden // by operation object's stdio property.
Operation Object
operation : 'download' // Operation name. Something that relates to the command. command : 'lftp command' // The LFTP command to run. Refer to man page. user : 'user' // OPTIONAL: User name. If !persistent, overrides constructor value. pw : 'password' // OPTIONAL: Password. If !persistent, overrides constructor value. protocol : 'ftp' // OPTIONAL: Protocol. If !persistent, overrides constructor value. host : 'some.host.com' // OPTIONAL: Host name. If !persistent, overrides constructor value. port : '21' // OPTIONAL: Port number. If !persistent, overrides constructor value. settings : // OPTIONAL: Do stuff based on configuration. sync : // OPTIONAL: If !persistent and sync = true, call fn(child) on finish. stdio : stdio // OPTIONAL: Override stdio configuration of child process. This overrides // constructor object's stdio property.
stdio Object
// https://nodejs.org/api/child_process.html#child_process_options_stdio stdio_config : stdio: // See above link for custom configuration. // Custom behavior for child's stdout event. // Custom behavior for child's stderr event. // Custom behavior for child's close event. // Custom behavior for child's error event.
Examples
Fancy level 0: The following example will use a minimal configuration to demonstrate basic functionality. Open a non-persistent connection to ftp.host.com
, execute operation 1 and exit.
// client.jsconst RemoteSync = ;const config = operations : operation : 'List' command : 'nlist files' user : 'kurt' pw : 'foobar' host : 'ftp.host.com';const client = config;client; // Returns a remote listing of files at ftp.host.com/files/
Fancy level 1: Add lftp_settings
to the constructor object to customize the session. Open a non-persistent connection to ftp.host.com
using FTPES (Explicit FTP over TLS), set parallel transfer count to 5, execute operation 1 and exit. (Read about the flags passed to mirror
at LFTP.)
// client.jsconst RemoteSync = ;const command = 'mirror -c --only-missing <source> <dest>';const config = operations : operation : 'mirror directory' command : command user : 'kurt' pw : 'foobar' host : 'ftp.host.com' lftp_settings : 'ftp:ssl-force':'true' 'ftp:ssl-protect-data':'true' 'ssl:verify-certificate':'false' 'net:max-retries':'2' 'net:timeout':'10' 'net:connection-limit':'5' 'net:reconnect-interval-base':'5' 'net:reconnect-interval-multiplier':'1' 'mirror:parallel-transfer-count':'5' ;const client = config;client; // Mirror only missing files from remote source to local disk.
Fancy level 2: Create a conditional chain of operations where you create the condition for further execution. Open a non-persistent connection to ftp.host.com
and execute operation 1. If operation 1's status is not 0 (success) exit the parent process, halting any further execution. If operation 1 is successful, operations 2 & 3 will be executed.
// client.jsconst RemoteSync = ;const mirror = 'mirror -c --only-missing <source> <dest>';const remove = 'rm -r <source>';const list = 'nlist files';const config = operations : operation : 'mirror directory' command : mirror settings : { if childstatus != 0 process; } operation : 'delete directory' command : remove operation : 'list directory' command : list user : 'username' pw : 'password' host : 'other.host.com' user : 'kurt' pw : 'foobar' host : 'ftp.host.com' lftp_settings : settings_obj // omitted for brevity sync : true;const client = config;client; // Mirror only missing files from remote source to local disk. // If successful, delete the remote source and get remote listing // from other.host.com/files/
Fancy level 3: Open a persistent connection (persistent: true
) to the designated host and perform operations in series. In this configuration a connection is established to ftp.host.com
, each operation's command
is executed in series followed by severing the connection (exit: true
).
// client.jsconst RemoteSync = ;const mirror = 'mirror -c --only-missing <source> <dest>';const upload = 'mirror -R -c --only-newer --overwrite --exclude .git/ <local> <remote>'; // reverse mirror -Rconst config = operations : operation : 'mirror directory' command : mirror operation : 'upload directory' command : upload user : 'kurt' pw : 'foobar' host : 'ftp.host.com' lftp_settings : settings_obj // omitted for brevity persistent : true exit : true;const client = config;client; // Connection stays open until `mirror` and `upload` complete
General Purpose LFTP Client
'use strict';const RemoteSync = ;const rs = user : 'kurt' pw : 'foobar' host : 'ftp.host.com';// Define an iterator and execute commands in seriesrsSymboliterator = { rs; rs; rs;};...rs;
Common Commands
Mirror directory from source
to dest
const mirror = 'mirror -c --only-missing <source> <dest>';
Upload directory from local
to remote
const upload = 'mirror -R -c --only-newer --overwrite --exclude .git/ <local> <remote>';
Delete remote source
const remove = 'rm -r <source>';
Get remote listing source
const list = 'nlist <source>';
Cronjob
# Run client.js every 15 minutes 0/15 * * * * path/to/node path/to/client.js