Comparing Data Transfer Utilities

Technology professionals and end users rely on a myriad of utilities for moving data between locations. This guide discusses several of these data transfer protocols and utilities:

• rsync
• rclone
• SCP
• SFTP
• FTP/SFTP clients

Details of each utility’s core features, use cases, key considerations, strengths, and limitations are described.

What Is a Data Transfer Utility?

A data transfer utility’s primary purpose is to facilitate the movement of data between different systems/devices, networks, and IT environments. At its core, a data transfer utility must transmit data from origin to destination efficiently and securely, with minimal disruption and errors. Depending on the use case, different data transfer tools may work better than others.

Common data transfer scenarios range from simple file transfers and sharing to full-scale system backups, real-time data replication, and data migrations between different on-premise or cloud environments. Each use case often benefits from a different type of data transfer utility.

Transfer Considerations

The following are some general transfer considerations to keep in mind when selecting a data transfer utility for a particular use case or scenario.

Egress and Ingress Cost

Copying a single file from a desktop client to a cloud server calls for a different data transfer approach than migrating large volumes of data at scale between enterprise IT environments. When evaluating data transfer utilities, you should consider relative egress and ingress costs and the volume of data being transferred.

Data Storage Size and Type

Along with how much data is being transferred, the kind of data and the destination are equally important (e.g., your data’s source and target destination, their respective file systems, and how much storage is required once the data is at rest).

Most end users only store data in traditionally structured file systems when using their local environment, with files organized hierarchically into folders. These file systems are implemented on top of block storage devices. Block storage devices are an abstraction created by an operating system for the hardware of a hard drive. Block storage is also offered as a product by cloud platforms, where these devices can be attached to a compute instance, and the user can install a file system on them.

Cloud platforms also offer an alternative service called object storage. Object storage saves files in a non-hierarchical, self-contained “flat” format. This allows for increased scalability and sizable reductions in cost and complexity.

File Type and Size

Different file types and sizes also call for different data transfer utilities. For example, server log files grow in size quickly but can be easily compressed, whereas media files (e.g., MP4 video files, WAV sounds files) are usually quite large, aren’t easily compressed, and typically do not change once saved. Your choice of data transfer utility will depend on what kind of files are being transferred.

One-time Transfer vs. Recurring Syncs

A specific utility may be well-suited for your use case depending on whether the data transfer is a single event or a recurring job. For example, an ad-hoc file transfer to a cloud server may only require a desktop-based data transfer utility, while a regularly scheduled backup job may require an automated tool or process that works at scale.

Comparing Data Transfer Utilities

Rsync

Rsync is a standard Linux-based data transfer utility designed for syncing files between remote and local servers. Known for its flexibility, speed, and scriptable configuration, rsync is often a go-to utility for Linux administrators looking to quickly transfer and synchronize files between systems. rsync is used to transfer data between file systems and does not interact with object storage.

Use Cases

Rsync is ideal for syncing files between local and remote Linux machines in incremental backup and transfer use cases, as well as customizing large and complicated sync jobs with specific options and settings. rsync can also be used to sync files between two directories that are both stored locally. Its delta-transfer algorithm reduces network traffic during syncs by only sending parts of a file that differ from the files on the recipient machine.

Pros and Cons

Rsync excels in use cases that require a proven method for transferring files and situations that call for scheduling and automation capabilities, such as using rsync with Linux cron jobs. However, it also requires familiarity with the command line.

Rclone

Rclone is a command-line utility for syncing files and directories to and from different cloud storage providers, servers, and workstations. A number of rclone’s commands are similar to common Linux commands (e.g., cp, mv, mount, ls), so it can feel familiar to use. Rclone was designed for file transfers between cloud servers and on-premises servers and workstations, whereas rsync can be configured for more sophisticated file synchronization capabilities. Rclone can transfer data to and from both file systems and object storage.

Use Cases

Rclone is used for basic cloud synchronization and the copying and managing files in the cloud. For example, if you’re copying database files or directories from a cloud server to an on-premises system, rclone is an ideal data transfer utility for the job.

Pros and Cons

Rclone supports leading cloud services like Akamai, Amazon S3, Microsoft OneDrive, Google Drive/Cloud Storage, Microsoft Azure Blob/File Storage, DropBox, and more. Rclone can also recover from interrupted connections during data transfers. As with rsync, rclone can perform a sync operation, but it also supports a bidirectional sync operation.

SCP

Secure copy (SCP) is a Linux system command for securely copying files between hosts. SCP was designed to be a more secure version of remote copy (RCP), relying on the secure shell (SSH) protocol to encrypt network traffic between data transfers. SCP is used to transfer data between file systems and does not interact with object storage.

Use cases

SCP is ideal for general file transfer use cases that require copying files from a local computer to a remote server, or from remote servers to a local computer, in a secure and resilient manner. Unlike rsync, SCP is not used to copy files between directories that are both local.

Pros and Cons

Some users may find SCP to be less complicated to use than rsync. SCP is not capable of restarting transfers as rsync can, because it lacks a similar delta-transfer algorithm.

SFTP

Like SCP, secure file transfer protocol (SFTP) is commonly used for transferring files over an SSH-encrypted channel. However, SFTP can be used with a range of interactive commands (e.g, creating/deleting directories and files, settings permissions), while SCP is used for transferring files in a non-interactive manner. SFTP is used to transfer files between file systems and is generally not natively supported by object storage services, but some services may expose a compatibility layer that allows SFTP access.

Use Cases

SFTP is ideal when strong file encryption is required or mandated for transferring files over the internet. For example, compliance regulations like HIPAA, PCI-DSS, and GDPR require the use of SFTP to meet data integrity, confidentiality, and security requirements for safe data transmission. SFTP is widely regarded as the secure replacement for FTP.

Pros and Cons

SFTP works like traditional FTP, but over an encrypted, secure connection. SFTP supports both username and password and SSH key authentication, with some SFTP clients also supporting MFA and role-defined access. When using an SFTP client, it’s important to make sure it’s not configured to use outdated encryption protocols like MD5 or DES (versus AES-128 or AES-256). This could result in a false sense of regulatory compliance and security.

SFTP can be used to transfer files without requiring shell access to the user on the remote host (by using the ForceCommand SSH config parameter). In other words, a user on the remote host can be configured that has no shell access while still accepting file transfers. This can be relevant because limiting shell access is an important tool for securing a server. If a user has shell access, then if it is compromised, an attacker can run any permitted command for the user. By comparison, SFTP has a narrower range of commands that can be executed.

SFTP can be further secured by creating an SFTP jail, which limits which directory can be accessed on the remote server.

FTP/SFTP Clients

One of the earliest data transfer mechanisms, FTP is a widely used protocol for transferring files over the internet. FTP clients like Cyberduck and FileZilla are readily available for download; however, because FTP is an unencrypted protocol, its usage is now commonly discouraged for general purpose file transfers over the internet. SFTP is often recommended instead of FTP, with a variety of supporting SFTP clients (e.g., WinSCP and Tectia). Many legacy FTP clients now also support SFTP. As well, many of these clients also support transfers to and from object storage services, but they generally do not use the SFTP protocol to perform these transfers.

Use Cases

Most modern FTP/SFTP clients are capable of providing a unified GUI for accessing FTP, FTP over TLS (FTPS), and SFTP services. This allows users to copy files from local environments to remote servers with an interface that supports both legacy systems and newer machines running SFTP services. For users that require drag-and-drop functionality, FTP/SFTP clients enable them to transfer files remotely without having to use the command line.

FTP is sparingly used on legacy systems and machines that are unable to support standard encryption protocols. Aside from these edge cases, FTP is commonly disabled on modern systems, with SFTP becoming the norm for transferring files securely over the internet.

Pros and Cons

FTP/SFTP clients are considered to have a low learning curve in comparison to other data transfer utilities. Users familiar with Windows File Explorer, macOS Finder, or File Manager on Linux desktops may find them intuitive and familiar due to their graphic interfaces.

Not all FTP/SFTP clients share the same functionality, and some may lack certain features associated with other file transfer utilities. Users who require the ability to incorporate FTP/SFTP commands into scripts or automated jobs will be unable to do so with a GUI-based FTP/SFTP client, and are better suited choosing a command line-based utility.

More Information

You may wish to consult the following resources for additional information on this topic. While these are provided in the hope that they will be useful, please note that we cannot vouch for the accuracy or timeliness of externally hosted materials.

• Introduction to Rsync
• Transfer Files With the scp Command on Linux
• Transfer Files with SFTP
• What Is FTP and How Does It Work?
• Transfer Files with FileZilla