When you need to move a large set of files from one remote location to another, especially between different cloud services, copying all the files from the source to your local machine and then uploading them to the new destination can be a nightmare. Imagine handling a terabyte of files. Do you have the storage space for that on your machine?
Luckily, there’s a better way to handle this: rclone. Rclone is a CLI tool that allows you to perform batch file operations directly between remote locations, syncing them efficiently from source to destination.
In this post, we’ll guide you through using rclone to copy files from a cloud storage drive to Linode Object Storage, making the process as seamless as possible.
What is rclone?
Rclone is a command-line program that manages files on cloud storage. It’s like a Swiss army knife for cloud storage, handling tasks such as syncing, copying, and moving files.
How does it work?
Rclone sets up “remotes” for each cloud service you want to interact with. Once configured, you can use rclone commands to transfer data between them. Although files do pass through your local machine, they aren’t stored there long term.
What can you do with it?
With rclone, you can:
- Sync files between different cloud services.
- Copy and move files efficiently.
- Mount cloud storage as a file system on your local machine.
- Encrypt files for secure storage.
Rclone supports many cloud services, including Linode Object Storage, Google Drive, AWS S3, Nextcloud, and more. It’s a versatile tool for managing your cloud data.
Now that we know what rclone can do, let’s demonstrate how you can use it to sync data between Google Drive and Linode Object Storage.
Step 1: Create a Linode (optional but recommended)
Using rclone involves transferring files from the remote source to your local machine before moving them to the remote destination. While these files aren’t held locally, the data still travels through your internet connection.
To optimize the process and reduce the load on your home network, we recommend creating a Linode VM. By doing this, you shorten the data transfer path.
To optimize your setup, choose a Linode in the same region as your Linode Object Storage bucket.
Step 2: Install rclone
After setting up your Linode VM, the next step is to install rclone. First, use SSH to connect to your Linode instance. From the Linode dashboard, you can connect via the LISH Console.
Run the following command to download and install rclone:
~# curl https://rclone.org/install.sh | sudo bash |
Check that rclone is installed correctly by running the following:
~# rclone version |
After confirming that the installation was successful, you’re ready to configure rclone with your cloud storage remotes.
Step 3: Add a new remote (Google Drive) to rclone
To start transferring files, you need to set up rclone to access your Google Drive. To do this, you’ll need to obtain credentials. Follow these instructions for creating a service account and obtaining a public/private key for your service account. Save this file on your local machine as service-account.json. Then, use SFTP to transfer this file to your Linode VM (since rclone will need to reference it).
With the service account key file on your Linode VM, use the following command to configure the Google Drive remote in rclone:
~# rclone config |
Follow the prompts to set up a new remote:
- Choose “n” for a
new remote. Namethe remote (for example: “gdrive”).- For
storage type, select “Google Drive” from the list of options. - Leave client_id and client_secret blank.
- Choose the appropriate
scope. For our purposes, we will choose “Read-only access to file metadata and contents”. - For the service_account file, enter the path to the JSON file with the service account credentials that you downloaded above.
- Choose the defaults for any remaining questions.
For more detailed steps, you can also refer to the rclone documentation for Google Drive.
Once completed, rclone will have access to your Google Drive, ready for file operations.
Step 4: Add a new remote (Linode) to rclone
The next step is to set up a second remote in rclone, pointing to Linode Object Storage. Just as in the previous step, you’ll need to get your cloud storage credentials.
Linode Object Storage credentials are similar to AWS S3 credentials. You can get your Access Key and Secret Key from the Linode Cloud Manager. On the Object Storage page, click Create Access Key.
After creating your Access Key and Secret Key, copy them down.
Use the following command to configure the Linode remote in rclone:
~# rclone version |
Follow the prompts to set up a new remote:
- Choose “n” for a
new remote. Namethe remote (for example: “linode”).- For
storage type, select “Amazon S3 Compliant Storage Providers” from the list of options. - For
provider, select “Linode Object Storage” from the list of options. - Select “Enter AWS credentials in the next step”.
- For access_key_id, paste the Access Key you copied above.
- For secret_access_key, paste the Secret Key you copied above.
- Select the region for your Linode Object Storage bucket.
- Choose the defaults for any remaining questions.
For detailed steps, refer to the rclone documentation for Linode Object Storage.
Once completed, rclone will have access to your Linode Object Storage, ready for syncing files from Google Drive.
Step 5: Sync
With both Google Drive and Linode Object Storage configured as remotes in rclone, you can now sync files between them.
For our demo, we want to sync the contents of data in our Google Drive to a folder called data in our Linode Object Storage bucket.
Let’s start by taking a look at what’s in our Google Drive folder:
~# rclone size gdrive:/data |
Here is what we see in our Linode Object Storage bucket:
~# rclone size linode:/data |
As expected, the data folder doesn’t exist (yet).
To sync our files, we could use the following basic command:
~# rclone sync gdrive:/data linode:/data |
This command will copy files from your Google Drive (source) to your Linode Object Storage (destination).
If we wanted to monitor the transfer process, we would add the –progress flag:
~# rclone sync \ gdrive:/data \ linode:/data \ --progress |
We can further optimize the sync process by configuring the number of concurrent transfers with the –transfers flag:
~# rclone sync \ gdrive:/data \ linode:/data \ --progress –transfers=8 |
As our sync begins, we see rclone constantly update the transfer statuses:
Transferred: 6.152 KiB / 145.195 KiB, 4%, 421 B/s, ETA 5m37s |
After the initial sync, we can rerun the command to ensure all files are up-to-date. Rclone will only transfer new or changed files, making subsequent syncs faster.
~# rclone sync gdrive:/data linode:/data --progress |
This step ensures your files are efficiently transferred and kept in sync between Google Drive and Linode Object Storage.
Conclusion
In this post, we’ve walked you through using rclone to efficiently transfer files from Google Drive to Linode Object Storage. Although we specifically used Google Drive and Linode Object Storage for our demo, rclone supports dozens of cloud storage providers.
To recap:
- Rclone allows you to perform batch file operations directly between cloud services.
- Setting up a Linode VM in the same region as your Linode Object Storage can optimize your transfers.
- Configuring rclone with Google Drive and Linode Object Storage remotes is straightforward and powerful.
- Using rclone’s sync feature helps keep your files up-to-date without manual intervention.
Ready to streamline your file transfers? Try Linode Object Storage today and see how easy managing your cloud data can be!
Comments