Basics of rclone
Stairway to the Cloud
As a simple and reliable command-line backup utility that supports popular cloud storage services, rclone makes a perfect tool for maintaining an off-site backup of your data. This article can help you to get started.
Cloud storage is cheap nowadays, and you have plenty of storage providers to choose from. So, you have no excuse for not having an off-site backup system to keep your files safe. There is a fly in the ointment, however. Many cloud storage services want you to use their own proprietary graphical client applications. Worse still, some services don't provide Linux clients at all. Fortunately, there is rclone, a small open source utility that can talk to many popular cloud storage services, including Google Drive, Amazon S3, and hubiC. Additionally, rclone can handle local filesystems, so you can use it for local backup, too. The utility is straightforward in use, so there is no learning curve to speak of.
Deploying rclone
Written in Go, rclone is distributed as a self-contained binary file with no dependencies, and it will happily run on the x86, AMD64, and ARM platforms. Installing rclone is not difficult, but it does require a bit of manual work. Grab the latest release of the tool for the appropriate Linux platform from the project's website [1]. Unpack the downloaded archive and switch to the resulting directory in the terminal. Then, copy the binary executable to the /usr/local/bin/
directory and change the file's permissions:
sudo cp rclone /usr/local/bin/ sudo chown root:root /usr/local/bin/rclone sudo chmod 755 /usr/local/bin/rclone
To install man pages containing rclone's documentation, run the following commands:
sudo mkdir -p /usr/local/share/man/man1 sudo cp rclone.1 /usr/local/share/man/man1/ sudo mandb
That's all there is to it. Alternatively, you can compile rclone from source, which is also simple to do. First, make sure that the Go programming language is installed on your system. On Debian and Ubuntu, you can install Go by running the apt-get install golang
command as root. Create the ~/go
directory, and point $GOPATH
to it using the export GOPATH=$HOME/go
command. Then, run the following command to download and compile rclone:
go get github.com/ncw/rclone
Once the operation is finished, you'll find the compiled binary file in the ~/go/bin
directory. Copy the rclone file and change its permissions as described above.
Configuring rclone
To make rclone work with your preferred cloud storage service, you need to create a configuration file. The utility includes the special config
subcommand that allows you to do just that. For example, suppose you want to use rclone with the hubiC service [2]. Run the rclone config
command, press n when prompted to set a configuration password, and name the configuration profile remote
. Then, enter the number corresponding to the hubiC option in the list of supported services. Leave the Hubic Client Id and Hubic Client Secret options empty and select the auto-configuration option. This will open the default browser and obtain an access token. Check and confirm the generated settings, and you are done.
Rclone uses a default browser for authenticating with cloud storage services, which works well on any system with a graphical desktop environment. But, what if you want to configure rclone on a remote machine without a graphical desktop? It can be done, but you still need another machine with a browser. Here is how it works.
Run the rclone config
command and configure the options as described above until you reach the Use auto config? prompt. Press n and note the exact rsync authorize
command (in the case of hubiC, the command is rsync authorize "hubic"
). Run this command on the machine with a browser, copy the obtained access token, and paste it into the result > prompt on the remote machine.
If you've already configured rclone on a local machine, you can simply copy the .rclone.conf
file to the remote machine. Usually, the configuration file is stored in the home directory (i.e., ~/.rclone.conf
), but you can find the exact path by running the rclone -h
command; The actual path to the configuration file will be shown next to the --config
option.
Once rclone has been configured, run the rclone lsd remote:
command, which returns all containers. Listing 1 shows the output of this command in the case of hubiC.
Listing 1
Output of rclone lsd remote
01 7639189 0001-01-01 00:00:00 21 default 02 0 0001-01-01 00:00:00 0 default_segments 03 04 Transferred: 0 Bytes ( 0.00 kByte/s) 05 Errors: 0 06 Checks: 0 07 Transferred: 0 08 Elapsed time: 600ms
Using rclone
The rclone utility supports a few simple subcommands and options that allow you to access, manage, and use the remote storage; the copy
and sync
subcommands are probably the most important among them. As the name suggests, the copy
subcommand copies the contents of the source directory to the remote destination.
This subcommand doesn't transfer unchanged files (it checks them by size, modification time, and md5sum hashes), and it doesn't delete files from the destination directory. If the destination directory doesn't exist, rclone automatically creates it. Most rclone commands have the simple rclone [OPTION] [SUBCOMMAND] <source> <destination>
syntax, and here is what the command that copies the contents of a specified directory to the remote destination looks like:
rclone copy /path/to/source remote:destination
Similar to copy
, the sync
subcommand transfers files from the source directory to the destination, skipping unchanged files. When sync
encounters files that don't exist in the source directory, the subcommand deletes them from the destination. In other words, sync
keeps both source and destination directories in sync by modifying the destination. Because this operation (as well as some other rclone actions) is irreversible, it makes sense to test it first, for which rclone provides the handy --dry-run
option. Add this option to the rclone sync
command to check what files will be copied and deleted:
rclone --dry-run sync /path/to/source remote:destination
When using rclone, keep in mind that it copies and syncs the contents of directories and not the directories themselves. So, the rclone sync /home/user/Documents remote:Backup
command copies the contents of the Documents
directory (and not the directory and files in it) to the remote Backup
directory.
The check
subcommand can come in handy when you need to ensure that the files in the source and destination directories match:
rclone check /path/to/source remote:destination
This command compares files by their sizes and md5sum hashes and then shows a list of files that don't match.
Besides copy
and sync
, rclone supports several subcommands that let you view and manage remote storage. The lsd
subcommand, for example, can be used to list all directories (also called containers and buckets) in the remote destination, and the ls
subcommand shows all files in a specified remote directory:
rclone lsd remote: rclone ls remote:dir
If you need to create or delete a remote directory, you can use the mkdir
and rmdir
subcommands for that:
rclone mkdir remote:new_dir rclone rmdir remote:old_dir
The last command can remove a directory only if it's empty. If you want to delete a directory and its contents, use the purge
subcommand:
rclone purge remote:old_dir
As a command-line tool, rclone supports a number of options that control its behavior. The --bwlimit
option, for example, lets you limit the bandwidth available to rclone. This can be useful when the machine running rclone shares the Internet connections with other clients. Limiting the bandwidth ensures that other machines can access the Internet at a reasonable speed during the copy or sync operations. The bandwidth limit can be specified in kilobytes, megabytes, or gigabytes using the k
, M
, and G
suffixes:
rclone --bwlimit=15M sync /path/to/source remote:destination
The command above limits the bandwidth to 15MBps.
By default, rclone runs four simultaneous file transfer operations, and you can adjust this number using the --transfers
option. If you have a fast connection and remote storage service, you can increase the number of transfers:
rclone --transfers=7 copy /path/to/source remote:destination
Conversely, you might want to reduce the number of transfers if the remote service frequently times out or your Internet connection is on the slow side.
The --dry-run
option mentioned previously allows you to test rclone operations without applying any changes. Finally, if you run rclone unattended, you might want to use the --log-file
option as shown here
rclone --log-file=rclone.log sync /path/to/source remote:destination
to save rclone's output to a file for later reference.
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
Latest Cinnamon Desktop Releases with a Bold New Look
Just in time for the holidays, the developer of the Cinnamon desktop has shipped a new release to help spice up your eggnog with new features and a new look.
-
Armbian 24.11 Released with Expanded Hardware Support
If you've been waiting for Armbian to support OrangePi 5 Max and Radxa ROCK 5B+, the wait is over.
-
SUSE Renames Several Products for Better Name Recognition
SUSE has been a very powerful player in the European market, but it knows it must branch out to gain serious traction. Will a name change do the trick?
-
ESET Discovers New Linux Malware
WolfsBane is an all-in-one malware that has hit the Linux operating system and includes a dropper, a launcher, and a backdoor.
-
New Linux Kernel Patch Allows Forcing a CPU Mitigation
Even when CPU mitigations can consume precious CPU cycles, it might not be a bad idea to allow users to enable them, even if your machine isn't vulnerable.
-
Red Hat Enterprise Linux 9.5 Released
Notify your friends, loved ones, and colleagues that the latest version of RHEL is available with plenty of enhancements.
-
Linux Sees Massive Performance Increase from a Single Line of Code
With one line of code, Intel was able to increase the performance of the Linux kernel by 4,000 percent.
-
Fedora KDE Approved as an Official Spin
If you prefer the Plasma desktop environment and the Fedora distribution, you're in luck because there's now an official spin that is listed on the same level as the Fedora Workstation edition.
-
New Steam Client Ups the Ante for Linux
The latest release from Steam has some pretty cool tricks up its sleeve.
-
Gnome OS Transitioning Toward a General-Purpose Distro
If you're looking for the perfectly vanilla take on the Gnome desktop, Gnome OS might be for you.