Basics of rclone
Stairway to the Cloud

As a simple and reliable command-line backup utility that supports popular cloud storage services, rclone makes a perfect tool for maintaining an off-site backup of your data. This article can help you to get started.
Cloud storage is cheap nowadays, and you have plenty of storage providers to choose from. So, you have no excuse for not having an off-site backup system to keep your files safe. There is a fly in the ointment, however. Many cloud storage services want you to use their own proprietary graphical client applications. Worse still, some services don't provide Linux clients at all. Fortunately, there is rclone, a small open source utility that can talk to many popular cloud storage services, including Google Drive, Amazon S3, and hubiC. Additionally, rclone can handle local filesystems, so you can use it for local backup, too. The utility is straightforward in use, so there is no learning curve to speak of.
Deploying rclone
Written in Go, rclone is distributed as a self-contained binary file with no dependencies, and it will happily run on the x86, AMD64, and ARM platforms. Installing rclone is not difficult, but it does require a bit of manual work. Grab the latest release of the tool for the appropriate Linux platform from the project's website [1]. Unpack the downloaded archive and switch to the resulting directory in the terminal. Then, copy the binary executable to the /usr/local/bin/
directory and change the file's permissions:
sudo cp rclone /usr/local/bin/ sudo chown root:root /usr/local/bin/rclone sudo chmod 755 /usr/local/bin/rclone
To install man pages containing rclone's documentation, run the following commands:
sudo mkdir -p /usr/local/share/man/man1 sudo cp rclone.1 /usr/local/share/man/man1/ sudo mandb
That's all there is to it. Alternatively, you can compile rclone from source, which is also simple to do. First, make sure that the Go programming language is installed on your system. On Debian and Ubuntu, you can install Go by running the apt-get install golang
command as root. Create the ~/go
directory, and point $GOPATH
to it using the export GOPATH=$HOME/go
command. Then, run the following command to download and compile rclone:
go get github.com/ncw/rclone
Once the operation is finished, you'll find the compiled binary file in the ~/go/bin
directory. Copy the rclone file and change its permissions as described above.
Configuring rclone
To make rclone work with your preferred cloud storage service, you need to create a configuration file. The utility includes the special config
subcommand that allows you to do just that. For example, suppose you want to use rclone with the hubiC service [2]. Run the rclone config
command, press n when prompted to set a configuration password, and name the configuration profile remote
. Then, enter the number corresponding to the hubiC option in the list of supported services. Leave the Hubic Client Id and Hubic Client Secret options empty and select the auto-configuration option. This will open the default browser and obtain an access token. Check and confirm the generated settings, and you are done.
Rclone uses a default browser for authenticating with cloud storage services, which works well on any system with a graphical desktop environment. But, what if you want to configure rclone on a remote machine without a graphical desktop? It can be done, but you still need another machine with a browser. Here is how it works.
Run the rclone config
command and configure the options as described above until you reach the Use auto config? prompt. Press n and note the exact rsync authorize
command (in the case of hubiC, the command is rsync authorize "hubic"
). Run this command on the machine with a browser, copy the obtained access token, and paste it into the result > prompt on the remote machine.
If you've already configured rclone on a local machine, you can simply copy the .rclone.conf
file to the remote machine. Usually, the configuration file is stored in the home directory (i.e., ~/.rclone.conf
), but you can find the exact path by running the rclone -h
command; The actual path to the configuration file will be shown next to the --config
option.
Once rclone has been configured, run the rclone lsd remote:
command, which returns all containers. Listing 1 shows the output of this command in the case of hubiC.
Listing 1
Output of rclone lsd remote
01 7639189 0001-01-01 00:00:00 21 default 02 0 0001-01-01 00:00:00 0 default_segments 03 04 Transferred: 0 Bytes ( 0.00 kByte/s) 05 Errors: 0 06 Checks: 0 07 Transferred: 0 08 Elapsed time: 600ms
Using rclone
The rclone utility supports a few simple subcommands and options that allow you to access, manage, and use the remote storage; the copy
and sync
subcommands are probably the most important among them. As the name suggests, the copy
subcommand copies the contents of the source directory to the remote destination.
This subcommand doesn't transfer unchanged files (it checks them by size, modification time, and md5sum hashes), and it doesn't delete files from the destination directory. If the destination directory doesn't exist, rclone automatically creates it. Most rclone commands have the simple rclone [OPTION] [SUBCOMMAND] <source> <destination>
syntax, and here is what the command that copies the contents of a specified directory to the remote destination looks like:
rclone copy /path/to/source remote:destination
Similar to copy
, the sync
subcommand transfers files from the source directory to the destination, skipping unchanged files. When sync
encounters files that don't exist in the source directory, the subcommand deletes them from the destination. In other words, sync
keeps both source and destination directories in sync by modifying the destination. Because this operation (as well as some other rclone actions) is irreversible, it makes sense to test it first, for which rclone provides the handy --dry-run
option. Add this option to the rclone sync
command to check what files will be copied and deleted:
rclone --dry-run sync /path/to/source remote:destination
When using rclone, keep in mind that it copies and syncs the contents of directories and not the directories themselves. So, the rclone sync /home/user/Documents remote:Backup
command copies the contents of the Documents
directory (and not the directory and files in it) to the remote Backup
directory.
The check
subcommand can come in handy when you need to ensure that the files in the source and destination directories match:
rclone check /path/to/source remote:destination
This command compares files by their sizes and md5sum hashes and then shows a list of files that don't match.
Besides copy
and sync
, rclone supports several subcommands that let you view and manage remote storage. The lsd
subcommand, for example, can be used to list all directories (also called containers and buckets) in the remote destination, and the ls
subcommand shows all files in a specified remote directory:
rclone lsd remote: rclone ls remote:dir
If you need to create or delete a remote directory, you can use the mkdir
and rmdir
subcommands for that:
rclone mkdir remote:new_dir rclone rmdir remote:old_dir
The last command can remove a directory only if it's empty. If you want to delete a directory and its contents, use the purge
subcommand:
rclone purge remote:old_dir
As a command-line tool, rclone supports a number of options that control its behavior. The --bwlimit
option, for example, lets you limit the bandwidth available to rclone. This can be useful when the machine running rclone shares the Internet connections with other clients. Limiting the bandwidth ensures that other machines can access the Internet at a reasonable speed during the copy or sync operations. The bandwidth limit can be specified in kilobytes, megabytes, or gigabytes using the k
, M
, and G
suffixes:
rclone --bwlimit=15M sync /path/to/source remote:destination
The command above limits the bandwidth to 15MBps.
By default, rclone runs four simultaneous file transfer operations, and you can adjust this number using the --transfers
option. If you have a fast connection and remote storage service, you can increase the number of transfers:
rclone --transfers=7 copy /path/to/source remote:destination
Conversely, you might want to reduce the number of transfers if the remote service frequently times out or your Internet connection is on the slow side.
The --dry-run
option mentioned previously allows you to test rclone operations without applying any changes. Finally, if you run rclone unattended, you might want to use the --log-file
option as shown here
rclone --log-file=rclone.log sync /path/to/source remote:destination
to save rclone's output to a file for later reference.
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Direct Download
Read full article as PDF:
Price $2.95
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Find SysAdmin Jobs
News
-
Kubuntu Focus Announces XE Gen 2 Linux Laptop
Another Kubuntu-based laptop has arrived to be your next ultra-portable powerhouse with a Linux heart.
-
MNT Seeks Financial Backing for New Seven-Inch Linux Laptop
MNT Pocket Reform is a tiny laptop that is modular, upgradable, recyclable, reusable, and ships with Debian Linux.
-
Ubuntu Flatpak Remix Adds Flatpak Support Preinstalled
If you're looking for a version of Ubuntu that includes Flatpak support out of the box, there's one clear option.
-
Gnome 44 Release Candidate Now Available
The Gnome 44 release candidate has officially arrived and adds a few changes into the mix.
-
Flathub Vying to Become the Standard Linux App Store
If the Flathub team has any say in the matter, their product will become the default tool for installing Linux apps in 2023.
-
Debian 12 to Ship with KDE Plasma 5.27
The Debian development team has shifted to the latest version of KDE for their testing branch.
-
Planet Computers Launches ARM-based Linux Desktop PCs
The firm that originally released a line of mobile keyboards has taken a different direction and has developed a new line of out-of-the-box mini Linux desktop computers.
-
Ubuntu No Longer Shipping with Flatpak
In a move that probably won’t come as a shock to many, Ubuntu and all of its official spins will no longer ship with Flatpak installed.
-
openSUSE Leap 15.5 Beta Now Available
The final version of the Leap 15 series of openSUSE is available for beta testing and offers only new software versions.
-
Linux Kernel 6.2 Released with New Hardware Support
Find out what's new in the most recent release from Linus Torvalds and the Linux kernel team.