Rclone as a helper for external backups

First Test

An initial test will show you if everything worked out so far. Select a small local text file and copy it to the cloud repository with:

rclone copy mytext.txt gdrv:mytext.txt

This may take a while, because Rclone works very slowly without options. My text only reached a transmission speed of 58bps without parameters. I learned this by specifying the parameter -P (Figure 3). The "Accelerating Rclone" box explains parameters that you need to pass in to accelerate the Rclone commands.

Accelerating Rclone

You can specify the number of parallel file transfers via --transfers. For example, to back up a folder with 50 files, specify --transfers=25. As a rule, a value that is too high does not do any damage.

The --checkers parameter defines the number of test processes running in parallel. The processes identify all the files to be uploaded and compare them with what is already in the cloud store. In this way, the system excludes unchanged data from the transfer. The value is based on the --transfers parameter.

Using --drive--chunk-size, you can specify how much memory Rclone uses when uploading. The developers recommend 16MB, which you specify as --drive--chunk-size=16384k. However, experimenting with higher values will not do any harm. If you also specify the --progress (-P) parameters, you can see the exact values for the upload's duration. Rclone provides even more information with the -v or -vv options.

Figure 3: Rclone works very slowly without optimization. Google will slow you down here unless you specify additional parameters.

Step on the Gas

In order to investigate the effect of the acceleration options, I had Rclone back up a folder with 10 images with a size of 40MB in a folder to be created on the target (Listing 1). The average data rate during the backup fluctuated in the test at around 500KBps; doubling of the chunk size returned about 620KBps.

Listing 1

Testing Acceleration Options

 

Then I added three files with a size of 12MB to the locally stored image folder and restarted the backup with the same command. Since Rclone now only saved the newly added data, the process was completed in about 25 seconds. When restoring Google Drive to the local instance, no parameters are required. The command is otherwise the same as the copy command; you only have to swap source and destination.

Copy and Sync

In addition to copying, Rclone also lets you synchronize files between a client and a server. The difference between copy and sync is that the latter compares the target with the source file by file and deletes everything on the target that is not in the source directory from which you issue the command.

If you call a sync command from the wrong directory, you risk losing important data. The sync option should therefore be tested with the --dry-run parameter. copy does not delete any data locally; however, the process overwrites the backup in the cloud. If you want to keep a backup, you can bypass this process by using the backup directory mentioned above.

Another way to connect to Rclone is to mount Google Drive as a filesystem in your home directory using Filesystem in Userspace (FUSE) [6]. In my example, I created the cloud/ folder there and then mounted the previously configured Google Drive account (Listing 2).

Listing 2

Mounting with FUSE

 

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy Linux Magazine

SINGLE ISSUES
 
SUBSCRIPTIONS
 
TABLET & SMARTPHONE APPS
Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

  • Workspace – rclone

    As a simple and reliable command-line backup utility that supports popular cloud storage services, rclone makes a perfect tool for maintaining an off-site backup of your data. This article can help you to get started.

  • The sys admin's daily grind: Rclone

    Having a good backup is a matter of course for sys admin columnist Charly Kühnast, but devices could still fall victim to fire or theft some day. Because he has enough free space on Google Drive, he doesn't need to search long for a solution. The only thing missing is the right tool.

  • Google Drive File Sync with rclone
  • Website Backup

    Shared hosting is the best way for first-time webmasters to get started. But what do you do about backup?

  • FOSSPicks

    Graham nearly made it through an entire month of FOSSPicks without an esoteric audio discovery. And then he found VeeSeeVSTRack.

comments powered by Disqus

Direct Download

Read full article as PDF:

Price $2.95

News