Secure Cloud

The topic of online backups took on a whole new aspect after the Snowden revelations of recent weeks and months. Even before the NSA scandal, however, people were wondering how safe their data would be with a cloud provider (see the "Persistent Criticism" box). Although manual encryption of the data up front can help mitigate concern, the process also involves a huge amount of effort and makes the entire workflow complex.

A remedy has appeared in the form of a small open source tool named Duplicati [1], which kills two birds with one stone: It includes back ends for many important cloud services, so you can forget about using the native clients, and it encrypts all your data before transferring.

In fact, the program offers a third distinct advantage over manually backing up data via the provider's native cloud client: As true backup software, Duplicati not only flexibly helps you select the data to be backed up, it also backs up data on the fly that you can't easily access as a user. Additionally, clients are available for Linux, Windows, and Mac OS X, so the program can be deployed seamlessly on multiple platforms.

Duplicati for Linux

The open source program Duplicati was originally penned by Danish developer Kenneth Skovhede, who is still is one of the main developers. Since 2011, Germany's René Stach has also coordinated public relations and managed communications with external developers.

The core of the software is under the GPL. Other licenses apply for some of the SDKs, tools, and libraries, such as AWSSDK, GPG, SQLite, or PuTTY. Version 1.3.4, which was published in February 2013, can be downloaded from the project website [2] with a choice of DEB and RPM packages for Linux distros, binaries for Windows and Mac OS X, and packed binary packages for Linux and Windows.

The program quickly demonstrates its roots in Windows: The interface in the Linux and Mac versions is based on the free .NET implementation Mono and the WinForms libraries. The tool for the command line, however, is similar for all three versions. In the upcoming version 2.0, the developers are planning some radical streamlining (see the "Duplicati 2.0" box).

The advantage of Duplicati is that it includes an impressive range of back ends for the major cloud providers, including 1&1 SmartDrive, Amazon S3, Google Drive, ownCloud, Windows SkyDrive, Strato HiDrive, or T-Online Media Center. For more information on the supported cloud providers, see the Beginner's Guide [3]. Before transmission, the software uses AES-256 or GPG to encrypt all the data, and it supports incremental backups.

Installing the packages for the distributions proves to be the easiest approach, because you do not have to worry about resolving dependencies yourself. Alternatively, you can simply download the tarball and unpack it in any directory. Before doing so, install the mono-runtime, libmono2.0-cil, and libmono-winforms2.0-cil packages. For more information about installation, see Google Code [4].

Open Files

Whereas Windows uses Microsoft's Volume Shadow Copy service to back up open or locked files, Linux supports snapshots via the Logical Volume Manager. Using the snapshot method on Linux means having tools for LVM in place and additional administrative privileges. A copy-on-write procedure is used.

A redirect-on-write permanently reroutes all changes to the snapshot, but copy-on-write keeps back the changes in the metadata until the original data has been fully transferred to the snapshot. When reading a snapshot, the operating system can first determine whether the part to be read already exists and then use it. Thus, administrative privileges are essential for scenarios involving snapshots.

You can control the way snapshots are handled with the --snapshot-policy and --open-file-policy options; the former is disabled by default, because the function needs the above-mentioned, far-reaching rights and preconditions. Additionally, the on, auto, and required parameters are available for this option. The on parameter tries to use an existing snapshot. In the absence of such, the program outputs an extensive warning for any files that it cannot back up. The auto option does the same thing but without a warning.

If a snapshot exists, Duplicati ignores the --open-file-policy, if set. It is only used if no snapshot exists and the snapshot policy is not set to required. The --open-file-policy lets you specify how the program handles open files: ignore basically excludes locked files from the backup. The default setting is snapshot, which gets a copy of the file as-is and a warning in the log if the file changes during the backup. copy tells Duplicati to try and make a copy of the open file before backing up if the lock mechanisms allow it. Because this action only works with a full copy, problems occur with very large files. If a snapshot exists, Duplicati ignores --open-file-policy and uses the snapshot.

The Linux version of Duplicati also backs up various metadata typical of Unix-style operating systems. You can configure three different types of behavior for handling symlinks with the store, follow, or ignore parameters.

For example, if you choose store, the tool only saves the symlink. Using follow, on the other hand, adds the linked files to the backup, whereas ignore does exactly that to the symbolic link. Duplicati version 1.34 saves only the timestamp of metadata such as pipes or FIFOs.

Use

To invoke the wizard, type Duplicati at the command line, or, after installing via the package sources, you can start the program via the KDE menu or the Gnome/Unity dash, depending on your distribution. If you want to benefit from the advanced options for backing up open or locked files via LVM snapshots, you will need to launch Duplicati with administrative privileges.

The first time you use the program, the wizard asks whether you want to Setup a new backup, Restore files from a backup, or Restore settings from a previous Duplicati installation (Figure 1). You then enter a name for the backup and optionally assign the backup to a group that you can create, if needed, by pressing the Add a Group button.

Figure 1: Whether you want to set up a new backup or restore data from an existing backup – the Duplicati wizard is there to help.

When you select the data to be backed up, the wizard suggests a couple of typical locations on the filesystem: My Documents, Files on the desktop, and My Pictures. You can customize this choice by selecting Custom folder list.

If you choose to Include application settings, you'll find only the $HOME/.config/ directory. The Custom folder list contains a button labeled … that lets you add more directories. After adding a directory, another line appears in the wizard, which you can use to select other directories in the same way. You can use the trash can icon to the right to delete them again.

This approach, however, does not give you access to hidden files. Moreover, you cannot exclude subdirectories or files in this step. That is only possible later after enabling Advanced settings.

The next step is all about encryption: If you uncheck the box for Protect the backups with this password, which is checked by default, Duplicati does not encrypt the backed up data. Otherwise, under Encryption method, you can choose between the integrated AES-256 encryption, built-in and GNU Privacy Guard, external (Figure 2).

Figure 2: Trust is good; encryption is better. You can protect the backed up data with the internal encryption mechanisms or with your own GPG key.

After confirming the password, you can move on to select the desired back end in the next step. Besides the services mentioned previously, you can choose some pretty straightforward functions here, such as FTP, SSH, or file-based. The parameters needed in each case depend on the method you choose.

For the Google back end, you just need the account information and the name of a new or existing Google Docs Collection; however, you can Create Collection to set up a new one if needed. The fact that the service is now called Google Drive has gone unnoticed by this interface, but this does not detract from the function.

The back end for Amazon S3, on the other hand, operates on the basis of buckets. Buckets simplify the process of distributing and optimizing the stored data for Amazon. A bucket is a kind of folder that is given a unique URL. For Amazon S3, you need an AWS Access ID, along with the associated key and region code (Figure 3). You can create a bucket in your AWS console if you have an account. Access IDs and passwords for individual user accounts are set in AWS Identity and Access Management. Note that the dialogs designed for Amazon can also be used to accommodate credentials for other bucket-oriented cloud services, such as Host Europe or Dunkel.

Figure 3: To back up data with Amazon, you need extensive information about the containers (buckets) you will be using.

In the next step, Advanced Settings, for each option you enable, the wizard treats you to an additional dialog. If you enable all six options (Figure 4), you then need to define three things: when the software should create a backup, the interval for removing old backups, and whether to take limits, such as the volume or bandwidth, into consideration. If these options are not sufficient, you can set others manually, such as with the parameters described for --snapshot-policy or --open-file-policy.

Figure 4: During the configuration process, the wizard gives you detailed options for influencing the data backup behavior.

After you select your options, the wizard schedules your backups and implements your choice of strategy. By default, Duplicati backs up every day at 1:00pm, always creates a full backup the first time out and incremental backups thereafter, and then creates another full backup every month. Additionally, you have the options Always perform an incremental backup, never full and Always perform a full backup, never incremental. However, not even the program itself recommends these variants.

The options for deleting in the next step help save space. You can define, among other things, how many backups the program keeps (default=4). Alternatively, you can specify a date at which the software removes backups. Another option lets you tell Duplicati to ignore timestamps for incremental backups.

If so desired, the wizard will help you comply with limits for uploading and downloading to and from the chosen cloud service. For example, if you want to avoid exceeding the quotas imposed in the free versions of Amazon S3 (5GB) or Google Drive (15GB), you can set a volume size limit.