Cloud backup with MCrypt and S3cmd
Pay for What You Use
Because you have given up your sacred AWS access credentials, you also have the ability to control other AWS services and, in the case of S3cmd, that includes the superb "pay for what you use" service called CloudFront [5]. If you're unfamiliar with Content Delivery Networks (CDNs), CloudFront is undeniably both powerful and affordable. You can use S3cmd to query, create, and modify several of CloudFront's functions.
I'll take an extremely brief look at AWS's widely used CDN integration with S3cmd. Each instance of a CDN configuration assigned to your account is referred to as a "distribution" (probably because it efficiently distributes your data around the globe). You can display a list of your configured distributions with the command:
# s3cmd cflist
If you want a bit more detail about the configured parameters for all your distributions, use:
# s3cmd cfinfo
You can also query a specific distribution and its parameters by referencing its distributionID
as follows (adding your own ID accordingly):
# s3cmd cfinfo cf://<distributionID>
You can also use cfcreate
, cfdelete
, and cfmodify
to create, delete, and change a distribution without messing about with web interfaces.
Please, Sir, I Want Some More
The list of S3cmd features is really comprehensive for such a diminutive utility. The list includes a --force
overwrite option, which should be used with great care, and a very useful --dry-run
flag, which lets you display the files that will be uploaded and downloaded – without it actually happening. If you're ever worried about breaking things horribly by getting a regex entry incorrect, you will appreciate the --dry-run
feature.
The useful --continue
option only works with downloads, but it should, in theory at least, resume a partially downloaded file so you don't have to bother starting a big download again from scratch. It's fair to say that HTTP resumes have been around for a while and this capability is not a new, earth-shattering feature, but it is still a nice and well-needed touch.
Stop me if I have already mentioned the -r
parameter (otherwise known as --recursive
), which works on uploads, downloads, and deletions if you want to affect subdirectories. Use this with caution if you want to avoid incurring massive data transfer fees or accidental deletions.
Reduced Cost
If you are storing large amounts of data, you might be concerned about minimizing expenses. Apparently (and please be warned that specifications, configurations, and procedures change frequently with emerging technologies, so don't take this as gospel), the standard repository for an uploaded Amazon S3 file spans three data centers. In other words, your file is copied three times across three geographically disparate buildings.
If you only use two data centers, you can use "Reduced Redundancy," and AWS will lower its storage fees. The theory is that these files won't be as critical to you; therefore, you might tolerate losing one or two. Perhaps you will have local backups available or maybe the files have a limited shelf life.
The -rr
switch, or --reduced-redundancy
in longhand, allows you to instruct the clever S3cmd that you need to watch your costs and only use two data centers for storage.
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
Canonical Releases Ubuntu 24.04
After a brief pause because of the XZ vulnerability, Ubuntu 24.04 is now available for install.
-
Linux Servers Targeted by Akira Ransomware
A group of bad actors who have already extorted $42 million have their sights set on the Linux platform.
-
TUXEDO Computers Unveils Linux Laptop Featuring AMD Ryzen CPU
This latest release is the first laptop to include the new CPU from Ryzen and Linux preinstalled.
-
XZ Gets the All-Clear
The back door xz vulnerability has been officially reverted for Fedora 40 and versions 38 and 39 were never affected.
-
Canonical Collaborates with Qualcomm on New Venture
This new joint effort is geared toward bringing Ubuntu and Ubuntu Core to Qualcomm-powered devices.
-
Kodi 21.0 Open-Source Entertainment Hub Released
After a year of development, the award-winning Kodi cross-platform, media center software is now available with many new additions and improvements.
-
Linux Usage Increases in Two Key Areas
If market share is your thing, you'll be happy to know that Linux is on the rise in two areas that, if they keep climbing, could have serious meaning for Linux's future.
-
Vulnerability Discovered in xz Libraries
An urgent alert for Fedora 40 has been posted and users should pay attention.
-
Canonical Bumps LTS Support to 12 years
If you're worried that your Ubuntu LTS release won't be supported long enough to last, Canonical has a surprise for you in the form of 12 years of security coverage.
-
Fedora 40 Beta Released Soon
With the official release of Fedora 40 coming in April, it's almost time to download the beta and see what's new.