Cloud backup with MCrypt and S3cmd

The Name's Ex, Regex

A very appealing feature of S3cmd is its ability to use regular expressions. This functionality means you can exclude certain files from upload or download commands. Try using --rexclude=.*\.log if you want to catch all files with the extension .log. This functionality opens up several other possibilities, especially within scripts. You can effectively run cron jobs and grab precisely the files you want (whether for upload or download) without fear of accidentally adding an extra 10GB of data to your payload.

Examining the Viscera

A directory is called a bucket in the AWS environment. To create a bucket with mb, where linux-is-the-best is the bucket name, do:

# s3cmd mb s3://linux-is-the-best

Use rb to delete a bucket. List the contents of your remote filesystem as follows:

# s3cmd ls s3://linux-is-the-best/chris-says

This command lists the contents of the chris-says subdirectory. You can also use s3cmd la to show all the files in all buckets.

Put It There

You might be surprised at how absurdly simple it is to upload and download with S3cmd. For uploading to Amazon S3, you simply use the put command:

# s3cmd put chrisbinnie.tar.gz.nc s3://linux-is-the-best

And, if you can't guess, you can use get for downloading:

# s3cmd get chris_backup_14411441.nc s3://only-use-linux

A final basic option is deleting a single file using del:

# s3cmd del rubbish.nc s3://temp-bucket

Another very handy function is du; for those unfamiliar, du is a command-line disk usage tool on Unix-like systems that lists the disk space used by each bucket. The output from this command is well suited to scripts for quashing the hungry appetites of certain filesystem hogs. You can use it as follows:

# s3cmd du

Additionally, you can optionally add a file path to the end of that command.

A useful pair of options let you move files within the Amazon S3 filesystem without uploading or downloading the file. As in Unix, you can move files (using mv) or duplicate them (by using cp). The move and copy functions are as follows:

# s3cmd mv s3://linux_bucket/chris.png s3://older_linux_bucket
# s3cmd cp s3://linux_bucket/binnie.jpg s3://older_linux_bucket

Note that you can also change the name of a file in one swift move with an elegant maneuver by adding something like /newer_chrisbinnie.jpg as a object name melded to the s3://older_linux_bucket destination.

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy Linux Magazine

SINGLE ISSUES
 
SUBSCRIPTIONS
 
TABLET & SMARTPHONE APPS
Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

  • Command Line – crypt

    If you just need to encrypt a file or two, a descendant of crypt can do the job. Which one you choose depends on your objective.

  • Duplicity Cloud Backup

    If you're looking for a secure and portable backup technique, try combining the trusty command-line utility Duplicity with an available cloud account.

  • Manage Amazon S3 with s3cmd
  • Charly’s Column: S3QL

    Sys admin Charly has been an enthusiastic amateur photographer for many years. Recently, he started worrying about something happening to his rapidly expanding photo collection. Can the cloud save the day?

  • Duplicati

    The free backup tool Duplicati simplifies the process of backing up data with cloud providers while at the same time protecting backups with strong cryptography.

comments powered by Disqus
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters

Support Our Work

Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.

Learn More

News