Needle in a Haystack
Tutorials – Attachment Extraction
If your inbox is full of email messages with important attachments, retrieving those attachments manually can be a tedious task. The script presented in this article does this task automatically and can even save the email as a plain text file.
Do you ever find yourself urgently searching for a file that you know you received as an email attachment but do not remember who sent it or when? Has your company saved all the important documents received via email somewhere easily retrievable? Would you like to save the content of all your email messages automatically as separate, plain text files?
Being able to copy automatically, into one folder and as separate files, all the email attachments and message bodies hidden in your email archives might save your day in situations like these. This tutorial explains how to do it with one relatively simple shell script and tools available from the standard repositories of most Linux distributions. Only basic knowledge of shell scripts is necessary. Additionally, patching the script to make it save just the attachment is also very easy.
MIME and Mailbox Formats
To process email messages, you need to know how files are attached to email and how email messages are archived inside digital mailboxes. To extract attachments from one email message, you need a MIME-aware processor that can split all the email's parts into separate files. Multipurpose Internet Mail Extensions (MIME) [1] is the open standard that describes how to:
- encode non-ASCII character sets in email bodies and headers,
- format message bodies with multiple parts (e.g., content in both HTML and plain text formats), and
- encode and attach to email any non-textual content (e.g., images to generic binary files).
To archive multiple messages as one mailbox, the simplest format is MBOX, which just concatenates all messages into one plain text file. Because MBOX is as simple as it is inefficient, other formats that store each message in a separate file inside a folder were developed [2]. One of the most common formats, and the one used in this tutorial, is called Maildir, which has the internal structure shown in Figure 1.
![](/var/linux_magazin/storage/images/issues/2018/216/tutorials-attachment-extraction/figure-1/735828-1-eng-US/Figure-1_large.png)
Preliminary Work
Unless the mailbox you want to scan for attachments already is in Maildir format (or in any other format that puts every email into a separate file), you have to convert it. You can manually, but quickly, convert any mailbox to Maildir format in any Linux terminal using the Mutt email client:
1. Type:
mutt -R -m Maildir -f <YOUR MAILBOX>
2. Type T followed by the Enter key.
3. Type a semicolon ( ; ) and then s.
The -R
switch opens in read-only mode the mailbox defined with -f
, and -m
sets Maildir as the default mailbox format. By doing the second step inside Mutt, you tag all the messages in the mailbox. The third step makes Mutt ask you in which other mailbox all those messages should be saved, (i.e., copied, because the original mailbox is in read-only mode). Pass Mutt any mailbox name you want; when it has finished copying the messages, type q to exit.
If you have many mailboxes to convert, you can run Mutt from a script, as explained in an article online [3]. In any case, I recommend only working with copies of your mailbox, just to be on the safe side (not to mention that changing the access times of email files may confuse some email clients).
A Redundant, Overcautious Extractor
By default, the email extractor script that I introduce here extracts everything from email messages, not just attachments: message bodies, digital signatures, embedded images, and so on. With some very minor modifications, however, you can make it extract attachments only.
The code for this article is a shell script that contains three parts. For readability, I present them in three separate listings. The first part (Listing 1) defines all the necessary variables and folders, plus a shell function that generates unique folder names for each message.
Listing 1
Preparation
01 #! /bin/bash 02 03 MAILBOX="$1" 04 TARGET="$2" 05 TMPDIR="$HOME/extractor-tmp" 06 CNT=0 07 declare -A CHECKSUMS 08 09 emaildirname() { 10 MSGTS=`stat -c %Y $EMAIL` 11 ORIGTS=`date -d @$MSGTS '+%Y%m%d%H%M.%S'` 12 FILENAMETS=`date -d @$MSGTS '+%Y%m%d%H%M%S'` 13 MSGDATE=`grep '^Date: ' $EMAIL | cut -c7- | head -1` 14 MSGSUBJ=`grep '^Subject: ' $EMAIL | cut -c10-210 | head -1` 15 DIR=`echo $MSGDATE-$MSGSUBJ | sed -r 's/[^a-zA-Z0-9]+/-/g' | sed -r 's/-+$//'` 16 DIR="$FILENAMETS-$DIR-$CNT" 17 let "CNT=CNT+1" 18 } 19 20 rm -rf $TMPDIR $TARGET 21 mkdir -p $TMPDIR $TARGET/tmp 22 23 echo "EXTR mbox : $MAILBOX" 24 printf "EXTR contains : %s messages ( %s KBytes )\n" `find $MAILBOX -type f | wc -l ` `du -sk $MAILBOX | cut -f1` 25 echo "EXTR target : $TARGET"
I call the script "redundant" because its second part (Listing 2) does the actual job of saving all attachments and email bodies as separate files a total of four times, with four different tools. The final section (Listing 3) removes any empty or duplicate files produced by that process.
Listing 2
Extraction
26-28 <these lines only contained comments...> 29 for TOOL in mu uudeview munpack ripmime 30 do 31 mkdir -p $TMPDIR/$TOOL/ 32 printf "EXTR\nEXTR %-10s: %-9s start\n" $TOOL `date +%H:%M:%S` 33 CNT=0 34 35 for EMAIL in `find $MAILBOX/ -type f` 36 do 37 emaildirname 38 mkdir $TMPDIR/$TOOL/$DIR 39 cd $TMPDIR/$TOOL/$DIR 40 case $TOOL in 41 42 ripmime) 43 TOOLNUM=3 44 ripmime -i $EMAIL --paranoid ;; 45 46 munpack) 47 TOOLNUM=2 48 munpack -t -q $EMAIL > /dev/null ;; 49 50 uudeview) 51 TOOLNUM=1 52 uudeview +a -m -n -q -i $EMAIL ;; 53 54 mu) 55 TOOLNUM=0 56 mu extract -a $EMAIL ;; 57 esac 58 59 NUMFILES=`find . -type f | wc -l` 60 if [[ "$NUMFILES" -gt "0" ]] 61 then 62 find . -type f | cut -c3- > /tmp/file_list.txt 63 while IFS= read -r file 64 do 65 NEWNAME=`echo ${file%.*}` 66 NEWNAME=`echo $NEWNAME | sed -r 's/[^a-zA-Z0-9]+/-/g' | sed -r 's/-+$//'` 67 EXT=`echo $file | awk -F . '{print $NF}'` 68 69 if [ "$EXT" == "$NEWNAME" ] 70 then 71 EXT='probablyemailbody.txt' 72 fi 73 mv -- "$file" $FILENAMETS-$TOOLNUM-$CNT-$NEWNAME.$EXT 74 touch -t $ORIGTS $FILENAMETS-$TOOLNUM-$CNT-$NEWNAME.$EXT 75 done < /tmp/file_list.txt 76 fi 77 78 done 79 printf "EXTR %-10s: %-9s end\n" $TOOL `date +%H:%M:%S` 80 printf "EXTR %-10s: %-9s files extracted (%7s empty)\n" $TOOL `find $TMPDIR/$TOOL/ -type f | wc -l` `find $TMPDIR/$TOOL/ -type f -empty| wc -l` 81 82 find $TMPDIR/$TOOL/ -type f -exec mv -i {} $TARGET/tmp \; 83 84 done
Listing 3
Cleanup
85 86 find $TARGET/tmp -type f -empty -exec rm {} \; 87 88 printf "EXTR\nEXTR cleaning : %-9s start\n" `date +%H:%M:%S` 89 90 CNT=0 91 92 for F in `find $TARGET/tmp -type f | sort` 93 do 94 CK=`md5sum $F | sed 's/ .*$//' ` 95 if [ "${CHECKSUMS[$CK]}" == "found" ] 96 then 97 echo "DUPX: removing $F" 98 let "CNT=CNT+1" 99 else 100 echo "DUPX: keeping $F" 101 mv $F $TARGET/ 102 CHECKSUMS[$CK]='found' 103 fi 104 done 105 106 mv $TARGET/tmp $TARGET-tmp 107 108 printf "EXTR total : %-9s files found, after removing %s duplicates\n" `find $TARGET -type f | wc -l` $CNT 109 printf "EXTR cleaning : %-9s end\n" `date +%H:%M:%S` 110 >
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
![Learn More](https://www.linux-magazine.com/var/linux_magazin/storage/images/media/linux-magazine-eng-us/images/misc/learn-more/834592-1-eng-US/Learn-More_medium.png)
News
-
NVIDIA Released Driver for Upcoming NVIDIA 560 GPU for Linux
Not only has NVIDIA released the driver for its upcoming CPU series, it's the first release that defaults to using open-source GPU kernel modules.
-
OpenMandriva Lx 24.07 Released
If you’re into rolling release Linux distributions, OpenMandriva ROME has a new snapshot with a new kernel.
-
Kernel 6.10 Available for General Usage
Linus Torvalds has released the 6.10 kernel and it includes significant performance increases for Intel Core hybrid systems and more.
-
TUXEDO Computers Releases InfinityBook Pro 14 Gen9 Laptop
Sporting either AMD or Intel CPUs, the TUXEDO InfinityBook Pro 14 is an extremely compact, lightweight, sturdy powerhouse.
-
Google Extends Support for Linux Kernels Used for Android
Because the LTS Linux kernel releases are so important to Android, Google has decided to extend the support period beyond that offered by the kernel development team.
-
Linux Mint 22 Stable Delayed
If you're anxious about getting your hands on the stable release of Linux Mint 22, it looks as if you're going to have to wait a bit longer.
-
Nitrux 3.5.1 Available for Install
The latest version of the immutable, systemd-free distribution includes an updated kernel and NVIDIA driver.
-
Debian 12.6 Released with Plenty of Bug Fixes and Updates
The sixth update to Debian "Bookworm" is all about security mitigations and making adjustments for some "serious problems."
-
Canonical Offers 12-Year LTS for Open Source Docker Images
Canonical is expanding its LTS offering to reach beyond the DEB packages with a new distro-less Docker image.
-
Plasma Desktop 6.1 Released with Several Enhancements
If you're a fan of Plasma Desktop, you should be excited about this new point release.