A download manager for the shell
Short and Painless
Listing 5 shows a function that sensibly splits up the download selection. You make the selection after listing the downloads by typing 1,2,3,4-10
, for example. In this case, the script takes the downloads from 1
to 10
, which are then sent to an array in the form of (1:*2 3:*4 ...)
.
Listing 5
Splitter Function
function splitter () { sed 's/,/\n/g' <<<$* | sed -r '/-/ s/([0-9]+)-([0-9]+)/seq \1 \2/e' | sort -nu }
The splitter
function strips the string to remove the commas and creates a sorted, space-separated string of numbers from ranges such as 4-10
. You can use these selection numbers later on to find the downloads contained in the indexed_downloads
array.
Keep in mind that the count for Bash arrays always starts at
. For example, to select file number 5
, you need to find it by querying indexed_indexes[4]
. You can use the same index with the indexed_downloads
array to retrieve the associated value.
Listing 6 shows the function that renames the files at download time. You need to pass in the base name of the URL as a parameter. At this point, the script is already in the right directory for the file extension (e.g., jpg/
). The parameters starting in line 3 find other files that are already in the directory before the download starts. Bash uses the command from line 4 to check if there are files with the same name. If so, the script inserts an underscore (_
) between the name and the dot that separates it from the file extension. This will also tell you how many times the file has been renamed. If the name contains one underscore, the file was renamed once; if it contains two, the file was renamed twice; and so on.
Listing 6
Rename Function
01 function rename () { 02 filename=$1 03 other_filenames=`echo ${@:2}` 04 while grep -q -F "${filename}" <<<${other_filenames}; do 05 filename=$(sed -r 's/(.+)(\.)(.+)/\1_\2\3/' <<<${filename}) 06 done 07 echo ${filename} 08 }
At this point, you should debug the functions in detail by isolating them. For example, using sed
and the command from Listing 7, you could write the warn
and rename
functions to a separate file, where you would then subsequently debug them with, for example,
bash -x debug
Listing 7
Debug Function
$ cat <(sed -r -n '/function (warn|rename)/,/^}/p' downloader_optimized2.bash) > debug
by renaming the function within the file and calling it with the associated parameters.
Listing 8 shows the download function, which first filters out the base name from the download link. To do this, it deletes all path information, as well as http://...
or https://...
, until only the actual file name remains. The script then finds the file extension and, if it does not already exist, creates a directory with this name. Then it changes to the directory and starts the download after running the rename
function.
Listing 8
Download Function
function download () { name=$(basename $1) suffix=$(cut -f 2 -d "." <<<${name}) [ ! -e ${download_directory}/${suffix} ] && mkdir ${download_directory}/${suffix} cd ${download_directory}/${suffix} && files_in_directory=$(ls) future_name=$(rename $name $files_in_directory) wget -O $future_name $1 }
Listing 9 generates a menu that lists the available downloads. This function starts a loop that iterates across the indexed_downloads
array and outputs the size and the base name from the array index one line at a time. At the end of the loop in line 6, everything is piped to gawk
.
Listing 9
Menu Function
01 function menu () { 02 for index in ${!indexed_downloads[@]}; do 03 local base_name=$(basename ${indexed_downloads[$index]}) 04 local size=${index} 05 echo "${size} ${base_name}" 06 done | gawk --assign free=${free} -F " " -f cutter.awk 07 }
Thanks to the -f cutter.awk
option, gawk
knows which AWK file to use as the program text. The call has an additional --assign free=${free}
option, which ensures that the gawk
script is aware of the free disk space previously determined in Bash. gawk
then examines the file size and the base name one line at a time and evaluates both line by line.
Formatted Displays
The gawk
script, cutter.awk
, in Listing 10 starts with two functions I defined myself. The first function, cutter
, truncates long basic names for the display by cutting them into two parts and dropping three dots into the middle. The second function, separating_line
, generates separating lines in the display to improve clarity for download pages with a large number of links.
Listing 10
cutter.gawk
function cutter( word ){ l = length(word) part1 = substr(word,1,8) part2 = substr(word,l-22) return part1"..."part2 } function separating_line ( lesser_equal ) { for ( p = 0; p <= lesser_equal ; p++){ printf "%s" (p == lesser_equal ? "\n" : "") ,"=" } } BEGIN { i = 1 printf "%8s %18s %10s %13s %s\n", "Download", "Kilobytes", "Megabytes", "Gigabytes", "Filename" printf "%-5s %21.2f %10.2f %13.2f %s\n", "Disc:", free, free/1024, free/(1024*1024),"home or /" separating_line(75) } { if ( length($2) > 40 ) { $2 = cutter($2) } printf "%2i => %21.2f %13.2f %13.2f %s\n", i++, $1/1024, $1/(1024*1024), $1/(1024*1024*1024), $2 total += $1 } END { separating_line(75) printf "Totals: %19.2f %13.2f %13.2f All downloads together\n", total/1024, total/(1024*1024), total/(1024*1024*1024) }
The BEGIN
block defines some basics as well as header formats. In addition, it shows the free space on the hard disk, or in your home directory, in the line following the header.
Finally, the command block without a pattern specification in lines 21 to 27 iterates through the individual lines, displaying the download size (in kilobytes, megabytes, and gigabytes) in the first field and the basic name in the second. This can be useful if you are downloading smaller files, such as wallpapers, ebooks, or MP3 files – output in gigabytes only, for example, would not make much sense here. This command block also computes the total size of all downloads and finally outputs it via the END
block (Figure 1).
Control
Finally, Listing 11 shows the main function that controls the entire program flow. As soon as you copy a URL from the browser by pressing Ctrl+C, xclip
accesses it (line 1).
Listing 11
Main Function
01 url=$(xclip -o 2>/dev/null) 02 03 if [ -z $url ]; then 04 warn "No URL present." 05 warn "Use [Ctrl]+[C] to copy the URL from the browser to the clipboard." 06 exit 1 07 fi 08 09 capture 10 11 if [ ${#download_links[@]} -gt 0 ]; then 12 menu 13 read -p "Select files (Example: 1,2,3-8,10 ): " selection 14 else 15 warn "NO DOWNLOADS PRESENT" && exit 1 16 fi 17 18 declare -i total_size_dowloads=0 19 20 for selection in $(splitter $selection); do 21 total_size_dowloads+=${indexed_indexes[((select - 1))]} 22 current_download=${indexed_downloads[${indexed_indexes[((select - 1))]}]} 23 if [[ ${frei}-5000 -lt ${total_size_dowloads}/1024 ]]; then 24 warn "Not enough free disc space." 25 warn "Canceling ${Current_download}." 26 exit 1 27 else 28 download $current_download 29 fi 30 done
If the graphical interface uses multiple clipboards (like FVWM), you need to use the -selection primary
or secondary
option to explicitly specify which clipboard xclip
should use. After capturing the clipboard, the script checks whether the URL was also filled with the clipboard contents. If the URL has a length of zero, the program cancels the operation.
Lines 11 to 16 then check whether any download links have been captured on the page and whether they can be output. If there are no files to download, the script also terminates at this point.
Finally, a for
loop processes all the downloads, checking their sizes and comparing them against the free disk space. If the size of the selected downloads exceeds the free disk space size, the script terminates.
Because all you need to do is copy the URL of the desired site for downloading files from the browser's address bar for the script to start automatically, this script works with all web browsers. The script retrieves the URL from the clipboard, evaluates the page for files that can be downloaded, and shows them to you sorted by size. You can then conveniently choose which of them you want to download.
« Previous 1 2 3 Next »
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
New Steam Client Ups the Ante for Linux
The latest release from Steam has some pretty cool tricks up its sleeve.
-
Gnome OS Transitioning Toward a General-Purpose Distro
If you're looking for the perfectly vanilla take on the Gnome desktop, Gnome OS might be for you.
-
Fedora 41 Released with New Features
If you're a Fedora fan or just looking for a Linux distribution to help you migrate from Windows, Fedora 41 might be just the ticket.
-
AlmaLinux OS Kitten 10 Gives Power Users a Sneak Preview
If you're looking to kick the tires of AlmaLinux's upstream version, the developers have a purrfect solution.
-
Gnome 47.1 Released with a Few Fixes
The latest release of the Gnome desktop is all about fixing a few nagging issues and not about bringing new features into the mix.
-
System76 Unveils an Ampere-Powered Thelio Desktop
If you're looking for a new desktop system for developing autonomous driving and software-defined vehicle solutions. System76 has you covered.
-
VirtualBox 7.1.4 Includes Initial Support for Linux kernel 6.12
The latest version of VirtualBox has arrived and it not only adds initial support for kernel 6.12 but another feature that will make using the virtual machine tool much easier.
-
New Slimbook EVO with Raw AMD Ryzen Power
If you're looking for serious power in a 14" ultrabook that is powered by Linux, Slimbook has just the thing for you.
-
The Gnome Foundation Struggling to Stay Afloat
The foundation behind the Gnome desktop environment is having to go through some serious belt-tightening due to continued financial problems.
-
Thousands of Linux Servers Infected with Stealth Malware Since 2021
Perfctl is capable of remaining undetected, which makes it dangerous and hard to mitigate.