Easy steps for optimizing shell scripts
Computation
You have several different options for calculating the sum of numbers that a text file contains. In the example in Listing 14, every line containing the fiftieth
string is interesting. The script evaluates a file that contains one million lines. Every fiftieth
line contains the string.
Listing 14
Looking for a String
#!/bin/bash typeset -i sum=0 while read line; do set $line # Totaling field 6 sum=$((sum+$6)) # Parse output from grep done < <(grep " fiftieth " largefile) echo "Sum total: $sum"
Here, too, you can use Awk as a tool for quick summation (Listing 15). But Awk does not work directly in machine language. For this reason, it makes sense to pass the search for the character string to the grep
command and then add the lines found using Awk (Listing 16).
Listing 15
Faster with Awk
# time awk '/ fiftieth / {sum += $6} END {print "Sum total:", sum}' largefile
Listing 16
Fastest: Awk with grep
# time awk '{sum += $6} END {print "Sum total:", sum}' < <(grep " fiftieth " largefile)
In this example, too, optimization achieved significant speed gains. The variant from Listing 15 reduces the runtime to about one third; the variant from Listing 16 runs twenty times faster than the first alternative (see Table 3).
Table 3
Awk Timer
Category | |||
---|---|---|---|
real |
0m4.471s |
0m1.408s |
0m0.231s |
user |
0m2.374s |
0m1.348s |
0m0.050s |
sys |
0m1.956s |
0m0.013s |
0m0.010s |
Conclusions
The examples in this article show that you can drastically increase the speed of your scripts by skillfully using multifunctional tools such as Awk, Python, or Perl and by avoiding complex constructs with Tr, Sed, or Grep: You will thus consistently avoid many context changes.
However, not every anonymous pipe is detrimental to throughput, as the last example shows. Instead, it is more important to use the strengths of the various tools and keep the number of subprocesses as low as possible.
It is also important to remember that you can improve the readability and thus the maintainability of the scripts by doing without complex chains of commands.
« Previous 1 2
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
Red Hat Adds New Deployment Option for Enterprise Linux Platforms
Red Hat has re-imagined enterprise Linux for an AI future with Image Mode.
-
OSJH and LPI Release 2024 Open Source Pros Job Survey Results
See what open source professionals look for in a new role.
-
Proton 9.0-1 Released to Improve Gaming with Steam
The latest release of Proton 9 adds several improvements and fixes an issue that has been problematic for Linux users.
-
So Long Neofetch and Thanks for the Info
Today is a day that every Linux user who enjoys bragging about their system(s) will mourn, as Neofetch has come to an end.
-
Ubuntu 24.04 Comes with a “Flaw"
If you're thinking you might want to upgrade from your current Ubuntu release to the latest, there's something you might want to consider before doing so.
-
Canonical Releases Ubuntu 24.04
After a brief pause because of the XZ vulnerability, Ubuntu 24.04 is now available for install.
-
Linux Servers Targeted by Akira Ransomware
A group of bad actors who have already extorted $42 million have their sights set on the Linux platform.
-
TUXEDO Computers Unveils Linux Laptop Featuring AMD Ryzen CPU
This latest release is the first laptop to include the new CPU from Ryzen and Linux preinstalled.
-
XZ Gets the All-Clear
The back door xz vulnerability has been officially reverted for Fedora 40 and versions 38 and 39 were never affected.
-
Canonical Collaborates with Qualcomm on New Venture
This new joint effort is geared toward bringing Ubuntu and Ubuntu Core to Qualcomm-powered devices.