Easy steps for optimizing shell scripts
Computation
You have several different options for calculating the sum of numbers that a text file contains. In the example in Listing 14, every line containing the fiftieth string is interesting. The script evaluates a file that contains one million lines. Every fiftieth line contains the string.
Listing 14
Looking for a String
#!/bin/bash typeset -i sum=0 while read line; do set $line # Totaling field 6 sum=$((sum+$6)) # Parse output from grep done < <(grep " fiftieth " largefile) echo "Sum total: $sum"
Here, too, you can use Awk as a tool for quick summation (Listing 15). But Awk does not work directly in machine language. For this reason, it makes sense to pass the search for the character string to the grep command and then add the lines found using Awk (Listing 16).
Listing 15
Faster with Awk
# time awk '/ fiftieth / {sum += $6} END {print "Sum total:", sum}' largefile
Listing 16
Fastest: Awk with grep
# time awk '{sum += $6} END {print "Sum total:", sum}' < <(grep " fiftieth " largefile)
In this example, too, optimization achieved significant speed gains. The variant from Listing 15 reduces the runtime to about one third; the variant from Listing 16 runs twenty times faster than the first alternative (see Table 3).
Table 3
Awk Timer
| Category | |||
|---|---|---|---|
| real |
0m4.471s |
0m1.408s |
0m0.231s |
| user |
0m2.374s |
0m1.348s |
0m0.050s |
| sys |
0m1.956s |
0m0.013s |
0m0.010s |
Conclusions
The examples in this article show that you can drastically increase the speed of your scripts by skillfully using multifunctional tools such as Awk, Python, or Perl and by avoiding complex constructs with Tr, Sed, or Grep: You will thus consistently avoid many context changes.
However, not every anonymous pipe is detrimental to throughput, as the last example shows. Instead, it is more important to use the strengths of the various tools and keep the number of subprocesses as low as possible.
It is also important to remember that you can improve the readability and thus the maintainability of the scripts by doing without complex chains of commands.
« Previous 1 2
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
US / Canada
UK / Australia
Direct Download
Read full article as PDF:
Price $2.95
News
-
Mozilla VPN Now Available for Linux
The promised subscription-based VPN service from Mozilla is now available for the Linux platform.
-
Wayland and New App Menu Coming to KDE
The 2021 roadmap for the KDE desktop environment includes some exciting features and improvements.
-
Deepin 20.1 has Arrived
Debian-based Deepin 20.1 has been released with some interesting new features.
-
CloudLinux Commits Over 1 Million Dollars to CentOS Replacement
An open source, drop-in replacement for CentOS is on its way.
-
Linux Mint 20.1 Beta has Been Released
The first beta of Linux Mint, Ulyssa, is now available for downloading.
-
Manjaro Linux 20.2 has Been Unleashed
The latest iteration of Manjaro Linux has been released with a few interesting new features.
-
Patreon Project Looks to Bring Linux to Apple Silicon
Developer Hector Martin has created a patreon page to fund his work on developing a port of Linux for Apple Silicon Macs.
-
A New Chrome OS-Like Ubuntu Remix is Now Available
Ubuntu Web looks to be your Chrome OS alternative.
-
System76 Refreshes the Galago Pro Laptop
Linux hardware maker has revamped one of their most popular laptops.
-
Dell Will Soon Enable Privacy Controls for Linux Hardware
Dell makes it possible for Linux users to disable webcams and microphones.