Custom Shell Scripts

Text Processing with Variables

The most popular interpreters, and corresponding scripting languages for processing text strings, are probably Perl and Python. The current versions of Bash, however, include enough string processing operators to be more than adequate for the simplest and most common tasks. The main ones are:

#> NAME=Marco
#> echo ${NAME%co}
Mar
#> echo ${NAME/co/36}
Mar36
#>  echo ${NAME^^c}
MarCo
#> NAME=MARCO
echo ${NAME,,*}
marco

The statements above show how the shell can perform removal, substitution, and case modification, respectively, on parts of a variable.

The % operator removes the shortest matching pattern (co in the example) from a variable, starting from its end. The slashes delimit which part of the string should be replaced, and which other string to use. The caret (^) and comma (,) operators convert any occurrence of the matched characters to uppercase (c) or lowercase, respectively (the asterisk is a shortcut for "any character"). All these operators have other formats and variants: To know how they work, check the Bash man page [3].

It is also possible to extract and replace specific parts of a string, whatever their value. I will show how to do it, together with another very useful feature of the shell, with a final, simple script (Listing 2).

Listing 2

Replacing Specific Parts of a String

01  #! /bin/bash
02
03  FIRSTNAME=$1
04  SURNAME=$2
05  PHONE=$3
06
07  cat<<INITIAL_DATA
08
09  Initial Data:
10   First Name   : $FIRSTNAME
11   Surname : $SURNAME
12   Phone Number : $PHONE
13
14  INITIAL_DATA
15
16  FIRSTNAME="${FIRSTNAME,,}"
17  FIRSTNAME="${FIRSTNAME^}"
18
19
20  SURNAME="${SURNAME,,}"
21  SURNAME="${SURNAME^}"
22
23  PHONE_1="${PHONE:0:3}"
24  PHONE_2="${PHONE:4:3}"
25  PHONE_3="${PHONE:6}"
26
27  cat<<FORMATTED_DATA
28
29  Formatted Data:
30   First Name   : $FIRSTNAME
31   Surname : $SURNAME
32   Phone Number : ($PHONE_1) $PHONE_2-$PHONE_3
33
34  FORMATTED_DATA
35 exit

This code is a really bare-bones formatter of names and phone numbers: No matter how you write them, the script returns name and surname with the first letter capitalized and the phone number in a standard format.

Listing 3 shows what you get when you run the script from Listing 2.

Listing 3

Output from

#> phone-formatter LEia ORGaNA 5553236856
Initial Data:
                First Name   : LEia
                Surname      : ORGaNA
                Phone Number : 5553236856
Formatted Data:
                First Name   : Leia
                Surname      : Organa
                Phone Number : (555) 236-6856

As is, the script is not complete, because it only formats one entry at a time. In a future installment, I will show how to work on multiple records. Even in this form, however, the code is useful as a first approach to text formatting and parsing, because the same tricks may be adapted, for example, to process generic spreadsheets in CVS format.

In Listing 2, lines 3 to 5 show how you can pass different parameters to a script every time you run it. The variables $1, $2, and $3 contain the parameters that were passed in from the command line. You may use them directly, but for readability, it is much better to copy them into $FIRSTNAME, $SURNAME, and $PHONE.

Line 7 marks the beginning of a so-called "here document," which ends on line 14; cat, a Linux command, simply prints out whatever you pass to it.

A here document is a really cool way to use variables and generate shell output. You can think of a here document as a pseudo-file or, more appropriately, template that is embedded into a shell script. The template ends with a line containing the same string used to mark its beginning (INITIAL_DATA).

Inside the template, you can put as many variables as you like, and every time you use it, it will contain the current values of all those variables.

You can use here documents to custom generate any kind of complex, pre-formatted data on the fly, from database records to sequences of commands to pass to any other program.

Lines 16 to 21 just apply the case modification tricks already presented, and lines 23 to 25 show how to extract substrings. The final part of the script uses another here document to display the reformatted input values.

Conclusion

In this installment, I have covered the basic components that make shell scripts so flexible and usable. Next month, I'll play with more complex data structures like shell arrays and hashes. In the meantime, I highly recommend that you copy the provided script [5] and experiment both with the string processing commands and the here document layout to determine how much you can customize them. Happy scripting!

Infos

  1. Bash: http://www.gnu.org/software/bash/
  2. "Tutorial – Desktop News Feeds" by Marco Fioretti, Linux Magazine, issue 217, December 2018, pp. 90-95: http://www.linuxpromagazine.com/Issues/2018/217/Read-Me
  3. Seven expansion types: https://tiswww.case.edu/php/chet/bash/bash.html
  4. Generate random numbers in a given range: https://bytefreaks.net/gnulinux/bash/bash-get-random-number-that-belongs-in-a-range
  5. Code in this article: ftp://ftp.linux-magazine.com/pub/listings/linux-magazine.com/219/

The Author

Marco Fioretti (http://mfioretti.com) is a freelance author, trainer, and researcher based in Rome, Italy. He has been working with free and open source software since 1995 and on open digital standards since 2005. Marco also is a board member of the Free Knowledge Institute (http://freeknowledge.eu).

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy Linux Magazine

SINGLE ISSUES
 
SUBSCRIPTIONS
 
TABLET & SMARTPHONE APPS
Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

  • Tutorials – Shell Scripts

    Letting your scripts ask complex questions and give user feedback makes them more effective.

  • Bash Alternatives

    Don't let your familiarity with the Bash shell stop you from exploring other options. We take a look at a pair of alternatives that are easy to install and easy to use: Zsh and fish.

  • Bash vs. Vista PowerShell

    Microsoft’s new PowerShell relies on .NET framework libraries and thus has access to a treasure trove of functions and objects. How does PowerShell measure up to traditional shells like Bash?

  • Bash 4

    Despite the Bourne-again shell's biblical age and high level of maturity, developers continue to work on it. We take a look at the latest Bash release.

  • Bash Tuning

    In the old days, shells were capable of little more than calling external programs and executing basic, internal commands. With all the bells and whistles in the latest versions of Bash, however, you hardly need the support of external tools.

comments powered by Disqus
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters

Support Our Work

Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.

Learn More

News