Text Processing with Variables

The most popular interpreters, and corresponding scripting languages for processing text strings, are probably Perl and Python. The current versions of Bash, however, include enough string processing operators to be more than adequate for the simplest and most common tasks. The main ones are:

#> NAME=Marco
#> echo ${NAME%co}
Mar
#> echo ${NAME/co/36}
Mar36
#>  echo ${NAME^^c}
MarCo
#> NAME=MARCO
echo ${NAME,,*}
marco

The statements above show how the shell can perform removal, substitution, and case modification, respectively, on parts of a variable.

The % operator removes the shortest matching pattern (co in the example) from a variable, starting from its end. The slashes delimit which part of the string should be replaced, and which other string to use. The caret (^) and comma (,) operators convert any occurrence of the matched characters to uppercase (c) or lowercase, respectively (the asterisk is a shortcut for "any character"). All these operators have other formats and variants: To know how they work, check the Bash man page [3].

It is also possible to extract and replace specific parts of a string, whatever their value. I will show how to do it, together with another very useful feature of the shell, with a final, simple script (Listing 2).

01  #! /bin/bash
02
03  FIRSTNAME=$1
04  SURNAME=$2
05  PHONE=$3
06
07  cat<<INITIAL_DATA
08
09  Initial Data:
10   First Name   : $FIRSTNAME
11   Surname : $SURNAME
12   Phone Number : $PHONE
13
14  INITIAL_DATA
15
16  FIRSTNAME="${FIRSTNAME,,}"
17  FIRSTNAME="${FIRSTNAME^}"
18
19
20  SURNAME="${SURNAME,,}"
21  SURNAME="${SURNAME^}"
22
23  PHONE_1="${PHONE:0:3}"
24  PHONE_2="${PHONE:4:3}"
25  PHONE_3="${PHONE:6}"
26
27  cat<<FORMATTED_DATA
28
29  Formatted Data:
30   First Name   : $FIRSTNAME
31   Surname : $SURNAME
32   Phone Number : ($PHONE_1) $PHONE_2-$PHONE_3
33
34  FORMATTED_DATA
35 exit

This code is a really bare-bones formatter of names and phone numbers: No matter how you write them, the script returns name and surname with the first letter capitalized and the phone number in a standard format.

Listing 3 shows what you get when you run the script from Listing 2.

#> phone-formatter LEia ORGaNA 5553236856
Initial Data:
                First Name   : LEia
                Surname      : ORGaNA
                Phone Number : 5553236856
Formatted Data:
                First Name   : Leia
                Surname      : Organa
                Phone Number : (555) 236-6856

As is, the script is not complete, because it only formats one entry at a time. In a future installment, I will show how to work on multiple records. Even in this form, however, the code is useful as a first approach to text formatting and parsing, because the same tricks may be adapted, for example, to process generic spreadsheets in CVS format.

In Listing 2, lines 3 to 5 show how you can pass different parameters to a script every time you run it. The variables $1, $2, and $3 contain the parameters that were passed in from the command line. You may use them directly, but for readability, it is much better to copy them into $FIRSTNAME, $SURNAME, and $PHONE.

Line 7 marks the beginning of a so-called "here document," which ends on line 14; cat, a Linux command, simply prints out whatever you pass to it.

A here document is a really cool way to use variables and generate shell output. You can think of a here document as a pseudo-file or, more appropriately, template that is embedded into a shell script. The template ends with a line containing the same string used to mark its beginning (INITIAL_DATA).

Inside the template, you can put as many variables as you like, and every time you use it, it will contain the current values of all those variables.

You can use here documents to custom generate any kind of complex, pre-formatted data on the fly, from database records to sequences of commands to pass to any other program.

Lines 16 to 21 just apply the case modification tricks already presented, and lines 23 to 25 show how to extract substrings. The final part of the script uses another here document to display the reformatted input values.

Conclusion

In this installment, I have covered the basic components that make shell scripts so flexible and usable. Next month, I'll play with more complex data structures like shell arrays and hashes. In the meantime, I highly recommend that you copy the provided script [5] and experiment both with the string processing commands and the here document layout to determine how much you can customize them. Happy scripting!