Manipulating Binary Data with Bash
ASCII to Unicode
In some situations, you might need to convert an ASCII character into Unicode. ASCII is an 8-bit character set, whereas as Unicode starts at a 16-bit length. Converting from ASCII to Unicode might seem complicated, but it is actually quite simple thanks to the backward compatibility built into the Unicode standard. To convert ASCII to Unicode, you just need to prepend the value of 0 onto each ASCII character, thus making it a 16-bit character (see Listing 3).
Listing 3
ASCII to Unicode
01 ascii2unicode() { 02 echo "$1" | sed 's/\(.\)/\1\n/g' | awk '/^.$/{ printf("%c%c",0,$0) }' 03 } 04 05 command> ascii2unicode jello 06 output> jello
In Listing 3, the output of the command appears to show no noticeable change. To get a better view of the binary data behind this text, pipe the output into xxd
:
command> ascii2unicode jello | xxd output> 0000000: 006a 0065 006c 006c 006f .j.e.l.l.o
As you can see, the ASCII values have been prepended with "00," which converts them to 16-bit Unicode characters. Take a closer look at Listing 3 to see what's happening: The output of the echo
statement is piped into the sed
statement, which places each character of the output on a separate line. The awk
command reads the input from the sed
command line-by-line, and when the line contains a single character, it prints the character prepended by the character value "0".
URL Encoding and Decoding
Hexadecimal data is something you see every day, but it often goes unnoticed. When data is passed as a query string in a URL, it may be encoded using special formatting. This formatting consists of a percent sign followed by the hexadecimal value of an ASCII character. For example, the URL encoded string of "%61%62%63," when decoded, becomes "abc." Listing 4 shows a function for performing URL encoding and decoding.
Listing 4
URL Encoding and Decoding
01 urlencode() { 02 echo -n "$1" | xxd -p | tr -d '\n' | sed 's/\(..\)/%\1/g' 03 } 04 05 urldecode() { 06 tr -d '%' <<< "$1" | xxd -r -p 07 } 08 09 command> urlencode name 10 output> %6e%61%6d%65 11 command> urldecode %64%6f%6e%65 12 output> done
The function in Listing 4 uses the standard functionality of xxd
. When encoding a string, the output of xxd
is split into 1-byte chunks and prepended with a "%" by the sed
command. When decoding, all percent signs are stripped and the output is piped into xxd
to revert the hexadecimal string to ASCII.
Calculating IP Subnets
On an IP network, the subnet mask specifies how many bits of the IP address will be dedicated to the network ID and how many will be used for the host ID. The size of the host ID address space will tell you how many host IP addresses are available. Listing 5 shows how to convert the subnet mask to a binary string and determine the host ID count.
Listing 5
Converting a Subnet Mask
01 subnetcalc() { 02 echo -n "$1" | \ 03 awk 'BEGIN { FS="." ; printf("obase=2;ibase=A;") } { printf("%s;%s;%s;%s;\n",$1,$2,$3,$4) }' | \ 04 bc | sed 's/^0$/00000000/g;s/\(.\)/\1\n/g' | \ 05 awk 'BEGIN { ht = 0; nt = 0; } 06 /[01]/ { if ($0=="1") nt++; if ($0=="0") ht++; } 07 END { printf("Network bits: %s\nHost bits: %s\nHost IP Count: %d\n",nt,ht,2^ht); }' 08 } command> subnetcalc 255.255.192.0 output> Network bits: 18 output> Host bits: 14 output> Host IP Count: 16384
The output of the echo
statement is fed into the awk
statement. This first awk
command will generate the statement that is piped into the following bc
command. The statement will include ibase
, obase
, and each individual octet of the subnet mask. Once bc
evaluates the statements, it returns four lines: one line for each octet. The following sed
statement finds lines containing only "0" and extends them to 8-bits of zeros. The sed
statement also puts each bit on a line by itself. This will be necessary to properly evaluate the host bit length. The awk
statement has three sections. The first section initializes the ht
and nt
variables, which store the host total bits and network total bits, respectively. The next section searches for lines containing 0 or 1. If the value is 1, the network total is incremented, and if the value is 0, the host total is incremented. The final section of the awk
statement prints the summary data for the network, including the host and network bit counts, along with the host IP count.
« Previous 1 2 3 Next »
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Direct Download
Read full article as PDF:
Price $2.95
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Find SysAdmin Jobs
News
-
KDE Plasma 5.27 Beta is Ready for Testing
The latest beta iteration of the KDE Plasma desktop is now available and includes some important additions and fixes.
-
Netrunner OS 23 Is Now Available
The latest version of this Linux distribution is now based on Debian Bullseye and is ready for installation and finally hits the KDE 5.20 branch of the desktop.
-
New Linux Distribution Built for Gamers
With a Gnome desktop that offers different layouts and a custom kernel, PikaOS is a great option for gamers of all types.
-
System76 Beefs Up Popular Pangolin Laptop
The darling of open-source-powered laptops and desktops will soon drop a new AMD Ryzen 7-powered version of their popular Pangolin laptop.
-
Nobara Project Is a Modified Version of Fedora with User-Friendly Fixes
If you're looking for a version of Fedora that includes third-party and proprietary packages, look no further than the Nobara Project.
-
Gnome 44 Now Has a Release Date
Gnome 44 will be officially released on March 22, 2023.
-
Nitrux 2.6 Available with Kernel 6.1 and a Major Change
The developers of Nitrux have officially released version 2.6 of their Linux distribution with plenty of new features to excite users.
-
Vanilla OS Initial Release Is Now Available
A stock GNOME experience with on-demand immutability finally sees its first production release.
-
Critical Linux Vulnerability Found to Impact SMB Servers
A Linux vulnerability with a CVSS score of 10 has been found to affect SMB servers and can lead to remote code execution.
-
Linux Mint 21.1 Now Available with Plenty of Look and Feel Changes
Vera has arrived and although it is still using kernel 5.15, there are plenty of improvements sure to please everyone.