Manipulating Binary Data with Bash
Bit Trip
Bash is known for admin utilities and text manipulation tools, but the venerable command shell included with most Linux systems also has some powerful commands for manipulating binary data.
One of the most versatile scripting environments available on Linux is the Bash shell. The core functionality of Bash includes many mechanisms for tasks such as string processing, mathematical computation, data I/O, and process management. When you couple Bash with the countless commandline utilities available for everything from image processing to virtual machine (VM) management, you have a very powerful scripting platform.
One thing that Bash is not generally known for is its ability to process data at the bit level; however, the Bash shell contains several powerful commands that allow you to manipulate and edit binary data. This article describes some of these binary commands and shows them at work in some practical situations.
Viewing and Converting Data
Two tools you can use to represent data in hexadecimal and binary format are hexdump
and xxd
. The hexdump
utility provides many options for outputting hexadecimal data. Although xxd
lacks some of the options available with hexdump
, it does have one key feature hexdump
lacks: In addition to letting you output hexadecimal values, xxd
also has the ability to convert a hexadecimal string into binary data. The following command:
command> echo n "hello"  xxd p output> 68656c6c6f
outputs the binary values of the ASCII string "hello" as an ASCII string of hexadecimal values (refer to Table 1).
Table 1
ASCII Lowercase Alphabet
Char  Decimal  Hexadecimal 

a 
97 
61 
b 
98 
62 
c 
99 
63 
d 
100 
64 
e 
101 
65 
f 
102 
66 
g 
103 
67 
h 
104 
68 
i 
105 
69 
j 
106 
6A 
k 
107 
6B 
l 
108 
6C 
m 
109 
6D 
n 
110 
6E 
o 
111 
6F 
p 
112 
70 
q 
113 
71 
r 
114 
72 
s 
115 
73 
t 
116 
74 
u 
117 
75 
v 
118 
76 
w 
119 
77 
x 
120 
78 
y 
121 
79 
z 
122 
7A 
In the following command:
command> echo n "6162"  xxd p r output> ab
The hexadecimal string "6162" is converted into what the xxd
man page calls a "mailsafe ASCII representation" of the binary data. Because hexadecimal 61 and 62 correspond to ASCII characters "a" and "b," respectively, the binary data is expressed in the form of the string "ab."
Bash makes it easy to build these commands into functions. The following functions, pack
and unpack
(named because of their similarity to the PHP pack
function) use the preceding commands to convert a hexadecimal string to binary and convert binary to hex.
pack() { echo n "$1"  xxd p r } unpack() { echo n "$1"  xxd p }
All the examples so far have used the p
switch to specify that the command will use plain hexdump style, representing each 8bit byte as a twodigit hexadecimal number. To output data as a binary string (containing 1s and 0s), you need to use different switches. The following function returns a binary string representation of binary data, along with a sample command and output:
tobin() { echo n "$1"  xxd b g0  awk '{ printf("%s",$2) }' } command> tobin ab output> 0110000101100010
The twocharacter string "ab" is converted into a binary string containing two 8bit values: 01100001 and 01100010 (97 and 98), corresponding to the base 10 values for characters "a" and "b."
Transforming Binary Data
Bash also offers some commands for transforming binary data using mathematical and logical operations. This process is call arithmetic expansion. In the following example:
command> echo $(( 4*5 )) output> 20
the output of the echo
command is the value calculated by the statement enclosed in $((
and ))
.
Another tool for performing mathematical operations is bc
. bc
is a commandline calculator that can read statements from standard input. For example:
command> echo "4*5"  bc output> 20
Note that the preceding operation is performed in base 10. bc
does not provide native functionality for arithematic operations on hexadecimal numbers, so you need to convert the numbers to base 10 and then perform the operation. For instance, if you want to perform simple addition and subtraction operations against hexadecimal numbers, use a function like the following:
hexadd() { echo "obase=16;ibase=A;$((16#$1))+$2"  bc } command> hexadd A 2 output> C
The hexadd
function provides the ability to add or subtract (add a negative number) to a valid hexadecimal number in a single command. The function pipes a string of commands into bc
. The first command, obase=16
, sets the base in which data will be output (in this case, hexadecimal or base 16). The second command, ibase=A
, sets the base used to read input data. The input base is set up A
, which corresponds to base 10. The third and final command is an addition statement comprised of an arithmetic expansion and the second function argument. The arithmetic expansion uses the #
operator to convert the number following it from the base specified before the #
to base 10. In the command, the hexadecimal value for "A" is converted to decimal 10 using arithmetic expansion, added to 2 using bc
, and then converted back into hexadecimal using bc
.
You can also perform bitwise operations on the data, including AND, OR, XOR, and shift. The AND operation (using the &
operator) returns the bits shared between the two numbers. The OR operation (using the 
operator) returns all bits used by either of the two numbers. The XOR operation (using the ^ operator) returns all bits that are unique to one of the numbers.
In Listing 1, the first command performs a binary AND on 3 (or 0011 in binary) and 6 (or 0110 in binary). Because the only shared bit is the second bit, 2 (or 0010 in binary), the output for the command is 2. The second command performs an OR operation against the same two numbers: 3 and 6. Between the two numbers, the lower three bits are used, so 7 (or 0111 in binary) is returned. In the third example an XOR is performed against the same two numbers. The second and fourth bits are the same between the two numbers, however, the third and first bit are different, therefore 5 (or 0101 in binary), is returned.
Listing 1
Bitwise Operations
01 logicand() { 02 echo $(($1&$2)) 03 } 04 05 logicor() { 06 echo $(($1$2)) 07 } 08 09 logicxor() { 10 echo $(($1^$2)) 11 } 12 13 command> logicand 3 6 14 output> 2 15 command> logicor 3 6 16 output> 7 17 command> logicxor 3 6 18 output> 5
The other bitwise operations is the shift operation, which shifts the bits in one direction or another (right or left). Listing 2 shows functions for shifting the bits shifting right (using the >>
operator) and left (using the <<
operator).
Listing 2
Shifting Bits
01 shiftr() { 02 echo $(($1>>$2)) 03 } 04 05 shiftl() { 06 echo $(($1<<$2)) 07 } 08 09 command> shiftl 4 2 10 output> 16 11 command> shiftr 16 4 12 output> 1
The first command in Listing 2 shifts the bits in 4 (0100) left by two spaces, returning a value of 16 (10000). The second command shifts the bits in 16 (10000) to the right four spaces, returning a value of 1 (0001). As you might have noticed, for each bit space shifted to the left, the number is multiplied by 2, and for each bit space shifted to the right, the number is divided by 2.
Examples
Switching from binary to hexadecimal, and moving bits around to change an A into a C, is certainly interesting, but does this capability have any uses in the real world? The following examples offer some hints for how you could use these tools in practical ways.
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News

Manjaro Team Tests Immutable Version of its ArchBased Distribution
If you're a fan of immutable operating systems, you'll be thrilled to know that the Manjaro team is working on an immutable spin that is now available for testing.

Nitrux 3.6.0 ("ie") Available for Download and Install
The latest version of the systemdfree Linux distribution, Nitrux, has been released for general usage and includes the beta of the upcoming NVIDIA 560 graphics driver.

Vanilla OS 2 Available for Download
The latest iteration of Vanilla OS (Orchid) has been released and uses a hybrid Debian base.

DebianBased eLxr Distribution Announced for Edge Deployments
Open source eLxr offers a seamless operating environment for edgetocloud deployments.

NVIDIA Released Driver for Upcoming NVIDIA 560 GPU for Linux
Not only has NVIDIA released the driver for its upcoming CPU series, it's the first release that defaults to using opensource GPU kernel modules.

OpenMandriva Lx 24.07 Released
If you’re into rolling release Linux distributions, OpenMandriva ROME has a new snapshot with a new kernel.

Kernel 6.10 Available for General Usage
Linus Torvalds has released the 6.10 kernel and it includes significant performance increases for Intel Core hybrid systems and more.

TUXEDO Computers Releases InfinityBook Pro 14 Gen9 Laptop
Sporting either AMD or Intel CPUs, the TUXEDO InfinityBook Pro 14 is an extremely compact, lightweight, sturdy powerhouse.

Google Extends Support for Linux Kernels Used for Android
Because the LTS Linux kernel releases are so important to Android, Google has decided to extend the support period beyond that offered by the kernel development team.

Linux Mint 22 Stable Delayed
If you're anxious about getting your hands on the stable release of Linux Mint 22, it looks as if you're going to have to wait a bit longer.