Redirect data streams with pipes

Using Pipes

Experienced Linux users are probably familiar with the use of pipes. For instance, the following example finds and counts all the subdirectories of the directory where the command is invoked:

$ ls -l | grep "^d" | wc -l
138

The output from a call to ls is routed to the grep command's input channel, while the output from grep itself is routed to the input of the wc command (Figure 2).

Figure 2: Pipes let you connect an arbitrary number of commands.

The shell defines which way redirection works for anonymous pipes. It is always from left to right. On completing the command sequence, all processes are terminated and the redirections are revoked.

For some command lines and scripts, you may need pipes for a longer period of time or for multiple processes. For these instances, you can use a named pipe. Named pipes reside in the filesystem, like other files, and persist after a reboot. Multiple processes can use named pipes. There are no restrictions to reading from or writing to the data flow.

Named Pipes

To create a permanent pipe, you can use the mkfifo command. The example shown in Listing 2 creates a pipe named /var/tmp/testpipe. The ls command shows you that the action worked. In the output from ls (line 3), the p flag on the left tells you that this is a pipe. You can now redirect the output of a command to this pipe. The command waits before completing processing until another process has read the data from the pipe (Listing 3).

Listing 2

Creating a Named Pipe

01 $ mkfifo /var/tmp/testpipe
02 $ ls -l /var/tmp/testpipe
03 prw-r--r-- 1 root root 0 Jan 4 23:35 /var/tmp/testpipe

Listing 3

Using Named Pipes

### Session 1: Write
$ echo "3.1415" >/var/tmp/testpipe
### Session 2: Read
$ ls -l /var/tmp/testpipe
prw-r--r- 1 root root 0 Jan 4 23:35 /var/tmp/testpipe
$ pi=$(cat /var/tmp/testpipe)
$ ls -l /var/tmp/testpipe
prw-r--r- 1 root root 0 Jan 4 23:42 /var/tmp/testpipe
$ echo $pi
3.1415

You can see that the call changes the timestamp from 23:35 to 23:42 after reading. However, the size of the file is still zero bytes because the pipe theoretically only transfers the data. What actually happens, however, is that the operating system provides a buffer, although this is not important for general usage.

A simple example (shown in Listings 4 and 5) clearly demonstrates that – from the processes' perspective – writing to a pipe only starts when the pipe is read. In this session, a command writes the current date and time to the start variable and then writes the variable's contents to the pipe. It then reads the date and time again and stores the values in the end variable. The process also immediately writes its value to the pipe. The second session reads both lines from the pipe and outputs them.

Listing 4

Defining Variables

# started=$(date +'%H:%M:%S') ; echo $started
# ended=$(date +'%H:%M:%S'); echo $ended
23:04:44
23:04:44

To show that the output is actually generated within next to no time, the first run does not use redirection (Listing 4). Now the two variables are again assigned timestamps and the output is written to the pipe in each case. Shortly after this, the pipe is read line by line in a second session and the results are output onscreen (Listing 5). You can see that there is a difference of several seconds between start and end. The difference results from the fact that the second timestamp is not created until the first line has been read from the pipe.

Listing 5

Reading from a Pipe

### Terminal 1:
$ started=`date +'%H:%M:%S'` ; echo $start >/var/tmp/testpipe ; ende=`date +'%H:%M:%S'` ; echo $ended >/var/tmp/testpipe
### Terminal 2:
$ read started </var/tmp/testpipe ; read ended </var/tmp/testpipe ; echo "Start: $started" ; echo "End: $ended"
Start: 23:08:40
End: 23:08:52

Sample Application

Listing 6 implements a network connection based on named pipes and uses the connection to execute commands on a remote computer.

Listing 6

Remote Command

01 $ ssh [-q] target "command1 [; command2 [;...]]"
02 $ rcmd "command1 [; command2 [;...]]"

The interesting aspect about the communication between two computers is that it does not matter, for the reading or writing process, what happens before or after the pipe, because the read and write actions do not change for the respective process. Listing 6 assumes that access between the two machines is not password protected and relies on SSH in the example.

The ssh command lets you stipulate both a command chain and the target computer (line 1, Listing 6). SSH now connects to the target system, executes the specified commands, and then terminates the connection. It would be simpler to call a function that is then handed the command to be executed, runs it on the other machine, and prints the output on the local screen (line 2).

The rmcd function is from the functions script (Listing 7). Each computer uses one pipe for sending and one pipe for receiving. A process permanently reads the sending pipe and redirects the lines it reads to a data flow. In this example, SSH sends the data flow to the remote computer. On the receiving side, the incoming data flow is redirected to the receiving pipe where it is read. The pipe processes the data that was read and writes the results to the sending pipe in order to send the results back to the originating computer (Figure 3).

Listing 7

The functions Script

001 setenv() {
002  # Set required environment variables
003  awkfile=/var/tmp/read.awk
004  BASENAME=${BASENAME:=/usr/bin/basename}
005  PYTHON=${PYTHON:=/usr/bin/python}
006  AWK=${AWK:=/usr/bin/awk}
007  TAR=${TAR:=/usr/bin/tar}
008  MKFIFO=${MKFIFO:=/usr/bin/mkfifo}
009  UNAME=${UNAME:=/usr/bin/uname}
010  TAIL=${TAIL:=/usr/bin/tail}
011  CAT=${CAT:=/usr/bin/cat}
012  SSH=${SSH:=/usr/bin/ssh}
013  NOHUP=${NOHUP:=/usr/bin/nohup}
014  RM=/usr/bin/rm
015  rhost=${REMOTEHOST:=localhost}
016  lhost=${LOCALHOST:=`$UNAME -n`}
017  sendpipe=/tmp/send.${rhost}
018  receivepipe=/tmp/receive.${rhost}
019  rsendpipe=/tmp/send.${lhost}
020  rreceivepipe=/tmp/receive.${lhost}
021 }
022
023 chkpipes() {
024  # Check whether reuqired pipes exist
025  for pipe in $sendpipe $receivepipe
026  do
027   if [ ! -p $pipe ]
028   then
029    echo "cannot communicate with $rhost" >&2
030    return 1
031   fi
032  done
033  return 0
034 }
035
036 createpipes() {
037  # Generate required pipes
038  setenv
039  for pipe in $sendpipe $receivepipe
040  do
041   if [ ! -p ${pipe} ]
042   then
043    $MKFIFO ${pipe}
044    if [ $? -ne 0 ]
045    then
046     echo "Cannot create ${pipe}" >&2
047     return 1
048    fi
049   else
050    echo "Pipe ${pipe} already exists"
051   fi
052  done
053  return 0
054 }
055
056 removepipes() {
057  # Delete pipes if so desired
058  setenv
059  for pipe in $sendpipe $receivepipe
060  do
061   if [ -p ${pipe} ]
062   then
063    rm -f ${pipe}
064    if [ $? -ne 0 ]
065    then
066     echo "Cannot remove ${pipe}" >&2
067     return 1
068    fi
069   else
070    echo "Pipe ${pipe} does not exist"
071   fi
072  done
073  return 0
074 }
075
076 listen() {
077  setenv
078  chkpipes   # Do required pipes exist
079  if [ $? -ne 0 ]
080  then
081   return 1
082  fi
083  # If present delete file with last directory used
084  if [ -w /tmp/lastpwd.${rhost} ]
085  then
086   $RM /tmp/lastpwd.${rhost}
087  fi
088  ( while read line
089    do
090     set $line
091     if [ "$1" = "BEGIN_CMD" ]
092     then
093      shift
094      # Run command
095      # Change to last directory
096      if [ -r /tmp/lastpwd.${rhost} ]
097      then
098       cdircmd="cd `$CAT /tmp/lastpwd.${rhost}` ; "
099      else
100       cdircmd=""
101      fi
102      # Redirect output to data stream to be sent
103      echo "BEGIN_CMD_OUT" >$sendpipe
104      echo "${cdircmd} $@" |/bin/bash >$sendpipe
105      echo "END_CMD_OUT" >$sendpipe
106     elif [ "$1" = "END_COMMUNICATION" ]
107     then
108      # Terminate process if END_COMMUNICATION string is received
109      exit 0
110     elif [ "$1" = "BEGIN_FILETRANSFER" ]
111     then
112      # If a file is to be received
113      filename=`$BASENAME $2`
114      tdir=$3
115      echo "Receiving file $filename..."
116      echo "Copying to $tdir..."
117      # Use awk to read lines until the END_FILETRANSFER
118      # string is received
119      $AWK '
120       {
121        if ( $0 == "END_FILETRANSFER" )
122         exit 0
123        else
124         print $0
125       # The received lines are decoded and redirected to the desired target file
126       }' $receivepipe | $PYTHON -c 'import sys,uu; uu.decode(sys.stdin, sys.stdout)' >${tdir}/$filename
127      echo "File $filename transferred"
128     elif [ "$1" = "BEGIN_CMD_OUT" ]
129     then
130      # If BEGIN_CMD_OUT string is received, output all further lines
131      # until END_CMD_OUT is received
132      $AWK '
133       {
134        if ( $0 == "END_CMD_OUT" )
135         exit 0
136        else
137         print $0
138       }' $receivepipe
139     fi
140    done <$receivepipe
141  ) &
142 }
143
144 establish() {
145  setenv
146  chkpipes # Do the required pipes exist
147  # Does the awk file exist
148  if [ ! -r $awkfile ]
149  then
150   echo "Cannot find $awkfile" >&2
151   return 1
152  fi
153  if [ $? -eq 0 ]
154  then
155   ( $TAIL -f $sendpipe | ( $SSH -q ${rhost} "$AWK -v pipe=${rreceivepipe} -f ${awkfile}" ) ) &
156  else
157   # If pipes do not exist, quit output and function
158   echo "Pipes for communication not present" >&2
159   return 1
160  fi
161 }
162
163 killall() {
164  # Send end communication to all processes
165  setenv
166  if [ -w $sendpipe ]
167  then
168   echo "END_COMMUNICATION" >$sendpipe
169   echo "." >$sendpipe
170   $RM -f /tmp/sendpipe_${rhost}.pid
171  fi
172  if [ -w $receivepipe ]
173  then
174   echo "END_COMMUNICATION" >$receivepipe
175   $RM -f /tmp/listener_${rhost}.pid
176  fi
177 }
178
179 rcmd() {
180  setenv
181  # Check whether connection to REMOTEHOST exists
182  chkpipes
183  if [ $? -ne 0 ]
184  then
185   return 1
186  fi
187  # Send parsed line
188  echo "BEGIN_CMD $@ ; { pwd >/tmp/lastpwd.$lhost ;}" >$sendpipe
189  return $?
190 }
191
192 sendfile() {
193  setenv
194  # Check whether Python can be executed
195  if [ -x $PYTHON -a ! -d $PYTHON ]
196  then
197   # If Python is not available, report and quit function
198   if [ $# -lt 1 -o $# -gt 2 ]
199   then
200    echo "usage: sendfile FILE [target directory]"
201    return 1
202   fi
203   # By default, copy file to /var/tmp
204   tdir=${REMOTETEMPDIR:=/var/tmp}
205   if [ $# -eq 1 ]
206   then
207    file=$1
208   else
209    file=$1
210    tdir=$2
211   fi
212   # Register and copy file
213   echo "BEGIN_FILETRANSFER $file $tdir" >$sendpipe
214   cat $file |$PYTHON  -c 'import sys,uu; uu.encode(sys.stdin, sys.stdout)' >$sendpipe
215   echo "END_FILETRANSFER" >$sendpipe
216  else
217   echo "No python executable found. File transfer not possible." >&2
218   return 1
219  fi
220 }
Figure 3: Two computers communicate using pipes and various on-board tools on Linux.

The communication in Figure 3 is broken down into several segments. Essentially, it is all about receiving data via a data stream, processing the data, and sending back a data stream in response. But how do you maintain a continuous data stream between two computers in order to send commands and other information?

« Previous 1 2 3 4 Next »

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy Linux Magazine

SINGLE ISSUES
Print Issues
Digital Issues
 
SUBSCRIPTIONS
Print Subs
Digisubs
 
TABLET & SMARTPHONE APPS
Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

  • Optimizing Shell Scripts

    Shell scripts are often written for simplicity rather than efficiency. A closer look at the processes can lead to easy optimization.

    more »
  • Parallel Bash

    You don't need a heavy numeric mystery to benefit from the wonders of parallel processing. This article describes some simple techniques for parallelizing everyday bash scripts.

    more »
  • Command Line: Data Flow

    Working in the shell has many benefits. Pipelines, redirectors, and chains of commands give users almost infinite options.

    more »
  • Binary Data in Bash

    Bash is known for admin utilities and text manipulation tools, but the venerable command shell included with most Linux systems also has some powerful commands for manipulating binary data.

    more »
  • Apache StreamPipes

    You don't need to be a stream processing expert to create useful custom solutions with Apache StreamPipes. We'll use StreamPipes to build a simple app that calculates when the International Space Station will fly overhead.

    more »
comments powered by Disqus
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters

Support Our Work

Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.

Learn More

News

Tag Cloud

Administration Community Events Hardware Linux Linux Pro Magazine Mobile Programming Security Software Ubuntu Web Development Windows free software open source