Piping Input to read

65
■ ■ ■
CHAPTER 10
Piping Input to read
T
his chapter deals with a gotcha that I came across while porting a script from ksh to
bash. It was a gotcha only because at the time I wasn’t aware of a fairly crucial difference
in the behavior of the two shells. In both pdksh and bash, the last command of a pipeline is
performed in a subshell. This means that a variable assigned within the subshell is not
available to the parent shell. In ksh, the last command of a pipeline is executed in the orig-
inal shell.
This isn’t an issue when using the pipe to set a variable, but if the result of a pipe is sent
to a loop structure that then populates variables you will use later, that is more of a prob-
lem. Once the loop completes, the variables you were going to rely on don’t exist.
Included here are a few of examples of code that you might expect to work, but they
actually don’t. I also include some workarounds that will perform the intended tasks.
The following is the part of the code that had problems when I ported it. It was used to
process a file of extended output one line at a time. To perform this task in ksh, I would use
the following:
cat somefile | while read line
do
# Process the $line variable in some form.
if [ "`echo $line | awk '{print $3}'`" = "somevalue" ]
then
all="$all $line"
fi
done
If everything within the loop is self-contained and none of the variables in it are
accessed outside the loop, this will work fine. However, the bash code parsed each line
in the output of the piped command, and populated some variables based on that output.
Once the loop completed, I wanted to access those values ($all in this example) for other

purposes, and found that they were undefined.
The following code is the first workaround that I found to overcome the problem.
Unfortunately it isn’t quite as elegant or intuitive as the original code because it uses a
temporary file. To keep the code clean I try to avoid using temporary files, but in this case
I had no choice.
66
CHAPTER 10
■
PIPING INPUT TO READ
while read line
do
# Process the $line variable in some form.
if [ "`echo $line | awk '{print $3}'`" = "somevalue" ]
then
all="$all $line"
fi
done < somefile
First the data originally piped to the while read loop is sent to a temporary file. The file is
then redirected into the back end of the loop. This functions the same way as the original
code, but allows the variables populated within the loop to remain usable once the loop
completes. Chapter 8 offers another example of this technique.
The following is a modified form of the previous example:
THE_INPUT=`ps -ef`
while read line
do
# Process the $line variable in some form.
if [ "`echo $line | awk '{print $3}'`" = "somevalue" ]
then
all="$all $line"
fi

done <<EOF
$THE_INPUT
EOF
This slight modification of the earlier example eliminates the need for a temporary file.
Instead of redirecting a file into the back of the loop, we start a here-document and feed it
the data we want to process through the loop. A here-document is where the shell reads
input from the current source until it reaches the matching tag alone on a single line, in
this case EOF. This solution works in the same way as a real file with both bash and pdksh.
The following sections show four methods for reading input one line at a time. With
each method, I explain what variables are available within the code for each of the four
shells (bash, ksh, pdksh, and Bourne sh).
Line-by-Line Option 1
The original method of piping input to a read loop looks like this:
ps -ef | while read firstvar
do
echo firstvar within the loop: $firstvar
secondvar=$firstvar
echo secondvar within the loop: $secondvar
done
CHAPTER 10
■
PIPING INPUT TO READ
67
echo firstvar outside the loop: $firstvar
echo secondvar outside the loop: $secondvar
KornShell (ksh): Both firstvar and secondvar are available within the loop. Only
secondvar is available outside the loop. This is useful because even though you can’t
use the original read variable, you can assign it to some other variable, which is then
available when the loop completes.
Bash (bash): Both firstvar and secondvar are available within the loop. Neither

firtvar nor secondvar is available after the loop completes.
Public Domain Korn Shell (pdksh): Both firstvar and secondvar are available within
the loop. Neither firstvar nor secondvar is available after the loop completes.
Bourne (sh): Both firstvar and secondvar are available within the loop. Neither
firstvar nor secondvar is available after the loop completes.
Line-by-Line Option 2
This is the workaround option I discussed originally. The input will be sent to a temporary
file and then redirected to the back of the loop.
ps -ef > /tmp/testfile
while read firstvar
do
echo firstvar within the loop: $firstvar
secondvar=$firstvar
echo secondvar within the loop: $secondvar
done < /tmp/testfile
echo firstvar outside the loop: $firstvar
echo secondvar outside the loop: $secondvar
KornShell (ksh): Both firstvar and secondvar are available within the loop. Only
secondvar is available outside the loop.
Bash (bash): Both firstvar and secondvar are available within the loop. Only secondvar
is available outside the loop. This version now performs in the same manner as the ksh
version.
Public Domain Korn Shell (pdksh): Both firstvar and secondvar are available within
the loop. Only secondvar is available outside the loop.
Bourne (sh): Both firstvar and secondvar are available within the loop. Neither vari-
able is available outside the loop.
68
CHAPTER 10
■
PIPING INPUT TO READ

Line-by-Line Option 3
This is the here-document workaround option where we remove the need for a temporary
file. This functions in the same way as Option 2.
the_input=`ps -ef`
while read firstvar
do
echo firstvar within the loop: $firstvar
secondvar=$firstvar
echo secondvar within the loop: $secondvar
done <<EOF
$the_input
EOF
echo firstvar outside the loop: $firstvar
echo secondvar outside the loop: $secondvar
KornShell (ksh): Both firstvar and secondvar are available within the loop. Only
secondvar is available outside the loop.
Bash (bash): Both firstvar and secondvar are available within the loop. Only secondvar
is available outside the loop. This version now performs in the same manner as the ksh
version.
Public Domain Korn Shell (pdksh): Both firstvar and secondvar are available within
the loop. Only secondvar is available outside the loop.
Bourne (sh): Both firstvar and secondvar are available within the loop. Neither vari-
able is available outside the loop.
Line-by-Line Option 4
This last option removes the pipe (|) from the loop and processes an input file manually.
If you have only the Bourne shell at your disposal, this is your only option, and the script
will be somewhat slower. With this option, all set variables from both inside and outside
the loop will be available following loop completion. This option is valid for all the shells
I’ve mentioned.
ps -ef > /tmp/testfile

filecount=`wc -l /tmp/testfile`
count=0
while [ $count -le $filecount ]
CHAPTER 10
■
PIPING INPUT TO READ
69
do
firstvar=`tail +$count /tmp/testfile | head -1`
echo firstvar within the loop: $firstvar
secondvar=$firstvar
echo secondvar within the loop: $secondvar
count=`echo $count+1 | bc`
done < /tmp/testfile
echo firstvar outside the loop: $firstvar
echo secondvar outside the loop: $secondvar
The following tables summarize all of the previously discussed scenarios. Table 10-1
displays the availability of the variable that is initially set in the loop (firstvar). Note that
in all shells except the manual loop method used by the Bourne shell, this variable is
unavailable for use following the loop’s completion.
Table 10-1. Availability of Variables That Are Initially Set in a Loop, After Loop Completion
Table 10-2 displays availability of variables that are assigned within the loop
(secondvar) once the loop has completed. These variables can have values assigned
to them from the initial variable (firstvar), since that variable is accessible within
the loop or from any other assignment inside the loop.
Table 10-2. Availability of Variables That Are Set Within a Loop, After Loop Completion
ksh bash pdksh Bourne
Opt. 1: Pipe to
while read
No No No No

Opt. 2: Redirected file to back of loop No No No No
Opt. 3: Redirected here-document to back of loop No No No No
Opt. 4: Manual iteration through loop Yes Yes Yes Yes
ksh bash pdksh Bourne
Opt. 1: Pipe to
while read
Yes No No No
Opt. 2: Redirected file to back of loop Yes Yes Yes No
Opt. 3: Redirected here-document to back of loop Yes Yes Yes No
Opt. 4: Manual iteration through loop Yes Yes Yes Yes