Indirect Reference Variables

43
■ ■ ■
CHAPTER 7
Indirect Reference Variables
I
t is possible to set a normal (direct) variable in three ways:
• Directly, by assigning it a value
• By storing the output of a command
• By storing the results of some type of calculation
In all of these cases, although you might not know a variable’s value in advance of any
given point, you do know the name of the variable that is to receive the value.
For example, AGE=36 is a direct variable assignment. The value of 36 might change at a
later stage and it may also have some calculation applied to it, but the variable name AGE
will not change.
In some cases, however, you may need the ability to generate variable names on the fly.
You may not know the number or names of variables you are going to need at the time of
execution. These could be referred to as variable variables, or variable variable names.
This chapter shows how to create and use such indirect variables.
Log File Monitoring with Indirect Variables
The following example monitors log files and notifies the user when specified string val-
ues show up in the file. The script is designed to run continuously while keeping track
of where it last left off in the file; thus, it knows where to start the next time it looks. The
configuration value at the beginning of the script points to the log files it needs to watch
and the string values to be tracked.
This configuration value can consist of many entries, each specifying strings the script
needs to watch for in all the tracked files. There may be several entries specifying different
strings for the same log file. Our example script is configured in this way.
#!/bin/sh
#set -x
debug=0
DELAY=120

44
CHAPTER 7
■
INDIRECT REFERENCE VARIABLES
LOGCHKS="/var/log/messages:authentication%20failure:\
rbpeters:warn /var/log/messages:recv%20failure::error"
This LOGCHKS variable configures which log files will be monitored. You could create a
separate configuration file to hold this information, but to keep things straightforward,
I’ve included the configuration variables as part of the script.
Each entry consists of four fields separated by colons (:). Here are the meanings of the
four fields:
First: The full path to the log file being watched; nothing special here.
Second: The string or strings to watch for. Multiple strings can be specified in this
field by separating each with a pipe character, |. Because the entries in the configu-
ration string are themselves separated by spaces, you can’t have a space within the
watch string. If you want to watch for a phrase that includes spaces, the spaces need
to be replaced with %20, as shown in the LOGCHKS variable assignment.
Third: Exception strings that are to be ignored. In our example, the script will watch for
any authentication-failure messages in the /var/log/messages file, with the exception
of those containing the string rbpeters. The subfields of this field, like those of the sec-
ond, are pipe-separated, and any spaces should be replaced with %20.
Fourth: Notification level. The two values here are warn and error. If the notification is
not an error, the script defaults to warn. The notification strings are left undefined
because they will be determined by the user implementing them.
The Main Monitor Loop
To begin, start the infinite loop in which the script will be running. Then you can look at
each configuration entry.
while :
do
entry_count=0

for LOGCHK in `echo $LOGCHKS`
do
Now all values of the configured entry have to be parsed and then assigned to a direct
variable. The script also replaces %20 characters with real spaces.
logfile=`echo $LOGCHK | cut -d: -f1`
strings="`echo $LOGCHK | cut -d: -f2`
strings="`echo $strings | sed -e \"s/%20/ /g\"`"
exceptions=`echo $LOGCHK | cut -d: -f3`
exceptions="`echo $exceptions | sed -e \"s/%20/ /g\"`"
notify=`echo $LOGCHK | cut -d: -f4`
CHAPTER 7
■
INDIRECT REFERENCE VARIABLES
45
entry_count represents the number assigned to the specific entry in the configuration
string. If two log files are configured to be watched, their entry_count values will be 1 and
2, respectively. This variable will be used later to create a new variable on the fly.
entry_count=`expr $entry_count + 1`
The suffix is nothing more than the name of the log file. Slashes (/) and dots (.) are
replaced with underscores (_). You’ll also use the suffix value later to build indirect vari-
able names. The combination of this suffix and the entry_count allows us to create
unique variable names specific to the log files that the script is working with.
suffix=`echo $logfile | sed -e s/\\\//_/g`
suffix=`echo $suffix | sed -e s/\\\./_/g`
Next comes the first reference to an indirect variable. The shell evaluates a normal line
of code within a script so that any variables are replaced with their values before any com-
parisons or calculations are performed.
The eval command is used when you want the shell to perform an additional evalua-
tion prior to the normal evaluation. This allows you to construct names for new variables
using the values of existing variables.

In this case, the line in the script is as follows:
if [ "`eval echo '$COUNT'${suffix}_$entry_count`" = "" ]
After the first explicit evaluation (in the embedded eval), to the shell, the line would
look like this:
if [ "$COUNT_var_log_messages_1" = "" ]
Then, when the shell evaluates the line normally, it sees a direct variable, although we
know that the direct variable has been conjured up by a prior evaluation.
Now back to the code logic: the code then checks whether the log file’s base line count
is null. The only time it will be null is the first time the line count is tested. If this is the
case, the base line count is set to the current file length (in number of lines).
if [ "`eval echo '$COUNT'${suffix}_$entry_count`" = "" ]
then
eval BASE${suffix}_$entry_count=`wc -l $logfile | awk '{ print $1 }'`
fi
Resetting the value of the line count the first time the loop is executed is a safe way of
not having the monitor find any previous string entries being watched for in the log file.
We don’t want to see strings that were there prior to the monitor ever running.
Now the line count of the log file is set. The line count is different from the base count.
Let us assume the script starts up to find 10 lines in the log file. The log’s base is then set
to 10. The script sets the line count of the file to 10, sees there is no difference between the
line count and the base count, and completes running this script segment.
eval COUNT${suffix}_$entry_count=`wc -l $logfile | awk '{ print $1 }'`
46
CHAPTER 7
■
INDIRECT REFERENCE VARIABLES
The script then sleeps for the number of seconds specified at the beginning of the
script and wakes up again. Suppose that it now finds 13 lines in the file. The difference
between the file’s line count and the base count is used to detect new log entries.
The following code checks to see whether the log file has grown. If it has, we use the

tail command to check the newly added lines inside the log file for the desired strings.
The script then resets the file’s base count to whatever the current line count happens to
be so you don’t look at lines that have already been checked.
if [ `eval echo '$COUNT'${suffix}_$entry_count` -gt
`eval echo '$BASE'${suffix}_$entry_count` ]
then
LINES=`eval expr '$COUNT'${suffix}_$entry_count - '$BASE'${suffix}_$entry_count`
eval \ BASE${suffix}_$entry_count='$COUNT'${suffix}_$entry_count
if [ "$exceptions" != "" ]
then
MSGS=`tail -$LINES $logfile | egrep -i "\"$strings\"" | egrep -iv "$exceptions"`
test $debug -gt 0 && echo "MSGS is $MSGS"
else
MSGS=`tail -$LINES $logfile | egrep -i "$strings"`
test $debug -gt 0 && echo "MSGS is $MSGS"
fi
If any messages found in the log file match what you’re looking for, the script should
send a notification. As mentioned earlier, there are two possible forms of notification: a
warning and an error. In the environments I’ve worked with, both notification methods
would normally result in an e-mail with a warning status message, or in an alphanumeric
page if it were a more critical message. You must decide how to configure the notifica-
tions, so the code here simply echoes a message depending on the notification type.
if [ ! -z "$MSGS" ]
then
if [ "$notify" != "error" ]
then
echo Send a warning notification...
else
echo Send an error notification...
fi

fi
If the file’s line count is less than the base value (the value from the previous loop
through the code), you need to reset the base value.
elif [ `eval echo '$COUNT'${suffix}_$entry_count` -lt
`eval echo '$BASE'${suffix}_$entry_count` ]
then
# This resets the tracked size of the
# log if the log size gets smaller
eval BASE${suffix}_$entry_count='$COUNT'${suffix}_$entry_count
CHAPTER 7
■
INDIRECT REFERENCE VARIABLES
47
if [ "$exceptions" != "" ]
then
MSGS=`cat $logfile | egrep -i "\"$strings\"" | egrep -iv "$exceptions"`
test $debug -gt 0 && echo "MSGS is $MSGS"
else
MSGS=`cat $logfile | egrep -i "$strings"`
test $debug -gt 0 && echo "MSGS is $MSGS"
fi
if [ ! -z "$MSGS" ]
then
if [ "$notify" != "error" ]
then
echo Send a warning notification...
else
echo Send an error notification...
fi
fi

A likely scenario for this occurrence is when the log file is trimmed to a preset size to
save disk space, such as with the logrotate utility. If this is the case, we want to check the
whole file for the strings we’re looking for. If we don’t, we might miss something.
If there is no change in the file size, nothing needs to be done. We just complete the
loop, and go back and repeat the same operations for all the other log files in the configu-
ration string. Finally, the script should sleep for the specified amount of time before
starting over again.
else
test $debug -gt 0 && echo "No change in size of $logfile"
fi
done
sleep $DELAY
done