Managing File Counts

211
■ ■ ■
CHAPTER 32
Managing File Counts
S
ystem administrators have to manage the computing resources available to their users.
One resource consists of the directory queues for e-mail and print servers. These directo-
ries contain the files waiting to be processed by the server. The number of files in these
directories varies depending on the volume of work on the server. There are times when
no one is using the service and the queue empties out. There are also times when the
server is heavily loaded and the queue will be full. Usually, though, the number of files will
stay within stipulated limits.
Of course, from time to time you will have problems with a queue filling up faster than
the server can process the requests. This might be due to a bubble of extremely heavy load
or because of some type of problem that prevents the server from processing the requests.
You will want to monitor the queue directories so you can take the proper measures when
a problem arises.
The script presented in this chapter is a simple monitor that counts the files in the
specified directories. The script gives you the necessary core functionality; you will, how-
ever, want to modify the directory locations and notification methods for your own site
and needs.
File-Count Monitor
First we define the directories to be monitored. It would be a bit cleaner to place this
definition in a separate file so you don’t have to modify the script to make configuration
changes. However, to keep the script as simple as possible I have not done that. Each
entry in the FILENUM variable is a colon-delimited set of fields. The fields specify the direc-
tory name to monitor, the warning threshold, the maximum limit, and finally the e-mail
address to which to send notifications.
#!/bin/sh
FILENUM="/var/spool/mqueue:50:100:root \
/var/spool/lp:50:100: /:50:100:rbpeters"

212
CHAPTER 32
■
MANAGING FILE COUNTS
Now we start looping through each of the configured directories to check the file
counts. First sed swaps colons with spaces in the configuration entry; then the set com-
mand assigns, to each of the positional parameters $1 through $4, the corresponding
element of the output, which consists of the items in the original colon-separated fields.
Once the fields are assigned to the variables, the script determines the number of files in
the directory.
for monitor in $FILENUM
do
set -- `echo $monitor | sed -e 's/:/ /g'`
file_count=`ls $1 | wc -l`
The script completes by comparing the current file count to the three possible
situations:
Everything is normal. The file count is lower than the warning threshold.
A warning state. The count is between the warning threshold and maximum limit.
An error state. The count is above the maximum limit.
In either the warning or error state, the script will send a notification via e-mail specify-
ing the condition of the directory. There is one inherent problem with the e-mail
notification type. If you’re monitoring a mail queue directory that is filling up, it is likely
that the e-mail notification will not be sent. In this case, you may want to change your
notification method to use something other than e-mail such as an alpha-numeric
pager or phone.
if [ $file_count -lt $2 ]
then
continue
elif [ $file_count -ge $2 -a $file_count -lt $3 ]
then

echo "Warning: File count in $1 is $file_count, should be less \
than $2" | mail -s "Directory file count warning for $1" $4
else
echo "Error: File count in $1 is $file_count, should be less \
than $2" | mail -s "Directory file count error for $1" $4
fi
done
To make this script more useful, you would need to call it with cron or possibly sur-
round the code with an infinite loop that a perpetual process would run to continually
monitor your machine. The script’s main function is carried out in the assignment of the
file_count variable by performing an ls on the target directory and then piping the out-
put to the wc utility. This is one of a number of ways to get queue-status information. Two
CHAPTER 32
■
MANAGING FILE COUNTS
213
other ways of determining the file count in a directory are to use the echo and find com-
mands. Here are some possible solutions:
file_count=`ls $1 | wc -l`
file_count=`find $1 -type f | wc -l`
file_count=`echo $1/* | wc -w`
Note that the wc -l switch has been replaced with -w for the last solution. This is
because the result of the shell’s expansion of the asterisk will appear in the form of a single
line of filenames separated by spaces.
Testing File-Count Methods
The different methods of counting the files in a directory are not all created equal. Some
older operating systems cannot handle large numbers of files when the ls or find com-
mand is used. Sometimes the find-oriented solution will return incorrect results on
directories containing between 12,000 and 15,000 files. Also, I have worked on systems
with file counts of more than one million in a single directory; in these cases ls returns

an “argument list too long” error. I have not yet found a situation in which the echo solu-
tion fails.
On my current Gentoo Linux system, both find and ls seem not to suffer from these
earlier limitations, and I have used both to successfully count files in a single directory
numbering well above 800,000 files. While testing, I noticed that there is a speed differ-
ence between the three solutions.
The find solution is by far the fastest. It took about .4 seconds with just over 300,000
files in a directory. The second fastest is echo *. It took about 1.5 seconds. This measure is
most accurate if you work in the directory whose files are to be counted. If a full path is
supplied with the asterisk (e.g., /some/full/path/*), the command takes longer because
each element in the shell expansion contains the full path and all that extra text needs to
be processed. The slowest in my tests was ls. It took about 4 seconds to complete. Both
find and ls seemed to take similar amounts of time regardless of whether the full path is
supplied.
Realistically speaking, with a current operating system any of these solutions should
work fine. It is not very likely that you will deal with this extreme number of files in a single
directory, so the performance differences between the solutions will almost surely be
insignificant.