Removing Large Files and Log Rolling

261
■ ■ ■
CHAPTER 40
Removing Large Files and
Log Rolling
T
his chapter gives a few tips relating to moving or removing files that are consuming
large amounts of disk space. In some cases you may have a file system that’s filling up
because a process adds copious amounts of entries to a log file. When you run into a large
log file that is found to be the primary cause of a full file system, your first inclination may
be to remove the file to reclaim space. However, this may not work as you might intend.
Before I go into why, let me first talk a little about files and their structure. The file name
is the visible representation of an inode that allows you to access the data in the file. The
inode contains all the important information about a file, such as its ownership, permis-
sions, modification times, and other items of interest. An inode also holds information
about the location of the data on the hard disk. A file name is a user-friendly way of access-
ing an inode, which is represented by a number. You can determine the inode number of
any file by running the command ls -i filename.
Here is an example of a full directory:
$ ls -i
474322 37l5152.pdf 214111 rsync.tar.gz
474633 770tref.pdf 215939 yum-2.0.5-1.fd.fr.noarch.rpm
215944 openbox-3.1-1.i386.rpm
Note that the inode number precedes each of the five files listed here. As just men-
tioned, part of the data contained in an inode is a pointer to the data on the physical disk.
If the file has been opened by a process for writing, the process writes to this location on
the hard disk.
The potential problem with removing a large file to clean up disk space is that if a user
or an administrator removes the file, the inode may still be kept open by a process that is
writing the data, and the disk space will not be returned to the system. The operating sys-
tem won’t realize it should release the disk space for reuse until the process closes the file.

At that time the disk space will definitely be reclaimed.
One way of finding out if a process is keeping a file open is to use the fuser command,
which will display a list of process IDs that are accessing a given file. You also can use
the lsof command to find this information. lsof is designed to list open files and the
262
CHAPTER 40
■
REMOVING LARGE FILES AND LOG ROLLING
processes that are accessing them. Once you know the processes that are using the file,
you can stop them before deleting the file. However, you may not be able (or want) to do
this. The other problem with needing to stop a process that is holding a file open is that
doing so may be against existing site policies. The process or application that is holding
the file open might be production-critical and impact business needs if it is halted.
Another way to clean up the space used by a file is to zero out the file by redirecting
/dev/null into the file. This trims the file down to zero bytes while leaving the file itself in
place. The file remains open and accessible to any process that might be using it. How-
ever, the operating system will release the disk space in a timely fashion. Keep in mind
that some processes may keep a file open for writing for a very long time.
Here is a sample of a directory listing, including an offending log file that is consuming
large amounts of disk space. We have a choice of several possible commands that will zero
out the file.
$ ls -l
total 113172
-rw-rw-r-- 1 rbpeters rbpeters 1057862 Jun 7 18:21 37l5152.log
-rw-rw-r-- 1 rbpeters rbpeters 449184 Jun 7 18:21 770tref.log
-rw-rw-r-- 1 rbpeters rbpeters 1104249096 Jun 7 21:22 really_big.log
The following command is one, cp /dev/null really_big.log is another, and echo >
really_big.log is yet another. All of these commands overwrite a large file with nothing.
$ >really_big.log
This is the resulting directory listing after the file has been zeroed out:

$ ls -l
total 1484
-rw-rw-r-- 1 rbpeters rbpeters 1057862 Jun 7 18:21 37l5152.log
-rw-rw-r-- 1 rbpeters rbpeters 449184 Jun 7 18:21 770tref.log
-rw-rw-r-- 1 rbpeters rbpeters 0 Jun 7 21:25 really_big.log
This procedure can be used to rotate log files. For example, you may have an applica-
tion for which you would like to keep one week’s worth of log information. You would
then rotate it to an older version to keep up to one month. After a month, you would delete
the file. One solution includes the logrotate command, which is part of multiple Linux
distributions.
The following is a simple script that copies the existing log file to a backup with a ver-
sion number and then zeroes out the file. During the time the script is being run, any
existing processes that are writing to the file can still access it.
The code first defines the file in question and then checks whether it exists. If it does
exist, execution continues.
CHAPTER 40
■
REMOVING LARGE FILES AND LOG ROLLING
263
#!/bin/sh
LOGFILE=./my.log
if [ ! -f $LOGFILE ]
then
echo "Nothing to do... exiting."
else
You’ve already seen the use of the redirect (>) to zero a file. The main addition to this
idea here is the cp -p command.
cp -p $LOGFILE ${LOGFILE}.0
> $LOGFILE
fi

This command preserves the original file’s ownership and time stamp for the new
copy. Thus the new copy maintains the attributes of the original file and it appears as if
the file had been moved. This is also a useful technique when modifying scripts, configu-
ration files, or any other file for which preserving the original file’s attributes would be
valuable.
265
■ ■ ■
CHAPTER 41
Core Finder
I
n a production environment, it is a good idea to know whether your applications are
dumping core often. It is also considered good housekeeping to know about your core
files so your hard disk won’t fill up with unnecessary files. The small script in this chapter
tracks down and cleans up core files. The script was intended to be run as an hourly cron
job, although you could change the schedule to fit your needs. The job also has its priority
lowered using the nice command so that it won’t interfere with the performance of
regular processes on the machine. The notifications I’ve received from this script have
characterized chronic issues with applications on more than one occasion. Without the
script, I would have never seen the patterns.
This script steps through each of the locally mounted file systems and finds all core
files. It determines the applications that created the core files and moves the core files to
a central location for later examination. The script also logs its actions and cleans up old
saved files.
First we have to set up some straightforward variables.
#!/bin/sh
HOWOLD=30
FSLIST="/ /boot"
UNAME=`uname -n`
DATE=`date +%m%d%I%M`
DATADIR="/usr/local/data/cores"

LOGFILE="$DATADIR/cor_report"
The HOWOLD variable is used as the maximum age, in days, for saved core files. Any files
older than this will be removed.
The FSLIST variable contains the list of file systems that will be checked for core files.
The list will vary from system to system. You could set FSLIST dynamically by using the
df command to determine the locally mounted file systems. The command might look
something like this: FSLIST=`df -l | grep '^/dev' | awk '{print $6}'`, which gathers
the lines starting with /dev and then prints the field containing the file-system name.
The other four variables contain the name of the system that the script is running
on (UNAME), the current date (DATE), the directory to which the core files will be saved
(DATADIR), and the name of the log file (LOGFILE).
266
CHAPTER 41
■
CORE FINDER
Now we determine whether the data directory exists. This is the directory where the
core files will be saved. If it doesn’t exist, you have to create it. The -p option to mkdir adds
any omitted parent directories to the path of the directory being created.
if [ ! -d $DATADIR ]
then
mkdir -p $DATADIR
fi
Then you need to find all previously saved core files that don’t need to be kept around
anymore, and remove them.
find $DATADIR -name \*\.core\.\* -mtime +$HOWOLD -exec rm {} \;
The following is the script’s main loop. It finds all the core files created since the last
time the script was run.
find $FSLIST -mount -type f -name core -print | while read file
do
if [ -s $file ]

then
coretype=`file $file | cut -d\' -f2`
mv $file $DATADIR/$UNAME.core.$coretype.$DATE
echo "$coretype $UNAME $DATADIR/$UNAME.core.$coretype.$DATE" | \
tee -a $LOGFILE | mailx -s "core_finder: $coretype core file \
found on $UNAME" sysadmins
else
rm $file
fi
done
If a core file’s size is zero bytes, we remove it. If it is larger than zero bytes, we use the
file command to determine the application that created the core file. Then the core file
is saved in the archive directory and its name is changed to include the type of core and
the date when the file was found. Then we add an entry to the log file noting the action
that was taken, and send an e-mail notice to the administrators.