Automation and Observation

Systems Administration Chapter 15: Automation and Observation
Page 359
Chapter
Automation and Observation

Introduction
Setting up a machine is one part of Systems Administration. Another somewhat more
important part is keeping that machine going. Central to achieving these aims are the
two activities we look at in this chapter (there are more):
· Automation
Any tasks which occur more than once must be automated. The primary tool on
UNIX systems for achieving this is shell programs. This chapter looks at the use
of
cron
(the Linux scheduler) for automatically scheduling shell programs and
other tasks.
· Observation
People will do things to your computer. Some of them will do nasty things.
Observation is the act of keeping an eye on your machine so you know there is
something wrong with it before the users complain about something not working.
We look at observation from two perspectives: historical and current. Historical
observation tells you what has happened on your system. Current observation tells
you what is happening now.
Other resources
Other resources which discuss similar topics include:
· LAME
A section called Automatic tasks with Cron and Crontab files.
· USAIL
A section on automating tasks with Cron.

Automation and cron

A number of the responsibilities of a Systems Administrator are automated tasks that
must be carried out at the regular times every day, week or hour. Examples include,
early every morning freeing up disk space by deleting entries in the
/tmp
directory,
performing backups every night or compressing and archiving log files.
Most of these responsibilities require no human interaction other than to start the
command. Rather than have the Systems Administrator start these jobs manually,
UNIX provides a mechanism that will automatically carry out certain tasks at set
times. This mechanism relies on the
cron
system.
For example, the mirror of the Linux Documentation Project (LDP) on the Systems
Administration website is kept up to date with a
cron
job (a task scheduled with
cron
). This particular
cron
job, run every Sunday night, connects to the central LDP
site and transfers any updated data.
Systems Administration Chapter 15: Automation and Observation
Page 360
Components of
cron

The
cron
system consists of the following three components:
·

crontab
(the
cron
configuration) files
These are the files which tell the
cron
system which tasks to perform and when.
· the
crontab
command
This is the command used to modify the
crontab
files. Even though the
crontab

files are text files, they should not be edited using a text editor.
· the daemon,
crond

The
cron
daemon is responsible for reading the
crontab
file and then performing
the required tasks at the specified times. The
cron
daemon is started by a system
startup file.
crontab
format

crontab
files are text files with each line consisting of six fields separated by spaces.
The first five fields specify when to carry out the command, and the sixth field
specifies the command. Table 15.1 outlines the purpose of each of the fields.
Field Purpose
minute

Minute of the hour,
00
to
59

hour

Hour of the day,
00
to
24
(military time)
day

Day of the month,
1
to
31

month

Month of the year,
1

to
12

weekday

Day of the week. Linux uses three letter abbreviations,
sun
,
mon
,
tue
,....
command

The actual command to execute
Table 15.1
crontab
fields
Comments can be used and are indicated using the
#
symbol just as with shell
programs. Anything that appears after a
#
symbol until the end of that line is
considered a comment and is ignored by
crond
.
The five time fields can also use any one of the following formats:
· a
*

(matches all possible values)
· a single integer (matches that exact value)
· a list of integers separated by commas (no spaces) used to match any one of the
values
· two integers separated by a dash (a range) used to match any value within the
range

For example
Some example
crontab
entries include (all but the first two examples are taken from
the Linux manual page for
crontab
):
0 * * * * echo Cuckoo Cuckoo > /dev/console 2>&1
Every hour (when
minutes=0
), display
Cuckoo

Cuckoo
on the system console.
30 9-17 * 1 sun,wed,sat echo `date` >> /date.file 2>&1
At half past the hour, between 9 and 5, for every day of January which is a Sunday,
Wednesday or Saturday, append the date to the file
date.file

0 */2 * * * date
Every two hours at the top of the hour, run the
date

command.
Systems Administration Chapter 15: Automation and Observation
Page 361
0 23-7/2,8 * * * date
Every two hours from 11p.m. to 7am, and at 8am, run the
date
command.
0 11 4 * mon-wed date
At 11:00 am on the 4th and on every Monday, Tuesday and Wednesday, run the
date

command.
0 4 1 jan * date
At 4:00 am on January 1st, run the
date
command.
0 4 1 jan * date >> /var/log/messages 2>&1
Once an hour, all output appended to log file.
Output
When commands are executed by the
crond
daemon, there is no terminal associated
with the process. This means that standard output and standard error, which are
usually set the terminal, must be redirected somewhere else. In this case the output is
emailed to the person whose
crontab
file the command appears in. It is possible to
use I/O redirection to redirect the output of the commands to files. Some of the
examples above use output redirection to send the output of the commands to a log
file.

Exercises
15.1.
Write
crontab
entries for the following:
- run the program
date
every minute of every day and send the output to a
file called
date.log

- remove all the contents of the directory
/tmp
at 5:00am every morning
- execute a shell script
/root/weekly.job
every Wednesday
- run the program
/root/summary
at 3pm, 6pm and 9 pm for the first five
days of a month
Creating
crontab
files
crontab
files should not be modified using an editor. Instead, they should be created
and modified using the
crontab
command. Refer for the manual page for
crontab

for more information. The following are two of the basic methods for using the
command:
· crontab [file]
·
crontab [-e | -r | -l ] [username]


The first method above is used to replace an existing
crontab
file with the contents of
standard input or the specified file. The second method makes use of one of the
following command line options:
·
-e

Allows the user to edit the
crontab
file using an editor (the command will
perform some additional actions to make it safe to do so).
·
-r

Remove the user's
crontab
file.
·
-l

Display the user's

crontab
file onto standard output.

Systems Administration Chapter 15: Automation and Observation
Page 362
By default all actions are carried out on the user's own
crontab
file. Only the root
user can specify another username and modify that user's
crontab
file.
Exercises
15.2.
Using the
crontab
command to add the following to your
crontab
file
and observe what happens:
- Run the program
date
every minute of every day and send the output to a
file called
date.log

Current Observation
A part of the day-to-day operation of a system is keeping an eye on the systems’
current state. This section introduces a number of commands and tools that can be
used to examine the current state of the system.
The tools are divided into two sections based on what they observe. The sections are:

· disk and file system observation
The commands
du
and
df
.
· process observation and manipulation
The commands
ps
,
kill
,
nice
and
top
.
df

df
(disk free) summarises that amount of free disk space. By default,
df
will display
the following information for all mounted file systems:
· total number of disk blocks
· number of disk blocks used
· number available
· percentage of disk blocks used
· where the file system is mounted

df

also has an option
-i
to display Inode usage rather than disk block usage. What an
Inode is will be explained in a later chapter. Simply, every file that is created must
have an Inode. If all the Inodes are used, you can't create any more files, even if you
have disk space available.
The
-T
option will cause
df
to display each file system’s type.
Exercises
15.3.
Use the
df
command to answer the following questions:
- how many partitions do you have mounted
- how much disk space do you have left on your Linux partition
- how many more files can you create on your Linux partition
du

The
du
command (disk usage) is used to discover the amount of disk space used by a
file or directory. By default,
du
reports file size as a number of 1Kb blocks. There are
options to modify the command so it reports size in bytes (
-b
) or kilobytes (

-k
).
If you use
du
on a directory, it will report back the size of each file and directory
within it and recursively descend down any subdirectories. The
-s
switch is used to
produce the total amount of disk space used by the contents of a directory.
Systems Administration Chapter 15: Automation and Observation
Page 363
There are other options that allow you to modify the operation of
du
with respect to
partitions and links. For this information, refer to the
du
manual page.

Exercises
15.4.
Use the
du
command to answer the following questions:
- how many blocks does the
/etc/passwd
file use?
- how large (in bytes) is the
/etc/passwd
file?
- how much disk space is used by the

/etc/
directory?
- how much disk space is used by the
/usr
directory?
System Status
Table 15.2 summarises some of the commands that can be used to examine the
current state of your machine. Some of the information they display includes:
· amount of free and used memory
· the amount of time the system has been up (available)
· the load average of the system
Load average is the number of processes ready to be run, and is used to give some
idea of how busy your system is.
· the number of processes, and amount of resources they are consuming

Some of the commands are explained below. For those that aren't, use your system's
manual pages to discover more.
Command Purpose
free
Display the amount of free and used memory
uptime
How long has the system been running and what the current load
average is
ps/pstree
One-off snap shot of the current processes
top
Continual listing of current processes
uname
Display system information including the hostname, operating system
and version, and current date and time

gtop
The Gnome system monitor, a GUI which provides a view of running
processes, memory and file system usage (see chapter 5)
Table 15.2
System status commands
ps
The
ps
command (process state) displays a list of information about the process that
were running at the time the
ps
command was executed.
ps
has a number of options that modify what information it displays. Table 15.3 lists
some of the more useful or interesting options that the Linux version of
ps
supports.
Table 15.4 explains the headings used by
ps
for the columns it produces.
For more information on the
ps
command, refer to the manual page.
Systems Administration Chapter 15: Automation and Observation
Page 364

Option Purpose
l
Long format
u

Displays username (rather than UID) and the start time of the process
m
Display process memory information
a
Display processes owned by other users (by default
ps
only shows
your processes)
x
Shows processes that aren't controlled by a terminal
f
Use a tree format to show parent/child relationships between processes
w
Don't truncate lines to fit on screen
Table 15.3
ps
options
Field Purpose
NI

The
nice
value
SIZE

Memory size of the process’ code, data and stack
RSS

Kilobytes of the program in memory (the resident set size)
STAT

The status of the process (
R
-runnable,
S
-sleeping,
D
-uninterruptable sleep,
T
-stopped,
Z
-zombie)
TTY

The controlling terminal
Table 15.4
ps
fields
Exercises
15.5.
Use the
ps
command to answer the following questions:
- how many processes do you currently own?
- how many processes are running on your system?
- how much RAM does the
ps
command use?
- what is the current running process?
top

ps
provides a one-off snapshot of the processes on your system. For an ongoing look
at the processes, Linux generally comes with the
top
command. This command also
displays a collection of other information about the state of your system including:
· uptime
The amount of time the system has been up.
· the load average
· the total number of processes
· percentage of CPU time in user and system mode
· memory usage statistics
· statistics on swap memory usage

Refer to the manual page for
top
for more information.
top
is not a standard UNIX command, however it is generally portable and available
for most platforms.
top
displays the process on your system ranked in order from the most CPU intensive
down, and updates that display at regular intervals. It also provides an interface by
which you can manipulate the
nice
value and send processes signals.