The Practice of System and Network Administration Second Edition phần 10 pdf

906 Chapter 36 Firing System Administrators
36.2.2 System File Changes
If someone suspects that he may be fired, he may create a back door—a secret
way to get into the system—or plant a logic bomb, or software that causes
damage once he has gone. Ideally, you can take a snapshot of all software
before the person becomes suspicious and compare it to the running system
on a regular basis. However, that is time consuming, requires a lot of storage,
and would easily tip off the person that something is about to happen.
However, no suspicions can be raised if such a thing is always done.
Programs that checksum system filesand report changes are commonly found.
The earliest to achieve popularity is named Tripwire. If this process is an
automated system that is used regularly to notice external intruders, system
failures, or other problems, it will be much easier to use it without raising
suspicion. However, care must be taken to make sure that that person being
fired doesn’t update the database so that his changes aren’t noticed.
Such a system is an excellent measure to detect any kind of intrusions.
However, it is time consuming to process all the false-positives. The issue
becomes scaling it to too many machines.
36.3 Conclusion
Firing SAs isn’t fun or easy, but sometimes it has to happen. The basics are
very simple. The most important rule is to follow the policies of your HR de-
partment. HR personnel are the experts, and you are supporting their process.
Three tiers of access must be removed: physical access, remote access, and
service access. The primary benefit of the three-tier model is that it provides
a structured, rather than ad hoc, approach and is tolerant of mistakes made
at any one level. Architectures that seek to minimize the number of access
databases and a well-maintained inventory ease the process considerably.
In creating a checklist for all the manners of access to be disabled, one
might begin with the new-hire procedure as a starting point: Whatever is
done for a new hire must be undone for a termination. Although no checklist
is complete, we have assembled several checklists of things to disable in the

event of termination:
•
Physical access. Change combination locks, all applicable safe combi-
nations, and locks on doors with keys, even if they are returned. Re-
move access for all buildings: for example, remote locations, shacks, and
utility buildings.
36.3 Conclusion 907
•
Property surrender. Have the ex-employee turn in keys, card-keys,
badges, HHAs, PDAs, and any company-owned equipment at home.
•
Remote access. Modem pools, ISDN pool, VPN servers, in-bound net-
work access—that is,
ssh, telnet, rlogin—cable modem access, xDSL
X.25 access.
•
Service access. Remove access from database servers, NIS domains,
NT domains, superuser access IDs, Netnews IDs, password files, and
RADIUS servers.
The icing is a set of design and operational factors that better prepare
a site for these unlikely but important tasks. The fewer the administrative
databases, the easier the task will be, but if they are all tied to a single au-
thentication database, the entire process becomes much simpler. Regularly
maintained file checksum histories provide a way to detect and prevent back
doors and logic bombs.
Dividing the process into HR policy and physical, remote, and service
access brings clarity to the process. The process can be explained easily. The
staff can be divided into a physical team, a remote team, and a service team.
Each team can then work with complete focus because it has only one task.
This process works best when one can leverage the infrastructure that

should be in any system. A solid security infrastructure keeps the wrong
people out. Having a single (or few) administrative databases, such as a well-
implemented HHA architecture, makes disabling all access from a central
place a snap. Properly documented environments and well-maintained in-
ventory improve one’s ability to disable all access quickly. Routine Tripwire
runs and system-monitoring processes are some of the automation that may
already be in place at a site. The better the infrastructure is, the easier this
process becomes.
The process described in this chapter handles the extreme case of ter-
minating an SA but is also a useful model to consider when anyone leaves
a company, simply leaves your domain of support, or changes jobs within
a company and should no longer have privileged access to the systems she
previously administered. We don’t cover those topics directly. We felt that it
would be more interesting to cover one extreme case and leave the others as
an exercise to the reader.
Our discussion of this topic has been restricted to the technical side of the
process. The nontechnical side, the human side, is equally important. You are
changing this person’s life in a very profound way. The person has bills to pay,
908 Chapter 36 Firing System Administrators
a family to support, and a life to live. Corporate policies range from “get them
out the door immediately” to “we’re laying you off in six months.” There are
potential problems with both, but from our point of view, the latter not only
works best but also shows trust and respect. This issue is not so much for the
benefit of the person being laid off as for the benefit of those remaining.
Exercises
1. When someone is fired, does HR know whom to contact in the IT orga-
nization to have the person’s access disabled?
2. In your current environment, what must be disabled if you were to be
fired? Outside of checking individual hosts for local accounts, how many
individual administrative systems did you have to touch?

3. What improvements to your system could make it easier to disable your
access when you are fired?
4. A system like Tripwire causes periodic points of filesystem I/O. How does
that affect the planning and deployment of such a system? How is this
different for a file server, an e-commerce server, and a database server?
Epilogue
We began this book asking for a concise definition of system administration.
Now we’re no closer to an answer. If anything, we’ve broadened the defi-
nition. Rather than building a crisper definition of system administration,
we’ve discussed customer support, repair, operations, architecture definition,
deployment, disaster planning, and even management skills. System admin-
istration is an extremely broad field, and no simple definition can cover it all.
We hope that you’ve learned a lot from reading this book. We’ve certainly
learned a lot by writing it. Having to put into words things that had become
second nature has forced us to think hard about everything we do, every
habit we’ve developed. The peer-review process stands us naked in front of
our mentors and comrades to receive criticism of our fundamental beliefs.
We’re better for writing this book, and we hope you are better for reading it.
We hope that some day, you write a book and enjoy the same exhilaration.
The most exciting part of this book has been to record, in such a
permanent form, the rants and anecdotes that we have accumulated over our
careers. We respond to certain technical and nontechnical issues by getting
on our soapboxes to expound our opinions. These monologues are refined
every time we repeat them, until we find ourselves repeating them word for
word, over and over again. We can honestly say that this book includes every
tub-thumping rant we authors blurt out with Pavlovian predictability. With a
little bit of luck, these rants will stand the test of time. This book also captures
every useful anecdote in our library of experience. Each anecdote teaches an
important lesson or two. We can rest assured that these anecdotes will not be
lost, and we can safely look forward to all the new anecdotes we will accrue

in the future.
System administration is a culture. Every culture has its anecdotes, myths,
and stories. It is how we pass our history to new generations and propagate
909
910 Epilogue
the lessons and values that are important to us. We learn best from hearing
our culture’s stories and anecdotes. We enrich the culture every time we share
a new one.
We’d like to share with you one final anecdote.
A Concise Definition
A facility had several researchers from a variety of universities visiting for the summer.
That autumn, after they left, the SAs had to decommission their computers and clean the
large room they had been sharing. The SAs found a scrap of paper that had been taped
near the phone. It simply said, “Makes things work,” followed by the phone number of
the SAs.
It was absolutely the highest compliment they had ever received.
Appendixes
This page intentionally left blank
Appendix A
The Many Roles of a System
Administrator
This appendix is heavy on philosophy. If that turns you off, you can skip it,
but we think that it will help you think about your place in the universe or at
least your role within your company, organization, or SA team. Examining
your own role within an organization helps you focus, which helps you do
a better job. It can give you a long-term perspective on your career, which
can help you make the big career decisions necessary for having a happy and
successful life.
This can also give your organization a framework for thinking about
what roles they want you to play. Each of these roles in some way affects

your organization. This is by no means a complete list; however, it is a very
good starting point. You should use this list to consider what roles are missing
in your organization and perhaps to start on a quest to fill them.
It is interesting to think about which and how many of these roles you
are asked to play as your career moves forward. The list can help you plan
your career. Some entry-level SAs are asked to play single roles and grow into
more roles as they gain experience. Sometimes, SAs start out flooded with
many roles and specialize as time goes on.
A small site may require its single SA to take on many roles. As the organi-
zation grows, certain roles can be transferred to newly hired SAs. Sometimes,
you discover that you don’t enjoy a particular role and look to avoid it when
you change jobs. Thinking about these roles may also help guide your career
with respect to what kinds of companies you decide to work for: Small com-
panies tend to require people to fill multiple roles, larger companies tend to
require people to specialize, and megacorporations have people so special-
ized that it can seem bizarre to outsiders. Technology companies respect and
913
914 Appendix A The Many Roles of a System Administrator
reward those who play the role of pushing for new technology, whereas other
companies often discourage too much change.
A.1 Common Positive Roles
Some roles within a company are more critical than others; some are good
and some are bad. Here we list many common roles, the value they provide
to the company, how those people derive satisfaction from the job, and what
customers tend to expect from them.
A.1.1 The Installer
Some people view an SA as the person who installs “stuff.” This is the role
that customers see most often, and so is most often associated with the career
of system administration. The customer rarely sees the other, possibly more
critical, positions, such as the people who design the infrastructure.

The value to the company of Installers is their ability to follow through
and see that the job gets done. They are often the final and most critical link
in the deployment chain.
When installation is being done on a large scale, the item that is being in-
stalled is usually preconfigured at some central location. Installers are trained
on the specific situations they are expected to see and have a second-tier re-
source to call on if they come across an unexpected situation. In that case, the
kind of person who makes a good Installer is one who enjoys meeting and
helping the customers and gets satisfaction from doing the same task well
many times over. On the other hand, in smaller deployments, the Installer is
often expected to be a higher-skilled person because more unexpected situa-
tions will be encountered.
When you are the Installer, it is important to be friendly and polite. The
Installer is the public face of the organization; people will assume that the
entire organization acts the same way that you do.
A.1.2 The Repair Person
Things break. Some people view an SA as a Repair Person. Just as people call
a dishwasher repair person when their dishwasher breaks, they call a com-
puter Repair Person when their computer breaks. SAs also repair bigger and
sometimes more nebulous things, such as “the Internet” and “the database.”
Whether the real problem is simply a broken cable or a much larger problem
is of little interest to the customer.
A.1 Common Positive Roles 915
The value to the company of Repair People is their ability to bring the
company back to life when technological problems stall a business. Repair
People receive satisfaction from knowing they’ve helped one person or the
entire company. They enjoy the challenge of a good puzzle or mystery.
When you are the Repair Person, customers want to know that you are
concerned about their problems. They want to feel as though their problems
are the most important problems in the world.

A.1.3 The Maintainer
The Maintainer is the person who keeps previously built systems going. Main-
tainers are very good at following the instructions presented to them, either
in a written manual or through training. They do not seek to improve the
system; they are willing to maintain it as it is.
The value to the company of Maintainers is bringing stability to the
environment. These people are not going to break things trying to improve
them or replace them; nor do they spend all day reading magazines about
new things to install. Once companies spend money to install something,
they need it to be stable long enough to pay for itself before it is replaced
with something newer.
Maintainers receive satisfaction from knowing that their work is part of
the big picture that keeps the organization working. They tend to be glad that
they aren’t the people who have to figure out how to design and install the
next generation of systems and may even have disdain for those who wish to
replace their stable system with something new.
When you are the Maintainer, customers want two opposing things: They
want to know that you are maintaining the stability of their world, and they
want you to be flexible when they seek customizations.
A.1.4 The Problem Preventer
A role that is invisible to most customers is the Problem Preventer, who looks
for problems and fixes them before they become visible. Problem preventers
do the behind-the-scenes planning and preventive maintenance that keeps
problems from occurring at all. A good Problem Preventer collects metrics to
find trends but also has an ear to the ground to know what future problems
may arise.
The value to the company of Problem Preventers averting problems,
which is less costly than fixing problems when they happen.
916 Appendix A The Many Roles of a System Administrator
Problem Preventers receive satisfaction from knowing that their work

prevented problems that no one even knows could have happened. Their
joy is private. They enjoy thinking in the longer term rather than getting
immediate satisfaction from solving an emergency.
Typical customers do not know that this person exists, but their manage-
ment does. The managers expect this person to have the same priorities that
they do.
A.1.5 The Hero
The SA can be the Hero who saves the day. Like the firefighter who pulls
people out of a burning building, the Hero receives adulation and praise. The
network was down, but now it is up. The demo wasn’t going to be ready, but
the SA worked all weekend to bring the network to that part of the building.
Heroes get satisfaction out of their jobs from the praise they receive after
the fact.
The value to the company of Heroes is huge: Management always re-
wards a hero. Ironically, Problem Preventers often must struggle to get the
same positive visibility, though their contribution may be as or more valuable.
Heroes receive satisfaction from knowing that they hold the key to some
knowledge that the company could not live without. The Hero role is not
one that promotes a healthy nonwork life. Heroes give up nights, weekends,
and vacations, often with no notice. A personal life takes second priority.
Eventually, Heroes burn out and become Martyrs, unless management finds
some way to help them manage their stress.
Customers expect the Hero to be anywhere at any time. Customers would
prefer to deal only with the Hero, because this dashing superstar has become
someone on whom they can rely. However, customers need to learn that if
they get what they want, the Hero will burn out. New Heroes take a while
to find.
A.1.6 The “Go To” Person
This person has gained the reputation of being the one to solve any problem.
Go-to people are a little like Heroes, but are more coordinated and more

infrastructure-related. Instead of running around putting out fires or repairing
a server starting at 3
PM Friday and finishing at 3AM Sunday morning, this is
the person management will go to when large-scale deep-knowledge issues are
involved. Management knows that the go to person will get to the bottom
of the problem, work out the underlying problems, and fix them. It could
A.1 Common Positive Roles 917
be an infrastructure issue—tuning a parameter on a database—or a process
issue: how to ensure that new users have a common configuration, the need
to create a new automation system, or almost anything.
The value of having a go to person around is to get things done when
others don’t.
Like the Hero, this person can burn out if overused, but when he does
the job, he gets the satisfaction of knowing that his solution will become part
of the standard procedures going forward.
Customers expect a go to person to follow through when they agree to
solve a problem and to be able to give accurate time estimates, or at least
periodic status updates until a time estimate can be given.
A.1.7 The Infrastructure Builder
A corporate network depends on a lot of infrastructure: DNS, directories,
databases, scripts, switches, and so on. None of this is seen by the typical
customer, except when an outage is explained after the fact with mysterious
phrases, such as “It was a problem with the DNS server.”
The larger the company, the more valuable Infrastructure Builders be-
come. A good infrastructure is like a solid foundation on which a house
can be built. You can build a house on a shaky foundation and adjust for
it with more complicated and costly house designs, but in the long run, it
is cheaper to have started with a solid foundation. A tiny company has al-
most no infrastructure. Larger companies get benefits from amortizing the
cost of a quality infrastructure over larger and larger customer bases. When

small companies grow to become large companies, often what makes this go
smoothly is having had the foresight to employ SAs who “think big” about
infrastructure.
Infrastructure Builders get satisfaction from doing long-term planning,
taking existing systems and improving them, scaling large systems into hu-
mongous systems, and overhauling legacy systems and replacing them with
newer systems. Infrastructure Builders are proud of their ability to not only
build extremely huge systems but also coordinate elegant ways to transition
to them.
When you are the Infrastructure Builder, you have two groups of cus-
tomers. The general customer population wants the computer infrastructure
to be reliable and wants new infrastructure to be deployed yesterday. Your
other customers are the SAs whose systems sit on top of the infrastructure
you are building. The SAs want documentation and an infrastructure that
918 Appendix A The Many Roles of a System Administrator
is reliable and easy for them to understand, and they want it now, because
when you miss a deadline, it makes their projects late too.
A.1.8 The Policy Writer
Policies are the backbone of IT. They communicate the wishes of the top
corporate officials and dictate how things should be done, tell when they
should be done, and explain why they are done. SAs are often asked to
write policies on behalf of management. Social problems cannot be solved by
technology. Some social problems can be only solved by written policy.
The value to the company of Policy Writers is that they solve some prob-
lems and prevent new ones. Policies are a communication tool. As a company
grows, communication becomes more difficult and more important.
Policy Writers gain satisfaction from knowing that their knowledge,
skills, and personal experiences contributed to a policy that improved an
organization. They also enjoy being facilitators who can obtain buy-in from
many different communities.

When you are the Policy Writer, customers expect you to seek their input.
This should be done at the beginning of the process. It disempowers people
to ask for their opinion after the major decisions have been made. Your
willingness to listen will be appreciated.
A.1.9 The System Clerk
System Clerks have very little power or decision-making responsibilities.
These SAs are given instructions to be followed, such as “Create an account
for Fred” and “Allocate an IP address.” If the System Clerk works as an
assistant to a higher-level SA, this can be a fine arrangement. In fact, it is an
excellent way to start a career. However, we have seen System Clerks who
report to nontechnical managers, who get frustrated when the Clerk is not
able to tackle things outside his normal duties.
The value to the company of System Clerks comes from performing the
tasks that would otherwise distract senior SAs from more specialized tasks
and providing coverage for SAs when they are away. A System Clerk is also an
excellent candidate to fill a more senior SA position as it opens. The Clerk al-
ready knows the environment, and the hiring manager knows his personality.
However, if the environment has no senior SAs, the value provided is often
that of a scapegoat for a bad computing environment, when the real problem
is management’s lack of understanding about how to manage technology.
A.1 Common Positive Roles 919
The Clerk receives satisfaction from a job well done, from learning new
skills, and from looking forward to the excellent growth path ahead of him.
When you are the Clerk, customers want their requests to be performed
immediately, whether that is reasonable or not. Chapter 31 has more infor-
mation about dealing with this situation.
Case Study: Site with Only System Clerks
A site needs a balance of senior-level SAs and Clerks. There once was a site that had
only System Clerks. Their training included rudimentary U
NIX skills: perform back-

ups, create accounts, allocate IP addresses and IP subnets, install new software, and
add new hosts. The Clerks fell victim to the ‘‘we can always add one more’’ syn-
drome: new allocations were blindly made as requested, with no overall plan for in-
creasing capacity. For example, a new host would be added to a subnet without any
network-capacity planning. This worked for a while but eventually led to overloaded
subnets.
Customers complained of slow networks, but the Clerks did not have the network
engineering skills to fix the problem. Customers solved this problem themselves by
requesting private subnets to gain their own private dedicated local bandwidth. The
Clerks would happily allocate a new IP subnet, and users would connect it to the rest of
the network via a routing-enabled workstation with two NICs. These interconnections
were unreliable because hosts route packets slowly, especially when they become
overloaded. The more overloaded the main networks became, the more dedicated
subnets were created. Eventually, much of the slowness of the network was caused by
the slow interconnections between these private pools of bandwidth. The company’s
compute servers also suffered from the same lack of capacity planning. The customers
installed their own compute servers, even though the performance problems they
were trying to work around were most likely related to the slow host-based routing.
These new, fast servers overpowered the 10Mb network, particularly because they
were often an order of magnitude faster than the hosts doing the routing.
By the time the organization hired a senior-level SA, the network was a swamp of
unreliable subnets, badly configured compute servers, and antique file servers. The
network had 50 subnets for about 500 users. It took nearly 2 years to clean up the
mess and modernize the network.
A.1.10 The Lab Technician
The Lab Technician is an SA for highly specialized equipment. For exam-
ple, in a chemical research firm, the Lab Technician may be responsible for
a small network that connects all the scopes and monitoring devices. At a
920 Appendix A The Many Roles of a System Administrator
telecommunications manufacturer, the Lab Technician may maintain all the

equipment in a protocol-interoperability room having one of every version
of a product, the competition’s products, and a suite of traffic generators.
The Lab Technician is responsible for installing new equipment, integrating
systems together for ad hoc projects,
1
and being able to understand enough
of her customers’ specialties to translate their needs into the tasks she must
perform. The Lab Technician usually has a small network or group of net-
works that connect to the main corporate network and depend on the main
corporate network for most services; if she is smart, she also makes friends
in the corporate services area to pick their brains for technical knowledge.
The value to the company of Lab Technicians is letting the researchers
focus on designing the experiments rather than executing them. Lab Techni-
cians also add value by their vast knowledge base of technical information.
The Lab Technician derives satisfaction from getting an experiment or
demo successfully completed on time. However, if she does not get direct
congratulations from the researchers she serves, she may grow resentful. Lab
Technicians need to remember that their researchers are grateful, whether
they express it or not. Researchers will have Technicians who stay with them
longer if the Technicians are included in recognition ceremonies, awards,
dinners, and so on.
When you are the Lab Technician, customers want to know that some-
thing can be done, not how it will be done. They want their requirements met,
though it is your responsibility to draw out of them what those requirements
are. Active listening skills can greatly help in this area.
A.1.11 The Product Finder
The Product Finder reads every technology magazine and review so that when
someone asks, “Is there a software package that compresses widgets?” he can
recommend not only a list of widget compressors but also ways to determine
which is the most appropriate for the particular application. He also knows

where to find such a product or where to look for one.
The value to the company of the Product Finder is his ability to stay on
top of what’s new. Managers should not watch this kind of person closely,
because they will be appalled to discover that he spends half his workday
surfing the web and reading magazines. Managers must weigh that against
the time this person saves for everyone else.
1. In most labs, they are all ad hoc.
A.1 Common Positive Roles 921
Product Finders receive satisfaction from having all the right resources.
These people can be annoying to others in the group, even those they help,
because everyone would like to have the time to surf the web and keep in
touch, but most people (necessarily) have other priorities.
When you are the Product Finder, customers want summaries rather than
details. If you provide them with every detail that you’ve learned on the subject
in a long rambling story that takes hours to read, they will shy away from
you. Be concise.
A.1.12 The Solution Designer
Solution Designers play a key role in a company. On hearing of a problem,
they soon have a solution that is better than anyone would have expected.
This may solve a small issue, such as installing an e-fax server to make it easier
to send faxes, or it may resolve a large issue, such as creating an electronic
version of a paper process. Unlike Product Finders, Solution Designers are
more likely to build something from scratch or to integrate some smaller
packages.
The value to the company of Solution Designers is their ability to remove
roadblocks and simplify bureaucratic processes.
The Solution Designer receives satisfaction from knowing that her solu-
tions are used, because usage indicates that people like it.
When you are the Solution Designer, customers want to see their aspect
of the problem solved, not what you may perceive as the problem or what

would save the company money. For example, if expense reports are faxed
to headquarters (HQ), you might create a way for the data to be entered
electronically so that HQ doesn’t have to retype all the data. However, your
customers aren’t helped by saving time at HQ; they simply want the prepa-
ration to be made easier. That would be solved with a better user interface or
a system that could download their corporate credit card bill off the service
provider’s web site. Your customers wouldn’t even care if the output was then
e-faxed to HQ for manual reentry.
A.1.13 The Ad Hoc Solution Finder
The Ad Hoc Solution Finder can, on an emergency basis, create a solution
to a seemingly impossible problem. This is the person who magically yet
securely gets network connectivity to the moon for your big demo to the
moon men. These people may know more about the tools than the aver-
age person who uses the tools, possibly from dissecting them. Unlike the
922 Appendix A The Many Roles of a System Administrator
Hero, who usually puts out fires by fixing problems, this person builds
solutions.
The value to the company of Ad Hoc Solution Finders is their ability
to find solutions that work around the fact that technology is not as flex-
ible as some special situations require or that your corporate network has
weaknesses that you have not invested in fixing. The former is a situation
that gets better over time. The latter indicates a lack of proper technology
management.
The Ad Hoc Solution Finder receives satisfaction from saving the day.
Like the Hero, the Ad Hoc Solution Finder can get burned out from being
overloaded.
When you are the Ad Hoc Solution Finder, customers want miracles
to happen and don’t want to be reminded that the emergency could have
been prevented through better planning by the company, which is rarely
their fault.

A.1.14 The Unrequested Solution Person
Some SAs find themselves providing solutions that weren’t requested. This
can be a good thing and a bad thing. One SA was rewarded for installing
for his users a paperless fax system that wasn’t requested but soon became
a major productivity enhancement. It was based on free software and used
their existing modem pool, so the tangible cost was zero. This same SA was
once reprimanded for spending too much time on “self-directed projects”
and was encouraged to focus on his assigned tasks.
The value to the company of Unrequested Solution people is, they are
usually close to the customers and positioned to see needs that upper manage-
ment wouldn’t see or understand. These SAs may also be more aware of new
products than their less technical customers.
Individuals in this role receive satisfaction from discovering that their
guesses of what might be useful turn out to be correct.
When you are in this role, customers want you to guess correctly what
will or won’t be useful to them; talking with them regularly at appropriate
times is critical. They will be concerned that these new projects don’t interfere
with your assigned project’s deadlines, especially when that would result in
their missing their deadlines. Management will be concerned about the cost
of your time and of any tangible costs, especially when an unrequested new
service does not get used.
A.1 Common Positive Roles 923
A.1.15 The On-Call Expert
The On-Call Expert is always available to give advice. This person has estab-
lished herself as knowledgeable in all or most aspects of the system. Some-
times the On-Call Expert has a narrow focus; other times, she is an all-around
expert.
The value to the company of On-Call Experts is that people have someone
to call when they need advice, whether for an exact answer or simply a good
starting point for research.

The On-Call Expert receives satisfaction from helping people and from
the ego trip that is inherent to the role. Because technology changes quickly,
she requires time to maintain her knowledge, whether that is time spent
reading magazines, networking at conferences, or experimenting with new
products.
When you are the On-Call Expert, you must remember to help people
help themselves. If you don’t, you will find yourself overcommitted.
A.1.16 The Educator
The Educator teaches customers to use the services available. The Educator
may stop by to fix a problem with a printer but stays to teach the customer
how to better use the spreadsheet software and finds himself writing most of
the user documentation.
The Educator is valuable to the company because his work results in
people working more efficiently with the tools they have. The Educator has
close interactions with customers and therefore learns what problems people
are having. He becomes a resource for finding out what the customers need.
The Educator receives satisfaction from knowing that his documentation
is used and appreciated and from knowing that people work better because
of his efforts.
When you are the Educator, customers want you to understand their jobs,
how they work, and, most important, what it is in their tools that they find
confusing. They want documentation that answers the questions they have,
not what the developers think is important.
A.1.17 The Policy Enforcer
The Policy Enforcer is responsible for saying no when someone wants to do
something that is against policy and also shuts down violators. The Policy
Enforcer depends on two tools equally: written policies and management
924 Appendix A The Many Roles of a System Administrator
support. Policies must be written and published for all to see. If the policies
are not written, enforcement will be inconsistent because he will have to make

up the rules as he goes along, and his peers may enforce different ideas of
what is right and wrong. The second tool is management support. The policy
has no teeth if management bends the rules every time someone requests an
exception. A manager shouldn’t sign off on a policy and then continually sign
off on requests for exceptions. Often, the Policy Enforcer has the authority
to disconnect a network jack if the violation is creating a global problem
and the violator cannot be contacted in a reasonable amount of time. If the
management does not support the Enforcer’s decision, he can’t do his job.
If management approves a policy but then permits an exception after the
Enforcer says no, he loses authority and the will or reason to continue.
The value to the company of the Policy Enforcer is that company policies
are carried out. Lack of follow-through on an important policy defeats the
point of having a policy.
The Policy Enforcer receives satisfaction from knowing that he is actively
trying to keep the company following the direction set by the management
and from being chartered to steamroller through the site ensuring compliance.
When you are the Policy Enforcer, customers want to get their jobs done
and don’t understand why so many roadblocks (policies) are preventing them
from doing that. Rather than saying no, it can be more useful to help them by
understanding what they are trying to achieve and helping them reach that
goal and stay within policy. If you do not like to be in this role but feel trapped
in it, you might consider assertiveness training or such books as When I Say
No I Feel Guilty (Smith 2000).
A Policy with Exceptions
A site had a security policy that created a lot of extra work for anyone who wanted to
abide by it. For a web site to be accessible from outside the firewall, the site had to be
replicated on the outside rather than by poking a hole in the firewall to let outsiders access
the internal host. This replicated site could not make connections back into the company.
If it needed access to an internal service, such as a database, that service also had to
be replicated. Making a service completely self-sufficient was very difficult. Therefore,

when the Policy Enforcer rejected a request, the employee would cry to management,
and an exception would be granted. Eventually, enough holes were poked in the firewall
that the policy didn’t mean anything.
The Policy Enforcer proposed a revision to the policy that simply reflected manage-
ment’s behavior: Holes would be poked if the cost of replication would exceed a certain
A.1 Common Positive Roles 925
number of hours of work. Management was in a furor at the proposal because it was not
how it wanted security to be done. The Policy Enforcer pointed out all the exceptions
management had made. Although old exceptions were grandfathered, management be-
came much better at supporting the Policy Enforcer after the revision. If management
wasn’t going to support the policy, the Policy Enforcer shouldn’t have to, either.
A.1.18 The Disaster Worrier
Someone in the group should be worried about things going wrong. When a
solution is being proposed, this person asks, “What is the failure mode?” Of
course, the Disaster Worrier can’t drive all decisions, or projects will never
be completed or will be over budget. This person needs to be balanced by
an optimist. However, without someone keeping an eye out for potential
disasters, a team can create a house of cards.
The value to the company of the Disaster Worrier is felt only in times
of emergency. Half the system is failing, but the other half keeps working
because of controls put in place. General system robustness can be the result
of this person.
This person receives satisfaction from ensuring safety and stability.
When you are in this role, others around you may get tired of your con-
stant push for belts and suspenders. It is important to pick your battles rather
than have an opinion at every turn. Nobody likes to hear such laments as,
“That wouldn’t have failed if people had listened to me” or “Next time,
you won’t be so quick to ignore me!” It may be better to share responsibil-
ity rather than place blame and refer to future improvement rather than
gloat about your expertise: “In the future, we need to write scripts that

handle disk-full situations.” Gentle one-on-one coaching is more effective
than public bemoaning.
A.1.19 The Careful Planner
The Careful Planner takes the time to plan each step of the project in which
she is involved. She builds good test plans and is never flustered when things
go wrong, because she has already figured out what to do.
The value to the company of the Careful Planner is that she completes
important tasks reliably and flawlessly.
This person derives satisfaction from completing a task and knowing that
it is really finished and watching the first customers use it without a hitch.
She takes pride in her work.
926 Appendix A The Many Roles of a System Administrator
When you are in this role, others come to rely on your work being flaw-
less. You are often given the tasks that cannot afford to fail. Continue to
work as you always did, and don’t let the importance of the tasks weigh you
down. Be aware that your meticulous work takes time and that others are
always in a hurry and may get agitated watching you work. Make sure that
you develop a talent for predicting how long you will need to complete a
task. You don’t want to be seen as someone who couldn’t meet a deadline if
it walked up and introduced itself.
A.1.20 The Capacity Planner
The Capacity Planner makes the system scale as it grows. This person notices
when things are getting full, running out, or becoming overloaded. Good
Capacity Planners pay attention to utilization patterns and are in tune with
business changes that may affect them. Great Capacity Planners install sys-
tems that do this monitoring automatically and produce graphs that predict
when capacity will run out. Vendors can help Capacity Planners by docu-
menting data that they would find useful, such as how much RAM and disk
space are required as a function of the number of users.
The value to the company of Capacity Planners is that traffic jams are

prevented. This is another role that goes unnoticed if the job is done properly.
This person also helps the company fix the correct problem the right way. (Too
many times, we’ve seen departments trying to speed up a server by adding
more RAM when the real problem was an overloaded network connection,
or vice versa.)
The Capacity Planner receives satisfaction from knowing that problems
are prevented, that people heed warnings, and from finding the real source
of problems.
When you are the Capacity Planner, customers want you to have accurate
data and solutions that won’t cost any money. It is your job to justify costs.
As always, explaining things in the customer’s language is critical.
A.1.21 The Budget Administrator
The Budget Administrator keeps tabs on how much money is left in the budget
and helps write the budget for next year. This person knows what the money
is meant to be spent on, when it is meant to be spent, and how to make the
budget stretch farther.
The value to the company of the Budget Administrator is to keep
SA expenses under control, ensuring that the tasks that need doing are
A.1 Common Positive Roles 927
funded—within reason—even if they are unexpected, and providing reli-
able figures so management can perform financial planning for the coming
year.
The Budget Administrator receives satisfaction from staying within
budget and still managing to fund extra, important projects that were not
budgeted for.
When you are the Budget Administrator, customers want you to stay in
budget, to prepare a good budget plan for the next year, to accurately evaluate
what the most important projects are, to make sure that all the critical tasks
have funding, and to show how the money they let you spend is benefiting
them.

A.1.22 The Customer’s Advocate
The Customer’s Advocate can help a person speak up for her needs. He is
the translator and lobbyist positioned between the customer and her manage-
ment. The Advocate doesn’t simply recommend a solution but also coaches
the customer on how to sell the idea to her boss and stands by during the
presentation in case she needs help.
The value to the company of the Customer’s Advocate is to help the
customers get what they need despite red tape and communication barriers.
The Advocate receives satisfaction from knowing that he has helped
someone. He also knows that by interfacing with management, he is able
to put his SA team in a good light and to perform the role of the helpful facil-
itator. Often, you help a customer get what she needs by working the system
rather than going around it. This is especially valuable if you also created the
system.
When you are the Advocate, customers want you to understand them
before you start suggesting solutions. The customers want you to under-
stand their technical needs as well as soft issues, such as schedules and
budgets.
A.1.23 The Technocrat
The Technocrat is the advocate for new technology. When a system needs
to be repaired or replaced, he puts more value in the new system because
it is new, even if it still has bugs. He disdains those who seek comfort in
old systems that may be “good enough.” The Technocrat can provide good
counterbalance to the Disaster Worrier.
928 Appendix A The Many Roles of a System Administrator
The value to the company of the Technocrat is that he prevents the com-
pany from becoming technically stagnant.
The Technocrat receives satisfaction from being surrounded by the latest
new technology—dare we say new-toy syndrome.
When you are the Technocrat, customers want you to focus on the real

value of a solution rather than that newer is better.
A.1.24 The Salesperson
The Salesperson is not limited to tangible items. She may be selling a particular
policy, new service, or proposal. She may be selling the SA team itself, either
to upper management or to the customers. A Salesperson is concerned with
finding the needs of customers and then convincing them that what she has
to sell meets those needs. New services are easier to sell if the customers were
involved in the specification and selection process.
The value to the company of the Salesperson is that she makes the SA
team’s job easier. A great system that is never accepted by the customers is
not useful to the company. A great policy that saves the company money is
not helpful if the customers work around it because they don’t understand
the benefits.
The Salesperson receives short-term satisfaction from “making the sale,”
but for real, lasting satisfaction, the Salesperson must develop a relationship
with the customers and find herself feeling that she truly helps the customers
in a meaningful way.
When you are the Salesperson, customers want to have their needs un-
derstood and appreciated. They want to be talked with, not to.
A.1.25 The Vendor Liaison
The Vendor Liaison maintains a relationship with one or more vendors. She
may know a vendor’s product line better than anyone else the in the group
and be privy to upcoming products. She is a resource for the other SAs, thus
saving calls to the vendor’s salesperson.
The value to the company of the Vendor Liaison is having someone who
understands and is dedicated to the company’s needs dealing with a vendor.
Having a single point of contact saves resources.
The Vendor Liaison receives satisfaction from being the expert that ev-
eryone respects, from being the first to know about vendor news, and from
the free lunches and shirts she receives.

A.1 Common Positive Roles 929
When you are the Vendor Liaison, customers want you to be all-knowing
about the vendor, open-minded about competing vendors, and a harsh
negotiator when getting prices.
A.1.26 The Visionary
The Visionary looks at the big picture and has a vision of where the group
should go.
The value to the company of the Visionary is keeping the group focused
on what’s next.
The Visionary receives satisfaction when he looks back over the years
and sees that in the long term, he made a difference. All those incremental
improvements accumulated to meet major goals.
When you are the Visionary, customers want to know what’s happening
next and may not be too concerned with the long term. Your team’s reputation
for being able to execute a plan affects your ability to sell your vision to the
customers.
A.1.27 The Mother
The Mother nurtures the customers. It’s difficult to explain except through
example. One SA spent her mornings walking through the halls, stopping by
each person’s office to see how things were. She would fix small problems
and note the bigger problems for the afternoon. She would answer many
user-interface questions that a customer might have felt were too small to
ask the helpdesk. The customers were making a big paradigm change (from
X Terminals to PCs running X Terminal emulators), and this mothering was
exactly what they needed. In her morning walks, she would answer hundreds
of questions and resolve dozens of problems that would otherwise have been
tickets submitted to the helpdesk. The customers got very used to this level
of service and soon came to rely on her morning visits as part of what kept
them productive.
The value to the company of the Mother is her high degree of hand-

holding, which can be critical at times of great change or with nontechnical
customers. The personal contact also ensures a more precise understanding
of the customers’ needs.
The Mother receives satisfaction from the personal relationships she
develops with her customers.
930 Appendix A The Many Roles of a System Administrator
When you are the Mother, customers want to know that their immediate
needs are being met and will put less emphasis on the long-term strategy. You
must remember to keep an eye on the future and not get too absorbed in the
present.
A.1.28 The Monitor
The Monitor notices how well things are running. Sometimes, the Moni-
tor uses low-tech methods, using the same services that his customers use.
Although the SAs may have a private file server, this person stores his files
on the file server that the customers use, so he can “feel their pain.” As this
person becomes more sophisticated, he automates his monitoring but then
watches the monitoring system’s output and takes the time to fix things rather
than simply clear the alarms.
The value to the company of the Monitor is that problems are noticed
before customers start complaining. This can give the perception of a trouble-
free network.
The Monitor receives satisfaction from being the first to notice a problem,
from knowing that he’s working on fixing a problem before customers report
it, and from knowing that problems are prevented by monitoring capacity
issues.
When you are the Monitor, customers most likely don’t know that you
exist. If they did, they would want your testing to simulate their real work-
loads: end-to-end testing. For example, it isn’t good enough to know that a
mail server is up. You must test that a message can be submitted, relayed,
delivered, and read.

A.1.29 The Facilitator
The Facilitator has excellent communication skills. He tends to turn im-
promptu discussions into decision-making meetings. He is often asked to run
meetings, especially large meetings in which keeping focus can be difficult.
The Facilitator adds value by making processes run more smoothly. He
may not take on a lot of action items, but he gets groups of people to agree to
what needs to be done and who is going to do it. He keeps meetings efficient
and fun.
The Facilitator receives satisfaction from seeing people come to agree-
ment on goals and taking initiative to see the goals completed.
When you are the Facilitator, the other members on your team want you
to facilitate all their discussions. It is important to coach other people into