red hat linux security and optimization

S
E
C
U
R
I
T
Y
T
O
O
L
S
O
N
C
D
-
R
O
M
®
PRESS
®
Linux Solutions from the Experts at Red Hat
Mohammed J. Kabir
®
®
™
Red Hat


Linux

Security and
Optimization
Mohammed J. Kabir
Hungry Minds, Inc.
New York, NY
●
Indianapolis, IN
●
Cleveland, OH
Trademarks: are trademarks or registered trademarks of Hungry Minds, Inc. All other trademarks are the
property of their respective owners. Hungry Minds, Inc., is not associated with any product or vendor
mentioned in this book.
LIMIT OF LIABILITY/DISCLAIMER OF WARRANTY: THE PUBLISHER AND AUTHOR HAVE USED THEIR
BEST EFFORTS IN PREPARING THIS BOOK. THE PUBLISHER AND AUTHOR MAKE NO REPRESENTATIONS
OR WARRANTIES WITH RESPECT TO THE ACCURACY OR COMPLETENESS OF THE CONTENTS OF THIS
BOOK AND SPECIFICALLY DISCLAIM ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS
FOR A PARTICULAR PURPOSE. THERE ARE NO WARRANTIES WHICH EXTEND BEYOND THE
DESCRIPTIONS CONTAINED IN THIS PARAGRAPH. NO WARRANTY MAY BE CREATED OR EXTENDED BY
SALES REPRESENTATIVES OR WRITTEN SALES MATERIALS. THE ACCURACY AND COMPLETENESS OF
THE INFORMATION PROVIDED HEREIN AND THE OPINIONS STATED HEREIN ARE NOT GUARANTEED OR
WARRANTED TO PRODUCE ANY PARTICULAR RESULTS, AND THE ADVICE AND STRATEGIES
CONTAINED HEREIN MAY NOT BE SUITABLE FOR EVERY INDIVIDUAL. NEITHER THE PUBLISHER NOR
AUTHOR SHALL BE LIABLE FOR ANY LOSS OF PROFIT OR ANY OTHER COMMERCIAL DAMAGES,
INCLUDING BUT NOT LIMITED TO SPECIAL, INCIDENTAL, CONSEQUENTIAL, OR OTHER DAMAGES.
FULFILLMENT OF EACH COUPON OFFER IS THE SOLE RESPONSIBILITY OF THE OFFEROR.
Red Hat

Linux


Security and Optimization
Published by
Hungry Minds, Inc.
909 Third Avenue
New York, NY 10022
www.hungryminds.com
Copyright © 2002 Hungry Minds, Inc. All rights
reserved. No part of this book, including interior
design, cover design, and icons, may be reproduced
or transmitted in any form, by any means
(electronic, photocopying, recording, or otherwise)
without the prior written permission of the publisher.
Library of Congress Control Number: 2001092938
ISBN: 0-7645-4754-2
Printed in the United States of America
10 9 8 7 6 5 4 3 2 1
1B/SX/RR/QR/IN
Distributed in the United States by Hungry Minds,
Inc.
Distributed by CDG Books Canada Inc. for Canada;
by Transworld Publishers Limited in the United
Kingdom; by IDG Norge Books for Norway; by IDG
Sweden Books for Sweden; by IDG Books Australia
Publishing Corporation Pty. Ltd. for Australia and
New Zealand; by TransQuest Publishers Pte Ltd. for
Singapore, Malaysia, Thailand, Indonesia, and Hong
Kong; by Gotop Information Inc. for Taiwan; by ICG
Muse, Inc. for Japan; by Intersoft for South Africa;
by Eyrolles for France; by International Thomson

Publishing for Germany, Austria, and Switzerland;
by Distribuidora Cuspide for Argentina; by LR
International for Brazil; by Galileo Libros for Chile;
by Ediciones ZETA S.C.R. Ltda. for Peru; by WS
Computer Publishing Corporation, Inc., for the
Philippines; by Contemporanea de Ediciones for
Venezuela; by Express Computer Distributors for the
Caribbean and West Indies; by Micronesia Media
Distributor, Inc. for Micronesia; by Chips
Computadoras S.A. de C.V. for Mexico; by Editorial
Norma de Panama S.A. for Panama; by American
Bookshops for Finland.
For general information on Hungry Minds’ products
and services please contact our Customer Care
department within the U.S. at 800-762-2974, outside
the U.S. at 317-572-3993 or fax 317-572-4002.
For sales inquiries and reseller information,
including discounts, premium and bulk quantity
sales, and foreign-language translations, please
contact our Customer Care department at
800-434-3422, fax 317-572-4002 or write to Hungry
Minds, Inc., Attn: Customer Care Department, 10475
Crosspoint Boulevard, Indianapolis, IN 46256.
For information on licensing foreign or domestic
rights, please contact our Sub-Rights Customer Care
department at 212-884-5000.
For information on using Hungry Minds’ products
and services in the classroom or for ordering
examination copies, please contact our Educational
Sales department at 800-434-2086 or fax

317-572-4005.
For press review copies, author interviews, or other
publicity information, please contact our Public
Relations department at 317-572-3168 or fax
317-572-4168.
For authorization to photocopy items for corporate,
personal, or educational use, please contact
Copyright Clearance Center, 222 Rosewood Drive,
Danvers, MA 01923, or fax 978-750-4470.
is a trademark of Hungry Minds, Inc.
About the Author
Mohammed Kabir is the founder and CEO of Evoknow, Inc. His company specializes
in open-source solutions and customer relationship management software develop-
ment. When he is not busy managing software projects or writing books, he enjoys
traveling around the world. Kabir studied computer engineering at California State
University, Sacramento. He is also the author of Red Hat Linux Server and Apache
Server Bible. He can be reached at
Credits
ACQUISITIONS EDITOR
Debra Williams Cauley
PROJECT EDITOR
Pat O’Brien
TECHNICAL EDITORS
Matthew Hayden
Sandra “Sam” Moore
COPY EDITORS
Barry Childs-Helton
Stephanie Provines
EDITORIAL MANAGER
Kyle Looper

RED HAT PRESS LIAISON
Lorien Golaski, Red Hat
Communications Manager
SENIOR VICE PRESIDENT, TECHNICAL
PUBLISHING
Richard Swadley
VICE PRESIDENT AND PUBLISHER
Mary Bednarek
PROJECT COORDINATOR
Maridee Ennis
GRAPHICS AND PRODUCTION
SPECIALISTS
Karl Brandt
Stephanie Jumper
Laurie Petrone
Brian Torwelle
Erin Zeltner
QUALITY CONTROL TECHNICIANS
Laura Albert
Andy Hollandbeck
Carl Pierce
PERMISSIONS EDITOR
Carmen Krikorian
MEDIA DEVELOPMENT SPECIALIST
Marisa Pearman
PROOFREADING AND INDEXING
TECHBOOKS Production Services

This book is dedicated to my wife, who proofs my writing, checks my facts,
and writes my dedications.

Preface
This book is focused on two major aspects of Red Hat Linux system administration:
performance tuning and security. The tuning solutions discussed in this book will
help your Red Hat Linux system to have better performance. At the same time, the
practical security solutions discussed in the second half of the book will allow you
to enhance your system security a great deal. If you are looking for time saving,
practical solutions to performance and security issues, read on!
How This Book is Organized
The book has five parts, plus several appendixes.
Part I: System Performance
This part of the book explains the basics of measuring system performance, cus-
tomizing your Red Hat Linux kernel to tune the operating system, tuning your
hard disks, and journaling your filesystem to increase file system reliability and
robustness.
Part II: Network and Service Performance
This part of the book explains how to tune your important network services,
including Apache Web server, Sendmail and postfix mail servers, and Samba and
NFS file and printer sharing services.
Part III: System Security
This part of the book covers how to secure your system using kernel-based Linux
Intrusion Detection System (LIDS) and Libsafe buffer overflow protection mecha-
nisms. Once you have learned to secure your Red Hat Linux kernel, you can secure
your file system using various tools. After securing the kernel and the file system,
you can secure user access to your system using such tools as Pluggable
Authentication Module (PAM), Open Source Secure Socket Layer (OpenSSL), Secure
Remote Password (SRP), and xinetd.
Part IV: Network Service Security
This part of the book shows how to secure your Apache Web server, BIND DNS
server, Sendmail and postfix SMTP server, POP3 mail server, Wu-FTPD and
ProFTPD FTP servers, and Samba and NFS servers.

vi
Part V: Firewalls
This part of the book shows to create packet filtering firewall using iptables, how to
create virtual private networks, and how to use SSL based tunnels to secure access
to system and services. Finally, you will be introduced to an wide array of security
tools such as security assessment (audit) tools, port scanners, log monitoring and
analysis tools, CGI scanners, password crackers, intrusion detection tools, packet
filter tools, and various other security administration utilities.
Appendixes
These elements include important references for Linux network users, plus an
explanation of the attached CD-ROM.
Conventions of This Book
You don’t have to learn any new conventions to read this book. Just remember the
usual rules:
◆
When you are asked to enter a command, you need press the Enter or the
Return key after you type the command at your command prompt.
◆
A monospaced font is used to denote configuration or code segment.
◆
Text in italic needs to be replaced with relevant information.
Watch for these icons that occasionally highlight paragraphs.
The Note icon indicates that something needs a bit more explanation.
The Tip icon tells you something that is likely to save you some time and
effort.
Preface vii
The Caution icon makes you aware of a potential danger.
The cross-reference icon tells you that you can find additional information
in another chapter.
Tell Us What You Think of This Book

Both Hungry Minds and I want to know what you think of this book. Give us your
feedback. If you are interested in communicating with me directly, send e-mail
messages to I will do my best to respond promptly.
viii Red Hat Linux Security and Optimization
Acknowledgments
While writing this book, I often needed to consult with many developers whose
tools I covered in this book. I want to specially thank a few such developers who
have generously helped me present some of their great work.
Huagang Xie is the creator and chief developer of the LIDS project. Special
thanks to him for responding to my email queries and also providing me with a
great deal of information on the topic.
Timothy K. Tsai, Navjot Singh, and Arash Baratloo are the three members of the
Libsafe team who greatly helped in presenting the Libsafe information. Very special
thanks to Tim for taking the time to promptly respond to my emails and providing
me with a great deal of information on the topic.
I thank both the Red Hat Press and Hungry Minds teams who made this book a
reality. It is impossible to list everyone involved but I must mention the following
kind individuals.
Debra Williams Cauley provided me with this book opportunity and made sure I
saw it through to the end. Thanks, Debra.
Terri Varveris, the acquisitions editor, took over in Debra’s absence. She made
sure I had all the help needed to get this done. Thanks, Terri.
Pat O’Brien, the project development editor, kept this project going. I don’t know
how I could have done this book without his generous help and suggestions every
step of the way. Thanks, Pat.
Matt Hayden, the technical reviewer, provided numerous technical suggestions,
tips, and tricks — many of which have been incorporated in the book. Thanks, Matt.
Sheila Kabir, my wife, had to put up with many long work hours during the few
months it took to write this book. Thank you, sweetheart.
ix

Contents at a Glance
Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vi
Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . ix
Part I System Performance
Chapter 1 Performance Basics . . . . . . . . . . . . . . . . . . . . . . . . . 3
Chapter 2 Kernel Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Chapter 3 Filesystem Tuning . . . . . . . . . . . . . . . . . . . . . . . . . 39
Part II Network and Service Performance
Chapter 4 Network Performance . . . . . . . . . . . . . . . . . . . . . . 75
Chapter 5 Web Server Performance . . . . . . . . . . . . . . . . . . . . 89
Chapter 6 E-Mail Server Performance . . . . . . . . . . . . . . . . . 125
Chapter 7 NFS and Samba Server Performance . . . . . . . . . . 141
Part III System Security
Chapter 8 Kernel Security . . . . . . . . . . . . . . . . . . . . . . . . . . 155
Chapter 9 Securing Files and Filesystems . . . . . . . . . . . . . . 179
Chapter 10 PAM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
Chapter 11 OpenSSL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263
Chapter 12 Shadow Passwords and OpenSSH . . . . . . . . . . . . 277
Chapter 13 Secure Remote Passwords . . . . . . . . . . . . . . . . . . 313
Chapter 14 xinetd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323
Part IV Network Service Security
Chapter 15 Web Server Security . . . . . . . . . . . . . . . . . . . . . . 351
Chapter 16 DNS Server Security . . . . . . . . . . . . . . . . . . . . . . 399
Chapter 17 E-Mail Server Security . . . . . . . . . . . . . . . . . . . . 415
Chapter 18 FTP Server Security . . . . . . . . . . . . . . . . . . . . . . . 443
Chapter 19 Samba and NFS Server Security . . . . . . . . . . . . . 473
Part V Firewalls
Chapter 20 Firewalls, VPNs, and SSL Tunnels . . . . . . . . . . . . 491
Chapter 21 Firewall Security Tools . . . . . . . . . . . . . . . . . . . . 541

Appendix A IP Network Address Classification . . . . . . . . . . . . 589
Appendix B Common Linux Commands . . . . . . . . . . . . . . . . . 593
Appendix C Internet Resources . . . . . . . . . . . . . . . . . . . . . . . . 655
Appendix D Dealing with Compromised Systems . . . . . . . . . . 661
Appendix E What’s On the CD-ROM? . . . . . . . . . . . . . . . . . . . 665
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 669
End-User License Agreement . . . . . . . . . . . . . . . . 691
Contents
Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vi
Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . ix
Part I System Performance
Chapter 1 Performance Basics . . . . . . . . . . . . . . . . . . . . . . . . . 3
Measuring System Performance . . . . . . . . . . . . . . . . . . . . . . . 4
Monitoring system performance with ps . . . . . . . . . . . . . . . . . . . . . 4
Tracking system activity with top . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Checking memory and I/O with vmstat . . . . . . . . . . . . . . . . . . . . . . 8
Running Vtad to analyze your system . . . . . . . . . . . . . . . . . . . . . . 9
Chapter 2 Kernel Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Compiling and Installing a Custom Kernel . . . . . . . . . . . . . . 11
Downloading kernel source code (latest distribution) . . . . . . . . . . 11
Creating the /usr/src/linux symbolic link . . . . . . . . . . . . . . . . . . . 12
Selecting a kernel-configuration method . . . . . . . . . . . . . . . . . . . 13
Using menuconfig . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Compiling the kernel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
Booting the new kernel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
Running Demanding Applications . . . . . . . . . . . . . . . . . . . . 35
Chapter 3 Filesystem Tuning . . . . . . . . . . . . . . . . . . . . . . . . . 39
Tuning your hard disks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
Tuning ext2 Filesystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
Changing the block size of the ext2 filesystem . . . . . . . . . . . . . . . 44

Using e2fsprogs to tune ext2 filesystem . . . . . . . . . . . . . . . . . . . . 45
Using a Journaling Filesystem . . . . . . . . . . . . . . . . . . . . . . . 48
Compiling and installing ReiserFS . . . . . . . . . . . . . . . . . . . . . . . . 50
Using ReiserFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Benchmarking ReiserFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Managing Logical Volumes . . . . . . . . . . . . . . . . . . . . . . . . . 54
Compiling and installing the LVM module for kernel . . . . . . . . . . 54
Creating a logical volume . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
Adding a new disk or partition to a logical volume . . . . . . . . . . . 62
Removing a disk or partition from a volume group . . . . . . . . . . . 65
Using RAID, SAN, or Storage Appliances . . . . . . . . . . . . . . 66
Using Linux Software RAID . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
Using Hardware RAID . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
Using Storage-Area Networks (SANs) . . . . . . . . . . . . . . . . . . . . . . 67
Using Storage Appliances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
Using a RAM-Based Filesystem . . . . . . . . . . . . . . . . . . . . . . 68
Part II Network and Service Performance
Chapter 4 Network Performance . . . . . . . . . . . . . . . . . . . . . . 75
Tuning an Ethernet LAN or WAN . . . . . . . . . . . . . . . . . . . . 75
Using network segmentation technique for performance . . . . . . . 77
Using switches in place of hubs . . . . . . . . . . . . . . . . . . . . . . . . . . 80
Using fast Ethernet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
Using a network backbone . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
Understanding and controlling network traffic flow . . . . . . . . . . . 83
Balancing the traffic load using the DNS server . . . . . . . . . . . . . . 85
IP Accounting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
IP accounting on a Linux network gateway . . . . . . . . . . . . . . . . . 86
Chapter 5 Web Server Performance . . . . . . . . . . . . . . . . . . . . 89
Compiling a Lean and Mean Apache . . . . . . . . . . . . . . . . . . 89
Tuning Apache Configuration . . . . . . . . . . . . . . . . . . . . . . . 95

Controlling Apache processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
Controlling system resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
Using dynamic modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
Speeding Up Static Web Pages . . . . . . . . . . . . . . . . . . . . . . 103
Reducing disk I/O for faster static page delivery . . . . . . . . . . . . . 104
Using Kernel HTTP daemon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
Speeding Up Web Applications . . . . . . . . . . . . . . . . . . . . . 105
Using mod_perl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
Using FastCGI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
Installing and configuring FastCGI module for Apache . . . . . . . . 115
Using Java servlets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
Using Squid proxy-caching server . . . . . . . . . . . . . . . . . . . . . . . . 118
Chapter 6 E-Mail Server Performance . . . . . . . . . . . . . . . . . 125
Choosing Your MTA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
Tuning Sendmail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
Controlling the maximum size of messages . . . . . . . . . . . . . . . . 127
Caching Connections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
Controlling simultaneous connections . . . . . . . . . . . . . . . . . . . . 130
Limiting the load placed by Sendmail . . . . . . . . . . . . . . . . . . . . . 131
xiv Contents
Saving memory when processing the mail queue . . . . . . . . . . . . 131
Controlling number of messages in a queue run . . . . . . . . . . . . . 132
Handling the full queue situation . . . . . . . . . . . . . . . . . . . . . . . . 132
Tuning Postfix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
Installing Postfix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
Limiting number of processes used . . . . . . . . . . . . . . . . . . . . . . . 134
Limiting maximum message size . . . . . . . . . . . . . . . . . . . . . . . . . 135
Limiting number of messages in queue . . . . . . . . . . . . . . . . . . . . 135
Limiting number of simultaneous delivery to a single site . . . . . 135
Controlling queue full situation . . . . . . . . . . . . . . . . . . . . . . . . . 135

Controlling the length a message stays in the queue . . . . . . . . . . 136
Controlling the frequency of the queue . . . . . . . . . . . . . . . . . . . . 136
Using PowerMTA for High-Volume Outbound Mail . . . . . . 136
Using multiple spool directories for speed . . . . . . . . . . . . . . . . . . 137
Setting the maximum number of file descriptors . . . . . . . . . . . . 137
Setting a maximum number of user processes . . . . . . . . . . . . . . 138
Setting maximum concurrent SMTP connections . . . . . . . . . . . . 138
Monitoring performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
Chapter 7 NFS and Samba Server Performance . . . . . . . . . . 141
Tuning Samba Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
Controlling TCP socket options . . . . . . . . . . . . . . . . . . . . . . . . . . 142
Tuning Samba Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
Tuning NFS Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
Optimizing read/write block size . . . . . . . . . . . . . . . . . . . . . . . . . 146
Setting the appropriate Maximum Transmission Unit . . . . . . . . . 149
Running optimal number of NFS daemons . . . . . . . . . . . . . . . . . 149
Monitoring packet fragments . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
Part III System Security
Chapter 8 Kernel Security . . . . . . . . . . . . . . . . . . . . . . . . . . 155
Using Linux Intrusion Detection System (LIDS) . . . . . . . . . 155
Building a LIDS-based Linux system . . . . . . . . . . . . . . . . . . . . . . 156
Administering LIDS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
Using libsafe to Protect Program Stacks . . . . . . . . . . . . . . 173
Compiling and installing libsafe . . . . . . . . . . . . . . . . . . . . . . . . . 175
libsafe in action . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
Chapter 9 Securing Files and Filesystems . . . . . . . . . . . . . . 179
Managing Files, Directories, and
User Group Permissions . . . . . . . . . . . . . . . . . . . . . . . . . 179
Understanding file ownership & permissions . . . . . . . . . . . . . . . 180
Changing ownership of files and directories using chown . . . . . . 181

Contents xv
Changing group ownership of files and
directories with chgrp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
Using octal numbers to set file and directory permissions . . . . . 182
Using permission strings to set access permissions . . . . . . . . . . 185
Changing access privileges of files and
directories using chmod . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
Managing symbolic links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
Managing user group permission . . . . . . . . . . . . . . . . . . . . . . . . 188
Checking Consistency of Users and Groups . . . . . . . . . . . . 190
Securing Files and Directories . . . . . . . . . . . . . . . . . . . . . . 198
Understanding filesystem hierarchy structure . . . . . . . . . . . . . . . 198
Setting system-wide default permission model using umask . . . . 201
Dealing with world-accessible files . . . . . . . . . . . . . . . . . . . . . . . 203
Dealing with set-UID and set-GID programs . . . . . . . . . . . . . . . . 204
Using ext2 Filesystem Security Features . . . . . . . . . . . . . . 208
Using chattr . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
Using lsattr . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
Using a File Integrity Checker . . . . . . . . . . . . . . . . . . . . . . 210
Using a home-grown file integrity checker . . . . . . . . . . . . . . . . . 210
Using Tripwire Open Source, Linux Edition . . . . . . . . . . . . . . . . . 215
Setting up Integrity-Checkers . . . . . . . . . . . . . . . . . . . . . . 230
Setting up AIDE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230
Setting up ICU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
Creating a Permission Policy . . . . . . . . . . . . . . . . . . . . . . . 239
Setting configuration file permissions for users . . . . . . . . . . . . . 239
Setting default file permissions for users . . . . . . . . . . . . . . . . . . . 240
Setting executable file permissions . . . . . . . . . . . . . . . . . . . . . . . 240
Chapter 10 PAM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
What is PAM? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241

Working with a PAM configuration file . . . . . . . . . . . . . . . . . . . 243
Establishing a PAM-aware Application . . . . . . . . . . . . . . . 245
Using Various PAM Modules to Enhance Security . . . . . . . 248
Controlling access by time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
Restricting access to everyone but root . . . . . . . . . . . . . . . . . . . . 257
Managing system resources among users . . . . . . . . . . . . . . . . . . 258
Securing console access using mod_console . . . . . . . . . . . . . . . . 260
Chapter 11 OpenSSL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263
Understanding How SSL Works . . . . . . . . . . . . . . . . . . . . . 263
Symmetric encryption . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264
Asymmetric encryption . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264
SSL as a protocol for data encryption . . . . . . . . . . . . . . . . . . . . . 264
Understanding OpenSSL . . . . . . . . . . . . . . . . . . . . . . . . . . 266
Uses of OpenSSL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266
Getting OpenSSL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
xvi Contents
Installing and Configuring OpenSSL . . . . . . . . . . . . . . . . . 267
OpenSSL prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
Compiling and installing OpenSSL . . . . . . . . . . . . . . . . . . . . . . . 268
Understanding Server Certificates . . . . . . . . . . . . . . . . . . . 270
What is a certificate? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
What is a Certificate Authority (CA)? . . . . . . . . . . . . . . . . . . . . . 271
Commercial CA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
Self-certified, private CA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
Getting a Server Certificate from a Commercial CA . . . . . . 273
Creating a Private Certificate Authority . . . . . . . . . . . . . . . 275
Chapter 12 Shadow Passwords and OpenSSH . . . . . . . . . . . . 277
Understanding User Account Risks . . . . . . . . . . . . . . . . . . 278
Securing User Accounts . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
Using shadow passwords and groups . . . . . . . . . . . . . . . . . . . . . 280

Checking password consistency . . . . . . . . . . . . . . . . . . . . . . . . . 282
Eliminating risky shell services . . . . . . . . . . . . . . . . . . . . . . . . . . 283
Using OpenSSH for Secured Remote Access . . . . . . . . . . . . 285
Getting and installing OpenSSH . . . . . . . . . . . . . . . . . . . . . . . . . 285
Configuring OpenSSH service . . . . . . . . . . . . . . . . . . . . . . . . . . . 286
Connecting to an OpenSSH server . . . . . . . . . . . . . . . . . . . . . . . . 293
Managing the root Account . . . . . . . . . . . . . . . . . . . . . . . . 298
Limiting root access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
Using su to become root or another user . . . . . . . . . . . . . . . . . . . 300
Using sudo to delegate root access . . . . . . . . . . . . . . . . . . . . . . . 302
Monitoring Users . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307
Finding who is on the system . . . . . . . . . . . . . . . . . . . . . . . . . . . 308
Finding who was on the system . . . . . . . . . . . . . . . . . . . . . . . . . 309
Creating a User-Access Security Policy . . . . . . . . . . . . . . . 309
Creating a User-Termination Security Policy . . . . . . . . . . . 310
Chapter 13 Secure Remote Passwords . . . . . . . . . . . . . . . . . . 313
Setting Up Secure Remote Password Support . . . . . . . . . . . 313
Establishing Exponential Password System (EPS) . . . . . . . 314
Using the EPS PAM module for password authentication . . . . . . 315
Converting standard passwords to EPS format . . . . . . . . . . . . . . 316
Using SRP-Enabled Telnet Service . . . . . . . . . . . . . . . . . . . 317
Using SRP-enabled Telnet clients
from non-Linux platforms . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319
Using SRP-Enabled FTP Service . . . . . . . . . . . . . . . . . . . . . 319
Using SRP-enabled FTP clients
from non-Linux platforms . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322
Contents xvii
Chapter 14 xinetd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323
What Is xinetd? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323
Setting Up xinetd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325

Getting xinetd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325
Compiling and installing xinetd . . . . . . . . . . . . . . . . . . . . . . . . . 325
Configuring xinetd for services . . . . . . . . . . . . . . . . . . . . . . . . . . 329
Starting, Reloading, and Stopping xinetd . . . . . . . . . . . . . 333
Strengthening the Defaults in /etc/xinetd.conf . . . . . . . . . 334
Running an Internet Daemon Using xinetd . . . . . . . . . . . . 335
Controlling Access by Name or IP Address . . . . . . . . . . . . 337
Controlling Access by Time of Day . . . . . . . . . . . . . . . . . . 338
Reducing Risks of Denial-of-Service Attacks . . . . . . . . . . . 338
Limiting the number of servers . . . . . . . . . . . . . . . . . . . . . . . . . . 338
Limiting log file size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339
Limiting load . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339
Limiting the rate of connections . . . . . . . . . . . . . . . . . . . . . . . . . 340
Creating an Access-Discriminative Service . . . . . . . . . . . . 341
Redirecting and Forwarding Clients . . . . . . . . . . . . . . . . . . 342
Using TCP Wrapper with xinetd . . . . . . . . . . . . . . . . . . . . . 345
Running sshd as xinetd . . . . . . . . . . . . . . . . . . . . . . . . . . . 345
Using xadmin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346
Part IV Network Service Security
Chapter 15 Web Server Security . . . . . . . . . . . . . . . . . . . . . . 351
Understanding Web Risks . . . . . . . . . . . . . . . . . . . . . . . . . 351
Configuring Sensible Security for Apache . . . . . . . . . . . . . 352
Using a dedicated user and group for Apache . . . . . . . . . . . . . . . 352
Using a safe directory structure . . . . . . . . . . . . . . . . . . . . . . . . . . 352
Using appropriate file and directory permissions . . . . . . . . . . . . 354
Using directory index file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356
Disabling default access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 358
Disabling user overrides . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 358
Using Paranoid Configuration . . . . . . . . . . . . . . . . . . . . . . 359
Reducing CGI Risks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 360

Information leaks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 360
Consumption of system resources . . . . . . . . . . . . . . . . . . . . . . . . 360
Spoofing of system commands via CGI scripts . . . . . . . . . . . . . . 361
Keeping user input from making system calls unsafe . . . . . . . . . 361
User modification of hidden data in HTML pages . . . . . . . . . . . . 366
Wrapping CGI Scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . 372
suEXEC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 372
CGIWrap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 375
Hide clues about your CGI scripts . . . . . . . . . . . . . . . . . . . . . . . . 377
xviii Contents
Reducing SSI Risks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 378
Logging Everything . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 379
Restricting Access to Sensitive Contents . . . . . . . . . . . . . . 382
Using IP or hostname . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 382
Using an HTTP authentication scheme . . . . . . . . . . . . . . . . . . . . 385
Controlling Web Robots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 390
Content Publishing Guidelines . . . . . . . . . . . . . . . . . . . . . . 392
Using Apache-SSL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 394
Compiling and installing Apache-SSL patches . . . . . . . . . . . . . . 394
Creating a certificate for your Apache-SSL server . . . . . . . . . . . . 395
Configuring Apache for SSL . . . . . . . . . . . . . . . . . . . . . . . . . . . . 396
Testing the SSL connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 398
Chapter 16 DNS Server Security . . . . . . . . . . . . . . . . . . . . . . 399
Understanding DNS Spoofing . . . . . . . . . . . . . . . . . . . . . . 399
Checking DNS Configuring Using Dlint . . . . . . . . . . . . . . . 400
Getting Dlint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 401
Installing Dlint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 401
Running Dlint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 402
Securing BIND . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405
Using Transaction Signatures (TSIG) for zone transfers . . . . . . . . 405

Running BIND as a non-root user . . . . . . . . . . . . . . . . . . . . . . . . 409
Hiding the BIND version number . . . . . . . . . . . . . . . . . . . . . . . . 409
Limiting Queries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 410
Turning off glue fetching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 411
chrooting the DNS server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 412
Using DNSSEC (signed zones) . . . . . . . . . . . . . . . . . . . . . . . . . . . 412
Chapter 17 E-Mail Server Security . . . . . . . . . . . . . . . . . . . . 415
What Is Open Mail Relay? . . . . . . . . . . . . . . . . . . . . . . . . . 415
Is My Mail Server Vulnerable? . . . . . . . . . . . . . . . . . . . . . . 417
Securing Sendmail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 419
Controlling mail relay . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 422
Enabling MAPS Realtime Blackhole List (RBL) support . . . . . . . . 425
Sanitizing incoming e-mail using procmail . . . . . . . . . . . . . . . . 429
Outbound-only Sendmail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 437
Running Sendmail without root privileges . . . . . . . . . . . . . . . . . 438
Securing Postfix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 440
Keeping out spam . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 440
Hiding internal e-mail addresses by masquerading . . . . . . . . . . . 442
Chapter 18 FTP Server Security . . . . . . . . . . . . . . . . . . . . . . . 443
Securing WU-FTPD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 443
Restricting FTP access by username . . . . . . . . . . . . . . . . . . . . . . 445
Setting default file permissions for FTP . . . . . . . . . . . . . . . . . . . 447
Contents xix
Using a chroot jail for FTP sessions . . . . . . . . . . . . . . . . . . . . . . 448
Securing WU-FTPD using options in /etc/ftpaccess . . . . . . . . . . . 452
Using ProFTPD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 455
Downloading, compiling, and installing ProFTPD . . . . . . . . . . . . 456
Configuring ProFTPD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 456
Monitoring ProFTPD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 462
Securing ProFTPD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 462

Chapter 19 Samba and NFS Server Security . . . . . . . . . . . . . 473
Securing Samba Server . . . . . . . . . . . . . . . . . . . . . . . . . . . 473
Choosing an appropriate security level . . . . . . . . . . . . . . . . . . . . 473
Avoiding plain-text passwords . . . . . . . . . . . . . . . . . . . . . . . . . . 476
Allowing access to users from trusted domains . . . . . . . . . . . . . . 477
Controlling Samba access by network interface . . . . . . . . . . . . . 477
Controlling Samba access by hostname or IP addresses . . . . . . . 478
Using pam_smb to authenticate all users
via a Windows NT server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 479
Using OpenSSL with Samba . . . . . . . . . . . . . . . . . . . . . . . . . . . . 481
Securing NFS Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 483
Using Cryptographic Filesystems . . . . . . . . . . . . . . . . . . . . 487
Part V Firewalls
Chapter 20 Firewalls, VPNs, and SSL Tunnels . . . . . . . . . . . . 491
Packet-Filtering Firewalls . . . . . . . . . . . . . . . . . . . . . . . . . 491
Enabling netfilter in the kernel . . . . . . . . . . . . . . . . . . . . . . . . . . 496
Creating Packet-Filtering Rules with iptables . . . . . . . . . . . 498
Creating a default policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 498
Appending a rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 498
Listing the rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 499
Deleting a rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 500
Inserting a new rule within a chain . . . . . . . . . . . . . . . . . . . . . . . 500
Replacing a rule within a chain . . . . . . . . . . . . . . . . . . . . . . . . . . 500
Creating SOHO Packet-Filtering Firewalls . . . . . . . . . . . . . 501
Allowing users at private network access
to external Web servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 504
Allowing external Web browsers access to a Web server
on your firewall . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 505
DNS client and cache-only services . . . . . . . . . . . . . . . . . . . . . . 506
SMTP client service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 508

POP3 client service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 508
Passive-mode FTP client service . . . . . . . . . . . . . . . . . . . . . . . . . 509
SSH client service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 510
Other new client service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 510
xx Contents
Creating a Simple Firewall . . . . . . . . . . . . . . . . . . . . . . . . . 511
Creating Transparent, proxy-arp Firewalls . . . . . . . . . . . . . 512
Creating Corporate Firewalls . . . . . . . . . . . . . . . . . . . . . . . 514
Purpose of the internal firewall . . . . . . . . . . . . . . . . . . . . . . . . . . 515
Purpose of the primary firewall . . . . . . . . . . . . . . . . . . . . . . . . . . 515
Setting up the internal firewall . . . . . . . . . . . . . . . . . . . . . . . . . . 516
Setting up the primary firewall . . . . . . . . . . . . . . . . . . . . . . . . . . 518
Secure Virtual Private Network . . . . . . . . . . . . . . . . . . . . . 528
Compiling and installing FreeS/WAN . . . . . . . . . . . . . . . . . . . . . 529
Creating a VPN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 530
Stunnel: A Universal SSL Wrapper . . . . . . . . . . . . . . . . . . 536
Compiling and installing Stunnel . . . . . . . . . . . . . . . . . . . . . . . . 536
Securing IMAP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 536
Securing POP3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 538
Securing SMTP for special scenarios . . . . . . . . . . . . . . . . . . . . . . 539
Chapter 21 Firewall Security Tools . . . . . . . . . . . . . . . . . . . . 541
Using Security Assessment (Audit) Tools . . . . . . . . . . . . . . 541
Using SAINT to Perform a Security Audit . . . . . . . . . . . . . . . . . . 541
SARA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 549
VetesCan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 550
Using Port Scanners . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 550
Performing Footprint Analysis Using nmap . . . . . . . . . . . . . . . . 550
Using PortSentry to Monitor Connections . . . . . . . . . . . . . . . . . . 552
Using Nessus Security Scanner . . . . . . . . . . . . . . . . . . . . . . . . . . 558
Using Strobe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 561

Using Log Monitoring and Analysis Tools . . . . . . . . . . . . . 562
Using logcheck for detecting unusual log entries . . . . . . . . . . . . 562
Swatch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 565
IPTraf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 565
Using CGI Scanners . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 566
Using cgichk.pl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 566
Using Whisker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 568
Using Malice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 569
Using Password Crackers . . . . . . . . . . . . . . . . . . . . . . . . . . 569
John The Ripper . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 570
Crack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 571
Using Intrusion Detection Tools . . . . . . . . . . . . . . . . . . . . . 571
Tripwire . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 571
LIDS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 571
Using Packet Filters and Sniffers . . . . . . . . . . . . . . . . . . . . 572
Snort . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 572
GShield . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 575
Contents xxi
Useful Utilities for Security Administrators . . . . . . . . . . . . 575
Using Netcat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 575
Tcpdump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 580
LSOF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 581
Ngrep . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 586
Appendix A IP Network Address Classification . . . . . . . . . . . . 589
Appendix B Common Linux Commands . . . . . . . . . . . . . . . . . 593
Appendix C Internet Resources . . . . . . . . . . . . . . . . . . . . . . . . 655
Appendix D Dealing with Compromised Systems . . . . . . . . . . 661
Appendix E What’s On the CD-ROM? . . . . . . . . . . . . . . . . . . . 665
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 669
End-User License Agreement . . . . . . . . . . . . . . . . . . . . 691

xxii Contents
System Performance
CHAPTER 1
Performance Basics
CHAPTER 2
Kernel Tuning
CHAPTER 3
Filesystem Tuning
Part
I