1727 the book of IMAP

BUILD A RELIABLE
SERVER WITH IMAP

The Book of IMAP offers a detailed introduction to IMAP
and POP3, the two protocols that govern all modern
mail servers and clients. You’ll learn how the protocols
work as well as how to install, configure, and maintain
the two most popular open source mail systems, Courier
and Cyrus.
Authors Peer Heinlein and Peer Hartleben have set up
hundreds of mail servers and offer practical hints about
troubleshooting errors, migration, filesystem tuning,
cluster setups, and password security that will help you
extricate yourself from all sorts of tricky situations. You’ll
also learn how to:
• Create and use shared folders, virtual domains, and
user quotas

• Use built-in tools for server analysis, maintenance,
and repairs
• Implement complementary webmail clients like
Squirrelmail and Horde/IMP
• Set up and use the Sieve email filter
Thoroughly commented references to the POP and IMAP
protocols round out the book, making The Book of IMAP
an essential resource for even the most experienced
system administrators.
ABOUT THE AUTHORS

Peer Heinlein has been operating an independent ISP
in Berlin since 1992. He specializes in mail servers of

various sizes and enjoys ambitious Linux projects. Peer
Hartleben is a CTO and Linux Security Consultant with
a focus on Cyrus-based mail servers.

H A R T L E BE N

$49.95 ($54.95 CDN)
SHELVE IN:
EMAIL

w w w.nostarch.com

• Handle heavy traffic with load balancers and proxies

B U I L D I N G

A

M A I L

BOOK
IMAP
S E R V E R

A N D

COURIER

PEER HEINLEIN AND PEER HARTLEBEN

“ I L AY F L AT .”
This book uses RepKover — a durable binding that won’t snap shut.

W I T H

CYRUS

H E INL E IN A ND

T H E F I N E ST I N G E E K E N T E RTA I N M E N T ™

• Authenticate user data with PAM, MySQL, PostgreSQL,
and LDAP

THE BOOK OF IMAP

IMAP (the Internet Message Access Protocol) allows
clients to access their email on a remote server,
whether from the office, a remote location, or a cell
phone or other device. IMAP is powerful and flexible,
but it’s also complicated to set up; it’s more difficult to
implement than POP3 and more error-prone for both
client and server.

THE
OF

®

Printed on recycled paper

www.it-ebooks.info


www.it-ebooks.info


The Book of IMAP

www.it-ebooks.info


www.it-ebooks.info


Peer Heinlein

Peer Hartleben

The Book of IMAP
Building a Mail Server with Courier and Cyrus

Munich

San Francisco

www.it-ebooks.info


The Book of IMAP: Building a Mail Server with Courier and Cyrus.

Press GmbH

Copyright © 2008 Open Source

All rights reserved. No part of this work may be reproduced or transmitted in any form or by any means,
electronic or mechanical, including photocopying, recording, or by any information storage or retrieval
system, without the prior written permission of the copyright owner and the publisher.
Printed on recycled paper in the United States of America.
1 2 3 4 5 6 7 8 9 10 — 08 07 06 05
No Starch Press and the No Starch Press logo are registered trademarks of No Starch Press, Inc. Other
product and company names mentioned herein may be the trademarks of their respective owners.
Rather than use a trademark symbol with every occurrence of a trademarked name, we are using the
names only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark.
Publisher: William Pollock
Cover Design: Octopod Studios
U.S. edition published by No Starch Press, Inc.
555 De Haro Street, Suite 250, San Francisco, CA 94107
phone: 415.863.9900; fax: 415.863.9950; ;
Original edition © 2007 Open Source Press GmbH
Published by Open Source Press GmbH, Munich, Germany
Publisher: Dr. Markus Wirtz
Original ISBN 978-3-937514-11-6
For information on translations, please contact
Open Source Press GmbH, Amalienstr. 45 Rg, 80799 München, Germany
phone +49.89.28755562; fax +49.89.28755563; ;
The information in this book is distributed on an “As Is” basis, without warranty. While every precaution
has been taken in the preparation of this work, neither the author nor Open Source Press GmbH nor
No Starch Press, Inc. shall have any liability to any person or entity with respect to any loss or damage
caused or alleged to be caused directly or indirectly by the information contained in it.
Library of Congress Cataloging-in-Publication Data

Heinlein, Peer
[POP3 und IMAP. English]
The book of IMAP: building a mail server with Courier and Cyrus / Peer Heinlein
and Peer Hartleben.-p. cm.
Includes index.
ISBN-13: 978-1-59327-177-0
ISBN-10: 1-59327-177-8
1. Electronic mail systems-Computer programs. 2. Electronic mail
systems-Standards. 3. Computer network protocols. 4. Web servers. I.
Hartleben, Peer. II. Title.
TK5105.73.H45 2008
004.692-dc22
2008012396

www.it-ebooks.info


Contents

Introduction

13

I

15

How To Set Up and Maintain IMAP Servers

1 Protocols and Terms

17

1.1 Why Is IMAP So Complex? . . . . . . . . . . . . . . . . . . . . . . 19
1.2 Comparing Courier and Cyrus . . . . . . . . . . . . . . . . . . . . 20
2 POP3 and IMAP at the Protocol Level

23

2.1 POP3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.1.1

Test Session . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

2.1.2

Authentication via APOP and KPOP . . . . . . . . . . . . 27

2.2 IMAP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.2.1

The Design of the IMAP Protocol . . . . . . . . . . . . . . 29

2.2.2

Transcript of an IMAP Session . . . . . . . . . . . . . . . . 31

2.2.3

A Practical View of IMAP . . . . . . . . . . . . . . . . . . . 33

2.2.4

Subscribing to IMAP Folders . . . . . . . . . . . . . . . . . 41

3 Load Distribution and Reliability

43

3.1 Load Balancer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3.1.1

DNS Round Robin . . . . . . . . . . . . . . . . . . . . . . . 46

3.1.2

Round Robin via ÔØ

3.1.3

Linux Virtual Server . . . . . . . . . . . . . . . . . . . . . . 47

Ð × . . . . . . . . . . . . . . . . . 46

3.2 IMAP Proxies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

5

www.it-ebooks.info

Contents

4 Selecting a Filesystem
4.1 A Performance Test

53
. . . . . . . . . . . . . . . . . . . . . . . . . . 55

4.2 Tuning the Performance of the Filesystem . . . . . . . . . . . . . 57
4.2.1

The Ø Ñ . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

4.2.2

Access Control Lists . . . . . . . . . . . . . . . . . . . . . . 58

4.2.3

The Ext2/Ext3 Option

4.2.4

Journal Mode . . . . . . . . . . . . . . . . . . . . . . . . . . 60

4.2.5

Optimized ×Ø

Ö Ò Ü . . . . . . . . . . . . . . 58

Entries . . . . . . . . . . . . . . . . . . . 62

4.3 RAID . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
4.4 NFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
4.4.1

Disabling Ø Ñ and Optimizing Block Size . . . . . . . . 64

4.4.2

NFS Version 3 . . . . . . . . . . . . . . . . . . . . . . . . . . 64

4.4.3

Fast I/O . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

5 Complementary Webmail Clients

67

5.1 Squirrelmail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
5.2 Horde/IMP

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

5.3 Fast Access via the IMAP Cache Proxy . . . . . . . . . . . . . . . 73
6 Migrating IMAP servers
6.1 Migration Using Ñ Ô×ÝÒ

75
. . . . . . . . . . . . . . . . . . . . . . 76

6.2 Converting mbox to maildir . . . . . . . . . . . . . . . . . . . . . 78
6.3 Modifying Folder Names . . . . . . . . . . . . . . . . . . . . . . . 79
6.4 Determining Cleartext Passwords . . . . . . . . . . . . . . . . . . 81

II

Courier-IMAP

7 Structure and Basic Configuration

83
85

7.1 Installing the Software . . . . . . . . . . . . . . . . . . . . . . . . . 86
7.2 What Is Where? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
7.3 Initial Start-Up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
7.4 Courier and MTAs . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
7.4.1

Courier and Postfix . . . . . . . . . . . . . . . . . . . . . . 92

7.4.2

Courier and QMail . . . . . . . . . . . . . . . . . . . . . . . 94

7.4.3

Courier and Exim . . . . . . . . . . . . . . . . . . . . . . . 94

6

www.it-ebooks.info


Contents

7.5 Optimizing the Configuration . . . . . . . . . . . . . . . . . . . . 95
7.5.1

Real and “False” Configuration Parameters . . . . . . . . 96

Ö»ÔÓÔ¿ . . . . . . 96
7.5.3 Configuring the IMAP Daemon in » Ø » ÓÙÖ Ö»
Ñ Ô . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
7.5.2

POP3 Configuration in » Ø » ÓÙÖ

7.6 The Configuration Files for SSL . . . . . . . . . . . . . . . . . . . 102
8 Maildir as Email Storage Format

107

8.1 The IMAP Namespace . . . . . . . . . . . . . . . . . . . . . . . . . 110
8.2 Filenames of Emails . . . . . . . . . . . . . . . . . . . . . . . . . . 111
8.2.1

Keywords: Custom IMAP Flags . . . . . . . . . . . . . . . 115

9 User Data

119

ÙØ Ø ×Ø and
Í ÄÇ ÁÆ for Debugging Assistance . . . . . 121
ÑÓÒ . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
9.2 The ÙØ
9.1

9.3 Authentication via PAM . . . . . . . . . . . . . . . . . . . . . . . . 123
9.4 The ÙØ Ù× Ö

Module . . . . . . . . . . . . . . . . . . . . . . . 124

9.4.1

Converting Ô ××Û into a Ù× Ö

9.4.2

Maintaining Account Data with Ù× Ö

9.4.3

Creating a Binary Version of the User Database . . . . . 128

9.4.4

Separating the Ù× Ö

9.4.5

The Ø Ñ . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130

9.5 Using QMail’s Ú

. . . . . . . . . . . . . 125
. . . . . . . . . . 127

into Multiple Files . . . . . . . . . 129

ÔÛ Library for Authentication . . . . . . . . 130

9.6 Implementing Custom Authentication Methods . . . . . . . . . 130
9.7 Integrating External Authentication Programs

. . . . . . . . . . 131

9.8 Authentication via MySQL . . . . . . . . . . . . . . . . . . . . . . 133
9.9 Authentication via PostgreSQL . . . . . . . . . . . . . . . . . . . . 139
9.10 Authentication via LDAP . . . . . . . . . . . . . . . . . . . . . . . 140
9.11 Obsolete Authentication Modules . . . . . . . . . . . . . . . . . . 143
9.11.1 The ÙØ ÔÛ Module . . . . . . . . . . . . . . . . . . . . . 143
9.11.2 The ÙØ ×
9.11.3 The ÙØ

ÓÛ Module . . . . . . . . . . . . . . . . . . . 143
Ö Ñ Module . . . . . . . . . . . . . . . . . . . . 144

9.12 User Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
9.12.1 Saving User Options in the Ù× Ö

. . . . . . . . . . . . . 146

9.12.2 Individual User Options in an LDAP Directory . . . . . . 146

7

www.it-ebooks.info


Contents

9.12.3 Storing User Options in Dedicated Fields in an SQL
Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
9.13 Saving Passwords: Cleartext or Hash? . . . . . . . . . . . . . . . . 147
9.14 Username Selection When Maintaining Multiple Domains . . . 150
10 The Work of a Courier Administrator

153

10.1 Shared Folders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
10.1.1 Setting Up Virtual Shared Folders . . . . . . . . . . . . . . 154
10.1.2 Creating Filesystem-Based Shared Folders . . . . . . . . 163
10.2 Quotas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
10.2.1 Quotas for Courier . . . . . . . . . . . . . . . . . . . . . . . 167

10.2.2 Quotas and the MDA . . . . . . . . . . . . . . . . . . . . . 172
10.3 Building an IMAP Proxy with Courier . . . . . . . . . . . . . . . . 175
10.4 Push Instead of Pull: The Á Ä Command . . . . . . . . . . . . . 176
10.5 Sending Emails via the IMAP Server

III

Cyrus-IMAP

. . . . . . . . . . . . . . . . 178

181

11 Structure and Basic Configuration

183

11.1 Installing Cyrus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
11.1.1 OpenSuSE/SuSE Linux Enterprise Server (SLES) . . . . . 185
11.1.2 Fedora Core/Red Hat . . . . . . . . . . . . . . . . . . . . . 186
11.1.3 Debian . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
11.2 The Cyrus Hierarchy and Permissions System . . . . . . . . . . . 187
11.3 Features and Functions . . . . . . . . . . . . . . . . . . . . . . . . 188
11.4 Quick Start . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
11.4.1 Authentication and Mailboxes . . . . . . . . . . . . . . . . 194
11.4.2 Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
12 A Closer Look at the Configuration Files
12.1 » Ø » ÝÖÙ׺ ÓÒ

199

. . . . . . . . . . . . . . . . . . . . . . . . . . . 199

12.1.1 The ËÌ ÊÌß Section . . . . . . . . . . . . . . . . . . . . . 200
12.1.2 The Ë ÊÎÁ

Ëß Section . . . . . . . . . . . . . . . . . . . 200

12.1.3 The Î ÆÌËß Section . . . . . . . . . . . . . . . . . . . . 201
12.2 » Ø » Ñ Ô º ÓÒ

. . . . . . . . . . . . . . . . . . . . . . . . . . . 203

8

www.it-ebooks.info


Contents

13 Authentication and Safeguards

207

13.1 Encrypting with SSL/TLS . . . . . . . . . . . . . . . . . . . . . . . 208
13.1.1 SSL Transmission Types . . . . . . . . . . . . . . . . . . . . 208
13.1.2 Real and Fake Certificates . . . . . . . . . . . . . . . . . . 208
13.1.3 Creating and Integrating SSL Certificates . . . . . . . . . 209
13.2 Cyrus SASL

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211

13.2.1 Cyrus SASL Modules . . . . . . . . . . . . . . . . . . . . . . 212
13.2.2 The ÙÜÔÖÓÔ Module . . . . . . . . . . . . . . . . . . . . . 213
13.2.3 The Authentication Process . . . . . . . . . . . . . . . . . 214
13.3 Calling Different Data Sources . . . . . . . . . . . . . . . . . . . . 215
13.3.1 Standard Authentication Methods for Unix . . . . . . . . 215
13.3.2 × ×Ð

¾ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216

13.3.3 Cyrus and MySQL . . . . . . . . . . . . . . . . . . . . . . . 216
13.3.4 Cyrus and LDAP . . . . . . . . . . . . . . . . . . . . . . . . 220
13.3.5 Cyrus and Kerberos . . . . . . . . . . . . . . . . . . . . . . 223
14 Advanced Cyrus Configuration

225

14.1 Mailbox Quotas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
14.1.1 Automatic Quotas . . . . . . . . . . . . . . . . . . . . . . . 226
14.1.2 Manual Quotas . . . . . . . . . . . . . . . . . . . . . . . . . 228
14.2 Shared Folders and ACLs . . . . . . . . . . . . . . . . . . . . . . . 230
14.3 Virtual Domains . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232
14.3.1 The Underlying Concept . . . . . . . . . . . . . . . . . . . 232
14.3.2 Effects on ACLs . . . . . . . . . . . . . . . . . . . . . . . . . 236
14.3.3 Domain Administrators . . . . . . . . . . . . . . . . . . . . 237
14.4 Sorting Emails into Subdirectories

. . . . . . . . . . . . . . . . . 237

14.5 Email Partitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
14.6 The Sieve Email Filter . . . . . . . . . . . . . . . . . . . . . . . . . 240
14.6.1 The Email Filter Daemon Ø Ñ×

Ú

. . . . . . . . . . . 240

14.6.2 Configuring and Testing . . . . . . . . . . . . . . . . . . . 240
14.6.3 The ×

Ú ×

ÐÐ Administration Tool . . . . . . . . . . . 242

14.6.4 The Sieve Script Language . . . . . . . . . . . . . . . . . . 246
14.6.5 Setting Up Sieve Scripts Automatically for New Accounts 251
14.6.6 Adapting Sieve Scripts

. . . . . . . . . . . . . . . . . . . . 252

9

www.it-ebooks.info


Contents

14.7 The ÒÓØ

Ý Daemon . . . . . . . . . . . . . . . . . . . . . . . . . 252

14.7.1 Drums or Smoke Signals? . . . . . . . . . . . . . . . . . . . 253
14.8 Cyrus and Other MTAs

. . . . . . . . . . . . . . . . . . . . . . . . 254

14.9 Backing Up and Restoring Data . . . . . . . . . . . . . . . . . . . 255
14.9.1 Using Ö

ÓÒ×ØÖÙ Ø to Repair Mailboxes . . . . . . . . . 255

14.9.2 Restoring Quotas . . . . . . . . . . . . . . . . . . . . . . . . 257
14.10 Performance Tuning . . . . . . . . . . . . . . . . . . . . . . . . . 257
14.10.1 Parameters in » Ø » Ñ Ô º ÓÒ that influence performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258

15 Internal Structure and Modules

261

15.1 The Cyrus Daemons . . . . . . . . . . . . . . . . . . . . . . . . . . 262
15.2 Tools for Analysis, Maintenance, and Repairs . . . . . . . . . . . 263
15.2.1 Statistics and Analysis . . . . . . . . . . . . . . . . . . . . . 263
15.2.2 Maintenance and Repair . . . . . . . . . . . . . . . . . . . 266
15.2.3 Internal Tools . . . . . . . . . . . . . . . . . . . . . . . . . . 268
15.3 Other In-House Tools . . . . . . . . . . . . . . . . . . . . . . . . . 269
15.4 The ÝÖ

Ñ Administration Tool . . . . . . . . . . . . . . . . . . 271

16 Cyrus at the Filesystem Level

275

16.1 The Email Directory . . . . . . . . . . . . . . . . . . . . . . . . . . 275
16.2 The Administration Directory . . . . . . . . . . . . . . . . . . . . 277

17 Cyrus in a Cluster

281

17.1 The Cyrus Aggregator . . . . . . . . . . . . . . . . . . . . . . . . . 281
17.1.1 The Aggregator Concept . . . . . . . . . . . . . . . . . . . 282
17.1.2 The Cluster Setup . . . . . . . . . . . . . . . . . . . . . . . 283
17.2 Cyrus Replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291
17.2.1 Replicating the Authentication Data . . . . . . . . . . . . 291

10

www.it-ebooks.info


Contents

Appendixes

293

A IMAP Command Reference

295

A.1 Commands Always Available to Clients . . . . . . . . . . . . . . . 296
A.2 Commands Available in the Not-Authenticated Status . . . . . . 297
A.3 Commands Available in the Authenticated Status . . . . . . . . 298
A.4 Commands Available in the Selected Status . . . . . . . . . . . . 303
A.5 IMAP Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314
A.6 Experimental Commands . . . . . . . . . . . . . . . . . . . . . . . 316
B POP3 Command Reference

317

B.1 An Overview of All Commands . . . . . . . . . . . . . . . . . . . . 318
C Installing from the Source Code

321

C.1 Courier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321
C.2 Cyrus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325
C.2.1 Cyrus Sources . . . . . . . . . . . . . . . . . . . . . . . . . . 325
C.2.2 Creating a System User . . . . . . . . . . . . . . . . . . . . 325
C.2.3 Installing Cyrus SASL . . . . . . . . . . . . . . . . . . . . . 326
C.2.4 Installing the Cyrus IMAP Server . . . . . . . . . . . . . . 329
C.2.5 Convenient Starting and Stopping . . . . . . . . . . . . . 330

11

www.it-ebooks.info

www.it-ebooks.info


Introduction
There is very little specialist literature available on IMAP servers, and no
current documentation deals with the subject in sufficient depth.
There is a real need for a guide to IMAP. A quick look at relevant mailing
lists shows that they are full of questions and problems, indicating that the
software solutions now in use raise many issues. IMAP may seem to be a
simple affair and to require little in the way of configuration, but there are
plenty of pitfalls when an IMAP server is designed for a large number of
users or when elaborate additional features are added to a basic installation.
We have specialized in Courier and Cyrus during the last few years. Both
offer distinct advantages and disadvantages, so the appropriate choice of
software depends on the project. Peer Heinlein mainly works with Courier
IMAP, and he uses it to implement mail servers for large ISPs that are
designed to accommodate tens or hundreds of thousands of users. Peer
Hartleben uses Cyrus IMAP for mail servers in small and large companies, which require Cyrus user administration—sometimes via a console—
and server-based filtering of mail using Sieve. Peer Heinlein has therefore
written the introduction and the Courier section of this book, and Peer
Hartleben has written the section on Cyrus.
Neither Courier nor Cyrus have had suitable documentation (until now).
We have to admit: This book was hard work. There were many behaviors
and call parameters that we had to debug and test by trial and error, or
understand by analyzing the source code, because their significance was
not documented anywhere. The project mailing lists often were not helpful,
frequently containing more questions than answers.
The detailed work on this book took far longer than we had originally suspected it would, and there were repeated delays in publication. But, finally,
we have an exhaustive and up-to-date reference on the subjects of IMAP,
Courier, and Cyrus. Considering the importance of email communication,

we hope that this book will help many administrators and postmasters in
their work.
This book is in its first edition and is still not truly complete. We had to
postpone discussion of some small details until the second edition. Also,

13

www.it-ebooks.info


Introduction

when interpreting behavior that had no or insufficient documentation, we
ran as many tests as possible to try to gain an accurate picture; nevertheless, we cannot rule out errors and omissions.
We will therefore provide corrections and additions at ØØÔ »»ÛÛÛº Ñ Ô¹
Ù º ÓÑ». You are very welcome to leave helpful suggestions, references,
or corrections for us there. This kind of help is very important to us. Please
tell us which subjects you found interesting, which topics remained unclear
after you read our explanations, and where you suspect we made a mistake.
The website contains a link to the mailing list Ñ Ô¹ Ù , which, we hope,
will soon develop into a lively and competent discussion.
Once this book has been sent to the printers, many people will heave a
large sigh of relief. We are very grateful to these people. First, we have to
thank our editor, Patricia Jung, for her perseverance in adding the finishing
touches and questioning every detail. We authors often despaired of relief
from her scrutiny, but she is the reason for the high quality of this book.
Thanks to her specialist knowledge, she also was able to provide many suggestions and explanations.
The rest of the Open Source Press team, Markus Wirtz and Ulrich Wolf, also
played an important part in making this book a reality—and gained not a
few grey hairs during the process. (Sorry!)

Arnt Gulbrandsen and his detailed knowledge of IMAP were also a great
help, and we would like to thank him for his commitment. We would also
like to thank Frank Richter from TU Chemnitz for helping us in our battle
with the Cyrus cluster.
We also had behind-the-scenes help from members of the Heinlein Support
team, who did the preliminary work, made measurements (to be honest,
they did the heavy lifting), researched details, and did some of our own
tasks as well, so that we were able to concentrate on the book itself: Thank
you, Stefan, Holger, Chrizz, Henri, Christian, Matthias, and Christiane.
As customary when writing a book, we want to thank our loved ones, and
tell them that “it will all get better now.” So we would like to tell our four
ladies: Now we can spend more time with you again! Thanks to our grownup ladies, Anja and Ivonne, who have gone through the whole process with
us, and who have had to live with the book hanging over every moment
of spare time like the sword of Damocles. And to our tiny women: Lara
Hartleben, who does not yet sleep through the night, and Heinlein Junior,
who does not yet have a name but has already entered the first contest of
her life: Who will be born first—her or The Book of IMAP? It seems as if the
book will win . . .

Peer Heinlein and Peer Hartleben

14

www.it-ebooks.info

Berlin, September 2007


Part I

How To Set Up and Maintain IMAP
Servers

www.it-ebooks.info


www.it-ebooks.info


Chapter

1

Protocols and Terms
What is a mail server? This term could describe the particular machine in a
computer center that is responsible for sending and receiving users’ emails.
However, such a mail server actually consists of a variety of components
and programs, which use various protocols to communicate among themselves. The same is true of commercial software applications that combine
the many necessary functions into one product. In most cases, the different mail server tasks can be distributed to more than one computer, which
means that “the mail server” may in fact consist of several machines that
together fulfill the different functions.
Mail servers use the Simple Mail Transport Protocol (SMTP) to communicate and to deliver emails. Clients such as Outlook, KMail, Thunderbird,
and Evolution usually deliver emails to the relay server via SMTP. However, SMTP is suitable only for sending emails, not for receiving them. This
means that SMTP cannot be used to query a mailbox or create email directories in it. Likewise, a Mail Transfer Agent (MTA) transports emails received

17

www.it-ebooks.info

1 Protocols and Terms

from clients or other servers to their destination, but has nothing to do with
mail-receiving protocols such as POP3 or IMAP. This book will not deal with
SMTP servers such as Postfix, QMail, Exim, or Sendmail, except marginally.
We simply assume that they work correctly.1
Post Office Protocol Version 3 (POP3) is a comparatively simple protocol
with few configuration options, so pure POP3 servers require very little administration. Once they have been started or entered in the configuration
of the X(Inet) Daemon, emails can be retrieved on port 110 using POP3.
Internet Message Access Protocol (IMAP), the “grown-up” version of POP3, is
far more complex. This book will focus on the numerous ways that email
retrieval can be configured and on the administration of emails. Once you
have read it, you will be able to implement even demanding mail-handling
scenarios. Common IMAP servers also contain a small POP3 daemon, so
programmers who deal with the complexity of managing the IMAP protocol
will have no difficulty in providing a POP3 protocol “on the side.”
This book deals with the two most common open source IMAP protocols:
Courier IMAP and Cyrus IMAP. Both contain a POP3 server. When we refer to “IMAP servers,” we mean both services, unless we are specifically
discussing the features of one of them.
The Local Message Transfer Protocol (LMTP) is closely related to SMTP, but
it is only used locally; for example, to transfer an email from a Mail Transfer
Agent (such as Postfix) to another component of the mail system, specifically to a Mail Delivery Agent (MDA). In this case, LMTP has an advantage
over SMTP: With LMTP, it is possible to determine the email addresses for
which a local transfer succeeds. Unlike SMTP, LMTP returns a status message for each recipient after the Ì command. The status message specifies the mailbox in which the email was actually saved. SMTP only indicates
whether the server was able to place an email for the recipient in a queue
for delivery at a later point in time. LMTP is also better in high-performance
environments.
You should only use LMTP locally within your own network; for example, to
transfer emails from the front relay (which receives via SMTP) to the actual
mail backend that does the saving. The protocol can be used, for example,

to connect Cyrus to the MTA (see section 11.4 on page 191).
Groupware is software that manages tasks, calendars, email contacts, and
address books on behalf of multiple users. Depending on the version, it
can also manage resources, rooms, files, or other kinds of objects. Email is
thus only a part of the functionality offered by groupware, but groupware
usually contains an email service. The IMAP servers introduced here are
not groupware servers, but some free groupware solutions such as Kolab,
OpenGroupware, and eGroupWare are based on IMAP, so this book may be
helpful when adapting such software to your needs as well.
1

See The Book of Postfix (No Starch Press, 2005) by Ralf Hildebrandt and Patrick Koetter
for more information on this subject.

18

www.it-ebooks.info


1.1 Why Is IMAP So Complex?

1.1 Why Is IMAP So Complex?
The POP3 server waits until a user has logged on and then transfers the
unread messages that are saved in that user’s mailbox to the user’s mail
client. Depending on the client requirements, the messages are deleted
after transfer to save space, or flagged as read and retained. This is not
particularly demanding, so there is not much that can go wrong with the
software here.
An IMAP server operates differently: Not only does it deliver emails to users,
it also organizes the entire end-user email administration. The user’s email

client now functions as a kind of “remote control” for manipulating the
mailbox stored on the server.
An IMAP server provides storage space and stores all emails. For this reason, it makes sense to use quotas, which force users to clean up occasionally and free up valuable space. When a user creates folders for his or her
emails, the IMAP server has to represent this folder structure and sort the
emails correspondingly. IMAP also enables users to search messages for
specified senders or text and to flag emails, for example, as read, unread, or
answered. Users can also access shared folders in parallel.
An IMAP server enables a user to manage a mailbox from different computers; the contents of the mailbox always consist of the same data records
no matter where it is accessed from, and the mailbox does not need to be
synchronized among the machines.
All these features make great demands on the IMAP protocol and the programmer. The configuration of an IMAP server does not require much attention from the administrator once the server has been connected to a
user database. However, the operation of IMAP servers does contain a few
traps and technical difficulties, which we will examine in this book:
Performance
As the number of users increases, the load on the IMAP server becomes noticeable. In a large organization, the server has to manage
millions of emails, operate hundreds or thousands of IMAP connections in parallel, and deal with email searches and extensive copying
actions. Depending on the scenario, IMAP servers can consume considerable RAM or create high I/O loads on the data carriers.
Availability
Nowadays, email needs to be available around the clock, as any extended outage can endanger business. Once a certain number of
users has been reached, the infrastructure should be secured by using multiple servers, even if a robust IMAP server has been selected.
Storage
Email storage can increase to sizeable proportions, which necessi-

19

www.it-ebooks.info


1 Protocols and Terms

tates the use of an NAS or SAN. Also, when an IMAP server is part of a
server cluster, it is no longer sufficient to use a directly attached hard
disk for storage.
Quotas
Implementing storage restrictions as quotas is not always easy and
requires precise planning.
Legal Situation
Emails are subject to the laws on privacy of communications. Not
many people are aware that administrators can be prosecuted for
negligence. However, this is a general problem with all email management, and this book will not deal with the topic further.

1.2 Comparing Courier and Cyrus
The requirements and size of the installation play an important part in the
selection of an IMAP server. Both Courier and Cyrus make low demands on
the CPU; fast I/O is required in both cases if there are more than approximately 20,000 users.
Both Courier and Cyrus enable users to share IMAP folders and administrators to limit the number of simultaneous logins. The user data can be
transferred from Unix accounts, read out via PAM, and stored in a LDAP
directory or in a MySQL or a PostgreSQL database.
In terms of user management, the real difference between the two lies in
secure authentication via Simple Authentication and Security Layer (SASL),
which is specified as an Internet standard in RFC 2222 (and used by Postfix
and other MTAs). In Cyrus, it is simple to implement, but in Courier, it
requires numerous kludges and dodges that are too much even for good
administrators. Instead, Courier uses a specially developed authentication
ÑÓÒ daemon. Cyrus
library, Authlib, whose central program is the ÙØ
also supports the authentication library as an SASL module (see page 213).
Both servers allow POP3 and IMAP via SSL/TLS, either via the dedicated
ports 995 (POP3 via SSL) and 993 (IMAP via SSL) or via the commands
ËÌÄË (POP3, see page 319) and ËÌ ÊÌÌÄË (IMAP, see page 297), and both

support virtual domains.
The most noticeable difference between Courier and Cyrus is the way they
manage email accounts and metadata. Courier uses only the filesystem
and ASCII files. The benefit is that nothing can break down; as long as the
filesystem is okay, Courier works. Courier uses the maildir format, which is
suitable for use via NFS, as no file locking is required.
On a Courier IMAP server, accounts automatically exist as long as, and as
soon as, they are listed in the user database. On the other hand, once

20

www.it-ebooks.info


1.2 Comparing Courier and Cyrus

Cyrus knows the login data of a new user, it creates the account structure
automatically when the user first logs in. It is also possible to initialize new
mailboxes using the administration tool ÝÖ Ñ.
Courier administrators can use shell scripts to intervene in the system, but
Cyrus administrators always use ÝÖ Ñ to administer their servers.
Cyrus stores emails and administration information in small filesystembased databases. This is intended to speed up access when there are a
large number of emails, but the disadvantage is that accessing mail messages via an index is more prone to errors. Simple manipulations to the
email store, such as the deletion or addition of messages, are complex to
carry out. As NFS accesses internal databases, and index files are destroyed
if multiple Cyrus nodes access them in parallel, a functioning file locking
system is essential. The suitability of NFS therefore mainly depends on
the NFS version and the maturity of the locking mechanisms it provides;
however, most administrators prefer not to use this solution.
Cyrus has the advantage of being able to use the mail filter language Sieve.

It also provides a system of permissions for shared IMAP folders that is
easier to use to implement access control policies.

21

www.it-ebooks.info


www.it-ebooks.info


Chapter

2

POP3 and IMAP at the Protocol
Level
You should take the intended environment into account when choosing
whether to support POP3 or IMAP as the protocol for mail retrieval. One
is simple and robust, the other is powerful and flexible. Courier and Cyrus
speak both protocols, and by using them you can provide IMAP and POP3
to your users without any additional work.

2.1 POP3
Version 3 of the Post Office Protocol (POP3) is comparatively simple, and
only allows the user to download emails from the server to the client. The
user can log in to an account, view the contents of the mailbox, transfer
and delete emails, and log out, all via server port 110. This requires few
resources, and there is little to configure, which means few sources of error.

23

www.it-ebooks.info