applied cryptanalysis - breaking ciphers in the real world

APPLl ED CRYPTANALYSIS
Breaking Ciphers in the Real World
Mark
Stamp
Richard
M.
Low
San
Jose State University
San Jose,
CA
BICENTENNIAL
BICENTENNIAL
WILEY-INTERSCIENCE
A
JOHN
WILEY
&
SONS,
INC., PUBLICATION
This Page Intentionally Left Blank
APPLIED CRYPTANALYSIS
THE
WILEY
BICENTENNIAL-KNOWLEDGE
FOR GENERATIONS
ach generation has its unique needs and aspirations. When Charles Wiley first
opened his small printing shop in lower Manhattan in
1807,
it was a generation

of boundless potential searching for an identity. And we were there, helping to
define a new American literary tradition. Over half a century later, in the midst
of the Second Industrial Revolution, it was a generation focused on building the
future. Once again, we were there, supplying the critical scientific, technical, and
engineering knowledge that helped frame the world. Throughout the 20th
Century, and into the new millennium, nations began to reach out beyond their
own borders and a new international community was born. Wiley was there,
expanding its operations around the world to enable
a
global exchange of ideas,
opinions, and know-how.
For 200 years, Wiley has been an integral
part of each generation’s journey,
enabling the flow
of
information and understanding necessary to meet their needs
and fulfill their aspirations. Today, bold new technologies are changing the way
we live and learn. Wiley will be there, providing you the must-have knowledge
you need to imagine new worlds, new possibilities, and new opportunities.
Generations come and go, but you can always count on Wiley to provide you the
knowledge you need, when and where you need it!
6
WILLIAM
J.
PESCE
PETER
BOOTH
WILEY
PRESIDENT AND
CHIEF

EXECUTIVE OFFICER
CHAIRMAN
OF
THE
BOARD
APPLl ED CRYPTANALYSIS
Breaking Ciphers in the Real World
Mark
Stamp
Richard
M.
Low
San
Jose State University
San Jose,
CA
BICENTENNIAL
BICENTENNIAL
WILEY-INTERSCIENCE
A
JOHN
WILEY
&
SONS,
INC., PUBLICATION
Copyright
0
2007 by John Wiley
&
Sons, Inc.

All
rights reserved.
Published by John Wiley
&
Sons, Inc., Hoboken, New Jersey
Published simultaneously in Canada.
No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form
or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as
permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior
written permission of the Publisher, or authorization through payment
of
the appropriate per-copy fee to
the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax
(978) 750-4470, or on the web at www.copyright.com. Requests
to
the Publisher for permission should
be addressed to the Permissions Department, John Wiley
&
Sons, Inc.,
11
1
River Street, Hoboken, NJ
07030, (201) 748-601
1.
fax (201) 748-6008, or online at
go/permission.
Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in
preparing this book, they make no representations
or
warranties with respect to the accuracy or

completeness of the contents
of
this book and specifically disclaim any implied warranties of
merchantability
or
fitness for a particular purpose. No warranty may be created
or
extended by sales
representatives
or written sales materials. The advice and strategies contained herein may not be
suitable for your situation.
You
should consult with a professional where appropriate. Neither the
publisher nor author shall be liable for any
loss
of profit or any other commercial damages, including
but not limited to special, incidental, consequential, or other damages.
For general information on our other products and services
or
for technical support, please contact our
Customer Care Department within the United States at (800) 762-2974, outside the United States at
(317) 572-3993
or
fax (317) 572-4002.
Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may
not be available in electronic format.
For
information about Wiley products, visit our web site at
www.wiley.com.
Wiley Hicentennial

Logo:
Richard
J.
Pacific0
Library
of
Congress Cataloging-in-Publication Data:
Stamp, Mark.
Richard
M.
Low.
Applicd cryptanalysis
:
breaking ciphers in the real world
/
Mark Stamp,
p. cm.
lncludes bibliographical references and index.
1,
Computer security. 2. Data encryption (Computer science) 3.
QA76.9.A25S687 2007
005.8'24~22 2007001277
ISBN 978-0-470-1 1486-5 (pbk.)
Cryptography.
I.
Low, Richard
M.,
1967-
11.
Title.

Printed in the United States
of
America
10987654321
To
Melody, Austin, and Males
~
MSS
TO
Amy
-
RML
This Page Intentionally Left Blank
Contents
Preface xiii
About the Authors xvii
Acknowledgments xix
1
Classic Ciphers
1
1.1 Introduction

1
1.2
Good Guys and Bad Guys

1
1.3
Terminology

2
1.4 Selected Classic Crypto Topics

4
1.4.1 Transposition Ciphers

5
1.4.2 Subst. itution Ciphers

8
1.4.3 One-Time Pad

18
1.4.4 Codebook Ciphers

20
1.5 Summary

21
1.6 Problems

22
2
World War
I1
Ciphers
2.1 Introduction

2.2
Enigma

2.2.2
Enigma Keyspace

2.2.3 Rotors

2.2.4 Enigma Attack

2.2.5 More Secure Enigma?

2.3 Purple

2.3.1 Purple Cipher Machine

2.3.2 Purple Keyspace

2.3.3 Purple Diagnosis

2.3.4 Decrypting Purple

2.3.5 Purple versus Enigma

2.4 Sigaba

2.2.1 Enigma Cipher Machine

25
25
26
26

29
31
34
37
38
38
44
45
49
50
52
vii

Vlll
CONTENTS
2.4.1 Sigaba Cipher Machine

52
2.4.2 Sigaba Keyspace

57
2.4.3 Sigaba Attack

59
2.4.4 Sigaba Conclusion

67
2.5 Summary

68

2.6 Problerns

69
3
Stream
Ciphers
79
3.1
Introduction

79
3.2 Shift Registers

81
3.2.1 Berlekamp-Massey Algorithm

83
3.2.2 Cryptographically Strong Sequences

85
3.2.3 Shift Register-Based Stream Ciphers

89
3.2.4 Correlation Attack

90
3.3
ORYX

93

3.3.1
ORYX Cipher

94
3.3.2 ORYX Attack

97
3.3.3
Secure ORYX?

102
3.4
RC4

103
3.4.1
RC4
Algorithm

105
3.4.2 RC4 Attack 105
3.4.3 Preventing the
RC4
Attack

110
3.5 I’KZIP

110
3.5.1 PKZIP Cipher

111
3.5.2
PKZIP
Attack

113
3.5.3 Improved PKZIP?

120
3.6 Summary

120
3.7 Problems

121
4
Block
Ciphers
127
4.1 Introduction

127
4.2 Block Cipher Modes

128
4.3 Feistel Cipher

131
4.4 Hellman’s Time-Memory Trade-off

133
4.4.1 Cryptanalytic TMTO

133
4.4.2 Bad Chains

137
4.4.3 Succcss Probability

141
4.4.4 Distributed TMTO

142
4.4.5 TMTO Conclusioris

143
4.5 CMEA

144
4.5.1 CMEA Cipher

144
4.5.2 SCMEA Cipher

146
4.5.3 SCMEA Chosen Plaintext Attack

147
CONTENTS

ix
4.5.4 CMEA Chosen Plaintext Attack

148
4.5.5 SCMEA Known Plaintext Attack

151
4.5.6 CMEA Known Plaintext Attack

158
4.5.7 More Secure CMEA?

159
4.6 Akelarre

160
4.6.1 Akelarre Cipher

160
4.6.2 Akelarre Attack

166
4.6.3 Improved Akelarre?

169
4.7 FEAL

170
4.7.1
FEAL-4

Cipher

171
4.7.2 FEAL-4 Differential Attack

172
4.7.3 FEAL-4 Linear Attack

177
4.7.4 Confusion and Diffusion

182
4.8 Summary

183
4.9 Problems

183
5
Hash
Functions
193
5.1 Introduction

193
5.2 Birthdays and Hashing

200
5.2.1 The Birthday Problem

200
5.2.2 Birthday Attacks on Hash Functions

201
5.2.3 Digital Signature Birthday Attack

202
5.2.4 Nostradamus Attack

203
5.3 MD4

208
5.3.1
MD4
Algorithm

208
5.3.2 MD4 Attack

210
5.3.3 A Meaningful Collision

224
5.4 MD5

225
5.4.1 MD5 Algorithm

225

5.4.2 A Precise Differential

231
5.4.3 Outline
of
Wang’s Attack

233
5.4.4 Wang’s MD5 Differentials

235
5.4.5 Reverse Engineering Wang’s Attack

238
5.4.6 Stevens’ Implementation
of
Wang’s Attack

252
5.4.7
A Practical Attack

253
5.5 Summary

256
5.6 Problems

257
6

Public
Key
Systems
265
6.1 Introduction

265
6.2 MerkleeHellman Knapsack

267
6.2.1 Lattice-Reduction Attack

270
6.2.2 Knapsack Conclusion

275
X
CONTENTS
6.3
Difie-Hellman Key Exchange

6.3.1
Man-in-the-Middle Attack

6.3.2
Diffie-Hellman Conclusion

6.4
Arithmetica Key Exchange

6.4.1
Hughes-Tannenbaum Length Attack

6.4.2
Arithmetica Conclusion

6.5
RSA

6.5.1
Mathematical Issues

6.5.2
RSA Conclusion

6.6
Rabin Cipher

6.6.1
Chosen Ciphertext Attack

6.6.2
Rabin Cryptosystenl Conclusion

6.7
NTRU
Cipher

6.7.1
Meet-in-the-Middle Attack

6.7.2
Multiple Transmission Attack

6.7.3
Chosen Ciphertext Attack

6.7.4
NTRU
Conclusion

6.8
ElGarnal Signature Scheme

6.8.1
Mathematical Issues

6.8.2
ElGamal Signature Conclusioil

6.9
Summary

6.10
Problems

7
Public
Key
Attacks

7.1
Introduction

7.2
Factoring Algorithms

7.2.1
Trial Division

7.2.2
Dixon’s Algorithm

7.2.3
Quadratic Sieve

7.2.4
Factoring Conclusions

7.3
Discrete Log Algorithms

7.3.1
Trial Multiplication

7.3.2
Baby-Step Giant-Step

7.3.3
Index Calculus

7.3.4
Discrete Log Conchlsions

7.4
RSA Iniplenieritation Attacks

7.4.1
Tinling Attacks

7.4.2
Glitchirlg Attack

7.4.3
Implementatiorl Attacks Conclusiorls

7.5
Summary

7.6
Problems

275
277
278
279
283
284
284
285
288

289
291
292
293
299
301
302
304
305
308
308
309
309
315
315
316
316
317
323
327
330
330
331
332
333
334
334
353
354
355

355
CONTENTS
xi
Appendix
361
A-1
MD5Tables

361
A-2
Math

371
A-2.1
Number
Theory

371
A-2.2
Group Theory

372
A-2.3
Ring
Theory

372
A-2.4
Linea.
r

Algebra

373
Annotated Bibliography
375
Index
393
This Page Intentionally Left Blank
Preface
To
paraphrase Barbie, “cryptanalysis is hard”
[6].
Unfortunately, many
cryptanalysis papers seem to be written in their own impenetrable secret
code, making the subject appear to be even more difficult than it really is.
In this book, we strive to present applied cryptanalytic attacks in an acces-
sible form. Here, we are focused on practical attacks that actually break real-
world systems, not attacks that merely indicate some theoretical weakness
in
a
cipher. Consequently, we consider real ciphers and, primarily, modern
ciphers. Many attacks that satisfy our criteria are scattered throughout the
literature.’ With a few notable exceptions, these papers require
a
Herculean
effort to digest and understand. One of our goals is to lift this unintentional
veil on the exciting and fascinating field of cryptanalysis.
Most of the topics presented in this book require only
a
modest math-

ematical background. Some of the public key topics are inherently more
mathematical, but in every case we have strived to minimize the advanced
mathematics. We also believe that we have provided enough background in-
formation
so
that the book is essentially self-contained. Some of the more
advanced mathematical topics are treated briefly in the Appendix. Any moti-
vated upper-division undergraduate student-in any technical field of study-
should be able to tackle this book. Some of the material is not easy, but those
who persist
will
be rewarded with
a
solid understanding of cryptanalysis, as
well
as
the knowledge, tools, and experience to confidently explore cutting-
edge cryptanalytic topics.
We have provided an extensive set of problems for each chapter.
A
few
of
these problems are relatively easy, but most range from moderate to some-
what challenging. Generally, we have tried to avoid obvious problems
of
the
“implement such-and-such attack” variety.
Of
course, it is useful and instruc-
tive

to
implement an attack, but the problems are intended to reinforce
and
expand on material presented in the text, without placing an overwhelming
burden on the reader.
A
fairly complete solutions manual is available to
instructors directly froni your Wiley representative.
’A large percentage of the cryptanalysis literature
is
informal in the sense that many
papers never receive any formal peer review. Although the academic peer-review process
suffers from
a
multitude
of
sins, no peer review is no better.
xiii
xiv
PREFACE
To really understand the material in this book,
it
is necessary to work
a
significant number of the problems. Cryptarialysis is definitely not
a
spectator
sport. We believe that the computer is an essential cryptanalytic tool. It is
riot coincidental that many
of

the homework problems require some computer
programming.
For the terminally cryptanalytically insane, we have created
it
collection
of
challenge problems. These problems, which are posted on the textbook
website at

consist primarily
of
cryptanalytic challenges based on the ciphers and attacks
presented in the text.
A
few research-oriented problems are
also
included.
Each problem carries
a
difficulty rating
so
that you will have some idea
of
what
you
might be getting into. For each challenge problem,
a
small prize2 is
offered to the first solver. We promise to update the website as the challenge
problems are solved.

The website includes source code arid test vectors for
many of the ciphers discussed here. In addition, a complete set of quality
PowerPoint slides is available.
The text is organized around four major themes, namely, classic ciphers
(Chapters
1
and
a),
symmetric ciphers (Chapters
3
and
4),
hash functions
(Chapter
5),
and public key crypto (Chapters
6
and
7).
The specific topics
covered in each chapter are summarized below:
Chapter Topics
1.
Classic Ciphers Pen-and-paper systems
2.
World War I1 Ciphers
3.
Stream Ciphers
4.
Block Ciphers

5.
Hash Functions
6.
Public Key Systems
7.
Public Key Attacks
Enigma, Purple, Sigaba
Shift registers,
correlation at tacks,
ORYX.
RC4,
PKZIP
Block cipher modes,
MAC, Hellman's TMTO,
CMEA, Akelarre,
FEAL
HMAC, birthday attacks,
Nostrasamus at tack,
MD4, MD5
Knapsack, Diffie-Hellman,
Arithmetica, RSA
Rabin, NTRU, EIGamal
Factoring, discrete log,
RSA timing attacks,
RSA ditching attack
Y
-
'The
emphasis
here

is
on
'?,mall
''
PREFACE
xv
The first author wrote Chapters
2
through
5
and
7,
while the second
author wrote the majority of Chapters
1
and 6. The first author extensively
edited all chapters to give the book
a
more consistent “look and feel.” The
first author did his best to resist including too many bad jokes, but some
proved irresistible. Most of these have, mercifully, been relegated to footnotes.
The majority of the book consists of
a
series of cryptanalytic vignettes,
organized by topic. Chapters
3,
4,
and
5
each begin with

a
relatively generic
method of attack (correlation attacks, Hellman’s TMTO and birthday at-
tacks, respectively). These attacks are interesting in their own right, but
each also serves as an introduction to the type of cipher under consideration.
Each of these chapters then segues into the cryptanalysis of specific ciphers.
For public key crypto, the introductory material has been expanded to
an entire chapter. In Chapter
6,
several public key systems are introduced
and discussed from the perspective of relatively straightforward attacks or
implementation issues that can lead to weaknesses. Then selected public key
attacks are covered in depth in Chapter
7.
The chapters are highly independent of each other, as are many of the sec-
tions within chapters. The most dependent chapters are
6
and
7,
which cover
public key crypto. In addition, some familiarity with hashing (Chapter 5)
would be useful before diving into the public key material. The terminology
and background covered in Chapter
1
is used throughout the text. Regardless
of your background in cryptography, we recommend that you read Chapter
1
first, since terminology is not consistent throughout the crypto world. Not
only is crypto terminology inconsistent, but notation is even worse. Notation-
wise, we have tried to be as internally consistent as possible. Consequently,

our notation often differs from the original source.
The first author’s information security textbook [142] covers four ma-
jor topics, one of which is cryptography. The only significant overlap be-
tween [142] and this book is Hellman’s time-memory trade-off attack, dis-
cussed here in Section
4.4.
A
brief section on the knapsack attack is also
included in both books; here, in Section 6.2.
Finally, we apologize in advance for the inevitable “bugs” in this book.
Any computer program
of
sufficient size has bugs and it is more difficult to
debug a textbook than
a
program, since there is at least some hope of getting
a
program to misbehave during testing. There is no method to “exercise” a
textbook other than to proofread it and to teach from it,-the more times the
better. The first author has taught virtually all of the material in this text,
and several careful proofreadings have been done. Nevertheless, it is a sure
bet that errors remain. Please tell us of any bugs you find. We would also
appreciate any other comments you have regarding this book.
Mark Stamp
Richard
M,
Low
San Jose State University
This Page Intentionally Left Blank
About the

Authors
Mark Stamp has an extensive background in information security in general
and cryptography in particular, having spent more than seven years
as
a
Cryptologic Mathematician
at
the National Security Agency. His other rele-
vant experience includes two years as Chief Cryptologic Scientist at
a
small
Silicon Valley startup company. Since the demise of his startup company
in
2002,
he has been
a
faculty member in the department
of
computer science
at San Jose State University, where he primarily teaches courses in informa-
tion security. In
2005,
Dr. Stamp published his first textbook,
Information
Security: Principles
a.nd
Practice
(Wiley Interscience).
Richard
M.

Low has
a
PhD
in mathematics and is a faculty member in
the department of mathematics at San Jose State University. His research
interests include cryptography, combinatorics and group theory. In addition
to teaching mathematics, he has conducted
a
popular cryptography seminar
at SJSU.
xvii
Acknowledgments
I
want to thank the following San Jose State University students who con-
tributed significantly to the indicated sections: Heather Kwong (Enigma);
Thang Dao (Purple); Wing On Chan and Ethan Le (Sigaba); Thuy Nguyen-
phuc
(ORYX);
Bevan Jones and Daniel
Tu
(Akelarre); Tom Austin, Ying
Zhang, and Narayana Kashyap (MD5); Ramya Venkataramu (RSA timing
attack); Natalia Khuri (RSA); Edward Yin (Chapter
2
solutions).
As always, thanks to my PhD advisor, Clyde
F.
Martin. Clyde is the one
who introduced me to cryptography.

Richard Low deserves credit for enthusiastically signing on to this project
and for convincing
me
to persevere at
a
couple of points where
I
was ready
to throw in the towel. He also tolerated my occasional rants very well.
A very special thanks to Wan-Teh Chang for his careful reading of most
sections of this book. Wan-Teh has an excellent eye for detail and he provided
numerous corrections and useful suggestions.
Thanks are due to all of the people at Wiley who were involved with this
book. In particular,
I
want to thank Paul Petralia, Whitney A. Lesch, and
Kellsee Chu who were extremely helpful throughout.
Last but certainly not least, thanks to my lovely wife, Melody, and my
two boys, Austin and Miles, for their patience during the seemingly endless
hours
I
spent working on this project.
~-
MSS
My
love of mathematics was cultivated by many
of
my former math teach-
ers (from junior high school to graduate school). Those that come particularly
to mind include: Joseph Buckley, Gary Chartrand, Daniel Goldston, Doug

Harik, Philip Hsieh, Sin-Min Lee, John Martino, John Mitchem, Thomas
Richardson, Gerhard Ringel, Jerome Schroeder, Michael Slack, Arthur Stod-
dart, Sandra Swanson, Arthur White, Gregg Whitnah, and Kung-Wei Yang.
Thank you for showing me the way.
-
RML
xix
This Page Intentionally Left Blank
Chapter
1
Classic
Ciphers
You
are in a maze
of
twisty little passages, all alike.
-
Adventure
1.1
Introduction
Most of this chapter is devoted to introducing terminology and discussing
a
select few classic “pen and paper” ciphers. Our goal here is not to cover clas-
sical cryptography in detail, since there are already many excellent sources of
information on such ciphers. For example, Kahn’s history [74] has
a
general
discussion of virtually every cipher developed prior to its original publica-
tion date of
1967,

Barr
[7]
presents a readable introduction to cryptography,
Spillman [139] nicely covers the cryptanalysis of several classic cipher systems
and Bauer
[8]
provides rigorous coverage of a large number of classical crypto
topics. The ciphers we discuss in this chapter have been selected to illustrate
a
few important points that arise in upcoming chapters.
Even if you are familiar with classical cryptosystems, you should read
the next two sections where terminology is discussed, since the terminology
in cryptography is not always consistent. In addition, the material in Sec-
tions 1.4.3 and 1.4.4 is directly referenced in upcoming chapters.
1.2
Good
Guys
and Bad
Guys
In cryptography, it is traditional that Alice and Bob are the good guys who
are trying to communicate securely over an insecure channel. We employ
Trudy (the “intruder”)
as
our generic bad guy. Some books have a whole
cast of bad guys with the name indicating the particular evil activity (Eve,
the eavesdropper, for example), but we use Trudy
as
our all-purpose bad
“guy”
.

1
2
CLASSIC
CIPHERS
Since this is a cryptanalysis book, we often play the role
of
Trudy. Trudy
is an inherently more interesting character than boring old Alice and Bob, and
this is part of what makes cryptanalysis
so
much more fun than cryptography.
Trudy does not have to play by any preconceived set of rules. However, it
is important to remember that attacks
on
real systems are almost certainly
illegal,
so
do not attempt to play Trudy in the real world.
1.3
Terminology
Oryptology
is
the art and science of making and breaking “secret codes.”
Cryptology can be subdivided irito
cryptography
(the art and science
of
mak-
ing secret codes) and
cryptanalysis

(the breaking
of
secret codes). The secret
codes themselves are known
as
ciphers
or
cryptosystems.
In this book, we are
focused on cryptanalysis, but many topics in cryptography naturally arise.
It is common practice to use the term cryptography as a synonym for
cryptology, and we generally follow this practice. In fact, we often use
crypto
as shorthand for cryptology, cryptography, cryptanalysis, or any variety of
other crypto-related topics. The precise meaning should be clear from the
context.
The original readable message is the
plaintext,
while the
ciphertext
is the
unreadable text that results from
encrypting
the plaintext.
Decryption
is the
inverse process, where the ciphertext is converted into plaintext.
A
key
is used to configure

a
cryptosystem.
All
classic systems are
sym-
metric ciphers,
meaning that the same key is used to encrypt
as
to decrypt.
In
so-called
public key cryptography
the encryption and decryption keys are
different, which means that the encryption key can be be made public, but
the decryption key must remain private.
We
cover public key cryptosystems
in Chapters
6
and
7,
while
all
of
the remaining chapters including the re-
maining sections
of
this chapter~ deal with symmetric ciphers.
Note that decryption is distinct from cryptanalysis, since cryptanalysis
implies an attack of

some
sort has been used to read the messages, while
decryption implies that the plaintext has been retrieved using the key by the
expectcd process. Of course,
if
Trudy recovers the key via cryptanalysis, then
she can simply decrypt
a
particular ciphertext.
The typical encryption and decryption process is illustrated in Figure
1.1,
where
Pi
is
the ith unit
of
plaintext (which may be
a
bit, a letter,
a
word, or
a
la.rger block, depending
on
the particular cipher),
Ci
is the corresponding
unit
of
ciphertext, and the squiggly line represenh the transmission

of
the
ciphertext over an insecure channel.
In a
ciphertext
only
attack, the attacker attempts to recover the key or plaintext from the
ciphertext.
In
particular, in
a
ciphertext-only attack, the cryptanalyst does
There
are several generic types of attacks on ciphers.