John wiley sons concurrent and distributed computing in java 2004 (by laxxuss)

Concurrent and Distributed
Computing in Java

Vijay K. Garg
University of Texas at Austin

IEEE PRESS

A JOHN WILEY & SONS, INC., PUBLICATION


This Page Intentionally Left Blank


Concurrent and Distributed
Computing in Java


This Page Intentionally Left Blank


Concurrent and Distributed
Computing in Java

Vijay K. Garg
University of Texas at Austin

IEEE PRESS

A JOHN WILEY & SONS, INC., PUBLICATION

Copyright 02004 by John Wiley & Sons, lnc. All rights reserved.
Published by John Wiley & Sons, inc., Hoboken, New Jersey.
Published simultaneously in Canada.
No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or
by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except as
permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior
written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to
the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax
(978) 646-8600, or on the web at www.copyright.com. Requests to the Publisher for permission should
be addressed to the Permissions Department, John Wiley & Sons, Inc., 11 1 River Street, Hoboken, NJ
07030, (201) 748-601 I , fax (201) 748-6008.
Limit of LiabilityiDisclaimer of Warranty: While the publisher and author have used their best efforts in
preparing this book, they make no representation or warranties with respect to the accuracy or
completeness of the contents of this book and specifically disclaim any implied warranties of
merchantability or fitness for a particular purpose. No warranty may be created or extended by sales
representatives or written sales materials. The advice and strategies contained herein may not be
suitable for your situation. You should consult with a professional where appropriate. Neither the
publisher nor author shall be liable for any loss of profit or any other commercial damages, including
but not limited to special, incidental, consequential, or other damages.
For general information on our other products and services please contact our Customer Care
Department within the U S . at 877-762-2974, outside the U.S. at 317-572-3993 or fax 317-572-4002.
Wiley also publishes its books in a variety of electronic formats. Some content that appears in print,
however, may not be available in electronic format.
Library of Congress Cataloging-in-Publication Data:
Garg, Vijay Kumar, 1938Concurrent and distributed computing in Java / Vijay K. Garg.
p. cm.
Includes bibliographical references and index.
ISBN 0-471 -43230-X (cloth)

I . Parallel processing (Electronic computers). 2. Electronic data processing-Distributed
processing. 3. Java (Computer program language). 1. Title.
QA76.58G35 2004
005.2'756~22
Printed in the United States of America.
1 0 9 8 7 6 5 4 3 2 1

2003065883


To

my teachers and
my students


This Page Intentionally Left Blank


Contents
List of Figures

xiii

Preface

xix

1 Introduction
1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1.2 Distributed Systems versus Parallel Systems . . . . . . . . . . . . . .
1.3 Overview of the Book . . . . . . . . . . . . . . . . . . . . . . . . . .
1.4 Characteristics of Parallel and Distributed Systems . . . . . . . . . .
1.5 Design Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.6 Specification of Processes and Tasks . . . . . . . . . . . . . . . . . .
1.6.1 Runnable Interface . . . . . . . . . . . . . . . . . . . . . . . .
1.6.2 Join Construct in Java . . . . . . . . . . . . . . . . . . . . . .
1.6.3 Thread Scheduling . . . . . . . . . . . . . . . . . . . . . . . .
1.7 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.8 Bibliographic Remarks . . . . . . . . . . . . . . . . . . . . . . . . . .

11
11
13
13
15

2 Mutual Exclusion Problem
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2 Peterson’s Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.3 Lamport’s Bakery Algorithm . . . . . . . . . . . . . . . . . . . . . .
2.4 Hardware Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.4.1 Disabling Interrupts . . . . . . . . . . . . . . . . . . . . . . .
2.4.2 Instructions with Higher Atomicity . . . . . . . . . . . . . . .
2.5 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.6 Bibliographic Remarks . . . . . . . . . . . . . . . . . . . . . . . . . .

17
17
20

24
27
27
27
28
30

3 Synchronization Primitives
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2 Semaphores . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

31
31
31

vii

1
1
3
4
6
7

8


...

CONTENTS

Vlll

The Producer-Consumer Problem . . . . . . . . . . . . . . .
The Reader-Writer Problem . . . . . . . . . . . . . . . . . . .
The Dining Philosopher Problem . . . . . . . . . . . . . . . .
3.3 Monitors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.4 Other Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.5 Dangers of Dea.dlocks . . . . . . . . . . . . . . . . . . . . . . . . . .
3.6 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.7 Bibliographic Remarks . . . . . . . . . . . . . . . . . . . . . . . . . .

3.2.1
3.2.2
3.2.3

33
36
36
42
46
49
50
51

4 Consistency Conditions
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2 System Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.3 Sequential Consistency . . . . . . . . . . . . . . . . . . . . . . . . . .
4.4 Linearizabilky . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4.5 Other Consistency Conditions . . . . . . . . . . . . . . . . . . . . . .
4.6 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.7 Bibliographic Remarks . . . . . . . . . . . . . . . . . . . . . . . . . .

53
53
54
55
57
60
62
63

5 Wait-Free Synchronization
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.2 Safe, Regular, and Atomic Registers . . . . . . . . . . . . . . . . . .
5.3 Regular SRSW R.egist,er . . . . . . . . . . . . . . . . . . . . . . . . .
5.4 SRSW Multivalued R.egister . . . . . . . . . . . . . . . . . . . . . . .
5.5 MRSW Register . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.6 MRMW Register . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.7 Atomic Snapshots . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.8 Consensus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.9 Universal Constructions . . . . . . . . . . . . . . . . . . . . . . . . .
5.10 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.1 1 Bibliographic Remarks . . . . . . . . . . . . . . . . . . . . . . . . . .

65

6 Distributed Programming
6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6.2 InetAddress Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.3 Sockets based on UDP . . . . . . . . . . . . . . . . . . . . . . . . . .
6.3.1 Datagram Sockets . . . . . . . . . . . . . . . . . . . . . . . .
6.3.2 Datagrampacket Class . . . . . . . . . . . . . . . . . . . . . .
6.3.3 Example Using Dat#agraIns . . . . . . . . . . . . . . . . . . .
6.4 Sockets Rased on TCP . . . . . . . . . . . . . . . . . . . . . . . . . .
6.4.1 Server Sockets . . . . . . . . . . . . . . . . . . . . . . . . . .

89
89
89
90
90
91
92
94
96

65
66
70
71
73
74
76
78
84
87
87

CONTENTS

ix

6.4.2 Example 1: A Name Server . . . . . . . . . . . . . . . . . . . 96
6.4.3 Example 2: A Linker . . . . . . . . . . . . . . . . . . . . . . .
100
6.5 Remote Method Invocations . . . . . . . . . . . . . . . . . . . . . . .
101
6.5.1 Remote Objects . . . . . . . . . . . . . . . . . . . . . . . . .
105
107
6.5.2 Parameter Passing . . . . . . . . . . . . . . . . . . . . . . . .
108
6.5.3 Dealing with Failures . . . . . . . . . . . . . . . . . . . . . .
108
6.5.4 Client Program . . . . . . . . . . . . . . . . . . . . . . . . . .
6.6 Other Useful Classes . . . . . . . . . . . . . . . . . . . . . . . . . . .
109
109
6.7 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.8 Bibliographic Remarks . . . . . . . . . . . . . . . . . . . . . . . . . .
110
7 Models and Clocks
111
7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
111
7.2 Model of a Distributed System . . . . . . . . . . . . . . . . . . . . .
112

7.3 Model of a Distributed Computation . . . . . . . . . . . . . . . . . . 114
7.3.1 Interleaving Model . . . . . . . . . . . . . . . . . . . . . . . .
114
114
7.3.2 Happened-Before Model . . . . . . . . . . . . . . . . . . . . .
115
7.4 Logical Clocks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
117
7.5 Vector Clocks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
122
7.6 Direct-Dependency Clocks . . . . . . . . . . . . . . . . . . . . . . . .
125
7.7 Matrix Clocks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
126
7.8 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7.9 Bibliographic Remarks . . . . . . . . . . . . . . . . . . . . . . . . . .
127
8 Resource Allocation

8.1
8.2
8.3
8.4
8.5
8.6
8.7
8.8
8.9
8.10

129
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
129
Specification of the Mutual Exclusion Problem . . . . . . . . . . . . 130
132
Centralized Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . .
135
Lamport’s Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . .
136
Ricart and Agrawala’s Algorithm . . . . . . . . . . . . . . . . . . . .
138
Dining Philosopher Algorithm . . . . . . . . . . . . . . . . . . . . . .
Token-Based Algorithms . . . . . . . . . . . . . . . . . . . . . . . . .
142
144
Quorum-Based Algorithms . . . . . . . . . . . . . . . . . . . . . . . .
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
146
Bibliographic Remarks . . . . . . . . . . . . . . . . . . . . . . . . . .
147

9 Global Snapshot
149
9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
149
9.2 Chandy and Lamport’s Global Snapshot Algorithm . . . . . . . . . . 151
9.3 Global Snapshots for non-FIFO Channels . . . . . . . . . . . . . . . 154


X

CONTENTS

9.4
9.5
9.6
9.7

Channel Recording by the Sender . . . . . . . . . . . . . . . . . . . .
154
Application: Checkpointing a Distributed Application . . . . . . . . 157
Problenis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
161
Bibliographic Remarks . . . . . . . . . . . . . . . . . . . . . . . . . .
162

10 Global Properties
10.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
10.2 Unstable Predicate Detection . . . . . . . . . . . . . . . . . . . . . .
10.3 Application: Distributed Debugging . . . . . . . . . . . . . . . . . .
10.4 A Token-Based Algorithm for Detecting Predicates . . . . . . . . . .
10.5 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
10.6 Bibliographic Remarks . . . . . . . . . . . . . . . . . . . . . . . . . .

163
163
164
169
169
173

176

11 Detecting Termination and Deadlocks
11.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
11.2 Diffusing Computation . . . . . . . . . . . . . . . . . . . . . . . . . .
11.3 Dijkstra and Scholten’s Algorithm . . . . . . . . . . . . . . . . . . .
11.3.1 An Optimization . . . . . . . . . . . . . . . . . . . . . . . . .
11.4 Termination Detection without Acknowledgment Messages . . . . . .
11.5 Locally Stable Predicates . . . . . . . . . . . . . . . . . . . . . . . .
11.6 Application: Deadlock Detection . . . . . . . . . . . . . . . . . . . .
11.7 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
11.8 Bibliographic Remarks . . . . . . . . . . . . . . . . . . . . . . . . . .

177

12 Message Ordering
12.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
12.2 Causal Ordering . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
12.2.1 Application: Causal Chat . . . . . . . . . . . . . . . . . . . .
12.3 Synchronous Ordering . . . . . . . . . . . . . . . . . . . . . . . . . .
12.4 Total Order for Multicast Messages . . . . . . . . . . . . . . . . . . .
12.4.1 Centralized Algorithm . . . . . . . . . . . . . . . . . . . . . .
12.4.2 Lamport’s Algorithm for Total Order . . . . . . . . . . . . .
12.4.3 Skeen’s Algorithm . . . . . . . . . . . . . . . . . . . . . . . .
12.4.4 Application: Replicated State Machines . . . . . . . . . . . .
12.5 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
12.6 Bibliographic Remarks . . . . . . . . . . . . . . . . . . . . . . . . . .

191
191

193
196
196
203
203
204
204
205
205
207

13 Leader Election
13.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
13.2 Ring-Based Algorithms . . . . . . . . . . . . . . . . . . . . . . . . .

209
209
210

177
177
180
181
182
185
188
189
189

xi

CONTENTS

13.2.1 Chang-Roberts Algorithm . . . . . . . . . . . . . . . . . . . .
13.2.2 Hirschberg-Sinclair Algorithm . . . . . . . . . . . . . . . . .
13.3 Election on General Graphs . . . . . . . . . . . . . . . . . . . . . . .
13.3.1 Spanning Tree Construction . . . . . . . . . . . . . . . . . . .
13.4 Application: Computing Global Functions . . . . . . . . . . . . . . .
13.5 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
13.6 Bibliographic Remarks . . . . . . . . . . . . . . . . . . . . . . . . . .
14 Synchronizers
14.1 Introduction . . . . . . . . . . . . . . . . . . .
14.2 A Simple synchronizer . . . . . . . . . . . . .
14.2.1 Application: BFS Tree Construction
14.3 Synchronizer CY . . . . . . . . . . . . . . . . .
14.4 Synchronizerp . . . . . . . . . . . . . . . . .
14.5 Synchronizer y . . . . . . . . . . . . . . . . .
14.6 Problems . . . . . . . . . . . . . . . . . . . .
14.7 Bibliographic Remarks . . . . . . . . . . . . .

210
212
213
213
215
217
219

221

221
..........
223
. . . . . . . . . . . 225
..........
226
..........
228
..........
230
..........
232
..........
232

.............

...

. . .

...

...

...

...
...

15 Agreement
233
15.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
233
15.2 Consensus in Asynchronous Systems (Impossibility) . . . . . . . . . 234
15.3 Application: Terminating Reliable Broadcast . . . . . . . . . . . . . 238
15.4 Consensus in Synchronous Systems . . . . . . . . . . . . . . . . . . . 239
15.4.1 Consensus under Crash Failures . . . . . . . . . . . . . . . . . 240
15.4.2 Consensus under Byzantine Faults . . . . . . . . . . . . . . . 243
15.5 Knowledge and Common Knowledge . . . . . . . . . . . . . . . . . . 244
15.6 Application: Two-General Problem . . . . . . . . . . . . . . . . . . . 248
249
15.7 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
250
15.8 Bibliographic Remarks . . . . . . . . . . . . . . . . . . . . . . . . . .

16 Transactions
253
16.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
253
16.2 ACID Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254
16.3 Concurrency Control . . . . . . . . . . . . . . . . . . . . . . . . . . .
255
16.4 Dealing with Failures . . . . . . . . . . . . . . . . . . . . . . . . . . .
256
257
16.5 Distributed Commit . . . . . . . . . . . . . . . . . . . . . . . . . . .
16.6 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
261
262

16.7 Bibliographic Remarks . . . . . . . . . . . . . . . . . . . . . . . . . .


xii

CONTENTS

17 Recovery
17.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
17.2 Zigzag Relation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
17.3 Communication-Induced Checkpointing . . . . . . . . . . . . . . .
17.4 Optimistic Message Logging: Main Ideas . . . . . . . . . . . . . . .
17.4.1 Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
17.4.2 Fault-Tolerant Vector Clock . . . . . . . . . . . . . . . . . .
17.4.3 Version End Table . . . . . . . . . . . . . . . . . . . . . . . .
17.5 An Asynchronous Recovery Protocol . . . . . . . . . . . . . . . . .
17.5.1 Message Receive . . . . . . . . . . . . . . . . . . . . . . . . .
17.5.2 On R.estart after a Failixe . . . . . . . . . . . . . . . . . . . .
17.5.3 On Receiving a Token . . . . . . . . . . . . . . . . . . . . . .
17.5.4 On Rollback . . . . . . . . . . . . . . . . . . . . . . . . . . .
17.6 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
17.7 Bibliographic Remarks . . . . . . . . . . . . . . . . . . . . . . . . . .

.
.
.

.

263

263
265
267
268
269
270
272
272
274
274
274
276
277
278

18 Self-stabilization
18.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
18.2 Mutual Exclusion with K-State Machines . . . . . . . . . . . . . . .
18.3 Self-St abilizing Spanning Trce Construction . . . . . . . . . . . . . .
18.4 Problenw . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
18.5 Bibliographic Remarks . . . . . . . . . . . . . . . . . . . . . . . . . .

279
279
280
285
286
289

A . Various Utility Classes

291

Bibliography

297

Index

305


List of Figures
1.1
1.2
1.3
1.4
1.5
1.6

A parallel system . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A distributed system . . . . . . . . . . . . . . . . . . . . . . . . . .
A process with four threads . . . . . . . . . . . . . . . . . . . . . . .
HelloWorldThread.java . . . . . . . . . . . . . . . . . . . . . . . . .
FooBar.java
...............................
Fibonacci.java . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2
2

9
11
12
14

2.1
2.2
2.3
2.4
2.5
2.6
2.7
2.8
2.9
2.10
2.11

Interface for acccssing the critical section . . . . . . .
A program t o test mutual exclusion . . . . . . . . . . . . . . . . . .
An attempt that violates mutual exclusion . . . . . . . . . . . . . .
An attempt that can deadlock . . . . . . . . . . . . . . . . . . . . . .
An attempt with strict alternation . . . . . . . . . . . . . . . . . . .
Peterson’s algorithm for mutual exclusion . . . . . . . . . . . . . . .
Lamport’s bakery algorithm . . . . . . . . . . . . . . . . . . . . . . .
TestAndSet hardware instruction . . . . . . . . . . . . . . . . . . . .
Mutual exclusion using TestAndSet . . . . . . . . . . . . . . . . . .
Semantics of swap operation . . . . . . . . . . . . . . . . . . . . . .
Dekker.java . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

18

19
20
21
21
22
25
27
28
28
29

3.1
3.2
3.3
3.4
3.5
3.6
3.7
3.8
3.9
3.10
3.11

Binary semaphore . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Counting semaphore . . . . . . . . . . . . . . . . . . . . . . . . . . .
A shared buffer implemented with a circular array . . . . . . . . . .
Bounded buffer using semaphores . . . . . . . . . . . . . . . . . . .
Producer-consumer algorithm using semaphores . . . . . . . . . . .
Reader-writer algorithm using semaphores . . . . . . . . . . . . . . .
The dining philosopher problem . . . . . . . . . . . . . . . . . . . .

Dining Philosopher . . . . . . . . . . . . . . . . . . . . . . . . . . .
Resource Inkrface . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Dining philosopher using semaphores . . . . . . . . . . . . . . . . . .
A pictorial view of a Java monitor . . . . . . . . . . . . . . . . . . .

32
33
34
35
37
38
39
40
41
41
44

xiii


xiv

LIST OF FIGURES

3.12 Bounded buffer monitor . . . . . . . . . . . . . . . . . . . . . . . . .
3.13 Dining philosopher using monitors . . . . . . . . . . . . . . . . . . .
3.14 Linked list . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

45
47

48

4.1 Concurrent. histories illustrating sequential consistency . . . . . . . .
4.2 Sequential consistency does not satisfy locality . . . . . . . . . . . .
4.3 Summary of consistency conditions . . . . . . . . . . . . . . . . . . .

56
58
62

5.1
5.2
5.3
5.4
5.5
5.6
5.7
5.8
5.9
5.10
5.11
5.12
5.13
5.14
5.15
5.16
5.17

Safe and unsafe read-write registers . . . . . . . . . . . . . . . . . .
Concurrent histories illustrating regularity . . . . . . . . . . . . . .

Atomic and nonatomic registers . . . . . . . . . . . . . . . . . . . .
Construction of a regular boolean regist.er . . . . . . . . . . . . . . .
Const.ruction of a multivalued register . . . . . . . . . . . . . . . . .
Construction of a niultireader register . . . . . . . . . . . . . . . . .
Construction of a mukiwriter register . . . . . . . . . . . . . . . . .
Lock-free atomic snapshot algorithm . . . . . . . . . . . . . . . . . .
Consensus Interface . . . . . . . . . . . . . . . . . . . . . . . . . . .
Impossibility of wait-free consensus with atomic read-write registers
TestAndSet class . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Consensus using TestAndSet object . . . . . . . . . . . . . . . . . .
CompSwap object . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Consensus using CompSwap object . . . . . . . . . . . . . . . . . .
Load-Linked and Store-Conditional object . . . . . . . . . . . . . . .
Sequential queue . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Concurrent queue . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

67
68
69
71
72
75
76
77
78
80

6.1
6.2
6.3

6.4
6.5
6.6
6.7
6.8
6.9
6.10
6.11
6.12

A datagram server . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A datagram client . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Simple name table . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Name server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A client for name server . . . . . . . . . . . . . . . . . . . . . . . . .
Topology class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Connector class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Message class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Linker class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
R.emote interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A name service implementation . . . . . . . . . . . . . . . . . . . .
A RMI client program . . . . . . . . . . . . . . . . . . . . . . . . . .

7.1
7.2

An example of topology of a distributed system . . . . . . . . . . . . 113
A simple distributed program with two processes . . . . . . . . . . . 113

81

82
82
83
84
85
86
93
95
97
98
99
100
102
103
104
105
106
109


xv

LIST OF FIGURES

A run in the happened-before model . . . . . . . . . . . . . . . . . .
A logical clock algorithm . . . . . . . . . . . . . . . . . . . . . . . . .
A vector clock algorithm . . . . . . . . . . . . . . . . . . . . . . . .
The VCLinker class that extends the Linker class . . . . . . . . . .
A sample execution of the vector clock algorithm . . . . . . . . . . .
A direct-dependency clock algorithm . . . . . . . . . . . . . . . . . .

A sample execution of the direct-dependency clock algorithm . . . . .
The matrix clock algorithm . . . . . . . . . . . . . . . . . . . . . . .

115
117
119
120
121
122
123
124

Testing a lock implementation . . . . . . . . . . . . . . . . . . . . .
ListenerThread . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Process.java . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A centralized mutual exclusion algorithm . . . . . . . . . . . . . . .
Lamport’s mutual exclusion algorithm . . . . . . . . . . . . . . . . .
Ricart and Agrawala’s algorithm . . . . . . . . . . . . . . . . . . . .
(a) Conflict graph; (b) an acyclic orientation with P2 and P4 as
sources; (c) orientation after I3 and P4 finish eating . . . . . . . . .
8.8 An algorithm for dining philosopher problem . . . . . . . . . . . . .
8.9 A token ring algorithm for the mutual exclusion problem . . . . . .

131
132
133
134
137
139

7.3
7.4
7.5
7.6
7.7
7.8
7.9
7.10

8.1
8.2
8.3
8.4
8.5
8.6
8.7

9.1
9.2
9.3
9.4
9.5
9.6

Consistent and inconsistent cuts . . . . . . . . . . . . . . . . . . .
Classification of messages . . . . . . . . . . . . . . . . . . . . . . .
Chandy and Lamport’s snapshot algorithm . . . . . . . . . . .
Linker extended for use with Sendercamera . . . . . . . . . . .
A global snapshot algorithm based on sender recording . . . . .
Invocation of the global snapshot algorithm . . . . . . . . . . .

.
.
...
. . .

151
153
155
158
. . . 159
. . . 160

10.1 WCP (weak conjunctive predicate) detection algorithm-checker process. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
10.2 Circulating token with vector clock . . . . . . . . . . . . . . . . . .
10.3 An application that runs circulating token with a sensor . . . . . .
10.4 Monitor process algorithm at Pi . . . . . . . . . . . . . . . . . . . . .
10.5 Token-based WCP detection algorithm . . . . . . . . . . . . . . . . .
11.1
11.2
11.3
11.4
11.5

141
143
145

167
170

171
172
174

A diffusing computation for the shortest path . . . . . . . . . . . . 179
Interface for a termination detection algorithm . . . . . . . . . . . . 179
183
Termination detection algorithm . . . . . . . . . . . . . . . . . . . .
A diffusing computation for the shortest path with termination . . . 184
Termination detection by token traversal . . . . . . . . . . . . . . . . 186

12.1 A FIFO computation that is not causally ordered . . . . . . . . . . . 191


xvi

LIST OF FIGURES

12.2
12.3
12.4
12.5
12.6
12.7
12.8
12.9

An algorithm for causal ordering of messages at Pi . . . . . . . . .
Structure of a causal message . . . . . . . . . . . . . . . . . . . . . .
CausalLinker for causal ordering of messages . . . . . . . . . . . .

A chat program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A computation t.hat is synchronously ordered . . . . . . . . . . . .
A computation that is not synchronously ordered . . . . . . . . . .
The algorithm at. Pi for synchronous ordering of messages . . . . .
The algorithm for synchronous ordering of messages . . . . . . . .

13.1
13.2
13.3
13.4
13.5
13.6
13.7

The leader election algorithm . . . . . . . . . . . .
Configurat.ions for the worst case (a) and the best
A spanning tree construction algorithm . . . . .
A convergecast algorithm . . . . . . . . . . . . . .
A broadcast algorithm . . . . . . . . . . . . . . .
Algorithm for computing a global function . . . .
Compiit.ing t.he global sum . . . . . . . . . . . . . .

14.1
14.2
14.3
14.4
14.5

Algorithm for the simple synchronizer at Pj . . . . . . . . . . . . .
Irnplementatiori of the simple synchronizer . . . . . . . . . . . . . .

An algorithm that generates a tree on an asynchronous network .
BFS tree algorithm using a synchronizer . . . . . . . . . . . . . . .
Alpha synchronizer . . . . . . . . . . . . . . . . . . . . . . . . . . . .

15.1
15.2
15.3
15.4
15.5
15.6

(a) Commutativity of disjoint events; (b) asynchrony of messages
(a) Case 1: proc(e) # p r o c ( f ) ; (b) case 2: proc(e) = p r o c ( f ) . .

194

. 195

197

. 198
. 198
. 201
. 202

..........

211
case (b) . . . . . . 212
. . . . . . . . . . . 214

216
..........
216
. . . . . . . . . .
. . . . . . . . . . . 218
219
..........

.

. 223

. 224

. 226
. 227
229

. 234

. . 237
Algorithm at Pi for consensus under crash failures . . . . . . . . . . 241
Consensus in a synchronous environment . . . . . . . . . . . . . . . . 242
Consensus tester . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
243
An algorithm for Byzantine General Agreement . . . . . . . . . . . 245

16.1 Algorithm for the coordinator of the two-phase commit protocol
16.2 Algorithm for the participants in the two-phase commit protocol
17.1

17.2
17.3
17.4
17.5
17.6

. 193

.
.

. 259
. 260

An example of the domino effect . . . . . . . . . . . . . . . . . . . .
Examples of zigzag paths . . . . . . . . . . . . . . . . . . . . . . . .
A distributed computation . . . . . . . . . . . . . . . . . . . . . . . .
Formal description of the fault-tolerant vector clock . . . . . . . . .
Formal description of the version end-table mechanism . . . . . . . .
An optimistic protocol for asynchronous recovery . . . . . . . . . . .

18.1 K-state self-stabilizing algorithm . . . . . . . . . . . . . . . . . . . .

264
266
271
273
273
275
280

xvii

LIST OF FIGURES

18.2 A move by the bottom machine in the K-state algorithm . . . . . .
18.3 A move by a normal machine in the K-state algorithm . . . . . . .
18.4 Self-stabilizing algorithm for mutual exclusion in a ring for the bottom
machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
18.5 Self-stabilizing algorithm for mutual exclusion in a ring for a normal
machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
18.6 Self-stabilizing algorithm for (BFS) spanning tree . . . . . . . . . . .
18.7 Self-stabilizing spanning tree algorithm for the root . . . . . . . . . .
18.8 Self-stabilizing spanning tree algorithm for nonroot nodes . . . . . .
18.9 A Java program for spanning tree . . . . . . . . . . . . . . . . . . . .
A.l
A.2
A.3
A.4
A.5
A.6

Util.java . . . . . .
Symbols.java . . .
Matrix.java . . . .
MsgList.java. . . .
1ntLinkedList.java
PortAddr.java . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . .
............................
............................
............................
............................
............................

280
281
283
284
285
286
287
288
292
293
293
294
294
295


This Page Intentionally Left Blank


Preface
This book is designed for a senior undergraduate-level course or an introductory
graduate-level course on concurrent and distributed computing. This book grew
out of my dissatisfaction with books on distributed systems (including books authored by me) that included pseudocode for distributed algorithms. There were two

problems with pseudocode. First, pseudocode had many assumptions hidden in it
making it more succinct but only at the expense of precision. Second, translating
pseudocode into actual code requires effort and time, resulting in students never actually running the algorithm. Seeing the code run lends a n extra level of confidence
in one’s understanding of the algorithms.
It must be emphasized that all of the Java code provided in this book is for
educational purposes only. I have deliberately avoided error checking and other
software engineering principles to keep the size of the code small. In the majority
of cases, this led to Java code, that kept the concepts of the algorithm transparent.
Several examples and exercise problems are included in each chapter to facilitate
classroom teaching. I have made an effort t o include some programming exercises
with each chapter.
I would like to thank the following people for working with me on various projects
discussed in this book: Craig Chase (weak predicates), Om Damani (message logging), Eddy Fromentin (predicate detection), Joydeep Ghosh (global computation),
Richard Kilgore (channel predicates), Roger Mitchell (channel predicates), Neeraj
Mittal (predicate detection and control, slicing, self-stabilization, distributed shared
memory), Venkat Murty (synchronous ordering), Michel Raynal (control flow properties, distributed shared memory), Alper Sen (slicing), Chakarat Skawratonand
(vector clocks), Ashis Tarafdar (message logging, predicate control), Alexander Tomlinson (global time, mutual exclusion, relational predicates, control flow properties)
and Brian Waldecker (weak and strong predicates). Anurag Agarwal, Arindam
Chakraborty, Selma Ikiz, Neeraj Mittal, Sujatha Kashyap, Vinit Ogale, and Alper
Sen reviewed parts of the book. I owe special thanks t o Vinit Ogale for also helping
me with figures.
I thank the Department of Electrical and Computer Engineering at The Unixix


xx
v<:rsit,yof Texas at Austin, where I was given the opportunity to develop and teach
courses on concurrent and distributed systems. Students in these courses gave me
very useful fedback.
I was support,ed in part by many grants from the National Science Foundat.ion
over the last, 14 years. Many of t,he results reported in this book would not have

been discovered by me and my research group without that support. I also thank
John Wiley & Sons, Inc. for supporting the project.
Finally, I thank my parents, wife and children. Without their love and support,
this book would not have been even conceived.
There are many concurrent and distributed programs in this book. Although I
have tried t,o ensure t,hat t’liere are no “bugs” in these programs, some are, no doubt,,
st,ill lurking in the code. I would be grat,eful if any bug that is discovered is reported
t,o me. The list of known errors and the supplerrientary material for the book will
be rriaint,ained on my homepage:
/>
Included in tjheWebsite is a program that allows animation of most of the algorithms
in the book. It also includes all the source code given in the book. The reader can
access t,he source code with the user name as guest and the password as utexas.
Vijay K. Garg
Austin. Texas


Chapter 1

Introduction

1.1

Introduction

Parallel and distributed computing systems are now widely availabli . A parullel system consists of multiple processors that communicate with each otl er using shared
memory. As the number of transistors on a chip increases, miiltipro essor chips will
become fairly common. With enough parallelism available in applicE :ions, such systerns will easily beat sequential systems in performance. Figure 1.1 5 lows a parallel
system with multiple processors. These processors communicate P ith each other
using the shared memory. Each processor may also have local mem Iry that is not

shared with other processors.
We define distributed systems as those computer syst.ems that co kain mult.iple
processors connected by a communication network. In these syste 11s processors
communicate with each other using messages that are sent over the I 2twork. Such
systems are increasingly available because of decrease in prices of c o a iuter processors and the high-bandwidth links t o connect them. Figure 1.2 shows t distributed
system. The communication network in the figure could be a local ,rea network
such as an Ethernet, or a wide area network such as the Internet.
Programming parallel and distributed systems requires a different set of tools
and techniques than that required by the traditional sequential software. The focus
of this book is on these techniques,.
1


2

CHAPTER 1. INTRODUCTION

I

Shared memory

1

CPU

CPU

CPU

CPU

CPU

Local
memory

Local
memory

Local
memory

Local
memory

Local
memory

Figure 1.1: A parallel system

Figure 1.2: A distributed system