Distibuted systems

Distributed Systems

www.it-ebooks.info


Distributed Systems
design and algorithms

Edited by
Serge Haddad
Fabrice Kordon
Laurent Pautet
Laure Petrucci

www.it-ebooks.info


First published 2011 in Great Britain and the United States by ISTE Ltd and John Wiley & Sons, Inc.
Apart from any fair dealing for the purposes of research or private study, or criticism or review, as
permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced,
stored or transmitted, in any form or by any means, with the prior permission in writing of the publishers,
or in the case of reprographic reproduction in accordance with the terms and licenses issued by the
CLA. Enquiries concerning reproduction outside these terms should be sent to the publishers at the
undermentioned address:
ISTE Ltd
27-37 St George’s Road
London SW19 4EU
UK

John Wiley & Sons, Inc.
111 River Street

Hoboken, NJ 07030
USA

www.iste.co.uk

www.wiley.com

© ISTE Ltd 2011
The rights of Serge Haddad, Fabrice Kordon, Laurent Pautet and Laure Petrucci to be identified as the
authors of this work have been asserted by them in accordance with the Copyright, Designs and Patents
Act 1988.
____________________________________________________________________________________
Library of Congress Cataloging-in-Publication Data
Distributed systems : design and algorithms / edited by Serge Haddad ... [et al.].
p. cm.
Includes bibliographical references and index.
ISBN 978-1-84821-250-3
1. Electronic data processing--Distributed processing. 2. Peer-to-peer architecture (Computer networks)
3. Computer algorithms. 4. Embedded computer systems. 5. Real-time data processing. I. Haddad,
Serge.
QA76.9.D5D6144 2011
004'.33--dc22
2011012243
British Library Cataloguing-in-Publication Data
A CIP record for this book is available from the British Library
ISBN 978-1-84821-250-3
Printed and bound in Great Britain by CPI Antony Rowe, Chippenham and Eastbourne.

www.it-ebooks.info

Contents

Foreword . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

9

Chapter 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Serge H ADDAD, Fabrice KORDON, Laurent PAUTET and Laure P ETRUCCI

13

F IRST PART. L ARGE S CALE P EER - TO -P EER D ISTRIBUTED S YSTEMS . .

19

Chapter 2. Introduction to Large-Scale Peer-to-Peer Distributed Systems
Fabrice KORDON

21

2.1. “Large-Scale” distributed systems? . . . . . . .
2.2. Consequences of “large-scale” . . . . . . . . .
2.3. Some large-scale distributed systems . . . . . .
2.4. Architectures of large scale distributed systems
2.5. Objective of Part 1 . . . . . . . . . . . . . . . .
2.6. Bibliography . . . . . . . . . . . . . . . . . . .

.
.

.
.
.
.

21
22
23
26
30
31

Chapter 3. Design Principles of Large-Scale Distributed System . . . . . .
Xavier B ONNAIRE and Pierre S ENS

33

3.1. Introduction to peer-to-peer systems
3.2. The peer-to-peer paradigms . . . . .
3.3. Services on structured overlays . . .
3.4. Building trust in P2P systems . . . .
3.5. Conclusion . . . . . . . . . . . . . .
3.6. Bibliography . . . . . . . . . . . . .

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.

.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.

.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.

.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.

.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.

.
.

33
34
41
43
52
53

Chapter 4. Peer-to-Peer Storage . . . . . . . . . . . . . . . . . . . . . . . . .
Olivier M ARIN, Sébastien M ONNET and Gaël T HOMAS

59

4.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

59

www.it-ebooks.info

.
.
.
.
.
.

.
.

.
.
.
.

.
.
.
.
.
.

v

.
.
.
.
.
.

.
.
.
.
.
.

.
.

.
.
.
.


6

Distributed Systems

4.2. BitTorrent . .
4.3. Gnutella . . .
4.4. Conclusion .
4.5. Bibliography

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

60
66
79
79

Chapter 5. Large-Scale Peer-to-Peer Game Applications . . . . . . . . . . .
Sébastien M ONNET and Gaël T HOMAS

81

5.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.2. Large-scale game applications: model and specific requirements . .
5.3. Overview of peer-to-peer overlays for large-scale game applications
5.4. Overlays for FPS games . . . . . . . . . . . . . . . . . . . . . . . . .
5.5. Overlays for online life-simulation games . . . . . . . . . . . . . . .
5.6. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.7. Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

.
.
.
.

.
.
.
.
.
.
.

. 81
. 83
. 90
. 93

. 95
. 100
. 101

S ECOND PART. D ISTRIBUTED , E MBEDDED AND R EAL -T IME S YSTEMS . 105
Chapter 6. Introduction to Distributed
Embedded and Real-time Systems . . . . . . . . . . . . . . . . . . . . . . . . 107
Laurent PAUTET
6.1. Distributed real-time embedded systems . . . . . . .
6.2. Safety critical systems as examples of DRE systems
6.3. Design process of DRE systems . . . . . . . . . . .
6.4. Objectives of Part 2 . . . . . . . . . . . . . . . . . . .
6.5. Bibliography . . . . . . . . . . . . . . . . . . . . . .

.
.
.
.
.

.
.
.
.
.

.
.
.
.

.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.

.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

108
109
112
114
115

Chapter 7. Scheduling in Distributed Real-Time Systems . . . . . . . . . . 117
Emmanuel G ROLLEAU, Michaël R ICHARD, and Pascal R ICHARD
7.1. Introduction . . . . . . . . . . . . . .
7.2. Generalities about real-time systems

7.3. Temporal correctness . . . . . . . . .
7.4. WCRT of the tasks . . . . . . . . . .
7.5. WCRT of the messages . . . . . . .
7.6. Case study . . . . . . . . . . . . . . .
7.7. Conclusion . . . . . . . . . . . . . .
7.8. Bibliography . . . . . . . . . . . . .

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.

.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.

.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.

.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.

.
.
.
.
.
.

117
118
122
126
142
149
154
155

Chapter 8. Software Engineering for Adaptative Embedded Systems . . . 159
Etienne B ORDE
8.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
8.2. Adaptation, an additional complexity factor . . . . . . . . . . . . . . . . 160
8.3. Theoretical aspects of adaptation management . . . . . . . . . . . . . . 163

www.it-ebooks.info


Contents

8.4. Technical solutions for the design of adaptative embedded systems .
8.5. An example of adaptative system from the robotic domain . . . . .
8.6. Applying MDE techniques to the design of the robotic use-case . .

8.7. Exploitation of the models . . . . . . . . . . . . . . . . . . . . . . . .
8.8. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
8.9. Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

.
.
.
.
.
.

.
.
.
.
.
.

7

171
176
177
184
188
189

Chapter 9. The Design of Aerospace Systems . . . . . . . . . . . . . . . . . . 191
Maxime P ERROTIN, Julien D ELANGE, and Jérôme H UGUES
9.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

9.2. Flight software typical architecture . . . . . . . . . . . . . . . .
9.3. Traditional development methods and their limits . . . . . . . .
9.4. Modeling a software system using TASTE: philosophy . . . .
9.5. Common solutions . . . . . . . . . . . . . . . . . . . . . . . . .
9.6. What TASTE specifically proposes . . . . . . . . . . . . . . . .
9.7. Modeling process and tools . . . . . . . . . . . . . . . . . . . .
9.8. Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
9.9. Model transformations . . . . . . . . . . . . . . . . . . . . . . .
9.10. The TASTE run-time . . . . . . . . . . . . . . . . . . . . . . .
9.11. Illustrating our process by designing heterogeneous systems .
9.12. First user feedback and TASTE future . . . . . . . . . . . . .
9.13. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
9.14. Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . .
T HIRD PART. S ECURITY IN D ISTRIBUTED S YSTEMS

.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.

191
193
195
197
199
200
201
208
209
213
215
224
225
226

. . . . . . . . . . . . 229

Chapter 10. Introduction to Security Issues in Distributed Systems
Laure P ETRUCCI
10.1. Problem . . . . . . . . . . . . . . . . .
10.2. Secure data exchange . . . . . . . . . .
10.3. Security in specific distributed systems
10.4. Outline of Part III . . . . . . . . . . . .
10.5. Bibliography . . . . . . . . . . . . . . .

.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

. . . . 231
.
.
.
.
.

.
.
.
.
.

.
.
.
.

.

.
.
.
.
.

.
.
.
.
.

231
233
234
234
235

Chapter 11. Practical Security in Distributed Systems . . . . . . . . . . . . 237
Benoît B ERTHOLON, Christophe C ÉRIN, Camille C OTI,
Jean-Christophe D UBACQ and Sébastien VARRETTE
11.1. Introduction . . . . . . . . . . .
11.2. Confidentiality . . . . . . . . . .
11.3. Authentication . . . . . . . . . .
11.4. Availability and fault tolerance
11.5. Ensuring resource security . . .

.

.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.

.
.
.
.

.
.
.
.
.

www.it-ebooks.info

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

237
249
252
261
278


8

Distributed Systems

11.6. Result checking in distributed computations . . . . . . . . . . . . . . . 283
11.7. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291
11.8. Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292
Chapter 12. Enforcing Security with Cryptography . . . . . . . . . . . . . . 301
Sami H ARARI and Laurent P OINSOT
12.1. Introduction . . . . . . . . . . . . . . . . . . .

12.2. Cryptography: from a general perspective . .
12.3. Symmetric encryption schemes . . . . . . . .
12.4. Prime numbers and public key cryptography .
12.5. Conclusion . . . . . . . . . . . . . . . . . . . .
12.6. Bibliography . . . . . . . . . . . . . . . . . . .

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.

.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.

.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

301
303
308
324
328
329

List of Authors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333

www.it-ebooks.info


Foreword

It is hard to imagine today a single computation that does not rely on at least one

distributed system directly or indirectly. It could be a distributed file system, a distributed database, a content distribution network, a peer-to-peer game, a remote malware detection service, a sensor network, or any other distributed computation. Distributed systems have become the equivalent of economic globalization in the world
of computing. Adopted for economic reasons, powered by highly efficient and ubiquitous networking, distributed systems define the default architecture for almost every
computing infrastructure in use today.
Over the last two decades, distributed systems have taken many shapes and forms.
Clusters of computers were among the earliest generations of distributed systems,
whose goal was to provide a cost-effective alternative to highly expensive parallel
machines. File servers were first to evolve from the cluster-based distributed system
model to serve an increasing hunger for storage. The World Wide Web introduced
the web server and, with it, the client-server distributed system model, on which millions of other Internet services have been built. Peer-to-peer systems appeared as an
“anti-globalization movement”, in fact an anti-corporate globalization movement that
fought against the monopoly of the service provider in the client-server model. Cloud
computing turned distributed systems into a utility that offers computing and storage
as services over the Internet. One of the emerging and least expected beneficiaries
of cloud computing will be the mobile world of smart phones and personal devices,
whose resource limitation can be solved through computation offloading. At the other
end, wireless networking has initiated the use of distributed systems in sensor networks and embedded devices. Finally, online social networking is providing a novel
use for distributed systems.
With this multitude of realizations, distributed systems have generated a rich set
of research problems and goals. Performance was the first one. However, although
the performance of distributed systems has increased, there has been a resultant increase in the programming burden. For a decade, research in distributed systems

ix

www.it-ebooks.info


10

Distributed Systems

had tried to reconcile performance and programmability by making the distribution
of computation transparent to the programmer through software distributed shared
memory. In the end, things have not become simpler as achieving performance under
distributed shared memory comes with a non-negligible semantic cost caused by the
relaxed memory consistency models.
With the shift of distributed systems towards file systems and Internet-based services, the research changed focus from performance to fault tolerance and availability.
More recently, the ubiquity of distributed system architecture has resulted in an increased research interest in manageability aspects. Concerns of sustainability resulted
in energy-aware distributed servers, which essentially proposed dynamic reconfiguration for energy saving without performance loss. In the mobile arena, wireless networking introduced the important issues of location-awareness, ad-hoc networking,
and distributed data collection and processing. Finally, as computation and storage is
increasingly offloaded to the cloud, issues of security and privacy have recently gained
momentum.
This book is a journey into three domains of this vast landscape of distributed systems: large-scale peer-to-peer systems, embedded and real-time systems, and security
in distributed systems. The authors have recognized expertise in all three areas, and,
more importantly, the experience of building real distributed systems. This book reflects the expertise of its authors by balancing algorithms and fundamental concepts
with concrete examples.
Peer-to-peer systems have generated a certain fascination amongst researchers. I
see at least two reasons for this. First, peer-to-peer systems come from the position of
the challenger who wants to take away the crown from the long-reigning client-server
model. Essentially, the challenge is whether it is possible for a democratic society
of systems to function efficiently without leadership. I am not sure whether history
has ever proven that this is possible, but the peer-to-peer systems researchers have
shown it to be possible. They employed efficient peer-to-peer data structures called
distributed hash tables (DHT) to achieve scalable data retrieval when peers come and
go, fail or misbehave.
Tribal instinct might also be responsible for our interest in peer-to-peer systems: it
is more likely to seek help from our peers whenever possible rather than from the outsiders. This may explain the popularity of peer-to-peer applications, such as Gnutella,
BitTorrent, and the peer-to-peer games discussed in the book, some of them (Gnutella)
developed even before researchers showed how to design peer-to-peer systems efficiently.
However, take heed, occasionally, peer-to-peer systems can be an illusion. Popular
social networks today may look like peer-to-peer systems to the user, but, in reality,

their implementation is heavily centralized. Recent concerns of data ownership and

www.it-ebooks.info


Foreword

11

privacy have triggered an appetite for building truly peer-to-peer online social networks. It is better to understand how peer-to-peer systems work rather than be fooled
again.
The distributed embedded and real-time systems, which make the middle part of
the book, take distributed systems’ computing labs or centers, into the real, uncontrollable world. Whether embedded in cars, buildings, or our own bodies, embedded
systems must function without continuous operator assistance, adapting their functionality to the changing demands of the physical systems they assist or control. Physical
systems may also incorporate highly inter-connected embedded computers in order
to become cyber-physical systems. Computer scientists have always been good at designing systems for themselves: languages, operating systems, and network protocols.
However, embedded systems are about others. They represent a prerequisite in implementing Mark Weiser’s vision of pervasive computing, according to which computers
will not just become ubiquitous, but also invisible.
Embedded computing often demands real-time guarantees, a requirement that has
been shown to be challenging for any kind of computing, not just for distributed systems. This part of the book covers distributed real-time systems, how to build adaptive
embedded systems from a software engineering perspective, and concludes with an
interesting real-world example of software design for an aerospace system using the
modeling tool they developed. After reading this book, whenever you fly, I am sure
you will hope that the engineer who designed the plane’s software has read it too.
Finally, the last part of the book covers security in distributed systems. Distributed
systems inherently require security. Whether they are clients and servers or just peers,
these parties, as in real life, rarely trust each other. The authors present key aspects
of grid systems’ security and dependability such as confidentiality, authentication,
availability, and integrity. With the increasing popularity of cloud computing, security
and privacy issues will be an even greater concern. Virtual machine environments are

shown not to be sufficiently trustworthy as long as they are in the hands of the cloud
providers. Users are likely to ask for stronger assurances, which may come from
using the Trusted Platform Module (TPM) support, presented in this book, as well as
from intelligent auditing techniques. The book’s last section is about cryptography,
the mystical part of computer science, which we always rely on when it comes to
protecting the confidentiality of our communications.
Who should read the book? The authors recommend it for engineers and masters
students. I am inclined to agree with them that this book is certainly not for the
inexperienced. It requires some background knowledge, but also the maturity to read

www.it-ebooks.info


12

Distributed Systems

further from the recommended bibliography in order to fully internalize the material.
If you finish the book and want to read more, stay tuned; this is just the first book:
more is coming.

Professor Liviu Iftode
Rutgers University

www.it-ebooks.info


Chapter 1

Introduction

Problematics
Most do not know but, in 1946, ENIAC, one of the first known computers (after
Z3 in 1941 and Colossus in 1943), was already a parallel machine [WIK 11a]. The
very basic programming mechanisms 1 of this computer hid this capability that was
never really exploited.
Then, for years, programming remained sequential. In the 1960s, interest in parallel programming increased again. Two approaches were then explored:
– supercomputers;
– multi-processor computers.
Parallel programming on supercomputers
In the 1960s, the notion of supercomputers emerged. It was a brute-force 2 computer able to perform complex scientific calculi such as meteorologic predictions.
Technologies of this time made such computers extremely costly. As an example,
the Cray-1 (1976), which constituted a major achievement in this domain, cost US$

Introduction written by Serge H ADDAD, Fabrice KORDON, Laurent PAUTET and Laure
P ETRUCCI.
1. From documents and pictures, it is clear that ENIAC looked more like an old-fashioned telephone
center than a modern computer. It was programmed by means of cables wiring switches.
2. As a comparison, a modern laptop is far more efficient than the Cray Y-MP from 1988, the most
powerful computer at this time.

Distributed Systems: Design and Algorithms
Edited by Serge Haddad, Fabrice Kordon, Laurent Pautet and Laure Petrucci
© 2011 ISTE Ltd. Published 2011 by ISTE Ltd.

13

www.it-ebooks.info

14

Distributed Systems

8.8 million [CRA 11]. It weighed 5 tons and performed 166 MegaFLOPS (FLOPS
means FLoating point Operations Per Second). It required a dedicated electrical
power supply and had a very complex cooling system.
Parallelism in such computers was mainly based on the “vector calculus” principle.
The idea is to apply a given operator to a vector instead of a scalar. This objective
was achieved thanks to pipe-line in processors, which enabled “pseudo-parallelism”
to process several data simultaneously. The same operation was performed on several
data but at different stages in the pipe-line.
The Cray Y-MP, in 1988, was based on this principle but also comprised parallel
configurations from four up to eight vector units. Then, two types of parallelism coexist: vector-based and multi-processor based.
FORTRAN was the traditional programming language for numerical applications.
It was rapidly extended to enable parallel programming. Other languages were enriched with new instructions or associated libraries to also handle parallelism. However, to get all benefits from such computers, a deep understanding of their architecture was required. Then, when a computer was replaced by a new one implementing
a more efficient architecture, programs had to be thoroughly rewritten to exploit the
new capabilities. This additional cost could only be supported by large institutions
such as the army, aircraft manufactories, state research agencies, etc.
Parallel programming on multi-processor machines
During the second half of the 1990s, when network usage was growing, some
started to build clusters of computers. It was first a way to build “supercomputers for
the poor” by assembling PC motherboards connected via a dedicated local network.
The time of the first configurations involving 64 to 128 nodes is now gone: current
clusters are gathering thousands of machines via high-speed networks such as glass
reinforced plastic fibers. The Jaguar machine 3 is made of 18,688 processors, each one
being an hex-core (that makes 112,128 cores in total) [WIK 11b]. Its peak computing
capacity is 1.75 PetaFLOPS (1015 FLOPS).
The cost of clusters dramatically increased the affordability of supercomputers
and thus almost all the fastest machines in the world are of this type. Most companies

selling “traditional supercomputers” have reduced their activity.
Another aspect made this new generation of supercomputers popular; their programming is much easier and reusable compared with the old ones. This is because
the programming paradigm is that of a distributed application that is not architecture

3. This was the fastest computer recorded in June 2010 [TOP 11].

www.it-ebooks.info


Introduction

15

dependent. Hence, “classical” distributed programming techniques can be used and
preserved when transferring the program from one machine to another.
In fact, the Internet itself can be seen as a gigantic supercomputer, as applications
like SETI@home [UCB 11] do. Thus, some experiments involve dozens of thousands
of machines over the Internet. The main difference with clusters is that the nodes
are not connected via a high-speed network. There is also a need to check for trust
between nodes.
Thus, the problem of distributed computing is now mainly a software problem.
However, distributed applications belong to a difficult class of problems.

Objectives of this book
This book is aimed at engineers or masters students or anyone familiar with algorithmic and programming who wants to know more about distributed systems.
We deliberately chose, in this first book, to group the presentation of distributed
systems in relation to their design and their main principles. To do so, we present both
the main algorithms and replace them in their application context (i.e. consistency
management and the way they are used in distributed file systems).

Description of chapters
The first part is dedicated to large-scale peer-to-peer distributed systems. This
is currently a very active area with new improvements (especially those induced by
mobility and the numerous small devices we use daily):
– Chapter 3 presents the main principles of large-scale distributed peer-to-peer
systems. It details the main algorithms used to communicate and ensure trust in such
systems.
– Chapter 4 deals with peer-to-peer storage, an application domain which has already accumulated several years of experience. Some well-known protocols and tools,
such as BitTorrent and Gnutella, are detailed.
– Chapter 5 presents another hot application domain for such systems: gaming.
Once again, the principles adapted to this class of applications are put into a practical
perspective.
The second part is dedicated to distributed real-time embedded systems: a domain
that has always been very active. The topic of distributed systems is now gaining
importance in the design of the next real-time systems generation.

www.it-ebooks.info


16

Distributed Systems

– Chapter 7 presents the holistic analysis, a well-known method used to compute
the schedulability of distributed real-time systems. This chapter provides some background knowledge of scheduling analysis in the distributed real-time systems area.
– Chapter 8 deals with the design of adaptative real-time embedded systems. This
second contribution provides the results for some schedulability theories, it also details some of the fundamental insights of the design process of adaptative real-time
embedded systems.
– Chapter 9 presents an innovative approach to designing the new generation space
systems. This approach is supported by a toolset which is required to automate the

process where possible.
The third part is devoted to security issues in distributed systems. This is a critical
area that is of the utmost importance in such systems where trust among communicating entities and confidentiality of data are key issues:
– Chapter 11 presents the main characteristics of grid computing, with a focus on
security. The security properties that have to be guaranteed are detailed, and how they
are achieved in practice is presented through several case studies.
– Chapter 12 tackles the issue of data confidentiality using cryptography. It describes the core techniques which use symmetric key and public key protocols, and
details their main characteristics.
The MeFoSyLoMa community
MeFoSyLoMa (Méthodes Formelles pour les Systèmes Logiciels et Matériels 4) is
an association gathering several world-renowned research teams from various laboratories in the Paris area [MEF 11]. It is composed of people from LIP6 5 (P. & M.
Curie University), LIPN 6 (University of Paris 13), LSV 7 (École Normale Supérieure
de Cachan), LTCI 8 (Telecom ParisTech), CÉDRIC 9, (CNAM), IBISC 10 (University
of Évry-Val-d’Esssone), and LACL 11 (University of Paris 12). Its members, approximately 80 researchers and PhD students, all have common interest in the construction
of distributed systems and promote a software development cycle based on modeling,
analysis (formal), and model-based implementation. This community was founded in
2005 and is federated by regular seminars from well-known researchers (inside and

4.
5.
6.
7.
8.
9.
10.
11.

This acronym stands for Formal Methods for Software and Hardware Systems (in French).
Laboratoire d’Informatique de Paris 6.
Laboratoire d’Informatique de Paris Nord.

Laboratoire de Spécification et de Vérification.
Laboratoire Traitement et Communication de l’Information.
Centre d’Études et de Recherche en Informatique du CNAM.
Informatique, Biologie Intégrative et Systèmes Complexes.
Laboratoire d’Algorithmique, Complexité et Logique.

www.it-ebooks.info


Introduction

17

outside the community) as well as by common research activities and the organization
of events in their domains such as conferences, workshops, or book writing.
The editors of this book, as well as most authors, are from this community.
Bibliography
[CRA 11] C RAY- RESEARCH, “Cray history”, />2011.
[MEF 11] M E F O S Y L O M A, “MeFoSyLoMa, home-page”, www.mefosyloma.fr 2011.
[TOP 11] TOP500. ORG, “BlueGene/L”, 2011.
[UCB 11] UCB, “SETI@home”, 2011.
[WIK 11a] W IKIPEDIA, “ENIAC”, 2011.
[WIK 11b] W IKIPEDIA, “Jaguar (computer)”, />2011.

www.it-ebooks.info


F IRST PART

Large Scale Peer-to-Peer Distributed Systems

19

www.it-ebooks.info


Chapter 2

Introduction to Large-Scale Peer-to-Peer
Distributed Systems

2.1. “Large-Scale” distributed systems?
For several years now, the term “Large-Scale” has applied to distributed systems.
It indicates that they involve a very large number of computers that are usually spread
worldwide.
As a typical example, we are all regular users of a large-scale distributed system: the Internet. So, the “Internet universe” provides a rough idea of the way such
systems work. How programs contact and cope with other programs (such as the relationship between a web browser and a web server) can be easily imagined. Moreover,
it is possible to feel how the Internet is a worldwide parallel machine of which each
connected computer is a part. The structure of such distributed systems can thus be
comprehended by anyone regularly using the Internet.
However, only a few really understand the implication of “large scale” regarding
such distributed systems. Creating a program to be executed on a few (or several
dozens) nodes is completely different from the same exercise for a few thousand nodes
or more. This is a problem when ubiquitous systems 1 become more and more present
in our day-to-day life.
Chapter written by Fabrice KORDON.
1. This denotes the set of devices (computer, PDA, cellular and smart phones, specialized devices, etc.)
connected into a network in a transparent way. Thus, users deal with several media in their day-to-day life
to accomplish personal or professional matters.

Distributed Systems: Design and Algorithms
Edited by Serge Haddad, Fabrice Kordon, Laurent Pautet and Laure Petrucci
© 2011 ISTE Ltd. Published 2011 by ISTE Ltd.

21

www.it-ebooks.info


22

Distributed Systems

2.2. Consequences of “large-scale”
What are the consequences of having “large-scale” distributed systems? Simply
the number of components in the involved systems (to illustrate the concepts, we consider here that all components are programs, even if the human factor is important
too). The following are some of the problems encountered:
– communication;
– fault tolerance;
– decision making.

2.2.1. Communication in large-scale distributed systems
For distributed systems, every component can be connected to any other. However,
such a connectivity cannot be maintained when the system grows. It is obvious that
the management of dozens of thousands of connections cannot be handled by a single
program.
Point-to-point communication can no longer be adapted; new mechanisms are required, such as broadcast or multicast. It should be noted that such mechanisms already exist (for example, they are used in the Internet, when a new router is connected
to the global network).
Thus, one of the main characteristics of communications in large-scale distributed
systems is their anonymous aspect. Components of such systems communicate without knowledge of their location or identity (e.g. IP 2 address). This is due to the fact

that during the system execution, due to the network load, or faults mentioned in the
next section, the location of the components may change. Thus detecting a component
via its IP address is absurd.

2.2.2. Fault tolerance in large-scale distributed systems
No problems are usually encountered during one hour of computing on a single
host. This is not the case when this computation time employs 10,000 machines. The
probability of the occurrence of failure tends to 100%. In fact, the more CPUs 3 that
are involved in computing, the higher the probability that a failure will occur in the
involved machines (host crash, network breakdown, etc.).

2. IP stands for “Internet protocol”.
3. CPU stands for “central processing units”.

www.it-ebooks.info


Introduction to Large-Scale Peer-to-Peer Distributed Systems

23

Thus, it is necessary to create dedicated mechanisms to handle such failures and
ensure that the program is able to overcome these. Such mechanisms have a dramatic
impact on the software architecture of large-scale distributed systems.
2.2.3. Decision making in large scale distributed systems
Decision making in large-scale distributed systems is a difficult task. If, when parallelism is limited, each component (piece of program) has a full copy of the involved
data, such hypothesis cannot be considered when the number of hosts increases: managing and updating data would require too many resources.
Thus, each component in the distributed system is expected to handle decisions
based on a (very) partial view of the global system state. Collection of the involved
data remains a difficult task that also has far-reaching impacts on the software architecture.

2.3. Some large-scale distributed systems
Earlier, we mentioned the Internet as an example of a large-scale distributed system. In fact, this example is not typical because the Internet itself consists of a worldwide network of machines (evaluation in June 2010 was 1.97 billion users according
to [IWS 10]); most users only operate a very limited number of machines simultaneously (typically, a web server, composed of a cluster of machines to handle services –
most online vendors have such an architecture).
Thus, to provide a more accurate view of large-scale distributed systems, we briefly
present two examples that emphasize their main characteristics.
2.3.1. SETI@home
Our first example is an application launched in 1999. An improved version is still
in use today (the first version is now called SETI@home “classic”).
When SETI@home [UCB 11] was released, it raised a huge interest from both Internet users and computer science professionals. The main objective of this application
is to exploit the useless CPU time of machines connected to the Internet to compute
and analyze radio signals coming from outer space. The objective is to look for traces
of non-human intelligence 4.

4. SETI is the acronym for Search for Extra-Terrestrial Intelligence.

www.it-ebooks.info


24

Distributed Systems

The way SETI@home operates is quite simple. Users willing to offer unused CPU
capacity download an application to create an account on a server and set-up its configuration (enabling the conditions for SETI to activate, etc.).
Once launched, the application remains paused until the host computer becomes
inactive. Then, it connects to the server, downloads data to be analyzed and, once
computation is over, uploads the results to the server. If the host machine resumes its
activity, the application stops until the next inactive period.
SETI@home met great success in the Internet community and more than 5 million

people have used it since it was launched. From a performance point of view, it is
also a success as the computation strength has increased up to 364.8 TeraFLOPS 5 in
January 2008 with about 1.7 million simultaneous users. As a comparison, let us note
that the most powerful computer is about 1,759 TeraFLOPS 6 for an incredible cost.
However, no non-human activity has been detected so far.
Let us consider the three main problems noted in section 2.2 and the way they are
addressed by SETI@home:
– communications: SETI@home mainly relies on the classic Internet architecture,
the so-called client-server approach. The downloaded application connects to a server
to obtain data and sends back the results to this same server;
– fault tolerance: its management is simple. If a participating machine (a client)
does not send back results for a while, the server may resend the set of data to another
client. Let us note that users trust the system as they accept the program to run on
their computer, which communicates with another machine. They expect SETI@home
not the behave in a malicious way;
– decision making: all decisions are taken by servers, clients only analyze local
data they retrieve from the server.
2.3.2. The automated motorway
Our second example concerns intelligent transport systems (ITS). These remain a
challenge as there is no implementation yet. However, this is an important and active
research domain in the USA, Japan, and in Europe, with investments involving billions
of Euros and concerning numerous applications. Here, we will focus on one of them:
the automatic motorway (seen from the perspective of distributed systems).
The objective of the automated motorway is to let especially equipped vehicles
drive without human intervention. Such a system is of interest for main roads; it is
5. 1 TeraFLOP corresponds to 1012 FLOPS.
6. This is, in June 2010, the Jaguar system at NCSS, a 18,688 node computer (each node runs a dual hexcore AMD Opteron 2435 (Istanbul) processors running at 2.6 GHz, 16 GB of DDR2-800 memory [WIK 11].

www.it-ebooks.info

Introduction to Large-Scale Peer-to-Peer Distributed Systems

25

also intended for trucks. Let us note that some experimentation started around the mid1990s in the context of the PATH project [UCC 04], whereby a platoon of automated
vehicles successfully drove along a dedicated freeway. However, due to the cost of
road adaptation, this solution got no further and was abandoned.
Based on these first successes and due to technological advances, the principle has
changed. Current solutions tends to minimize the adaptation of infrastructures and put
more “intelligence” in the vehicles.
Let us now draw a more up-to-date vision of the automated motorway. The components if the system are:
– the motorway infrastructure, which offers lanes, communication mechanisms,
and global information about the road itself (such as speed limit, information about
traffic or accidents, etc.). The network may only exist from time to time (e.g. WIFI
communication spots may be located alongside emergency phones);
– vehicles, which are equipped with sensors and a network interface enabling local communication with both other vehicles and the road infrastructure. Only local
communication are needed (e.g. with close vehicles only).
A priori, each vehicle communicates with its close neighbors. Information (e.g.
acceleration and speed data, emergency braking, etc.) is propagated to the surrounding
vehicles. Information then travels faster than it does from driver to driver (when they
notice other drivers’ behavior).
Vehicles can also receive messages from the road infrastructure.

section s

section s+1 section s+2 section s+3

“local”
level

“global”
level

Such a system can be structured as a hierarchy of subsystems (see Figure 2.1).
Here, there are two levels of hierarchy, each one dedicated to a given set of services.

Group i+1

Group i

Figure 2.1. Possible hierarchical architecture for the automated motorway

The first level (called “local” in Figure 2.1) deals with a group of circulating vehicles. Obviously, this group is composed dynamically and evolves when cars enter
or leave the motorway. It is also possible that some vehicles go from one group to
another when their speed becomes slightly higher than another group (overlapping

www.it-ebooks.info


26

Distributed Systems

between groups can also be considered, in which case, one vehicle may belong to two
groups for a while).
Inside the group, the security of each vehicle must be handled. To do so, information is shared between vehicles and possibly the road infrastructure. In which case,
decisions are made by vehicle controllers to avoid collisions and ensure safety.
The second level (called “global” in Figure 2.1) focuses on higher-level services
and does not handle collision avoidance at all. Its objective is to propagate information regarding unexpected events, such as an accident or bad weather conditions (e.g.

localized fog). Any change in the dynamic of the system (e.g. a sudden braking of a
group) must be analyzed and backward propagated to let later vehicles anticipate the
problem if necessary.
Traffic management (to prevent traffic-jams) can also be handled at that level. The
system may decide to reduce the speed limit of some sections to limit the number of
vehicles coming to a section where an unexpected event is occurring.
Let us consider the three main problems noted in the section 2.2 and the way they
are addressed here:
– communications: each group of vehicles must have its own broadcast mechanism. If a vehicle belongs to two groups, it receives information from the two groups
and may propagate data backward (e.g. from the front group to the back group).
Thus, information goes from hop to hop. When a vehicle sends information to the
group channel, it does not have an idea of which participant will get this information
(anonymous communication). The principle is that “involved vehicles” will get the
appropriate data to operate;
– fault-tolerance: such systems require fault-tolerance mechanisms. Typically, a
vehicle losing connection with the group must continue to behave safely within the
group (until communication is back).
Vehicles must trust each other: a situation where a component sends information
to go faster than other vehicles or cause a crash must be avoided.
– decision making: at the local level, decisions are probably taken by vehicles. At
the global level, decisions could be taken by server. There may only be one server
for the motorway or a set of servers, each one dealing with a set of sections (and then
communicating together or to a higher-level server that handles the motorway).
2.4. Architectures of large scale distributed systems
During the early stages of parallel programming, computer scientists tried to extend “classical” mechanisms to produce a distributed execution. Thus, the notion of
procedure call was extended to remote procedure call (RPC) [SUN 88].

www.it-ebooks.info

Introduction to Large-Scale Peer-to-Peer Distributed Systems

27

2.4.1. Remote procedure call
Figure 2.2 shows the behavior of a RPC. The two components involved have asymmetric behaviors. When the invoker is ready (1 in Figure 2.2), it generates a request
to the invoked that holds the code to be executed and waits for an answer. Once the
invoked receives the request (there may be a list of pending requests if the invoked
receives many of them), it is processed (2 in Figure 2.2) and then an answer is sent
back to the invoker. When the answer is received, the invoker conintues its execution
(3 in Figure 2.2).
The first message contains the parameter required by the code to be executed.
The second message contains return values (or modified parameters). When there is
no return value, an empty message is sent as the RCP is a synchronous mechanism
(the invoker must not continue its execution before the end of the remote procedure
execution).
invoker

Invoked

1
m

u
2
m

u
3

Figure 2.2. The Remote Procedure Call mechanism

Some typical characteristics of distributed systems must be outlined there:
– the two hosts executing the invoker and the invoked may have different architectures. the data format may thus differ (e.g. an integer is stored on 32 bits for the first
machine, on 64 bits for the second one). It is then necessary to encode data instead of
sending a memory segment. The marshaling operation (m in Figure 2.2) encodes data
into a message and the unmarshaling operation (u in Figure 2.2) decodes this message
to reconstitute the original data. Both marshaling and unmarshaling must respect the
same conventions of course;
– identification of the invoked is explicit. Thus, each component in the system
must “know” each actor to interact with;
– finally, a crash of the machine hosting the invoked makes the RPC unavailable.

www.it-ebooks.info


28

Distributed Systems

RPC rely on two principles: message passing and point-to-point communication.
This is the case for any high-level communication mechanism that requires the exchange of messages over the network. A mechanism like RPC aims to structure interactions between components into a protocol.
Parallel to the creation of RPC, scientists were formulating the main Internet mechanisms based on simple message passing such as broadcast and multicast. So, if the
Internet follows the client/server protocol (an extension of RPC presented in the next
section), it is based on message passing (this is a peer-to-peer approach that is also
presented later in this chapter).
2.4.2. The client/server model
The client/server (see Figure 2.3) is a natural extension of the RPC. In the early
1990s, middleware such as CORBA [OMG 06] extended this initial notion to the object model. Registration mechanisms enable the dynamic identification of the server
address when necessary and it is not necessary to encode its IP address in the program.

This is an important evolution that increases the portability and enables evolution of
distributed applications.
Client1

Client

2

Server

...
Client

N

Figure 2.3. Architecture of the client/server model

However, the client/server model remains a point-to-point protocol in which clients
initiate contact with a server. The server is reactive and only answers requests.
The context of the system (that enables decision making) is usually centralized in
the server. Thus, to tolerate faults, the server must be replicated on another machine.
This approach is often used for large servers (e-commerce, Google, etc.).
2.4.3. The master/slaves model
The master/slaves model (see Figure 2.4) is a variant of the client/server model. In
that model, all initiative is taken by the “master” that provides jobs to the “slaves”.
In that model, the “slaves” are the reactive ones and the communication employs
the point-to-point model. The “master” handles the context of the application and

www.it-ebooks.info