Lecture Operating systems: A concept-based approach (2/e): Chapter 18 - Dhananjay M. Dhamdhere

  
PROPRIETARY MATERIAL. ©  2007 The McGraw­Hill Companies, Inc. All rights reserved. No part of this PowerPoint slide  may be displayed, reproduced or distributed 
in any form or by any means, without the prior written permission of the publisher, or used beyond the limited distribution to teachers and educators permitted by McGraw­Hill 
for their individual course preparation. If you are a student using this PowerPoint slide, you are using it without permission. 

Chapter 18: Recovery and
Fault Tolerance

Dhamdhere: Operating Systems—
A Concept­Based Approach, 2 ed

Slide No: 1
Copyright © 2008


Introduction

•

A fault may have several consequences
– It may damage state of some data or processes, leading to
* Malfunctioning of a server
* Non-availability of resources and services
* Disruption of system operation

– Its consequences can be avoided using three approaches

  

* Recovery
 Some data or processes are rolled back to previous states

* Fault tolerance
 Provides un-interrupted operation of a system despite faults
* Resiliency
 Reduces cost of re-execution following a fault

– Recovery is the generic term for all three approaches
Chapter 18: Recovery and
Fault Tolerance

Dhamdhere: Operating Systems—
A Concept­Based Approach, 2 ed

Slide No: 2
Copyright © 2008


Faults and failures
• A fault damages the state of a system
– We say it causes an error in state of a system
– It leads to unexpected behavior, which we call a failure
– Recovery restores the system to an error-free state

  

Chapter 18: Recovery and
Fault Tolerance

Dhamdhere: Operating Systems—
A Concept­Based Approach, 2 ed

Slide No: 3
Copyright © 2008


Recovery after a fault

•  System operation is initiated in state S0 at time 0
’
; a failure is detected at time t
when state is S
•  A fault occurs at time t
1
i   
i
  
•  Recovery puts the system into a new state Snew , which is error­free

Chapter 18: Recovery and
Fault Tolerance

Dhamdhere: Operating Systems—
A Concept­Based Approach, 2 ed

Slide No: 4
Copyright © 2008


Classes of faults

•

A fault model describes properties of a fault
– System fault is a system crash caused by a power outage or
component fault
* Amnesia fault
 The system loses its state completely
* Fail-stop fault
 The system stops operating when a fault occurs
 This property permits an error in system state to be corrected

  

* Byzantine fault
 A process suffering this fault behaves maliciously
* Storage fault
 Bad block on a store medium

Chapter 18: Recovery and
Fault Tolerance

Dhamdhere: Operating Systems—
A Concept­Based Approach, 2 ed

Slide No: 5
Copyright © 2008


Recovery techniques

•

Four techniques of recovery
– Data recovery
* State of data or file is restored to that in its latest back-up

– Process recovery
* A checkpoint is a recording of the state of a system
* At a fault, a process is rolled back to the state in some checkpoint

– Fault tolerance
* The error in state is corrected without interruption of system
operation

– Resiliency

  

* Some results produced by a computation before a fault occurred are
used during recovery

Chapter 18: Recovery and
Fault Tolerance

Dhamdhere: Operating Systems—
A Concept­Based Approach, 2 ed

Slide No: 6
Copyright © 2008

Recovery techniques

•

Two classes of recovery techniques
– Backward recovery
* The state of an entity or application is reset to a prior state
 Simple to implement
 However, state recording should be both feasible and practical

– Forward recovery
* The erroneous state of an entity or application is repaired

  

Chapter 18: Recovery and
Fault Tolerance

Dhamdhere: Operating Systems—
A Concept­Based Approach, 2 ed

Slide No: 7
Copyright © 2008


Byzantine faults and agreement protocols

•

A faulty entity may perform wrong actions; a fault

tolerance technique is used to avoid their consequences
– The Byzantine agreement problem: A group of processes must
agree on a value
* Initiator broadcasts the value to all other processes
* On receiving the value, each process broadcasts it to all others
 A non-faulty process broadcasts the received value
 A faulty process may broadcast a different value

  

* All non-faulty processes must conclude the correct value
* Lamport’s protocol involves m+1 rounds of messages, where
number of faulty processes does not exceed m
* Impossibility result: a group of 3 processes involving one faulty
process cannot reach agreement

Chapter 18: Recovery and
Fault Tolerance

Dhamdhere: Operating Systems—
A Concept­Based Approach, 2 ed

Slide No: 8
Copyright © 2008


Recovery schemes

•

A recovery scheme consists of two algorithms:
– Checkpointing algorithm
* Decides when and how to create checkpoints of processes
* Cij denotes the j’th checkpoint of process Pi
* State (Cij) denotes the state of process Pi in checkpoint Cij

– Recovery algorithm
* When a failure is noticed, it decides which checkpoint a process
should be rolled back to

  

Chapter 18: Recovery and
Fault Tolerance

Dhamdhere: Operating Systems—
A Concept­Based Approach, 2 ed

Slide No: 9
Copyright © 2008


Checkpoints and recovery

•  At time tf , failure occurs in node N3 , where process P3 is in operation
  •  Rolling back P3 to C32 would be inconsistent because sending of 
   message m3 by P3  would be nullified, whereas P2 has received m3 
Chapter 18: Recovery and
Fault Tolerance

Dhamdhere: Operating Systems—
A Concept­Based Approach, 2 ed

Slide No: 10
Copyright © 2008


Orphan messages

•

Definition
– A message mk sent by process Pi to Pj is an orphan in the new
state of a system during recovery if the new state
* Records mk as received by Pj
* Does not record mk as sent by Pi

•

If rolling back of Pi makes mk an orphan, then Pj must be
rolled back to a state before it had received mk
  

Chapter 18: Recovery and
Fault Tolerance

Dhamdhere: Operating Systems—
A Concept­Based Approach, 2 ed

Slide No: 11

Copyright © 2008


Fault tolerance techniques

•

A fault tolerance technique ensures that
– Either a fault cannot cause an error
– Or the error can be removed easily
* It is typically achieved through data recovery

•

We discuss tolerance of storage faults
– Its two aspects are

  

* Fault tolerance for replicated data
 Data should remain available; processes should see correct
values
* Fault tolerance for distributed data
 Different parts of data should not become mutually inconsistent

Chapter 18: Recovery and
Fault Tolerance

Dhamdhere: Operating Systems—
A Concept­Based Approach, 2 ed

Slide No: 12
Copyright © 2008


Logs, forward recovery and backward recovery

•

A log is a record of activities in a process. Its contents
are used for fault tolerance and recovery
– Do log
* Records those actions that should be performed to ensure
correctness of state of an entity or system

– Undo log
* Records actions that should be undone to remove an error in state if
a fault occurs

– Write ahead logging
* A log record is written before an action is taken
* Ensures that the log contains correct entries when faults occur

  – Operation log contains a list of actions

– Value log contains a list of values of data
Chapter 18: Recovery and
Fault Tolerance

Dhamdhere: Operating Systems—

A Concept­Based Approach, 2 ed

Slide No: 13
Copyright © 2008


Reliable data replication

•

A quorum is a specified number of copies of data D that
should be accessed by a process
– Qr: quorum for reading, Qw: quorum for writing
– How to ensure that D can be updated by only one process but
can be read by many, and that a process would always read a
latest value of D?
*

2 x Qw > n

*
*

Qr + Qw > n
Use time-stamps to identify the latest value

– How to tolerate up to k failures?

  

*

Qr = k + 1

*
*

Qw = n – k
n>2xk

Chapter 18: Recovery and
Fault Tolerance

Dhamdhere: Operating Systems—
A Concept­Based Approach, 2 ed

Slide No: 14
Copyright © 2008


Reliable distributed data

•

Inability to update some parts of distributed data due to
link or node faults would cause data inconsistency
– A distributed transaction is used to avoid it
– The two phase commit protocol (also called 2 PC protocol) is
used to tolerate link or node faults
* It ensures one of the following conditions

 All sites containing parts of distributed data can perform
updates reliably
 None of the parts of the distributed data is updated

  

Chapter 18: Recovery and
Fault Tolerance

Dhamdhere: Operating Systems—
A Concept­Based Approach, 2 ed

Slide No: 15
Copyright © 2008


Two phase commit protocol
– Phase I
1. Actions of transaction coordinator:
a. Write a ‘Prepare Ti’ record in the log
b. Set a time-out and send a ‘Prepare Ti’ message to all nodes
participating in the transaction
2. Actions of a participating node:
a. If it is ready to commit, write updates in stable storage and a
‘Prepared Ti’ record in the log, and send a ‘Prepared Ti’
message to coordinator

  

b. Otherwise, write an ‘Abandoned Ti’ record in the log and send

an ‘Abandoned Ti’ message to coordinator

Chapter 18: Recovery and
Fault Tolerance

Dhamdhere: Operating Systems—
A Concept­Based Approach, 2 ed

Slide No: 16
Copyright © 2008


Two phase commit protocol
–

Phase II
1. Actions of transaction coordinator:
a. If it receives a ‘Prepared’ reply from all nodes before time-out
occurs, write a ‘Commit Ti’ record in the log and send ‘Commit
Ti’ messages to all nodes
b. Otherwise, write an ‘Abort Ti’ record in the log and send ‘Abort
Ti’ messages to all nodes
c. Wait for an acknowledgment from each node and write a
‘Complete Ti’ message in the log

  

2. Actions of a participating node: Depending on the coordinator’s
message,
a. Write a ‘Commit Ti’ record in the log and perform commit

processing
b. Write an ‘Abort Ti’ record in the log and abandon updates of Ti

Chapter 18: Recovery and
Fault Tolerance

Dhamdhere: Operating Systems—
A Concept­Based Approach, 2 ed

Slide No: 17
Copyright © 2008


Resiliency

•

Consider a distributed computation
– Parts of the computation are executed in different nodes
* These parts communicate through interprocess messages

– Faults can arise during its operation
* A node containing a part of the computation or its data fails
* A link fails

– However, failures are partial
* A node or link failure may affect only some part of the computation
* Resiliency techniques exploit this fact to reduce the recovery cost

  

 Some of the results computed before the failure are re-used
 Only some parts of the computation are re-executed

Chapter 18: Recovery and
Fault Tolerance

Dhamdhere: Operating Systems—
A Concept­Based Approach, 2 ed

Slide No: 18
Copyright © 2008


Nested transaction

•

A nested transaction Tik is a part of an atomic transaction
Ti
– It commits only if its parent transaction Ti commits
– It is implemented as follows
* When Tik completes, it is said to have reached a tentative commit
* When Ti wishes to commit, it checks whether all nested
transactions have reached a tentative commit and can participate in
commit processing
 It is implemented using a 2PC protocol

  

Chapter 18: Recovery and
Fault Tolerance

Dhamdhere: Operating Systems—
A Concept­Based Approach, 2 ed

Slide No: 19
Copyright © 2008


Nested transaction

•

Resiliency is implemented as follows:
– If a nested transaction Tik does not respond to a ‘Prepare’
message, the coordinator can retry Tik in the same node or in
some other node
* If Tik had reached tentative commit and its node had failed when
‘Prepare’ message was sent
* If the failed node recovers and the coordinator retries Tik in it
* The results of Tik, computed before the failure, can be used

  

Chapter 18: Recovery and
Fault Tolerance

Dhamdhere: Operating Systems—
A Concept­Based Approach, 2 ed

Slide No: 20
Copyright © 2008