Distributed Systems

F 6/7 - 1

Distributed Systems

DISTRIBUTED MUTUAL EXCLUSION

AND ELECTION

F 6/7 - 2

Mutual Exclusion

Mutual exclusion ensures that concurrent processes

make a serialized access to shared resources or data.
1. Mutual Exclusion in Distributed Systems
The well known critical section problem!
In a distributed system neither shared variables
(semaphores) nor a local kernel can be used in order
to implement mutual exclusion!
Thus, mutual exclusion has to be based exclusively
on message passing, in the context of unpredictable
message delays and no complete knowledge of the
state of the system.

2. Non-Token-Based Algorithms

3. Token-Based Algorithms

4. Distributed Election

5. The Bully and the Ring-Based Algorithms

Petru Eles, IDA, LiTH

Sometimes the resource is managed by a server

which implements its own lock together with the
mechanisms needed to synchronize access to the
resource mutual exclusion and the related
synchronization are transparent for the process
accessing the resource.
This is typically the case for database systems with
transaction processing (see concurrency controll in
Database course!)

Petru Eles, IDA, LiTH

Distributed Systems

F 6/7 - 3

Distributed Systems

Non-Token-Based Mutual Exclusion

Mutual Exclusion (contd)

Often there is no synchronization built in which
implicitely protects the resource (files, display
windows, peripheral devices, etc.).

Central Coordinator Algorithm

Ricart-Agrawala Algorithm

A mechanism has to be implemented at the level of

the process requesting for access.
Basic requirements for a mutual exclusion mechanism:
safety: at most one process may execute in the
critical section (CS) at a time;
liveness: a process requesting entry to the CS is
eventually granted it (so long as any process
executing the CS eventually leaves it).
Liveness implies freedom of deadlock and starvation.
There are two basic approaches to distributed mutual
exclusion:
1. Non-token-based: each process freely and
equally competes for the right to use the
shared resource; requests are arbitrated by a
central control site or by distributed agreement.
2. Token-based: a logical token representing the
access right to the shared resource is passed
in a regulated fashion among the processes;
whoever holds the token is allowed to enter the
critical section.

Petru Eles, IDA, LiTH

Petru Eles, IDA, LiTH

F 6/7 - 4

Distributed Systems

F 6/7 - 5

Distributed Systems

F 6/7 - 6

Central Coordinator Algorithm

A central coordinator grants permission to enter a CS.
P1

P2

P0

P2

P1
OK

P0

release

OK

request

P2

req
ues
t

P1

P0

Central Coordinator Algorithm (contd)

The scheme is simple and easy to implement.

The strategy requires only three messages per use
of a CS (request, OK, release).

Queue

Queue

Queue

Coordinator

Coordinator

Coordinator

To enter a CS, a process sends a request message

to the coordinator and then waits for a reply (during
this waiting period the process can continue with
other work).
The reply from the coordinator gives the right to
enter the CS.
After finishing work in the CS the process notifies
the coordinator with a release message.

Petru Eles, IDA, LiTH

Problems
The coordinator can become a performance
bottleneck.
The coordinator is a critical point of failure:
- If the coordinator crashes, a new coordinator
must be created.
- The coordinator can be one of the processes
competing for access; an election algorithm
(see later) has to be run in order to choose one
and only one new coordinator.

Petru Eles, IDA, LiTH

Distributed Systems

F 6/7 - 7

Distributed Systems

Ricart-Agrawala Algorithm

In a distributed environment it seems more natural to

implement mutual exclusion, based upon distributed
agreement - not on a central coordinator.
It is assumed that all processes keep a (Lamports)
logical clock which is updated according to the rules
in F. 5, slide 8.
The algorithm requires a total ordering of requests
requests are ordered according to their global logical
timestamps; if timestamps are equal, process identifiers
are compared to order them (see F. 5, slide 10).
The process that requires entry to a CS multicasts
the request message to all other processes competing
for the same resource; it is allowed to enter the CS
when all processes have replied to this message.
The request message consists of the requesting
process timestamp (logical clock) and its identifier.
Each process keeps its state with respect to the CS:
released, requested, or held.

F 6/7 - 8

Ricart-Agrawala Algorithm (contd)

The Algorithm

Rule for process initialization

/* performed by each process Pi at initialization */
[RI1]: statePi := RELEASED.

Rule for access request to CS

/* performed whenever process Pi requests an
access to the CS */
[RA1]: statePi := REQUESTED.
TPi := the value of the local logical clock
corresponding to this request.
[RA2]: Pi sends a request message to all processes; the
message is of the form (TPi, i), where i is an
identifier of Pi.
[RA3]: Pi waits until it has received replies from all other
n-1 processes.

Rule for executing the CS

/* performed by Pi after it received the n-1 replies */
[RE1]: statePi := HELD.
Pi enters the CS.

Petru Eles, IDA, LiTH

Petru Eles, IDA, LiTH

F 6/7 - 9

Distributed Systems

Ricart-Agrawala Algorithm (contd)

F 6/7 - 10

Ricart-Agrawala Algorithm (contd)

req
(11 ues
,1) t

P1

request
(15,2)

request
(11,1)

P2

P1

P0

OK

P2

P1

OK

P1 gets the
resource

P2
P2 gets the
resource

Problems

Petru Eles, IDA, LiTH

P0
OK

Rule for releasing a CS

/* performed by Pi after it finished work in a CS */
[RR1]: statePi := RELEASED.
Pi replies to all queued requests.

P0
st
ue )
req15,2
(

Rule for handling incoming requests

/* performed by Pi whenever it received a request
(TPj, j) from Pj */
[RH1]: if statePi = HELD or ((statePi = REQUESTED)
and ((TPi, i) < (TPj, j))) then
Queue the request from Pj without replying.
else
Reply immediately to Pj.
end if.

A request issued by a process Pj is blocked by

another process Pi only if Pi is holding the resource
or if it is requesting the resource with a higher
priority (this means a smaller timestamp) then Pj.

OK

Distributed Systems

The algorithm is expensive in terms of message

traffic; it requires 2(n-1) messages for entering a
CS: (n-1) requests and (n-1) replies.
The failure of any process involved makes progress
impossible if no special recovery measures are
taken.

Petru Eles, IDA, LiTH

Distributed Systems

F 6/7 - 11

Token-Based Mutual Exclusion

Ricart-Agrawala Second Algorithm

Token Ring Algorithm

Distributed Systems

F 6/7 - 12

Ricart-Agrawala Second Algorithm

A process is allowed to enter the critical section when
it got the token. In order to get the token it sends a
request to all other processes competing for the
same resource. The request message consists of the
requesting process timestamp (logical clock) and its
identifier.
Initially the token is assigned arbitrarily to one of the
processes.

When a process Pi leaves a critical section it passes

the token to one of the processes which are waiting
for it; this will be the first process Pj, where j is
searched in order [i+1, i+2, ..., n, 1, 2, ..., i-2, i-1] for
which there is a pending request.
If no process is waiting, Pi retains the token (and is
allowed to enter the CS if it needs); it will pass over
the token as result of an incoming request.

How does Pi find out if there is a pending request?

Each process Pi records the timestamp
corresponding to the last request it got from process
Pj, in requestPi[j]. In the token itself, token[j] records
the timestamp (logical clock) of Pjs last holding of the
token. If requestPi[j] > token[j] then Pj has a pending
request.

Petru Eles, IDA, LiTH

Petru Eles, IDA, LiTH

Distributed Systems

F 6/7 - 13

Distributed Systems

Ricart-Agrawala Second Algorithm (contd)

F 6/7 - 14

Ricart-Agrawala Second Algorithm (contd)

The Algorithm

Rule for process initialization

/* performed at initialization */
[RI1]: statePi := NO-TOKEN for all processes Pi, except
one single process Px for which
statePx := TOKEN-PRESENT.
[RI2]: token[k] initialized 0 for all elements k = 1 .. n.
requestPi[k] initialized 0 for all processes Pi and
all elements k = 1 .. n.
Rule for access request and execution of the CS
/* performed whenever process Pi requests an
access to the CS and when it finally gets it;
in particular Pi can already possess the token */
[RA1]: if statePi = NO-TOKEN then
Pi sends a request message to all processes;
the message is of the form (TPi, i), where
TPi = CPi is the value of the local logical clock,
and i is an identifier of Pi.
Pi waits until it receives the token.
end if.
statePi := TOKEN-HELD.
Pi enters the CS.

Rule for releasing a CS

/* performed by Pi after it finished work in a CS or
when it holds a token without using it and it got a
request */
[RR1]: statePi = TOKEN-PRESENT.
[RR2]: for k = [i+1, i+2, ..., n, 1, 2, ..., i-2, i-1] do
if requestPi[k] > token[k] then
statePi := NO-TOKEN.
token[i] := CPi, the value of the local
logical clock.
Pi sends the token to Pk.
break. /* leave the for loop */
end if.
end for.

Petru Eles, IDA, LiTH

Distributed Systems

Ricart-Agrawala Second Algorithm (contd)

Each process keeps its state with respect to the token:
NO-TOKEN, TOKEN-PRESENT, TOKEN-HOLD.

Ricart-Agrawala Second Algorithm (contd)

j
9

i+1 i i-1

Pj

es
t

CS left and no
other request there

Pi
t

es

u
eq

Pi-1

i+1 i i-1

Pi+1
9

j
12

12 11 6
token
i+1 i i-1

Pj

pass token

10 9 9
requestPi

Pi-1

Petru Eles, IDA, LiTH

es

request

TOKEN
PRESENT

Petru Eles, IDA, LiTH

qu

st
ue
q
re

qu

10 9 9
requestPi

ues

re

re

a
requ n extern
est c
a
ome l
s in

the process asks

again for the CS

NO
TOKEN

r
the
d o ists
n
ta x
lef st e
CS eque
r

12

12 8 6
token

req

Pi+1

st
ue
q
re

es
qu

ge

i+1 i i-1

re

TOKEN
HELD

n
oke
ts t

F 6/7 - 16

F 6/7 - 15

ue
s

Distributed Systems

re
q

Petru Eles, IDA, LiTH

Rule for handling incoming requests

/* performed by Pi whenever it received a request
(TPj, j) from Pj */
[RH1]: requestPi[j] := max(requestPi[j], TPj).
[RH2]: if statePi = TOKEN-PRESENT then
Pi releases the resource (see rule RR2).
end if.

Pi

Distributed Systems

F 6/7 - 17

Distributed Systems

F 6/7 - 18

Token Ring Algorithm

Ricart-Agrawala Second Algorithm (contd)

A very simple way to solve mutual exclusion

arrange the n processes P1, P2, .... Pn in a logical ring.
The complexity is reduced compared to the (first)
Ricart-Agrawala algorithm: it requires n messages for
entering a CS: (n-1) requests and one reply.

The failure of a process, except the one which holds

the token, doesnt prevent progress.

The logical ring topology is created by giving each

process the address of one other process which is its
neighbour in the clockwise direction.
The logical ring topology is unrelated to the physical
interconnections between the computers.

Pn
P1

P2

P3
P5

Petru Eles, IDA, LiTH

Petru Eles, IDA, LiTH

Distributed Systems

F 6/7 - 19

Distributed Systems

Token Ring Algorithm (contd)

The algorithm

P4

The token is initially given to one process.

The token is passed from one process to its
neighbour round the ring.
When a process requires to enter the CS, it waits
until it receives the token from its left neighbour and
then it retains it; after it got the token it enters the
CS; after it left the CS it passes the token to its
neighbour in clockwise direction.
When a process receives the token but does not
require to enter the critical section, it immediately
passes the token over along the ring.

F 6/7 - 20

Token Ring Algorithm (contd)

It can take from 1 to n-1 messages to obtain a token.

Messages are sent around the ring even when no
process requires the token additional load on the
network.

The algorithm works well in heavily loaded situations,

when there is a high probability that the process
which gets the token wants to enter the CS. It works
poorly in lightly loaded cases.
If a process fails, no progress can be made until a
reconfiguration is applied to extract the process from
the ring.
If the process holding the token fails, a unique
process has to be picked, which will regenerate the
token and pass it along the ring; an election algorithm
(see later) has to be run for this purpose.

Petru Eles, IDA, LiTH

Petru Eles, IDA, LiTH

Distributed Systems

F 6/7 - 21

Distributed Systems

F 6/7 - 22

Election

Election (contd)

Many distributed algorithms require one process to

act as a coordinator or, in general, perform some
special role.

We consider that it doesnt matter which process is

elected; what is important is that one and only one
process is chosen (we call this process the
coordinator) and all processes agree on this decision.

Examples with mutual exclusion

We assume that each process has a unique number

(identifier); in general, election algorithms attempt to
locate the process with the highest number, among
those which currently are up.

Central coordinator algorithm: at initialisation or

whenever the coordinator crashes, a new
coordinator has to be elected (see slide 6).

Token ring algorithm: when the process holding the

token fails, a new process has to be elected which
generates the new token (see slide 20).

Election is typically started after a failure occurs. The

detection of a failure (e.g. the crash of the current
coordinator) is normally based on time-out a
process that gets no response for a period of time
suspects a failure and initiates an election process.

An election process is typically performed in two

phases:
1. Select a leader with the highest priority.
2. Inform all processes about the winner.

Petru Eles, IDA, LiTH

Petru Eles, IDA, LiTH

Distributed Systems

F 6/7 - 23

Distributed Systems

The Bully Algorithm

A process has to know the identifier of all other
processes (it doesnt know, however, which one is
still up); the process with the highest identifier,
among those which are up, is selected.
Any process could fail during the election procedure.
When a process Pi detects a failure and a
coordinator has to be elected, it sends an election
message to all the processes with a higher identifier
and then waits for an answer message:
If no response arrives within a time limit, Pi
becomes the coordinator (all processes with
higher identifier are down) it broadcasts a
coordinator message to all processes to let
them know.
If an answer message arrives, Pi knows that
another process has to become the coordinator
it waits in order to receive the coordinator
message. If this message fails to arrive within a
time limit (which means that a potential
coordinator crashed after sending the answer
message) Pi resends the election message.
When receiving an election message from Pi, a
process Pj replys with an answer message to Pi and
then starts an election procedure itself, unless it has
already started one it sends an election message
to all processes with higher identifier.

The Bully Algorithm (contd)

The Algorithm

By default, the state of a process is ELECTION-OFF

Rule for election process initiator

/* performed by a process Pi, which triggers the
election procedure, or which starts an election after
receiving itself an election message */
[RE1]: statePi := ELECTION-ON.
Pi sends an election message to all processes
with a higher identifier.
Pi waits for answer message.
if no answer message arrives before time-out then
Pi is the coordinator and sends a coordinator
message to all processes.
else
Pi waits for a coordinator message to arrive.
if no coordinator message arrives before
time-out then
restart election procedure according to RE1
end if
end if.

Finlly all processes get an answer message, except

the one which becomes the coordinator.
Petru Eles, IDA, LiTH

F 6/7 - 24

Petru Eles, IDA, LiTH

Distributed Systems

F 6/7 - 25

Distributed Systems

The Bully Algorithm (contd)

F 6/7 - 26

The Bully Algorithm (contd)

P2
el

P5

el

P6

P7

ans

coo
rd

or

P1

coord

P6
co

P7

P5

d
coor
d
or

P4

P3
co

P5
P6

P6

P1

P2

P1

P5

el

P3

P4

P7

ans
ans
el

P4

P7

P1

P2

P3

el

P4
el

Rule for handling an incoming election message

/* performed by a process Pi at reception of an
election message coming from Pj */
[RH1]: Pi replies with an answer message to Pj.
[RH2]: if statePi := ELECTION-OFF then
start election procedure according to RE1
end if

P2

P3

If P6 crashes before sending the coordinator

message, P4 and P5 restart the election process.

The best case: the process with the second highest

identifier notices the coordinators failure. It can
immediately select itself and then send n-2
coordinator messages.
The worst case: the process with the lowest identifier
initiates the election; it sends n-1 election messages
to processes which themselves initiate each one an
election O(n2) messages.
Petru Eles, IDA, LiTH

Petru Eles, IDA, LiTH

Distributed Systems

F 6/7 - 27

Distributed Systems

The Ring-Based Algorithm

We assume that the processes are arranged in a

logical ring; each process knows the address of one
other process, which is its neighbour in the clockwise
direction.

F 6/7 - 28

The Ring-Based Algorithm (contd)

An optimization

Several elections can be active at the same time.

Messages generated by later elections should be
killed as soon as possible.

The algorithm elects a single coordinator, which is

the process with the highest identifier.
Election is started by a process which has noticed
that the current coordinator has failed. The process
places its identifier in an election message that is
passed to the following process.

Processes can be in one of two states: participant and

non-participant. Initially, a process is non-participant.

When a process receives an election message it

compares the identifier in the message with its own.
If the arrived identifier is greater, it forwards the
received election message to its neighbour; if the
arrived identifier is smaller it substitutes its own
identifier in the election message before forwarding it.

If the received identifier is that of the receiver itself

this will be the coordinator. The new coordinator
sends an elected message through the ring.

Petru Eles, IDA, LiTH

The process initiating an election marks itself

participant.
A participant process, in the case that the identifier
in the election message is smaller than the own,
does not forward any message (it has already
forwarded it, or a larger one, as part of another
simultaneously ongoing election).
When forwarding an election message, a process
marks itself participant.
When sending (forwarding) an elected message, a
process marks itself non-participant.

Petru Eles, IDA, LiTH

Distributed Systems

F 6/7 - 29

Distributed Systems

The Ring-Based Algorithm (contd)

F 6/7 - 30

The Ring-Based Algorithm (contd)

The Algorithm
By default, the state of a process is NON-PARTICIPANT
Rule for election process initiator
/* performed by a process Pi, which triggers the
election procedure */
[RE1]: statePi := PARTICIPANT.
[RE2]: Pi sends an election message with message.id := i
to its neighbour.
Rule for handling an incoming election message
/* performed by a process Pj, which receives an
election message */
[RH1]: if message.id > j then
Pj forwards the received election message.
statePj := PARTICIPANT.
elseif message.id < j then
if statePj = NON-PARTICIPANT then
Pj forwards an election message with
message.id := j.
statePj := PARTICIPANT
end if
else
Pj is the coordinator and sends an elected
message with message.id := j to its neighbour.
statePj := NON-PARTICIPANT.
end if.
Petru Eles, IDA, LiTH

Rule for handling an incoming elected message

/* performed by a process Pi, which receives an
elected message */
[RD1]: if message.id i then
Pi forwards the received elected message.
statePj := NON-PARTICIPANT.
end if.

Petru Eles, IDA, LiTH

Distributed Systems

F 6/7 - 31

Distributed Systems

Summary

The Ring-Based Algorithm (contd)

state = PARTICIPANT

15
9
10

If, in this stage,

process 8 initiates
another election, the
election message will
be killed by process 15

22
5

7
22

With one single election started:

On average: n/2 (election) messages needed to
reach maximal node; n (election) messages to
return to maximal node; n messages to rotate
elected message.
Number of messages: 2n + n/2.
Worst case: n-1 messages needed to reach
maximal node;
Number of messages: 3n - 1.
The ring algorithm is more efficient on average then
the bully algorithm.

Petru Eles, IDA, LiTH

F 6/7 - 32

In a distributed environment no shared variables

(semaphores) and local kernels can be used to
enforce mutual exclusion. Mutual exclusion has to
be based only on message passing.
There are two basic approaches to mutual
exclusion: non-token-based and token-based.
The central coordinator algorithm is based on the
availability of a coordinator process which handles
all the requests and provides exclusive access to
the resource. The coordinator is a performance
bottleneck and a critical point of failure. However,
the number of messages exchanged per use of a
CS is small.
The Ricart-Agrawala algorithm is based on fully
distributed agreement for mutual exclusion. A
request is multicast to all processes competing for a
resource and access is provided when all
processes have replied to the request. The
algorithm is expensive in terms of message traffic,
and failure of any process prevents progress.
Ricart-Agrawalas second algorithm is token-based.
Requests are sent to all processes competing for a
resource but a reply is expected only from the
process holding the token. The complexity in terms
of message traffic is reduced compared to the first
algorithm. Failure of a process (except the one
holding the token) does not prevent progress.

Petru Eles, IDA, LiTH

Distributed Systems

F 6/7 - 33

Summary (contd)

The token-ring algorithm very simply solves mutual

exclusion. It is requested that processes are
logically arranged in a ring. The token is
permanently passed from one process to the other
and the process currently holding the token has
exclusive right to the resource. The algorithm is
efficient in heavily loaded situations.
For many distributed applications it is needed that
one process acts as a coordinator. An election
algorithm has to choose one and only one process
from a group, to become the coordinator. All group
members have to agree on the decision.
The bully algorithm requires the processes to know
the identifier of all other processes; the process
with the highest identifier, among those which are
up, is selected. Processes are allowed to fail during
the election procedure.
The ring-based algorithm requires processes to be
arranged in a logical ring.The process with the
highest identifier is selected. On average, the ringbased algorithm is more efficient then the bully
algorithm.

Petru Eles, IDA, LiTH