Parallel Programming: for Multicore and Cluster Systems- P43 docx

7.3 Iterative Methods for Linear Systems 413
ˆ
A ·
ˆ
x =

D
R
F
ED
B

·

ˆ
x
R
ˆ
x
B

=

ˆ
b
1
ˆ
b
2

, (7.46)

where
ˆ
x
R
denotes the subvector of size n
R
of the first (red) unknowns and
ˆ
x
B
denotes
the subvector of size n
B
of the last (black) unknowns. The right-hand side b of the
original equation system is reordered accordingly and has subvector
ˆ
b
1
for the first
n
R
equations and subvector
ˆ
b
2
for the last n
B
equations. The matrix
ˆ
A consists

of four blocks D
R
∈ R
n
R
×n
R
, D
B
∈ R
n
B
×n
B
, E ∈ R
n
B
×n
R
, and F ∈ R
n
R
×n
B
.
The submatrices D
R
and D
B
are diagonal matrices and the submatrices E and F

are sparse banded matrices. The structure of the original matrix of the discretized
Poisson equation in Fig. 7.9 in Sect. 7.2.1 is thus transformed into a matrix
ˆ
A with
the structure shown in Fig. 7.17(c).
The diagonal form of the matrices D
R
and D
B
shows that a red unknown
ˆ
x
i
,
i ∈{1, ,n
R
}, does not depend on the other red unknowns and a black unknown
ˆ
x
j
, j ∈{n
R
+ 1, ,n
R
+ n
B
}, does not depend on the other black unknowns.
The matrices E and F specify the dependences between red and black unknowns.
The row i of matrix F specifies the dependences of the red unknowns
ˆ

x
i
(i <
n
R
) on the black unknowns
ˆ
x
j
, j = n
R
+ 1, ,n
R
+ n
B
. Analogously, a row of
matrix E specifies the dependences of the corresponding black unknowns on the red
unknowns.
The transformation of the original linear equation system Ax = b into the
equivalent system
ˆ
A
ˆ
x =
ˆ
b can be expressed by a permutation π : {1, ,n}→
{1, ,n}. The permutation maps a node i ∈{1, ,n} of the rowwise numbering
onto the number π(i) of the red–black numbering in the following way:
x
i

=
ˆ
x
π(i)
, b
i
=
ˆ
b
π(i)
, i = 1, ,n or x = P
ˆ
x and b = P
ˆ
b
with a permutation matrix P = (P
ij
)
i, j=1, ,n
, P
ij
=

1if j = π(i)
0 otherwise
.For
the matrices A and
ˆ
A the equation
ˆ

A = P
T
AP holds. Since for a permutation
matrix the inverse is equal to the transposed matrix, i.e., P
T
= P
−1
, this leads to
ˆ
A
ˆ
x = P
T
APP
T
x = P
T
b =
ˆ
b. The easiest way to exploit the red–black ordering
is to use an iterative solution method as discussed earlier in this section.
7.3.5.1 Gauss–Seidel Iteration for Red–Black Systems
The solution of the linear equation system (7.46) with the Gauss–Seidel iteration is
based on a splitting of the matrix
ˆ
A of the form
ˆ
A =
ˆ
D −

ˆ
L −
ˆ
U,
ˆ
D,
ˆ
L,
ˆ
U ∈ R
n×n
,
ˆ
D =

D
R
0
0 D
B

,
ˆ
L =

00
−E 0

,
ˆ

U =

0 −F
00

,
with a diagonal matrix
ˆ
D, a lower triangular matrix
ˆ
L, and an upper triangular
matrix
ˆ
U. The matrix 0 is a matrix in which all entries are 0. With this notation,
iteration step k of the Gauss–Seidel method is given by
414 7 Algorithms for Systems of Linear Equations

D
R
0
ED
B

·

x
(k+1)
R
x
(k+1)

B

=

b
1
b
2

−

0 F
00

·

x
(k)
R
x
(k)
B

(7.47)
for k = 1, 2, According to equation system (7.46), the iteration vector is split
into two subvectors x
(k+1)
R
and x
(k+1)

B
for the red and the black unknowns, respec-
tively. (To simplify the notation, we use x
R
instead of
ˆ
x
R
in the following discussion
of the red–black ordering.)
The linear equation system (7.47) can be written in vector notation for vectors
x
(k+1)
R
and x
(k+1)
B
in the form
D
R
· x
(k+1)
R
= b
1
− F · x
(k)
B
for k = 1, 2, , (7.48)
D

B
· x
(k+1)
B
= b
2
− E · x
(k+1)
R
for k = 1, 2, , (7.49)
in which the decoupling of the red subvector x
(k+1)
R
and the black subvector x
(k+1)
B
becomes obvious: In Eq. (7.48) the new red iteration vector x
(k+1)
R
depends only
on the previous black iteration vector x
(k)
B
and in Eq. (7.49) the new black iteration
vector x
(k+1)
B
depends only on the red iteration vector x
(k+1)
R

computed before in the
same iteration step. There is no additional dependence. Thus, the potential degree of
parallelism in Eq. (7.48) or (7.49) is similar to the potential parallelism in the Jacobi
iteration. In each iteration step k, the components of x
(k+1)
R
according to Eq. (7.48)
can be computed independently, since the vector x
(k)
B
is known, which leads to a
potential parallelism with p = n
R
processors. Afterwards, the vector x
(k+1)
R
is
known and the components of the vector x
(k+1)
B
can be computed independently
according to Eq. (7.49), leading to a potential parallelism of p = n
R
processors.
For a parallel implementation, we consider the Gauss–Seidel iteration of the red–
black ordering (7.48) and (7.49) written out in a component-based form:

x
(k+1)
R


i
=
1
ˆ
a
ii

ˆ
b
i
−

j∈N(i)
ˆ
a
ij
·(x
(k)
B
)
j

, i = 1, ,n
R
,

x
(k+1)
B


i
=
1
ˆ
a
i+n
R
,i+n
R

ˆ
b
i+n
R
−

j∈N(i)
ˆ
a
i+n
R
, j
·(x
(k+1)
R
)
j

, i = 1, ,n

B
.
The set N(i) denotes the set of adjacent mesh points for mesh point i. According to
the red–black ordering, the set N(i) contains only black mesh points for a red point
i and vice versa. An implementation on a shared memory machine can employ at
most p = n
R
or p = n
B
processors. There are no access conflicts for the par-
allel computation of x
(k)
R
or x
(k)
B
but a barrier synchronization is needed between
the two computation phases. The implementation on a distributed memory machine
requires a distribution of computation and data. As discussed before for the paral-
lel SOR method, it is useful to distribute the data according to the mesh structure
7.3 Iterative Methods for Linear Systems 415
such that the processor P
q
to which the mesh point i is assigned is responsible for
the computation or update of the corresponding component of the approximation
vector. In a row-oriented distribution of a squared mesh with
√
n ×
√
n = n mesh

points to p processors,
√
n/ p rows of the mesh are assigned to each processor P
q
,
q ∈{1, , p}. In the red–black coloring this means that each processor owns
1
2
n
p
red and
1
2
n
p
black mesh points. (For simplicity we assume that
√
n is a multiple of
p.) Thus, the mesh points
(q − 1) ·
n
R
p
+1, ,q ·
n
R
p
for q = 1, ,p and
(q − 1) ·
n

B
p
+1 + n
R
, ,q ·
n
B
p
+n
R
for q = 1, , p
are assigned to processor P
q
. Figure 7.18 shows an SPMD program implement-
ing the Gauss–Seidel iteration with red–black ordering. The coefficient matrix A
is stored according to the pointer-based scheme introduced earlier in Fig. 7.3. After
the computation of the red components xr, a function collect
elements(xr)
distributes the red vector to all other processors for the next computation.
Analogously, the black vector xb is distributed after its computation. The function
collect
elements() can be implemented by a multi-broadcast operation.
Fig. 7.18 Program fragment for the parallel implementation of the Gauss–Seidel method with the
red–black ordering. The arrays xr and xb denote the unknowns corresponding to the red or black
mesh points. The processor number of the executing processor is stored in me
416 7 Algorithms for Systems of Linear Equations
7.3.5.2 SOR Method for Red–Black Systems
An SOR method for the linear equation system (7.46) with relaxation parameter ω
can be derived from the Gauss–Seidel computation (7.48) and (7.49) by using the
combination of the new and the old approximation vectors as introduced in Formula

(7.41). One step of the SOR method has then the form
˜
x
(k+1)
R
= D
−1
R
·b
1
− D
−1
R
· F · x
(k)
B
,
˜
x
(k+1)
B
= D
−1
B
·b
2
− D
−1
B
· E · x

(k+1)
R
,
x
(k+1)
R
= x
(k)
R
+ω

˜
x
(k+1)
R
− x
(k)
R

, (7.50)
x
(k+1)
B
= x
(k)
B
+ω

˜
x

(k+1)
B
− x
(k)
B

, k = 1, 2, .
The corresponding splitting of matrix
ˆ
A is
ˆ
A =
1
ω
ˆ
D −
ˆ
L −
ˆ
U −
1−ω
ω
ˆ
D with the
matrices
ˆ
D,
ˆ
L,
ˆ

U introduced above. This can be written using block matrices:

D
R
0
ωED
B

·

x
(k+1)
R
x
(k+1)
B

(7.51)
= (1 − ω)

D
R
0
0 D
B

·

x
(k)

R
x
(k)
B

−ω

0 F
00

·

x
(k)
R
x
(k)
B

+ω

b
1
b
2

.
For a parallel implementation the component form of this system is used. On
the other hand, for the convergence results the matrix form and the iteration matrix
have to be considered. Since the iteration matrix of the SOR method for a given

linear equation system Ax = b with a certain order of the equations and the iter-
ation matrix of the SOR method for the red–black system
ˆ
A
ˆ
x =
ˆ
b are different,
convergence results cannot be transferred. The iteration matrix of the SOR method
with red–black ordering is
ˆ
S
ω
=

1
ω
ˆ
D −
ˆ
L

−1

1 − ω
ω
ˆ
D +
ˆ
U


.
For a convergence of the method it has to be shown that ρ(
ˆ
S
ω
) < 1 for the spectral
radius of
ˆ
S
ω
and ω ∈ R. In general, the convergence cannot be derived from the
convergence of the SOR method for the original system, since P
T
S
ω
P is not iden-
tical to
ˆ
S
ω
, although P
T
AP =
ˆ
A holds. However, for the specific case of the model
problem, i.e., the discretized Poisson equation, the convergence can be shown. Using
the equality P
T
AP =

ˆ
A,itfollowsthat
ˆ
A is symmetric and positive definite and,
thus, the method converges for the model problem, see [61].
Figure 7.19 shows a parallel SPMD implementation of the SOR method for
the red–black ordered discretized Poisson equation. The elements of the coeffi-
cient matrix are coded as constants. The unknowns are stored in a two-dimensional
structure corresponding to the two-dimensional mesh and not as vector so that
7.4 Conjugate Gradient Method 417
Fig. 7.19 Program fragment of a parallel SOR method for a red–black ordered discretized Poisson
equation
unknowns appear as x[i][j] in the program. The mesh points and the correspond-
ing computations are distributed among the processors; the mesh points belong-
ing to a specific processor are stored in myregion. The color red or black of
a mesh point (i, j) is an additional attribute which can be retrieved by the func-
tions is
red() and is black().Thevaluef[i][j] denotes the discretized
right-hand side of the Poisson equation as described earlier, see Eq. (7.15). The
functions exchange
red borders() and exchange black borders()
exchange the red or black data of the red or black mesh points between neighboring
processors.
7.4 Conjugate Gradient Method
The conjugate gradient method or CG method is a solution method for linear equa-
tion systems Ax = b with symmetric and positive definite matrix A ∈ R
n×n
, which
has been introduced in [86]. (A is symmetric if a
ij

= a
ji
and positive definite if
x
T
Ax > 0 for all x ∈ R
n
with x = 0.) The CG method builds up a solution x
∗
∈ R
n
in at most n steps in the absence of roundoff errors. Considering roundoff errors
more than n steps may be needed to get a good approximation of the exact solution
x
∗
. For sparse matrices a good approximation of the solution can be achieved in less
than n steps, also with roundoff errors [150]. In practice, the CG method is often
used as preconditioned CG method which combines a CG method with a precon-
ditioner [154]. Parallel implementations are discussed in [72, 133, 134, 154]; [155]
gives an overview. In this section, we present the basic CG method and parallel
implementations according to [23, 71, 166].
418 7 Algorithms for Systems of Linear Equations
7.4.1 Sequential CG Method
The CG method exploits an equivalence between the solution of a linear equation
system and the minimization of a function.
More precisely, the solution x
∗
of the linear equation system Ax = b, A ∈ R
n×n
,

b ∈ R
n
, is the minimum of the function Φ : M ⊂ R
n
→ R with
Φ(x) =
1
2
x
T
Ax −b
T
x , (7.52)
if the matrix A is symmetric and positive definite. A simple method to determine the
minimum of the function Φ is the method of the steepest gradient [71] which uses
the negative gradient. For a given point x
c
∈ R
n
the function decreases most rapidly
in the direction of the negative gradient. The method computes the following two
steps:
(a) Computation of the negative gradient d
c
∈ R
n
at point x
c
:
d

c
=−grad Φ(x
c
) =−

∂
∂x
1
Φ(x
c
), ,
∂
∂x
n
Φ(x
c
)

= b − Ax
c
.
(b) Determination of the minimum of Φ in the set
{x
c
+td
c
|t ≥ 0}∩M ,
which forms a line in R
n
(line search). This is done by inserting x

c
+ td
c
into
Formula (7.52). Using d
c
= b − Ax
c
and the symmetry of matrix A we get
Φ(x
c
+td
c
) = Φ(x
c
) −td
T
c
d
c
+
1
2
t
2
d
T
c
Ad
c

. (7.53)
The minimum of this function with respect to t ∈ R can be determined using the
derivative of this function with respect to t. The minimum is
t
c
=
d
T
c
d
c
d
T
c
Ad
c
. (7.54)
The steps (a) and (b) of the method of the steepest gradient are used to create a
sequence of vectors x
k
, k = 0, 1, 2, , with x
0
∈ R
n
and x
k+1
= x
k
+ t
k

d
k
.
The sequence
(
Φ(x
k
)
)
k=0,1,2,
is monotonically decreasing which can be seen by
inserting Formula (7.54) into Formula (7.53). The sequence converges toward the
minimum but the convergence might be slow [71].
The CG method uses a technique to determine the minimum which exploits
orthogonal search directions in the sense of conjugate or A-orthogonal vectors d
k
.
For a given matrix A, which is symmetric and non-singular, two vectors x, y ∈ R
n
are called conjugate or A-orthogonal, if x
T
Ay = 0. If A is positive definite, k
7.4 Conjugate Gradient Method 419
pairwise conjugate vectors d
0
, ,d
k−1
(with d
i
= 0, i = 0, ,k − 1 and k ≤ n)

are linearly independent [23]. Thus, the unknown solution vector x
∗
of Ax = b can
be represented as a linear combination of the conjugate vectors d
0
, ,d
n−1
, i.e.,
x
∗
=
n−1

k=0
t
k
d
k
. (7.55)
Since the vectors are orthogonal, d
T
k
Ax
∗
=

n−1
l=0
d
T

k
At
l
d
l
= t
k
d
T
k
Ad
k
.This
leads to
t
k
=
d
k
Ax
∗
d
T
k
Ad
k
=
d
T
k

b
d
T
k
Ad
k
for the coefficients t
k
. Thus, when the orthogonal vectors are known, the values t
k
,
k = 0, ,n − 1, can be computed from the right-hand side b.
The algorithm for the CG method uses a representation
x
∗
= x
0
+
n−1

i=0
α
i
d
i
(7.56)
of the unknown solution vector x
∗
as a sum of a starting vector x
0

and a term

n−1
i=0
α
i
d
i
to be computed. The second term is computed recursively by
Fig. 7.20 Algorithm of the CG method. (1) and (2) compute the values α
k
according to Eq. (7.58).
The vector w
k
is used for the intermediate result Ad
k
. (3) is the computation given in Formula
(7.57). (4) computes g
k+1
for the next iteration step according to Formula (7.58) in a recursive
way: g
k+1
= Ax
k+1
−b = A(x
k
+α
k
d
k

) −b = g
k
+ Aα
k
d
k
. This vector g
k+1
represents the error
between the approximation x
k
and the exact solution. (5) and (6) compute the next vector d
k+1
of
the set of conjugate gradients
420 7 Algorithms for Systems of Linear Equations
x
k+1
= x
k
+α
k
d
k
, k = 1, 2, , with (7.57)
α
k
=
−g
T

k
d
k
d
T
k
Ad
k
and g
k
= Ax
k
−b . (7.58)
Formulas (7.57) and (7.58) determine x
∗
according to Eq. (7.56) by computing α
i
and adding α
i
d
i
in each step, i = 1, 2, Thus, the solution is computed after at
most n steps. If not all directions d
k
are needed for x
∗
, less than n steps are required.
Algorithms implementing the CG method do not choose the conjugate vectors
d
0

, ,d
n−1
before computing the vectors x
0
, , x
n−1
but compute the next con-
jugate vector from the given gradient g
k
by adding a correction term. The basic
algorithm for the CG method is given in Fig. 7.20.
7.4.2 Parallel CG Method
The parallel implementation of the CG method is based on the algorithm given
in Fig. 7.20. Each iteration step of this algorithm implementing the CG method
consists of the following basic vector and matrix operations.
7.4.2.1 Basic Operations of the CG Algorithm
The basic operations of the CG algorithm are
(1) a matrix–vector multiplication Ad
k
,
(2) two scalar products g
T
k
g
k
and d
T
k
w
k

,
(3) a so-called axpy-operation x
k
+α
k
d
k
(The name axpy comes from axplus y describing the computation.),
(4) an axpy-operation g
k
+α
k
w
k
,
(5) a scalar product g
T
k+1
g
k+1
, and
(6) an axpy-operation −g
k+1
+β
k
d
k
.
The result of g
T

k
g
k
is needed in two consecutive steps and so the computation of one
scalar product can be avoided by storing g
T
k
g
k
in the scalar value γ
k
. Since there are
mainly one matrix–vector product and scalar products, a parallel implementation
can be based on parallel versions of these operations.
Like the CG method many algorithms from linear algebra are built up from
basic operations like matrix–vector operations or axpy-operations and efficient
implementations of these basic operations lead to efficient implementations of the
entire algorithms. The BLAS (Basic Linear Algebra Subroutines) library offers
efficient implementations for a large set of basic operations. This includes many
axpy-operations which denote that a vector x is multiplied by a scalar value a and
then added to another vector y. The prefixes s in saxpy or d daxpy denote axpy-
operations for simple precision and double precision, respectively. Introductory
descriptions of the BLAS library are given in [43] or [60]. A standard way to par-
allelize algorithms for linear algebra is to provide efficient parallel implementations
of the BLAS operations and to build up a parallel algorithm from these basic parallel
7.4 Conjugate Gradient Method 421
operations. This technique is ideally suited for the CG method since it consists of
such basic operations.
Here, we consider a parallel implementation based on the parallel implemen-
tations for matrix–vector multiplication or scalar product for distributed memory

machines as presented in Sect. 3. These parallel implementations are based on a data
distribution of the matrix and the vectors involved. For an efficient implementation
of the CG method it is important that the data distributions of different basic opera-
tions fit together in order to avoid expensive data re-distributions between the oper-
ations. Figure 7.21 shows a data dependence graph in which the nodes correspond
to the computation steps (1)–(6) of the CG algorithm in Fig. 7.20 and the arrows
depict a data dependency between two of these computation steps. The arrows are
annotated with data structures computed in one step (outgoing arrow) and needed
for another step with incoming arrow. The data dependence graph for one iteration
step k is a directed acyclic graph (DAG). There are also data dependences to the
previous iteration step k − 1 and the next iteration step k + 1, which are shown as
dashed arrows.
There are the following dependences in the CG method: The computation (2)
needs the result w
k
from computation (1) but also the vector d
k
and the scalar value
γ
k
from the previous iteration step k − 1; γ
k
is used to store the intermediate result
γ
k
= g
T
k
g
k

. Computation (3) needs α
k
from computation step (2) and the vectors
x
k
, d
k
from the previous iteration step k − 1. Computation (4) also needs α
k
from
( 3 )
x
k
x
k+1
( 2 )
( 4 )
( 5 )
α
k
β
k
γ
k+1
g
k+1
g
k
γ
k

w
k
w
k
( 6 )
d
k
d
k+1
( 1 )
α
k
g
k+1
k−1
k
k+1
Iteration step
Iteration step
Iteration step
Fig. 7.21 Data dependences between the computation steps (1)–(6) of the CG method in Fig. 7.20.
Nodes represent the computation steps of one iteration step k. Incoming arrows are annotated by
the data required and outgoing arrows are annotated by the data produced. Two nodes have an
arrow between them if one of the nodes produces data which are required by the node with the
incoming arrow. The data dependences to the previous iteration step k −1 or the next iteration step
k +1aregivenasdashed arrows. The data are named in the same way as in Fig. 7.20; additionally
the scalar γ
k
is used for the intermediate result γ
k

= g
T
k
g
k
computed in step (5) and required for
the computations of α
k
and β
k
in computation steps (2) and (5) of the next iteration step
422 7 Algorithms for Systems of Linear Equations
computation step (2) and vector w
k
from computation (1). Computation (5) needs
vector g
k+1
from computation (4) and scalar value γ
k
from the previous iteration
step k −1; computation (6) needs the scalar value from β
k
from computation (5) and
vector d
k
from iteration step k −1. This shows that there are many data dependences
between the different basic operations. But it can also be observed that computation
(3) is independent of the computations (4)–(6). Thus, the computation sequence
(1),(2),(3),(4),(5),(6) as well as the sequence (1),(2),(4),(5),(6),(3) can be used. The
independence of computation (3) from computations (4)–(6) is also another source

of parallelism, which is a coarse-grained parallelism of two linear algebra operations
performed in parallel, in contrast to the fine-grained parallelism exploited for a sin-
gle basic operation. In the following, we concentrate on the fine-grained parallelism
of basic linear algebra operations.
When the basic operations are implemented on a distributed memory machine,
the data distribution of matrices and vectors and the data dependences between oper-
ations might require data re-distribution for a correct implementation. Thus, the data
dependence graph in Fig. 7.21 can also be used to study the communication require-
ments for re-distribution in a message-passing program. Also the data dependences
between two iteration steps may lead to communication for data re-distribution.
To demonstrate the communication requirements, we consider an implementa-
tion of the CG method in which the matrix A has a row-blockwise distribution and
the vectors d
k
, ω
k
, g
k
, x
k
, and r
k
have a blockwise distribution. In one iteration
step of a parallel implementation, the following computation and communication
operations are performed.
7.4.2.2 Parallel CG Implementation with Blockwise Distribution
The parallel CG implementation has to consider data distributions in the following
way:
(0) Before starting the computation of iteration step k, the vector d
k

computed in
the previous step has to be re-distributed from a blockwise distribution of step
k − 1 to a replicated distribution required for step k. This can be done with a
multi-broadcast operation.
(1) The matrix–vector multiplication w
k
= Ad
k
is implemented with a row-
blockwise distribution of A as described in Sect. 3.6. Since d
k
is now replicated,
no further communication is needed. The result vector w
k
is distributed in a
blockwise way.
(2) The scalar product d
T
k
w
k
is computed in parallel with the same blockwise dis-
tribution of both vectors. (The scalar product γ
k
= g
T
k
g
k
is computed in the

previous iteration step.) Each processor computes a local scalar product for its
local vectors. The final scalar product is then computed by the root processor
of a single-accumulation operation with addition as reduction operation. This
processor owns the final result α
k
and sends it to all other processors by a
single-broadcast operation.
(3) The scalar value α
k
is known by each processor and thus the axpy-operation
x
k+1
= x
k
+α
k
d
k
can be done in parallel without further communication. Each