Digital Image Processing: Image Restoration Matrix Formulation - Duong Anh Duc

Digital Image Processing
Image Restoration
Matrix Formulation

21/11/15

Duong Anh Duc - Digital Image Processing

1


Matrix Formulation
of Image Restoration Problem
1-D Case:
 We will consider the 1-D version first, for simplicity:
g(m) = f(m)*h(m) +  (m) 
 We will assume that the arrays f and h have been zeropadded to be of size M, where M length(f) + length(h) 1.
 Henceforth, we will not explicitly mention the zero-padding.
 The degradation equation:
can be written in matrix-vector form as follows:
g = Hf + n, where

21/11/15

Duong Anh Duc - Digital Image Processing

2


Matrix Formulation
of Image Restoration Problem

H

H1

h0
h1
h2

hM 1

h 1
h0
h1

hM 2

h0
h1
h2

hM 1
21/11/15

h 2
h 1
h0

hM 3

0

h0
h1

hM 2

 h M 1
 h M 2
 h M 3



h0

0
0
h0

hM 3

Duong Anh Duc - Digital Image Processing

 0
 0
 0


 h0
3

Matrix Formulation
of Image Restoration Problem
 However, since the arrays f and h are zeropadded, we can equivalently set:
H

H2

h0
h1
h2

hM 1

hM 1
h0
h1

hM 2

hM 2
hM 1
h0

hM 3







h1
h2
h3

h0

 Notice that the (second) matrix H is circulant; i.e.,
each row of H is a circular shift of the previous
row.
21/11/15

Duong Anh Duc - Digital Image Processing

4


Matrix Formulation
of Image Restoration Problem
 Example:
A

= length of array f = 3
 B = length of array h = 2
M
A + B – 1 = 4, say M = 4.

f

21/11/15

f 0
f 1
f 2
0

h

h0
h1
0
0

Duong Anh Duc - Digital Image Processing

5


Matrix Formulation
of Image Restoration Problem
H1

21/11/15

h0
h1
h2
h3

h 1

h0
h1
h2

h0
h1
h2
h3

0
h0
h1
h2

h 2
h 1
h0
h1
0
0
h0
h1

h
h
h
h

0
0

0
h0

3
2
1
0
h0
h1
0
0

Duong Anh Duc - Digital Image Processing

0
h0
h1
0

0
0
h0
h1

0
0
0
h0

6

Matrix Formulation
of Image Restoration Problem
H2

h0
h1
h2
h3

hM 1
h0
h1
h2

h0
h1
h2
h3

h3
h0
h1
h2

h2
h3
h0
h1

hM 2
hM 1
h0
h1
h1
h2
h3
h0

hM 3
hM 2
hM 1
h0
h0
h1
0
0

0
h0
h1
0

0
0
h0
h1

h1

0
0
h0

Notice that H1f = H2f. Indeed
21/11/15

Duong Anh Duc - Digital Image Processing

7


Matrix Formulation
of Image Restoration Problem
H1f

h0
h1
0
0

0
h0
h1
0

0
0
h0
h1

0
0
0
h0

f 0
f 1
f 2
0

h0
0
0
h1 f 0
h1 h 0
0
0
f 1
H 2f
0
h1 h 0
0
f 2
0
0
h1 h 0
0
Henceforth, we will use H = H2 , so that we can apply
properties of circulant matrices to H.

21/11/15

Duong Anh Duc - Digital Image Processing

8


Matrix Formulation
of Image Restoration Problem
2-D Case:
 Suppose g, f, h, are M N arrays (after zeropadding). The degradation equation can be
written in matrix-vector format as follows:
g = Hf + n, where

21/11/15

Duong Anh Duc - Digital Image Processing

9


Matrix Formulation
of Image Restoration Problem

g

g 0,0


f 0,0



0,0


g 0, N 1
g 1,0

f 0, N 1
f 1,0

0, N 1
1,0


g 1, N 1


f 1, N 1


1, N 1

f

n







g M 1,0

f M 1,0

M 1,0


g M 1, N 1


f M 1, N 1


M 1, N 1

21/11/15

MN 1

MN 1

Duong Anh Duc - Digital Image Processing

MN 1
10

Matrix Formulation
of Image Restoration Problem
H

H0

H M­1

H M­ 2

 H1

H1

H0

HM

 H2

H2

H1

H0







HM

1

HM

2

HM

1

 H3

3



 H0

MN MN

 Note that H is a MN MN block-circulant
matrix with M M blocks.

21/11/15

Duong Anh Duc - Digital Image Processing

11


Matrix Formulation
of Image Restoration Problem
 Each block Hj is itself an N N circulant matrix.
Indeed, the matrix Hj is a circulant matrix formed
from the j-th row of array h(m,n):

Hj

21/11/15

h j ,0

h j, N 1

h j, N 2

 h j ,1

h j ,1
h j ,2

h j ,0
h j ,1

h j, N 1
h j ,0

 h j ,2
 h j ,3


h j, N 1


h j, N 2


h j, N 3



 h j ,0

Duong Anh Duc - Digital Image Processing

12


Matrix Formulation
of Image Restoration Problem
 Given the degradation equation:
g = Hf + n,
our objective is to recover f from observation g.
 We will assume that the array h(m,n) (usually
referred to as the blurring function) and statistics of
the noise (m,n) are known. The problem

becomes very complicated if array h(m,n) is
unknown and this case is usually referred to as
blind restoration or blind deconvolution.
21/11/15

Duong Anh Duc - Digital Image Processing

13


Matrix Formulation
of Image Restoration Problem
 Notice that, even when there is no noise; i.e.
(m,n) = 0, or the values of (m,n) were
exactly known, and matrix H is invertible,
computing
^f = H-1(g-n)
directly would not be practical.
 Example: Suppose M = N = 256. Therefore
MN = 65536 and H would be a 65536 by
65536 matrix to be inverted!
21/11/15

Duong Anh Duc - Digital Image Processing

14


Matrix Formulation
of Image Restoration Problem

 Naturally, direct inversion of H would not be feasible.
 But H has several useful properties; in particular:



H is block circulant.
H is usually sparse (has very few non-zero entries).

We will exploit these properties to obtain ˆf more efficiently.
 In particular, we will derive the theoretical solutions to the
restoration problem using matrix algebra. However, when it
comes to implementing the solution, we can resort to the
Fourier domain, thanks to the properties of circulant
matrices.

21/11/15

Duong Anh Duc - Digital Image Processing

15


Constrained least squares
filtering (restoration)
 Recall that the knowledge of blur function
h(m, n) is essential to obtain a meaningful
solution to the restoration problem.
 Often, knowledge of h(m, n) is not perfect
and subject to errors.
 One way to alleviate sensitivity of the result

to errors in h(m, n) is to base optimality of
restoration on a measure of smoothness,
such as the second derivative of the image.
21/11/15

Duong Anh Duc - Digital Image Processing

16


Constrained least squares
filtering (restoration)
 We will approximate the second derivative
(Laplacian) by a matrix Q. Indeed, we will
first formulate the constrained restoration
problem and obtain its solution in terms of a
general matrix Q.

21/11/15

Duong Anh Duc - Digital Image Processing

17


Constrained least squares
filtering (restoration)
 Later different choices of matrix Q will be
considered, each giving rise to a different
restoration filter.

 Suppose Q is any matrix (of appropriate
dimension). In constrained image restoration, we
choose ^f to minimize ||Qˆf||2, subject to the
constraint,
||g-Hˆf||2= ||n||2.
(Recall the degradation equation
g=Hˆf +n g-Hˆf = n.)
21/11/15

Duong Anh Duc - Digital Image Processing

18


Constrained least squares
filtering (restoration)
 Introduction of matrix Q allows considerable
flexibility in the design of appropriate
restoration filters (we will discuss specific
choices of Q later). So our problem is
formulated as follows:
min ||Qˆf||2
subject to ||g-Hˆf||2= ||n||2 or ||g-Hˆf||2- ||n||2=0

21/11/15

Duong Anh Duc - Digital Image Processing

19

A brief review of matrix
differentiation
 Suppose
x

x1
x2

and f(x1,x2) is a function of two variables. Then
f x1 , x2
x

21/11/15

f x1 , x2
x1
f x1 , x2
x2

Duong Anh Duc - Digital Image Processing

20


A brief review of matrix
differentiation
 If

f(x1,x2) = ||Ax-b||2 = (Ax­b)T (Ax­b)

for some matrix A and some vector b , then

f x1 , x2
x

T

2A Ax b

where superscript T denotes matrix
transpose.
21/11/15

Duong Anh Duc - Digital Image Processing

21


A brief review of matrix
differentiation
 Recall from calculus that such a constrained
minimization problem can be solved by means of
Lagrange multipliers. We need to minimize the
augmented objective function J(ˆf):
J(ˆf) = ||Qˆf||2+ (||g-Hˆf||2- ||n||2),
where is a Lagrange multiplier.
 We set the derivative of J(ˆf) with respect to ˆf to
zero.
J(ˆf) = 2QT Qˆf-2 HT(g-Hˆf) = 0

(QT Q + HTH)ˆf = HTg
21/11/15

Duong Anh Duc - Digital Image Processing

22


A brief review of matrix
differentiation
 Therefore,
ˆf = (QT Q + HTH)-1 HTg
= (QT Q/ + HTH)-1HTg
= ( QT Q + HTH)-1HTg
where = 1/ is chosen to satisfy the
constraint ||g-Hˆf||2= ||n||2.
 We will now use the above formulation to
derive a number of restoration filters.
21/11/15

Duong Anh Duc - Digital Image Processing

23


Pseudo-inverse Filtering
 The pseudo-inverse filter tries to avoid the
pitfalls of applying an inverse filter in the
presence of noise.
 Consider the constrained restoration

solution,
^f = ( QT Q + HTH)-1HTg
with Q = I. This gives,
^f = ( I + HTH)-1HTg
21/11/15

Duong Anh Duc - Digital Image Processing

24


Pseudo-inverse Filtering
 It can be implemented in the Fourier domain by
the following equation:
^F(u,v) = R(u,v)G(u,v), where
R u, v

*

H u, v
H u, v

2

1
H u, v

H u, v
H u, v

2

2

 The parameter is a constant to be chosen.
 Note that = 0 gives us back the inverse filter. For
> 0, the denominator of R(u, v) is strictly positive
and the pseudo-inverse filter is well defined.
21/11/15

Duong Anh Duc - Digital Image Processing

25