(Undergraduate texts in mathematics) michael w frazier an introduction to wavelets through linear algebra springer (1999)

An Introduction
to Wavelets Through
Linear Algebra

Michael W. Frazier

Springer


www.pdfgrip.com


www.pdfgrip.com


www.pdfgrip.com


www.pdfgrip.com


Preface

Mathematics majors at Michigan State University take a “Capstone”
course near the end of their undergraduate careers. The content
of this course varies with each offering. Its purpose is to bring
together different topics from the undergraduate curriculum and
introduce students to a developing area in mathematics. This text
was originally written for a Capstone course.
Basic wavelet theory is a natural topic for such a course. By name,
wavelets date back only to the 1980s. On the boundary between

mathematics and engineering, wavelet theory shows students that
mathematics research is still thriving, with important applications
in areas such as image compression and the numerical solution
of differential equations. The author believes that the essentials of
wavelet theory are sufficiently elementary to be taught successfully
to advanced undergraduates.
This text is intended for undergraduates, so only a basic
background in linear algebra and analysis is assumed. We do not
require familiarity with complex numbers and the roots of unity.
These are introduced in the first two sections of chapter 1. In the
remainder of chapter 1 we review linear algebra. Students should be
familiar with the basic definitions in sections 1.3 and 1.4. From our
viewpoint, linear transformations are the primary object of study;

v
www.pdfgrip.com


vi

Preface

a matrix arises as a realization of a linear transformation. Many
students may have been exposed to the material on change of basis
in section 1.4, but may benefit from seeing it again. In section 1.5,
we ask how to pick a basis to simplify the matrix representation of
a given linear transformation as much as possible. We then focus on
the simplest case, when the linear transformation is diagonalizable.
In section 1.6, we discuss inner products and orthonormal bases. We
end with a statement of the spectral theorem for matrices, whose

proof is outlined in the exercises. This is beyond the experience of
most undergraduates.
Chapter 1 is intended as reference material. Depending on
background, many readers and instructors will be able to skip or
quickly review much of this material. The treatment in chapter 1 is
relatively thorough, however, to make the text as self-contained as
possible, provide a logically ordered context for the subject matter,
and motivate later developments.
The author believes that students should be introduced to Fourier
analysis in the finite dimensional context, where everything can be
explained in terms of linear algebra. The key ideas can be exhibited
in this setting without the distraction of technicalities relating to
convergence. We start by introducing the Discrete Fourier Transform
(DFT) in section 2.1. The DFT of a vector consists of its components
with respect to a certain orthogonal basis of complex exponentials.
The key point, that all translation-invariant linear transformations
are diagonalized by this basis, is proved in section 2.2. We turn to
computational issues in section 2.3, where we see that the DFT can
be computed rapidly via the Fast Fourier Transform (FFT).
It is not so well known that the basics of wavelet theory can
also be introduced in the finite dimensional context. This is done
in chapter 3. The material here is not entirely standard; it is an
adaptation of wavelet theory to the finite dimensional setting. It has
the advantage that it requires only linear algebra as background. In
section 3.1, we search for orthonormal bases with both space and
frequency localization, which can be computed rapidly. We are led
to consider the even integer translates of two vectors, the mother and
father wavelets in this context. The filter bank arrangement for the
computation of wavelets arises naturally here. By iterating this filter
bank structure, we arrive in section 3.2 at a multilevel wavelet basis.

www.pdfgrip.com


Preface

vii

Examples and applications are discussed in section 3.3. Daubechies’s
wavelets are presented in this context, and elementary compression
examples are considered. A student familiar with MatLab, Maple, or
Mathematica should be able to carry out similar examples if desired.
In section 4.1 we change to the infinite dimensional but discrete
setting 2 (Z), the square summable sequences on the integers.
General properties of complete orthonormal sets in inner product
spaces are discussed in section 4.2. This is first point where analysis
enters our picture in a serious way. Square integrable functions
on the interval [−π, π) and their Fourier series are developed in
section 4.3. Here we have to cheat a little bit: we note that we
are using the Lebesgue integral but we don’t define it, and we
ask students to accept certain of its properties. We arrive again at
the key principle that the Fourier system diagonalizes translationinvariant linear operators. The relevant version of the Fourier
transform in this setting is the map taking a sequence in 2 (Z)
to a function in L2 ([−π, π)) whose Fourier coefficients make up
the original sequence. Its properties are presented in section 4.4.
Given this preparation, the construction of first stage wavelets on
the integers (section 4.5) and the iteration step yielding a multilevel
basis (section 4.6) are carried out in close analogy to the methods
in chapter 3. The computation of wavelets in the context of 2 (Z)
is discussed in section 4.7, which includes the construction of

Daubechies’s wavelets on Z. The generators u and v of a wavelet
system for 2 (Z) reappear in chapter 5 as the scaling sequence and
its companion.
The usual version of wavelet theory on the real line is presented
in chapter 5. The preliminaries regarding square integrable functions and the Fourier transform are discussed in sections 5.1 and 5.2.
The facts regarding Fourier inversion in L2 (R) are proved in detail,
although many instructors may prefer to assume these results. The
Fourier inversion formula is analogous to an orthonormal basis representation, using an integral rather than a sum. Again we see that
the Fourier system diagonalizes translation-invariant operators. Mallat’s theorem that a multiresolution analysis yields an orthonormal
wavelet basis is proved in section 5.3. The aformentioned relation
between the scaling sequence and wavelets on 2 (Z) allows us to
make direct use of the results of chapter 4. The conditions under

www.pdfgrip.com


viii

Preface

which wavelets on 2 (Z) can be used to generate a multiresolution
analysis, and hence wavelets on R, are considered in section 5.4.
In section 5.5, we construct Daubechies’s wavelets of compact support, and show how the wavelet transform is implemented using
filter banks.
We briefly consider the application of these results to numerical
differential equations in chapter 6. We begin in section 6.1 with
a discussion of the condition number of a matrix. In section
6.2, we present a simple example of the numerical solution of a
constant coefficient ordinary differential equation on [0, 1] using
finite differences. We see that although the resulting matrix is

sparse, which is convenient, it has a condition number that grows
quadratically with the size of the matrix. By comparison, in section
6.3, we see that for a wavelet-Galerkin discretization of a uniformly
elliptic, possibly variable-coefficient, differential equation, the
matrix of the associated linear system can be preconditioned to be
sparse and to have bounded condition number. The boundedness
of the condition number comes from a norm equivalence property
of wavelets that we state without proof. The sparseness of the
associated matrix comes from the localization of the wavelet system.
A large proportion of the time, the orthogonality of wavelet basis
members comes from their supports not overlapping (using wavelets
of compact support, say). This is a much more robust property,
for example with respect to multiplying by a variable coefficient
function, than the delicate cancellation underlying the orthogonality
of the Fourier system. Thus, although the wavelet system may not
exactly diagonalize any natural operator, it nearly diagonalizes (in
the sense of the matrix being sparse) a much larger class of operators
than the Fourier basis.
Basic wavelet theory includes aspects of linear algebra, real
and complex analysis, numerical analysis, and engineering. In
this respect it mimics modern mathematics, which is becoming
increasingly interdisciplinary.
This text is relatively elementary at the start, but the level
of difficulty increases steadily. It can be used in different ways
depending on the preparation level of the class. If a long time is
required for chapter 1, then the more difficult proofs in the later
chapters may have to be only briefly outlined. For a more advanced

www.pdfgrip.com

Preface

ix

group, most or all of chapter 1 could be skipped, which would leave
time for a relatively thorough treatment of the remainder. A shorter
course for a more sophisticated audience could start in chapter
4 because the main material in chapters 4 and 5 is technically,
although not conceptually, independent of the content of chapters
2 and 3. An individual with a solid background in Fourier analysis
could learn the basics of wavelet theory from sections 4.5, 4.7, 5.3,
5.4, 5.5, and 6.3 with only occasional references to the remainder of
the text.
This volume is intended as an introduction to wavelet theory
that is as elementary as possible. It is not designed to be a thorough
reference. We refer the reader interested in additional information
to the Bibliography at the end of the text.
Michigan State University
April 1999

www.pdfgrip.com

M. Frazier


This page intentionally left blank

www.pdfgrip.com

Acknowledgments

This text owes a great deal to a number of my colleagues and
students. The discrete presentation in Chapters 3 and 4 was
developed in joint work (Frazier and Kumar, 1993) with Arun Kumar,
in our early attempt to understand wavelets. This was further
clarified in consulting work done with Jay Epperson at Daniel H.
Wagner Associates in California. Many of the graphs in this text
are similar to examples done by Douglas McCulloch during this
consulting project. Additional insight was gained in subsequent work
with Rodolfo Torres.
My colleagues at Michigan State University provided assistance
with this text in various ways. Patti Lamm read a preliminary version
in its entirety and made more than a hundred useful suggestions,
including some that led to a complete overhaul of section 6.2. She
also provided computer assistance with the figures in the Prologue.
Sheldon Axler supplied technical assistance and made suggestions
that improved the style and presentation throughout the manuscript.
T.-Y. Li made a number of helpful suggestions, including providing
me with Exercise 1.6.20. Byron Drachman helped with the index.
I have had the opportunity to test preliminary versions of
this text in the classroom on several occasions. It was used at
Michigan State University in a course for undergraduates in spring

xi
www.pdfgrip.com


xii

Acknowledgments

1996 and in a beginning graduate course in summer 1996. The
administration of the Mathematics Department, especially Jon Hall,
Bill Sledd, and Wei-Eihn Kuan, went out of their way to provide these
opportunities. The students in these classes made many suggestions
and corrections, which have improved the text. Gihan Mandour,
Jian-Yu Lin, Rudolf Blazek, and Richard Andrusiak made large
numbers of corrections.
This text was also the basis for three short courses on wavelets.
One of these was presented at the University of Puerto Rico at
ă in the spring of 1997. I thank Nayda Santiago for helping
Mayaguez
arrange the visit, and Shawn Hunt, Domingo Rodr´iguez, and Ram´on
V´asquez for inviting me and for their warm hospitality. Another
short course was given at the University of Missouri at Columbia
in fall 1997. I thank Elias Saab and Nakhl´e Asmar for making
this possible. The third short course took place at the Instituto de
Matem´aticas de la UNAM in Cuernavaca, Mexico in summer 1998.
I thank Professors Salvador P´erez-Esteva and Carlos Villegas Blas
for their efforts in arranging this trip, and for their congeniality
throughout. The text in preliminary form has also been used in
courses given by Cristina Pereyra at the University of New Mexico
and by Suzanne Tourville at Carnegie-Mellon University. Cristina,
Suzanne, and their students provided valuable feedback and a
number of corrections, as did Kees Onneweer.
My doctoral students Kunchuan Wang and Mike Nixon made
many helpful suggestions and found a number of corrections in the
manuscript. My other doctoral student, Shangqian Zhang, taught me

the mathematics in Section 6.3. I also thank him and his son Simon
Zhang for Figure 35.
The fingerprint examples in Figures 1–3 in the Prologue were
provided by Chris Brislawn of the Los Alamos National Laboratories.
I thank him for permission to reproduce these images. Figures
36e and f were prepared using a program (Summus 4U2C 3.0)
provided to me by Bjăorn Jawerth and Summus Technologies, Inc,
for which I am grateful. Figures 36b, c, and d were created using the
commercially available software WinJPEG v.2.84. The manuscript
and some of the figures were prepared using LaTEX. The other figures
were done using MatLab. Steve Plemmons, the computer manager
in the mathematics department at Michigan State University, aided

www.pdfgrip.com


Acknowledgments

xiii

in many ways, particularly with regard to the images in Figure
36. I thank Ina Lindemann, my editor at Springer-Verlag, for her
assistance, encouragement, and especially her patience.
I take this opportunity to thank the mathematicians whose aid
was critical in helping me reach the point where it became possible
for me to write this text. The patience and encouragement of my
thesis advisor John Garnett was essential at the start. My early
collaboration with Bjăorn Jawerth played a decisive role in my career.
My postdoctoral advisor Guido Weiss encouraged and helped me in
many important ways over the years.

This text was revised and corrected during a sabbatical leave
provided by Michigan State University. This leave was spent at
the University of Missouri at Columbia. I thank the University of
Missouri for their hospitality and for providing me with valuable
resources and technical assistance.
At a time when academic tenure is under attack, it is worth
commenting that this text and many others like it would not have
been written without the tenure system.

www.pdfgrip.com


This page intentionally left blank

www.pdfgrip.com


Contents

Preface

v

Acknowledgments

xi

Prologue: Compression of the FBI Fingerprint Files
1 Background: Complex Numbers and Linear Algebra
1.1 Real Numbers and Complex Numbers . . . . . . .

1.2 Complex Series, Euler’s Formula, and the Roots of
Unity . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.3 Vector Spaces and Bases . . . . . . . . . . . . . . . .
1.4 Linear Transformations, Matrices, and Change of
Basis . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.5 Diagonalization of Linear Transformations and
Matrices . . . . . . . . . . . . . . . . . . . . . . . . .
1.6 Inner Products, Orthonormal Bases, and Unitary
Matrices . . . . . . . . . . . . . . . . . . . . . . . . .

1

.

7
7

.
.

16
29

.

40

.

56

.

79

2 The Discrete Fourier Transform
2.1 Basic Properties of the Discrete Fourier Transform .
2.2 Translation-Invariant Linear Transformations . . . .

101
101
128

xv
www.pdfgrip.com


xvi

Contents

2.3

The Fast Fourier Transform . . . . . . . . . . . . . . .

151

3 Wavelets on ZN
165
3.1 Construction of Wavelets on ZN : The First Stage . . 165

3.2 Construction of Wavelets on ZN : The Iteration Step . 196
3.3 Examples and Applications . . . . . . . . . . . . . . . 225
4 Wavelets on Z
2
4.1
(Z) . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2 Complete Orthonormal Sets in Hilbert Spaces . . .
4.3 L2 ([−π, π)) and Fourier Series . . . . . . . . . . . .
4.4 The Fourier Transform and Convolution on 2 (Z) .
4.5 First-Stage Wavelets on Z . . . . . . . . . . . . . . .
4.6 The Iteration Step for Wavelets on Z . . . . . . . .
4.7 Implementation and Examples . . . . . . . . . . .
5 Wavelets on R
5.1 L2 (R) and Approximate Identities . . . . . .
5.2 The Fourier Transform on R . . . . . . . . .
5.3 Multiresolution Analysis and Wavelets . . .
5.4 Construction of Multiresolution Analyses .
5.5 Wavelets with Compact Support and Their
Computation . . . . . . . . . . . . . . . . . .

.
.
.
.
.
.
.

265
265

271
279
298
309
321
330

.
.
.
.

349
349
362
380
398

. . . . .

429

.
.
.
.

.
.
.

.

.
.
.
.

.
.
.
.

6 Wavelets and Differential Equations
451
6.1 The Condition Number of a Matrix . . . . . . . . . . 451
6.2 Finite Difference Methods for Differential
Equations . . . . . . . . . . . . . . . . . . . . . . . . . 459
6.3 Wavelet-Galerkin Methods for Differential
Equations . . . . . . . . . . . . . . . . . . . . . . . . . 470
Bibliography

484

Index

491

www.pdfgrip.com

Prologue: Compression of
the FBI Fingerprint Files

When your local police arrest somebody on a minor charge, they
would like to check whether that person has an outstanding warrant,
possibly in another state, for a more serious crime. To check, they
can send his or her fingerprints to the FBI fingerprint archive
in Washington, D.C. Unfortunately, the FBI cannot compare the
received fingerprints with their records rapidly enough to make
an identification before the suspect must be released. A criminal
wanted on a serious charge will most likely have vacated the area
by the time the FBI has provided the necessary identification.
Why does it take so long? The FBI fingerprint files are stored
on fingerprint cards in filing cabinets in a warehouse that occupies
about an acre of floor space. The logistics of the search procedure
make it impossible to proceed sufficiently rapidly.
The solution to this seems obvious—the FBI fingerprint data
should be computerized and searched electronically. After all, this
is the computer age. Why hasn’t this been done long ago?
Data representing a fingerprint image can be stored on a
computer in such a way that the image can be reconstructed with
sufficient accuracy to allow positive identification. To do this, the
fingerprint image is scanned and digitized. Each square inch of the
fingerprint image is broken into a 500 by 500 grid of small boxes,

1
www.pdfgrip.com


2

Prologue: Compression of the FBI Fingerprint Files

FIGURE 1 Original fingerprint image (Courtesy of Chris Brislawn, Los
Alamos National Laboratory)

called pixels. Each pixel is given a gray-scale value corresponding to
its darkness, on a scale from 0 to 255. Because the integers from 0
to 255 can be represented in base 2 using eight places (that is, each
integer between 0 and 255 corresponds to an 8-digit sequence of
zeros and ones), it takes eight binary data bits to specify the darkness
of one pixel. (One digit in base 2 represents a single data bit, which
electronically corresponds to the difference between a switch being
on or off.) A portion of a fingerprint scanned in this way is exhibited
in Figure ??.

www.pdfgrip.com


Prologue: Compression of the FBI Fingerprint Files

3

Consider the amount of data required for a single fingerprint
card. Each rolled fingerprint is about 1.5 inches by 1.6 inches, with
5002 250,000 pixels per square inch, each requiring eight data bits
(one data byte). So each fingerprint requires about 600,000 data bytes.
A card includes all 10 rolled fingerprints, plus 2 unrolled thumb
impressions and 2 impressions of all 5 fingers on a hand. The result
is that each card requires about 10 megabytes of data (a megabyte is

one million bytes). This is still manageable for modern computers,
which frequently have several gigabytes of memory (a gigabyte is a
billion, or 109 , bytes). Electronic transmission of the data on a card
is feasible, although slow. So it is possible for the police to send the
necessary data electronically to the FBI while the suspect is still in
custody.
However, the FBI has about 200 million fingerprint cards in its
archive. (Many are for deceased individuals, and there are some
duplications—apparently the FBI is not good at throwing things
away.) Hence digitizing the entire archive would require roughly 2
× 1015 data bytes, or about 2,000 terabytes (a terabyte is 1012 bytes)
of memory. This represents more data than current computers can
store. Even if we restrict to cards corresponding to current criminal
suspects, we are dealing with about 29 million cards (with some
duplications due to aliases), or roughly 2 × 1014 data bytes. Thus it
would require about 60,000 3-gigabyte hard drives to store. This is
too much, even for the FBI. Even if this large of a data base could be
stored, it could not be rapidly searched. Yet it is not astronomically
too large. If the amount of data could be cut by a factor of about 20, it
could be stored on roughly 3,000 3-gigabyte hard drives. This is still a
lot, but not an unimaginable amount for a government agency. Thus
what is needed is a method to compress the data, that is, to represent
the information using less data while retaining enough accuracy to
allow positive identification.
Data compression is a major field in signal analysis, with a long
history. The current industry standard for image compression was
written by the Joint Photographic Experts Group, known as JPEG.
Many, perhaps most, of the image files that are downloaded on the
Internet are compressed with this standard, which is why they end
in the suffix “jpg.” The FBI solicitated proposals for compressing

their fingerprint files a few years ago. Different groups proposing

www.pdfgrip.com


4

Prologue: Compression of the FBI Fingerprint Files

different methods responded to the FBI solicitation. The contract
was awarded to a group at the Los Alamos National Laboratory,
headed by Jonathan Bradley and Christopher Brislawn; the project
leader was Tom Hopper from the FBI. They proposed compression
using the recently developed theory of wavelets. An account of this
project can be found in Brislawn (1995).
To see the reason the wavelet proposal was accepted instead of
proposals based on JPEG, consider the images in Figures ?? and ??.
Both contain compressions by a factor of about 13 of the fingerprint
image in Figure ??. Figure ?? shows the compression using JPEG,
and Figure ?? exhibits the wavelet compression. One feature of JPEG
is that it first divides a large image into smaller boxes, and then
compresses in these smaller boxes independently. This provides
some advantages due to local homogeneities in the image, but the
disadvantage is that the subimages may not align well at the edges of
the smaller boxes. This causes the regular pattern of horizontal and
vertical lines seen in Figure ??. These are called block artifacts, or
block lines for short. These are not just a visual annoyance, they also
are an impediment to machine recognition of fingerprints. Wavelet
compression methods do not require dividing the image into smaller
blocks because the desired localization properties are naturally built

into the wavelet system. Hence the wavelet compression in Figure
?? does not show block lines. This is one of the main reasons
that the FBI fingerprint compression contract was awarded to the
wavelet group. We introduce both Fourier compression and wavelet
compression in section 3.3 of this text.
The examples of fingerprint file compression in Figures ?? and
?? show that mathematics that has been developed recently (within
the last 10 or 12 years) has important practical applications.

www.pdfgrip.com


Prologue: Compression of the FBI Fingerprint Files

5

FIGURE 2 JPEG compression (Courtesy of Chris Brislawn, Los Alamos
National Laboratory)

www.pdfgrip.com


6

Prologue: Compression of the FBI Fingerprint Files

FIGURE 3 Wavelet compression (Courtesy of Chris Brislawn, Los
Alamos National Laboratory)

www.pdfgrip.com

1

C H A P T E R

...
...
...
...
...
...
...
...
...
...
...
...
...
...
.

Background:
Complex
Numbers and
Linear Algebra

1.1 Real Numbers and Complex
Numbers
We start by setting some notation. The natural numbers {1, 2, 3, 4, . . .}

will be denoted by N, and the integers {. . . , −3, −2, −1, 0, 1, 2, 3, . . .}
by Z. Complex numbers will be introduced later. We assume
familiarity with the real numbers R and their properties, which we
briefly summarize here. The basic algebraic properties of R follow
from the fact that R is a field.
Definition 1.1
A field (F, +, ·) is a set F with operations +
(called addition) and · (called multiplication) satisfying the following
properties:
A1. (Closure for addition) For all x, y ∈ F, x + y is defined and is an
element of F.
A2. (Commutativity for addition) x + y

y + x, for all x, y ∈ F.

A3. (Associativity for addition) x + (y + z)
x, y, z ∈ F.

(x + y) + z, for all

A4. (Existence of additive identity) There exists an element in F,
denoted 0, such that x + 0 x for all x ∈ F.

7
www.pdfgrip.com


8

1. Background: Complex Numbers and Linear Algebra

A5. (Existence of additive inverse) For each x ∈ F, there exists an
element in F, denoted −x, such that x + (−x) 0.
M1. (Closure for multiplication) For all x, y ∈ F, x · y is defined and is
an element of F.
M2. (Commutativity for multiplication) x · y

y · x, for all x, y ∈ F.

M3. (Associativity for multiplication) x · (y · z)
x, y, z ∈ F.

(x · y) · z, for all

M4. (Existence of multiplicative identity) There exists an element in
F, denoted 1, such that 1 0 and x · 1 x for all x ∈ F.
M5. (Existence of multiplicative inverse) For each x ∈ F such that
x
0, there exists an element in F, denoted x−1 (or 1/x), such
that x · (x−1 ) 1.
D. (Distributive law) x · (y + z)

(x · y) + (x · z), for all x, y, z ∈ F.

We emphasize that in principle the operations + and · in
Definition 1.1 could be any operations satisfying the required
properties. However, in our main examples R and C, these are
the usual addition and multiplication. In particular, with the usual
meanings of + and · , (R, +, ·) forms a field. We usually omit · and
write xy in place of x · y. All of the usual basic algebraic properties

(such as −(−x)
x) of R follow from the field properties. This
is shown in most introductory analysis texts. We assume all these
familiar properties in this text.
An ordered field is a field F with a relation < satisfying properties
O1–O4. The first two properties state that F is an ordered set.
O1. (Comparison principle) If x, y ∈ F, then one and only one of the
following holds:
x < y,

y < x,

y

x.

O2. (Transitivity) If x, y, z ∈ F, with x < y and y < z, then x < z.
The remaining two properties state that the operations + and ·
defined on F are consistent with the ordering <:
O3. (Consistency of + with <) If x, y, z ∈ F and y < z, then
x + y < x + z.
O4. (Consistency of · with <) If x, y ∈ F, with 0 < x and 0 < y, then
0 < xy.

www.pdfgrip.com