lecture notes on algorithm analysis and computation complexity 4th ed - ian parberry

Lecture Notes on Algorithm Analysis
and Computational Complexity
(Fourth Edition)
Ian Parberry
1
Department of Computer Sciences
University of North Texas
December 2001
1
Author’s address: Department of Computer Sciences, University of North Texas, P.O. Box 311366, Denton, TX
76203–1366, U.S.A. Electronic mail:
License Agreement
This work is copyright Ian Parberry. All rights reserved. The author offers this work, retail value US$20,
free of charge under the following conditions:
No part of this work may be made available on a public forum (including, but not limited to a web
page, ftp site, bulletin board, or internet news group) without the written permission of the author.
●
No part of this work may be rented, leased, or offered for sale commercially in any form or by any
means, either print, electronic, or otherwise, without written permission of the author.
●
If you wish to provide access to this work in either print or electronic form, you may do so by
providing a link to, and/or listing the URL for the online version of this license agreement:
You may not link directly to the PDF file.
●
All printed versions of any or all parts of this work must include this license agreement. Receipt of
a printed copy of this work implies acceptance of the terms of this license agreement. If you have
received a printed copy of this work and do not accept the terms of this license agreement, please
destroy your copy by having it recycled in the most appropriate manner available to you.
●
You may download a single copy of this work. You may make as many copies as you wish for
your own personal use. You may not give a copy to any other person unless that person has read,

understood, and agreed to the terms of this license agreement.
●
You undertake to donate a reasonable amount of your time or money to the charity of your choice
as soon as your personal circumstances allow you to do so. The author requests that you make a
cash donation to The National Multiple Sclerosis Society in the following amount for each work
that you receive:
$5 if you are a student,❍
$10 if you are a faculty member of a college, university, or school,❍
$20 if you are employed full-time in the computer industry.❍
Faculty, if you wish to use this work in your classroom, you are requested to:
encourage your students to make individual donations, or❍
make a lump-sum donation on behalf of your class.❍
If you have a credit card, you may place your donation online at
Otherwise, donations may be sent to:
National Multiple Sclerosis Society - Lone Star Chapter
8111 North Stadium Drive
Houston, Texas 77054
If you restrict your donation to the National MS Society's targeted research campaign, 100% of
your money will be directed to fund the latest research to find a cure for MS.
For the story of Ian Parberry's experience with Multiple Sclerosis, see
/>●
Preface
These lecture notes are almost exact copies of the overhead projector transparencies that I use in my CSCI
4450 course (Algorithm Analysis and Complexity Theory) at the University of North Texas. The material
comes from
• textbooks on algorithm design and analysis,
• textbooks on other subjects,
• research monographs,
• papers in research journals and conferences, and
• my own knowledge and experience.

Be forewarned, this is not a textbook, and is not designed to be read like a textbook. To get the best use
outofityoumust attend my lectures.
Students entering this course are expected to be able to program in some procedural programming language
such as C or C++, and to be able to deal with discrete mathematics. Some familiarity with basic data
structures and algorithm analysis techniques is also assumed. For those students who are a little rusty, I
have included some basic material on discrete mathematics and data structures, mainly at the start of the
course, partially scattered throughout.
Why did I take the time to prepare these lecture notes? I have been teaching this course (or courses very
much like it) at the undergraduate and graduate level since 1985. Every time I teach it I take the time to
improve my notes and add new material. In Spring Semester 1992 I decided that it was time to start doing
this electronically rather than, as I had done up until then, using handwritten and xerox copied notes that
I transcribed onto the chalkboard during class.
This allows me to teach using slides, which have many advantages:
• They are readable, unlike my handwriting.
• I can spend more class time talking than writing.
• I can demonstrate more complicated examples.
• I can use more sophisticated graphics (there are 219 figures).
Students normally hate slides because they can never write down everything that is on them. I decided to
avoid this problem by preparing these lecture notes directly from the same source files as the slides. That
way you don’t have to write as much as you would have if I had used the chalkboard, and so you can spend
more time thinking and asking questions. You can also look over the material ahead of time.
To get the most out of this course, I recommend that you:
• Spend half an hour to an hour looking over the notes before each class.
iii
iv PREFACE
• Attend class. If you think you understand the material without attending class, you are probably kidding
yourself. Yes, I do expect you to understand the details, not just the principles.
• Spend an hour or two after each class reading the notes, the textbook, and any supplementary texts
you can find.
• Attempt the ungraded exercises.

• Consult me or my teaching assistant if there is anything you don’t understand.
The textbook is usually chosen by consensus of the faculty who are in the running to teach this course. Thus,
it does not necessarily meet with my complete approval. Even if I were able to choose the text myself, there
does not exist a single text that meets the needs of all students. I don’t believe in following a text section
by section since some texts do better jobs in certain areas than others. The text should therefore be viewed
as being supplementary to the lecture notes, rather than vice-versa.
Algorithms Course Notes
Introduction
Ian Parberry
∗
Fall 2001
Summary
• What is “algorithm analysis”?
• What is “complexity theory”?
• What use are they?
TheGameofChess
According to legend, when Grand Vizier Sissa Ben
Dahir invented chess, King Shirham of India was so
taken with the game that he asked him to name his
reward.
The vizier asked for
• One grain of wheat on the first square of the
chessboard
• Two grains of wheat on the second square
• Four grains on the third square
• Eight grains on the fourth square
• etc.
How large was his reward?
How many grains of wheat?
63


i=0
2
i
=2
64
− 1=1.8 × 10
19
.
A bushel of wheat contains 5 × 10
6
grains.
∗
Copyright
c
 Ian Parberry, 1992–2001.
Therefore he asked for 3.7 × 10
12
bushels.
The price of wheat futures is around $2.50 per
bushel.
Therefore, he asked for $9.25 × 10
12
= $92 trillion
at current prices.
The Time Travelling Investor
A time traveller invests $1000 at 8% interest com-
pounded annually. How much money does he/she
have if he/she travels 100 years into the future? 200
years? 1000 years?

Years Amount
100 $2.9 × 10
6
200 $4.8 × 10
9
300 $1.1 × 10
13
400 $2.3 × 10
16
500 $5.1 × 10
19
1000 $2.6 × 10
36
The Chinese Room
Searle (1980): Cognition cannot be the result of a
formal program.
Searle’s argument: a computer can compute some-
thing without really understanding it.
Scenario: Chinese room = person + look-up table
The Chinese room passes the Turing test, yet it has
no “understanding” of Chinese.
Searle’s conclusion: A symbol-processing program
cannot truly understand.
Analysis of the Chinese Room
1
How much space would a look-up table for Chinese
take?
A typical person can remember seven objects simul-
taneously (Miller, 1956). Any look-up table must
contain queries of the form:

“Which is the largest, a <noun>
1
,a
<noun>
2
,a<noun>
3
,a<noun>
4
,a<noun>
5
,
a <noun>
6
,ora<noun>
7
?”,
There are at least 100 commonly used nouns. There-
fore there are at least 100 · 99 · 98 · 97 · 96 · 95 · 94 =
8 × 10
13
queries.
100 Common Nouns
aardvark duck lizard sardine
ant eagle llama scorpion
antelope eel lobster sea lion
bear ferret marmoset seahorse
beaver finch monkey seal
bee fly mosquito shark
beetle fox moth sheep

buffalo frog mouse shrimp
butterfly gerbil newt skunk
cat gibbon octopus slug
caterpillar giraffe orang-utang snail
centipede gnat ostrich snake
chicken goat otter spider
chimpanzee goose owl squirrel
chipmunk gorilla panda starfish
cicada guinea pig panther swan
cockroach hamster penguin tiger
cow horse pig toad
coyote hummingbird possum tortoise
cricket hyena puma turtle
crocodile jaguar rabbit wasp
deer jellyfish racoon weasel
dog kangaroo rat whale
dolphin koala rhinocerous wolf
donkey lion salamander zebra
Size of the Look-up Table
The Science Citation Index:
• 215 characters per line
• 275 lines per page
• 1000 pages per inch
Our look-up table would require 1.45 × 10
8
inches
=2, 300 miles of paper = a cube 200 feet on a side.
The Look-up Table and the Great Pyramid
Computerizing the Look-up Table
Use a large array of small disks. Each drive:

• Capacity 100 × 10
9
characters
• Volume 100 cubic inches
• Cost $100
Therefore, 8 × 10
13
queries at 100 characters per
query:
• 8,000TB = 80, 000 disk drives
• cost $8M at $1 per GB
• volume over 55K cubic feet (a cube 38 feet on
aside)
Extrapolating the Figures
Our queries are very simple. Suppose we use 1400
nouns (the number of concrete nouns in the Unix
spell-checking dictionary), and 9 nouns per query
(matches the highest human ability). The look-up
table would require
• 1400
9
=2× 10
28
queries, 2 × 10
30
bytes
• astackofpaper10
10
light years high [N.B. the
nearest spiral galaxy (Andromeda) is 2.1 × 10

6
light years away, and the Universe is at most
1.5 × 10
10
light years across.]
• 2× 10
19
hard drives (a cube 198 miles on a side)
• if each bit could be stored on a single hydrogen
atom, 10
31
use almost seventeen tons of hydro-
gen
Summary
We have seen three examples where cost increases
exponentially:
2
• Chess: cost for an n × n chessboard grows pro-
portionally to 2
n
2
.
• Investor: return for n years of time travel is
proportional to 1000 × 1.08
n
(for n centuries,
1000 × 2200
n
).
• Look-up table: cost for an n-term query is pro-

portional to 1400
n
.
Algorithm Analysis and Complexity Theory
Computational complexity theory = the study of the
cost of solving interesting problems. Measure the
amount of resources needed.
• time
• space
Two aspects:
• Upper bounds: give a fast algorithm
• Lower bounds: no algorithm is faster
Algorithm analysis = analysis of resource usage of
given algorithms
Exponential resource use is bad. It is best to
• Make resource usage a polynomial
• Make that polynomial as small as possible
Polynomial Good
Exponential Bad
Linear
Quadratic
Cubic
Exponential
Factorial
Y x 10
3
X
0.00
0.50
1.00

1.50
2.00
2.50
3.00
3.50
4.00
4.50
5.00
50.00 100.00 150.00
Motivation
Why study this subject?
• Efficient algorithms lead to efficient programs.
• Efficient programs sell better.
• Efficient programs make better use of hardware.
• Programmers who write efficient programs are
more marketable than those who don’t!
Efficient Programs
Factors influencing program efficiency
• Problem being solved
• Programming language
• Compiler
• Computer hardware
• Programmer ability
• Programmer effectiveness
• Algorithm
Objectives
What will you get from this course?
• Methods for analyzing algorithmic efficiency
• A toolbox of standard algorithmic techniques
• A toolbox of standard algorithms

3
J
Just when YOU
thought it was
safe to take CS
courses
Discrete Mathematics
What’s this
class like?
CSCI 4450
Welcome To
CSCI 4450
Assigned Reading
CLR, Section 1.1
POA, Preface and Chapter 1.
/>4
Algorithms Course Notes
Mathematical Induction
Ian Parberry
∗
Fall 2001
Summary
Mathematical induction:
• versatile proof technique
• various forms
• application to many types of problem
Induction with People
. . .
Scenario 1:
Fact 1: The first person is Greek.

Fact 2: Pick any person in the line. If they are
Greek, then the next person is Greek too.
Question: Are they all Greek?
Scenario 2:
Fact: The first person is Ukranian.
Question: Are they all Ukranian?
Scenario 3:
∗
Copyright
c
 Ian Parberry, 1992–2001.
Fact: Pick any person in the line. If they are Nige-
rian, then the next person is Nigerian too.
Question: Are they all Nigerian?
Scenario 4:
Fact 1: The first person is Indonesian.
Fact 2: Pick any person in the line. If all the people
up to that point are Indonesian, then the next person
is Indonesian too.
Question: Are they all Indonesian?
Mathematical Induction
. . .
1
2
3
4
5
6
7
8

9
. . .
To prove that a property holds for all IN, prove:
Fact 1: The property holds for 1.
Fact 2: For all n ≥ 1, if the property holds for n,
then it holds for n +1.
Alternatives
There are many alternative ways of doing this:
1
1. The property holds for 1.
2. For all n ≥ 2, if the property holds for n − 1,
then it holds for n.
There may have to be more base cases:
1. The property holds for 1, 2, 3.
2. For all n ≥ 3, if the property holds for n,then
it holds for n +1.
Strong induction:
1. The property holds for 1.
2. For all n ≥ 1, if the property holds for all 1 ≤
m ≤ n, then it holds for n +1.
Example of Induction
An identity due to Gauss (1796, aged 9):
Claim:
For all n ∈ IN ,
1+2+···+ n = n(n +1)/2.
First:
Prove the property holds for n =1.
1 = 1(1 + 1)/2
Second:
Prove that if the property holds for n,then

the property holds for n +1.
Let S(n)denote1+2+···+ n.
Assume: S(n)=n(n +1)/2 (the induction hypoth-
esis).
Required to Prove:
S(n +1)=(n +1)(n +2)/2.
S(n +1)
= S(n)+(n +1)
= n(n +1)/2+(n + 1) (by ind. hyp.)
= n
2
/2+n/2+n +1
=(n
2
+3n +2)/2
=(n +1)(n +2)/2
Second Example
Claim:
For all n ∈ IN , i f 1 + x>0, then
(1 + x)
n
≥ 1+nx
First:
Prove the property holds for n =1.
Both sides of the equation are equal to 1 + x.
Second:
Prove that if the property holds for n,then
the property holds for n +1.
Assume: (1 + x)
n

≥ 1+nx.
Required to Prove:
(1 + x)
n+1
≥ 1+(n +1)x.
(1 + x)
n+1
=(1+x)(1 + x)
n
≥ (1 + x)(1 + nx) (by ind. hyp.)
=1+(n +1)x + nx
2
≥ 1+(n +1)x (since nx
2
≥ 0)
More Complicated Example
Solve
S(n)=
n

i=1
(5i +3)
This can be solved analytically, but it illustrates the
technique.
Guess: S(n)=an
2
+ bn + c for some a, b, c ∈ IR .
Base:
S(1) = 8, hence guess is true provided a + b +
c =8.

Inductive Step:
Assume: S(n)=an
2
+ bn + c.
Required to prove: S(n+1) = a(n+1)
2
+b(n+1)+c.
Now,
S(n +1) = S(n)+5(n +1)+3
2
=(an
2
+ bn + c)+5(n +1)+3
= an
2
+(b +5)n + c +8
We want
an
2
+(b +5)n + c +8
= a(n +1)
2
+ b(n +1)+c
= an
2
+(2a + b)n +(a + b + c)
Each pair of coefficients has to be the same.
an + (b+5)n + (c+8) = an + (2a+b)n + (a+b+c)
2
2

The first coefficient tells us nothing.
The second coefficient tells us b+5 = 2a+b, therefore
a =2.5.
We know a + b + c = 8 (from the Base), so therefore
(looking at the third coefficient), c =0.
Since we now know a =2.5, c =0,anda +b+ c =8,
we can deduce that b =5.5.
Therefore S(n)=2.5n
2
+5.5n = n(5n + 11)/2.
Complete Binary Trees
Claim
: A complete binary tree with k levels has ex-
actly 2
k
− 1nodes.
Proof
: Proof by induction on number of levels. The
claim is true for k = 1, since a complete binary tree
with one level consists of a single node.
Suppose a complete binary tree with k levels has 2
k
−
1 nodes. We are required to prove that a complete
binary tree with k + 1 levels has 2
k+1
− 1nodes.
A complete binary tree with k +1 levels consists of a
root plus two trees with k levels. Therefore, by the
induction hypothesis the total number of nodes is

1+2(2
k
− 1) = 2
k+1
− 1
as required.
Tree with k
levels
Tree with k
levels
Another Example
Prove that for all n ≥ 1,
n

i=1
1/2
i
< 1.
The claim is clearly true for n = 1. Now assume
that the claim is true for n.
n+1

i=1
1/2
i
=
1
2
+
1

4
+
1
8
+ ···+
1
2
n+1
=
1
2
+
1
2

1
2
+
1
4
+
1
8
+ ···+
1
2
n

=
1

2
+
1
2
n

i=1
1/2
i
<
1
2
+
1
2
· 1 (by ind. hyp.)
=1
AGeometricExample
Prove that any set of regions defined by n lines in the
plane can be coloured with only 2 colours so that no
two regions that share an edge have the same colour.
3
L
Proof by induction on n.Trueforn = 1 (colour one
side light, the other side dark). Now suppose that
thehypothesisistrueforn lines.
Suppose we are given n + 1 lines in the plane. Re-
move one of the lines L, and colour the remaining
regions with 2 colours (which can be done, by the in-
duction hypothesis). Replace L. Reverse all of the

colours on one side of the line.
Consider two regions that have a line in common. If
that line is not L, then by the induction hypothe-
sis, the two regions have different colours (either the
same as before or reversed). If that line is L,then
the two regions formed a single region before L was
replaced. Since we reversed colours on one side of L
only, they now have different colours.
A Puzzle Example
A triomino is an L-shaped figure formed by the jux-
taposition of three unit squares.
An arrangement of triominoes is a tiling of a shape
if it covers the shape exactly without overlap. Prove
by induction on n ≥ 1thatany2
n
× 2
n
grid that
is missing one square can be tiled with triominoes,
regardless of where the missing square is.
Proof by induction on n.Trueforn =1:
Now suppose that the hypothesis is true for n. Sup-
pose we have a 2
n+1
× 2
n+1
grid with one square
missing.
NW NE
SW

SE
Divide the grid into four 2
n
× 2
n
subgrids. Suppose
the missing square is in the NE subgrid. Remove
the squares closest to the center of the grid from the
other three subgrids. By the induction hypothesis,
all four subgrids can be tiled. The three removed
squares in the NW, SW, SE subgrids can be tiled
with a single triomino.
A Combinatorial Example
AGraycode
is a sequence of 2
n
n-bit binary num-
bers where each adjacent pair of numbers differs in
exactly one bit.
4
n =1 n =2 n =3 n =4
0 00 000 0000
1 01 001 0001
11 011 0011
10 010 0010
110 0110
111 0111
101 0101
100 0100
1100

1101
1111
1110
1010
1011
1001
1000
Binary Reflected Gray Code on n Bits
Binary Reflected Gray Code on n Bits
0
1
0000
0001
0011
0010
0110
0111
0101
0100
1100
1101
1111
1110
1010
1011
1001
1000
Claim: the binary reflected Gray code on n bits is a
Gray code.
Proof by induction on n. The claim is trivially true

for n = 1. Suppose the claim is true for n bits.
Suppose we construct the n + 1 bit binary reflected
Gray code as above from 2 copies of the n bit code.
Now take a pair of adjacent numbers. If they are in
the same half, then by the induction hypothesis they
differ in exactly one bit (since they both start with
the same bit). If one is in the top half and one is
in the bottom half, then they only differ in the first
bit.
Assigned Reading
Re-read the section in your discrete math textbook
or class notes that deals with induction. Alterna-
tively, look in the library for one of the many books
on discrete mathematics.
POA, Chapter 2.
5
Algorithms Course Notes
Algorithm Correctness
Ian Parberry
∗
Fall 2001
Summary
• Confidence in algorithms from testing and cor-
rectness proof.
• Correctness of recursive algorithms proved di-
rectly by induction.
• Correctness of iterative algorithms proved using
loop invariants and induction.
• Examples: Fibonacci numbers, maximum, mul-
tiplication

Correctness
How do we know that an algorithm works?
Modes of rhetoric (from ancient Greeks)
• Ethos
• Pathos
• Logos
Logical methods of checking correctness
• Testing
• Correctness proof
Testing vs. Correctness Proofs
Testing: try the algorithm on sample inputs
Correctness Proof: prove mathematically
Testing may not find obscure bugs.
Using tests alone can be dangerous.
Correctness proofs can also contain bugs: use a com-
bination of testing and correctness proofs.
∗
Copyright
c
 Ian Parberry, 1992–2001.
Correctness of Recursive Algorithms
To prove correctness of a recursive algorithm:
• Prove it by induction on the “size” of the prob-
lem being solved (e.g. size of array chunk, num-
ber of bits in an integer, etc.)
• Base of recursion is base of induction.
• Need to prove that recursive calls are given sub-
problems, that is, no infinite recursion (often
trivial).
• Inductive step: assume that the recursive calls

work correctly, and use this assumption to prove
that the current call works correctly.
Recursive Fibonacci Numbers
Fibonacci numbers: F
0
=0,F
1
=1,andforall
n ≥ 2, F
n
= F
n−2
+ F
n−1
.
function fib(n)
comment return F
n
1. if n ≤ 1 then return(n)
2. else return(fib(n − 1)+fib(n − 2))
Claim
: For all n ≥ 0, fib(n) returns F
n
.
Base: for n = 0, fib(n) returns 0 as claimed. For
n = 1, fib(n) returns 1 as claimed.
Induction: Suppose that n ≥ 2andforall0≤ m<
n, fib(m) returns F
m
.

RTP fib(n) returns F
n
.
What does fib(n) return?
fib(n − 1) + fib(n − 2)
1
= F
n−1
+ F
n−2
(by ind. hyp.)
= F
n
.
Recursive Maximum
function maximum(n)
comment Return max of A[1 n].
1. if n ≤ 1 then return(A[1]) else
2. return(max(maximum(n − 1),A[n]))
Claim
: For all n ≥ 1, maximum(n) returns
max{A[1],A[2], ,A[n]}. Proof by induction on
n ≥ 1.
Base: for n =1,maximum(n) returns A[1] as
claimed.
Induction: Suppose that n ≥ 1 and maximum(n)
returns max{A[1],A[2], ,A[n]}.
RTP maximum(n + 1) returns
max{A[1],A[2], ,A[n +1]}.
What does maximum(n + 1) return?

max(maximum(n),A[n +1])
= max(max{A[1],A[2], ,A[n]},A[n +1])
(by ind. hyp.)
=max{A[1],A[2], ,A[n +1]}.
Recursive Multiplication
Notation
:Forx ∈ IR , ⌊x⌋ is the largest integer not
exceeding x.
function multiply(y,z)
comment return the product yz
1. if z =0then return(0) else
2. if z is odd
3. then return(multiply(2y, ⌊z/2⌋)+y)
4. else return(multiply(2y, ⌊z/2⌋))
Claim
: For all y,z ≥ 0, multiply(y, z) returns yz.
Proof by induction on z ≥ 0.
Base: for z = 0, multiply(y, z) returns 0 as claimed.
Induction: Suppose that for z ≥ 0, and for all 0 ≤
q ≤ z, multiply(y, q) returns yq.
RTP multiply(y,z + 1) returns y(z +1).
What does multiply(y, z + 1) return?
There are two cases, depending on whether z +1 is
odd or even.
If z + 1 is odd, then multiply(y,z + 1) returns
multiply(2y, ⌊(z +1)/2⌋)+y
=2y⌊(z +1)/2⌋ + y (by ind. hyp.)
=2y(z/2) + y (since z is even)
= y(z +1).
If z + 1 is even, then multiply(y, z + 1) returns

multiply(2y, ⌊(z +1)/2⌋)
=2y⌊(z +1)/2⌋ (by ind. hyp.)
=2y(z +1)/2(sincez is odd)
= y(z +1).
Correctness of Nonrecursive Algorithms
To prove correctness of an iterative algorithm:
• Analyse the algorithm one loop at a time, start-
ing at the inner loop in case of nested loops.
• Foreachloopdevisealoop invariant that re-
mains true each time through the loop, and cap-
tures the “progress” made by the loop.
• Prove that the loop invariants hold.
• Use the loop invariants to prove that the algo-
rithm terminates.
• Use the loop invariants to prove that the algo-
rithm computes the correct result.
Notation
We will concentrate on one-loop algorithms.
The value in identifier x immediately after the ith
iteration of the loop is denoted x
i
(i =0means
immediately before entering for the first time).
2
For example, x
6
denotes the value of identifier x after
the 6th time around the loop.
Iterative Fibonacci Numbers
function fib(n)

1. comment Return F
n
2. if n =0then return(0) else
3. a := 0; b := 1; i := 2
4. while i ≤ n do
5. c := a + b; a := b; b := c; i := i +1
6. return(b)
Claim
: fib(n) returns F
n
.
Facts About the Algorithm
i
0
=2
i
j+1
= i
j
+1
a
0
=0
a
j+1
= b
j
b
0
=1

b
j+1
= c
j+1
c
j+1
= a
j
+ b
j
The Loop Invariant
For all natural numbers j ≥ 0, i
j
= j +2,a
j
= F
j
,
and b
j
= F
j+1
.
The proof is by induction on j. The base, j =0,is
trivial, since i
0
=2,a
0
=0=F
0

,andb
0
=1=F
1
.
Now suppose that j ≥ 0, i
j
= j +2, a
j
= F
j
and
b
j
= F
j+1
.
RTP i
j+1
= j +3,a
j+1
= F
j+1
and b
j+1
= F
j+2
.
i
j+1

= i
j
+1
=(j + 2) + 1 (by ind. hyp.)
= j +3
a
j+1
= b
j
= F
j+1
(by ind. hyp.)
b
j+1
= c
j+1
= a
j
+ b
j
= F
j
+ F
j+1
(by ind. hyp.)
= F
j+2
.
Correctness Proof
Claim

: The algorithm terminates with b containing
F
n
.
The claim is certainly true if n =0. Ifn>0, then
we enter the while-loop.
Termination: Since i
j+1
= i
j
+ 1, eventually i will
equal n + 1 and the loop will terminate. Suppose
this happens after t iterations. Since i
t
= n +1and
i
t
= t + 2, we can conclude that t = n − 1.
Results: By the loop invariant, b
t
= F
t+1
= F
n
.
Iterative Maximum
function maximum(A, n)
comment Return max of A[1 n]
1. m := A[1]; i := 2
2. while i ≤ n do

3. if A[i] >mthen m := A[i]
4. i := i +1
4. return(m)
Claim
:maximum(A, n) returns
max{A[1],A[2], ,A[n]}.
3
Facts About the Algorithm
m
0
= A[1]
m
j+1
=max{m
j
,A[i
j
]}
i
0
=2
i
j+1
= i
j
+1
The Loop Invariant
Claim
: For all natural numbers j ≥ 0,
m

j
=max{A[1],A[2], ,A[j +1]}
i
j
= j +2
The proof is by induction on j. The base, j =0,is
trivial, since m
0
= A[1] and i
0
=2.
Now suppose that j ≥ 0, i
j
= j +2and
m
j
=max{A[1],A[2], ,A[j +1]},
RTP i
j+1
= j +3 and
m
j+1
=max{A[1],A[2], ,A[j +2]}
i
j+1
= i
j
+1
=(j + 2) + 1 (by ind. hyp.)
= j +3

m
j+1
=max{m
j
,A[i
j
]}
=max{m
j
,A[j +2]} (by ind. hyp.)
=max{max{A[1], ,A[j +1]},A[j +2]}
(by ind. hyp.)
=max{A[1],A[2], ,A[j +2]}.
Correctness Proof
Claim
: The algorithm terminates with m containing
the maximum value in A[1 n].
Termination: Since i
j+1
= i
j
+ 1, eventually i will
equal n+1 and the loop will terminate. Suppose this
happens after t iterations. Since i
t
= t+2, t = n−1.
Results: By the loop invariant,
m
t
=max{A[1],A[2], ,A[t +1]}

=max{A[1],A[2], ,A[n]}.
Iterative Multiplication
function multiply(y,z)
comment Return yz,wherey, z ∈ IN
1. x := 0;
2. while z>0 do
3. if z is odd then x := x + y;
4. y := 2y; z := ⌊z/2⌋;
5. return(x)
Claim
:ify,z ∈ IN, then multiply(y,z) returns the
value yz. That is, when line 5 is executed, x = yz.
A Preliminary Result
Claim
: For all n ∈ IN ,
2⌊n/2⌋ +(n mod 2) = n.
Case 1
. n is even. Then ⌊n/2⌋ = n/2, n mod 2 = 0,
and the result follows.
Case 2
. n is odd. Then ⌊n/2⌋ =(n − 1)/2, n mod
2 = 1, and the result follows.
Facts About the Algorithm
Write the changes using arithmetic instead of logic.
From line 4 of the algorithm,
y
j+1
=2y
j
z

j+1
= ⌊z
j
/2⌋
4
From lines 1,3 of the algorithm,
x
0
=0
x
j+1
= x
j
+ y
j
(z
j
mod 2)
The Loop Invariant
Loop invariant: a statement about the variables that
remains true every time through the loop.
Claim
: For all natural numbers j ≥ 0,
y
j
z
j
+ x
j
= y

0
z
0
.
The proof is by induction on j. The base, j =0,is
trivial, since then
y
j
z
j
+ x
j
= y
0
z
0
+ x
0
= y
0
z
0
Suppose that j ≥ 0and
y
j
z
j
+ x
j
= y

0
z
0
.
We are required to prove that
y
j+1
z
j+1
+ x
j+1
= y
0
z
0
.
By the Facts About the Algorithm
y
j+1
z
j+1
+ x
j+1
=2y
j
⌊z
j
/2⌋ + x
j
+ y

j
(z
j
mod 2)
= y
j
(2⌊z
j
/2⌋ +(z
j
mod 2)) + x
j
= y
j
z
j
+ x
j
(by prelim. result)
= y
0
z
0
(by ind. hyp.)
Correctness Proof
Claim
: The algorithm terminates with x containing
the product of y and z.
Termination: on every iteration of the loop, the
value of z is halved (rounding down if it is odd).

Therefore there will be some time t at which z
t
=0.
At this point the while-loop terminates.
Results: Suppose the loop terminates after t itera-
tions, for some t ≥ 0. By the loop invariant,
y
t
z
t
+ x
t
= y
0
z
0
.
Since z
t
=0,weseethatx
t
= y
0
z
0
. Therefore, the
algorithm terminates with x containing the product
of the initial values of y and z.
Assigned Reading
Problems on Algorithms: Chapter 5.

5
Algorithms Course Notes
Algorithm Analysis 1
Ian Parberry
∗
Fall 2001
Summary
• O, Ω, Θ
• Sum and product rule for O
• Analysis of nonrecursive algorithms
Implementing Algorithms
Algorithm
Program
Executable
Programmer
Compiler
Computer
Constant Multiples
Analyze the resource usage of an algorithm to within
a constant multiple.
Why
? Because other constant multiples creep in
when translating from an algorithm to executable
code:
• Programmer ability
• Programmer effectiveness
• Programming language
• Compiler
• Computer hardware
∗

Copyright
c
 Ian Parberry, 1992–2001.
Recall: measure resource usage as a function of input
size.
Big Oh
We need a notation for “within a constant multiple”.
Actually, we have several of them.
Informal definition: f(n)isO(g(n)) if f grows at
most as fast as g.
Formal definition: f(n)=O(g(n)) if there exists
c, n
0
∈ IR
+
such that for all n ≥ n
0
, f(n) ≤ c · g(n).
cg(n)
f(n)
n
0
time
n
Example
Most big-Os can be proved by induction.
Example: log n = O(n).
Claim: for all n ≥ 1, log n ≤ n. The proof is by
induction on n. The claim is trivially true for n =1,
since 0 < 1. Now suppose n ≥ 1andlogn ≤ n.

Then,
log(n +1)
≤ log 2n
=logn +1
≤ n + 1 (by ind. hyp.)
1
Second Example
2
n+1
= O(3
n
/n).
Claim
: for all n ≥ 7, 2
n+1
≤ 3
n
/n. The proof is
by induction on n. The claim is true for n =7,
since 2
n+1
=2
8
= 256, and 3
n
/n =3
7
/7 > 312.
Now suppose n ≥ 7and2
n+1

≤ 3
n
/n.RTP2
n+2
≤
3
n+1
/(n +1).
2
n+2
=2· 2
n+1
≤ 2 · 3
n
/n (by ind. hyp.)
≤
3n
n +1
·
3
n
n
(see below)
=3
n+1
/(n +1).
(Note that we need
3n/(n +1)≥ 2
⇔ 3n ≥ 2n +2
⇔ n ≥ 2.)

Big Omega
Informal definition: f(n)isΩ(g(n)) if f grows at
least as fast as g.
Formal definition: f (n)=Ω(g(n)) if there exists
c>0 such that there are infinitely many n ∈ IN
such that f(n) ≥ c · g(n).
cg(n)
f(n)
Alternative Big Omega
Some texts define Ω differently: f(n)=Ω
′
(g(n)) if
there exists c, n
0
∈ IR
+
such that for all n ≥ n
0
,
f(n) ≥ c · g(n).
cg(n)
f(n)
Is There a Difference?
If f(n)=Ω
′
(g(n)), then f (n)=Ω(g(n)), but the
converse is not true. Here is an example where
f(n)=Ω(n
2
), but f(n) =Ω

′
(n
2
).
f(n)
0.6 n
2
Does this come up often in practice? No.
Big Theta
Informal definition: f(n)isΘ(g(n)) if f is essentially
the same as g, to within a constant multiple.
Formal definition: f(n)=Θ(g(n)) if f (n)=
O(g(n)) and f(n)=Ω(g(n)).
2
cg(n)
f(n)
n
0
dg(n)
Multiple Guess
True or false?
• 3n
5
− 16n +2=O(n
5
)?
• 3n
5
− 16n +2=O(n)?
• 3n

5
− 16n +2=O(n
17
)?
• 3n
5
− 16n +2=Ω(n
5
)?
• 3n
5
− 16n +2=Ω(n)?
• 3n
5
− 16n +2=Ω(n
17
)?
• 3n
5
− 16n +2=Θ(n
5
)?
• 3n
5
− 16n +2=Θ(n)?
• 3n
5
− 16n +2=Θ(n
17
)?

Adding Big Ohs
Claim
.Iff
1
(n)=O(g
1
(n)) and f
2
(n)=O(g
2
(n)),
then f
1
(n)+f
2
(n)=O(g
1
(n)+g
2
(n)).
Proof
: Suppose for all n ≥ n
1
, f
1
(n) ≤ c
1
·g
1
(n)and

for all n ≥ n
2
, f
2
(n) ≤ c
2
· g
2
(n).
Let n
0
=max{n
1
,n
2
} and c
0
=max{c
1
,c
2
}.Then
for all n ≥ n
0
,
f
1
(n)+f
2
(n) ≤ c

1
· g
1
(n)+c
2
· g
2
(n)
≤ c
0
(g
1
(n)+g
2
(n)).
Claim
.Iff
1
(n)=O(g
1
(n)) and f
2
(n)=O(g
2
(n)),
then f
1
(n)+f
2
(n)=O(max{g

1
(n),g
2
(n)}).
Proof
: Suppose for all n ≥ n
1
, f
1
(n) ≤ c
1
·g
1
(n)and
for all n ≥ n
2
, f
2
(n) ≤ c
2
· g
2
(n).
Let n
0
=max{n
1
,n
2
} and c

0
= c
1
+ c
2
. Then for all
n ≥ n
0
,
f
1
(n)+f
2
(n) ≤ c
1
· g
1
(n)+c
2
· g
2
(n)
≤ (c
1
+ c
2
)(max{g
1
(n),g
2

(n)})
= c
0
(max{g
1
(n),g
2
(n)}).
Multiplying Big Ohs
Claim
.Iff
1
(n)=O(g
1
(n)) and f
2
(n)=O(g
2
(n)),
then f
1
(n) · f
2
(n)=O(g
1
(n) · g
2
(n)).
Proof: Suppose for all n ≥ n
1

, f
1
(n) ≤ c
1
·g
1
(n)and
for all n ≥ n
2
, f
2
(n) ≤ c
2
· g
2
(n).
Let n
0
=max{n
1
,n
2
} and c
0
= c
1
· c
2
. Then for all
n ≥ n

0
,
f
1
(n) · f
2
(n) ≤ c
1
· g
1
(n) · c
2
· g
2
(n)
= c
0
· g
1
(n) · g
2
(n).
Types of Analysis
Worst case
: time taken if the worst possible thing
happens. T (n) is the maximum time taken over all
inputs of size n.
Average Case: The expected running time, given
some probability distribution on the inputs (usually
uniform). T (n) is the average time taken over all

inputs of size n.
Probabilistic: The expected running time for a ran-
dom input. (Express the running time and the prob-
ability of getting it.)
Amortized
: The running time for a series of execu-
tions, divided by the number of executions.
Example
Consider an algorithm that for all inputs of n bits
takes time 2n, and for one input of size n takes time
n
n
.
Worst case
: Θ(n
n
)
Average Case
:
Θ(
n
n
+(2
n
− 1)n
2
n
)=Θ(
n
n

2
n
)
Probabilistic
: O(n) with probability 1 − 1/2
n
3
Amortized: A sequence of m executions on different
inputs takes amortized time
O(
n
n
+(m − 1)n
m
)=O(
n
n
m
).
Time Complexity
We’ll do mostly worst-case analysis. How much time
does it take to execute an algorithm in the worst
case?
assignment O(1)
procedure entry O(1)
procedure exit O(1)
if statement time for test plus
O(max of two branches)
loop sum over all iterations of
the time for each iteration

Put these together using sum rule and product rule.
Exception — recursive algorithms.
Multiplication
function multiply(y,z)
comment Return yz,wherey, z ∈ IN
1. x :=0;
2. while z>0 do
3. if z is odd then x := x + y;
4. y :=2y ; z := ⌊z/2⌋;
5. return(x)
Suppose y and z have n bits.
• Procedure entry and exit cost O(1) time
• Lines 3,4 cost O(1) time each
• The while-loop on lines 2–4 costs O(n) time (it
is executed at most n times).
• Line 1 costs O(1) time
Therefore, multiplication takes O(n)time(bythe
sum and product rules).
Bubblesort
1. procedure bubblesort(A[1 n])
2. for i :=1to n − 1 do
3. for j :=1to n − i do
4. if A[j ] >A[j +1]then
5. Swap A[j ]withA[j +1]
• Procedure entry and exit costs O (1) time
• Line 5 costs O(1) time
• The if-statement on lines 4–5 costs O (1) time
• The for-loop on lines 3–5 costs O (n − i)time
• The for-loop on lines 2–5 costs O (


n−1
i=1
(n − i))
time.
O (
n−1

i=1
(n − i)) = O(n(n − 1) −
n−1

i=1
i)=O(n
2
)
Therefore, bubblesort takes time O(n
2
)intheworst
case. Can show similarly that it takes time Ω(n
2
),
hence Θ(n
2
).
Analysis Trick
Rather than go through the step-by-step method of
analyzing algorithms,
• Identify the fundamental operation
used in the
algorithm, and observe that the running time

is a constant multiple of the number of funda-
mental operations used. (Advantage: no need
to grunge through line-by-line analysis.)
• Analyze the number of operations exactly. (Ad-
vantage: work with numbers instead of sym-
bols.)
This often helps you stay focussed, and work faster.
Example
In the bubblesort example, the fundamental opera-
tion is the comparison done in line 4. The running
time will be big-O of the number of comparisons.
• Line 4 uses 1 comparison
• The for-loop on lines 3–5 uses n−i comparisons
4
• The for-loop on lines 2–5 uses

n−1
i=1
(n−i)com-
parisons, and
n−1

i=1
(n − i)=n(n − 1) −
n−1

i=1
i
= n(n − 1) − n(n − 1)/2
= n(n − 1)/2.

Lies, Damn Lies, and Big-Os
(Apologies to Mark Twain.)
The multiplication algorithm takes time O(n).
What does this mean? Watch out for
• Hidden assumptions: word model vs. bit model
(addition takes time O(1))
• Artistic lying: the multiplication algorithm
takes time O(n
2
) is also true. (Robert Hein-
lein: There are 2 artistic ways of lying. One is
to tell the truth, but not all of it.)
• The constant multiple: it may make the algo-
rithm impractical.
Algorithms and Problems
Big-Os mean different things when applied to algo-
rithms and problems.
• “Bubblesort runs in time O(n
2
).” But is it
tight? Maybe I was too lazy to figure it out,
or maybe it’s unknown.
• “Bubblesort runs in time Θ(n
2
).” This is tight.
• “The sorting problem takes time O(n log n).”
There exists an algorithm that sorts in time
O(n log n), but I don’t know if there is a faster
one.
• “The sorting problem takes time Θ(n log n).”

There exists an algorithm that sorts in time
O(n log n), and no algorithm can do any better.
Assigned Reading
CLR, Chapter 1.2, 2.
POA, Chapter 3.
5
Algorithms Course Notes
Algorithm Analysis 2
Ian Parberry
∗
Fall 2001
Summary
Analysis of iterative (nonrecursive) algorithms.
The heap: an implementation of the priority queue
• InsertionintimeO(log n)
• Deletion of minimum in time O(log n)
Heapsort
• Build a heap in time O(n log n).
• Dismantle a heap in time O(n log n).
• Worst case analysis — O(n log n).
• How to build a heap in time O(n).
The Heap
A priority queue
is a set with the operations
• Insert an element
• Delete and return the smallest element
A popular implementation: the heap
.Aheapisa
binary tree with the data stored in the nodes. It has
two important properties:

1. Balance
. It is as much like a complete binary tree
as possible. “Missing leaves”, if any, are on the last
level at the far right.
∗
Copyright
c
 Ian Parberry, 1992–2001.
2. Structure
. Thevalueineachparentis≤ the
values in its children.
20
3
5
24
124030
15
21
11
9
10
Note this implies that the value in each parent is ≤
the values in its descendants (nb. includes self).
To Delete the Minimum
1. Remove the root and return the value in it.
20
3
3
5
24

124030
15
21
11
9
10
?
Butwhatwehaveisnolongeratree!
2. Replace root with last leaf.
1
20
5
24
124030
15
21
11
9
10
12
20
5
24
4030
15
21
11
9
10
12

But we’ve violated the structure condition!
3. Repeatedly swap the new element with its small-
est child until it reaches a place where it is no larger
than its children.
20
5
24
4030
15
21
11
9
10
12
20
5
24
4030
15
21
11
9
10
12
20
5
24
4030
15
21

11
9
10
12
20
5
24
4030
15
21
11
9 10
12
Why Does it Work?
Why does swapping the new node with its smallest
child work?
cb
a
c b
a
or
Suppose b ≤ c and a is not in the correct place. That
is, either a>bor a>c. In either case, since b ≤ c,
we know that a>b.
c
b
a
c b
a
or

Then we get
c
b
a c
b
a
or
respectively.
Is b smaller than its children? Yes, since b<aand
b ≤ c.
2