Duality in convex optimization

Chapter 5 Duality in convex optimization
Chapter 5.
Duality in convex optimization
tvnguyen (University of Science) Convex Optimization 81 / 108
Chapter 5 Duality in convex optimization
The Fermat Rule
Proposition. Let f : IR
n
→ IR ∪ {+∞} be a closed convex and proper
function. Then, for an element x
∗
∈ IR
n
the two following statements
are equivalent :
(i) f (x
∗
) ≤ f (x) for all x ∈ IR
n
(ii) 0 ∈ ∂f (x
∗
)
The necessary and sufficient condition 0 ∈ ∂f (x
∗
) is an extension of the
classical optimality condition for convex C
1
functions : ∇f (x
∗
) = 0.
So finding the optimal solutions of f can be attacked by solving the

generalized equation 0 ∈ ∂f (x)
tvnguyen (University of Science) Convex Optimization 82 / 108
Chapter 5 Duality in convex optimization
The constrained convex problem
Consider the following optimization problem
(P) min {f
0
(x) | x ∈ C}
where f
0
: IR
n
→ IR ∪ {+∞} is a closed convex and proper function (called
the objective function) and C is a closed convex nonempty subset of IR
n
(set of constraints). Assume dom f
0
∩ C = ∅.
Setting f = f
0
+ δ
C
, this problem can be written in the equivalent form
min {f (x) | x ∈ IR
n
}
When dom f
0
∩ int C = ∅, we have ∂f (x) = ∂f
0

(x) + ∂δ
C
(x). So
x
∗
optimal solution of (P) ⇔ 0 ∈ ∂f
0
(x
∗
) + ∂δ
C
(x
∗
)
To describe ∂δ
C
(x), we need to introduce the notion of tangent and
normal cone to C at x.
tvnguyen (University of Science) Convex Optimization 83 / 108
Chapter 5 Duality in convex optimization
The tangent and normal cones
Definition. Let C be a closed convex nonempty subset of IR
n
and let
x ∈ C.
(a) The tangent cone to C at x, denoted T
C
(x) is defined by
T
C

(x) = ∪
λ≥0
λ (C − x)
It is the closure of the cone spanned by C − x.
(b) The normal cone N
C
(x) to C at x is the polar cone of T
C
(x) :
N
C
(x) = {x
∗
∈ IR
n
| x
∗
, y ≤ 0 ∀y ∈ T
C
(x)}
= {x
∗
∈ IR
n
| x
∗
, y − x ≤ 0 ∀y ∈ C}
tvnguyen (University of Science) Convex Optimization 84 / 108
Chapter 5 Duality in convex optimization
Illustration

Tangent cones Normal cones
tvnguyen (University of Science) Convex Optimization 85 / 108
Chapter 5 Duality in convex optimization
Properties
Proposition. Let C be a closed convex nonempty subset of IR
n
and let
x ∈ C. Then
(i) T
C
(x) is a closed convex cone containing 0
(ii) T
C
(x) = IR
n
when x ∈ int C
(iii) N
C
(x) is a closed convex cone containing 0
(iv) N
C
(x) = {0} when x ∈ int C
Proposition. Let C be a closed convex nonempty subset of IR
n
and let
x ∈ C. Then
∂δ
C
(x) = N
C

(x)
tvnguyen (University of Science) Convex Optimization 86 / 108
Chapter 5 Duality in convex optimization
The constrained convex problem
Consider again the following optimization problem
(P) min {f
0
(x) | x ∈ C}
where f
0
: IR
n
→ IR ∪ {+∞} is a closed convex and proper function and C
is a closed convex nonempty subset of IR
n
.
Proposition. Assume that the following qualification assumption is
satisfied :
dom f
0
∩ int C = ∅
Then the following statements are equivalent :
(i) x
∗
is an optimal solution to (P) ;
(ii) x
∗
is a solution to the equation 0 ∈ ∂f
0
(x

∗
) + N
C
(x
∗
) ;
(iii) x
∗
∈ C and ∃ s ∈ ∂f
0
(x
∗
) such that s, x − x
∗
 ≥ 0 ∀x ∈ C
tvnguyen (University of Science) Convex Optimization 87 / 108
Chapter 5 Duality in convex optimization
The mathematical programming problem
Consider the problem
(P)

min f (x)
s.t. g
i
(x) ≤ 0, i = 1, . . . , m
where f : IR
n
→ IR ∪ {+∞} is closed convex and proper, and
g
1

, . . . , g
m
: IR
n
→ IR, are convex.
Here the constraint C has the following specific form
C = { x ∈ IR
n
| g
i
(x) ≤ 0, i = 1, . . . , m}
This problem is of fundamental importance : a large number of problems
in decision sciences, engineering, and so forth can be written as
mathematical programming problems.
tvnguyen (University of Science) Convex Optimization 88 / 108
Chapter 5 Duality in convex optimization
N
C
(x) when C = {x ∈ IR
n
| g(x) ≤ 0}
Proposition. Let C = {x ∈ IR
n
| g(x) ≤ 0} where g : IR
n
→ IR is
convex (and thus also continuous). Assume that C satisfies the
following Slater property :
there exists some x
0

∈ C such that g(x
0
) < 0
Then, for every x ∈ C
N
C
(x) =

{0} if g(x) < 0,
IR
+
∂g(x) if g(x) = 0.
As a consequence,
s ∈ N
C
(x) ⇔ ∃ λ ≥ 0 such that s ∈ λ ∂g (x) and λg (x) = 0
tvnguyen (University of Science) Convex Optimization 89 / 108
Chapter 5 Duality in convex optimization
N
C
(x) when C = {x ∈ IR
n
| g
i
(x) ≤ 0, i = 1, . . . , m}
Proposition. Let C = ∩
1≤i≤m
C
i
where for each i = 1, . . . , m

C
i
= {x ∈ IR
n
| g
i
(x) ≤ 0} and g
i
: IR
n
→ IR, i = 1, . . . , m is convex.
Assume that C satisfies the following Slater property :
there exists some x
0
∈ C such that g
i
(x
0
) < 0, i = 1, . . . , m
Then x
0
∈ ∩
i
int C
i
, δ
C
= δ
C
1

+ · · · + δ
C
m
, and (by the subdifferential
rule for the sum of convex functions)
∂δ
C
= ∂δ
C
1
+ · · · + ∂δ
C
m
As a consequence, for every x ∈ C,
N
C
(x) = N
C
1
(x) + · · · + N
C
m
(x)
tvnguyen (University of Science) Convex Optimization 90 / 108
Chapter 5 Duality in convex optimization
Karush-Kuhn-Tucker optimality conditions
Theorem. Suppose that f : IR
n
→ IR ∪ {+∞} is closed convex and
proper and that g

1
, . . . , g
m
: IR
n
→ IR are convex. Suppose also that the
Slater qualification assumption is satisfied :
∃ x
0
∈ IR
n
such that f (x
0
) < +∞ and g
i
(x
0
) < 0 ∀i = 1, . . . , m
Then the following statements are equivalent :
(i) x
∗
is a solution of problem (P) ;
(ii) there exist λ
1
, . . . , λ
m
such that








0 ∈ ∂f (x
∗
) + λ
1
∂g
1
(x
∗
) + · · · + λ
m
∂g
m
(x
∗
),
λ
i
≥ 0, i = 1, . . . , m
λ
i
g
i
(x
∗
) = 0, i = 1, . . . , m
g

i
(x
∗
) ≤ 0, i = 1, . . . , m.
tvnguyen (University of Science) Convex Optimization 91 / 108