ORF363 COS323 F14 Lec4

Bạn đang xem bản rút gọn của tài liệu. Xem và tải ngay bản đầy đủ của tài liệu tại đây (3.38 MB, 15 trang )

(1)<div class='page_container' data-page=1>

Instructor:
Amir Ali Ahmadi

Fall 2014

Convex sets
○

Convex functions
○

Convex optimization problems
○

Why convex optimization? Why so early in the course?
○

Convex optimization
•

This lecture:

Recall the general form of our optimization problems:

s.t.

In the last lecture, we focused on unconstrained optimization:
•

We saw the definitions of local and global optimality, and, first and
second order optimality conditions.

•

TAs: Y. Chen,
G. Hall,
J. Ye

In this lecture, we consider a very important special case of constrained
optimization problems known as "convex optimization problems".

•

will be a "convex function".
○

will be a "convex set".
○

These notions are defined formally in this lecture.
○

For these problems,
•

Convex optimization problems are pretty much the broadest class of
optimization problems that we know how to solve efficiently.

○

e.g., a local minimum is automatically a global minimum.


They have nice geometric properties;
○

Numerous important optimization problems in engineering,
operations research, machine learning, etc. are convex.

○

You should take advantage of this!


There is available software that can take (a large subset of) convex
problems written in very high-level language and solve it.

○

Convex optimization is one of the biggest success stories of modern
theory of optimization.

○

Roughly speaking, the high-level message is this:
•

</div>
(2)<div class='page_container' data-page=2>

Convex sets

Definition.A set is convex, if for all the line segment 
connecting and is also in In other words,

A point of the form , is called a convex combination
of and

•

Note that when we are at when we are at for

intermediate values of we are on the line segment connecting and
•

Convex:

Not convex:

</div>
(3)<div class='page_container' data-page=3>

Convex sets & Midpoint Convexity

Midpoint convexity is a notion that is equivalent to convexity in most practical
settings, but it is a little bit cleaner to work with.

Definition.A set is midpoint convex, if for all the midpoint 
between and is also in In other words,

Obviously, convex sets are midpoint convex.
•

e.g., a closed midpoint convex sets is convex.
○

What is an example of a midpoint convex set that is not convex?
(The set of all rational points in )

○

Under mild conditions, midpoint convex sets are convex
•

The nonconvex sets that we had are also not midpoint convex (why)?:

</div>
(4)<div class='page_container' data-page=4>

Common convex sets in optimization

Hyperplanes: 
•

Halfspaces: 
•

Euclidean balls: 2-norm)
•

Ellipsoids: )
•

(Prove convexity in each case.)

( here is an symmetric matrix)

Proof hint: Wait until you see convex functions
and quasiconvex functions. Observe that

</div>
(5)<div class='page_container' data-page=5>

Fancier convex sets

Many fundamental objects in mathematics have surprising convexity
properties.

The set of (symmetric) positive semidefinite matrices:

•

The set of nonnegative polynomials in variables and of degree
•

(A polynomial ) is nonnegative, if

Image credit: [BV04]
For example, prove that the following two sets are convex.

e.g.,

e.g.,

</div>
(6)<div class='page_container' data-page=6>

Intersections of convex sets

Easy to see that intersection of two convex sets is convex:
•

convex, convex convex.
Proof:

Obviously, the union may not be convex:
•

Polyhedra

Ubiquitous in optimization theory.
○

Feasible sets of "linear programs" (an upcoming subject).
○

A polyhedron is the solution set of finitely many linear inequalities.
•

Such sets are written in the form:
•

where is an matrix, and is an vector.
These sets are convex: intersection of halfspaces

where is the -th row of
•

e.g.,

</div>
(7)<div class='page_container' data-page=7>

Convex functions

Definition.A function is convex if its domain is a convex set and for 
all , in its domain, and all we have

In words: take any two points ; evaluated at any convex combination
should be no larger than the same convex combination of and
•

If interpretation is even easier: take any two points ;

evaluated at the midpoint should be no larger than the average of
and

•

Geometrically, the line segment connecting to sits
above the graph of

•

( )

</div>
(8)<div class='page_container' data-page=8>

Concave, if

•

Strictly convex, if

•

Strictly concave, if

•

Definition.A function is

Note: is concave if and only if is convex. Similarly, is

strictly concave if and only if is strictly convex.

The only functions that are both convex and concave are affine functions; i.e.,

functions of the form:

convex

(and strictly convex)

concave
(and strictly
concave)

neither convex
nor concave

both convex and
concave (but not
strictly)

Examples of univariate convex functions ( ):

•

(defined on ) or
•

(defined on )
•

•

(defined on )

•

Let's see some examples of convex functions (selection from [BV04]; see this
reference for many more examples).

Try to plot the functions above and convince yourself of convexity visually.
•

Can you formally verify that these functions are convex?
•

We will soon see some characterizations of convex functions that make the task
of verifying convexity a bit easier.

</div>
(9)<div class='page_container' data-page=9>

Examples of convex functions

( )

Affine functions: (for any
•

(convex, but not strictly convex; also concave)

Some quadratic functions:

•

Convex if and only if
○

Strictly convex if and only if
○

Concave iff Strictly concave iff
○

Proofs are easy from the second order
characterization of convexity (coming up).
○

a.

b.

c.

Any norm: meaning, any function satisfying:

•

•

•

•
Examples:

</div>
(10)<div class='page_container' data-page=10>

Midpoint convex functions

Same idea as what we saw for midpoint convex sets.

Obviously, convex functions are midpoint convex.
•

Continuous, midpoint convex functions are convex.
•

Definition.A function is midpoint convex if its domain is a convex 
set and for all , in its domain, we have

Convexity = Convexity along all lines

Theorem.A function is convex if and only if the function
given by is convex (as a univariate function), for all in
domain of and all (The domain of here is all for which is

in the domain of

The notion of convexity is defined based on line segments.
○

This should be intuitive geometrically:
•

The theorem simplifies many basic proofs in convex analysis.
•

But it does not usually make verification of convexity that much easier;
the condition needs to hold for all lines (and we have infinitely many).
•

Many of the algorithms we will see in future lectures work by iteratively
minimizing a function over lines. It's useful to remember that the

</div>
(11)<div class='page_container' data-page=11>

Epigraph

We will see a couple; via epigraphs, and sublevel sets.
•

Is there a connection between convex sets and convex functions?

Definition.The epigraph of a function is a subset of
defined as

Theorem.A function is convex if and only if its epigraph is convex
(as a set).

</div>
(12)<div class='page_container' data-page=12>

Convexity of sublevel sets

Definition.The -sublevel set of a function is the set

Several sublevel
sets (for different
values of

Theorem.If a function is convex, then all its sublevel sets are
convex sets.

Converse not true.
•

A function whose sublevel sets are
all convex is called quasiconvex.
•

</div>
(13)<div class='page_container' data-page=13>

Convex optimization problems

A convex optimization problem is an optimization problem of the form

s.t.

where are convexfunctions and are affinefunctions.

Observe that for a convex optimization problem is a convex set
(why?)

○

Consider for example, Then is a convex set,
but minimizing a convex function over is not a convex

optimization problem per our definition.


However, the same set can be represented as
and then this would be a convex optimization problem with our
definition.



But the converse is not true:
○

Let denote the feasible set:
•

Here is another example of a convex feasible set that fails our definition of a

convex optimization problem:

•

</div>
(14)<div class='page_container' data-page=14>

Convex optimization problems (cont'd)

We require this stronger definition because otherwise many abstract and
complex optimization problems can be formulated as optimization

problems over a convex set. (Think, e.g., of the set of nonnegative
polynomials.) The stronger definition is much closer to what we can
actually solve efficiently.

•

• The software CVX that we'll be using ONLY accepts convex optimization
problems defined as above.

• Beware that [CZ13] uses the weaker and more abstract definition for a
convex optimization problem (i.e., the definition that simply asks to be
a convex set.)

Acceptable constraints in CVX:

• Convex Concave
• Affine Affine

This is really the same as:
• Convex 0

• Affine 0

Why?

</div>
(15)<div class='page_container' data-page=15>

• Further reading for this lecture can include the first few pages of Chapters
2,3,4 of [BV04]. Your [CZ13] book defines convex sets in Section 4.3. Convex
optimization appears in Chapter 22. The relevant sections are 22.1-22.3.