Optimal control theory applications to management science and economics, 3rd edition

Suresh P. Sethi

Optimal
Control Theory
Applications to Management Science
and Economics
Third Edition


Optimal Control Theory


Suresh P. Sethi

Optimal Control Theory
Applications to Management Science
and Economics
Third Edition

123


Suresh P. Sethi
Jindal School of Management, SM30
University of Texas at Dallas
Richardson, TX, USA

ISBN 978-3-319-98236-6
ISBN 978-3-319-98237-3 (eBook)
/>Library of Congress Control Number: 2018955904
2nd edition: © Springer-Verlag US 2000

© Springer Nature Switzerland AG 2019
This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of
the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation,
broadcasting, reproduction on microfilms or in any other physical way, and transmission or information
storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology
now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication
does not imply, even in the absence of a specific statement, that such names are exempt from the relevant
protective laws and regulations and therefore free for general use.
The publisher, the authors and the editors are safe to assume that the advice and information in this book
are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the
editors give a warranty, express or implied, with respect to the material contained herein or for any errors or
omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in
published maps and institutional affiliations.
This Springer imprint is published by the registered company Springer Nature Switzerland AG
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland


This book is dedicated to the memory of
my parents
Manak Bai and Gulab Chand Sethi


Preface to Third Edition
The third edition of this book will not see my co-author Gerald L.
Thompson, who very sadly passed away on November 9, 2009. Gerry
and I wrote the first edition of the 1981 book sitting practically side by
side, and I learned a great deal about book writing in the process. He
was also my PhD supervisor and mentor and he is greatly missed.
After having used the second edition of the book in the classroom

for many years, the third edition arrives with new material and many
improvements. Examples and exercises related to the interpretation of
the adjoint variables and Lagrange multipliers are inserted in Chaps. 2–
4. Direct maximum principle is now discussed in detail in Chap. 4 along
with the existing indirect maximum principle from the second edition.
Chattering or relaxed controls leading to pulsing advertising policies are
introduced in Chap. 7. An application to information systems involving
chattering controls is added as an exercise.
The objective function in Sect. 11.1.3 is changed to the more popular
objective of maximizing the total discounted society’s utility of consumption. Further discussion leading to obtaining a saddle-point path on the
phase diagram leading to the long-run stationary equilibrium is provided
in Sect. 11.2. For this purpose, a global saddle-point theorem is stated
in Appendix D.7. Also inserted in Appendix D.8 is a discussion of the
Sethi-Skiba points which lead to nonunique stable equilibria. Finally,
a new Sect. 11.4 contains an adverse selection model with continuum of
the agent types in a principal-agent framework, which requires an application of the maximum principle.
Chapter 12 of the second edition is removed except for the material
on differential games and the distributed parameter maximum principle.
The differential game material joins new topics of stochastic Nash differential games and Stackelberg differential games via their applications to
marketing to form a new Chap. 13 titled Differential Games. As a result,
Chap. 13 of the second edition becomes Chap. 12. The material on the
distributed parameter maximum principle is now Appendix D.9.
The exposition is revised in some places for better reading. New
exercises are added and the list of references is updated. Needless to say,
the errors in the second edition are corrected, and the notation is made
consistent.
vii


viii

Preface to Third Edition

Thanks are due to Huseyin Cavusoglu, Andrei Dmitruk, Gustav Feichtinger, Richard Hartl, Yonghua Ji, Subodha Kumar, Sirong Lao, Helmut Maurer, Ernst Presman, Anyan Qi, Andrea Seidl, Atle Seierstad,
Xi Shan, Lingling Shi, Xiahong Yue, and the students in my Optimal
Control Theory and Applications course over the years for their suggestions for improvement. Special thanks go to Qi (Annabelle) Feng
for her dedication in updating and correcting the forthcoming solution
manual that went with the first edition. I cannot thank Barbara Gordon
and Lindsay Wilson enough for their assistance in the preparation of
the text, solution manual, and presentation materials. In addition, the
meticulous copy editing of the entire book by Lindsay Wilson is much
appreciated. Anshuman Chutani, Pooja Kamble, and Shivani Thakkar
are also thanked for their assistance in drawing some of the figures in
the book.
Richardson, TX, USA
June 2018

Suresh P. Sethi


Preface to Second Edition
The first edition of this book, which provided an introduction to optimal control theory and its applications to management science to many
students in management, industrial engineering, operations research and
economics, went out of print a number of years ago. Over the years we
have received feedback concerning its contents from a number of instructors who taught it, and students who studied from it. We have also kept
up with new results in the area as they were published in the literature.
For this reason we felt that now was a good time to come out with a
new edition. While some of the basic material remains, we have made
several big changes and many small changes which we feel will make the
use of the book easier.

The most visible change is that the book is written in Latex and the
figures are drawn in CorelDRAW, in contrast to the typewritten text
and hand-drawn figures of the first edition. We have also included some
problems along with their numerical solutions obtained using Excel.
The most important change is the division of the material in the
old Chap. 3, into Chaps. 3 and 4 in the new edition. Chapter 3 now
contains models having mixed (control and state) constraints, current
value formulations, terminal conditions and model types, while Chap. 4
covers the more difficult topic of pure state constraints, together with
mixed constraints. Each of these chapters contain new results that were
not available when the first edition was published.
The second most important change is the expansion of the material in
the old Sect. 12.4 on stochastic optimal control theory and its becoming
the new Chap. 13. The new Chap. 12 now contains the following advanced topics on optimal control theory: differential games, distributed
parameter systems, and impulse control. The new Chap. 13 provides a
brief introduction to stochastic optimal control problems. It contains
formulations of simple stochastic models in production, marketing and
finance, and their solutions. We deleted the old Chap. 11 of the first
edition on computational methods, since there are a number of excellent
references now available on this topic. Some of these references are listed
in Sect. 4.2 of Chap. 4 and Sect. 8.3 of Chap. 8.
ix


x

Preface to Second Edition

The emphasis of this book is not on mathematical rigor, but rather
on developing models of realistic situations faced in business and management. For that reason we have given, in Chaps. 2 and 8, proofs of the

continuous and discrete maximum principles by using dynamic programming and Kuhn-Tucker theory, respectively. More general maximum
principles are stated without proofs in Chaps. 3, 4 and 12.
One of the fascinating features of optimal control theory is its extraordinarily wide range of possible applications. We have covered some
of these as follows: Chap. 5 covers finance; Chap. 6 considers production
and inventory problems; Chap. 7 covers marketing problems; Chap. 9
treats machine maintenance and replacement; Chap. 10 deals with problems of optimal consumption of natural resources (renewable or exhaustible); and Chap. 11 discusses a number of applications of control
theory to economics. The contents of Chaps. 12 and 13 have been described earlier.
Finally, four appendices cover either elementary material, such as
the theory of differential equations, or very advanced material, whose
inclusion in the main text would interrupt its continuity. At the end
of the book is an extensive but not exhaustive bibliography of relevant
material on optimal control theory including surveys of material devoted
to specific applications.
We are deeply indebted to many people for their part in making this
edition possible. Onur Arugaslan, Gustav Feichtinger, Neil Geismar,
Richard Hartl, Steffen Jørgensen, Subodha Kumar, Helmut Maurer, Gerhard Sorger, and Denny Yeh made helpful comments and suggestions
about the first edition or preliminary chapters of this revision. Many
students who used the first edition, or preliminary chapters of this revision, also made suggestions for improvements. We would like to express
our gratitude to all of them for their help. In addition we express our
appreciation to Eleanor Balocik, Frank (Youhua) Chen, Feng Cheng,
Howard Chow, Barbara Gordon, Jiong Jiang, Kuntal Kotecha, Ming
Tam, and Srinivasa Yarrakonda for their typing of the various drafts of
the manuscript. They were advised by Dirk Beyer, Feng Cheng, Subodha Kumar, Young Ryu, Chelliah Sriskandarajah, Wulin Suo, Houmin
Yan, Hanqin Zhang, and Qing Zhang on the technical problems of using
LATEX.
We also thank our wives and children—Andrea, Chantal, Anjuli,
Dorothea, Allison, Emily, and Abigail—for their encouragement and understanding during the time-consuming task of preparing this revision.


Preface to Second Edition

xi

Finally, while we regret that lack of time and pressure of other duties prevented us from bringing out a second edition soon after the first
edition went out of print, we sincerely hope that the wait has been worthwhile. In spite of the numerous applications of optimal control theory
which already have been made to areas of management science and economics, we continue to believe there is much more that remains to be
done. We hope the present revision will rekindle interest in furthering
such applications, and will enhance the continued development in the
field.
Richardson, TX, USA
Pittsburgh, PA, USA
January 2000

Suresh P. Sethi
Gerald L. Thompson


Preface to First Edition
The purpose of this book is to exposit, as simply as possible, some
recent results obtained by a number of researchers in the application of
optimal control theory to management science. We believe that these results are very important and deserve to be widely known by management
scientists, mathematicians, engineers, economists, and others. Because
the mathematical background required to use this book is two or three
semesters of calculus plus some differential equations and linear algebra,
the book can easily be used to teach a course in the junior or senior
undergraduate years or in the early years of graduate work. For this
purpose, we have included numerous worked-out examples in the text,
as well as a fairly large number of exercises at the end of each chapter.
Answers to selected exercises are included in the back of the book. A
solutions manual containing completely worked-out solutions to all of

the 205 exercises is also available to instructors.
The emphasis of the book is not on mathematical rigor, but on modeling realistic situations faced in business and management. For that
reason, we have given in Chaps. 2 and 7 only heuristic proofs of the continuous and discrete maximum principles, respectively. In Chap. 3 we
have summarized, as succinctly as we can, the most important model
types and terminal conditions that have been used to model management problems. We found it convenient to put a summary of almost all
the important management science models on two pages: see Tables 3.1
and 3.3.
One of the fascinating features of optimal control theory is the extraordinarily wide range of its possible applications. We have tried to
cover a wide variety of applications as follows: Chap. 4 covers finance;
Chap. 5 considers production and inventory; Chap. 6 covers marketing;
Chap. 8 treats machine maintenance and replacement; Chap. 9 deals with
problems of optimal consumption of natural resources (renewable or exhaustible); and Chap. 10 discusses several economic applications.
In Chap. 11 we treat some computational algorithms for solving optimal control problems. This is a very large and important area that
needs more development.
xiii


xiv

Preface to First Edition

Chapter 12 treats several more advanced topics of optimal control: differential games, distributed parameter systems, optimal filtering,
stochastic optimal control, and impulsive control. We believe that some
of these models are capable of wider applications and further theoretical
development.
Finally, four appendixes cover either elementary material, such as
differential equations, or advanced material, whose inclusion in the main
text would spoil its continuity. Also at the end of the book is a bibliography of works actually cited in the text. While it is extensive, it is by no
means an exhaustive bibliography of management science applications
of optimal control theory. Several surveys of such applications, which

contain many other important references, are cited.
We have benefited greatly during the writing of this book by having discussions with and obtaining suggestions from various colleagues
and students. Our special thanks go to Gustav Feichtinger for his careful reading and suggestions for improvement of the entire book. Carl
Norstr¨
om contributed two examples to Chaps. 4 and 5 and made many
suggestions for improvement. Jim Bookbinder used the manuscript for
a course at the University of Toronto, and Tom Morton suggested some
improvements for Chap. 5. The book has also benefited greatly from various coauthors with whom we have done research over the years. Both of
us also have received numerous suggestions for improvements from the
students in our applied control theory courses taught during the past
several years. We would like to express our gratitude to all these people
for their help.
The book has gone through several drafts, and we are greatly indebted to Eleanor Balocik and Rosilita Jones for their patience and
careful typing.
Although the applications of optimal control theory to management
science are recent and many fascinating applications have already been
made, we believe that much remains to be done. We hope that this book
will contribute to the popularity of the area and will enhance future
developments.
Toronto, ON, Canada
Pittsburgh, PA, USA
August 1981

Suresh P. Sethi
Gerald L. Thompson


Contents
1 What Is Optimal Control Theory?
1.1 Basic Concepts and Definitions . . . . . . . . . . . . . .

1.2 Formulation of Simple Control Models . . . . . . . . . .
1.3 History of Optimal Control Theory . . . . . . . . . . .
1.4 Notation and Concepts Used . . . . . . . . . . . . . . .
1.4.1 Differentiating Vectors and Matrices with Respect
To Scalars . . . . . . . . . . . . . . . . . . . . . .
1.4.2 Differentiating Scalars with Respect to Vectors .
1.4.3 Differentiating Vectors with Respect to Vectors .
1.4.4 Product Rule for Differentiation . . . . . . . . .
1.4.5 Miscellany . . . . . . . . . . . . . . . . . . . . . .
1.4.6 Convex Set and Convex Hull . . . . . . . . . . .
1.4.7 Concave and Convex Functions . . . . . . . . . .
1.4.8 Affine Function and Homogeneous Function of
Degree k . . . . . . . . . . . . . . . . . . . . . . .
1.4.9 Saddle Point . . . . . . . . . . . . . . . . . . . .
1.4.10 Linear Independence and Rank of a Matrix . . .
1.5 Plan of the Book . . . . . . . . . . . . . . . . . . . . . .
2 The Maximum Principle: Continuous Time
2.1 Statement of the Problem . . . . . . . . . . . . . . .
2.1.1 The Mathematical Model . . . . . . . . . . .
2.1.2 Constraints . . . . . . . . . . . . . . . . . . .
2.1.3 The Objective Function . . . . . . . . . . . .
2.1.4 The Optimal Control Problem . . . . . . . .
2.2 Dynamic Programming and the Maximum Principle
2.2.1 The Hamilton-Jacobi-Bellman Equation . . .
2.2.2 Derivation of the Adjoint Equation . . . . . .

.
.
.
.

.
.
.
.

.
.
.
.
.
.
.
.

1
2
4
9
11
12
13
14
16
16
20
20
22
22
23
23

27
27
28
28
29
29
32
32
36
xv


xvi

CONTENTS
2.2.3
2.2.4

2.3
2.4
2.5

The Maximum Principle . .
Economic Interpretations of
Principle . . . . . . . . . .
Simple Examples . . . . . . . . . .
Sufficiency Conditions . . . . . . .
Solving a TPBVP by Using Excel .

. .

the
. .
. .
. .
. .

. . . . . . .
Maximum
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .

. . .

39

.
.
.
.

40
42
53
57

.
.
.

.

.
.
.
.

3 The Maximum Principle: Mixed Inequality
Constraints
3.1 A Maximum Principle for Problems with Mixed Inequality
Constraints . . . . . . . . . . . . . . . . . . . . . . . . .
3.2 Sufficiency Conditions . . . . . . . . . . . . . . . . . . .
3.3 Current-Value Formulation . . . . . . . . . . . . . . . .
3.4 Transversality Conditions: Special Cases . . . . . . . . .
3.5 Free Terminal Time Problems . . . . . . . . . . . . . . .
3.6 Infinite Horizon and Stationarity . . . . . . . . . . . . .
3.7 Model Types . . . . . . . . . . . . . . . . . . . . . . . .
4 The Maximum Principle: Pure State and Mixed
Inequality Constraints
4.1 Jumps in Marginal Valuations . . . . . . . . . . . . .
4.2 The Optimal Control Problem with Pure and Mixed
Constraints . . . . . . . . . . . . . . . . . . . . . . .
4.3 The Maximum Principle: Direct Method . . . . . . .
4.4 Sufficiency Conditions: Direct Method . . . . . . . .
4.5 The Maximum Principle: Indirect Method . . . . . .
4.6 Current-Value Maximum Principle:
Indirect Method . . . . . . . . . . . . . . . . . . . .
5 Applications to Finance
5.1 The Simple Cash Balance Problem . . . . . . . .
5.1.1 The Model . . . . . . . . . . . . . . . . .

5.1.2 Solution by the Maximum Principle . . .
5.2 Optimal Financing Model . . . . . . . . . . . . .
5.2.1 The Model . . . . . . . . . . . . . . . . .
5.2.2 Application of the Maximum Principle . .
5.2.3 Synthesis of Optimal Control Paths . . .
5.2.4 Solution for the Infinite Horizon Problem

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

69
70
79
80
86

93
103
109

. .

125
127

.
.
.
.

.
.
.
.

129
132
136
137

. .

147

.
.

.
.
.
.
.
.

159
160
160
161
164
165
167
170
180

.
.
.
.
.
.
.
.


CONTENTS

xvii

6 Applications to Production and Inventory
6.1 Production-Inventory Systems . . . . . . . . . . . . . . .
6.1.1 The Production-Inventory Model . . . . . . . . .
6.1.2 Solution by the Maximum Principle . . . . . . .
6.1.3 The Infinite Horizon Solution . . . . . . . . . . .
6.1.4 Special Cases of Time Varying Demands . . . . .
6.1.5 Optimality of a Linear Decision Rule . . . . . . .
6.1.6 Analysis with a Nonnegative Production
Constraint . . . . . . . . . . . . . . . . . . . . . .
6.2 The Wheat Trading Model . . . . . . . . . . . . . . . .
6.2.1 The Model . . . . . . . . . . . . . . . . . . . . .
6.2.2 Solution by the Maximum Principle . . . . . . .
6.2.3 Solution of a Special Case . . . . . . . . . . . . .
6.2.4 The Wheat Trading Model with No Short-Selling
6.3 Decision Horizons and Forecast Horizons . . . . . . . . .
6.3.1 Horizons for the Wheat Trading Model with
No Short-Selling . . . . . . . . . . . . . . . . . .
6.3.2 Horizons for the Wheat Trading Model with No
Short-Selling and a Warehousing Constraint . . .

191
192
192
193
196
197
200

7 Applications to Marketing

7.1 The Nerlove-Arrow Advertising Model . . . . . . . . .
7.1.1 The Model . . . . . . . . . . . . . . . . . . . .
7.1.2 Solution by the Maximum Principle . . . . . .
7.1.3 Convex Advertising Cost and Relaxed Controls
7.2 The Vidale-Wolfe Advertising Model . . . . . . . . . .
7.2.1 Optimal Control Formulation for the
Vidale-Wolfe Model . . . . . . . . . . . . . . .
7.2.2 Solution Using Green’s Theorem When
Q Is Large . . . . . . . . . . . . . . . . . . . .
7.2.3 Solution When Q Is Small . . . . . . . . . . . .
7.2.4 Solution When T Is Infinite . . . . . . . . . . .

.
.
.
.
.

225
226
226
228
232
235

.

236

.

.
.

237
245
247

.
.
.
.
.

259
259
260
262
267
268

8 The Maximum Principle: Discrete Time
8.1 Nonlinear Programming Problems . . . . . . .
8.1.1 Lagrange Multipliers . . . . . . . . . . .
8.1.2 Equality and Inequality Constraints . .
8.1.3 Constraint Qualification . . . . . . . . .
8.1.4 Theorems from Nonlinear Programming

.
.
.

.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

202
204
205
206
206
208
213
214
214

xviii
8.2

8.3

CONTENTS
A Discrete Maximum Principle . . . . . . . . . . .
8.2.1 A Discrete-Time Optimal Control Problem
8.2.2 A Discrete Maximum Principle . . . . . . .
8.2.3 Examples . . . . . . . . . . . . . . . . . . .
A General Discrete Maximum Principle . . . . . .

.
.
.
.
.

.
.
.
.
.

.
.
.
.

.

269
269
270
272
276

.
.
.
.
.

.
.
.
.
.

283
284
284
285
287
289

.
.
.

.
.
.
.
.

9 Maintenance and Replacement
9.1 A Simple Maintenance and Replacement Model . . .
9.1.1 The Model . . . . . . . . . . . . . . . . . . .
9.1.2 Solution by the Maximum Principle . . . . .
9.1.3 A Numerical Example . . . . . . . . . . . . .
9.1.4 An Extension . . . . . . . . . . . . . . . . . .
9.2 Maintenance and Replacement for
a Machine Subject to Failure . . . . . . . . . . . . .
9.2.1 The Model . . . . . . . . . . . . . . . . . . .
9.2.2 Optimal Policy . . . . . . . . . . . . . . . . .
9.2.3 Determination of the Sale Date . . . . . . . .
9.3 Chain of Machines . . . . . . . . . . . . . . . . . . .
9.3.1 The Model . . . . . . . . . . . . . . . . . . .
9.3.2 Solution by the Discrete Maximum Principle
9.3.3 Special Case of Bang-Bang Control . . . . . .
9.3.4 Incorporation into the Wagner-Whitin
Framework for a Complete Solution . . . . .
9.3.5 A Numerical Example . . . . . . . . . . . . .

.
.
.
.
.

.
.
.

290
291
293
296
297
297
299
301

. .
. .

301
302

10 Applications to Natural Resources
10.1 The Sole-Owner Fishery Resource Model . .
10.1.1 The Dynamics of Fishery Models . .
10.1.2 The Sole Owner Model . . . . . . .
10.1.3 Solution by Green’s Theorem . . . .
10.2 An Optimal Forest Thinning Model . . . .
10.2.1 The Forestry Model . . . . . . . . .
10.2.2 Determination of Optimal Thinning
10.2.3 A Chain of Forests Model . . . . . .
10.3 An Exhaustible Resource Model . . . . . .
10.3.1 Formulation of the Model . . . . . .

10.3.2 Solution by the Maximum Principle

.
.
.
.
.
.
.
.
.
.
.

311
312
312
313
314
317
317
318
321
324
324
327

.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.

.
.
.
.
.
.
.
.


CONTENTS
11 Applications to Economics
11.1 Models of Optimal Economic Growth . . . . . . .
11.1.1 An Optimal Capital Accumulation Model
11.1.2 Solution by the Maximum Principle . . .
11.1.3 Introduction of a Growing Labor Force . .
11.1.4 Solution by the Maximum Principle . . .
11.2 A Model of Optimal Epidemic Control . . . . . .
11.2.1 Formulation of the Model . . . . . . . . .
11.2.2 Solution by Green’s Theorem . . . . . . .
11.3 A Pollution Control Model . . . . . . . . . . . .
11.3.1 Model Formulation . . . . . . . . . . . . .
11.3.2 Solution by the Maximum Principle . . .
11.3.3 Phase Diagram Analysis . . . . . . . . . .
11.4 An Adverse Selection Model . . . . . . . . . . . .
11.4.1 Model Formulation . . . . . . . . . . . . .
11.4.2 The Implementation Problem . . . . . . .
11.4.3 The Optimization Problem . . . . . . . .
11.5 Miscellaneous Applications . . . . . . . . . . . .

xix

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

335
335
336
336

338
339
343
343
344
346
347
348
349
352
352
353
354
360

.
.
.
.
.
.

365
366
370
372
375
377
383

.
.
.
.

385
386
387
388
388

. .

389

. .
of
. .

392

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

12 Stochastic Optimal Control
12.1 Stochastic Optimal Control . . . . . . . . . . . . . . .
12.2 A Stochastic Production Inventory Model . . . . . . .
12.2.1 Solution for the Production Planning Problem
12.3 The Sethi Advertising Model . . . . . . . . . . . . . .
12.4 An Optimal Consumption-Investment Problem . . . .
12.5 Concluding Remarks . . . . . . . . . . . . . . . . . . .
13 Differential Games
13.1 Two-Person Zero-Sum Differential Games . . . . . .
13.2 Nash Differential Games . . . . . . . . . . . . . . . .
13.2.1 Open-Loop Nash Solution . . . . . . . . . . .
13.2.2 Feedback Nash Solution . . . . . . . . . . . .
13.2.3 An Application to Common-Property Fishery
Resources . . . . . . . . . . . . . . . . . . . .
13.3 A Feedback Nash Stochastic Differential
Game in Advertising . . . . . . . . . . . . . . . . . .
13.4 A Feedback Stackelberg Stochastic Differential Game
Cooperative Advertising . . . . . . . . . . . . . . . .

.
.

.
.

395


xx

CONTENTS

A Solutions of Linear Differential Equations
A.1 First-Order Linear Equations . . . . . . . . . . . . . . .
A.2 Second-Order Linear Equations with
Constant Coefficients . . . . . . . . . . . . . . . . . . . .
A.3 System of First-Order Linear Equations . . . . . . . . .
A.4 Solution of Linear Two-Point Boundary Value Problems
A.5 Solutions of Finite Difference Equations . . . . . . . . .
A.5.1 Changing Polynomials in Powers of k into
Factorial Powers of k . . . . . . . . . . . . . . . .
A.5.2 Changing Factorial Powers of k into Ordinary
Powers of k . . . . . . . . . . . . . . . . . . . . .

409
409
410
410
413
414
415
416

B Calculus of Variations and Optimal Control Theory
419
B.1 The Simplest Variational Problem . . . . . . . . . . . . 420
B.2 The Euler-Lagrange Equation . . . . . . . . . . . . . . . 421
B.3 The Shortest Distance Between Two Points on the Plane 424
B.4 The Brachistochrone Problem . . . . . . . . . . . . . . . 424
B.5 The Weierstrass-Erdmann Corner
Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . 427
B.6 Legendre’s Conditions: The Second Variation . . . . . . 428
B.7 Necessary Condition for a Strong
Maximum . . . . . . . . . . . . . . . . . . . . . . . . . . 429
B.8 Relation to Optimal Control Theory . . . . . . . . . . . 430
C An Alternative Derivation of the Maximum Principle 433
C.1 Needle-Shaped Variation . . . . . . . . . . . . . . . . . . 434
C.2 Derivation of the Adjoint Equation and the Maximum
Principle . . . . . . . . . . . . . . . . . . . . . . . . . . . 436
D Special Topics in Optimal Control
D.1 The Kalman Filter . . . . . . . . . . . . . . . . . . .
D.2 Wiener Process and Stochastic Calculus . . . . . . .
D.3 The Kalman-Bucy Filter . . . . . . . . . . . . . . . .
D.4 Linear-Quadratic Problems . . . . . . . . . . . . . .
D.4.1 Certainty Equivalence or Separation Principle
D.5 Second-Order Variations . . . . . . . . . . . . . . . .
D.6 Singular Control . . . . . . . . . . . . . . . . . . . .

.
.
.
.

.
.
.

.
.
.
.
.
.
.

441
441
444
447
448
451
452
454


CONTENTS
D.7 Global Saddle Point Theorem . . . . . . . . . . . . . . .
D.8 The Sethi-Skiba Points . . . . . . . . . . . . . . . . . . .
D.9 Distributed Parameter Systems . . . . . . . . . . . . . .

xxi
456
458

460

E Answers to Selected Exercises

465

Bibliography

473

Index

547


List of Figures
1.1
1.2
1.3
1.4

The Brachistochrone problem . . .
Illustration of left and right limits
A concave function . . . . . . . . .
An illustration of a saddle point . .

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

9
18
21
23

2.1
2.2
2.3
2.4
2.5
2.6
2.7
2.8

An optimal path in the state-time space . . . . . . . .
Optimal state and adjoint trajectories for Example 2.2
Optimal state and adjoint trajectories for Example 2.3
Optimal trajectories for Examples 2.4 and 2.5 . . . . .
Optimal control for Example 2.6 . . . . . . . . . . . .
The flowchart for Example 2.8 . . . . . . . . . . . . .
Solution of TPBVP by excel . . . . . . . . . . . . . . .
Water reservoir of Exercise 2.18 . . . . . . . . . . . . .

.
.
.
.
.
.
.
.

34
44

46
48
53
58
60
63

3.1
3.2

State and adjoint trajectories in Example 3.4 . . . . . .
Minimum time optimal response for Example 3.6 . . . .

93
101

4.1
4.2
4.3
4.4
4.5

Feasible state space and optimal state trajectory
for Examples 4.1 and 4.4 . . . . . . . . . . . . . .
State and adjoint trajectories in Example 4.3 . .
Adjoint trajectory for Example 4.4 . . . . . . . .
Two-reservoir system of Exercise 4.8 . . . . . . .
Feasible space for Exercise 4.28 . . . . . . . . . .

.

.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

128
143
147
151
157

5.1

5.2
5.3
5.4
5.5
5.6

Optimal policy shown in (λ1 , λ2 ) space .
Optimal policy shown in (t, λ2 /λ1 ) space
Case A: g ≤ r . . . . . . . . . . . . . . .
Case B: g > r . . . . . . . . . . . . . . .
Optimal path for case A: g ≤ r . . . . .
Optimal path for case B: g > r . . . . .

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.

.
.
.

.
.
.
.
.
.

163
164
169
170
174
179

.
.
.
.
.
.

.
.
.
.

.
.
.
.
.
.

.
.
.
.

.
.
.
.
.
.

.
.
.
.

.
.
.
.
.
.

.
.
.
.

.
.
.
.
.
.

.
.
.
.

.
.
.
.

xxiii


xxiv
5.7
5.8

LIST OF FIGURES
Solution for Exercise 5.4 . . . . . . . . . . . . . . . . . .
Adjoint trajectories for Exercise 5.5 . . . . . . . . . . .

186
187

Solution of Example 6.1 with I0 = 10 . . . . . . . . . . .
Solution of Example 6.1 with I0 = 50 . . . . . . . . . . .
Solution of Example 6.1 with I0 = 30 . . . . . . . . . . .
Optimal production rate and inventory level with different
initial inventories . . . . . . . . . . . . . . . . . . . . . .
6.5 The price trajectory (6.56) . . . . . . . . . . . . . . . . .
6.6 Adjoint variable, optimal policy and inventory in the
wheat trading model . . . . . . . . . . . . . . . . . . . .
6.7 Adjoint trajectory and optimal policy for the wheat trading model . . . . . . . . . . . . . . . . . . . . . . . . . .
6.8 Decision horizon and optimal policy for the wheat trading
model . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.9 Optimal policy and horizons for the wheat trading model
with no short-selling and a warehouse constraint . . . .
6.10 Optimal policy and horizons for Example 6.3 . . . . . .
6.11 Optimal policy and horizons for Example 6.4 . . . . . .

199
199
200

6.1
6.2
6.3

6.4

7.1
7.2

7.12
7.13
7.14

Optimal policies in the Nerlove-Arrow model . . . . . .
A case of a time-dependent turnpike and the nature of
optimal control . . . . . . . . . . . . . . . . . . . . . . .
A near-optimal control of problem (7.15) . . . . . . . . .
Feasible arcs in (t, x)-space . . . . . . . . . . . . . . . .
Optimal trajectory for Case 1: x0 ≤ xs and xT ≤ xs . .
Optimal trajectory for Case 2: x0 < xs and xT > xs . .
Optimal trajectory for Case 3: x0 > xs and xT < xs . .
Optimal trajectory for Case 4: x0 > xs and xT > xs . .
Optimal trajectory (solid lines) . . . . . . . . . . . . . .
Optimal trajectory when T is small in Case 1: x0 < xs
and xT > xs . . . . . . . . . . . . . . . . . . . . . . . . .
Optimal trajectory when T is small in Case 2: x0 > xs
and xT > xs . . . . . . . . . . . . . . . . . . . . . . . . .
Optimal trajectory for Case 2 of Theorem 7.1 for Q = ∞
Optimal trajectories for x(0) < x
ˆ . . . . . . . . . . . . .
Optimal trajectory for x(0) > x
ˆ . . . . . . . . . . . . . .

8.1

8.2

Shortest distance from point (2,2) to the semicircle . . .
Graph of Example 8.5 . . . . . . . . . . . . . . . . . . .

7.3
7.4
7.5
7.6
7.7
7.8
7.9
7.10
7.11

204
207
209
212
215
216
218
219
230
231
233
238
240
241
241

242
243
243
244
244
249
250
266
267


LIST OF FIGURES

xxv

8.3
8.4

Discrete-time conventions . . . . . . . . . . . . . . . . .
∗
Optimal state xk and adjoint λk . . . . . . . . . . . . .

270
275

9.1
9.2

Optimal maintenance and machine resale value . . . . .
Sat function optimal control . . . . . . . . . . . . . . . .

289
291

10.1 Optimal policy for the sole owner fishery model . . . . .
10.2 Singular usable timber volume x
¯(t) . . . . . . . . . . . .
10.3 Optimal thinning u∗ (t) and timber volume x∗ (t) for the
forest thinning model when x0 < x
¯(t0 ) . . . . . . . . . .
10.4 Optimal thinning u∗ (t) and timber volume x∗ (t) for the
chain of forests model when T > tˆ . . . . . . . . . . . .
10.5 Optimal thinning and timber volume x∗ (t) for the chain
of forests model when T ≤ tˆ . . . . . . . . . . . . . . . .
10.6 The demand function . . . . . . . . . . . . . . . . . . . .
10.7 The profit function . . . . . . . . . . . . . . . . . . . . .
10.8 Optimal price trajectory for T ≥ T¯ . . . . . . . . . . . .
10.9 Optimal price trajectory for T < T¯ . . . . . . . . . . . .

316
320

11.1
11.2
11.3
11.4
11.5
11.6
11.7

Phase diagram for the optimal growth model .
Optimal trajectory when xT > xs . . . . . . . .
Optimal trajectory when xT < xs . . . . . . .
Food output function . . . . . . . . . . . . . . .
Phase diagram for the pollution control model .
Violation of the monotonicity constraint . . . .
Bunching and ironing . . . . . . . . . . . . . .

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.

.

.
.
.
.
.
.
.

320
322
323
324
326
329
330

.
.
.
.
.
.
.

340
346
347
348

351
358
359

12.1 A sample path of optimal production rate It∗ with
I0 = x0 > 0 and B > 0 . . . . . . . . . . . . . . . . . . .

374

13.1 A sample path of optimal market share trajectories . . .
13.2 Optimal subsidy rate vs. (a) Retailer’s margin and (b)
Manufacturer’s margin . . . . . . . . . . . . . . . . . . .

396

B.1 Examples of admissible functions for the problem . . . .
B.2 Variation about the solution function . . . . . . . . . . .
B.3 A broken extremal with corner at τ . . . . . . . . . . . .

420
421
428

404


xxvi

LIST OF FIGURES

C.1 Needle-shaped variation . . . . . . . . . . . . . . . . . .
C.2 Trajectories x∗ (t) and x(t) in a one-dimensional case . .

434
434

D.1 Phase diagram for system (D.73) . . . . . . . . . . . . .
D.2 Region D with boundaries Γ1 and Γ2 . . . . . . . . . . .

457
461


List of Tables
1.1
1.2
1.3

The production-inventory model of Example 1.1 . . . .
The advertising model of Example 1.2 . . . . . . . . . .
The consumption model of Example 1.3 . . . . . . . . .

3.1
3.2
3.3

Summary of the transversality conditions
State trajectories and switching curves . .
Objective, state, and adjoint equations for
types . . . . . . . . . . . . . . . . . . . . .

4
6
8

. . . . . . . .
. . . . . . . .
various model
. . . . . . . .

89
100

Characterization of optimal controls with c < 1 . . . . .

168

13.1 Optimal feedback Stackelberg solution . . . . . . . . . .

403

A.1 Homogeneous solution forms for Eq. (A.5) . . . . . . . .
A.2 Particular solutions for Eq. (A.5) . . . . . . . . . . . . .

411
411

5.1

111

xxvii


Chapter 1

What Is Optimal Control
Theory?
Many management science applications involve the control of dynamic
systems, i.e., systems that evolve over time. They are called continuoustime systems or discrete-time systems depending on whether time varies
continuously or discretely. We will deal with both kinds of systems in this
book, although the main emphasis will be on continuous-time systems.
Optimal control theory is a branch of mathematics developed to find
optimal ways to control a dynamic system. The purpose of this book is
to give an elementary introduction to the mathematical theory, and then
apply it to a wide variety of different situations arising in management
science. We have deliberately kept the level of mathematics as simple as
possible in order to make the book accessible to a large audience. The
only mathematical requirements for this book are elementary calculus,
including partial differentiation, some knowledge of vectors and matrices, and elementary ordinary and partial differential equations. The last
topic is briefly covered in Appendix A. Chapter 12 on stochastic optimal control also requires some concepts in stochastic calculus, which are
introduced at the beginning of that chapter.
The principle management science applications discussed in this book
come from the following areas: finance, economics, production and inventory, marketing, maintenance and replacement, and the consumption
of natural resources. In each major area we have formulated one or more
simple models followed by a more complicated model. The reader may

© Springer Nature Switzerland AG 2019
S. P. Sethi, Optimal Control Theory,
1

1