ELEMENTS OF DYNAMIC OPTIMIZATION Alpha C. Chiang Professor of Economics The University of Connecticut New York Caracas New Delhi Lisbon Paris St. Louis London San Juan McGraw-Hill, Inc. San Francisco Madrid Singapore Mexico Auckland Sydney · Milan Tokyo Bogota Montreal Toronto ELEMENTS O F DYNAMIC OPTIMIZATION International Editions 1992 Exclusive rights by McGraw-Hill Book Co-Singapore for manufacture and export. This book cannot be re-exported from the country to which it is consigned by McGraw-Hill. Copyright © 1992 by McGraw-Hill, Inc. All rights reserved. Except as permitted under the United States Copyright Act of 1976, no part of this publication may be reproduced or distributed in any form or by any means, or stored in a data base or retrieval system, without the prior written permission of the publisher. 5 6 7 8 9 0 KHL SW 9 6 5 Library of Congress Cataloging-in-Publicaton Data Chiang, Alpha C., (date). Elements of dynamic optimization p. em. I Alpha C. Chiang. Includes bibliographical references and index. ISBN 0-07-010911-7 1. Mathematical optimization. I. Title. HB143.7.C45 2. Economics, Mathematical. 1992 330'.01'51'- dc20 91-37803 This book was set in Century Schoolbook by Science Typographers, Inc. The editor was Scott D. Stratford; the production supervisor was Denise L. Puryear. The cover was designed by John Hite. Project supervision was done by Science Typographers, Inc. Art was prepared electronically by Science Typographers, Inc. When ordering this title, use ISBN 0-07-112568-X. Printed in Singapore. CONTENTS Preface Part 1 1 1.1 1.2 1.3 1.4 Part 2 2 2.1 2.2 2.3 2.4 2.5 3 3.1 3.2 3.3 3.4 Introduction The Nature of Dynamic Optimization xi 1 3 Salient Features of Dynamic Optimization Problems Variable Endpoints and Transversality Conditions The Objective Functional Alternative Approaches to Dynamic Optimization 4 8 12 17 The Calculus of Variations 25 The Fundamental Problem of the Calculus of Variations 27 The Euler Equation Some Special Cases Two Generalizations of the Euler Equation Dynamic Optimization of a Monopolist Trading Off Inflation and Unemployment 28 37 45 49 54 Transversality Conditions for Variable-Endpoint Problems 59 The General Transversality Condition Specialized Transversality Conditions Three Generalizations The Optimal Adjustment of Labor Demand 59 63 72 75 vii viH CONTENTS 4 4.1 4.2 4.3 4.4 5 5.1 5.2 5.3 5.4 5.5 6 6.1 6.2 6.3 Part 3 7 Second-Order Conditions 79 Second-Order Conditions First and Second Variations 79 81 91 95 Infinite Planning Horizon 98 The Concavity ;convexity Sufficient Condition The Legendre Necessary Condition Methodological Issues of Infinite Horizon The Optimal Investment Path of a Firm The Optimal Social Saving Behavior Phase-Diagram Analysis The Concavity /Convexity Sufficient Condition Again 98 103 111 117 130 Constrained Problems 133 Four Basic Types of Constraints 134 144 148 Some Economic Applications Reformulated The Economics of Exhaustible Resources Optimal Control Theory 159 Optimal Control: The Maximum Principle 161 7.1 7.2 7.3 7.4 7.5 The Simplest Problem of Optimal Control 162 167 177 181 7.6 7.7 The Political Business Cycle 8 8.1 8.2 8.3 8.4 8.5 9 9.1 9.2 9.3 9.4 10 10.1 The Maximum Principle The Rationale of the Maximum Principle Alternative Terminal Conditions The Calculus of Variations and Optimal Control Theory Compared Energy Use and Environmental Quality 191 193 200 More on Optimal Control 205 An Economic Interpretation of the Maximum Principle Problems with Several State and Control Variables Antipollution Policy 205 210 214 221 234 Infinite-Horizon Problems 240 Transversality Conditions 240 244 253 264 The Current-Value Hamiltonian Sufficient Conditions Some Counterexamples Reexamined The Neoclassical Theory of Optimal Growth Exogenous and Endogenous Technological Progress Optimal Control with Constraints 275 Constraints Involving Control Variables 275 '· PRJ:P'A.CE 10.2 10.3 10.4 10.5 The Dynamics of a Revenue-Maximizing Firm State-Space Constraints Economic Examples of State-Space Constraints Limitations of Dynamic Optimization Answers to Selected Exercise Problems Index ix 292 298 307 313 315 323 PREFAC E In recent years I have received many requests to expand my Fundamental Methods of Mathematical Economics to include the subject of dynamic optimization. Since the existing size of that book would impose a severe space constraint, I decided to present the topic of dynamic optimization in a separate volume. Separateness notwithstanding, the present volume can be considered as a continuation of Fundamental Methods of Mathematical Economics. As the title Elements of Dynamic Optimization implies, this book is intended as an introductory text rather than an encyclopedic tome. While the basics of the classical calculus of variations and its modern cousin, optimal control theory, are explained thoroughly, differential games and stochastic control are not included. Dynamic programming is explained in the discrete-time form; I exclude the continuous-time version because it needs partial differential equations as a prerequisite, which would have taken us far afield. Although the advent of optimal control theory has caused the calculus of variations to be overshadowed, I deem it inadvisable to dismiss the topic of variational calculus. For one thing, a knowledge of the latter is indispens­ able for understanding many classic economic papers written in the calcu­ lus-of-variations mold. Besides, the method is used even in recent writings. Finally, a background in the calculus of variations facilitates a better and fuller understanding of optimal control theory. The reader who is only interested in optimal control theory may, if desired, skip Part 2 of the present volume. But I would strongly recommend reading at least the following: Chap. 2 (the Euler equation), Sec. 4.2 (checking concavity1 con­ vexity), and Sec. 5.1 (methodological issues of infinite horizon, relevant also to optimal control theory). Certain features of this book are worth pointing out. In developing the Euler equation, I supply more details than most other books in order that the reader can better appreciate the beauty of the logic involved (Sec. 2.1). xi xii PREFACE In connection with infinite-horizon problems, I attempt to clarify some common misconceptions about the conditions for convergence of improper integrals (Sec. 5.1). I also try to argue that the alleged counterexamples in optimal control theory against infinite-horizon transversality conditions may be specious, since they involve a failure to recognize the presence of implicit fixed terminal states in those examples (Sec. 9.2). To maintain a sense of continuity with Fundamental Methods of Mathematical Economics, I have written this volume with a comparable level of expository patience, and, I hope, clarity and readability. The dis­ cussion of mathematical techniques is always reinforced with numerical illustrations, economic examples, and exercise problems. In the numerical illustrations, I purposely present the shortest-distance problem-a simple problem with a well-known solution-in several different alternative formu­ lations, and use it as a running thread through the book. In the choice of economic examples, my major criterion is the suitabil­ ity of the economic models as illustrations of the particular mathematical techniques under study. Although recent economic applications are natural candidates for inclusion, I have not shied away from classic articles. Some classic articles are not only worth studying in their own right, but also turn out to be excellent for illustrative purposes because their model structures are uncluttered with secondary complicating assumptions. As a by-product, the juxtaposition of old and new economic models also provides an interest­ ing glimpse of the development of economic thought. For instance, from the classic Ramsey growth model (Sec. 5.3) through the neoclassical growth model of Cass (Sec. 9.3) to the Romer growth model with endogenous technological progress (Sec. 9.4), one sees a progressive refinement in the analytical framework. Similarly, from the classic Hotelling model of ex­ haustible resources (Sec. 6.3) to the Forster models of energy use and pollution (Sec. 7. 7 and Sec. 8.5), one sees the shift in the focus of societal concerns from resource-exhaustion to environmental quality. A comparison of the classic Evans model of dynamic monopolist (Sec. 2.4) with the more recent model of Leland on the dynamics of a revenue-maximizing firm (Sec. 10.2) also illustrates one of the many developments in microeconomic reorientation. In line with my pedagogical philosophy, I attempt to explain each economic model in a step-by-step manner from its initial construction through the intricacies of mathematical analysis to its final solution. Even though the resulting lengthier treatment requires limiting the number of economic models pre<;ented, I believe that the detailed guidance is desirable because it serves to minimize the trepidation and frustration often associ­ ated with the learning of mathematics. In the writing of this book, I have benefited immensely from the numerous comments and suggestions of Professor Bruce A. Forster of the University of Wyoming, whose keen eyes caught many sins of commission and omission in the original manuscript. Since I did not accept all his i· ' PREFACE xiii suggestions, however, I alone should be held responsible for the remaining imperfections. Many of my students over the years, on whom I tried the earlier drafts of this book, also helped me with their questions and reac­ tions. Scott D. Stratford, my editor, exerted the right amount of encourage­ ment and pressure at critical moments to keep me going. And the coopera­ tive efforts of Joseph Murphy at McGraw-Hill, and Sarah Roesser, Cheryl Kranz, and Ellie Simon at Science Typographers, Inc., made the production process smooth as well as pleasant. Thanks are also due to Edward T. Dowling for ferreting out some typographical errors that lurked in the initial printing of the book. Finally, my wife Emily again offered me unstinting assistance on manuscript preparation. To all of them, I am deeply grateful. Alpha C. Chiang ELEMENTS OF DYNAMIC OPTIMIZATION PART 1 INTRODUCTION CHAPTER 1 THE NATURE OF DYNAMIC OPTIMIZATION Optimization i s a predominant theme i n economic analysis. For this reason, the classical calculus methods of finding free and constrained extrema and the more recent techniques of mathematical programming occupy an impor­ tant place in the economist's everyday tool kit. Useful as they are, such tools are applicable only to static optimization problems. The solution sought in single such problems usually consists of a optimal magnitude for every choice variable, such as the optimal level of output per week and the optimal price to charge for a product. It does not call for a schedule of optimal sequential action. In contrast, a optimization problem poses the question of dynamic what is the optimal magnitude of a choice variable in each period of time within the planning period (discrete-time case) or at each point of time in a T] (continuous-time case). It is even possible to given time interval, say [0, consider an infinite planning horizon, so that the relevant time interval is oo)-literally "from here to eternity." The solution of a dynamic opti­ [0, mization problem would thus take the form of an optimal time path for every choice variable, detailing the best value of the variable today, tomor­ row, and so forth, till the end of the planning period. Throughout this book, we shall use the asterisk to denote optimality. In particular, the optimal time path of a (continuous-time) variable y will I be denoted by y*(t). 3 4 PART 1: INTRODUCTION 1.1 SALIENT FEATURES OF DYNAMIC OPTIMIZATION PROBLEMS Although dynamic optimization is mostly couched in terms of a sequence of time, it is also possible to envisage the planning horizon as a sequence of stages in an economic process. In that case, dynamic optimization can be viewed as a problem of multistage decision making. The distinguishing feature, however, remains the fact that the optimal solution would involve more than one single value for the choice variable. Multistage Decision Making _ The multistage character of dynamic optimization can be illustrated with a simple discrete example. Suppose that a firm engages in transforming a certain substance from an i nitial state A (raw material state) into a termi­ nal state Z (finished product state) through a five-stage production process. In every stage, the firm faces the problem of choosing among several possible alternative subprocesses, each entailing a specific cost. The ques­ tion is: How should the firm select the sequence of subprocesses through the five stages in order to minimize the total cost? In Fig. 1.1, we illustrate such a problem by plotting the stages horizontally and the states vertically. The initial state A is shown by the leftmost point (at the beginning of state 1); the terminal state Z is shown by the rightmost point (at the end of stage 5). The other points B, C, . . . , K show the various intermediate states into which the substance may be transformed during the process. These points (A, B, ... , Z) are referred to as vertic es. To indicate the possibility of transformation from state A to state B, we draw an arc from point A to point B. The other arc AC shows !� -· State 0 '-----+----+--1- Stage 1 2 3 4 5 �'-y---'-y----1 ----J'-y-----J Stage 1 FIGURE 1.1 Stage 2 Stage 3 Stage 4 Stage 5 5 CHAPTER 1: THE NATURE OF DYNAMIC OPTIMIZATION that the substance can also be transformed into state C instead of state B. Each arc is assigned a specific value-in the present example, a cost-shown in a circle in Fig. 1.1. The first-stage decision is whether to transform the raw material into state B (at a cost $2) or into state C (at a cost of $4), that is, whether to choose arc AB or arc AC. Once the decision is made, there will arise another problem of choice in stage 2, and so forth, till state Z is reached. Our problem is to choose a connected sequence of arcs going from left to right, starting at A and terminating at Z, such that the sum of the values of the component arcs is minimized. Such a sequence of arcs will constitute an optimal path. The example in Fig. 1.1 is simple enough so that a solution may be found by enumerating all the admissible paths from A to Z and picking the one with the least total arc values. For more complicated problems, how­ ever, a systematic method of attack is needed. This we shall discuss later when we introduce dynamic programming in Sect. 1.4. For the time being, let us just note that the optimal solution for the present example is the path ACEHJZ, with $14 as the minimum cost of production. This solution serves to point out a very important fact: A myopic, one-stage-at-a-time optimiza­ tion procedure will not in general yield the optimal path! For example, a myopic decision maker would have chosen arc AB over arc AC in the first stage, because the former involves only half the cost of the latter; yet, over the span of five stages, the more costly first-stage arc AC should be selected instead. It is precisely for this reason, of course, that a method that can take into account the entire planning period needs to be developed. The Continuous-Variable Version The example in Fig. 1. 1 is characterized by a discrete stage variable, which takes only integer values. Also, the state variable is assumed to take values belonging to a small finite set, {A, B, . . , Z). If these variables are continu. State z I I I I A I I I I I o�------_,--8� T FIGURE 1.2 PART 1: INTRODUCTION 6 ous, we may instead have a situation as depicted in Fig. 1.2, where, for illustration, we have drawn only five possible paths from A to Z. Each possible path is now seen to travel through an infinite number of stages in the interval [0,T]. There is also an infinite number of states on each path, each state being the result of a particular choice made in a specific stage. For concreteness, let us visualize Fig. 1.2 to be a map of an open terrain, with the stage variable representing the longitude, and the state variable representing the latitude. Our assigned task is to transport a load of cargo from location A to location Z at minimum cost by selecting an appropriate travel path. The cost associated with each possible path de­ pends, in general, not only on the distance traveled, but also on the topography on that path. However, in the special case where the terrain is completely homogeneous, so that the transport cost per mile is a constant, the least-cost problem will simply reduce to a shortest-distance problem. The solution in that case is a straight-line path, because such a path entails the lowest total cost (has the lowest path value). The straight-line solution is, of course, well known-so much so that one usually accepts it without demanding to see a proof of it. In the next chapter (Sec. 2.2, Example 4), we shall prove this result by using the calculus of variations, the classical approach to the continuous version of dynamic optimization. For most of the problems discussed in the following, the stage variable will represent time; then the curves in Fig. 1.2 will depict time paths. As a concrete example, consider a firm with an initial capital stock equal to A at time 0, and a predetermined target capital stock equal to Z at timeT. Many alternative investment plans over the time interval [0,T] are capable of achieving the target capital at time T. And each investment plan implies a specific capital path and entails a specific potential profit for the firm. In this case, we can interpret the curves in Fig. 1.2 as possible capital paths and their path values as the corresponding profits. The problem of the firm is to identify the investment plan-hence the capital path-that yields the maximum potential profit. The solution of the problem will, of course, depend crucially on how the potential profit is related to and determined by the configuration of the capital path. From the preceding discussion, it should be clear that, regardless of whether the variables are discrete or continuous, a simple type of dynamic optimization problem would contain the following basic ingredients: 1 initial point and a given terminal point; 2 a set of admissible paths from the initial point to the terminal point; 3 a set of path values serving as performance indices (cost, profit, etc.) 4 a given associated with the various paths; and a specified objective-either to maximize or to minimize the path value or performance index by choosing the optimal path . CHAPTER 1, THE NATURE OF DYNAMIC OPTIMIZATION 7 The Concept of a Functional The relationship between paths and path values deserves our close atten­ tion, for it represents a special sort of mapping-not a mapping from real numbers to real numbers as in the usual fu nction, but a mapping from paths (curves) to real numbers (performance indices). Let us think of the paths in question as time paths, and denote them by y 1(t), Y n(t), and so on. Then the mapping is as shown in Fig. 1.3, where V1, Vn represent the associated path values. The general notation for the mapping should there­ fore be V(y(t)]. But it must be emphasized that this symbol fundamentally differs from the composite-function symbol g( f(x )]. In the latter, g is a function of f, and f is in turn a function of x; thus, g is in the final analysis a function of x. In the symbol V(y(t)), on the other hand, the y(t) component comes as an integral unit-to indicate time paths-and there­ fore we should not take V to be a function of t. Instead, V should be understood to be a function of "y(t)" as such. To make clear this difference, this type of mapping is given a distinct name: fu nctio nal . To further avoid confusion, many writers omit the "(t)" part of the symbol, and write the functional a s V[y] or V{y), thereby underscoring the fact that it is the change in the position of the entire y path-the variatio n in they path-as against the change in t, that results in a change in path value V. The symbol we employ is V[y ]. Note that when the symbol y is used to indicate a certain state, it is suffixed, and appears as, say, y(O) for the initial state or y(T ) for the terminal state. In contrast, in the path connotation, the t in y(t) is not assigned a specific value. In the following, when we want to stress the specific time interval involved in a Set of admissible paths (curves) Set of path values (real line) FIGURE 1.8 . PART 1: 8 $ $ MC INTRODUCTION MC PI t-------J'--.!"'RI P2._------�'--t- MR2 1 I I I I I I _._.-4�>-Q Q'2 Q\ 0 _______ 0 __ I .____._! ...._ Q _ _ (b) (a) FIGURE I I I I 1.4 path or a segment thereof, we shall use the notation y[O, T] or y[O, T ). More often, however, we shall simply use the y(t) symbol, or use the term "y path". The optimal time path is then denoted by y*(t), or they* path. As an aside, we may note that although the concept of a functional takes a prominent place primarily in dynamic optimization, we can find examples of it even in elementary economics. In the economics of the firm, the profit-maximizing output is found by the decision rule MC = MR (marginal cost = marginal revenue). Under purely competitive conditions, where the MR curves are horizontal as in Fig. 1.4a, each MR curve can be represented by a specific, exogenously determined price, P0• Given the MC curve, we can therefore express the optimal output as Q* = Q*(P0), which is a function, mapping a real number (price) into a real number (optimal output). But when the MR curves are downward-sloping under imperfect competition, the optimal output of the firm with a given MC curve will depend on the specific position of the MR curve. In such a case, since the output decision involves a mapping from curves to real numbers, we in fact have something in the nature of a functional: Q* Q*[MR]. It is, of course, precisely because of this inability to express the optimal output as a function of price that makes it impossible to draw a supply curve for a firm under imperfect competition, as we can for its competitive counterpart. = 1 .2 VARIABLE ENDPOINTS AND TRANSVERSALITY CONDITIONS In our earlier statement of the problem of dynamic optimization, we simpli­ fied matters by assuming a given initial point (a given ordered pair (0, A)} and a given terminal point [a given ordered pair (T, Z)]. The assumption of CHAPTER 1: THE NATURE OF DYNAMIC OPTIMIZATION 9 a given initial point may not be unduly restrictive, because, in the usual problem, the optimizing plan must start from some specific initial position, say, the current position. For this reason, we shall retain this assumption throughout most of the book. The terminal position, on the other hand, may very well turn out to be a flexible matter, with no inherent need for it to be predetermined. We may, for instance, face only a fixed terminal time, but have complete freedom to choose the terminal state (say, the terminal capital stock). On the other hand, we may also be assigned a rigidly specified terminal state (say, a target inflation rate), but are free to select the terminal time (when to achieve the target). In such a case, the terminal point becomes a part of the optimal choice. In this section we shall briefly discuss some basic types of variable terminal points. We shall take the stage variable to be continuous time. We shall also retain the symbols 0 andT for the initial tim e and terminal tim e, and the symbols A and Z for the initial and terminal states. When no confusion can arise, we may also use A and Z to designate the initial and terminal points (ordered pairs), especially in diagrams. Types of Variable Terminal Points As the first type of variable terminal point, we may be given a fixed terminal timeT, but a free terminal state. In Fig. 1.5a, while the planning horizon is fixed at time T, any point on the vertical line t T is acceptable as a terminal point, such as Z1, Z2, and Z3• In such a problem, the planner obviously enjoys much greater freedom in the choice of the optimal path and, as a consequence, will be able to achieve a better-or at least no worse -optimal path value, V*, than if the terminal point is rigidly specified. This type of problem is commonly referred to in the literature as a fixed-time-horizon p robl em, or fixed-tim e probl em, meaning that the termi­ nal time of the problem is fixed rather than free. Though explicit about the time horizon, this name fails to give a complete description of the problem, since nothing is said about the terminal state. Only by implication are we to understand that the terminal state is free. A more informative characteriza­ tion of the problem is contained in the visual image in Fig. 1.5a . In line with this visual image, we shall alternatively refer to the fixed-time problem as the vertical-terminal -line probl em . T o give an economic example of such a problem, suppose that a monopolistic firm is seeking to establish a (smooth) optimal price path over a given planning period, say, 12 months, for purpose of profit maximization. The current price enters into the problem as the initial state. If there is no legal price restriction in force, the terminal price will be completely up to the firm to decide. Since negative prices are inadmissible, however, we must eliminate from consideration all P < 0. The result is a truncated v ertical term inal lin e, which is what Fig. 1.5a in fact shows. If, in addition, an = 10 PART 1: INTRODUCTION y y t=T T (a) (b) y Z=,P(T) (c) FIGURE 1.5 official price ceiling is expected to be in force at the terminal time t = T, then further truncation of the vertical terminal line is needed. The second type of variable-terminal-point problem reverses the roles played by the terminal time and terminal state; now the terminal state Z is stipulated, but the terminal time is free. In Fig. 1.5b, the horizontal line y = Z constitutes the set of admissible terminal points. Each of these, depending on the path chosen, may be associated with a different terminal time, as exemplified by T1, T 2, and T3• Again, there is greater freedom of choice as compared with the case of a fixed terminal point. The task of the planner might, for instance, be that of producing a good with a particular quality characteristic (steel with given tensile strength) at minimum cost, but there is complete discretion over the length of the production period. This permits the rise of a lengthier, but less expensive production method that might not be feasible under a rushed production schedule. CHAPTER 1: THE NATURE OF DYNAMIC OPTIMIZATION 11 This type of problem i s commonly referred t o as a fixed-endpoint A possible confusion can arise from this name because the word "endpoint" is used here to designate only the terminal state Z, not the entire endpoint in the sense of the ordered pair (T, Z). To take advantage of the visual image of the problem in Fig. 1.5b, we shall alternatively refer to the fixed-endpoint problem as the horizontal-terminal-line problem . Turning the problem around, it is also possible in a problem of this type to have minimum production time (rather than minimum cost) as the objective. In that case, the path with T1 as the terminal time becomes preferable to the one ending at T2, regardless of the relative cost entailed. This latter type of problem is called a time-optimal problem. In the third type of variable-terminal-point problem, neither the termi­ nal time T nor the terminal state Z is individually preset, but the two are tied together via a constraint equation of the form Z = c/J(T). As illustrated in Fig. 1.5c, such an equation plots as a terminal curve (or, in higher dimension, a terminal surface) that associates a particular terminal time (say, T1) with a corresponding terminal state (say, Z1). Even though the problem leaves both T and Z flexible, the planner actually has only one degree of freedom in the choice of the terminal point. Still, the field of choice is obviously again wider than if the terminal point is fully prescribed. We shall call this type of problem the terminal-curve (or terminal-surface) problem . problem . To give an economic example of a terminal curve, we may cite the case of a custom order for a product, for which the customer is interested in having both ( 1) an early date of completion, and (2) a particular quality characteristic, say, low breakability. Being aware that both cannot be attained simultaneously, the customer accepts a tradeoff between the two considerations. Such a tradeoff may appear as the curve Z = c/J(T) in Fig. 1.5c, where y denotes breakability. The preceding discussion pertains to problems with finite planning horizon, where the terminal time T is a finite number. Later we shall also encounter problems where the planning horizon is infinite (T--+ oo) . Transversality Condition The common feature of variable-terminal-point problems is that the planner has one more degree of freedom than in the fixed-terminal-point case. But this fact automatically implies that, in deriving the optimal solution, an extra condition is needed to pinpoint the exact path chosen. To make this clear, let us compare the boundary conditions for the optimal path in the fixed- versus the variable-terminal-point cases. In the former, the optimal path must satisfy the boundary (initial and terminal) conditions y(O) =A and y(T) = Z In the latter case, the initial condition ( T, A, and Z all given) y(O) =A still applies by assumption. 12 PART 1 , INTRODUCTION But since T andjor Z are now variable, the terminal condition y(T) = Z is no longer capable of pinpointing the optimal path for us. As Fig. 1.5 shows, all admissible paths, ending at Z1, Z2, or other possible terminal positions, equally satisfy the condition y(T) Z. What is needed, therefore, is a terminal condition that can_conclusively distinguish the optimal path from the other admissible paths. Such a condition is referred to as a transversal­ ity condition, because it normally appears as a description of how the optimal path crosses the terminal line or the terminal curve (to "transverse" means to "to go across"). = Variable Initial Point Although we have assumed that only the terminal point can vary, the discussion in this section can be adapted to a variable initial point as well. Thus, there may be an initial curve depicting admissible combinations of the initial time and the initial state. Or there may be a vertical initial line t = 0, indicating that initial time 0 is given, but the initial state is unrestricted. As an exercise, the reader is asked to sketch suitable diagrams silnilar to those in Fig. 1.5 for the case of a variable initial point. If the initial point is variable, the characterization of the optimal path must also include another transversality condition in place of the equation y(O) = A, to describe how the optimal path crosses the initial line or initial curve. EXERCISE 1.2 1 2 Sketch suitable diagrams similar to Fig. 1.5 for the case of a variable initial point. In Fig. 1.5a, suppose that y represents the price variable. How would an official price ceiling that is expected to take effect at time t = T affect the diagram? 3 In Fig. 1.5c, let y denote heat resistance in a product, a quality which takes longer production time to improve. How would you redraw the terminal curve to depict the tradeoff between early completion date and high heat resistance? 1.3 THE OBJECTIVE FUNCTIONAL The Integral Form of Functional An optimal path is, by definition, one that maximizes or minimizes the path value V[y ]. Inasmuch as any y path must perforce travel through an interval of time, its total value would naturally be a sum. In the discrete- CHAPTER 1: THE NATURE OF DYNAMIC OPTIMIZATION 13 stage framework of Fig. 1.1, the path value is the sum of the values of its component arcs. The continuous-time counterpart of such a sum is a definite integral, J;{ (arc value) dt. But how do we express the "arc value" for the continuous case? To answer this, we must first be able to identify an "arc" on a continuous-time path. Figure 1.1 suggests that three pieces of information are needed for arc identification: (1) the starting stage (time), (2) the starting state, and (3) the direction in which the arc proceeds. With continu­ ous time, since each arc is infinitesimal in length, these three items are represented by, respectively: (1) t, (2) y(t), and (3) y' (t) = dy jdt. For instance, on a given path y1, the arc associated with a specific point of time t0 is characterized by a unique value y1(t0) and a unique slope y{(t0). If there exists some function, F, that assigns arc values to arcs, then the value of the said arc can be written as F[t0, y1(t0), y {( t0 )] . Similarly, on another path, y11, the height and the slope of the curve at t = t0 are y11(t0) and yJ(t0), respectively, and the arc value is F[t0, y11(t0), y1{(t0)]. It follows that the general expression for arc values is F[t, y(t), y'(t)}, and the path-value functional-the sum of arc values-can generally be written as the definite integral (1.1) V[yJ = f TF [ t , y ( t ) , y'(t) ] dt 0 It bears repeating that, as the symbol V[y] emphasizes, it is the variation in they path (y1 versus y11) that alters the magnitude of V. Each different y path consists of a different set of arcs in the time interval [0, T], which, through the arc-value-assigning function F, takes a different set of arc values. The definite integral sums those arc values on each y path into a path value. If there ¥e two state variables, y and z, in the problem, the arc values on both the y and z paths must be taken into account. The objective functional should then appear as ( 1 .2) V[y, z ] = foTF [ t , y ( t ) , z(t),y'( t ) , z' ( t ) ] dt A problem with an objective functional in the form of (1.1) or ( 1.2) constitutes the standard problem. For simplicity, we shall often suppress the time argument (t) for the state variables and write the integrand function more concisely as F(t, y, y') or F(t, y, z, y', z'). A Microeconomic Example (1.1) may arise, for instance, in the case of a profit-maximizing, long-range-planning monopolistic firm with a dynamic demand function Qd = D(P, P'), where P' = dPjdt. In order to A functional of the standard form in 14 PART 1: INTRODUCTION set Q, = Qd (to allow no inventory accumulation or decumulation), the firm's output should be Q = D(P, P'), so that its total-revenue function is R =PQ =R(P,P') Assuming that the total-cost function depends only on the level of output, we can write the composite function C = C(Q) = C[D(P, P')] It follows that the total profit also depends on P and P': 7r = R - C Summing functional 7r = R(P, P')- C[D(P, P')] = 7r(P, P') over, say, a five-year period, results in the objective t71'(P, P') dt 0 t which conforms to the general form of (1.1), except that the argument in the F function happens to be absent. However, if either the revenue function or the cost function can shift over time, then that function should contain t as a separate argument; in that case the 71' function would also have t as an argument. Then the corresponding objective functional t7T (t, P, P') dt 0 would be exactly in the form of (1.1). As another possibility, the variable t can enter into the integrand via a discount factor e-pt. To each price path in the time interval [0, 5), there must correspond a particular five-year profit figure, and the objective of the firm is to find the optimal price path P*[O, 5] that maximizes the five-year profit figure. A Macroeconomic Example Let the social welfare of an economy at any time be measured by the utility from consumption, U = U(C). Consumption is by definition that portion of output not saved (and not invested). If we adopt the production function Q = Q(K, L), and assume away depreciation, we can then write C = Q( K, L) - I = Q(K, L) - K' where K' = I denotes net investment. This implies that the utility function be rewritten as can U(C) = U[Q(K, L)- K'] CHAPTER 1: THE NATURE OF DYNAMIC OPTIMIZATION 15 If the societal goal is to maximize the sum of utility over a period [0, T], then its objective functional takes the form fTU [ Q( K, L) - K') dt 0 This exemplifies the functional in (1.2), where the two state variables y and refer in the present example to K and L, respectively. Note that while the integrand function of this example does contain both K and K' as arguments, the L variable appears only in its natural form unaccompanied by L'. Moreover, the t argument is absent from the F function, too. In terms of (1.2), the F function contains only three argu­ ments in the present example: F[y(t), z(t),y'(t)], or F[K, L, K']. z Other Forms of Functional Occasionally, the optimization criterion in a problem may not depend on any intermediate positions that the path goes through, but may rely exclu­ sively on the position of the terminal point attained. In that event, no definite integral arises, since there is no need to sum the arc values over an interval. Rather, the objective functional appears as V [y) (1.3) = G[T, y( T ) ] where the G function is based on what happens at the terminal time T only. It may also happen that both the definite integral in (1.1) and the terminal-point criterion in (1.3) enter simultaneously in the objective func­ tional. Then we have (1.4) V[y] = f0TF(t ,y(t) ,y' ( t) ] dt + G (T,y( T ) ] where the G function may represent, for instance, the scrap value of some capital equipment. Moreover, the functionals in (1.3) and ( 1.4) can again be expanded to include more than one state variable. For example, with two state variables y and z, (1.3) would become (1.5) V[y, z] = G [ T,y( T ) , z(T)] A problem with the type of objective functional i n (1.3) is called a problem of Mayer. Since only the terminal position matters in V, it is also known as a terminal-control problem. If (1.4) is the form of the objective functional, then we have a problem of BoZza. PART 1: lNTRODUCTION 16 Although the problem of Bolza may seem to be the more general formulation, the truth is that the three types of problems-standard, Mayer, and Bolza-are all convertible into one another. For example, the functional (1. 3) can be transformed into the form (1.1) by defining a new variable ( 1 . 6) z( t) = G[t, y( t)] It should be noted that it is function in (1.6). Since with initial condition "t" rather than "T" z( O) = 0 that appears in the G ( 1 .7) (z'( t ) dt =z ( t ) i� = z ( T) - z ( O) 0 = G[T,y(T)] = z ( T) [by the initial condition ] [by ( 1 .6) ] the functional in (1. 3), G[T, y(T)], can be replaced by the integral in (1.7) in the new variable z(t). The integrand, z'(t) = dG[t, y(t)]jdt, is easily recog­ nized as a special case of the function F[t, z(t), z'( t)], with the arguments t and z( t) absent; that is, the integral in (1.7) still falls into the general form of the objective functional (1.1). Thus we have transformed a problem of Mayer into a standard problem. Once we have found the optimal z path, the optimal y path can be deduced through the relationship in (1.6). By a similar procedure, we can convert a problem of Bolza into a standard problem; this will be left to the reader. An economic example of this type of problem can be found in Sec. 3.4. In view of this convertibility, we shall deem it sufficient to couch our discussion primarily in terms of the standard problem, with the objective functional in the form of an integral. EXERCISE 1.3 1 In a so-called "time-optimal problem," the objective is to move the state variable from a given initial value y(O) to a given terminal value y(T) in the least amount of time. In other words, we wish to minimize the functional V[y] = T - 0. (a) Taking it as a standard problem, write the specific form of the F function in (1.1) that will produce the preceding functional. (b) Taking it as a problem of Mayer, write the specific form of the G function in (1.3) that will produce the preceding functional. 2 Suppose we are given a function D(t) which gives the desired level of the state variable at every point of time in the interval [0, T ]. All deviations from D(t), positive or negative, are undesirable, because they inflict a negative payoff (cost, pain, or disappointment). To formulate an appropriate dynamic minimization problem, which of the following objective functionals , QJIAPTBJl l : 17 THE NATURE OF DYNAMIC OPTIMIZATION would be acceptable? Why? (a) f[!y(t) - D(t)) dt (b) /0T[y(t) - D(t))2 dt (c) f[ID(t) - y(t)P dt (d) f:iD(t) - y(t)l dt 3 4 Transform the objective functional (1.4) of the problem of Bolza into the format of the standard problem, as in (L 1) or (1.2). [ Hint: Introduce a new variable z(t) = G[t, y(t)), with initial condition z(O) 0.] = Transform the objective functional (1.4) of the problem of Bolza into the format of the problem of Mayer, as in (1.3) or (1.5). [Hint: Introdu�e a new variable z(t) characterized by z'(t) = F[t, y(t), y'(t)), with initial condition z(O) = 0.) 1.4 ALTERNATIVE APPROACHES TO DYNAMIC OPTIMIZATION To tackle the previously stated problem of dynamic optimization, there are three major approaches. We have earlier mentioned the calculus of varia­ tions and dynamic programming. The remaining one, the powerful modern generalization of variational calculus, goes under the name of optimal control theory. We shall give a brief account of each. ,, The Calculus of Variations Dating back to the late 17th century, the calculus of variations is the classical approach to the problem. One of the earliest problems posed is that of determining the shape of a surface of revolution that would encounter the least resistance when moving through some resisting medium (a surface of revolution with the minimum area).1 Issac Newton solved this problem and stated his results in his Principia, published in 1687. Other mathe­ maticians of that era (e.g., John and James Bernoulli) also studied problems of a similar nature. These problems can be represented by the following general formulation: ( 1.8) = j TF[ t, y( t ) , y' (t)] dt Maximize or minimize VJ:y] subject to y(O) =A and y( T ) 1This problem will be discussed in Sec. 2.2, Example 0 = Z 3. (A given) ( T, Z given) 18 PART [, INTRODUCTION Such a problem, with an integral functional in a single state variable, with completely specified initial and terminal points, and with no constraints, is known as the fundamental problem (or simplest problem) of calculus o£ variations. In order to make such problems meaningful, it is necessary that the functional be integrable (i.e., the integral must be convergent). We shall assume this condition is met whenever we write an integral of the general form, as in (1.8). Furthermore, we shall assume that all the functions that appear in the problem are continuous and continuously differentiable. This assumption is needed because the basic methodology underlying the calcu­ lus of variations closely parallels that of the classical differential calculus. The main difference is that, instead of dealing with the differential dx that changes the value of y = f(x), we will now deal with the " variation" of an entire curve y(t) that affects the value of the functional V[y ]. The study of variational calculus will occupy us in Part 2. Optimal Control Theory · The continued study of variational problems has led to the development of the more modern method of optimal control theory. In optimal control theory, the dynamic optimization problem is viewed as consisting of three (rather than two) types of variables. Aside from the time variable t and the state variable y(t), consideration is given to a control variable u(t). Indeed, it is the latter type of variable that gives optimal control theory its name and occupies the center of stage in this new approach to dynamic optimiza­ tion. To focus attention on the control variable implies that the state variable is relegated to a secondary status. This would be acceptable only if the decision on a control path u(t) will, once given an initial condition on y, unambiguously determine a state-variable path y(t) as a by-product. For this reason, an optimal control problem must contain an equation that relates y to u : dy dt = f [ t , y(t) , u(t)] Such an equation, called an equation of motion (or transition equation or state equation), shows how, at any moment of time, given the value of the state variable, the planner's choice of u will drive the state variable y over time. Once we have found the optimal control-variable path u*(t), the equation of motion would make it possible to construct the related optimal state-variable path y*(t). CHAPTER 1: THE NATURE OF DYNAMIC OPTIMIZATION 19 The optimal control problem corresponding tions problem (1.8) is as follows: Maximize or minimize ( 1 .9) = f [ t, y(t), u ( t)J y(O) = A y( T ) and the calculus-of-varia­ V[ u ] = foTF [ t , y(t), u ( t ) ) dt y'( t) subject to to = Z (A given) (T, Z given) Note that, in (1.9), not only does the objective functional contain u as an argument, but it has also been changed from V[y] to V[ u ]. This reflects the fact that u is now the ultimate instrument of optimization. Nonetheless, this control problem is intimately related to the calculus-of-variations prob­ lem (1.8). In fact, by replacing y'(t) with u(t) in the integrand in (1.8), and adopting the differential equation y'(t) = u(t) as the equation of motion, we immediately obtain ( 1.9). The single most significant development in optimal control theory is known as the maximum principle. This principle is commonly associated with the Russian mathematician L. S. Pontryagin, although an American mathematician, Magnus R. Hestenes, independently produced comparable work in a Rand Corporation report in 1949.2 The powerfulness of that principle lies in its ability to deal directly with certain constraints on the control variable. Specifically, it allows the study of problems where the admissible values of the control variable u are confined to some closed, bounded convex set �. For instance, the set � may be the closed interval [0, 1 ], requiring 0 � u(t) � 1 throughout the planning period. If the marginal propensity to save is the control variable, for instance, then such a con­ straint, 0 � s(t) � 1 , may very well be appropriate. In sum, the problem 2The maximum principle is the product of the joint efforts of L. S. Pontryagin and his associates V. G. Boltyanskii, R. V. Gamkrelidze, and E. F. Mishchenko, who were jointly awarded the 1962 Lenin Prize for Science and Technology. The English translation of their work, done by K. N. Trirogoff, is The Mathematical Theory of Optimal Processes, Interscience, New York, 1962. Hestenes' Rand report is titled A General Problem in the Calculus of Variations with Applications to Paths of Least Time. But his work was not easily available until he published a paper that extends the results of Pontryagin: "On Variational Theory and Optimal Control Theory," Journal of SIAM, Series A, Control , Vol. 3, 1 96 5, pp. 23-48. The expanded version of this work is contained in his book Calculus of Variations and Optimal Control Theory, Wiley, New York, 1966 . Some writers prefer to call the principle the minimum principle, which is the more appropriate name for the principle in a slightly modified formulation of the problem. We shall use the original name, the maximum principle. 20 PART 1: INTRODUCTION addressed by optimal control theory is (in its simple form) the same as in (1.9), except that an additional constraint, u ( t) E � for 0 :::;; t ::5 T may be appended to it. In this light, the control problem (1.9) constitutes a special (unconstrained) case where the control set � is the entire real line. The detailed discussion of optimal control theory will be undertaken in Part 3. Dynamic Programming Pioneered by the American mathematician Richard Bellman,3 dynamic programming presents another approach to the control problem stated in (1.9). The most important distinguishing characteristics of this approach are two: First, it embeds the given control problem in a family of control problems, with the consequence that in solving the given problem, we are actually solving the entire family of problems. Second, for each member of this family of problems, primary attention is focused on the optimal value of the functional, V*, rather than on the properties of the optimal state path y*(t) (as in the calculus of variations) or the optimal control path u*(t) (as in optimal control theory). In fact, an optimal value function-assigning an optimal value to each individual member of this family of problems-is used as a characterization of the solution. All this is best explained with a specific discrete illustration. Referring to Fig. 1.6 (adapted from Fig. 1.1), let us first see how the "embedding" of a problem is done. Given the original problem of finding the least-cost path from point A to point Z, we consider the larger problem of finding the least-cost path from each point in the set {A, B, C, . . . , Z} to the terminal point Z. There then exists a family of component problems, each of which is associated with a different initial point. This is, however, not to be confused with the variable-initial-point problem in which our task is to select one initial point as the best one. Here, we are to consider every possible point as a legitimate initial point in its own right. That is, aside from the genuine initial point A, we have adopted many pseudo initial points ( B, C, etc). The problem involving the pseudo initial point Z is obviously trivial, for it does not permit any real choice or control; it is being included in the general problem for the sake of completeness and symmetry. But the component 3Richard E. Beilman, Dynamic Programming, Princeton University Press, Princeton, NJ, 1957. CHAPTER 1 : THE NATURE OF DYNAMIC OPTIMIZATION 21 State A o ., I I I I I ____ ____ _ f .L__ _ ; 1 '---y---1 Stage .. 2 f. ___ - ---,.-- + 3 + 4 5 '---y---1'-----y-J Stage 4 1 . ,. s� Stage 5 FIGURE 1.6 problems for the other pseudo initial points are not trivial. Our Origi­ nal problem has thus been " embedded" in a family of meaningful problems. Since evecy component problem has a unique optimal path value, it is possible to write an optimal value function V* = V*( i ) ( i = A , B, . . . , Z) which says that we can determine an optimal path value for evecy possible initial point. From this, we can also construct an optimal policy function, which will tell us how best to proceed from any specific initial point i, in order to attain V*(i) by the proper selection of a sequence of arcs leading from point i to the terminal point Z. The purpose of the optimal value function and the optimal policy function is easy to grasp, but one may still wonder why we should go to the trouble of embedding the problem, thereby multiplying the task of solution. The answer is that the embedding process is what leads to the development of a systematic iterative procedure for solving the original problem. Returning to Fig. 1 .6, imagine that our immediate problem is merely that of determining the optimal values for stage 5, associated with the three initial points I, J, and K. The answer is easily seen to be ( 1 .10) V* ( I ) = 3 V*(J) = 1 V* ( K ) =2 Having found the optimal values for I, J, and K, the task of finding the least-cost values V*(G) and V*(H) becomes easier. Moving back to stage 4 and utilizing the previously obtained optimal-value information in (1.10), we can determine V *(G) as well as the optimal path GZ (from G to Z) as PART 1: INTRODUCTION 22 follows: ( 1 . 11) V* ( G ) = min{value of arc GI + V* ( I ) , value of arc GJ = min{ 2 + 3, 8 ·+ 1 } = 5 + V*( J)} [The optimal path GZ is GIZ.] The fact that the optimal path from G to Z should go through I is indicated by the arrow pointing away from G; the numeral on the arrow shows the optimal path value V*(G). By the same token, we find ( 1 . 12) V* ( H ) = min{ value of arc HJ + V* ( J ) , value of arc HK + V* ( K) } = min{4 + 1 , 6 + 2 ) = 5 [The optimal path HZ is HJZ.] Note again the arrow pointing from H toward J and the numeral on it. The set of all such arrows constitutes the optimal policy function, and the set of all the numerals on the arrows constitutes the optimal value function. With the knowledge of V*(G) and V*(H), we can then move back one more stage to calculate V*(D), V*(E), and V*(F )-and the optimal paths DZ, EZ, and FZ-in a similar manner. And, with two more such steps, we will be back to stage 1, where we can determine V*(A) and the optimal path AZ, that is, solve the original given problem. The essence of the iterative solution procedure is captured in Bellman's principle of optimality, which states, roughly, that if you chop off the first arc from an optimal sequence of arcs, the remaining abridged sequence must still be optimal in its own right-as an optimal path from its own initial point to the terminal point. If EHJZ is the optimal path from E to Z, for example, then HJZ must be the optimal path from H to Z. Conversely, if HJZ is already known to be the optimal path from H to Z, then a longer optimal path that passes through H must use the sequence HJZ at the tail end. This reasoning is behind the calculations in ( 1 . 1 1) and (1.12). But note that in order to apply the principle of optimality and the iterative procedure to delineate the optimal path from A to Z, we must find the optimal value associated with every possible point in Fig. 1.6. This explains why we must embed the original problem. Even though the essence of dynamic programming is sufficiently clari­ fied by the discrete example in Fig. 1.6, the full version of dynamic program­ ming includes the continuous-time case. Unfortunately, the solution of continuous-time problems of dynamic programming involves the more ad­ vanced mathematical topic of partial differential equations. Besides, partial differential equations often do not yield analytical solutions. Because of this, we shall not venture further into dynamic programming in this book. The rest of the book will be focused on the methods of the calculus of variations and optimal control theory, both of which only require ordinary differential equations for their solution. CHAPTER 1: THE NATURE OF DYNAMIC OPTIMIZATION 23 EXERCISE 1.4 1 From Fig. 1.6, find V*(D), V*( E), and V*( F). Determine the optimal paths DZ, EZ, and FZ. 2 On the basis of the preceding problem, find V * ( B ) and V*(C). Determine the optimal paths BZ and CZ. 3 Verify the statement in Sec. 1 . 1 that the minimum cost of production for the example in Fig. 1.6 (same as Fig. 1.1) is $14, achieved on the path ACEHJZ. 4 Suppose that the arc values in Fig. 1.6 are profit (rather than cost) figures. For every point i in the set {A, B, . . . , Z}, find (a) (b) the optimal (maximum-profit) value V* ( i ), and the optimal path from i to Z. . 2 PART THE CALCULUS OF VARIATIONS - ·. - ,. . . ' ·: · CHAPTER 2 THE FUNDAMENTAL PROBLEM OF THE CALCULUS OF VARIATIONS We shall begin the study of the calculus of variations with the fundamental problem: ( 2. 1) Maximize or minimize V [y ) subject to y(O) = A and y( T ) = = j0TF[ t, y ( t) , y' ( t ) ) dt Z ( A given) ( T, Z given) The maximization and minimization problems differ from each other in the second-order conditions, but they share the same first-order condition. The task of variational calculus is to select from a set of admissible y paths (or trajectories) the one that yields an extreme value of V[y ]. Since the calculus of variations is based on the classical methods of calculus, requiring the use of first and second derivatives, we shall restrict the set of admissible paths to those continuous curves with continuous derivatives. A smooth y path that yields an extremum (maximum or minimum) of V[y] is called an extremal. We shall also assume that the integrand function F is twice differentiable. 27 PART 2: THE CALCULUS.OF VARIATIONS 28 y FIGURE 2.1 In locating an extremum of V[ y ], one may be thinking of either an absolute (global) extremum or a relative (local) extremum. Since the calcu­ lus of variations is based on classical calculus methods, it can directly deal only with relative extrema. That is, an extremal yields an extreme value of V only in comparison with the immediately " neighboring" y paths. 2.1 THE EULER EQUATION The basic first-order necessary condition in the calculus of variations is the Euler equation. Although it was formulated as early as 1744, it remains the most important result in this branch of mathematics. In view of its impor­ tance and the ingenuity of its approach, it is worthwhile to explain its rationale in some detail. With reference to Fig. 2.1, let the solid path y*(t) be a known extremal. We seek to find some property of the extremal that is absent in the (nonextremal) neighboring paths. Such a property would constitute a necessary condition for an extremal. To do this, we need for comparison purposes a family of neighboring paths which, by specification in (2.1), must pass through the given endpoints (0, A) and (T, Z). A simple way of generating such neighboring paths is by using a perturbing curve, chosen arbitrarily except for the restrictions that it be smooth and pass through the points 0 and T on the horizontal axis in Fig. 2.1, so that ( 2 .2) p( O) = p(T) = 0 We have chosen for illustration one with relatively small p values and small slopes throughout. By adding Ep(t) to y*(t), where E is a small number, and by varying the magnitude of E, we can perturb the y*(t) path, displacing it to various neighboring positions, thereby generating the desired neighbor­ ing paths. The latter paths can be denoted generally as ( 2.3) y(t) = y* (t) + Ep(t) [implying y'(t) = y*' ( t ) + Ep ' (t)] f ,F CHAPTER 2: THE FUNDAMENTAL PROBLEM OF CALCULUS OF VARIATIONS 29 with the property that, as e -> 0, y(t) -> y*(t). To avoid clutter, only one of these neighboring paths has been drawn in Fig. 2.1. The fact that both y*(t) and p(t) are given curves means that each value of e will determine one particular neighboring y path, and hence one particular value of V[y ]. Consequently, instead of considering V as a functional of the y path, we can now consider it as a function of the variable e-V(e). This change in viewpoint enables us to apply the familiar methods of classical calculus to the function V = V(e). Since, by assump­ tion, the curve y*(t)-which is associated with e 0-yields an extreme value of V, we must have = - dV de (2.4) \ .�o = 0 This constitutes a defining property o f the extremal. It follows that dV1 de = 0 is a necessary condition for the extremal. As written, however, condition (2.4) is not operational because it involves the use of the arbitrary variable E as well as the arbitrary perturb­ ing function p(t). What the Euler equation accomplishes is to express this necessary condition in a convenient operational form. To transform (2.4) into an operational form, however, requires a knowledge of how to take the derivatives of a definite integral. Differentiating a Definite Integral Consider the definition integral (2.5) I( x ) = tF( t, x ) dt a where F(t, x ) is assumed to have a continuous derivative Fx(t, x) in the time interval [a, b]. Since any change in x will affect the value of the F function and hence the definite integral, we may view the integral as a function of x-I(x). The effect of a change in x on the integral is given by the derivative formula: ( 2 .6) dl d = X b J Fx(t, x ) dt a [ Leibniz 's rule] In words, to differentiate a definite integral with respect to a variable x which is neither the variable of integration (t) nor a limit of integration (a or b), one can simply differentiate through the integral sign with respect to x. The intuition behind Leibniz's rule can be seen from Fig. 2.2a, where the solid curve represents F(t, x), and the dotted curve represents the displaced position of F(t, x) after the change in x. The vertical distance between the two curves (if the-change is infinitesimal) measures the partial .. . 30 PART 2: THE CALCULUS OF VARIATIONS F F 0 0 b a (a) 1 a (b) F(t, x) b F F(t,x) 0 b a (c) FIGURE 2.2 derivative F/t, x) at each value of t. It follows that the effect of the change in x on the entire integral, dljdx, corresponds to the area between the two curves, or, equivalently, the definite integral of Fx(t, x) over the interval [a, b]. This explains the meaning of (2.6). The value of the definite integral in (2.5) can also be affected by a change in a limit of integration. Defining the integral alternatively to be J( b, a) ( 2 .7) = fF(t, x) dt a we have the following pair of derivative formulas: (2 .8) iJJ (2 .9) iJJ F ( t, x ) i t-b = F( b, x ) ab = aa = - F (t, x ) l t -a = - F( a , x ) In words, the derivative of a definite integral with respect to its upper limit 31 CHAPTER 2 : THE FUNDAMENTAL PROBLEM OF CALCULUS OF VARIATIONS o f integration b is equal to the integrand evaluated a t t = derivative with respect to its lower limit of integration a is the the integrand evaluated at t = a. b; and the negative of In Fig. 2.2b, an increase in b is reflected in a rightward displacement of the right-hand boundary of the area under the curve. When the displace­ ment is infinitesimal, the effect on the definite integral is measured by the value of the F function at the right-hand boundary-F(b, x). This provides the intuition for (2.8). For an increase in the lower limit, on the other hand, the resulting displacement, as illustrated in Fig. 2.2c, ment of the left-hand boundary, which the area under the curve. This is why there is a negative sign reduces in (2.9). is a rightward move­ The preceding derivative formulas can also be used in combination. If, for instance, the definite integral takes the form K( X ) ( 2 . 10) where x = fb(x)F( t, X ) dt a F, but also affects the (2.6) and (2.8), to get the not only enters into the integrand function upper limit of integration, then we can apply both total derivative ( 2.11) dK b(x) - = f Fx ( t, x ) dt + F [ b(x) , x ] b'( x ) dX a The first term on the right, an integral, follows from (2.6); the second ter-m, representing the chain EXAMPLE 1 dK db( x ) · db( x ) � The derivative of rule, EXAMPLE 2 J;e-x dt (2.8). with respect to x is, by Leibniz's Similarly, .!!_ dx EXAMPLE 3 is based on }2f3x1 dt = J2[3 !_x1 dt = £23tx1-1 dt dx To differentiate J:x3t2 dt with respect to x which appears in the upper limit of integration, we need the chain rule as well as result is d dx i2x3t2 dt = [-d i2x l d(2x) 3t2 dt -- = 3(2x } 2 0 d(2x} 0 dx • (2.8). 2 = 24x 2 The PART 2, THE CALCULUS OF VARIA'nONS 32 Development of the Euler Equation For ease of understanding, the Euler equation will be developed in four steps. Step i Let us first express V in terms of E, and take its derivative. Substituting (2.3) into the objective functional in (2.1), we have [ ] . V( e ) = (F t, y* ( t) + ep(t) , y*' (t) + ep' (t) dt ----- .____.... 0 y(t) y'(t) obtain the derivative dVjd e , Leibniz's rule tells u s t o differentiate ( 2.12 ) To through the integral sign: ( 2 .13) [ by ( 2 .3 ) ] Breaking the last integral in (2.13) into two separate integrals, and setting dV1de = 0, we get a more specific form of the necessary condition for an extremal as follows: ( 2 . 1 4) While this form of necessary condition is already free of the arbitrary variable e , the arbitrary perturbing curve p(t ) is still present along with its derivative p ' (t). To make the necessary condition fully operational, we must also eliminate p(t) and p '(t ) . Step ii To that end, we first integrate the second integral in (2.14) by parts, by using the formula: ( 2.15 ) Let f.t-b du = vu ,t-b f.t-b dv t-a V t-a - t-a U v = Fy' and [ u = u(t), v = v(t)} u = p(t). Then we have dv dFy' dv = dt = dt dt dt and du = du dt = p' ( t ) dt dt Substitution of these expressions into (2.15)-with a = 0 and b = T- CHAPTER 2: TQE FUNDAMENTAL PROBLEM OF CALCULUS OF VARIATIONS - 33 gives us ( 2 . 16) since the first term to the right of the first equals sign must vanish under assumption (2.2). Applying (2. 16) to (2.14) and combining the two integrals therein, we obtain another version of the necessary condition for the extremal: ( 2 . 17) Step iii Although p'(t) is no longer present in (2. 17), the arbitrary p(t) still remains. However, precisely because p(t) enters in an arbitrary way, we may conclude that condition (2. 1 7) can be satisfied only if the bracketed expression [ FY - dF .jdt] is made to vanish for every value of t on the Y extremal; otherwise, the integral may not be equal to zero for some admissi­ ble perturbing curve p(t). Consequently, it is a necessary condition for an extremal that ( 2 . 18) Fy - d dt - F., = 0 for all J t E [0, T ] [Euler equation] Note that the Euler equation is completely free of arbitrary expressions, and thus be applied as soon as one is given a differentiable F(t, y, y') function. The Euler equation is sometimes also presented in the form can ( 2 . 18' ) which is the result of integrating (2. 18) with respect to t. Step iv The nature of the Euler equation (2. 18) can be made clearer when we expand the derivative dFy•/dt into a more explicit form. Because F is a function with three arguments (t, y, y'), the partial derivative Fy' should also be a function of the same three arguments. -The-total derivative dFY.jdt -therefore consists of three terms: dFy' dt -- aFy' aFy' dy aFy' dy' + - + -at ay dt ay' dt = F1y + Fyy•Y' ( t ) + Fy•y•Y" ( t) = - - 34 PART 2: THE CALCULUS OF VARIATIONS Substituting this into (2. 18), multiplying through by - 1, and rearranging, we arrive at a more explicit version of the Euler equation: Fy•y•Y"(t) + Fyy'Y' ( t) + Fey' - Fy = 0 ( 2 . 19) for all [ Euler equation ] t E [0, T ] This expanded version reveals that the Euler equation is in general a second-order nonlinear differential equation. Its general solution will thus contain two arbitrary constants. Since our problem in (2.1) comes with two boundary conditions (one initial and one terminal), we should normally possess sufficient information to definitize the two arbitrary constants and obtain the definite solution. EXAMPLE 4 Find the extremal of the functional V(y] = with boundary conditions have the derivatives fo\ 12ty + y'2 ) dt y(O) = 0 and y(2) = 8. FY. = 2y' By (2.19), Since F = 12ty + y'2, we and the Euler equation is 2y" (t) - 12t = 0 which, upon integration, yields or y"(t) = 6t y'(t) = 3t 2 + c1, and [ general solution] To definitize the arbitrary constants c1 and c2 , we first set t = 0 in the general solution to get y(O) = c2 ; from the initial condition, it follows that c2 = 0. Next, setting t = 2 in the general solution, we get y(2) = 8 + 2c1; from the terminal condition, it follows that c1 = 0. The extremal is thus the cubic function y*(t) = t3 EXAMPLE 5 [ definite solution ] Find the extremal of the functional V[y] = with boundary conditions 3t + (y' )ll2 • Thus, ' - 1 /2 Fy' = l( 2 y ) f [ st + (y') 1 12] dt 1 y(1) = 3 and y(5) = 7. ' - /2 Fy'y' = - . ( y' ) 3 Here we have and F= 35 CHAPTER 2 : THE FUNDAMENTAL PROBLEM OF CALCULUS OF VARIATIONS The Euler equation (2.19) now reduces to The only way to satisfy this equation is to have a constant y', in order that y" = 0. Thus, we write y'(t) = c1, which integrates to the solution [general solution 1 To definitize the arbitrary constants c1 and c 2 , we first set t = 1 to find y(1) = c1 + c 2 = 3 (by the initial condition), and then set t = 5 to find y(5) = 5c1 + c2 7 (by the terminal condition). These two equations give us c1 = 1 and c2 = 2. Therefore, the extremal takes the form of the linear = function y* ( t ) EXAMPLE 6 =t+2 [definite solution1 Find the extremal of the functional = t(t + y 2 + 3y') dt with boundary conditions y(O) 0 and y(5) = 3. Since F V[y1 have = 0 = t + y2 + 3y', we and By (2.18), we may write the Euler equation as 2y y* ( t ) =0 = 0, with solution Note, however, that although this solution is consistent with the initial condition y(O) = 0, it violates the terminal condition y(5) 3. Thus, we must conclude that there exists no extremal among the set of continuous curves that we consider to be admissible. This last example is of interest because it serves to illustrate that certain variational problems with given endpoints may not have a solution. More specifically, it calls attention to one of two peculiar results that can arise when the integrand function F is linear in y'. One result, as illus­ trated in Example 6, is that no solution exists. The other possibility, shown in Example 7, is that the Euler equation is an identity, and since it is automatically satisfied, any admissible path is optimal. = EXAMPLE 7 Find the extremal of the functional PART 2o THE CALCULUS OF VARIATIONS, 36 with boundary conditions y(O) = a and y(t) = {3. With F = y', we have and d dt - Fy, = O It follows that the Euler equation (2. 18) is always satisfied. In this example, it is in fact clear from straight integration that V ( y ] = [y(t ) ] � = y(T) - y(O) = {3 - a The value of V depends only on the given terminal and initial states, regardless of the path adjoining the two given endpoints. The reason behind these peculiarities is that when F is linear in y', Fy' is a constant, and Fy'y' = 0, so the first term in the Euler equation (2.19) vanishes. The Euler equation then loses its status as a second-order differ­ ential equation, and will not provide two arbitrary constants in its general solution to enable us to adapt the time path to the given boundary condi­ tions. Consequently, unless the solution path happens to pass through the fixed endpoints by coincidence, it cannot qualify as an extremal. The only circumstance under which a solution can be guaranteed for such a problem (with F linear in y' and with fixed endpoints) is when FY = 0 as well, which, together with the fact that Fy' = constant (implying dFy'jdt = 0), would turn the Euler equation (2.18) into an identity, as in Example 7. EXERCISE 2.1 1 In discussing the differentiation of definite integrals, no mention was made of the derivative with respect to the variable t. Is that a justifiable omission? Find the derivatives of the following definite integrals with respect to 2 I = J.bx4 dt 3 I = J.be -"'' dt 4 I = Jo2xe' dt 5 I = j02"'tex dt 6 V[y 1 Find the extremals, if any, of the following functionals: 7 8 = g(ty + 2y'2) dt, with y(O) = 1 and V[y 1 = gtyy' dt, with y(O) = 0 and y(l) = V[y 1 = j;(2ye ' + y2 + y(l) = 2 1 y'2) dt, with y(O) = 2 and y(2) 9 V[y1 = J;(y2 + t2y') dt, with y(O) = 0 and y(2) = 2 = 2e2 + e-2 x: CHAPTER 2, THE FUNDAMENTAL PROBLEM OF CALCULUS OF VARIATIONS 2.2 37 SOME SPECIAL CASES We have written the objective functional in the general form f;{F(t, y, y') dt, in which the integrand function F has three arguments: t, y, and y'. For some problems, the integrand function may not contain all three argu­ ments. For such special cases, we can derive special versions of the Euler equation which may often (though not always) prove easier to solve. Special case 1: F y, implying that F:t with the solution F( t, y' ) = In this special case, the F function is free of = 0. Hence, the Euler equation reduces to dF:t.jdt = 0, Fy' = constant It may be noted that Example 5 of the preceding section falls under this special case, although at that time we just used the regular Euler equation for its solution. It is easy to verify that the application of (2.20) will indeed lead to the same result. Here is another example of this special case. (2.20) EXAMPLE 1 Find the extremal of the functional with boundary conditions y(O) = y(1) F = ty' + y'2 (2.20) gives us t + 2y (t ) ' = and = constant, or y'( t) = 1. Since Ft' : = t + 2y' - 4t + c1 Upon direct integration, we obtain y* ( t ) = - tt2 + c1t + c2 [general solution] = With the help of the boundary conditions y(O) = y(1) 1, it is easily verified that c1 t and c 1. The extremal is therefore the quadratic path 2 - i't2 + tt + 1 y*(t) [definite solution] = = = Special case II: F = F(y, y' ) Since F is free of t in this case, we have �:t' = 0, so the Euler equation (2.19) simplifies to F:t' t'y"(t) + F:t:t•Y'(t) - F:t : = 0 The solution to this equation is by no means obvious, but it turns out that if we multiply through by y', the left-hand-side expression in the resulting 38 PART z, THE CALCULUS OF VARIATIONS new equation will be exactly the derivative d(y'Fy' - F)jdt, for d d d F(y, y') ) (y'Fy F) = (y'Fy . • dt dt dt = Fy•Y'� + y'(FyyY' + Fh.y") - (Fyy' + Fy·Y" ) y'(Fy•y·Y" + Fyy•Y' - Fy) Consequently, the Euler equation can be written as d(y'Fy' - F)jdt = 0, with the solution y'Fy' - F = constant, or, what amounts to be the same = thing, F - y'Fy' = constant ( 2 .21) This result-the simplified Euler equation already integrated once:-is a first-order differential equation, which may under some circumstances be easier to handle than the original Euler equation (2. 19). Moreover, in analytical (as against computational) applications, (2.21) may yield results that would not be discernible from (2.19), as will be illustrated in Sec. 2.4. EXAMPLE 2 Find the extremal of the functional with boundary conditions y(O) = 1 and y( and direct substitution in 7T /2) = 0. Since Fy' - 2y' = (2.21) gives us y ' 2 + y 2 = constant This last constant is nonnegative because the left-hand-side terms are all squares; thus we can justifiably denote it by 2 a , where a is a nonnegative real number. 2 The reader will recognize that the equation + = a can be plotted as a circle, as in Fig. 2.3, with radius a and with center at the point of origin. Since is plotted against in the diagram, this circle constitutes the phase line for the differential equation. Moreover, the circular loop y'2 y2 y' y shape of this phase line suggests that it gives rise to a cyclical time path, 1 1 Phase diagrams are explained in Alpha C. Chiang, Fundamental Methods of Mathematical Economics, 3d ed., McGraw-Hill, New York, 1984, Sec. 14.6. The phase line here is similar to phase line C in Fig. 14.3 in that section; the time path it implies is shown in Fig. 14.4c. CHAPTER 2: THE FUNDAMENTAL PROBLEM OF CALCULUS OF VARIATIONS 39 y'(t) FIGURE 2.3 with the y values bounded in the closed interval [ - a, a], as in a cosine function with amplitude a and period 27T. Such a cosine function can be represented in general by the equation y(t) = a cos( bt + c) where, aside from the amplitude parameter a , we also have two other parameters b and c which relate to the period and the phase of the function, respectively. In our case, the period should be 27T; but since the cosine function shows a period of 27T jb (obtained by setting the bt term to 27T ), we infer that b = 1. But the values of a and c are still unknown, and they must be determined from the given boundary conditions. At t = 0, we have y(O) = a cos c = 1 [by the initial condition] When t = 7T/2, we have Y( i) = a cos ( i + c ) = 0 [by the terminal condition] To satisfy this last equation, we must have either a = 0 or cos( 7T/2 + c) = 0. But a cannot be zero, because otherwise a cos c cannot possibly be equal to 1. Hence, cos( 7T /2 + c) must vanish, implying two possibilities: either c 0, or c 7T. With c = 0, the equation a cos c = 1 becomes a cos 0 = 1 or a(1) = 1, yielding a 1. With c = 7T, however, the equation a cos c = 1 becomes a( - 1) = 1, giving us a = - 1, which is inadmissible because we have restricted a to be a nonnegative number. Hence, we conclude that the boundary conditions require a = 1 and c = 0, and that the time path of y which qualifies as the extremal is = = = y * ( t ) = cos t The same solution can, as we would expect, also be obtained in a straightforward manner from the original Euler equation (2.18) or (2.19). PART 2, THE CALCULUS OF VARIATIONS 40 After normalizing and rearranging, the Euler equation can be expressed the following homogeneous second-order linear differential equation: as y" + y = O Since this equation has complex characteristic roots r1 , r 2 solution takes the form of2 y* ( t ) = a = ± i, the general cos t + f3 sin t The boundary conditions fix the arbitrary constants a and f3 at the values of 1 and 0, respectively. Thus, we end up with the same definite solution: y * ( t) = cos t For this example, the original Euler equation (2.18) or (2.19) turns out to be easier to use than the special one in (2.21). We illustrate the latter, not only to present it as an alternative, but also to illustrate some other techniques (such as the circular phase diagram in Fig. 2.3). The reader will have to choose the appropriate version of the Euler equation to apply to any particular problem. EXAMPLE 3 Among the curves that pass through two fixed points A and Z, which curve will generate the smallest surface of revolution when rotated about the horizontal axis? This is a problem in physics, but it may be of interest because it is one of the earliest problems in the development of the calculus of variations. To construct the objective functional, it may be helpful to consider the curve AZ in Fig. 2.4 as a candidate for the desired extremal. When rotated about the t axis in the stipulated manner, each point on curve AZ traces out a circle parallel to the xy plane, with its center on the t axis, and with radius R equal to the value of y at that point. Since the circumference of such a circle is 27TR (in our present case, 27Ty ) all we have to do to calculate the surface of revolution is to sum (integrate) the circumference over the entire length of the curve AZ. An expression for the length of the curve AZ can be obtained with the help of Fig. 2.5. Let us imagine that M and N represent two points that are located very close to each other on curve AZ. Because of the extreme proximity of M and N, arc MN can be approximated by a straight line, with its length equal to the differential ds. In order to express ds in terms of the variables y and t, we resort to Pythagoras' theorem to write (ds)2 (dy )2 + (dt)2• It follows that (ds)2j(dt)2 = (dy )2 j(dt)2 + 1. Taking the , = 2 For an explanation of complex roots, see Alpha C. Chiang, Fundamental Metlwds of Mathe­ matical Economics, 3d ed., McGraw-Hill, New York, 1984, Sec. 15.3. Here we have h = 0, and h v = 1; thus, e ' = 1 and vt = t. CHAP'l'Bil 2: THE FUNDAMENTAL PROBLEM OF CALCULUS OF VAlUATIONS y 41 z FIGURE 2.4 �- root o'ft both sides, we get ds dt - = .vhich yields the desired expression for ds in terms of y and t as follows: [arc length] fhe entire length of curve AZ must then be the integral of ds, namely f:(l + y'2 ) 1 12 dt. To sum the circumference 2'1Ty over the length of curve AZ will therefore result in the functional V [y ] = 2'1Tfzy(l a + 0 1/2 ..... y - + y'2) - t . PIGURE lU dt 42 PART 2, THE CALCULUS OF VARIATIONS The reader should note that, while it is legitimate to " factor out" the "2'1T" (constant) part of the expression 2'1TR, the (variable) part must remain within the integral sign. "y" For purposes of minimization, we may actually disregard the constant to _be the expression for the function, with 2'1T and take y(1 + y'2)112 F_y = yy ( 1 +y,2)-l/2 F ' Since F is free of t, we may apply (2.21) to get This equation can be simplified by takin� the following steps: (1) multiply through by (1 + (2) cancel out and on the left-hand side, in terms of y and c, and (4) take the (3) square both sides and solve for square root. The result is y' 2)112 , y'2 yy' -yy'2 y'(= dydt ) "!:.c .. Jy2 - c2 cdy ...jy2 c2 dt = or --= = = = _ y In this last equation, we note that the variables and have been separated, so that each side can be integrated by itself. The right-hand side t t + k, poses no problem, the integral of being in the simple form where k is an arbitrary constant. But the left-hand side is more complicated. In fact, to attempt to integrate it "in the raw" would take too much effort; hence, one should consult prepared tables of integration formulas to find the result:3 dt f cdy � vr = c ln - c� t+k Equating this result with (y + � ) c + c1 (and subsuming the constant c1 under the 3See, for example, CRC Standard Mathematical Tables, 28th ed., ed. by William H. Beyer, CRC Press, Boca Raton, FL, 1987, Formula 157. The c in the denominator of the log expression on the right may be omitted without affecting the validity of the formula, because its presence or absence merely makes a difference amounting to the constant - c In c, which can be absorbed into the arbitrary constant of integration c1, at any rate. However, by inserting the c in the denominator, our result will come out in a more symmetrical form. 43 CHAPTER 2: THE FUNDAMENTAL PROBLEM OF CALCULUS OF VARIATIONS constant k), we have ln ( y + Jy2 - c2 ) c e <t+ k )/c = y + VY 2 - c2 or = t+k c -- [ by definition of natural log ] c Multiplying both sides by c, subtracting y from both sides, squaring both sides and canceling out the y2 term, and solving for y in terms of t, we finally find the desired extremal to be y* ( t ) c = - [ e<t + k )jc 2 + e - ( t +k)/ c ] [ general solution ] where the two constants c and k are to be definitized by using the boundary conditions. This extremal is a modified form of the so-called catenary curve, ( 2.23) [catenary ] whose distinguishing characteristic is that it involves the average of two exponential terms, in which the exponent of one term is the exact negative of the exponent of the other. Since the positive-exponent term gives rise to a curve that increases at an increasing rate, whereas the negative-exponent term produces an ever-decreasing curve, the average of the two has a general shape that portrays the way a flexible rope will hang on two fixed pegs. (In fact, the name catenary comes from the Latin word catena , meaning "chain.") This general shape is illustrated by curve AZ in Fig. 2.4. Even though we have found our extremal in a curve of the catenary family, it is not certain whether the resulting surface of revolution (known as a catenoid) has been maximized or minimized. However, geometrical and intuitive considerations should make it clear that the surface is indeed minimized. With reference to Fig. 2.4, if we replace the AZ curve already drawn by, say, a new AZ curve with the opposite curvature, then a larger surface of revolution can be generated. Hence, the catenoid cannot possibly be a maximum. Special case III: F = F(y') When the F function depends on y' alone, many of the derivatives in (2.19) will disappear, including Fyy'' Fty • and FY . In fact, only the first term remains, so the Euler equation becomes ' ( 2.24) = To satisfy this equation, we must either have y"(t) 0 or Fy'y' 0. If = c1 and y(t) c1t + c2, indicating that the y"(t) = 0, then obviously y'(t) = = 44 PART 2: THE CALCULUS OF VARIATIONS general solution is a two-parameter family of straight lines. If, on the other hand, Fy'y' 0, then since Fy'y' is, like F itself, a function of y' alone, the solution of Fy'y' = 0 should appear as specific values of y'. Suppose there are one or more real solutions y' ki, then we can deduce that y = kit + c, which again represents a family of straight lines. Consequently, given an integrand function that depends on y' alone, we can always take its ex­ tremal to be a straight line. = = · EXAMPLE 4 Find the extremal of the functional ( 2 .25) with boundary conditions y(O) = A and y(T) = Z. The astute reader will note that this functional has been encountered in a different guise in Example 3. Recalling the discussion of arc length leading to (2.22), we know that (2.25) measures the total length of a curve passing through two given points. The problem of finding the extremal of this functional is therefore that of finding the curve with the shortest distance between those two points. The integrand function, F (1 + y' 2) 112, is dependent on y' alone. We can therefore conclude immediately that the extremal is a straight line. But if it is desired to examine this particular example explicitly by the Euler equation, we can use (2.18). With FY 0, the Euler equation is simply dFY.jdt = 0, and its solution is Fy' constant. In view of the fact that Fy' = y'/(1 + y'2 )112, we can write (after squaring). = = · ,_ :M\llqp)yjng both aides by .qan -� y' iJ1 �rms of . . •· . . .: ·. , = and factoring out y', we (1 + y' 2) rearranging, 2 as follows: y' c 2j(l - c 2). Equivalently, , c y' = = ( 1 c - c z) 1/2 = constant Inasmuch as y' (t), the slope of y(t), is a constant, the desired extremal y*(t) must be a straight line. Strictly speaking, we have only found an "extremal" which may either maximize or minimize the given functional. However, it is intuitively obvi­ ous that the distance between the two given points is indeed minimized rather than maximized by the straight-line extremal, because there is no such thing as " the longest distance" between two given points. :>:,. CHAPTER 2' THE FUNDAMENTAL PROBLEM OF CALCULUS OF VARIATIONS 45 Special case IV: F = F( t, y ) In this special case, the argument y ' i s missing from the F function. Since w e now have Fy' = 0, the Euler equa­ tion reduces simply to F 0, or, more explicitly, Y = Fy( t , y ) The fact that the derivative = 0 y' does not appear in this equation means that the E uler equation is not a differential equation. The problem is degenerate. Since there are no arbitrary constants in its solution to be definitized in accordance with the given boundary conditions, the extremal may not satisfy the boundary conditions except by sheer coincidence. This situation is very much akin to the case where the F function is linear in the argument y' (Sec. 2.1, Example 6). The reason is that the function F(t, y) can be considered as a special case of F(t, y, y') with y' entering via a single additive term, Oy', with a zero coefficient. Thus, F(t, y) is, i n a special sense, "linear" i n y'. EXERCISE 2.2 Find the extremals of the following functionals: 1 V[y] 2 V[y] 3 V[y] 4 5 6 V[y] V[y] V[y] = g(t2 = f;7y'3 dt, with y(O) = fcJ<y + yy' + y' + h'2) dt, with y(O) = 2 and y(l) = fabr3y'2 dt = = + y'2) dt, with y(O) = 0 and y(l) = 2 = 9 and y(2) = 11 = 5 (Find the general solution only.) JcJ<y2 + 4yy' + 4y'2) dt, with y(O) = 2e 1 12 f0" 12(y 2 - y'2) dt, with y(O) = 0 and and y( 1T/2) = y(l) = 1 + e 1 2.3 TWO GENERALIZATIONS OF THE EULER EQUATION The previous discussion of the Euler equation is based on an integral functional with the integrand F(t, y, y'). Simple generalizations can be made, however, to the case of several state variables and to the case where higher-order derivatives appear as arguments in the F function. The Case of Several State Variables With n (2.26) > 1 state variables in a given problem, the functional becomes .;' ,'_: �··· -. 46 PART 2: THE CALCULUS OF VARIATIONS and there will be a pair of initial condition and terminal condition for each of the n state variables. Any extremal y/ ( ), (j = 1, . . . , n ), must by definition yield the ex­ treme path value relative to all the neighboring paths. One type of neighbor­ ing path arises from varying _only one of the functions y ) at a time, say, y 1( , while all the other functions are kept fixed. Then the functional depends only on the variation in y1( ) as if we are dealing with the case of a single state variable. Consequently, the Euler equation (2.18) must still hold as a necessary condition, provided we change the y symbol to y1 to reflect the new problem. Moreover, this procedure can be similarly used to vary the other yi functions, one at a time, to generate other Euler equations. Thus, for the case of n state variables, the single Euler equation (2.18) should be replaced by a set of n simultaneous Euler equations: t t) ( 2 .27) }t y}t) FYj - dtd FY/ = 0 t for all t E [0, T ] (j = . 1 , 2, . . , n) [Euler equations] These n equations will, along with the boundary conditions, enable us to determine the solutions y 1 ( ) , Although (2.27) is a straightforward generalization of the Euler equa­ tion (2. 18)-replacing the symbol y by yi-the same procedure cannot be used to generalize (2.19). To see why not, assume for simplicity that there are only two state variables, y and in our new problem. The function will then be a function with five arguments, y, y', and the partial derivatives and will be, too. Therefore, the total derivative of with respect to will include five terms: y', * t . . . , Yn*(t). z, F(t, z, z'), F Fy' Fz'• t F/t, y, z, z') d t dt Fy.( , y, z ,y', z') = Fty' + Fyy•Y'( t ) + Fzy•z'(t) + Fy•yY"( t) + Fzyz" ( t) with a similar five-term expression for dF jdt. The expanded version of •. simultaneous Euler equations corresponding more complicated than (2. 19) itself: (2 .28) to (2.19) thus looks much FHy"(t) + Fz'y'z"( t ) + Fyy•Y'( t ) + Fzy'z'( t ) + Fty' - FY = 0 Fyz•Y"( t ) + Fz'z'z"( t ) + Fyz•Y'( t ) + Fzz'z'( t ) + F;z' - Fz = 0 for all t E EXAMPLE 1 Find the extremal of From the integrand, we find that Fy = 1 Fy' = 2y' F11 = 2z' [0, T ] CHAPTER 2: Thus, by 47 THE FUNDAMENTAL PROBLEM OF CALCULUS OF VARIATIONS (2.27), we have two simultaneous Euler equations 2y" z 11-2 " 0 0 = = � or " y = or Z, - !2 The same result can also be obtained from (2.28). In this particular case, it happens that each of the Euler equations · contains one variable exclusively. Upon integration, the first yields + and hence, �t c1, y* ( t) it2 + c1t + c2 z z* ( t) tt2 + c3t + c4 (c1, , c4) y(O), z(O), y(T), y' = = Analogously, the extremal of is = The four arbitrary constants can be definitized with the help of . • . four boundary conditions relating to and z(T). The Case of Higher-Order Derivatives another generalization, consider a functional that contains high-order derivatives of Generally, this can be written as As y(t). V[y] = f TF(t,y,y',y", . . . ,y<n>) dt ( 2 .29) 0 Since many derivatives are present, the boundary conditions should in this case prescribe the initial and terminal values not only of but also of the derivatives total of 2n y',y", .. . . , up to and including the derivative boundary conditions. y, y<n-I>, making a F To tackle this case, we first note that an function with a single state and derivatives of y up to the n th order can be transformed into variable y an equivalent form containing n state variables and their first-order deriva­ tives only. In other words, the functional in (2.29) can be transformed into the form in (2.26). Consequently, the Euler equation (2.27) or (2.28) can again be applied. Moreover, the said transformation can automatically take care of the boundary conditions as well. EXAMPLE 2 Transform the functional y(O) y(T) y'(O) y'(T) A, = Z, = a, and /3 , with boundary conditions into the format o f (2.26). T o achieve this task, we only have t o introduce a new variable z y' == = [ implying z' y"] == = .. . PART Then we cah rewrite the integrand as 2: THE CALCIJLUSOF VARIAT� y which now contains two state variables and z, with no derivatives higher than the first order. Substituting the new F into the functional will turn the latter into the format of What about the boundary conditions? For the original state variable = A and = Z can be kept intact. The other two the conditions conditions for can be directly rewritten as the conditions for the new = a and state variable z: = {3. This complete the transformation. (2.26). y, y(T) y(O) y' z (O) z(T) Given the functional (2.29), it is also possible to deal with it directly instead of transforming it into the format of (2.26). By a procedure similar to that used in deriving the Euler equation, a necessary condition for an extremal can be found for (2.29). The condition, known as the Eu ler­ Poisson equation, is d2 dn d + ( - l} n FY - - FY. + -Fy (2.30) dt dt2 " dtn FYc., 0 = · · · for all [ Euler-Poisson equation ) t E [0, T) 2n. This equation is in general a differential equation of order Thus its solution will involve arbitrary constants, which can be definitized with the help of the boundary conditions. 2n EXAMPLE 3 2n Find an extremal of the functional in Example have FY = F,. 2ty + y' = 2. Since we 2y" the Euler-Poisson equation is dy + -d22y" = 0 2ty + y' - dt dt2 or 2ty + 2y<•> - 0 which is a fourth-order differential equation. EXERCISE 2.3 1 2 Find the extremal of y(l) = y'(l) = 1. Find the extremal of only). V[y] = gu V[y, z] = + y"2) dt, with y(O) = 0 J:<y' 2 + z'2 + y'z') dt and y'(O) = (general solution 1·. . OHAPTER 2: THE FUNDAMENTAL PROBLEM OF CALCULUS OF VARIATIONS 49 3 Find the extremal of V[y, z] = j0"'12(2yz + y'2 + z'2) dt, with y(O) = z(O) = 0 and y(rr/2) = z(rr/2) = 1. 4 Example 3 of this section shows that, for the problem stated in Example 2, a necessary condition for the extremal is 2ty + 2y<4l = 0. Derive the same result by applying the definition that problem. 2.4 z = y' and the Euler equations (2.27) to DYNAMIC OPTIMIZATION OF A MONOPOLIST Let us turn now to economic applications of the Euler equation. As the first example, we shall discuss the classic Evans model of a monopolistic firm, one of the earliest applications of variational calculus to economics. 4 A Dynamic Profit Function Consider a monopolistic firm that produces a single commodity with a quadratic total cost function5 C = aQ 2 + [3Q + y (2.31) ( a , {3 , y > O) Since no inventory is considered, the output Q is always set equal to the quantity demanded. Hence, we shall use a single symbol Q(t) to denote both quantities. The quantity demanded is assumed to depend not only on price P(t), but also on the rate of change of price P'(t): ( a , b > O ; h *- 0) (2.32) Q = a - bP( t) + hP'(t) The firm's profit is thus 1r = PQ - C = P ( a - bP + hP' ) - a( a - bP + hP') 2 - f3(a - bP + bP') - y which is a function of P and P'. Multiplying out the above expression and collecting terms, we have the dynamic profit function (2.33) 1r(P, P') = -b(l + ab)P 2 + ( a + 2aab + f3b) P - ah2P' 2 - h(2aa + f3 ) P' + h ( l + 2ab) PP' [profit function] - ( aa2 + {3a + y) 4G. C. Evans, " The Dynamics ofMonopoly," American Mathematical Monthly, February 1924, pp. 77-83. Essentially author, the same material appears in Chapters 14 and 15 of a book by the same Mathematical Introduction to Economics, McGraw-Hill, New York, 1930, pp. 143-153. 6This quadratic cost function plots as a U-shaped curve. But with f3 positive, only the right-hand segment of the U appears in the first quadrant, giving us an upward-sloping total-cost curve for Q � 0. 50 PART 2: THE CALCULUS OF VARIATIONS The Problem P The objective of the firm is to find an optimal path of price that maximizes the total profit fi over a finite time period [0, T ]. This period is assumed to be sufficiently short to justify the assumption of fixed demand and cost functions, as well as- the omission of a discount factor. Besides, as the first approach to the problem, both the initial price and the terminal price are assumed to be given. The objective of the monopolist is therefore to P0 Pr fi(P] = j0T1r(P, P') dt P( O) = P0 (P0 given) P(T ) = Pr (T, Pr given) Maximize subject to (2.34) and The Optimal Price Path (2.34) Although pertains to Special Case II, it turns out that for computa­ tion with specific functions it is simpler to use the original Euler equation (2.19), where we should obviously substitute 7T for F, and for y. From the profit function it is easily found that P (2.33), 7Tp = -2b(l + ab)P + (a + 2aab + {3b) + h(l + 2ab ) P' = -2ah2P' - h (2aa + /3) + h(l + 2ab ) P = h ( l + 2ab ) 7TP'P' = - 2ah2 · 7TP' and 7Tpp• These expression turn (2. 19)-after normalizing-into the specific form + f3b (2.35 ) P" - b(la+h2ab) p = - a + 2aab 2 2a h [Euler equation] This is a second-order linear differential equation with constant ficients and a constant term, in the general format of y " coef­ + a iy' + a 2y = a a Its general solution is known to be6 6 This type of differential equation is discussed in Alpha C. Chiang, Fundamental Methods of Mathematical Econcmics, 3d ed., McGraw-Hill, New York, 1984, Sec. 15.1. CHAPTER 2: 51 THE FUNDAMENTAL PROBLEM OF CALCULUS OF VARIATIONS where the characteristic .roots r 1 and r2 take the values r1, r2 = t { -a1 ± Jai - 4 a 2 ) and the particular integral y is Thus, for the Euler equation of the present model (where a1 = (2 . 36) where _ p and = b ( 1 + ab) ah2 a + 2aab + f3b ____;__ 2b( 1 + ab) 0), we have [ characteristic roots J (particular integral] _ _ _ Note that the two characteristic roots are real and distinct under our sign specifications. Moreover, they are the exact negatives of each other. We may therefore let r denote the common absolute value of the two roots, and rewrite the solution as (2.36') (general solution ] The two arbitrary constants A1 and A2 in the boundary conditions P(O) = P0 and (2.36' ) can be definitized via P(T) = PT . When we set t = 0 and t = T, successively, in (2.36'), we get two simultaneous equations in the two variables A1 and A2: Po = A l + A z + p PT = Al e rT + A 2e- rT + p with solution values (2 .37) A1 - - -P -p - ) e rT ( Pr p 1 - e 2 rT o :.. -� - _!__....:.., ...;.� · A2 - P0 - P - (PT - P) e - rT 1 e -2 r T _ The determination of these two constants completes the solution of the problem, for the entire price path P*(t) is now specified in terms of the known parameters T, P0, PT , {3, y, and Of these parameters, all have specified signs except But inasmuch as the parameter enters into the solution path (2.36') only through r, and only in the form of a squared term we can see that its algebraic sign will not affect our results, h. a, h2 , although its numerical value will. a, b, h. h 52 PART 2: THE CALCULUS OF VARIATIONS At this moment, we are not equipped to discuss whether the solution path indeed maximizes (rather than minimizes) profit. Assuming that it indeed does, then a relevant question is: What can we say about the general configuration of the price path (2.36')? Unfortunately, no simple answer can be given. If Pr > P0, the monopolist's price may optimally rise steadily in the interval [0, T ] from P0 ··to Pr or it may have to be lowered slightly before rising to the level of Pr at the end of the period, depending on parameter values. In the opposite case of P0 > Pr, the optimal price path is characterized by a similar indeterminacy. Although this aspect of the prob­ lem can be pursued a little further (see Exercise 2.4, Probs. 3 and 4), specific parameter values are needed before more definite ideas about the optimal price path can be formed. A More General View of the Problem Evans' formulation specifies a quadratic cost function and a linear demand function. In a more general study of the dynamic-monopolist problem by Tintner, these functions are left unspecified. 7 Thus, the profit function is .. simply written as rr(P, P'). In such a general formulation, it turns out that :':\ formula (2.21) [for Special Case II] can be used to good advantage. It directly yields a simple necessary condition rr - ( 2 .38 ) arr P'aP' = c that can he given a clear-cut economic interpretation. To see this, first look into the economic meaning of the constant c. If profit rr does not depend on the rate of change of price P'-that is, if we are dealing with the static monopoly problem as a special case of the dynamic model-then arr;aP' = 0, and (2.38) reduces to rr = c . So the constant c represents the static monopoly profit. Let us therefore denote it by 7T8 (subscript s for static). Next, we note that if (2.38) is divided through by rr, the second term on the left-hand side will involve arr P' aP' -.;;:- = E ,.p' which represents the partial elasticity of rr with respect to P'. Indeed, after performing the indicated division, the equation can he rearranged into the 7 Gerhard Tintner, " Monopoly over Time," Econometrica , 1937, pp. 160- 170. Tintner also tried to generalize the Evans model to the case where rr depends on higher-order derivatives of P, but the economic meaning of the result is difficult to interpret. CHAPTER 2: THE FUNDAMENTAL PROBLEM OF CALCULUS OF VARIATIONS optimization rule €-rr p' ( 2 .38' ) 7Ts = 1 - 7T This rule states that the monopolistic firm should always select the price in such a way that the elasticity of profit with respect to the rate of change of price be equal to 1 - 7T5j7T. The reader will note that this analytical result would not have emerged so readily if we had not resorted to the special-case formula (2.21). It is of interest to compare this elasticity rule with a corresponding elasticity rule for the static monopolist. Since in the latter case profit depends only on P, the first-order condition for profit maximization is simply d 7TjdP = 0. If we multiply through by Pj7T, the rule becomes couched in terms of elasticity as follows: E.,P = 0. Thus, while the state monopolist watches the elasticity of profit with respect to price and sets it equal to zero, the dynamic monopolist must instead watch the elasticity of profit with respect to the rate of change of price and set it equal to 1 - 7TJ7T. The Matter of the Terminal Price The foregoing discussion is based on the assumption that the terminal price given. In reality, however, the firm is likely to have discretionary control over P(T) even though the terminal time T has been preset. If so, it will face the variable-terminal-point situation depicted in Fig. 1.5a, where the boundary condition P( T) = Pr must be replaced by an appropriate transversality condition. We shall develop such transversality conditions in the next chapter. P( T ) is EXERCISE 2.4 1 If the monopolistic firm in the Evans model faces a static linear demand (h = 0), what price will the firm charge for profit maximization? Call that price P., and check that it has the correct algebraic sign. Then compare the values of P. and P, and give the particular integral in (2.36) an economic interpretation. A1 A2 should indeed have the values shown in (2.37). 2 Verify that and 3 Show that the extremal P*(t) will not involve a reversal in the direction of price movement in the time interval [0, T ] unless there is a value 0 < t0 < T 4 A1er1• A 2e-rto A1 such that = [i.e., satisfying the condition that the first two terms on the right-hand side of (2.36') are equal at t = t0]. If the coefficients and A 2 in (2.36' ) are both positive, the P*(t) curve will take the shape of a catenary. Compare the location of the lowest point 54 PART 2: THE CALCULUS OF VARIATIONS 5 on the price curve in the following three cases: (a) A1 > A2, (b) A1 = A2, and (c) A1 < A2• Which of these cases can possibly involve a price reversal in the interval [0, T ]? What type of price movement characterizes the remaining cases? Find the Euler equation for the Evans problem by using formula (2.18). 6 If a discount rate p > t:J is used to convert the profit 7T(P, P' ) at any point of time to its present value, then the integrand in (2.34) will take the general form F = 7T(P, P')e - P1 • ·� (a) In that case, is formula (2.21) still applicable? F expression to derive the new Euler equation. (c) Apply formula (2.18) to derive the Euler equation, and express the result as a rule regarding the rate of growth of 7Tp• · (b) Apply formula (2.19) to this 2.5 TRADING OFF INFLATION AND UNEMPLOYMENT The economic maladies of both inflation and unemployment inflict social losses. When a Phillips tradeoff exists between the two, what would be the best combination of inflation and unemployment over time? The answer to this question may be sought through the calculus of variations. In this section we present a simple formulation of such a problem adapted from a paper by Taylor.8 In this formulation, the unemployment variable as such is not included; instead, it is proxied by (Y1 - Y)-the shortfall of current national income Y from its full-employment level Y1 . The Social Loss Function Let the economic ideal consist of the income level Y1 coupled with the inflation rate 0. Any deviation, positive or negative, of actual income Y from Y1 is considered undesirable, and so is any deviation of actual rate of inflation p from zero. Then we can write the social loss function as follows: ( 2.39) (a > 0) Because the deviation expressions are squared, positive and negative devia­ tions are counted the same way (cf. Exercise 1.3, Prob. 2). However, Y deviations and p deviations do enter the loss function with different weights, in the ratio of 1 to a, reflecting the different degrees of distaste for the two types of deviations. 8Dean Taylor, "Stopping Inflation in the Dornbusch Model: Optimal Monetary Policies with Alternate Price-A(ijustment Equations," Journal ofMacroecorwmics, Spring 1989, pp. 199-216. In our adapted version, we have altered the planning horizon from oo to a finite T. lI l 55 CHAPTER 2: THE FUNDAMENTAL PROBLEM OF CALCULUS OF VARIATIONS The expectations-augmented Phillips tradeoff between is captured in the equation (Yr - Y) and p p = -f3(Yr - Y) + 1r ( f3 > 0) (2.40) where 7T, unlike its usage in the preceding section, now means the expected rate of inflation. The formation of inflation expectations is assumed to be adaptive: ' 7T (2.41) ( �;) =j(p = - 1T ) (0 < j :5 1) p If the actual rate of inflation exceeds the expected rate of inflation 1T (proving 7T to be an underestimate), then 1r' > and 1T will be revised 0, upward; if, on the other hand, the actual rate p falls short of the expected rate 1T (proving 1T to be an over estimate), then 1r' < and 1T will be revised downward. The last two equations together imply that which can be rearranged as - 7T (2.40), (2.42) yields 1T (2.43) ' p = -;- + 7T J (2.42) (2.43) into (2.39), we are able to express the loss And, by plugging and function entirely in terms of (2 .44) ' {3j Yr - Y = - (2 .42) When substituted into 0, A(1T, 1r') = ( {3j 7T' 1T and 1r': )2 ( j )2 + a 1T' + 1T [loss function] The Problem The problem for the government is then to find an optimal path of [0, T). 1T that minimizes the total social loss over the time interval The initial (current) value of 1T is given at 1r0, and the terminal value of 1r, a policy target, is assumed to be set at 0. To recognize the importance of the present over the future, all social losses are discounted to their present values via a positive discount rate p. PART 2, THE CALCULUS OF VARIATIONS In view of these considerations, the objective of the policymaker is to .> ''� A[ 1T ) Minimize (2.45) subject to � r: and = · . .--·-:� ·:- • j TA ( 1r, 1r')e-pt dt 0 ! l l ( 1r0 > 0 given) 1r(O) = 1r0 1r(T) 0 (T given) = . I d' f The Solution Path On the basis of the loss function yields the first derivatives ' (2.44), the integrand function A(1T, 1T')e7', . with second derivatives and ( , F -rrr. = ) 2a -e-pt j a -pt 1 + af32 , Ft1T' - - 2p f3 2 · 2 1T + . 1T e - J J Consequently, formula (2.19) gives us (after simplification) the specific necessary condition (2 .46) 1r" - p1r' - il1r = 0 [�uler equation) af32j (p + j ) > 0 where n 1 + af3 2 == Since this differential equation is homogeneous, its particular integral is zero, and its general solution is simply its complementary function: (2.47) [general solution) The characteristic roots are real and distinct. Moreover, inasmuch square root has a numerical value greater than we know that p, (2.48) as the and ' L. ! CHAPTER 2: THE FUNDAMENTAL I'ROBLEM OF CALCULUS OF VARIATIONS 57 FIGURE 2.6 t = To definitize the arbitrary constants A 1 and A 2 , we successively set t= in (2.4 7), and use the boundary conditions to get the pair of 0 and relations T A I + A2 Al = '1T o e r,T + A 2 e rzT = 0 Solved simultaneously, these relations give us (2.49) Because of the signs of r1 and r2 in (2.48), we know that (2.50) From the sign information in (2.50) and (2.48), we can deduce that the 2.6. With a 7r* (t) path should follow the general configuration in Fig. t negative A1 and positive r1, the A1er, component of the path emerges as the mirror image (with reference to the horizontal axis) of an exponential t growth curve. On the other hand, the A 2 e rz component is, in view of the positive A2 and negative r2, just a regular exponential decay curve. The '7T* path, the sum of the two component curves, starts from the point 7T0)-where 1r0 = A1 + A2-and descends steadily toward the point (0, (T, 0), which is vertically equidistant from the two component curves. fact that the 7r * path shows a steady downward movement can The also be 58 PART 2: THE CALCULUS OF VARIATIONS verified from the derivative ?T*' ( t ) = 1 r1A1er,t + r2A2er•t < 0 Having found ?T*(t) and ?T*'(t), we can also derive some conclusions regarding p and (Yr - Y). F?r these, we can simply use the relations in (2.43) and (2.42), respectively. EXERCISE 2.5 1 Verify the result in (2.46) by using Euler equation (2.18). 2 Let (a) (b) S the /:lA<7T, objective 7r')e-P1 dt. functional in problem (2.45) be changed to Do you think the solution of the problem will be different? Can you think of any advantage in including a coefficient ! in the integrand? Let the terminal condition in problem 0 < (a) (b) 7Tr < 7ro. (2.45) be changed to 1r(T) = What would be the values of A1 and A2? Can you ambiguously evaluate the signs of A1 and A2? 7Tr, j CHAPTER 3 TRANSVERSALITY CONDITIONS FOR VARIABLE­ ENDPOINT PROBLEMS · The Euler equation-the basic necessary condition in the calculus of varia­ tions-is normally a second-order differential equation containing two arbi­ trary constants. For problems with fixed initial and terminal points, the two given boundary conditions provide sufficient information to definitize the two arbitrary constants. But if the initial or terminal point is variable (subject to discretionary choice), then a boundary condition will be missing. Such will be the case, for instance, if the dynamic monopolist of the preceding chapter faces no externally imposed price at time T, and can treat the Pr choice as an integral part of the optimization problem. In that case, the boundary condition P(T ) = Pr will no longer be available, and the void must be filled by a transversality condition. In this chapter we shall develop the transversality conditions appropriate to various types of variable termi­ nal points. 3.1 THE GENERAL TRANSVERSALITY CONDITION For expository convenience, we shall assume that only the terminal point is variable. Once we learn to deal with that, the technique is easily extended to the case of a variable initial point. 59 PART 2: THE CALCULUS OF VARIATIONS 60 The Variable-Terminal-Point Problem Our new objective is to Maximize or minimize subject to (3.1) V{y] t0 F(t,y,y') dt y(O) A ( A given) y(T) Yr ( T, yr free) = = and = This differs from the previous version of the problem in that the terminal time and terminal state Yr are now " free" in the sense that they have become a part of the optimal choice process. It is to be understood that although is free, only positive values of are relevant to the problem. To develop the necessary conditions for an extremal, we shall, as before, employ a perturbing curve and use the variable to generate neighboring paths for comparison with the extremal. First, suppose that is the known optimal terminal time. Then any value of in the immediate neighborhood of can be expressed as T T T p(t), . E T T* T* T T* + o lT where E represents a small number, and llT represents an arbitrarily chosen (and fixed) small change in T. Note that since T* is known and llT is a prechosen magnitude, T can be considered as a function of E, T(E), with ( 3 .2) = derivative dT llT dE = (3.3) E The same i s used i n conjunction with the perturbing curve generate neighboring paths of the extremal y*(t): [ implying y'(t) y*'(t) + Ep'(t)] p(t) to y(t) y*(t) + Ep(t) However, although the p (t) curve must still satisfy the condition p(O) = 0 (3 .4) = = [see (2.2)] to force the neighboring paths to pass through the fixed · initial point, the other condition-p(tY= 0-should now be dropped, because Yr is free. By substituting (3.4) into the functional we get a function akin to (2.12), but since is a function of by (3.2), the upper limit of integration in the function will also vary with V[y], V E: V(E) foT(E)F[t, y*(t) + Ep(t) , y*'(t) + Ep'(t) ) dt (3 .5) y'(t) y(t) Our problem is to optimize this V function with respect to E. T = E V(E) CHAPTER 3: TRANSVERSAL!TY CONDJ'I'!ONS FOR VARIABLE-ENDPOINT PROBLEMS 61 Deriving the General Transversality Condition The first-order necessary condition for an extremum in V is simply dV1d c = 0. This latter derivative is a total derivative taking into account the direct effect of E on V, as well as the indirect effect of E on V through the upper limit of integration T _ Step i The definite integral (3.5) falls into the general form of (2.10). The derivative dVjdE is therefore, by (2. 11), (3.6) dV dc = fT<•> aF dt 0 ac + dT F [ T , y( T ) , y'( T ) ] dE The integral on the right closely resembles the one encountered in (2.13) during our earlier development of the Euler equation. In fact, much of the derivation process leading to (2.17) still applies here, with one exception. The expression [FY.p(t)J& in (2. 16) does not vanish in the present problem, but takes the value [ FY.p(t)]1_ T [ Fy,]t - Tp(T), since we have assumed that p(O) = 0 but p(T) i= 0. With this amendment to the earlier result in (2. 17), we have = By (3.3), we can also write Second term in ( 3.6) = [ F ] 1 = 7- .6. T Substituting these into (3.6), and setting dV/dE = 0, we obtain the new condition Ofthe three terms on the left-hand side of (3. 7), each contains its own independent arbitrary element: p(t) (the entire perturbing curve) in the first term, p(T) (the terminal value on the perturbing curve) in the second, and tiT (the arbitrarily chosen change in T ) in the third. Thus we cannot presume any offsetting or cancellation of terms. Consequently, in order to satisfy the condition (3. 7), each of the three terms must individually be set equal to zero. It is a familiar story that when the first term in (3.7) is set equal to zero, the Euler equation emerges, as in (2.18). This establishes the fact that the Euler equation remains valid as a necessary condition in the variable­ endpoint problem. The other two terms, which relate only to the terminal time T, are where we should look for the transversality condition. 62 y PART 2: THE CALCULUS OF VARIATIONS I ����-�- � - - - - - � - . . I : ,; · '· ' ' ' ·' ?:'' 1 ,.,, .,.. � -- -- -- -- _.. ..- .... .... .... (7') p Z" · :. l} y'(T) 4r : --· ,.... ..... A J..-�� p(t) I .. · • ·, . .;· ·:, 1 I I I I Z I ,.... .J ,.. I I ll T I � � � ----------------�--�.----0 �- T FIGURE 3.1 Step ii To this end, we first get rid of the arbitrary quantity p(T) by transforming it into terms of AT and Ayr-the changes in T and Yr• the two principal variables in the variable-terminal-point problem. This can be done with the help of Fig. 3.1. The · AZ' curve represents a neighboring path obtained by perturbing the AZ path by Ep(t), with E set equal to one for convenience. Note that while the two curves share the same initial point because p(O) = 0, they have different heights at t = T because p(T ) 'f. 0 by construction of the perturbing curve. 1 The magnitude of p(T), shown by the vertical distance ZZ', measures the direct change in Yr resulting from the perturbation. But inasmuch as T can also be altered by the amount E AT ( = AT since E = 1), the AZ' curve should, if A T > 0, be extended out 2 to Z". As a result, Yr is further pushed up by the vertical distance between Z' and Z". For a small AT, this second change in Yr can be approximated by y'( T ) AT. Hence, the total change in Yr from point Z to point Z" is Ayr = p ( T ) + y'( T ) A T Rearranging this relation allows us t o express p ( T ) i n terms o f Ayr and AT:3 ( 3 .8) p ( T ) = Ayr - y'( T ) AT 1 Technically, the point T on the horizontal axis in Fig. 3.1 should be labeled T*. We are omitting the • for simplicity. This is justifiable because the result that this discussion is leading to-(3.9)-is a transversality condition, which, as a standard practice, is stated in terms of T (without the * ) anyway. 2 Aithough we are initially interested only in the solid portion of the AZ path, the equation for that path should enable us also to plot the broken portion as an extension of AZ. The perturbing curve is then applied to the extended version of AZ. 3 The result in (3.8) is valid even if £ is not set equal to one as in Fig. 3.1. CHAPTER 3: TRANSVERSALITY CONDITIONS FOR VARIABLE-ENDPOINT PROBLEMS 63 Step iii Using (3.8) to eliminate p(T) in (3. 7), and dropping the first term in (3.7), we finally arrive at the desired general transversality condition ( 3.9) This condition, unlike the Euler equation, is relevant only to one point of time, T. Its role is to take the place of the missing terminal condition in the present problem. Depending on the exact specification of the terminal line or curve, however, the general condition (3.9) can be written in various specialized forms. 3.2 SPECIALIZED TRANSVERSALITY CONDITIONS In this section we consider five types of variable terminal points: vertical terminal line, horizontal terminal line, terminal curve, truncated vertical terminal line, and truncated horizontal terminal line. Vertical Terminal Line (Fixed-Time-Horizon Problem) The vertical-terminal-line case, as illustrated in Fig. 1.5a, involves a fixed T. Thus tlT 0, and the first term in (3.9) drops out. But since tlyr is arbitrary and can take either sign, the only way to make the second term in (3.9) vanish for sure is to have Fy' 0 at t T. This gives rise to the transversality condition = = = ( 3 1 0) . which is sometimes referred to as the natural boundary condition. Horizontal Terminal Line . (Fixed-Endpoint Problem) For the horizontal-terminal-line case, as illustrated in Fig. 1.5b, the situa­ tion is exactly reversed; we now have tlyr = 0 but tlT is arbitrary. So the second term in (3.9) automatically drops out, but the first does not. Since tlT is arbitrary, the only way to make the first term vanish for sure is to have the bracketed expression equal to zero. Thus the transversality condi­ tion is ( 3 . 1 1) It might be useful to give an economic interpretation to (3.10) and (3. 11). To fix ideas, let us interpret F(t, y, y') as a profit function, where y represents capital stock, and y' represents net investment. Net investment 64 PART 2: THE CALCULUS OF VARIATIONS entails taking resources away from the current profit-making business operation, so as to build up capital which will enhance future profit. Hence, there exists a tradeoff between current profit and future profit. At any time t, with a given capital stock y, a specific investment decision-say, a decision to select the investment rate y'0-will result in the current profit F(t, y, y'0). As to the effect o£ the investment decision on future profits, it enters through the intermediary of capital. The rate of capital accumulation is y'0; if we can convert that into a value measure, then we can add it to the current profit F(t, y, y'0) to see the overall (current as well as future) profit implication of the investment decision. The imputed (or shadow) value to the firm of a unit of capital is measured by the derivative Fy' · This means that if we decide to leave (not use up) a unit of capital at the terminal time, it will entail a negative value equal to -Fy'· Thus, at t = T, the value measure of y'0 is -y'0FY'o· Accordingly, the overall profit implication of the decision to choose the investment rate y'0 is F(t, y, y'0) - y0F:Io · The gen­ eral expression for this is F - y'Fy'• as in (3. 11). Now we can interpret the transversality condition (3.11) to mean that, in a problem with a free terminal time, the firm should select a T such that a decision to invest and accumulate capital will, at t = T, no longer yield any overall (current and future) profit. In other word!>, all the profit opportunities should have been fully taken advantage of by the optimally chosen terminal time. In addition, (3. 10)-which can equivalently be writ­ ten as [ - Fy,]t- T = 0-instructs the firm to avoid any sacrifice of profit that will be incurred by leaving a positive terminal capital. In other words, in a free-terminal-state problem, in order to maximize profit in the interval [0, T ] but not beyond, the firm should, at time T, us up all the capital it ever accumulated. Terminal Curve With a terminal curve Yr = l/J(T ), as illustrated in Fig. 1.5c, neither 11Yr nor 11T is assigned a zero value, so neither term in (3.9) drops out. However, for a small arbitrary 11T, the terminal curve implies that 11yr ¢' 11T. So it is possible to eliminate 11yr in (3.9) and combine the two terms into the form = Since 11T is arbitrary, we can deduce the transversality condition ( 3 . 12) [ F + ( cf>' - y' ) Fy· L - r = 0 Unlike the last two cases, which involve a single unknown in the terminal point (either Yr or T ), the terminal-curve case requires us to determine both Yr and T. Thus two relations are needed for this purpose. CHAPTER 3: TRANSVERSALITY CONDITIONS FOR VARIABLE-ENDPOINT PROBLEMS 65 The transversality condition (3.12) only provides one; the other is supplied by the equation Yr = cf>(T). Truncated Vertical Terminal Line The usual case of vertical terminal line, with !lT - 0, specializes (3.9) to ( 3.13 ) When the line is truncated-restricted by the terminal condition Yr � Ymin where Ymin is a minimum permissible level of y-the optimal solution can have two possible types of outcome: y/ > Yrnin or y/ = Ymin · If y/ > Yrnin • the terminal restriction is automatically satisfied; that is, it is nonbinding. Thus, the transversality condition is in that event the same as (3. 10): for Yr* > Y min ( 3 .14) The basic reason for this is that, under this outcome, there are admissible neighboring paths with terminal values both above and below y/, so that llyr = Yr - Yr* can take either sign. Therefore, the only way to satisfy (3.13) is to have Fy' = 0 at t = T. The other outcome, y/ = Ymin • on the other hand, only admits neigh­ boring paths with terminal values Yr � y/. This means that !lyT is no longer completely arbitrary (positive or negative), but is restricted to be nonnegative. Assuming the perturbing curve to have terminal value p(T ) > 0, as in Fig. 3.1, !lyT � 0 would mean that t: � 0 (t: = 1 in Fig. 3.1). The nonnegativity of t: , in turn, means that the transyersality condition (3.13)-which has its roots in the first-order condition dVIdE = 0- must , be changed to an inequality as in the Kuhn-Tucker conditions.4 For a maximization problem, the � type of inequality is called for, and (3.13) should become ( 3. 15) And since ( 3. 16 ) !lyr � 0, (3.15) implies condition for Yr* = Ymin . .. ·- , · .:- Combining (3.14) and (3.16), we may write the following summary statement of the transversality condition for a maximization problem: Y/ � Ymin ( 3 . 17) ( y/ - Ym;n) [ Fy· L -r = 0 [for maximization of V} 4 For an explanation of the Kuhn-Tucker conditions, see Alpha C . Chiang, Fundamental Methods of Mathematical Economics, 3d ed., McGraw-Hill, New York, 1984, Sec. 2 1 .2. ' � ·.;: . 66 PART 2: THE CALCULUS OF VARIATIONS If the problem is instead to minimize V, then the inequality sign in (3.15) must be reversed, and the transversality condition becomes [ F,. L - T ;:>: 0 ( 3 . 17') (y/ - Ymin ) [ Fy' L - T = 0 [ for minimization of V] The tripartite condition (3.17) o r (3.17') may seem complicated, but its application is not. In practical problem solving, we can try the [ = = 0 condition in (3. 14) first, and check the resulting value. If Fy•l, T y/ � Ymin• then the terminal restriction is satisfied, and the problem solved. If Yr* < Ymin• on the other hand, then the optimal Yr lies below the permissible range, and [ Fy•l,_r fails to reach zero on the truncated terminal line. So, in y/ order to satisfy the complementary-slackness condition in (3.17) or (3. 17'), we just set Ymin • treating the problem as one with a fixed terminal point Yr* = (T, Ymin ). Truncated Horizontal Terminal Line The horizontal terminal line may be truncated by the restriction :s; where max represents a maximum permissible time for completing a task-a deadline. The analysis of such a situation is very similar to the truncated vertical terminal line just discussed. By analogous reasoning, we can derive the following transversality condition for a maximization prob­ lem: T T (3.18 ) [ F - y'Fy•L � r ;:>: 0 ( T * - Tmax ) [ F - y'Fy•L - r = 0 If the problem is t o minimize V, T* :S; Tmw<> Tmax [ for maximization of V ] the first inequality in (3.18) must be changed, and the transversality condition is ( 3 . 18') [ F - y'F,.j , _ T :s; 0 ( T * - Tmax ) [ F - y'Fy·L -r 0 T* :S; Tmax [ for minimization of V ] = EXAMPLE 1 Find the curve with the shortest distance between the point and the line = 2 t. Referring to Fig. 3.2, we see that this is a problem with a terminal curve = 2 - T, but otherwise it is similar to the shortest-distance problem in Example 4 of Sec. 2.2. Here, the problem is to (0, 1) y - y(T) T Minimize V[y ] = 1 (1 subject to y(O) and y( T ) 0 = 1 = / + y'2 ) 1 2 dt 2-T CHAPTER 3: TRANSVERSALITY CONDITIONS FOR VARIABLE-ENDPOINT PROBLEMS 67 y '. {� r . y = 2 - ': : 0 ._------4---�-2 FIGURE 3.2 It has previously been shown that with the given integrand, the Euler equation yields a straight-line extremal, say, y* = at + b with two arbitrary constants a and b. From the initial condition y(O) = 1, we can readily determine that b = 1. To determine the other constant, a, we resort to the transversality condition (3.12). Since we have ¢' = -1 the transversality condition can be written as (at t = T ) 1/ 2 2 Multiplying through by ( 1 + y ' ) and simplifying, we can reduce this equation to the form y' = 1 (at t = T ). But the extremal actually has a constant slope, y*' = at all values of t. Thus, we must have a = 1. And a, the extremal is therefore y* ( t) = t + 1 As shown in Fig. 3.2, the extremal is a straight line that meets the terminal line at the point <t 1-V. Moreover, we note that the slopes of the terminal line and the extremal are, respectively, - 1 and + 1. Thus the two lines are orthogonal (perpendicular) to each other. What the transversality condition does in this case is to require the extremal to be orthogonal to the terminal line. EXAMPLE 2 Find the extremal of the functional PART 2: THE CALCULUS OF VARIATIONS 68 y (6, 10) 10 +------=�� 1 --- -------+-0 '---· 6 FIGURE 3.3 with boundary conditions y(O) = 1, Yr = 10, and T free. Here we have a horizontal terminal line as depicted in Fig. 3.3. The general solution of the Euler equation for this problem has previously been found to be a quadratic path (Sec. 2.2, Example 1): y*(t) = - {t2 + c1t + c2 Since y(O) = 1, we have c2 L But the other constant c1 must be defini­ tized with the help of the transversality condition (3. 1 1). Since F = ty' + y'2, so that Fy' = t + 2y', that condition becomes = ty' + y'2 - y' ( t + 2y') = (at t = T ) 0 which reduces to -y'2 0 or y' = 0 at t = T . That is, the extremal is required to have a zero slope at the terminal time. To meet this condition, we differentiate the general solution to get y*'(t), then set t = T, and let y*'(T) = 0. The result is the equation - T/2 + c1 = 0; it follows that c1 = Tj2. However, we still cannot determine the specific value of c1 without knowing the value of T. To find T, we make use of the information that Yr = 10 (horizontal terminal line). Substituting c 1 = T/2 and c2 = 1 into the general solution, setting t T, and letting the resulting expression take the stipulated value 10, we obtain the equation = = - iT 2 + iT2 + 1 = 10 The solution values of T are therefore ± 6. Rejecting the negative value, we end up with T* = 6. Then it follows that c1 = 3, and the extremal is y* ( t) = - it2 + 3t + 1 . �..: This time path is shown in Fig. 3.3. AB required by the transversality condition, the extremal indeed attains a zero slope at the terminal point (6, 10). Unlike in Example 1, where the transversality condition translates I ' 69 CHAPTER 3: TRANSVERSALITY CONDITIONS FOR VARIABLE-ENDPOINT PROBLEMS into an orthogonality requirement, the transversality condition in the pre­ sent example dictates that the extremal share a common slope with the given horizontal terminal line at t == T. Application to the Dynamic Monopoly Model Let us now consider the Evans model of a dynamic monopolist as a problem with a fixed terminal time T, but a variable terminal state Pr � Pmin· With the vertical terminal line thus truncated, the appropriate transversality condition is For simplicity, we shall in this discussion assign specific (3.17). numerical values to the parameters, for otherwise the solution expressions will become too unwieldy. Let the cost function and demand function be [ i .e . , a = -fo , {3 = 0, 'Y = 1000] [i . e . , a = 160, b = 8, h = 100] c = -foQ 2 + 1000 Q = 160 - BP + lOOP' ·. ..... : Then the profit function becomes 7T = PQ - C = 416P - 14.4P 2 + 260PP' - 1000P' 2 - 3200P' - 3500 which implies that (3.19) 7Tp• = 260P - 2000P' - 3200 This is the derivative needed in the transversality condition. :: . ··· Since the postulated parameter values yield the characterilltie , . and particular integral [see (2.36)]: p = 'mow · 14� the general solution of the Euler equation is (3.20) If we further assume the boundary conditions to be Pr = 15� then, according to (2.37), the constants T=2 and A1 and A2 should, in the fixed­ terminal-point problem, have the values (after rounding): A1 = 6.933 A 2 = - 9 .933 (3.20) P*(2) = 15� The reader can verify that substitution of these two constants into ' does (apart from rounding errors) produce the terminal price as required. 70 PART 2: THE CALCULUS OF VARIATIONS Now adopt a variable tenninal state PT � 10. Using the transversality condition (3.17), we first set the TTp· expression in (3.19) equal to zero at t = T = 2. Upon normalizing, this yields the condition ( 3.21 ) P'( T ) - 0.13P( T ) = - 1.6 The P(T) term here refers to the general solution P*(t) in (3.20) evaluated at t = T = 2. And the P'(T) is the derivative of (3.20) evaluated at the same point of time. That is, P(T ) = Ale o.24 + A 2 e -o.24 + 14� P' ( T ) = 0.12 A 1e0·24 - 0.12 A2 e-0 24 'l'hus, (3.21) can be rewritten more specifically as - 0.01A1e0·24 - 0.25A 2e -0·24 = 0.2778 To solve for A1 and A 2 , we need to couple this condition with. the oonmtioll that governs the initial point, A 1 + A2 = - 3 obtained from (3.20) by setting t = 0 and equating the result to P The solution values of A1 and A 2 turn out to be (after rounding): 0 and AI = 4.716 A 2 = - 7.716 = 11�. giving us the definite solution (3.22) P* ( t) = 4 . 716e0 ·121 - 7 .716e-0· 121 + 1 4� It remains to check whether this solution satisfies the terminal speci­ fication PT � 10. Setting t = T = 2 in (3.22), we find that P*(2) = 14.37. Since this does meet the stipulated restriction, the problem is solved.5 The Inflation-Unemployment Tradeoff Again The inflation-unemployment problem in Sec. 2.5 has a fixed terminal point that requires the expected rate of inflation, TT, to attain a value of zero at the given terminal time T: TT(T) = 0. It may be interesting to ask: What will happen if the terminal condition comes as a vertical terminal line instead? 5While Evan's original treatment does include a discussion of the variable terminal price, it neither makes use of the transversality condition nor gives a completely determined price path for that case. For his alternative method of treatment, see his Economics, McGraw-Hill, New York, 1930, pp. 150-153. Mathematical Introduction to 71 CHAPTER 3: TRANSVERSALITY CONDITIONS FOR VARIABLE-ENDPOINT PROBLEMS From (2.4 7), the general solution of the Euler equation is (3.23) = .,_ The initial condition, still assumed to be t 0 in the general solution) that (3.24) 'TT(O) = '!To T, [Fy,]t� T = F . = 2 ( 1 f3+z.ia2{321T' + �'!T)e-pt =0 j > .. 0, requires (by �ting ' ' H : With a vertical terminal line at the given however, we now must use the transversality condition 0. In the tradeoff model, this takes the form (3.25) Tf · (at t = T) which can be satisfied i f and only if the expression in the parentheses is zero. Thus, the transversality condition can be written (after simplification) as + = 0 (3.25') 'TT( T) 1T1(T) 1T'(T) a'TT( T) {3 2j where a = a 1 + a/32 The and expressions are, of course, to be obtained from the general solution 'TT* (t) and its derivative, both evaluated at t = T. Using (3.23), we can thus give condition (3.25') the specific form (3.25") When (3.24) and (3.25") are solved simultaneously, we finally find the definite values of the constants and A1 (3.26) A2 : 1To__:___; (rl ; + a)er•T A2 - (rl + a)er• T - (rz + a)e r•T ----= - ,----=-. These can be used to turn the general solution into the definite solution. Now that the terminal state is endogenously determined, what value of '!T* (T) emerges from the solution? To see this, we substitute (3.26) into the general solution (3.23), and evaluate the result at t The answer turns out to be = T. 'TT* (T) = 0 In view of the given loss function, this requirement-driving the rate of expected inflation down to zero at the terminal time-is not surprising. And PART 2: THE CALCULUS OF VARIATIONS 72 since the earlier version of the problem in Sec. 2.5 already has the target value of 7T' set at 7T'r = 0, the switch to the vertical-terminal-line formula­ tion does not modify the result. EXERCISE 3.2 1 For the functional V[y I = J;{(t2 + y'2) dt, the general solution to the Euler equation is y*(t) c1t + c2 (see Exercise 2.2, Prob. 1). (a) Find the extremal if the initial condition is y(O) = 4 and the terminal condition is T = 2, YT free. (b) Sketch a diagram showing the initial point, the terminal point, and the extremal. How will the answer to the preceding problem change, if the terminal condition is altered to: T = 2, YT :2: 3? Let the terminal condition in Prob. 1 be changed to: YT 5, T free. (a) Find the new extremal. What is the optimal terminal time T*? (b) Sketch a diagram showing the initial point, the terminal line, and the extremal. (a) For the functional V[y ) = f T(y'2 jt3) dt, the general solution to the Euler equation is y*(t ) = c 1 t2 + c2 (see Exercise 2.2, Prob. 4). Find the extremal(s) if the initial condition is y(O) = 0, and the terminal condition is YT = 3 - T . (b) Sketch a diagram to show the terminal curve and the extremal(s). The discussion leading to condition (3.16) for a truncated vertical terminal line is based on the assumption that p(T) > 0. Show that if the perturbing curve is such that p(T) < 0 instead, the same condition (3.16) will emerge. For the truncated-horizontal-terminal-line problem, use the same line of reasoning employed for the truncated vertical terminal line to derive transversality condition (3.18). = 2 3 4 5 6 = 3.3 THREE GENERALIZATIONS What we have learned about the variable terminal point can be generalized in three directions. A Variable Initial Point If the initial point is variable, then the boundary condition y(O) = A no longer holds, and an initial transversality condition is needed to fill the void. Since the transversality conditions developed in the preceding section can be applied, to the case of variable initial points, there is no need for further discussion. If a problem has the initial and terminal mutatis mutandis, both CHAPTER 3: TRANSVERSALITY CONDITIONS FOR VARIABLE-ENDPOINT PROBLEMS 73 points variable, then two transversality conditions have to be used to definitize the two arbitrary constants that arise from the Euler equation. The Case of Several State Variables When several state variables appear in the objective functional, so that the integrand is F(t , y l , · , yn , Y1 1 • · · • · ' , y'n ) the general (terminal) transversality condition (3.9) must be modified into . . +y:FYJL-r � T + [ FY {L= r �Y tr + . . · [ FY.: L - r �Yn r = 0 [ F - (y;Fy{ + (3.27) · + n= It should be clear that (3.9) constitutes a special case of (3.27) with 1. Given a fixed terminal time, the first term in (3.27) drops out because � T = 0. Similarly, if any variable Yi has a fixed terminal value, then �YiT 0 and the jth term in (3 . 27) drops out. For the terms that remain, however, we may expect all the � expressions to represent independently determined arbitrary quantities. Thus, there cannot be any presumption that the terms in (3.27) can cancel out one another. Consequently, each term that remains will give rise to a separate transversality condition. The following examples illustrate the application of (3.27) when 2, with the state variables denoted by y and z. The general transversality condition for 2 is = n= n= EXAMPLE 1 Assume that T is fixed, but Yr and Zr are both free. Then we have 0, but tlyr and tlzr are arbitrary. Eliminating the first term in (3.27') and setting the other two terms individually equal to zero, we get the transversality conditions !J. T = and which the reader should compare with (3.10). Suppose that the terminal values of y and · i Bre"� tib ·� ' 1 ·' ; .. ..; satisfy the restrictions EXAMPLE 2 ' Yr = c/J( T ) and Zr = , .·. ·: . ,. ' ·· · · 1/J(T) Then we have a pair of terminal curves. For a small following to hold: and ' tJ. T, we may expect the PART 2, 74 Using tlulse to eliminate t:.yr t:. zr and [ F + ( c/>' - y') Fy' + THE CALCULUS OF VARIATIONS in (3.27'), we obtain ( rf/ - z' ) F.·L� r t:.T = o Be� *T is arbitrary, the transversality condition emerges [F + ( c/>' - y')Fy + as ( rf/ - z') F.·L-r = 0 which the reader should compare with (3.12). With this transversality condition, the terminal curves Yr = cf>(T ) and Zr = 1/!(T), and the initial conditions y (O) = Yo and z(O) = z0 , we now have five equations to determine T* as well as the four arbitrary constants in the two Euler equations. The Case of Higher Derivatives When the functional V[y] has the integrand F(t, y, y', . . . , y<n >), the (termi­ nal) transversality condition again requires modification. In view of the rarity of high-order derivatives in economic applications, we shall state here the general transversality condition for the case of F(t, y, y', y" ) only: (3.28) [ F - y'Fy' - y"Fy" + y' :t Fy· ] t:.T + [ Fy. - t Fy" ] t:.yr + [ Fy"J r-r .t:.Y'r = O t-T d d t-T The new symbol appearing in this condition, t:.y'T• means the change in the terminal slope of the y path when it is perturbed. In terms of Fig. 3.1, t:.y'r would mean the difference between the slope of the AZ" path at Z" and the slope of the AZ path at Z. If the problem specifies that the terminal slope must remain unchanged, then t:.y'T = 0, and the last term in (3.28) drops out. If the terminal slope is free to vary, then the last term will call for the condition Fy'' = 0 at t T. = EXERCISE 3.3 1 ' For the functional V[y I = f{<y + yy y' + !y' 2 ) dt, the general solution of the Euler equation is y* ( t ) = !t2 + c1t + c (see Exercise 2.2, Prob. 3). 2 If we have a vertical initial line at t = 0 and a vertical terminal line at t = 1, write out the transversality conditions, and use them to definitize the constants in the general solution. + 2 Let the vertical initial line in the preceding problem be truncated with the restriction y*(O) 2: 1, but keep the terminal line unchanged. (a) Is the original solution still acceptable? Why? (b) Find the new extremal. 75 CHAPTER 3: TRANSVERSALITY CONDITIONS FOR VARIABLE-ENDPOINT PROBLEMS 3 In a problem with the functional f;{F(t, y, z, y', z') dt, suppose that y(O) A, z(O) = B, Yr = C, zr = D, T free (A, B, C, D are constants). = (a) How many transversality condition(s) does the problem require? Why? (b) Write out the transversality condition(s)_ 4 In the preceding problem, suppose that we have instead y(O) = A, z(O) Yr = C, and Zr = t/J(T), T free ( A , B, C are constants). = B, (a) How many transversality condition(s) does the problem require? Why? (b) Write out the transversality condition(s). 3.4 THE OPTIMAL ADJUSTMENT OF LABOR DEMAND Consider a firm that has decided to raise its labor input from L0 to a yet undetermined level Lr after encountering a wage reduction at t = 0. The adjustment of labor input is assumed to entail a cost C that varies with U(t), the rate of change of L. Thus the firm has to decide on the best speed of adjustment toward Lr as well as the magnitude of Lr itself. This is the essence of the labor adjustment problem discussed in a paper by Hamermesh.6 The Problem For simplicity, let the profit of the firm be expressed by the general function as illustrated in Fig. 3.4. The labor input is taken to be the sole determinant of profit because we have subsumed all aspects of production and demand in the profit function. The cost of adjusting L is assumed to be rr(L), with rr"(L) < 0, bL' 2 + k ( b > 0 , and k > 0 when L' * 0) Thus the net profit at any time is rr(L) - C(L'). (3 .29) C ( L' ) = The problem of the firm is to maximize the total net profit n over time during the process of changing the labor input. Inasmuch as it must choose not only the optimal Lr, but also an optimal time T* for completing the adjustment, we have both the terminal state and terminal time free. An­ other feature to note about the problem is that n should include not only the net profit from t 0 to t = T* (a definite integral), but also the capitalized value of the profit in the post-T* period, which is affected by the choice of Lr and T, too. Since the profit rate at time T is rr(Lr). its present value is rr(Lr)e-p1, where p is the given discount rate and T is to be set equal to T*- So the capitalized value of that present value is, = 6 Daniel S. Hamermesh, " Labor Demand and the Structure of Adjustment Costs," American Economic Review, September 1989, pp. 674-689. PART z, THE CALCULUS OF VARIATIONS 76 1C TC(L) 0 ..________ L FIGURE 3.4 :=::�=:�::��:;:� �:::·.::::�:�:w: I. (3.30) 0 L( O) L 0 L( T ) Lr = subject to and (L0 given) (Lr > L 0 free, T free) = p ' . If the last term in the functional were a constant, we could ignore it in the optimization process, because the same solution path would emerge either way. But since that term-call it z( T )-varies with the optimal choice of and T, we must explicitly take it into account. From our earlier discussion of the problem of Mayer and the problem of Bolza, we Lr have learned that z(T) (3.31) Thus, by defining z( t) = {0 z'(t) dt + z(O) 1 = -rr( L ) e -pt p so that we can write the post-T* term in (3.3Y ) _:rr ( Lr) e-pT p = = (3.30) as fT[ -rr(L) 0 z'(t) + [see (1 .7)] [ -rr( L) ] + ; 1T' (L)L' ] e -pt _:rr'(L ) L' e-pt dt + _:rr(L0) p p Substitution of (3.3Y) into the functional in (3.30) yields, after combining the two integrals, the new but equivalent function 77 CHAPTER 3: TRANSVERSALITY CONDITIONS FOR VARIABLE-ENDPOINT PROBLEMS While this functional still contains an extra term outside of the integral, that term is a constant. So it affects only the optimal n value, but not the optimal path, nO{ the optimal values of and T. L Lr The Solution To find the Euler equation, we see from [ -bL'2 - k + ;1T'(L)L}-pt F= Thus, from formula ( 3 .33) (3.32) that (2.18), we get the condition 1T'(L) L" - pL' + --2b = 0 [ Euler equation] The transversality condition in this problem is twofold. Both Lr and T being free, we must satisfy both transversality conditions (3.10) and (3.11).7 The condition [ Fvlt = T = 0 means that 1T'(L) = 0 L' - --2pb (3.34) And the condition [ F - L'FL']t-T ( 3.35) = or (at t = T) 0 means (after simplification) that ( at t = T) L where we take the positive square root because is supposed to increase from L0 to The two transversality conditions, plus the initial condition L(O) = £0, can provide the information needed for definitizing the arbitrary constants in the solution path, as well as for determining the optimal and T. In using (3.34) and (3 . 35), i t i s understood that the symbol refers to the derivative of the general solution of the Euler equation, evaluated at Lr. t = L' T. Lr 'IT(L). T o obtain a specific quantitative solution, i t i s necessary t o specify the form of the profit function Since our primary purpose of citing this example is to illustrate a case with both the terminal state and the terminal 7Technically, condition (3.17) for a truncated vertical terminal line should be used in lieu of (3. 10). However, since Lr sho ld be strictly greater than L0, the complementary-slackness u requirement would reduce (3.17) to (3. 10). PART 2o THE CALCULUS OF VARIATIONS 78 time free, and to illustrate the problem of Bolza, we · shall not delve into specific quantitative solution. EXERCISE 3.4 1 (a) From the two transversality conditions (3.34) and (3.35), deduce the location of the optimal terminal state L/ with reference to Fig. 3.4. (b) How would an increase in p affect the location of L/? How would an increase in b or k affect L/? (c) Interpret the economic meaning of your result in (b). 2 In the preceding problem, let the profit function be 1r ( L ) = (a) Find the value of L/. 2 mL - (O < n < m ) nL2 (b) At what L value does .,. reach a peak? (c) In light of (b), what can you say about the location of Lr* in relation to the .,.(L) curve? 3 (a) From the transversality condition (3.35), deduce the location of the optimal terminal time T * with reference to a graph of the solution path L*(t). (b) How would an increase in k affect the location of T * ? How would an increase in b affect T*? . . - ; .__ . CHAPTER 4 SECOND-ORDER CONDITIONS Our discussion has so far concentrated on the identification of the extremal(s) of a problem, without attention to whether they maximize or minimize the functional V[y ]. The time has come to look at the latter aspect of the problem. This involves checking the second-order conditions. But we shall also discuss a sufficient condition based on the concept of concavity and convexity. A simple but useful test known as the Legendre necessary conditions for maximum and minimum will also be introduced. 4.1 SECOND-ORDER CONDITIONS By treating the functional V[y] as a function V( E ) and setting the first derivative dVId t:. equal to zero, we have derived the Euler equation and transversality conditions as first-order nece�sary conditions for an extremal. To distinguish between maximization and minimization problems, we can take the second derivative d 2 Vjdt:. 2, and use the following standard second-order conditions in calculus: Second-order necessary conditions: [ for maximization of V ] [ for minimization of V] 79 80 PART 2: THE CALCULUS OF VARIATIONS Second-order sufficient conditions: d2V de2 < 0 d2V > de2 0 [for maximization of V] [for minimization of V] - The Second Derivative of V To find d2Vjde2, we differentiate the dVjde expression in (2.13) with respect to e, bearing in mind that (1) all the partial derivatives of F( t, y, y') are, like F itself, functions of t, y, and y', and (2) y and y' are, in turn, both functions of e, with derivatives ( 4 . 1) dy de = p(t) and dy de p'(t) - = ' [by (2.3) ] Thus, we have ( 4 2) . ( ) d zV = :!_ dV de2 de de = l[ : T :!..._ de }n{ [ FYp(t) + FY.p'(t)] dt 0 : ] = T p(t) e FY + p'(t) e Fy' dt In view of the fact that [by Leibniz's rule] [ by (4. 1 }] and, similarly, the second derivative (4 .2) emerges (after simplification) (4.2') lo as d2V de2 = T [ FyyP2(t) + 2Fyy•P (t)p'(t) + Fy'y'p'2(t)] dt .·,. ,·.�. ....., ,·;' The Quadratic-Form Test The second derivative in (4.2') is a definite integral with a quadratic form as its integrand. Since t spans the interval [0, T ], we have, of course, not one, but an infinite number of quadratic forms in the integral. Nevertheless, if it , , : . . .. ' 81 CHAPTER 4: SECOND-ORDER CONDITIONS can b e established that the quadratic form-with FYY ' Fyy'• and Fy'y' evalu­ ated on the extremal-is negative definite for every t, then < 0, and the extremal maximizes Similarly, positive definiteness of the quadratic form for every t is sufficient for minimization of V. Even if we can only establish sign semi definiteness, we can at least have the second­ order necessary conditions checked. For some reason, however, the quadratic-form test was totally ig­ nored in the historical development of the classical calculus of variations. In a more recent development (see the next section), however, the concav­ ity;convexity of the integrand function F is used in a sufficient condition. While concavity;convexity does not per se require differentiability, it is true that if the F function does possess continuous second derivatives, then its concavity1 convexity can be checked by means of the sign semidefiniteness of the second-order total differential of F. So the quadratic-form test definitely has a role to play in the calculus of variations. V. d 2VjdE 2 4.2 THE CONCAVITY I CONVEXITY SUFFICIENT CONDITION A Sufficiency Theorem for Fixed-Endpoint Problems Just as a concave (convex) objective function in a static optimization prob­ lem is sufficient to identify an extremum as an absolute maximum (mini­ mum), a similar sufficiency theorem holds in the calculus of variations: For the fixed-endpoint problem (2.1), if the integrand function is concave in the variables (y, y'), then the Euler ( 4 .3) equation is sufficient for an absolute maximum of V[y ]. Similarly, if F(t, y, y ' ) is convex in (y, y'), then the Euler equation is sufficient for an absolute minimum of V[y ]. F(t, y, y') It should be pointed out that concavity/convexity in (y, y ' ) means concav­ ity;convexity in the two variables y and y' jointly, not in each variable separately. We shall give here the proof of this theorem for the concave case.1 Central to the proof is the defining property of a differentiable concave function: The function F(t, y, y') is concave in (y, y') if, and only if, for any This proof is due to Akira Takayama. See his Mathematical Economics, 2d ed., Cambridge University Press, Cambridge, 1985, pp. 429-430. 1 PART 2o THE CALCULUS OF VARIATIONS 82 pair of distinct points in the domain, (t,y* , y*') and (t, y, y'), we have ( 4.4 ) F( t, y, y' ) - F( t,y* , y *') s Fy( t , y* , y*') (y - y* ) + Fy• ( t, y* , y*') (y' - y* ' ) Fy( t, y* , y*') €p ( t ) + Fy.( t, y * , y*')€p'( t ) [ by (2 .3)] Here y*(t) denotes the optimal path, and y(t) denotes any other path. By integrating both sides of (4.4) with respect to t over the interval [0, T ], we = obtain ( 4 .5) V[y ] - V[y* ] !T [ Fy( t, y* , y*')p( t ) + Fy. ( t, y* , y*' ) p' ( t ) j dt T [ d = € � p (t) Fy(t , y* , y*' ) - Fl( t, y * , y*' ) ] dt dt s € 0 [integration of the Fy• ( t, y* , y*') p' ( t ) term by parts as in (2. 16)] = 0 [ since y * ( t) satisfies the Eul�r equation (2.18)] In other words, V (y ] s V[y*), where y(t) can refer to any other path. We have thus identified y*(t) as a V-maximizing path, and at the same time demonstrated that the Euler equation is a sufficient condition, given the assumption of a concave F function. The opposite case of a convex F function for minimizing V can be proved analogously. Note that if the F function is strictly concave in (y, y' ), then the weak inequality ::;; in (4.4) and (4.5) will become the strict inequality < . The result, V[y] < V(y* ], will then establish V[y* ] to be a unique absolute maximum of V. By the same token, a strictly convex F will make V[y*) a unique absolute minimum. Generalization to Variable Terminal Point The proof of the sufficiency theorem (4. 3) is based on the assumption of fixed endpoints. But it can easily be generalized to problems with a vertical terminal line or truncated vertical terminal line. To show this, first recall that the integration-by-parts process in (2.16) originally produced an extra term [FY.p(t)Jr, which later drops out because it reduces to zero. The reason is that, with fixed endpoints, the perturbing curve p(t) is characterized by p(O) = p(T ) = When we switch to the problem with a variable terminal point, with T fixed but y(T) free, p(T) is no longer required to be zero. For this reason, we must admit an extra term 0. [by (2.3)] 83 CHAPTER 4: SECOND-ORDER CONDITJONS on the right-hand side of the second and the third lines of rearrangement, (4.5). Upon (4.5) now becomes FY. y(t) (y - y* ) where is to be evaluated along the optimal path, and represents the deviation of any admissible neighboring path from the optimal path y* (t). If the last term in the last inequality is zero, then obviously the original conclusion-that V[y* ] is an absolute maximum-still stands. Moreover, if the said term is negative, we can again be sure that [ Fiy - y* )]t=T [F/y - y* )]t-T· V[y * ] constitutes an absolute maximum. It is only when is posi­ tive that we are thrown into doubt. In short, the concavity condition on F(t, y, y') in (4.3) only needs to be supplemented in the present case by a nonpositivity condition on the expression But this supplementary condition is automatically met when the transversality condition is satisfied for the vertical-terminal-line problem, namely, [Fy.]t = T 0. As [Fy.Jt = T = calls for either for the truncated case, the transversality condition = 0 (when the minimum acceptable terminal value = Ym in (when that terminal value is binding, Ymin is nonbinding), or thereby in effect turning the problem into one with a fixed terminal point). y* Either way, the supplementary condition is met. Thus, if the integrand function F is concave (convex) in the variables (y, y') in a problem with a vertical terminal line or a truncated vertical terminal line, then the Euler equation plus the transversality condition are sufficient for an absolute maximum (minimum) of V[y ]. Checking Concavity ; convexity For any function (, whether differentiable or not, the concavity feature can be checked via the basic defining property that, for two distinct points and v in the domain, and for 0 < 8 < 1, F Of( u ) + ( 1 - 0) f( v ) For convexity, the inequality � is concave if and only if u f [ O u + ( 1 - O)v] � i s reversed t o � . Checking this property can, however, be very involved and tedious. For our purposes, since F(t, y, y') is already assumed to have continuous second derivatives, we can take the simpler alternative of checking the sign definiteness or semidefiniteness of the quadratic form ( 4 .6) This quadratic form can, of course, be equivalently rewritten (4.6' ) as 84 PART 2: THE CALCULUS OF VARIATIONS which would fit in better with the ensuing discussion of the Legendre condition. Once the sign of the quadratic form is ascertained, inferences can readily be drawn regarding concavity ;convexity as follows: The y, y') function is concave (convex) in (y, y') if, and only if, the quadratic form q is everywhere negative (positive) semidefinite; and the function is strictly concave (strictly convex) if (but not only if) q is everywhere negative (positive) definite. It should be noted that concavity1convexity is a global concept. This is why it is related to absolute extrema. This is also why the quadratic form q is required to be everywhere negative (positive) semidefinite for concavity (convexity) of F, meaning that the sign semidefiniteness of q should hold for all points in the domain (in the yy' space) for all t. This is a stronger condition than the earlier-mentioned quadratic-form criterion relating to (4.2'), because in the latter, the second derivatives of F are to be evalu­ ated on the extremal only. From another point of view, however, the concavity;convexity sufficient condition is less strong; it only calls for sign semi definiteness, whereas, for (4.2'), a sufficient condition would require a definite sign. The sign definiteness and semidefiniteness of a quadratic form can be checked with determinantal and characteristic-root tests. Since these tests are applicable not only in the present context of the calculus of variations, but also in that of optimal control theory later, it may prove worthwhile to present here a self-contained explanation of these tests. F F(t, The Determinantal Test for Sign Definiteness The determinantal test for the sign definiteness of the quadratic form the simplest to use. We first write the discriminant of q, ( 4.7) IDI - J FFy'y' yy' = q is [ from ( 4.6' )] and then define the following two principal minors: and ( 4.8) The test is ( 4.9) Negative defmiteness of q Positive definiteness of q ID11 < 0, ID2 1 > o ID11 > 0, ID2 1 > 0 (everywhere in the domain) 85 CHAPTER 4: SECOND-ORDER CONDITIONS Though easy to apply, this test may be i:werly stringent. It is inttmded for strict concavity;convexity, whereas the sufficient condition (4.3) merely stipulates weak concavity ;convexity. The Determinantal Test for Sign Semidefiniteness The determinantal test for the sign semi definiteness of q requires a larger number of determinants to be examined, because under this test we must consider all the possible ordering sequences in which the variables of the quadratic form can be arranged. For the present two-variable case, there are two possible ordering sequences for the two variables-(y' , y) and (y, y'). Therefore, we need to consider only one additional discriminant besides (4.7), namely, ( 4 . 10) IDol =I FyFYY'y FyFyy'y'' I [from ( 4.6)] whose principal minors are (4.11) ID01 I = I Fy) Fyy = ID021 and = I FyFYY'y FyyFy'y' 'I ID 1 1 ID02I For notational convenience, let us refer to the 1 X 1 principal minors and and collectively as I D 1 I , and the 2 X 2 principal minors collectively as D . Further, let us use the notation I Di l � 0 to mean that each member of I Di l is � 0. Then the test for sign semidefiniteness is as follows: ID01 I ( 4 . 12) I 21 ID21 Negative semidefiniteness of q l D 1 1 5: o , Positive semidefiniteness of q ID I I � ID2 1 �o 0 , ID21 �0 ( everywhere in the domain), The Characteristic-Root Test The alternative to the use of determinants is to apply the characteristic-root test, which can reveal sign semidefiniteness as well as sign definiteness all at once. To use this test, first write the characteristic equation [ from ( 4 . 7 ) ] ( 4 . 13) Then sobe for the characteristic roots r1 and r2 • The test revolves aro\Uld 86 PART 2: THE CALCULUS OF VARIATIONS the signs of these roots as follows: Negative definiteness of q Positive definiteness of q Negative semidefiniteness of q < = r1 = r1 > 0, r2 > 0 = r1 s 0 , r2 s 0 0, r2 < 0 ( at least one root = 0) Positive semidefiniteness of q (at least one root = 0) For this test, it is useful to remember that the two roots are tied to the determinant I D2 1 in (4.8) by the following two relations:2 ( 4.15) ( 4 . 1 6) r1 + r2 = sum of principal-diagonal elements = Fy'y' + FYY r1r2 = ID2 I For one thing, these two relations provide a means of double-checking the calculation of r1 and r2• More importantly, they allow us to infer that, if ID2 1 < 0, then r1 and r2 must be opposite in sign, so that q must be indefinite in sign. Consequently, once ID2 1 is found to be negative, there is no need to go through the steps in (4.13) and (4.14). The shortest-distance problem (Example 4 of Sec. 2.2) has • the integrand function F = (1 + y '2)112 Since there is only one variable, y ' , in this function, it· is easy to check the second-order condition. In so doing, bear in mind that the square root in the F expression is a positive square root because it represents a distance, which cannot be negative. This point becomes important when we evaluate the second derivative. The first derivative of F is, by the chain rule, EXAMPLE 1 2 - 1 /2 2y' Fy' = i { 1 + y' ) = (1 + y'2 ) - 1/2 ' y Further differentiation yields - 1/2 r2 - 3/2 r2 Fh' = - (l + y ) y + ( l + yr2 ) /2 = (1 + y'2 ) - 3 [ - y'2 + (1 + y' 2 ) ] = (1 + y'2 ) -3 2 /2 >0 [positive square root] See the discussion relating to local stability analysis in Sec. 18.6 of Alpha C. Chiang, Fundamental Methods of Mathematical Economics, 3d ed., McGraw-Hill, New York, 1984. 87 CHAPTER 4: SECOND-ORDER CONDITIONS The positive sign for Fy'y'-positive for all y' values and for all t-means that F is strictly convex in the only variable y' everywhere. We can thus conclude that the extremal found from the Euler equation yields a unique minimum distance between the two given points. EXAMPLE 2 Is the Euler equation sufficient for maximization or mini­ mization if the integrand of the objective functional is F(t, y , y') = 4y 2 + 4Y.Y' + y'2? To check the concavity;convexity of F, we first find the second derivatives of F. Since FY = By + 4y' and Fy' = 4y + 2y' the required second derivatives are Fy'y' =2 Thus, with reference to (4.8), we have The quadratic form q is not positive definite. However, with reference to (4.11), we find that So the condition for positive semidefiniteness in (4.12) is satisfied. To illustrate the characteristic-root test, we first write the chtp"acteris­ tic equation I 4 r2 8 - r = - lOr = 0 r1 = 10 and r2 = 0. [This result, incidentally, Its two roots are confirms and (4.16)_) Since the roots are unconditionally nonnegative, q is everywhere positive semidefinite according to (4.14). It follows that the F function is convex, and the Euler equation is indeed sufficient for the minimization of V[y ]. (4.15) EXAMPLE 3 Let us now check whether the Euler equation is sufficient for profit maximization in the dynamic monopoly problem of Sec. 2.4. The profit function (2.33) gives rise to the second derivatives 7T'p•p• = - 2ah2 7T'p•p = 7T'pP' = h ( l + 2ab) 7T'pp = - 2b(l + ab) Using the determinantal test (4.9), we see that even though here) is negative as required for the negative definiteness of q, ID1 1 ( rrp•p• ID 2 1 is also 88 negative: PART 2: THE CALCULUS OF VARIATIONS 17Tp• 7Tpp•p• Hence, by (4. 16), the characteristic roots r1 and r2 have opposite signs. The F function is not concave, and the Euler equation is not sufficient for profit maximization. As we shall see in the next section (Example 3 ), however, the problem does satisfy a second-order necessary condition for a maximum. :;�:�:::.:::.::::�:�l= b!&, the oomavity ond oonvoxity sufficiency conditions are still applicable. But, in that case, the F function must be concavejconvex (as the case may be) in all the n variables and their derivatives (y1, , yn, y{, . . . , y;), jointly. For the characteristic-root test, this would mean a higher-degree polynomial equation to solve. Once the roots are found, however, we need only to subject all of them to sign restrictions similar to those in (4. 14). In the determinantal test for sign definiteness, there would be a larger discriminant with more principal minors to be checked. But the technique is well known. a With regard to the determinantal test for sign semi definiteness, the procedure for the n-variable case would become more complex. This is because the number of principal minors generated by different ordering sequences of the variables will quickly multiply. Even with only two state variables, say, (y, z), concavity;convexity must now be checked with regard to as many as four variables (y, z, y', z' ). For notational simplicity, let us number these four variables, respectively, as the first, second, third, and fourth. Then we can arrange the second-order derivatives of F into the following 4 X 4 discriminant: • • . (4. 17) IDI = Fu F21 Fal F•1 F12 F22 F1a F2a Fa2 F42 Faa F4a F1• F2• Fa4 F44 = FYY Fzy Fy'y Fz'y Fyz F•• Fy'z F • • • Fyy' Fzy' Fyz' F••• Fy'y' Fz'y' Fy'z' Fz'z' •1f.· ·_ ... ,.: _ . Since each of the variables can be taken as the '' first" variable in a difterent 3See, for example, Alpha C. Chiang, Fundamental Methods of Mathematical Economics, 3d ed., McGraw-Hill, New York, 1984, Sec. 1 1 .3. -� 89 CHAPTER 4: SECOND-ORDER CONDITIONS 1X 1 ordering sequence, there exist four possible principal minors: ( 4.18) 11\ 1 - We shall denote these 1 X 1 principal minors collectively by The total number of possible 2 X 2 principal minors is 12, but only six distinct values can emerge from them. These values can be calculated from the following six principal minors: ( 4 . 19) I I FY• F•• FYY F.Y I I I I F.. Fzy' Fy' z Fy'y' FYY Fyy' Fy'y Fy'y' F.. F • • • F••. F • • • • I I I I FYY Fyz' Fz'y Fz'z' Fy'y' Fz'y' I Fy'z' Fz'z' l i·.' The reason why the other six can be ignored is that the principal minor with in the diagonal is always equal in value to the one with the two variables in reverse order, ( Fj1, Fi), in the diagonaL The 2 X 2 principal � minors in (4.19) will be collectively referred to as As to 3 X 3 principal minors, the total number available is 24. From these, however, only four distinct values can arise, and they can be calcu­ lated from the following principal minors: (Fii• Fj) I D2 1 . ( 4.20) >. Fyz F., F Fyz F•• Fy'z Fyy' Fzy' Fy'y' FYY Fzy Fz'y FYY Fyy' Fy'y Fy'y' Fz'y Fz'y' Fyz' Fy'z' Fz'z' F•• Fzy' F.,. Fy'z Fy'y' Fy'z' Fz'y' F F, FYY Fzy Fy'y • • • • Fyz' F Fz'z' • •• • • • • • We can ignore the others -because the principal minor with ( Fu , Fj1 , Fkk) in the diagonal always has the same value as those with F;i, Fj1 , Fkk arranged in any other order in the diagonal. We shall refer to the principal minors in (4.20) collectively as Finally, we need to consider only one 4 X 4 principal minor, in (4.17). which is identical in value with One question naturally arises: How does one ascertain the number of distinct values that can arise from principal minors of various dimensions? The answer is that we can just apply the formula for the number of combinations of n objects taken r at a time, denoted by C;': ID3 1 . I D4 1 , ID I ( 4.21) e n = r ----­ n! r!( n - r) ! ' . . .. ·· .·.• ' ...; .. .. 90 PART 2: THE CALCULUS OF VARIATIONS For (4.18), we have four variables taken one at a time; thus the number of combinations is ct = For 4! -4 1 ! 3! -- 2 X 2 principal minors, w e have .. -� ; '� - 4 c2 . = 4! 2, 2 , = s .Ud for 3 X 3 principal minors, the formula yields 4! c� = 3, 1 ! =4 4 principal minors, a unique V!Uue �. becaue Finally, for 4 X C4 = 4 4! 4! 0! -- = · ·-:;: .;;" 1 Once all the principal minors have been calculated, the test for sign semidefiniteness is similar to (4.12). Using the notation ID; I � 0 to mean that each member of ID; I is � 0, we have the following: Negative semidefiniteness of q � ( 4.22) ID1 1 ::; o, ID2 1 � o, ID31 ::; o, ID4 1 �o � o, � o, ID31 � o, �o Positive semidefiniteness of q ID1 1 ID21 ID41 ( everywhere in the domain) EXERCISE 4.2 1 I J For Prob. 1 of Exercise 2.2: F function is strictly concave/convex in (y,y') by (a) Check whether the (b) the determinantal test (c) (4.9). If that test fails, check for concavity1convexity by the determinantal test (4.12) or the characteristic-root test. Is the sufficient condition for a maximum/minimum satisfied? Perform the tests mentioned in Prob. Perform the tests mentioned in Prob. 1 above on Prob. 3 of Exercise 2.2. 1 above on Prob. 5 of Exercise 2.2. 91 CHAPTER 4: SECOND-ORDER CONDITIONS 4 (a) Show that, i n the inflation-unemployment tradeoff model of Sec. 2.5, the integrand function is strictly convex. ( b ) What specifically can you conclude from that fact? 4.3 THE LEGENDRE NECESSARY CONDITION The concavity feature described in (4.4) is a global concept. When that feature is present in the integrand function F, the extremal is guaranteed to maximize V[y]. But when F is not globally concave, as may often happen (such as in the dynamic monopoly problem), we would have to settle for some weaker conditions. The same is true for convexity. In this section, we introduce a second-order necessary condition known as the Legendre condi­ tion, which is based on local concavity;convexity. Though not as powerful as a sufficient condition, it is very useful and indeed is used frequently. The Legendre Condition The great merit of the Legendre condition lies in its extreme simplicity, for it involves nothing but the sign of Fy'y'· Legendre, the mathematician, thought at one time that he had discovered a remarkably neat sufficient condition: Fy'y' < 0 for a maximum of V, and Fy'y' > 0 for a minimum of Y. But, unfortunately, he was mistaken. Nonetheless, the weak-inequality version of this condition is indeed correct as a necessary condition: ( 4.23 ) Maximization of V[y] � Fy'y' � 0 for all t e [0, T ] Minimization of V[y] � Fy'y' ;;:: 0 for all t e [0, T ] [Legendre necessary condliOD] The Fy'y' derivative is to be evaluated along the extremal. The Rationale To understand the rationale behind (4.23), let us first transform the middle term in the integrand of (4.2') into a squared term by integration by parts. Let v = Fyy' and u = p2(t), so that dv = dFyy' dt dt -- and du = 2p( t)p'(t) dt 92 PART 2 : THE CALCULUS OF VARIATIONS Then the middle term in (4.2') can be rewritten as ( 4 .24) 0 = -! 0 T dFyy ' dt p 2 ( t) dt [ p( O ) = p(T) where all the limits of integration refer to values of t (not this into (4.2') yields ( 4.25) d2V = de2 ! [( T 0 FYY - dFyy' -;{t ) p 2 ( t) u + Fh'p' 2 ( t) ] = 0 assumed] or v ) . Plugging dt where the integrand now consists of two squared terms p2(t) and p'2(t). It turns out the p'2(t) term will dominate the other one over the interval [0, T ], and thus the honpositivity (nonnegativity) of its coefficient Fy'y' is necessary for the nonpositivity (nonnegativity) of d2Vjd e 2 for maximization (minimization) of V, as indicated in (4.23). But because of the presence of the other term in (4.25), the condition is not sufficient. That the p'2(t) term will dominate can be given an intuitive explanation with the help of the illustrative perturbing curves in Fig. 4.1. Figure 4.1a shows that p'(t), the slope of p(t), can take large absolute values even if p(t) maintain small values throughout; Fig. 4.1b shows that in order to attain large p(t) values, p'(t) would in general have to take large absolute values, too. These considerations suggest that the p' 2(t) term tends to dominate. Note that, to reach the result in (4.25), we have made the assumption that p(O) = p(T ) = 0 on the perturbing curve. What will happen if the terminal point is variable and p(T ) is not required to be zero? The answer p(t) p(t) ' . �a) FIGVU 4.1 , , .· (b) 93 CHAPTER 4: SECOND-ORDER CONDITIONS [Fyy'p 2(t)lt�T· is that the zero term on the last line of (4.24) will then be replaced by the new expression But even then, it is possible for some admissi­ ble neighboring path to have its terminal point located exactly where is, thereby duplicating the result of = 0, and reducing that new expression to zero. This means that the eventuality of (4.25) is still possible, and the Legendre condition is still needed. Hence, the Legendre necessary condition is valid whether the terminal point is fixed or variable. y*(T) p(T) EXAMPLE 1 In the shortest-distance problem (Example 1, preceding sec­ tion), we have already found that Fy'y' (1 + y' = 2 ) - 3/ 2 > 0 Thus, by (4.23), the Legendre necessary condition for a minimum is satis­ fied. This is only to be expected, since that problem has earlier been shown to satisfy the second-order s ufficient condition for minimization. EXAMPLE 2 In Example 3 of Sec. 2.2, we seek to find a curve between two fixed points that will generate the smallest surface of revolution. Does the extremal (pictured in Fig. 2.4) minimize the surface of revolution? Since that problem has FY. = yy'( l 2 ) - 1/2 +y ' we can, after differentiating and simplifying, get Fy'y' y( 1 + ,2) -3/2 = (1 + y'2) -312 y expression is positive, and y-represented by the height The of the AZ curve in Fig. 2.4-is positive along the entire path. Therefore, is unconditionally positive for all t in the interval and the Legendre condition for a minimum is satisfied. To make certain that is indeed positive for all t, we see from the general solution of the problem Fy'y' [a, z], y(t) y* ( t) = c [ e<t+k)/c + e-(t+k)/c] 2 y*(t) that takes the sign of the constant c, because the two terms in parentheses are always positive. This means that can only have a single sign throughout. Given that points A and Z show positive y values, must be positive for all t in the interval Unlike the shortest-distance problem, however, the minimum-surface­ of-revolution problem does not satisfy the sufficient condition. y*(t) y*(t) [a, z]. : :;; . 94 PART 2, THE CALCULUS OF VARIATIONS EXAMPLE 3 For the dynamic monopoly problem, we have found in the preceding section that the integrand function rr(P, P') is not concave, so it does not satisfy the sufficient condition (4.3). However, since rrp'p' = - 2ah2 < 0 the Legendre necessary condition for a maximum is indeed satisfied. The n-Variable Case The Legendre necessary condition can, with proper modification, also be applied to the problem with n state variables (y v . . . , yn ). But instead of checking whether Fi:l � 0 or F11 � 0, as in (4.23), we must now check whether the n X n matrix [ Fy; 'y ']-or, alternatively, the quadratic form whose coefficients are FY/Y _.-is negative semidefinite (for maximization of 1 V) or positive semidefinite (for minimization of V). For this purpose, first define ( 4 .26) where the second-order derivatives are to be evaluated along the extremals. Then write all the 1 X 1 principal minors I � 1 1 , all the 2 X 2 principal minors ) A 2 ) , and so forth, as explained in the text following (4.17). The Legendre second-order necessary condition is ( 4.27) for all t E [0, T ] for all t E [0, T ] [Legendre necessary condition] It is clear that (4.27) includes (4.23) as a special case. Note that the Legendre condition in (4.27), though apparently similar to the determinantal test for sign semidefiniteness in (4.22), differs from the latter in two essential respects. First, the sign-semidefiniteness test in (4.22) involves derivatives of the FYY and Fyy' types as well as Fy'y'• but the Legendre condition is based exclusively on the Fii type. This is why we employ the symbol ! Il l in (4.26) and (4.27), so as to keep it distinct from the IDI symbol used earlier in (4.22). Second, unlike the global concavity;con­ vexity characterization, the Legendre condition is local in nature; so the second-order derivatives in (4.26) are to be evaluated along the extremals only. 95 CHAPTER 4: SECOND-ORDER CONDITIONS EXAMPLE 4 In Exercise 2.3, Prob. integrand function F = y'2 Since Fy' = 2y' + z' 2, two ' state + z 2 + y'z' F2• and = (4.26) results in IAI 2z' + y' Fz'z' we find that Substitution of these into variables appear in the =2 = li �� In this case, we find that and Thus, by 1 �I = 2 I A2 1 = 1 3 (4.27), the Legendre condition for a minimum is satisfied. EXERCISE 4.3 4.4 1 Check whether the extremals obtained in Probs. 6, 8, and 9 of Exercise 2.1 satisfy the Legendre necessary condition for a maximum/minimum. 2 Check whether Prob. 3 of Exercise 2.3, a two-variable problem, satisfies the Legendre condition for a maximum/minimum. 3 Does the inflation-unemployment problem of Sec. 2.5 satisfy the Legendre condition for minimization of total social loss? FffiST AND SECOND VARIATIONS Our discussion of first-order and second-order conditions has hitherto been based on the concept of the first derivative dVIdE and the second derivative d2VjdE2• An alternative way of viewing the problems centers around the concepts of first variation and second variation, which are directly tied to the name calculus of variations itself. The calculus of variations involves a comparison of path values V[y] and V[y* ]. The deviation of V[y] from V[y* ] is (4.28) AV = V [y] - V ( y* ] = fTF(t , y , y') dt - f TF(t,y* , y*' ) dt 0 \. i 0 96 PART 2: THE CALCULUS OF VARIATIONS When the integrand of the first integral is expanded into a Taylor series around the point it would contain a term which would allow us to . cancel out the second integral in (4.28). That Taylor series is (t, y*, y*'), F(t, y*, y*'), F(t,y,y') F(t,y* ,y*') + [ �(t - t) + Fy( y y*) + Fy• (Y - y*')] ' 1 + -d Fu(t - t)2 + Fyy(Y - y*)2 + Fy•y• (Y' - y*')2 2. +2Fty(t - t)(y - y* ) + 2Fty•( t - t)(y' - y*') + 2 F y ( Y - y*)(y' - y*') ] + + Rn where all the partial derivatives of F are to be evaluated at (t,y*,y*'). We have, for the sake of completeness, included the terms that involve (t - t), ( 4.29) = - ' y • · · · but, of course, these would all drop out. Recalling (2.3), we can substitute y - y* = ep, and y' - y*' ep' , to get (4 .29') F(t,y, y') F(t,y*,y*') + Fyep + FY.ep' 1 + -d Fyy( ep )2 + Fy'y'( ep')2 + 2Fyy• ( ep )( ep') ] 2. + . . . + Rn = = .•·i Using thi.41 result in (4.28), we can then transform the latter into ( 4 .30) € 2 fT( FyyP2 + Fy'y'P,2 +- 2 0 + 2F:trPJI ) dt + h .o.t. (higher-order-terms) In the calculus-of-variations literature, the first integral in referred to as the denoted by (4.30) is first variation, 8V: ( 4 .3 1 ) BV = �T( F:tp + Frp') dt dV de [by (2. 13) ] Similarly, the second integral in (4.30) is known the second .variation, denoted by 8 2 V: = as [by ( 4 .2')] - . -·· <" . . . ... ' CHAPTER 4: SECOND-ORDER CONDITIONS 97 In a maximization problem, where .1V :,; 0, it is clear from (4.30) that it is necessary to have 8V = 0, since e can take either sign at every point of time. This is equivalent to the first-order condition dV1d e = 0, which in our earlier discussion has led to the Euler equation. Once the 8V = 0 condition is met, it is further necessary to have 8 2V :,; 0, since the coeffi­ cient e 2 /2 is nonnegative at every point of time. This condition is equiva­ lent to d 2 Vjd e 2 :,; 0, the second-order necessary condition of Legendre. Analogous reasoning would show that, for a minimization problem, it is necessary to have 8V = 0 and 8 2 V � 0. In sum, the first-order and second-order necessary conditions can be derived from either the derivative route or the variation route. We have chosen to follow the derivative route because it provides more of an intuitive grasp of the underlying reasoning process. CHAPTER 5 INFINITE PLANNING HORIZON ' ' For an individual, it is generally adequate to plan for a finite time interval [0, T ], for even the most far-sighted person probably would not plan too far beyond his or her expected lifetime. But for society as a whole, or even for a corporation, there may be good reasons to expect or assume its existence to be permanent. It may therefore be desirable to extend its planning horizon indefinitely into the future, and change the interval of integration in the objective functional from [0, T ] to [0, oo]. Such an extension of horizon has the advantage of rendering the optimization framework more comprehen­ sive. Unfortunately, it also has the offsetting drawback of compromising the plausibility of the often-made assumption that all parameter values in a model will remain constant throughout the planning period. More impor­ tantly, the infinite planning horizon entails some methodological complica­ tions. 5.1 METHODOLOGICAL ISSUES OF INFINITE HORIZON With an infinite horizon, at least two major methodological issues must be addressed. One has to do with the convergence of the objective functional, and the other concerns the matter of transversality conditions. 98 99 CHAPTER 5: INFINITE PLANNING HORIZON The Convergence of the Objective Functional The convergence problem arises because the objective functional, now in the form of J;F(t, y, y') dt, is an improper integral which may or may not have a finite value. In the case where the integral diverges, there may exist more than one y(t) path that yields an infinite value for the objective functional and it would be difficult to determine which among these paths is optimal. 1 It is true that even in the divergent situation, various ways have been proposed to make an appropriate selection of a time path from among all the paths with an infinite integral value. But that topic is complicated and, besides, it does not pertain to the calculus of variations as such, so we shall not delve into it here.2 Instead, as a preliminary to the discussion of economic applications in the two ensuing sections, we shall make some comments on certain conditions that are (or allegedly are) sufficient for convergence. Condition I Given the improper integral J;F<t, y, y ' ) dt, if the integrand F is finite throughout the interval of integration, and if F attains a zero value at some finite point of time, say, t0, and remains at zero for all t > t0, then the integral will converge. This is in the nature of a sufficient condition. Although the integral nominally has an infinite horizon, the effective upper limit of integration is a finite value, t0• Thus, the given improper integral reduces in effect to a proper one, with assurance that it will integrate to a finite value. Condition II (False) Given the improper integral J;F(t, y, y' ) dt, if F � 0 as t � oo, then the integral will converge. This condition is often taken to be a sufficient condition, but it is not. To see this, consider the two integrals =f 00 (5.1) I1 0 (t 1 + 1)2 dt and [ 2 = f0 t 1 1 dt 00 + -- Each of these has an integrand that tends to zero as t --> oo. But while 11 1For a detailed discussion of the problem of convergence, see S. Chakravarty, "The Existence of an Optimum Savings Program," Ecorwmetrica, January 1962, pp. 178-187. 2 The various optimality criteria in the divergent case are concisely summarized in Atle Seierstad and Knut Sydsreter, Optimal Control Theory with Economic Applications, Elsevier, New York, 1987, pp. 231-237. 100 PART 2: THE CALCULUS OF VARIATIONS converges, I2 does not: (5.2) /1 = lim b - oo [ -1 ] b 1 f+1 -­ 12 O = lim ln ( t b-+oo [ 1)]� = oo + What accounts for this sharp difference in results? The answer lies i n the speed at which the integrand falls toward zero. In the case of /1, where the denominator of the integrand is a squared term, the fraction falls with a sufficiently high speed as convergence. In the case of t takes increasingly larger values, resulting in /2 , on the other hand, the speed of decline is not great enough, and the integral diverges. 3 The upshot is that, in and of itself, the condition "F -> 0 t -> oo" as does not guarantee convergence. It is of interest also to ask the converse question: Is the condition " F -> 0 as t -> oo" a necessary condition for J';F(t, y, y') dt to converge? Such a necessary condition seems intuitively plausible, and it is in fact 4 commonly accepted as such. But counterexamples can be produced to show that an improper integral can converge even though F does not tend to zero, so the condition is not necessary. For instance, if F( t) � {: if n :<;; t :<;; n + 2n1 (n = 1,2, ... ) otherwise � F(t) th n has no limit as t becomes infinite; yet the integral converges to the magnitude J; sin t2 dt I:J: _ 1(1jk 2). 5 As J;F(t) dt another example, the integral can be shown to converge to the value /2 4 ../rr sine-function integrand does not have zero as its limit. 6 even though the Note, however, that these counterexamples involve either an unusual type of discontinuous integrand or an integrand that periodically changes sign, thereby allowing the contributions to the integral from neighboring time intervals to cancel out one another. Such functions are not usually 3 For an integral J;F(t) dt whose integrand maintains a single sign (say, positive), the following criterion can be used: The integral converges if F(t) vanishes at infinity to a higher order than the first, that is, if there exists a number a > 1 such that, for all values of matter how large), 0 < F(t) F(t) :S Mjta holds, where M is a constant. The integral t (no diverges if vanishes at infinity to an order not higher than the first, that is, if there is a positive constant N such that tF(t) " N > 0. See R. Courant, Differential and Integral Calculus, translated from German by E. J. McShane, 2d ed., lnterscience, New York, 1937, Vol. 1, pp. 249-250. 4 See, for example, S. Chakravarty, " The Existence of an Optimum Savings Program," Econo­ metrica, January 1962, p. 182, and G. Hadley and M. C. Kemp, Variational Methods in Economics, North-Holland, Amsterdam, 1971, pp. 52, 63. 5 See Watson Fulks, Mvanced Calculus: An Introduction to Analysis, 3d ed., Wiley, New York, 1978, pp. 570-571 . 6 See R. Courant, Differential and Integral Calculus, lnterscience, New York, Vol. 1 , p . 253. 101 CHAPTER 5: INFINITE PLANNING HORIZON used in the objective functionals in economic models. Typically, the inte­ grand in an economic model-representing a profit function, utility func­ tion, and the like-is assumed to be continuous and nonnegative. For such functions, the counterexamples would be irrelevant. Condition III In the integral J�F(t , y, y') dt, if the integrand takes the form of G(t, y, y')e -P1, where p is a positive rate of discount, and the G function is bounded, then the integral will converge. A distinguishing feature of this integral is the presence of the discount factor e - pt which, ceteris paribus, provides a dynamic force to drive the integrand down toward zero over time at a good speed. When the G(t, y, y') component of the integrand is positive (as in most economic applications) and has an upper hound, say, G, then the downward force of e -pt is sufficient to make the integral converge. More formally, since the value of the G function can never exceed the value of the constant G, we can write ( 5 .3) f G ( t, y , y' ) e -pt dt f Ge - 1 dt 00 0 ;s; 00 0 " = (; p - The last equality, based on the formula for the present value of a perpetual constant flow, shows that the second integral in (5.3), with the upper bound (; in the integrand, is convergent. It follows that the first integral, whose integrand is G ;s; G, must also be convergent. We have here another suffi­ cient condition for convergence. There also exist other sufficient conditions, but they are not simple to apply, and will not be discussed here. In the following, if the F function is given in the general form, we shall always assume that the integral is convergent. This would imply, for normal economic applications, that the integrand function F tends to zero as t tends to infinity. When specific functions are used, on the other hand, convergence is something to be explicitly checked. The Matter of Transversality Conditions Transversality conditions enter into the picture when either the terminal time or the terminal state, or both, are variable. When the planning horizon is infinite, there is no longer a specific terminal T value for us to adhere to. And the terminal state may also be left open. Thus transversality conditions are needed. To develop the transversality conditions, we can use the same proce­ dure as in the finite-horizon problem of Chap. 3. Essentially, these condi­ tions would emerge naturally as a by-product of the process of deriving the Euler equation, as in (3.9), which emanates from (3.7). In the present ""!! 102 ·- · � � M V-=� context, (3.9) must be modified to where each of the two terms must individually vanish. Since there is no fixed T in the present context, !::. T is perforce nonzero, and this necessitates the condition ( 5 .5) lim ( F t � O> - y'Fy') = 0 [transversality condition for infinite horizon] This condition has the same economic interpretation as transversality condition (3.11) in the finite-horizon framework. If the problem pertains to a profit-maximizing firm, for instance, the condition requires the firm to take full advantage of all profit opportunities. As to the second term in (5.4), if an asymptotic terminal state is specified in the problem: ( 5 .6) lim y(t) = y.. = a given constant t � ac [which is the infinite-horizon counterpart of y(T) = Z in the finite-horizon problem], then the second term in (5.4) will vanish on its own (!::. yr = 0) and no transversality condition is needed. But if the terminal state is free, then we should impose the additional condition ( 5 . 7) J' :j i ( 5.4) ( transversality condition for free terminal state] This condition can be given the same economic interpretation as transver­ sality condition (3.10) for the finite-horizon problem. If the problem is that of a profit-maximizing firm with the state variable y representing capital stock, the message of (5. 7) is for the firm to use up all its capital as t --> oo. It may be added that if the free terminal state is subject to a restric­ tion such as y.. � Y min• then the Kuhn-Tucker conditions should be used [cf. (3. 17) and (3.17')]. In practical applications, however, we can always apply (5. 7) first. If the restriction y.. � y min is satisfied by the solution, then we are done with the problem. Otherwise, we have to use Y min as a given terminal state. Although these transversality conditions are intuitively reasonable, their validity is sometimes called into question. This is because in the field of optimal control theory, to be discussed in Part 3, alleged counterexamples have been presented to show that standard transversality conditions are not necessarily applicable in infinite-horizon problems. And the doubts have spread to the calculus of variations. When we come to the topic of optimal control, however, we shall argue that these counterexamples are not really valid counterexamples. Nevertheless, it is only fair to warn the reader at this point that there exists a controversy surrounding this aspect of infinite-horizon problems. CHAPTER 5: INFINITE PLANNING HORIWN 103 One way out of the dilemma is to avoid the transversality conditions and use plain economic reasoning to determine what the terminal state should be, as t --+ oo. Such an approach is especially feasible in problems where the integrand function F either does not contain an explicit t argument (an " autonomous" problem in the mathematical sense or where the t argument appears explicitly only in the discount factor e-1>t (an "autonomous" problem in economists' usage). In a problem of this type, there is usually an implicit terminal state Yoo-a steady state-toward which the system would gravitate in order to fulfill the objective of the problem. If we can determine Yoo• then we can simply use the terminal condition (5.6) in place of a transversality condition. 5.2 THE OPTIMAL INVESTMENT PATH OF A FIRM The gross investment of a firm, Ic, has two components: net investment, dKjdt, and replacement investment, assumed equal to oK, where o is the depreciation rate of capital K. Since both components of investment are intimately tied to capital, the determination of an optimal investment path, 1/ (t), is understandably contingent upon the determination of an optimal capital path, K*(t). Assuming that investment plans are always realizable, then, once K*(t) has been found, the optimal investment path is simply I = ( 5 .8) I/ ( t) = K *'( t ) + oK*(t ) But if, for some reason, there exist obstacles to the execution of investment plans, then some other criterion must be used to determine I/ (t) from K*(t). In the present section, we present two models of investment that illustrate both of these eventualities. The Jorgenson Model7 In the Jorgenson neoclassical theory of investment, the firm is assumed to produce its output with capital K and labor L with the " neoclassical production function" Q = Q(K, L ) that allows substitution between the two inputs. This feature distinguishes it from the acceleration theory of investment in which capital is tied to output in a fixed ratio. The neoclassi­ cal production function is usually accompanied by the assumptions of positive but diminishing marginal products (QK, QL > 0; QKK• QLL < 0) and constant returns to scale (linear homogeneity). 7 Dale W. Jorgenson, " Capital Theory and Investment Behavior," American Economic Review, May 1963, pp. 247-259. The veraion given here omits certain less essential features of the original model. PART 2, THE CALCULUS OF VARIATIONS 104 The firm's cash revenue at any point of time is PQ, where P is the given product price. Its cash outlay at any time consists of its wage bill, WL ( W denotes the money wage rate), and its expenditure on new capital, mig ( m denotes the price of " machine"). Thus the net revenue at any point of time is PQ( K, L) - WL - m ( K' + oK) Applying the discount factor e -pt to this expression and summing over time, we can express the present-value net worth N of the firm as ( 5 .9) N[ K, L ] = {0 [ PQ( K, L ) - WL - m ( K' + oK)) e-p' dt The firm's objective is to maximize its net worth N by choosing an optimal K path and an optimal L path. The objective functional in (5.9) is an improper integral. In view of the presence of the discount factor e-pt, however, the integral will converge, according to Condition III of the preceding section, if the bracketed net-rev­ enue expression has an upper bound. Such would be the case if we can rule out an infinite value for K', that is, if K is not allowed to make a vertical jump. There are two state variables, K and L, in the objective functional; the other symbols denote parameters. There will be two Euler equations yield­ ing two optimal paths, K*(t) and L*(t), and the K*(t) path can then lead us to the 1/(t) path. We may expect the firm to have a given initial capital, K0, but the terminal capital is left open. Optimal Capital Stock In applying the Euler equations d FK - - FK' = 0 dt .".-··.�·. < . .' ' . • to the present model, we first observe that the integrand in (5.9) is linear in both K' and L' . (The L' term is altogether absent; i.e., it has a zero coefficient.) Thus, in line with our discussion in Sec. 2.1, the Euler equa­ tions will not be differential equations, and the problem is degenerate. The partial derivatives of the integrand are FK = ( PQK - m o)e-pt FL = ( PQL - W ) e -p1 and the Euler equations indeed emerge as the twin conditions (5. 10 ) QK = m(o + p -- p) and for all t ,,. ; , � 0 105 CHAPTER 5: INFINITE PLANNING HORIZON with no derivatives of t in them. In fact, they are nothing but the standard " marginal physical product real marginal resource cost" conditions 8 that arise in static contexts, but are now required to hold at every t. The important point about (5. 10) is that it implies constant K* and L* solutions. Taking a generalized Cobb-Douglas production function Q = K"L/3, (a + f3 * 1), for instance, we have Q K = aK" - 1L 13 and QL = {3K"L f3 - l. When these are substituted into (5. 10), we obtain (after eliminat­ ing L) = ( 5 . 1 1 ) K* = [ m(8 + aP p) ](f3 -l}j(l -n-f3)( w ) -f3/(l -a-f3) - {3 P = constant And substitution o f this K* back into (5.10) can give u s a similar constant expression for L* . The constant K* value represents a first-order necessary condition to be satisfied at all points of time, including t = 0. Therefore, unless the firm's initial capital, K0 , happens to be equal to K*, or the firm is a newly instituted enterprise free to pick its initial capital (variable initial state), the condition cannot be satisfied. The Flexible Accelerator If it is not feasible to force a jump in the state variable K from K0 to K*, the alternative is a gradual move t o K*. Jorgenson adopts what amounts to the SQ-called flexible-accelerator mechanism to effect a gradual adjustment in net investment. The essence of the flexible accelerator is to eliminate in a systematic way the existing discrepancy between a target capital, K, and the actual capital at time t, K(t):9 ( 5 . 1 2) I( t ) = i [ K - K(t)] ( 0 < j < 1) The gradual nature of the adjustment in the capital level would presumably enable the firm to alleviate the difficulties that might be associated with abrupt and wholesale changes. While the particular construct in (5.12) was originally introduced by economists merely as a convenient postulate, a theoretical justification has been provided for it in a classic paper by Eisner and Strotz. 8 This fact is immediately obvious from the second equation in (5.10). As for the first equation, mlJ means the marginal depreciation cost, and mp means the interest·earning opportunity cost of holding capital, and their sum constitutes the marginal user cost of capital, which is the exact counterpart of W on the labor side. The correct application of the flexible accelerator calls for setting the target K at the level in (5. 1 1 ), but the Jorgenson paper actually uses a substitute target that varies with time, which is suboptimal. For a discussion of this, see John P. Gould, " The Use of Endogenous Variables in Dynamic Models of Investment," Quarterly Journal of Economics, November 1969, pp. 580-599. 9 PART 2o THE CALCULUS OF VARIATIONS 106 The Eisner-Strotz Model10 The Eisner-Strotz model focuses on net investment as a process that expands a firm's plant size. Thus replacement investment is ignored. As­ suming that the firm has knowledge of the profit rate '1T associated with each plant size, as measur.ed by the capital stock K, we have a profit function 7T( K)-which, incidentally, provides an interesting contrast to the profit function 7T(L) in the labor-adjustment model of Sec. 3.4. To expand the plant, an adjustment cost C is incurred whose magnitude varies posi­ tively with the speed of expansion K'(t). So we have an increasing function C = C(K'), through which the firm's internal difficulties of plant adjust­ ment as well as the external obstacles to investment (such as pressure on the capital-goods industry supply) can be explicitly taken into account in the optimization problem of the firm. If the C function adequately reflects these adjustment difficulties, then once the K*(t) path is found, we may just take its derivative K*'(t) as the optimal net investment path without having to use some ad hoc expedient like the flexible-accelerator mechanism. The objective of the firm is to choose a K*(t) path that maximizes the total present value of its net profit over time: ( 5 . 13) Maximize TI[ K ) subject to K(O) {0 [ 7r ( K ) - C ( K')]e-P' dt = = K0 ( K0 given) This functional is again an improper integral, but inasmuch as the net return can be expected to be bounded from above, convergence should not be a problem by Condition III of the preceding section. Note that although we have specified a fixed initial capital stock, the terminal capital stock is left open. Note also that this problem is autonomous. The Quadratic Case It is possible to apply the Euler equation to the general-function formula­ tion in (5.13). For more specific results, however, let us assume that both the 7T and C function are quadratic, as graphed in Fig. 5. 1 : (5.14) '1T = a. K - {3K2 ( 5 . 15) C aK' 2 10 = + bK' ( a , f3 > 0) ( a , b > 0) Robert Eisner and Robert H. Strotz, "Determinants of Business Investment," in Monetary Policy, Impacts of A Series of Research Studies Prepared for the Commission on Money and Credit, Prentice-Hall, Englewood Cliffs, NJ, here is contained in Sec. 1963, pp. 60-233. The theoretical model discussed 1 of their paper; see also their App. A. 107 CHAPTER 5: INFINITE PLANNING HORIZON c 0 (b) (a) FIGuaE 5.1 The profit rate has a maximum, attained at K = a/2{3. The C curve has been drawn only in the first quadrant, where K' > 0, because we are confining our attention to the case of plant expansion. Considered together, these two curves should provide an upper bound for the net return 7T C, to guarantee convergence of the improper integral in (5.13). In this quadratic model, we have - F = ( a K - {3K2 - aK'2 - bK' )e-pt with derivatives: Fx = (a - 2{3K ) e -pt Fx• = - (2aK' + b ) e -P1 (5 . 16) �K' = p( 2aK' + b ) e -p1 Thus, by (2.19), we have the Euler equation f3 K" - p K ' - - K = a ( 5 .17) bp - a , . . ·� . �. 1 -- 2a with general solution11 ( 5.18 ) 11 The Euler equation given by Eisner and Strotz contains a minor error-the coefficient of the K term in (5.17) is shown as -f3j2a instead of -f3ja. As a result, their characteristic roots and particular integral do not coincide with (5.18). This fact, however, does not affect their qualitative conclusions. _\,· · PART 2, THE CALCULUS OF VARIATIONS 108 where [characteristic roots) and [particular integral] The characteristic roots r1 and r2 are both real, because the expression under the square-root sign is positive. Moreover, since the square root itself exceeds p, it follows that r1 > p > 0, but r2 < 0. AB to the particular integral K, it is uncertain in sign. Economic considerations would dictate, however, that it be positive, because it represents the intertemporal equilib­ rium level of K. For the problem to be sensible, therefore, we must stipulate that a > bp ( 5 . 19) The Definite Solution To get the definite solution, we first make use of the initial condition K(O) = K0, which enables us-after setting t = 0 in (5. 18)-to write ( 5 .20) We would like to see whether we can apply the terminal condition (5.6). Even though no explicit terminal state is given in the model, the au­ tonomous nature of the problem suggests that it has an implicit terminal state-the ultimate plant size (capital stock) to which the firm is to expand. AB Fig. 5.1a indicates, the highest attainable profit rate occurs at plant size K = aj2{3. Accordingly, we would not expect the firm to want to expand beyond that size. Mter the cost function C is taken into account, moreover, the firm may very well wish to select an ultimate size even smaller. In fact, from our knowledge of differential equations, the particular integral K would be the logical candidate for the ultimate plant size K�. Referring to (5. 18), however, we see that while the second exponential term (where r2 < 0) tends to zero as t --> oo, the first exponential term (where r1 > 0) tends to ± oo if A1 � 0. Since neither + co nor - co is acceptable as the terminal value for K on economic grounds, the only way out is to set A1 = 0. Coupling this information with (5.20), we then have A 2 = K0 - K, so that the optimal K path is ( 5.21) K * ( t ) = ( K0 - K)er•t + K A few things may be noted about this result. First, since r2 < 0, K*(t) converges to the particular integral K; thus K indeed emerges as the optimal terminal plant size Kx. Moreover, given that b and p are positive, K = (a - bp)/ 2{3 is less than a/2{3. Thus, after explicitly taking into account the adjustment cost, the firm indeed ends up with a plant size ��::.: ' -,( ;.. ,. 109 CHAPTER 5: INFINITE PLANNING HORIZON ± .•. ------- t .,· FIGURE 5.2 smaller than the 7T-mruamizing one shown in Fig. 5.1a . Finally, the discrep­ ancy between the initial and terminal sizes, K0 - K, is subject to expo­ nential decay. The K*(t) path should therefore have the general shape illustrated in Fig. 5.2. The Transversality Conditions Although we have been able to definitize the arbitrary constants without using any transversality conditions, it would be interesting to see where the transversality condition (5.5) would lead us. Using the information in (5.16), we know that ( 5.22) For the K and K' terms, we should use the K* in (5.18) and its derivative, namely, Plugging these into (5.22), we get the complicated result (5.22') F* - K*'FK*' A1(a - 2 f3 K)e<r, -p)t + A1 2 ( ar/ - f3 ) e <2 r, -p ll = + A 1 A 2 (2ar1 r2 - 2{3)e1'1 +r2-p)t + A 2(a - 2 f3 K )e <r. -p)t + A2 2 ( ar/ - f3 ) e 12'•-plt + ( a K - f3 K2 )e - P1 The transversality condition is that (5.22') vanish as t -> oo. Since r2 is negative, the last three terms cause no difficulty. But the exponential in the third term reduces to e0 = 1, because r1 + r = p [see (5.18)], and that term 2 will not tend to zero. Worse, the first two terms are explosive since r1 > p . In order to force the first three terms to go to zero, the only way is to set ·� "'� " m• =�om v.-unom 1 10 .: A 1 = 0. Thus, the transversality condition leads us to precisely what we have concluded via economic reasoning. As to transversality condition (5. 7), it is not really needed in this problem. But if we try to apply it, it will also tell us to set A 1 = 0 in order to tame an explosive exponential expression. - The Optimal Investment Path and the Flexible Accelerator If the adjustment-cost function C = C( K') fully takes into account the various internal and external difficulties of adjustment, then the optimal investment path is simply the derivative of the definite solution (5.21): ( 5 .23) But since (5.21) implies that ( K0 - K)e'•1 = K*(t) - K the investment path (5.23) can be simplified to ' :· (5.23') The remarkable thing about (5.23') is that it exactly depicts the flexible-accelerator mechanism in (5.12). The discrepancy between K, the target plant size, and K*(t), the actual plant size on the extremal at any point of time, is systematically eliminated through the positive coefficient - r2 • The only unsettled matter is that, in the flexible accelerator, - r2 must be less than one. This requirement is met if we impose the following additional restriction on the model: f3 - < ( 5 .24) a . 1 +p What makes the flexible accelerator in (5.23') fundamentally different from (5.12) is that the former is no longer a postulated behavior pattern, but an optimization rule that emerges from the solution procedure. The Eisner-Strotz model, in its quadratic version, has thus contributed a much needed theoretical justification for the popularly used flexible-accelerator mechanism. EXERCISE 5.2 1 Let the profit function and the adjustment-cost function (5.14) and (5.15) take the specific forms 1r = 50K - K 2 C = K'2 + 2K' . . ·; '. . Ill CHAPTER 5: INFINITE PLANNING HORIZON but leave the discount rate p in the parametric form. Find the optimal path of K (definite solution) by directly applying the Euler equation (2.19). 2 Use the solution formula (5.21) to check your answer to the preceding problem. 3 Find the Euler equation for the general formulation of the Eisner-Strotz model as given in (5.13). 7T 50K - K2 as in Prob. 1, hut change the 2 adjustment-cost function to C = K' • Use the general Euler equation you derived in the preceding problem for (5.13) to get the optimal K path. 4 Let the profit function be 5 5.3 = Demonstrate that the parameter restriction (5.24) is necessary to make - r2 a positive fraction. THE OPTIMAL SOCIAL SAVING BEHAVIOR Among the very first applications of the calculus of variations to economics is the classic paper by Frank Ramsey on the optimal social saving behavior.12 This paper has exerted an enormous if delayed influence on the current literature on optimal economic growth, and is well worth a careful review. The Ramsey Model The central question addressed by Ramsey is that of intertemporal resource allocation: How much of the national output at any point of time should be for current consumption to yield current utility, and how much should be saved (and invested) so as to enhance future production and consumption, and hence yield future utility? The output is assumed to be produced with two inputs, capital K and labor L. The production function, Q = Q(K,L), is time invariant, since no technological progress is assumed. Other simplifying assumptions include the absence of depreciation for capital and a stationary population. How­ ever, the amount of labor services rendered can still vary. The output can either be consumed or saved, but what is saved always results in investment and capital accumulation. Then, using the standard symbols, we have Q = C + S = C + K', or · (5.25) C = Q( K, L ) - K' Consumption contributes to social welfare via a social utility (index) function U(C ), with nonincreasing marginal utility, U"(C) :5 0. This speci12Frank P. Ramsey, "A Mathematical Theory of Saving," Economic Journal, December 1928, pp. 543-559. 1 12 PART 2, THE CALCULUS OF VARIATIONS u D�------- 1 U<Cl 0 u 0 c " c " c (a) c (b) u '·' (c) FIGURE 5.3 fication is consistent with all three utility functions illustrated in Fig. 5.3. In every case, there is an upper bound for utility, 0. That level of utility is attainable at a finite consumption level 6 in Figs. 5.3a and b, but can only be approached asymptotically in Fig. 5.3c. In order to produce its consumption goods, society incurs disutility of labor D(L), with nondecreasing marginal disutility, D''(L) � 0. The net social utility is therefore D(L), where and L-as K and Q-are functions of time. The economic planner's problem is to maximize the social utility for the current generation as well as for all generations to come: U{C) - ( 5 .26) Maximize U C {! U(C) - D( L)] dt 0 C, C In the integrand, depends solely on and, by (5.25), depends on K, L, and K', whereas D depends only on L. Thus, this problem has two state variables, K and L. Because there is no L' term in the integrand, however, the problem is degenerate on the L side, and in general it would be inappropriate to stipulate an initial condition on L. CHAPTER s, INFINITE PLANNING HORIZON 113 The Question of Convergence Unlike the functional of the Eisner-Strotz model, the improper integral in (5.26) does not contain a discount factor. This omission is not the result of neglect; it stems from Ramsey's view that it is " ethically indefensible" for the (current-generation) planner to discount the utility of future genera­ tions. While this may be commendable on moral grounds, the absence of a discount factor unfortunately forfeits the opportunity to take advantage of Condition III of Sec. 5.1 to establish convergence, even if the integrand has an upper bound. In fact, since the net utility is expected to be positive as t becomes infinite, the integral is likely to diverge. To handle this difficulty, Ramsey replaces (5.26) with the following substitute problem: ( 5 .26' ) · Minimize (0 [ B - U( C ) + D( L ) ] dt subject to K(O) = K0 ( K0 given) where B (for Bliss) is a postulated maximum attainable level of net utility. Since the new functional measures the amount by which the net utility U(C) - D(L ) falls short of Bliss, it is to be minimized rather than maxi­ mized. Intuitively, an optimal allocation plan should either take society to Bliss, or lead it to approach Bliss asymptotically. If so, the integrand in (5.26') will fall steadily to the zero level, or approach zero as t -+ co. As far as Ramsey is concerned, the convergence problem is thereby resolved. The substitution of (5.26) by (5.26'), referred to as " the Ramsey device," is widely accepted as sufficient for convergence. Our earlier discussion of Condition I in Sec. 5.1 would confirm that if the integrand attains zero at some finite time and remains at zero thenceforth, then convergence indeed is ensured. But, as underscored in Condition II, the mere fact that the integrand tends to zero as t becomes infinite does not in and of itself guarantee convergence. The integrand must also fall sufficiently fast over time. Even though the convergence problem is not clearly resolved by Ramsey, we shall now proceed on the assumption that the general-function integrand in (5.26') does converge. The Solution of the Model The problem as stated in (5.26') is an autonomous problem in the state variables K and L . From the integrand F = B - U( C ) + D( L ) where C = Q( K , L ) - K' PART 2: THE CALCULUS OF VARIATIONS 114 we can get, on the L side, the derivatives FL = - U'(C) ac + D' ( L ) = -J.LQL + D' ( L ) aL [J.L = U'(C)] Fu = 0 For notational simplicity, we are using J.L to denote marginal utility. (The abbreviation of marginal utility is " mu.") Like U(C), J.L is a function of C and hence indirectly a function of K, L, and K'. On the K side, we can obtain the derivatives ac FK = - U'(C) = Q aK - J.L K ac = - U' ( C ) ( - 1) = U' ( C ) = J.L FK. = - U'(C) aK' These derivatives will enable us to apply the Euler equations (2.27). We note first that since Fu = 0, the Euler equation for the L variable, FL - dFv!dt 0, reduces simply to FL = 0, or = for all t (5.27) � 0 The marginal disutility of labor must, at each point of time, be equated to the product of the marginal utility of consumption and the marginal prod­ uct of labor. On the K side, the Euler equation FK - dFK.jdt = 0 gives us the condition - J.LQK - d}Ljdt = 0, or (5.28) d}Ljdt -- = - QK J.L for all t � 0 This result prescribes a rule about consumption: J.L, the marginal utility of consumption, must at every point of time have a growth rate· equal to the negative of the marginal product of capital. Following this rule, we can map out the optimal path for J.L . Once the optimal J.L path is found, (5.27) can be used to determine an optimal L path. As mentioned earlier, however, the problem is degenerate in L, so it is not appropriate to preset an initial level L(O). The Optimal Investment and Capital Paths Of greater interest to us is the optimal K path and the related investment (and savings) path K ' . Instead of deducing these from the previous results, let us find this information by taking advantage of the fact that the present problem falls under Special Case II of Sec. 2.2. Without t as an explicit argument in the integrand function, the Euler equation for K is, according . . � ·.- ' ;.. ! ; �•. 115 CHAPTER 5: INFINITE PLANNING HORIZON to (2.21), (5.29) F - K'FK' = constant, or B - U( C ) + D( L ) - K'J.L = constant for all t � 0 This equation can be solved for K' as soon as the (arbitrary) constant on the right-hand side can be assigned a specific value. To find this value, we note that this constant is to hold for all t, including t --+ oo. We can thus make use of the fact that, as t -> oo , the economic objective of the model is to have U(C) - D( L ) tend to Bliss. This means that U must tend to 0 and J.L must tend to zero as t becomes infinite. It follows that the arbitrary constant in (5.29) has to be zero. If so, the optimal path of K' is B - U( C ) + D( L ) K*' = ------J.L (5.30) or, with the time argument explicitly written out, K*' ( t) (5.30' ) B - U ( C( t ) ] + D ( L ( t) ] J.L( t) = _____:_:.____:_:._.c..__ _ '---'---- This result i s known as the Ramsey rule. I t stipulates that, optimally, the rate of capital accumulation must at any point of time be equal to the ratio of the shortfall of net utility from Bliss to the marginal utility of consump­ tion. On the surface, this rule is curiously independent of the production function. And this led Ramsey to conclude (page 548) that the production function would matter only insofar as it may affect the determination of Bliss. But this conclusion is incorrect. According to (5.28), J.L is to be optimally chosen by reference to QK. This means that the denominator of (5.30') does depend crucially on the production function. A Look at the Transversality Conditions We shall now show that the use of the infinite-horizon transversality condition would also dictate that the constant in (5.29) be set equal to zero. When condition (5.5) is applied to the two state variables in the present problem, it requires that lim ( F - L'Fv ) = 0 t � oo and lim ( F - K'Fx, ) = t - «> 0 I n view of the fact that Fv 0, the first of these conditions reduces t o the condition that F -> 0 as t -> oo. This would mean the net utility U(C) ­ D(L) must tend to Bliss. Note that, by itself, this condition still leaves the constant in (5.29) uncertain. However, the other condition will fix the constant at zero because F - K'Fx' is nothing but the left-hand-side ex­ pression in (5.29). = 116 PART 2: THE CALCULUS OF VARIATIONS The present problem implicitly specifies the terminal state at Bliss. Consequently, the transversality condition (5. 7) is not needed. From the Ramsey rule, it is possible to go one step further to find the K * (t) path by integrating (5.30'). For that, however, we need specific forms of U(C) and D(L ) functions. The general solution of (5.30') will contain one arbitrary constant, which can be definitized by the initial condition K(O) = K0• And that would complete the solution of the model. We shall not attempt to illustrate the model with specific functions; rather, we shall show the application of phase-diagram analysis to it in the next section. EXERCISE 5.3 1 Let the production function and the utility function in the Ramsey model take the specific forms: Q( K ) = rK 1 U = u - - c ­b b • ( r > 0) ( b > 0) with initial capital K0. (a) Apply the condition (5.28) to derive a C*(t) path. Discuss the characteristics of this path. (b) Use the Ramsey rule (5.30') to find K*'(t). Discuss the characteristics of this path, and compare it with C*(t). How specifically are the two paths related to each other? [Hint: Since the D(L ) term is absent in the problem, we have B = 0.] (c) Bearing in mind that Q0 rK0 (production of Q0), and Q0 = C* 0 + K*'(O) (allocation of Q0), and using the relationship between K*'(O) and C*0 implied by the result in (b), find an expression for C*0 in terms of K0• Write out the definite solution for K*'(t). (d) Integrate K*'(t) to obtain the K*(t) path. What is the rate of growth of K * ? How does that rate vary over time? = 2 Find the K * (t) path for the preceding problem by the alternative procedure outlined below: (a) Express the integrand of the functional ( B U(C)] in terms of K and K', that is, as F(K, K'). (b) Apply the Euler equation (2.18) or (2.19) to get a second-order differential equation. Find the general solution K*(t), with arbitrary constants A1 and A2. Then derive the optimal savings path K*'(t); retain the arbitrary constants. (c) Find the savings ratio K*'(t)jQ*(t) = K*'(t)jrK*(t). Show that this ratio approaches one as t -> oo, unless one of the arbitrary constants is zero. - 117 CHAPTER 5: INFINITE PLANNING HORIZON (d) I s a unitary limit for the savings ratio economically acceptable? Use your answer to this question to definitize one of the constants. Then definitize the other constant by the initial condition K(O) == K 0 . Compare the resulting definite solution with the one obtained in the preceding problem. 5.4 PHASE-DIAGRAM ANALYSIS In models of dynamic optimization, two-variable phase diagrams are preva­ lently employed to obtain qualitative analytical resultsY This is particu­ larly true when general functions are used in the model, and the time horizon is infinite. For a calculus-of-variations problem with a single state variable, the Euler equation comes as a single second-order differential equation. But it is usually a simple matter to convert that equation into a system of two simultaneous first-order equations in two variables. The two-variable phase diagram can then be applied to the problem in a straight­ forward manner. We shall illustrate the technique in this section with the Eisner-Strotz model and the Ramsey model. We shall then adapt the phase diagram to a finite-horizon context. The Eisner-Strotz Model Once Again In their model of a firm's optimal investment in its plant, discussed in Sec. 5.2, Eisner and Strotz are able to offer a quantitative solution to the problem when specific quadratic profit and cost-of-adjustment functions are assumed. Writing these functions as 1r = C = a K - {3K 2 aK' 2 + bK' ( a, {3 > 0) ( a , b > O) [from (5.14)] [from (5.15)] they derive the Euler equation (5.31) where (5.32) {3 bp - a K" - p K' - - K = 2a a --- p [from ( 5 .17)] is a positive rate of discount. From the quantitative solution, K*(t) = ( K o - K)er•t + K a - bp where K = 2{3 [from (5.21)] _ -- 13The method of two-variable phase diagrams i s explained i n Alpha C . Chiang. Fundamental Methods of Mathematical Economics, 3d ed., McGraw-Hill, New York, 1984, Sec. 18.5. PART 2o THE CALCULUS OF VARIATIONS 118 we see that the capital stock K its specific target value K (the plant size) should optimally approach by following a steady time path that exponen­ tially closes the gap between the initial value and the target value of K. Now we shall show how the Euler equation can be analyzed by means I of a phase diagram. We begin by introducing a variable (net investment) (5.33) I( t) = K'( t ) I' ( t ) [implying = K"( t) ] (5.31) as a system which would enable u s to rewrite the Euler equation of two first-order differential equations: f3 bp - a I' = pi + - K + a 2a (5.34) [i.e., -- I' = f( I, K ) ] [ i.e., K ' K' = I = g( I, K ) ] ' This is an especially simple system, since both the linear in the two variables function. g I and K, ::::::: : = 0 f and g functions are and in fact K is even absent in the · demarcation curves, dO.w=. �' fi..,t oni"' of bu,;noM ;, to d,.w two and K' = Individually, each of these curves 0. I' = serves to delineate the subset of points in the able in question can be stationary (dljdt = 0 IK space where the vari­ dKjdt = 0). Jointly, the or two curves determine at their intersection the intertemporal equilib­ rium-steady state-of the entire system (dijdt = 0 and Setting I' = {) in (5.34) and solving for K, we get (5.35) K = a - bp 2 f3 -- Similarly, by setting K' (5.36) ap - -I f3 = [equation for I' = dKjdt = 0). 0 curve) 0, we get I=0 [equation for K' = 0 curve) These two equations both plot as straight lines in the phase space, as shown in Fig. 5.4, with the I' = curve sloping downward, and the K' = curve 0 = 0 coinciding with the vertical axis. The unique intertemporal equilibrium occurs at the vertical intercept of the I' 0 curve, which, by (5.35), is a positive magnitude by (5.19). Note that this equilib­ rium coincides exactly with the particular integral (target plant size) K = (a - found in bp)/2/3, (5.18). This is of course only to be expected. K By definition, all points on the I' = 0 curve are characterized by stationarity in the variable I. We have therefore drawn a couple of vertical " sketching bars" on the curve, to indicate that there should not be any east-west movement for points on the curve. Similarly, horizontal " sketch. ... ! l_: ' ; 1 19 CHAPTER 5: INFINITE PLANNING HORIZON K - K' = 0 + L . . � _,. ·.\. :.·;.;! ,;·.· FIGURE 6.4 ing bars" have been attached to the K' = 0 curve to indicate that the stationarity of K forbids any north-south movement on the curve. Points off the demarcation curves are, on the other hand, very much involved in dynamic motion. The direction of movement depends on the signs of the derivatives I' and K' at a particular point in the IK space; the speed of movement depends on the magnitudes of those derivatives. From (5.34), we find by differentiation that (5.37) ar = >O aI p and aK' iJ] -=1>0 The positive sign of ol'jiJl implies that, moving from west to east (with I increasing), I' should be going from a negative region, through zero, to a positive region (with I' increasing). To record this ( - , 0, + ) sign sequence, a minus sign has been added to the left of the I' = 0 label, and a plus sign has been added to the right of the I' = 0 label. Accordingly, we have also drawn leftward /-arrowheads ( I' < 0) to the left, and rightward /-arrowheads (I' > 0) to the right, of the I' = 0 curve. For K', the positive sign of oK'!oi similarly implies that, going from west to east, K' should be moving from a negative region, through zero, to a positive region. Hence, the sign sequence is again ( - , 0, + ), and this explains the minus sign to the left, and the plus sign to the right, of the K' = 0 label. The K-arrowheads are downward (K' < 0) to the left, and upward (K' > 0) to the right, of the K' = 0 curve. 120 PART 2o THE CALCULUS OF VARIATIONS Following the directional restrictions imposed by the IK-arrowheads and sketching bars, we can draw a family of streamlines, or trajectories, to portray the dynamics of the system from any conceivable initial point. Every point should be on some streamline, and there should be an infinite number of streamlines. But usually we only draw a few. The reader should verify in Fig. 5.4 that the streamlint'fs conform to the IK-arrowheads, and, in particu­ lar, that they are consistent with the sketching bars which require them to cross the I' = 0 curve with infinite slope and cross the K' = 0 curve with zero slope. The Saddle-Point Equilibrium The intertemporal equilibrium of the system occurs at point E, the intersec­ tion of the two demarcation curves. The way the streamlines are structured, we see two stable branches leading toward E (one from the southeast direction and the other from the northwest), two unstable branches leading away from E (one toward the northeast and the other toward the south­ west), and the other streamlines heading toward E at first but then turning away from it. As a result, we have a so-called saddle-point equilibrium . It is clear that the only way to reach the target level of capital at E is to get onto the stable branches. Given the initial capital K0, for instance, it is mandatory that we select I* 0 as the initial rate of investment, for only that choice will place us on the " yellow brick road" to the equilibrium. A different initial capital will, of course, call for a different I* 0• The restriction on the choice of I* 0 is thus the surrogate for a transversality condition. And this serves to underscore the fact that the transversality condition consti­ tutes an integral part of the optimization rule. While all the streamlines, being based on (5.34), satisfy the Euler equation, only the one that also satisfies the transversality condition or its equivalent-one that lies along a stable branch-qualifies as an optimal path. Traveling toward the northwest on the stable branch in Fig. 5.4, we find a steady climb in the level of K, implying a steady n·arrowing of the discrepancy between K0 and the target capital level. This is precisely what (5.32) tells us. Accompanying the increase in K is a steady decline in the rate of investment I. Again, this is consistent with the earlier quantitative solution, for by differentiating (5.32) twice with respect to t, we do find I*'(t) = K*"( t ) = ri ( Ko - K ) erzt < 0 since, in the present example, K0 < K. A Simplified Ramsey Model To simplify the analysis of the Ramsey model, we shall assume that the labor input is constant, thereby reducing the production function to Q(K). CHAPTER 5: INFINITE PLANNING HORIZON 121 Then we will have only one state variable and one Euler equation with which to contend. Another consequence of omitting L and D(L) is to make Bliss identical with the upper bound of the utility function, 0. The Euler equation for the original Ramsey model can be written as [ from (5 .28) ] (5.38) or, making explicit the dependence of the marginal product on K, (5 .38' ) This is a first-order differential equation in the variable J.l.. However, since the variable K also appears in the equation, we need a differential equation in the variable K to close the dynamic system. This need is satisfied by (5.25), C = Q(K, L) - K', which, when adapted to the present simplified context, becomes K' = Q( K ) - C(tL) (5 .39) The replacement o f Q(K, L ) by Q(K ) i s due to the elimination o f L. The substitution of C by the function C(J.l.) reflects the fact that in the usual marginal-utility function, 1-L is a monotonically decreasing function of C, so that C can be taken as an inverse function of marginal utility 1-L · Combining (5.38') and (5.39), we get a two-equation system K' ( 5 . 40 ) Ji = = Q( K ) - C ( tL ) [ i.e., K ' = {( K, tL)] - tL Q' ( K ) [ i.e., jl = g ( K, tL) ] containing two general functions, Q(K) and C(J.l.). Before we proceed, certain qualitative restrictions should be placed on the shapes of these functions. For the time being, let us assume that the U(C) function is shaped as in the top diagram of Fig. 5.5, with B as the Bliss level of utility and CB the Bliss level of consumption. There exists cons u mption saturation at CB, and further consumption beyond that level entails a reduction in U. From this function, we can derive the marginal­ utility function J.l.(C) in the diagram directly beneath, with 1-L = 0 at C CB. Since J.l.(C) is monotonic, the inverse function C(J.l.) exists, with C'(J.l.) < 0. The diagram in the lower-left corner, on the other hand, sets forth the assumed shape of Q(K ), the other general function, here plotted-for a good reason-with Q on the horizontal rather than the vertical axis. The underlying assumption for this curve is that the marginal product of capital Q'(K ) is positive throughout-there is no capital saturation -but the law of diminishing returns prevails throughout. = 122 PART 2: THE CALCULUS OF VARIATIONS u B Utility function 0 •_ _ _ _ _ _ _ _ _.___ Ca I 'II " U'(C) Marginalutility function I 1 \ c ,. · p = U'(C) Phase 1 apace- I I �------- - - - - - -r 1 I I I I I I K I I Production I function I ------�-T 45° line 1 I I I FIGURE 5.5 The Phase Diagram To construct the phase diagram, we need the K' = 0 and ,_,: From (5.40), the following two equations readily emerge: ( 5 .41 ) ( 5 .42) Q( K ) = C ( �J- ) 11- = 0 - 0 curves. [equation for K' = 0 curve] [equation for It' = 0 curve] The first of these is self-explanatory. The second results from our assump­ tion of no capital saturation: Since Q'(K) * 0, we can have It' = 0 if and only if 11- = 0. :�-l, 123 CHAPTER 5: INFINITE PLANNING HORIZON To find the proper shape of the K' 0 curve, we equate Q(K) and C(J-L) in accordance with (5.41). It is in order to facilitate this equating process that we have chosen to plot the Q(K) function with the Q variable on the horizontal axis in Fig. 5.5, for then we can equate Q and C = graphically by making a vertical movement across the two diagrams. Con­ sider, for instance, the case of J-L = 0, that is, the point C8 on the J-L(C) curve. To satisfy (5.41), we simply select point G on the Q(K) curve, the point that lies directly below point C8. The vertical height of point G may be interpreted as the Bliss level of K; thus we denote it by K8. The ordered pair (K = K8, J-L = 0) then clearly qualifies as a point on the K' 0 curve in the phase space. Other ordered pairs that qualify as points on the K' = 0 curve can be derived similarly. The easiest way to trace out the entire K' 0 curve is by means of the rectangularly aligned set of our diagrams in the lower part of Fig. 5.5. Aside from the J-L(C) and Q(K) diagrams, we include in this diagram set a 45° line diagram (with K on both axes) and a phase-space diagram which is to serve as the repository of the K' 0 curve being derived. These four diagrams are aligned in a way such that each pair of adjacent diagrams in the same " row" has a common vertical axis, and each pair of adjacent diagrams in the same " column" has a common horizontal axis-except that, in the left column, the C axis and the Q axis are not inherently the same, but rather to be made the same via a deliberate equating process. Now the point ( K = K8, J-L = 0) can be traced out via the dashed rectangle, C8GHK8, spanned by the J-L(C) curve, the Q(K) curve, and the 45° line. Indeed, starting from any selected point on the J-L(C) curve, if we go straight down to meet the Q(K ) curve [thereby satisfying the condition (5.41)], turn right to meet the 45° line, and then go straight up into the phase space (thereby translating the vertical height of point G, or K8, into an equal horizontal distance in the phase space) to complete the formation of a rectangle, then the fourth corner of this rectangle must be a point on the K' = 0 curve. Such a process of rectangle construction yields a K' 0 curve that slopes downward and cuts the horizontal axis of the phase space at K K 8• = = = = = The Saddle-Point Equilibrium We now reproduce the K' = 0 curve in Fig. 5.6, and add thereto the Ji = 0 curve, which, according to (5.42), should coincide with the K axis. At the intersection of these two curves is the unique equilibrium point E, which is characterized by K = K8 and J-L = 0, and which therefore corresponds to Bliss. 0 curve are vertical. Those for the The sketching bars for the K' Ji = 0 curve are, however, horizontal, and they lie entirely along that curve. Since the sketching bars coincide with the Ji = 0 curve, the latter will serve not only as a demarcation curve, but also as the locus of some streamlines. = 124 PART 2: THE CALCULUS OF VARIATIONS K' = O SU!.ble branch ._ r :--:> ··.,:; ·· ! -� '• , ::- . !J' = 0 0 K Stable branch FIGURE 5.6 The sign sequences for K' derivatives (5.43) aK' aK - = Q'( K ) > 0 and and Ji can be uceriained from the partial ap: aJ.L - = - Q'( K ) < 0 [ from (5.40)] These yield the ( , 0, + ) sequence rightward for K', and the ( + , 0, ) sequence upward for Jl. Consequently, the K-arrowheads point eastward in the area to the right of the K' = 0 curve, and they point westward in the area to its left, whereas the J.L-arrowheads point southward in the area above the Jl = 0 curve, and they point northward in the area below it. The resulting streamlines again produce a saddle point. Given any initial capital K0, it is thus necessary for us to select an initial marginal utility that will place us on one of the stable branches of the saddle point. A specific illustration, with K0 < KB , is shown in Fig. 5.6, where there is a unique J.Lo value, J.L* 0, that can land us on the " yellow brick road" leading to E. All the other streamlines will result ultimately in either (1) excessive capital accumulation and failure to attain Bliss, or (2) continual decumula­ tion ("eating up") of capital and, again, failure to reach Bliss. As in the Eisner-Strotz model, we may interpret the strict requirement on the proper selection of an initial J.L value as tantamount to the imposition of a transver­ sality condition. The fact that both the Eisner-Strotz model and the Ramsey model present us with the saddle-point type of equilibrium is not a mere coinci­ dence. The requirement to place ourselves onto a stable branch merely - - 125 CHAPTER 5: INFINITE PLANNING HORIZON implies that there exists an optimization rule to be followed, much as a profit-maximizing firm must, in the simple static optimization framework, select its output level according to the rule MC MR. If the equilibrium were a stable node or a stable focus-the "all roads lead to Rome" situation -there would be no specific rule imposed, which is hardly characteristic of an optimization problem. On the other hand, if the equilibrium were an unstable node or focus, there would be no way to arrive at any target level of the state variable at all. This, again, would hardly be a likely case in a meaningful optimization context. In contrast, the saddle-point equilibrium, with a target that is attainable, but attainable only under a specific rule, fits comfortably into the general framework of an optimization problem. = Capital Saturation Versus Consumption Saturation The preceding discussion is based on the assumption of consumption satu­ ration at C8, with U(C8) B and J.L(C8) = 0. Even though capital satura­ tion does not exist and Q'(K ) remains positive throughout, the economy would nevertheless refrain from accumulating capital beyond the level K8. As a variant of that model, we may also analyze the case in which capital, instead of consumption, is saturated. Specifically, let Q(K) be a strictly concave curve, with Q(O) 0, and slope Q'(K ) z 0 as K 'S K. Then we have capital saturation at K. At the same time, let U(C) be an increasing function throughout. Under these new assumptions, it becomes necessary to redraw all the curves in Fig. 5.5 except the 45° line. Consequently, the K' 0 curve in the phase space may be expected to acquire an altogether different shape. However, it is actually also necessary to replace the horizontal j.L = 0 curve, because with con­ sumption saturation assumed away and with J.L > 0 throughout, the hori­ zontal axis in the diagram is now strictly off limits. In short, there will be an entirely new phase diagram to analyze. Instead of presenting the detailed analysis here, we shall leave to the reader the task of deriving the new K' 0 and j.L 0 curves and pursuing the subsequent steps of determining the KJ.L-arrowheads and the stream­ lines. However, some of the major features of the new phase diagram may be mentioned here without spoiling the reader's fun: (1) In the new phase diagram, the K' 0 curve will appear as a U-shaped curve, and the j.L = 0 curve will appear as a vertical line. (2) The new equilibrium point will occur at K K, and the marginal utility corresponding to K (call it p.) will now be positive rather than zero. (3) The equilibrium value of the marginal utility, P, , implies a corresponding equilibrium consumption C which, in turn, is associated with a corresponding utility level 0. Even though the latter is not the highest conceivable level of utility, it may nevertheless be regarded as " Bliss" in the present context, for, in view of capital saturation, we will never wish to venture beyond 0. The magnitude of this adapted = = = = = = = 126 PART 2, THE CALCULUS OF VARIATIONS Bliss is not inherently fixed, but is dependent on the location of the capital-saturation point on the production function. rium is again a saddle point. (4) Finally, the equilib­ Yet another variant of the model arises when the U( C) curve is replaced by a concave curve asymptotic to B rather than with a peak at B , as i n Fig. 5.3c. In this new case, the J.L curve will become a convex curve asymptotic to the C axis. What will happen to the K' = 0 curve now depends on whether the Q(K) curve is characterized by capital saturation. If it is, the present variant will simply reduce to the preceding variant. If not, and if the Q(K) curve is positively sloped throughout, the K' = 0 curve will turn out to be a downward-sloping convex curve asymptotic to the K axis in the phase space. The other curve, because, according to (5.40), p! p! = 0, is now impossible to draw cannot attain a zero value unless either J.L = 0 or Q' ( K ) = 0, but neither of these two alternatives is available to us. Consequently, we cannot define an intertemporal equilibrium in this case. Finite Planning Horizon and Turnpike Behavior The previous variants of the Ramsey model all envisage an infinite planning horizon. But what if the planning period is finite, say, 100 years or 250 years? Can the phase-diagram technique still be used? The answer is yes. In particular, phase-diagram analysis is very useful in explaining the so-called " turnpike" behavior of optimal time paths when long, but finite, planning horizons are considered.14 In the finite-horizon context, we need to have not only a given initial capital K 0 as in the infinite-horizon case, but also a prespecified terminal capital, Kr, to make the solution definite. The magnitude of Kr represents the amount of capital we choose to bequeath to the generations that outlive our planning period. As such, the choice of Kr is purely arbitrary. If we avoid making such a choice and take the problem to be one with a truncated vertical terminal line subject to Kr � 0, for K will inevitably dictate that Kr however, the optimal solution path = 0, because the maximization of utility over our finite planning period, but not beyond, requires that we use up all the capital by time but positive Kr. In Fig. T. More reasonably, we should pick an arbitrary 5.7, we assume that Kr > K0• The specification of K0 and Kr is not sufficient to pinpoint a unique optimal solution. In Fig. 5.7a, the streamlines labeled 81, . • • , 84 (as well as others not drawn), all start with K0, and are all capable of taking us 14See Paul A. Samuelson, "A Catenary Turnpike Theorem Involving Consumption and the Golden Rule," American Economic Review, June 1965, pp. 486-496. ;� 127 CHAPTER 5: INFINITE PLANNING HORIZON /1 - . _ __ ,., ·-··: > • . -: . ·; - ·· FIGURE 5.7 eventually to the capital level KT. The missing consideration that can pinpoint the solution is the length of the planning period, T. Let the T1 curve be the locus of points that represent positions attained on the various streamlines after exactly T1 years have elapsed from t 0. Such a curve, referred to as an isochrone (equal-time curve), displays a positive slope here for the following reason. Streamline 81, being the farthest away from the K' = 0 curve and hence with the largest K' value among the four, should result in the largest capital accumulation at the end of T1 years (here, Kr), whereas streamline 84, at the opposite extreme, should end up at time T1 with the least amount of capital (here, = :.- . PART 2, THE CALCULUS OF VARIATIONS 128 not much above K0). If the planning period happens to be T1 years, streamline S1 will clearly constitute the best choice, because while all streamlines satisfy the Euler equation, only S 1 satisfies additionally the boundary conditions K(O) K0 and K(T1) = Kr, with the indicated jour­ ney completed in exactly T1 years. Streamlines are phase paths, not time paths. Nevertheless, each streamline does imply a specific time path for each of the variables K and IL· The time paths for K corresponding to the four streamlines are illus­ trated in Fig. 5.7b, where, in order to align the K axis with that of Fig. 5.7a, we plot the time axis vertically down rather than horizontally to the right. In deriving these time paths, we have also made use of other isochrones, such as T2 and T3 (with a larger subscript indicating a longer time). Moving down each streamline in Fig. 5.7a, we take note of the K levels attained at various points of time (i.e., at the intersections with various isochrones). When such information is plotted in Fig. 5.7b, the time paths P11 , P4 emerge, with Pi being the counterpart of streamline Si . For the problem with finite planning horizon T1, the optimal time path for K consists only of the segment K 0G on P1 . Time path P1 is relevant to this problem, because S 1 is already known to be the relevant streamline; we stop at point G on P1, because that is where we reach capital level Kr at time T1. However, if the planning horizon is pushed outward to T2 , then we should forsake streamline S1 in favor of S3. The corresponding optimal time path for K will then be P3, or, more correctly, the K0J segment of it. Other values of T can also be analyzed in an analogous manner. In each case, a different value of T will yield a different optimal time path for K. And, collectively, all the finite-horizon optimal time paths for K are differ­ ent from the infinite-horizon optimal K path P2-the Ramsey path-asso­ ciated with streamline s2, the stable branch of the saddle point. With this background, we can discuss the turnpike behavior of finite-horizon optimal paths. First, imagine a family traveling by car be­ tween two cities. If the distance involved is not too long, it may be the simplest to take some small, but direct, highway for the trip. But if the distance is sufficiently long, then it may be advantageous to take a more out-of-the-way route to get onto an expressway, or turnpike, and stay on it for as long as possible, till an appropriate exit is reached close to the destination city. It turns out that similar behavior characterizes the finite-horizon optimal time paths in Fig. 5.7b. Having specified K0 as the point of departure and Kr as the destina­ tion level of capital, if we successively extend out the planning horizon sufficiently far into the future, then the optimal time path can be made to arch, to any arbitrary extent, toward the Ramsey path P2 or toward the KB line-the "turnpike" in that diagram. To see this more clearly, let us compare points M and N. If the planning horizon is T4, the optimal path is segment K0M on P4, which takes us to capital level Kr (point M) at time T4• If the horizon is extended to T5, however, we ought to select segment = • • • 129 CHAPTER 5: INFINITE PLANNING HORIZON K0 N on P3 instead, which takes us to capital level Kr (point N) at time T5• The fact that this latter path, with a longer planning period, arches more toward the Ramsey path and the K8 line serves to demonstrate the turnpike behavior in the finite-horizon model. The reader will note that the turnpike analogy is by no means a perfect one. In the case of auto travel, the person actually drives on the turnpike. In the finite-horizon problem of capital accumulation, on the other .hand, the turnpike is only a standard for comparison, but is not meant to be physically reached. Nevertheless, the graphic nature of the analogy makes it intuitively very appealing. EXERCISE 5.4 1 · 2 3 In the Eisner-Strotz model, describe the economic consequences of not adhering to the stable branch of the saddle point: (a) With K0 as given in Fig. 5.4, what will happen if the firm chooses .an initial rate of investment greater than I * 0? (b) What if it chooses an initial investment less than I* 0? In the Ramsey model with an infinite planning horizon, let there be capital saturation at K, but no consumption saturation. (a) Would this change of assumption require any modification of the system (5.40)? Why? ( b ) Should (5.41) and (5.42) be modified? Why? How? (c) Redraw Fig. 5.5 and derive an appropriate new K' = 0 curve. (d) In the phase space, also add an appropriate new Ji = 0 curve. On the basis of the new K' = 0 and Ji = 0 curves derived in the preceding problem, analyze the phase diagram of the capital-saturation case. In what way(s) is the new equilibrium similar to, and different from, the consumption-saturation equilibrium in Fig. 5.6? [ Hint: The partial derivatives in (5.43) are not the only ones that can be taken.] 4 Show that when depreciation of capital is considered, the net product can have capital saturation even if the gross product Q( K ) does not. 5 In the finite-horizon problem of Fig. 5.7a, retain the original K0, but let capital Kr be located to the right of K8. (a) Would the streamlines lying below the stable branch remain relevant to the problem? Those lying above? Why? (b) Draw a new phase diagram similar to Fig. 5.7a , with four streamlines as follows: S1 (the stable branch), S2, S3, and S4 (three successive relevant streamlines, with S2 being the closest to S1). Using isochrones, derive time paths P1, , P4 for the K variable, corresponding to the four streamlines. (c) Is turnpike behavior again discernible? . • • 130 PART 2: THE CALCULUS OF VARIATIONS 5.5 THE CONCAVITY / CONVEXITY SUFFICIENT CONDITION AGAIN F(t , It was earlier shown in Sec. 4.2 that if the integrand function y, y') in a fixed-endpoint problem is concave (convex) in the variables (y, y'), then the Euler equation is sufficient for an absolute maximum (minimum) of V[y ]. Moreover, this sufficiency condition remains applicable when the terminal time is fixed but the terminal state is variable, provided that the supplemen­ tary condition is satisfied. For the infinite-horizon case, this supplementary condition becomes15 (5.44) . Fy' ! is to be evaluated along the optimal path, and (jt y!'_) In this condition, represents the deviation of any admissible neighboring path y(t) from the optimal path y*(t). - Application to the Eisner-Strotz Model In the Eisner-Strotz model, the integrand function F(t, K, K') yields the following second derivatives, as shown in (5.16): and where all the parameters are positive. Applying the sign-definiteness test in (4.9), we thus have FK'K FKK i = 4'Cil.e -:,IP', > ·o. . ' .. " · '· .• I t follows that the quadratic form q associated \lrith these second derivatives is negative definite, and the integrand functi(m F js strictly concave in the variables (K, K'). · 15A more formal statement of this condition can · be found in G. Hadley and M. C. Kemp, Variational Methods in Economics, North-Holland, Amsterdam, 1971, p. 102, Theorem 2. 17. 5. CHAPTER 5: INFINITE PLANNING HORIZON form 131 For the present model, the supplementary condition (5.44) takes the lim [ FK' ( K (5 .45) where the (5 .46) t -+ oo - ;, K* ) ] :S:: O FK' term should specifically be FK' - ( 2aK' + b ) e - P1 = [ from (5.16) ] Substituting the K*'(t) expression in (5.23) for the K' term in (5.46), have we ( 5 .46') It is clear that FK' tends to zero as t becomes infinite. As for the ( K - K * ) component of (5.45), the assumed form of the quadratic profit function depicted in Fig. 5.1a suggests that as t becomes infinite, the difference between the K value on any admissible neighboring path and the K* value is bounded. Thus, the vanishing of FK' assures us that the supplementary condition (5.45) can be satisfied as an equality. Consequently, the strict concavity of the integrand function would make the Euler equation suffi­ cient for a unique absolute maximum in the total profit II(K). Application to the Ramsey Model The integrand function of the (simplified) Ramsey model is F( t , K, K' ) = B - U( C ) = B - U [ Q( K ) - K '] We assume that U'(C) � 0, U"(C ) < 0, Q'(K) > 0, and Q"( K ) the following first derivatives of F: FK = - U'( C) Q'( K ) FK. = and the second derivatives are found to be - U' ( C ) ( - 1 ) = < 0. From U'(C ) - FK' K' U"( C ) ( - 1) = U"( C ) FKK' = FK'K U" ( C ) Q' ( K ) FKK - U"( C ) [ Q' ( K ) ] 2 - U'( C ) Q''( K ) = = = Again applying the sign-definiteness test in (4.9), we find that,_, , ID t l ( 5.47) But since ( 5 . 48 ) ID:al ! F = FFKKK'K' FK'K ' KK = I = FK'K' > 0 U"( C ) U'(C)Q"( K) . . . .. � o r!_ :;.�· 132 PART 2' THE CALCULUS OF VARIATIONS is not strictly positive, we cannot claim that the associated quadratic form q is positive definite and that the F function is strictly convex. Nevertheless, it is possible to establish the (nonstrict) convexity of the F function by the sign-semidefiniteness test in (4.12). With reference to the principal minors defined in (4.11), we find that and These, together with the information in (5.47) and (5.48), imply that and Thus, by (4. 12), we can conclude that the integrand function F is convex. Turning to the supplementary condition (5.44), we need to confirm that ( 5.49) lim t .... oo [ Fd K - K*)] � 0 Since Fx· = U'(C), it is easy to see that as t becomes infinite (as we move toward Bliss and as the marginal utility steadily declines), Fx· tends to zero. As for the ( K - K* ) term, it is unfortunately not possible to state generally that the deviation of K(t) from K*(t) tends to zero, or is bounded, as t ..... oo. Since the production function is assumed to have positive marginal product throughout, the Q(K) curve extends upward indefinitely, and there is no bound to the economically meaningful values of K. In this connection, we see that the assumption of capital saturation would help in the application of the sufficiency condition. If Q(K) contains a saturation point, then (K - K*) will be bounded as t --> oo, and (5.49) can be satisfied as an equality. In fact, when capital saturation is assumed, condition (5.49) can be satisfied even if there is no consumption saturation. With the capital-saturation point K now serving to define " Bliss," all admissible paths must end at K. Thus the ( K - K*) component of (5.49) must tend to zero as t ..... oo. Since Fx· U'(C) is bounded as we approach Bliss, (5.49) is satisfied as an equality. Consequently, given the convexity of the F function, the Euler equation is in this case sufficient for an absolute minimum of the integral J;[ B - U(C)] dt. = CHAPTER 6 CONSTRAINED PROBLEMS In the previous discussion, constraints have already been encountered several times, even though not mentioned by name. Whenever a problem has a specified terminal curve or a truncated vertical or horizontal terminal line, it imposes a constraint that the solution must satisfy. Such con­ straints, however, pertain to the endpoint. In the present chapter, we concern ourselves with constraints that more generally prescribe the behav­ ior of the state variables. As a simple example, the input variables K and L may appear in a model as the state variables, but obviously their time paths cannot be chosen independently of each other because K and L are linked to each other by technological considerations via a production function. Thus the model should make room for a production-function constraint. It is also possible to have a constraint that ties one state variable to the time derivative of another. For instance, a model may involve the expected rate of inflation 7T and the actual rate of inflation expected rate, 7T, p as state variables. The is subject to revision over time when, in light of experi­ ence, it proves to be off the mark either on the up side or on the down side. The revision of the expected rate of inflation may straint such as d7T - =j (p - 7T ) dt as previously encountered i n (2.41). (0 < j :$; be captured by a con­ 1) While such a constraint substitution, be expunged from a model, it can also be can , through retained and explic­ itly treated as a constraint. The present chapter explains how this is done. 133 134 6.1 PART 2: THE CALCULUS OF VAR!ATIONS FOUR BASIC TYPES OF CONSTRAINTS Four basic types of constraints will be introduced in this section. As in static optimization problems with constraints, the Lagrange-multiplier method plays an important role in -the treatment of constrained dynamic optimization problems. Equality Constraints Let the problem be that of maximizing (6.1) subject to a set o f m independent but consistent constraints (m < n) (6.2) and appropriate boundary conditions. By the " independence" of the m constraints, it is meant that there should exist a nonvanis,hing Jacobian determinant of order m , such as {6.3) But, of course, any m of the Yj variables can be used in IJI , not necessarily the first m of them. Note that, in this problem, the number of constraints, m , is required to be strictly less than the number of state variables, n. Otherwise, with (say) m = n, the equation system (6.2) would already uniquely determine the y/t) paths, and there would remain no degree of freedom for any optimizing choice. In view of this, this type of constrained dynamic optimization problem ought to contain at least two state variables, before a single constraint can meaningfully be accommodated. In line with our knowledge of Lagrange multipliers in static optimiza­ tion, we now form a Lagrangian integrand function, 9", by augmenting the original integrand F in (6.1) as follows: = m F + L, A;(t) ( C; - g i ) i- 1 ,. . ... _·> .:· CHAPTER s, CONSTRAINED PROBLEMS 135 Although the structure of this Lagrangian function appears to be identical with that used in static optimization, there are two fundamental differences. First, whereas in the static problem the Lagrange-multiplier term is added to the original objective function, here the terms with Lagrange multipliers A i are added to the integrand function F, not to the objective functional J0TFdt. Second, in the present framework, the Lagrange multipliers A; appear not as constants, but as functions of t. This is because each constraint gi in (6.2) is supposed to be satisfied at every point of time in the interval [0, T ), and to each value of t there may correspond a different value of the Lagrange multiplier A; to be attached to the (c; g i ) expression. To emphasize the fact that A i can vary with t, we write A;(t). The Lagrangian integrand !F has as its arguments not only the usual t, y1 , and yj, (j 1, . . , n), but also the multipliers A ; , (i = 1, . . . , m). The replacement of F by !F in the objective functional gives us the new functional - = . ( 6 .5) which we can maximize as if it is an un constrained problem. As long as all of the constraints in (6.2) are satisfied, so that c; gi = 0 for all i , then the value of !F will be identical with that of F, and the free extremum of the functional "Y in (6.5) will accordingly be identical with the constrained extremum of the original functional V. It is a relatively simple matter to ensure the requisite satisfaction of all the constraints. Just treat the Lagrange multipliers as additional state variables, each to be subjected to an Euler equation, or the Eu ler-Lagrange equation , as it is sometimes referred to in the present constrained frame­ work. To demonstrate this, let us first note that the Euler-Lagrange equa­ tions relating to the y1 variables are simply - (6.6) d !FYj - - :FY . = O dt j for all t E [0, T ] ( j = l , . . , n ) (cf. ( 2 .27)] . As applied to the Lagrange multipliers, they are, similarly, ( 6 . 7} .9;., - d 0 dt 5";.,· = for all t E [0, T ] ( i = 1, . . . , m) However, since !F is independent of any A.[, we have .9*;. '.- = 0 for every i, that the m equations in (6. 7) reduce to ( 6.7') 5";.. , = 0 or for all t E [ 0, T ] so (by ( 6 .64)) which coincide with the given constraints. Thus, by subjecting all the Ai . · ·'··. PART 2: THE CALCULUS OF VARIATIONS 136 variables to the Euler-Lagrange equation, we can guarantee the satisfaction of all the constraints. As a matter of practical procedure for solving the present problem, (1) therefore, we can apply the Euler-Lagrange equation to the n state yi only, as in (6.6), (2) take the m constraints exactly as originally given in (6 . 2 ) , and (3) use these n + m equations to determine the y/t) and the A ;(t) paths. In step (1), the resulting n differential equations will variables contain 2n arbitrary constants in their general solutions. These may be definitized by the boundary conditions on the state variables. EXAMPLE 1 Find the curve with the shortest distance between two given = points A = (0, z0) and = 0. Y r, zr) lying on a surface The distance between the given pair of points is measured bf. the integral f[(1 + y'2 + z'2) 1 /2 which resembles the integral f (1 + y'2) 112 for the distance between two points lying in a plane. Thus the problem is to y0, B dt ( T, dt, cf>(t, y, z ) 0 Minimize 4>( t,y, z) = 0 y(O) = y0 , y(T) subject to and = Yr• z ( O ) = z0 , z ( T ) This is a problem with two state variables (n 2) and (m = 1). As the first step, we form the Lagrangian integrand = .9'= F + A( t) (O - 4> ) = F - >.( t )4> = { 1 + y' 2 + z'2 )112 - A( t) cf>( t , y , z ) Partially differentiating .9', we .9;" = see that - A( t)lf>z These derivatives lead to the two Euler-Lagrange equations - A( t) 4>y - :t [ y' ( 1 + y'z + z' 2) - I/2] = 0 - A{t ) 4>. - :t [ z' (l + y'2 + z'zfi/2] =0 = Zr one constraint CHAPTER 6' CONSTRAINED PROBLEMS 137 which, together with the constraint 4>( t' y' z ) =0 give us three equations to determine the three optimal paths for y, z, and A . Differential-Equation Constraints Now suppose that the problem is to maximize (6. 1), subject to a consistent set of m independent constraints ( m < n) that are differential equations:1 ( 6 .8) and appropriate boundary conditions. Even though the nature of the con­ straint equations has been changed, we can still follow essentially the same procedure as before. The Lagrangian integrand function is still and the Euler-Lagrange equations with respect to the state variables still in the form of !F. Yj d - - !F_ , dt Yj = 0 for all t E [0, T ] (j = l, yJ are . . . , n) Moreover, similar equations with respect to the Lagrange multipliers A; are again nothing but a restatement of the given constraints. So we have a total of n Euler-Lagrange equations for the state variables, plus the m con­ straints to determine the n + m paths, y; ( t ) and A;(t), with the arbitrary constants to be definitized by the boundary conditions. Inequality Constraints When the constraints are characterized by inequalities, the problem can 1The independence among the m differential equations in (6.8) means that there should exist a nonvanishing Jacobian determinant of order m, such as Again, any m of the Y/ can be used in the Jacobian. 138 PART 2: THE CALCULUS OF VARIATIONS generally be stated as follows: Maximize f0TF(t, yl, . . . , yn , YI'• • . . , yn' ) dt subject to g1 ( t, y t• · · · • Yn , Yt', . . . , y:) and appropriate boundary conditions (6.9) :5 cl Since inequality constraints are much less stringent then equality con­ straints, there is no need to stipulate that m < n . Even if the number of constraints exceeds the number of state variables, the inequality constraints as a group will not uniquely determine the Yj paths, and hence will not eliminate all degrees of freedom from our choice problem. However, the inequality constraints do have to be consistent with one another, as well as with the other aspects of the problem. To solve this problem, we may again write the Lagrangian integrand as While the Euler-Lagrange equations for the Yj variables, for all t E [0, T ] {j = l , . . . , n ) are no different from before, the corresponding equations with respect to the Lagrange multipliers must be duly modified to reflect the inequality nature of the constraints. To ensure that all the Ai(tX c i - gi) terms vanish in the solution (so that the optimized values of .r and F are equal), we need a complementary-slackness relationship between the i th multiplier and the i th constraint, for every i (a set of m equations): for all t E [0, T ] (i = l, . . . , m) This complementary-slackness relationship guarantees that (1) whenever the i th Lagrange multiplier is nonzero, the i th constraint will be satisfied as a strict equality, and (2) whenever the ith constraint is a strict inequal­ ity, the i th Lagrange multiplier will be zero. It is this relationship that serves to maintain the identity between the optimal value of the original integrand F and that of the modified integrand .r in (6.4). "; ! ' . • CHAPTER 6, CONSTRAINED PROBLEMS 139 Isoperimetric Problem The final type of constraint to be as exemplified by the equation (6.11) considered bends thtHn:tegral constraint, · fa ( t , y , y' ) dt = k · (k 0 = constant) One of the earliest problems involving such a constraint is that of finding the geometric figure with the largest area that can be enclosed by a curve of some specified length. Since all the figures admissible in the problem must have the same perimeter, the problem is referred to as the isoperimetric problem . In later usage, however, this name has been extended to all problems that involve integral constraints, that is, to any problem of the general form ( TF( t, yt, Maximize ), subject to ), 0 , Yn• Yt • · · · , yn ) dt 1 · · · 1 ( TG ( t, Yt> · · · • Yn, Yt • · · · • Yn ) dt - kl 1 0 (6. 12) f I f0TGm( t, yl> . . . , yn , yl'• . . . , y: ) dt = k m appropriate boundary conditions and In the isoperimetric problem, there is again no need to require m < n , because even with m � n, freedom of optimizing choice is not ruled out. Two characteristics distinguish isoperimetric problems from other con­ strained problems. First, the constraint in (6. 11) is intended, not to restrict the y value at every point of time, but to force the integral of some function to attain a specific value. In some sense, therefore, the. constraint is more indirect. The other characteristic is that the solution values of the Lagrange multipliers A i (t) are all constants, so that they can simply be written as A ; . G To verify that the Lagrange multipliers indeed are constants in the solution, let us consider the case of a single state variable and a single integral constraint ( 6 .13) (m = n = 1). r ( t) y First, let us define a function = f0 a c t , y, y') dt T, Note that the upper limit of integration here is not a specific value but the variable t itself. In other words, this integral is an indefinite integral 140 PART and hence a function of t. At (6.14) f( O ) = f0 o dt = o t = 2' THE CALCULUS OF VARIATIONS 0 and t = T, respectively, we have and r( T ) = f0 o dt = k (by (6.11}1 It is clear that f(t) measures the accumulation o f G from time 0 to time t. For our immediate purpose, we note from (6.13) that the derivative of the f(t) function is simply the G function. So we can write G ( t , y , y' ) - f'( t ) (6.15) = 0 which is seen to conform to the general structure of the differential con­ straint g(t, y, y') = c, with g = G - f' and c = 0. Thus we have in effect transformed the given integral constraint into a differential-equation con­ straint. And we can then use the previously discussed approach of solution. Accordingly, we may write the Lagrangian integrand (6.16) F = F( t, y , y' ) - + A ( t ) (O - G ( t , y, y' ) + f'( t ) ] = F( t , y , y') - A ( t ) G ( t , y , y' ) + A ( t ) f' ( t ) (We denote this particular Lagrangian integrand by F rather than sr, because it is merely an intermediate expression which will later be replaced by a final Lagrangian integrand.) In contrast to the previously encountered sr expressions, we note that F involves an extra variable r -although the latter enters into F only in its derivative form f'(t). This extra variable, whose status is the same as that of a state variable, must also be subjected to an Euler-Lagrange equation. Thus, there arise two parallel conditions in this problem: ( 6.17) d Fy - t FY, = O d ( 6.18) Fr - t Fr' = O d - d - for all t for all t E [0, T ] E [0, T ] However, inasmuch as F is independent of r, and since Fr' that (6.18) reduces to the condition (6.18') d - - A( t ) dt = 0 - = A(t), we see A(t) = constant This then validates our earlier claim about the constancy of A. In view of this, we may write the Lagrange multiplier of the isoperimetric problem simply as A. CHAPTER 6: CONSTRAINED PROBLEMS Turning next to 141 (6. 17) and using (6.16), we can obtain a more specific version of that Euler-Lagrange equation: ( 6.19) But a moment's reflection will tell us that this same condition could have been obtained from a modified (abridged) version of F without the term in it, namely, f'(t) !F= F( t,y , y' ) ( 6 .20) - AG ( t, y,y' ) ( A = constant) Thus, in the present one-state-variable problem with a single integral constraint, we can as a practical procedure use the modified Lagrange integrand .r in (6.20) instead of F, and then apply the Euler-Lagrange equation to alone, knowing that the Euler-Lagrange equation for A will merely tell us that A is a constant, the value of which can be determined from the isoperimetric constraint. The curious reader may be wondering why the Lagrange-multiplier y term AG enters in .r in (6.20) with a minus sign. The answer is that this AC - Ag follows directly from the standard way we write the Lagrangian integrand, g) = has a minus sign attached to A g. When where the term A(c written in this way, the Lagrange multiplier A would come out with a positive sign in the solution, and can be assigned the economic interpreta­ tion of a shadow price. The foregoing procedure can easily be generalized to the n-state­ variable, m-integral-constraint case. For the latter, the modified Lagrangian function is ( A; are all constants) Find a curve AB, passing though two given points A = (0, and B = (T, Y r ) in the plane, and having a given length k, that maxi­ mizes the area under the curve. The problem, as illustrated in Fig. 6.1, is to EXAMPLE 2 Maximize subject to and For the problem to y0) ty V= f0Ty dt 1/2 fT (1 + y'2 ) dt = k 0 y( O) = Yo y( T ) = Yr be the meaningful, the constant k shall be assumed to be larger then the length of line segment AB, say, L . (If k < L , the problem has no solution, for the curve AB in question is impossible to draw. Also, if PART 2: THE CALCULUS OF VARIATIONS 142 y t nG\JRE 6.1 L, then the curve AB can only be drawn as a straight line, and no optimizing choice exists.) To find the necessary condition for the extremal, we first write the Lagrangian integrand a Ia (6.20): k = (6.22) (A = constant) Since this does not explicitly contain t as an argument, we mq · take advantage of formula (2.21), and write the Euler-Lagrange equatiort iD a form that has already been partly solved, namely, · S"- y'�· (6.23) Since �· ( 6.23' ) = = c1 · ( c1 = arbitrary constant) - Ay'(1 + y'2 )-112, (6.23) can be expressed more s� M - 1/2 1/2 y - A(1 + y' 2 ) + Ay'2 (1 + y'2 ) = c1 Via a series of elementary operations to be described, this equation solved for y' . Transposing terms and simplifying, we can write or can be · ·' - � $quaring both sides and subtracting 1 from both sides, we get Az - (y - cl)2 y' 2 = --'-�i"­ (y - cl)2 ;· , '· The squani root of this gives us the following expression of y' in tenns of A, y, and c1: · .· · :. ': CHAPTER 6o 143 CONSTRAINED PROBLEMS Since y' = dyjdt, however, the last result may alternatively be written as y - ci dy --;==== =7 z (6.23") V>..2 - ( Y - c i ) dt = which the reader will recognize as a nonlinear differential equation with separable variables, solvable by integrating each side in turn. The integral of dt, the right-hand-side expression in (6.23"), is simply t + constant. And the integral of the left-hand-side expression is J.? - (y - c1)2 + constant.2 Thus, by equating the two integrals, con­ solidating the two constants of integration into a single symbol, -c2 , squaring both sides, and rearranging, we finally obtain the general solution -V ( 6 .24) Since this is the equation for a family of circles, with center (c2, c 1) and radius >.. , the desired curve AB must be an arc of a circle. The specific values of the constants c 1 , c2 , and >.. can be determined from the boundary conditions and the constraint equation. EXERCISE 6.1 I Prepare a summary table for this section, with a column for each of the following: (a) type of constraint, (b) mathematical form of the constraint, (c) m < n or m � n? (d) A; varying or constant over time? 2 Work out all the steps leading from (6.23) to (6.23"). 3 Find the extremal (general solution) of V = 4 5 6 j[y'2 dt, subject to J0Ty dt = k. Find the y(t) and z(t) paths (general solutions) that give an extremum of the functional <y'2 + z'2 ) dt, subject to y - z' = 0. With reference to Fig. 6.1, find a curve AB , with a given length and passing through the given points A and B, that produces the largest area between the curve AB and the line segment AB. Jt Find a curve AB with the shortest possible length that passes thro1,1gh two given points A and B, and has the area under the curve equal to a given constant K. [Note: This problem serves to illustrate the principle of reciprocity in isoperimetric problems, which states that the problem of maximum area enclosed by a curve of a given perimeter and the problem of minimum perimeter for a curve enclosing that area are reciprocal to 2Let x ., y - c1, so that dx - dy. The integral of the left-hand-side expression of(6.23") is then Jxf .;J.2. - x2 dx. to standard tables of integrals (e. ., Formula 204 in CRC Raton, FL, 1987), this integral is equal to - .;A2 - x2 - - VA2 - (y - c1) 2 , plus constant of integration. - According g Standard Mathematical Tables, 28th ed., CRC Press, a Boca 144 PART 2, THE CALCULUS OF VARIATIONS each other and share the same extremal.) [ Hint: When working out this problem, it may prove convenient to denote the Lagrange multiplier by p., and later define A 1 /p., to facilitate comparison with Example 2 of this section.) = 6.2 SOME ECONOMIC APPLICATIONS REFORMULATED While the Lagrange-multiplier method can play a central role in constrained problems, especially in problems with general functions, it is also possible, and sometimes even simpler, to handle constrained problems by substitu­ tion and elimination of variables. Actually, most of the economic models discussed previously contain constraints, and in the solution process we have substituted out certain variables. In the present section, we shall reconsider the Ramsey model and the inflation-unemployment tradeoff model, to illustrate the method of Lagrange multipliers. The results would, of course, be identical regardless of the method used. The Ramsey Model In the Ramsey model (Sec. 5.3), the integrand function in the objective functional is F = B - U( C ) + D( L ) where C = Q( K, L) - K' Previously, the F function was considered to contain only two variables, K and L, because we substituted out the variable C. When we took the derivatives FK and FK'• in particular, the chain rule was used to yield the results The use of this rule reduces C to an intermediate variable that disappears from the final scene. To show how the Lagrange-multiplier method can be applied, let us instead treat C as another variable on the same level as K and L. We then recognize the constraint g ( C , L , K, K' ) = Q( K, L ) - K' - C = 0 and reformulate the problem as one with an explicit constraint: Minimize ( 6.25) ('lB - U( C ) + D( L ) ] dt subject to Q( K, L ) - K' - C = O and boundary conditions 145 CHAPTER 6: CONSTRAINED PROBLEMS By (6.4), we write the Lagrangian integrand function as ( 6 .26) fF= B - U( C ) + D( L ) + A[ - Q( K , L) + K ' + C ] Since we now have three variables (C, L, K ) and one Lagrange multiplier A (not a constant), there should be altogether four Euler-Lagrange equations: ( 6.2 7) ( 6.28) ( 6.29) ( 6.30) d !J'C, = - U'(C) + A = - J.L + A = 0 => 9(; - dt d .9'£ .9'£· = D'(L) - AQL = D'(L) - J.LQL = 0 dt d d dJ.L fFK - dt fFK' = - AQK - dt A = - J.L QK - dt = 0 d fFx = - Q( K, L) + K' + C = 0 � - dt A = J.L [by ( 6.2 7 ) ] [by (6.2 7 ) ] Condition (6.2 7) tells u s that the Lagrange multiplier A i s equal to J.L-the symbol we have adopted for the marginal utility of consumption. Condition (6.28) is identical with (5.2 7). Similarly, (6.29) conveys the same information as (5.28). Lastly, (6.30) merely restates the constraint. Thus the use of the Lagrange-multiplier method produces precisely the same conclu­ sions as before. The Inflation-Unemployment Tradeoff Model The inflation-unemployment tradeoff model of Sec. 2.5 will now be reformu­ lated as a problem with two constraints. As might be expected, the presence of two constraints makes the solution process somewhat more involved. For notational simplicity, we shall use the symbol y to denote Yr - Y (the deviation of the current national income Y from its full-employment level Y,). Then the integrand function F-the loss function (2.39)-can be written as (a > 0) The expectations-augmented Phillips relation (2.40) and the adaptive-expec­ tations equation (2.41), which were previously used to substitute out y and p in favor of the variable 1r, will now be accepted as two constraints: g1( t, y, p , 7r, 7r1 ) = f3y + p - 7r = g2( t, y, p , 7r , 7r1 ) = j( p - 7T ) - 7T' 0 = [ from ( 2 .40)] 0 [from ( 2 .4 1 ) ] The problem thus becomes one where one of the constraints involves a 146 PART 2, THE CALCULUS OF VARIATIONS simple equality, and the other constraint involves a differential equation: jT(y 2 + ap 2 )e-pt dt f3y + p - 'TT' = 0 j(p - Tr) - Tr' = O Minimize (6.3 1 ) " ·:.. 0 subject to ;,:,· · boundary conditions and The Lagrangian integrand function is (6.32 ) !F= (y2 + ap 2 ) e -pt + A l( -f3y - p + 'TT' ) + .\2( -jp + j'TT' + 7r' ) with three variables y, p, and and two (nonconstant' Lagrange multipli­ ers .\1 and A 2 . The Euler-Lagrange equations are 'TT' d ( 6.33) !F. y (6.34) !F_ - - !F_, = 2ape -P1 - A - J·A = 0 dt (6.35) p ( 6 . 3 7) .9:A t :1 dt 5'. d d � · , '· ' ·: ( 6.36) - - dt d = 2ye -pt - f3 A = 0 -� 1 1 p �· = A t + j.\ 2 - d dt sr,;, - j 2 A2 = 0 -dt .9: . = - f3y - p + Tr = 0 d ·! At 0 = p dt sr.;, -j + j'TT' + 'TT'' = ' . ·�· ·; ,..- ... To solve these equations simultaneously requires quite a few steps. One way of doing it is the following. First, solve (6.36) for y, substitute the result into (6. 33), and solve for .\1 to get 2 A I = f3 2 (7r - p)e-pt Next, solve (6.37) for the variable p: (6.38) � .. '· '· . 1 p = 'TT' + -;'TT' J ' ( 6.39) Substitution of (6.39) into (6. 38) results in th, ·'i�i!)aHan of the AI expression: ·· (6.40 ) . ; . .. � 'TT' and p in CHAPTER 6: CONSTRAINED PROBLEMS 147 Then substituting both (6.39) and (6.40) into (6.34) and solving for A 2 , we obtain (after simplification) ( 6.4 1) This result implies the total derivative (6.42) dA2 dt -- = Finally, we can substitute the A1 , A2 , and dA2jdt expressions of the last three equations into (6.35) to get a single summary statement of the five simultaneous Euler-Lagrange equations. After combining and canceling terms, the complicated expressions reduce to the surprisingly simple result ( 6.43) This result qualifies as the condensed statement of conditions (6.33) through (6.37) because all five equations have been used in the process and incorpo­ rated into (6.43). Since (6.43) is identical with the earlier result (2.46), we have again verified that the Lagrange-multiplier method leads to the same conclusion. Note, however, that this time the Lagrange-multiplier method entails more involved calculations. This shows that sometimes the more elementary elimination-of-variable approach may work better. It is in situations where the elimination of variables is unfeasible (e.g., with general functions) that the Lagrange-multiplier method shows its powerfulness in the best light. EXERCISE 6.2 1 The constraint in the Ramsey model (6.25) could have been equivalently written as C - Q(K, L ) + K' = 0. Write the new Lagrangian integrand function and the new Euler-Lagrange equations. How do the analytical results differ from those in (6.27) through (6.30)? ... ..,_ � ·. : 148 PART 2 2: THE CALCULUS OF VARIATIONS The Eisner-Strotz model of Sec. 5.2 can be reformulated as a constrained problem: Maximize subject to j�( 0 -rr - C and - -rr - C)e-P' dt 2 o:K + {3K 2 aK' - bK' = = 0 0 boundary conditions (a) How many variables are there in this problem? How many constraints? (b) Write the Lagrangian integrand function. (c) Write the Euler-Lagrange equations and solve them. Compare the result with (5.17). 6.3 THE ECONOMICS OF EXHAUSTIBLE RESOURCES In the discussion of production functions, there is usually a presumption that all the inputs are inexhaustible, so that as they are being used up, more of them can be obtained for future use. In reality, however, certain re­ sources-such as oil and minerals-are subject to ultimate exhaustion. In the absence of guaranteed success in the discovery of new deposits or substitute resources, the prospect of eventual exhaustion must be taken into account when such a resource is put into use. The optimal production (extraction) of an exhaustible resource over time offers a good illustration of the isoperimetric problem. The Hotelling Model of Socially Optimal Extraction In a classic article by Harold Hotelling,3 the notion of " the social value" of an exhaustible resource is used for judging the desirability of any extraction pattern of the resource. The gross value to society of a marginal unit of output or extraction of the resource is measured by the price society is willing to pay to call forth that particular unit of output, and the net value to society is the gross value less the cost of extracting that unit. If the price of the resource, P, is negatively related to the quantity demanded, as illustrated in Fig. 6.2, then the gross social value of an output Q0 is 3 Harold Hotelling, " The Economics of Exhaustible Resources," Journal of Political Economy, April 1931, pp. 137-175. - �· - '· ·•. �: 149 CHAPTER 6: CONSTRAINED PROBLEMS p P(Q) 0 ----- Q FIGURE 6.2 measured by the shaded area under the curve, or the integral f(/0P(Q) dQ. To find the net social value, we subtract from the gross social value the total cost of extraction C (Q). Generalizing from a specific output level Q0 to any output level Q, we may write the net social value of the resource, N, as:4 ( 6.44) N( Q ) = j QP( Q ) dQ - C( Q ) 0 To avoid any confusion caused by using the same symbol Q as the variable of integration as well as the upper limit of integration, we can alternatively write ( 6 .44) as (6.44') N( Q ) = jQP(x) dx - C( Q ) 0 Assuming that the total stock of the exhaustible resource is given at S0, we have the problem of finding an extraction path Q(t) so as to Maximize (6.45) subject to V= {N ( Q ) e-pt dt 0 {'Qdt = S0 0 This, of course, is an isoperimetric problem. Since it is reasonable to expect the net-social-value function to have an upper bound, the improper integral in the objective functional should be convergent. By (6.20), the Lagrangian integrand is ( 6.46) !F= N( Q ) e-pt - >.Q (A = constant) 4 ln Hotelling's paper, the extraction cost is subsumed under the symbol P, which is inter­ preted as the " net price" of the exhaustible resource. Accordingly, the C(Q) term does not appear in his treatment. 150 PART 2, THE CALCULUS OF VARIATIONS Note that the integrand sr does not contain the derivative of Q; hence the extremal to be obtained from the Euler-Lagrange equation may not fit given fixed endpoints. If no rigid endpoints are imposed, however, we can still apply the Euler-Lagrange equation, which, because SQ· = 0, reduces in the present case to the condition .9Q = 0, or � N'(Q)e-pt - A = 0 (6 .47) Upon applying the differentiation formula (2.8) to N(Q) in (6.44'), we have [ P( Q) - C'(Q)] e-P1 - A = 0 (6. 47') The economic interpretation of this condition becomes easier when we recall that, for an isoperimetric problem, the Lagrange multiplier A is a constant. Along the optimal extraction path, the value of P(Q) - C ' (Q) associated with any point of time must have a uniform present value, A. By slightly rearranging the terms, we can reinterpret the condition as requiring that P(Q) - C'(Q) grow at the rate p: (6 .4 7" ) P(Q) - C ' (Q) = AeP1 [social optimum] From the last equation, it is also clear that A has the connotation of " the initial value of P(Q) - C'(Q)." If the P(Q) and the C(Q) functions are specific, we can solve (6.47'') for Q in terms of A and t, say Q(A, t). The latter, when substituted into the constraint in (6.45), then enables us to solve for A. Pure Competition Versus Monopoly One major conclusion of the Hotelling paper is that pure competition can yield an extraction path identical with the socially optimal one, whereas a monopolistic firm will adopt an extraction path that is more conservationis­ tic, but socially suboptimal. We shall now look into this aspect of the problem. Assume for simplicity that there are n firms under pure competition. The th firm extracts at a rate Qi out of a known total stock of Si under its control. The firm's problem is to maximize the total discounted profits over time, taking the product price to be exogenously set at P0: i ( 6.48) . Maximize subject to ' {0 [ P0Qi - Ci(Qi ) ] e -pt dt (0 Qi dt = si The Lagrangian integrand now becomes , . ,_. 1,;; ' CHAPTER 6: CONSTRAINED PROBLEMS 151 and the application of the Euler-Lagrange equation yields the condition ( Po - C/(Q;)]e-pt - A = 0 or [pure competition] (6.49} This condition is perfectly consistent with that for social optimum, (6.47"), in that it, too, requires the difference between the price of the exhaustible resource and its marginal cost of extraction to grow exponentially at the rate p. In contrast, the problem of monopolistic profit maximization is Maximize ( 6.50) ( [ R(Q) - C(Q)]e -pt dt 0 subject to This time, �tb the Lagrangian integrand :T= [ R(Q) - C(Q)) e - pt - >.Q the Euler-Lagrange equation leads to the condition R'(Q) - C'( Q) = AeP1 (6.51) [monopoly) which differs from the rule for socially optimal extraction. Here, it is the difference between the marginal revenue (rather than price) and the marginal cost that is to grow at the rate p, with A now representing the initial value of the said difference. The Monopolist and Conservation · · . : · · The conclusion of the Hotelling paper is, however, not only that monopolis­ tic production of an exhaustible resource is suboptimal, but also that it is specifically biased toward excessive conservationism. How valid is the latter claim? The answer is: It is under certain circumstances, but not always. This issue has been examined by various people using different specific assumptions. As one might expect, differences in assumptions result in different conclusions. In a paper by Stiglitz,5 for example, it is shown that if the elasticity of demand increases with time (with the discovery of substi­ tutes), or if the cost of extraction is constant per unit of extraction but 5 Joseph E. Stiglitz, "Monopoly and the Rate of Extraction of Exhaustible Resources," Ameri., • can Economic Reuiew, September 1976, pp. 655-661 . .f i' ... 152 PART 2o THE CALCULUS OF VARIATIONS decreases with time (with better technology), then the monopolist tends to be more conservationistic than under the social optimum. But the opposite conclusion is shown to prevail in a paper by Lewis, Matthews, and Burness,6 under the assumption that the extraction cost does not vary with the rate of extraction (capital costs, leasing fees, etc., are essentially fixed costs), and that the elasticity of demand increases with consumption (sufficiently low prices can attract bulk users at the margin to switch from substitutes). In one of Stiglitz's submodels, the monopolist faces the (inverse) demand function ( 6 .52 ) ( 0 < a < 1) - with elasticity of demand 1/(1 a). The per-unit cost of extraction is constant at any given time, but it can decline over time: c = tj>(t)Q (6 .53) ( 4>' < 0 ) Thus the profit is PQ - C <= 1/J(t)Qa - tj>(t)Q And the monopolist's dynamic optimization problem i s to Maximize { f l/l ( t)Q" - tf>(t)Q] e-pt dt subject to {Q dt (6.54) 0 0 = 80 The Lewis-Matthews-Burness model, on the other hand, uses a general (inverse) demand function P(Q) that is stationary, although its elastidty of demand is assumed to increase with consumption. Since the cost of extrac­ tion is assumed to be a fixed cost ( = <I>), the monopolist's problem is to Maximize (6.55) ( [ P( Q ) Q - <l>]e-P1 dt subject to The Euler-Lagrange equation is applicable to each of these two prob­ lems in a manner similar to the Hotelling model. From that process, we can deduce an optimal rate of growth of Q for the monopolist. Comparing that Tracy R. Lewis, Steven A Matthews, and H. Stuart Burness, "Monopoly and the Rate of Extraction of Exhaustible Resources: Note," American Economic Review, March 1979, pp. 6 227-230. 1 53 CHAPTER 6: CONSTRAINED PROBLEMS rate to the rate of growth dictated by the social optimum will then reveal whether the monopolist is excessively or insufficiently conservationistic. Instead of analyzing problems (6.54) and (6.55) separately, however, we shall work with a single, more general formulation that consolidates the considerations present in (6.54) and (6.55). Let the demand function and the cost function be expressed e81D ( P ) ( 6.56) Q ( 6.57) C = C ( Q , t) = as [ D' ( P ) < 0] ( CQ � o, ctQ � o) These functions include those used by Stiglitz and Lewis, Matthews, and Burness as special cases. In the following, we shall use the standard abbreviations for marginal revenue and marginal cost, MR = R'(Q) and MC = C'(Q), as well as the notation rx = (dxjdt)jx for the rate of growth of any variable x. The condition for social optimum that arises from the Euler-Lagrange equation, (6.47"), requires that r< P - MC) = p. Using a familiar formula for the rate of growth of a difference, 7 we can rewrite this as p And this implies that - rP (6 .58) P r MC p = p(l - - _ p MC MC p ) r Mc MC + MC We can translate this rp into a corresponding linked to rp via the elasticity of demand follows:8 p =P rMC rQ by using the fact that rQ is E < 0, or its negative, E = \ E \ , as (6.59) 7 See Alpha C. Chiang, Fundamental Methods ofMathematical Economics, 3d ed., McGraw-Hill, New York, 1984, Sec. 10.7. 8 First take the natural log of both sides of (6.56) to get ln Q = gt + In D ( P ) Then derive rQ b y differentiating In Q with respect t o t : d d 1 d 1 dP rQ = - ln Q = g + - ln D ( P ) = g + -- - D ( P ) = g + -- D' ( P ) dt dt D ( P ) dt D(P) dt =g + D' ( P )P dPjdt -p- = g + trp = g - Erp D(P) (where E = lEI) 154 PART 2: THE CALCULUS OF VARIATIONS Substituting (6.58) into (6.59), we then find the socially optimal rate of growth of Q, denoted by rQ. (s for social optimum), to be ( 6 .60 ) [ ( �) � ] C rQ• = g - E p 1 - C + [social optimum] rMc The counterpart expression for rQm (m for monopoly) can be derived by a similar procedure. Condition (6.51) requires that r(MR - MC> = p , which may be rewritten as MR MR - MC rMR - MC rM MR - MC c = P And this implies that ( 6.61 ) However, since MR and P are related to each other by the equation ( 6 .62) we may derive another expression for ( 6 ·63) rMR "" 1 dMR MR ----;it = dPjdt 1 P ( 1 - 1 /E ) 1 [( rMR from (6.62), namely, ) + E2 dt J 1 dP 1E dt dEjdt P dE 1 rE = p - + (1 - = rp + E /E ) E E - 1 - 1 EquatiJW (6.61) and (6.63), solving for rp, and then substituting into (S.t)W, we get the rate of growth of Q under monopoly: · (6.64) [ ( �) :� rQm = g - E p 1 - + rMc E � r 1 E ] · [monopoly] A comparison of (6.64) with (6.60) reveals that we can have ifMC ( 6.65) = rMc = rE = 0 The situation of MC 0 occurs when extraction is costless. For rMc = 0, the marginal cost of extraction has to be time invariant [ C1Q = 0 in (6.57)]. And the condition rE = 0 is met when the demand elasticity is constant over time. This set of conditions-also considered by Stiglitz-is quite stringent, but if the conditions are met, then the monopolist will follow exactly the same extraction path as called for under the social optimum, which is in no = \ :�- CHAPTER 6: CON�NED PROBLEMS 155 Q Path with larger rQ 0 '----- t FIGURE 6.3 - way overconservationistic. Note that the Ep term is negative and implies a declining Q over time, unless offset by a sufficiently large positive value of g. If the demand for the exhaustible resource is stationary (g 0), or shifts downward over time (g < 0) as a result of, say, the discovery of substitutes, then rq must be negative. A different outcome emerges, however, when there is a positive MC that is output-invariant (MC k > 0) as well as time-invariant (rMc = 0). Retaining the assumption that rE = 0, we now find = = ( 6.66) rq . = g - [ ( �}] E p 1- and Since P > MR, it follows that kjP is less than kjMR, so that rQ• < rQ m at any level of Q. The social optimizer and the monopolist now follow different Q(t) paths. Recall, however, that both paths must have the same total area under the curve,. namely, 80. To satisfy this constraint, the monopolist's Q(t) path, with the larger rq value (e.g., - 0.03 as against - 0.05), must be characterized by a more gentle negative slope, as illustrated by the solid curve in Fig. 6.3. Since such a curve entails lower initial extraction and higher later extraction than the broken curve, the monopolist indeed emerges in the present case as a conservationist, relative to the social optimizer. We have reached the same conclusion here as Stiglitz, although the assump­ tions are different. Nevertheless, it is also possible to envisage the opposite case where the monopolist tends to undertake excessive early extraction. Assume again that MC = rMc 0. But let the elasticity of demand E vary with the rate of extraction Q. More specifically, let E '(Q) > 0-a case considered by Lewis, Matthews, and Burness. Then, as Q varies over time, E must vary accord­ ingly, even though E is not in itself a function of t. With these assump­ tions, we now have a nonzero rate of growth of E: = dEjdt E rE = -- = ( 6.67 ) ' . ·�- · ;; E' ( Q ) ( dQjdt) E E' ( Q)Q rQ E 156 PART 2o THE CALCULUS OF VARIATIONS Taking the we get rQ in this equation to be rQm = g - Ep + Collecting terms and solving for rQ m ( 6.68) rQ m • = rQ m • and substituting (6.67) into (6.64), E' ( Q ) Q r E - 1 Qm we finally obtain g - Ep 1 - E'( Q ) Q/( E - 1) which should be contrasted against rQ s = g - - (from ( 6.60) with MC = Ep rMc = 0] Suppose that g Ep < 0. Then, if the denominator in (6.68) is a positive fraction, we will end up with rQ m < rQ s (e.g., dividing - 0.04 by � results in - 0.08, a smaller number). And, in view of Fig. 6.3, the monopolist will in such a case turn out to be an anticonservationist. It can be shown9 that the denominator in (6.68) is a positive fraction when R"(Q) < 0, that is, when dMR/dQ < 0. But a declining marginal revenue is a commonly acknowl­ edged characteristic of a monopoly situation. Therefore, we can see from the present analysis that monopoly is by no means synonymous with conserva­ tionism. EXERCISE 6.3 1 2 In (6.47"), we interpret A to mean the initial value of P(Q) an alternative economic interpretation of A from (6.47'). - C'(Q). Find If the maximization problem (6.45) is subject instead to the weak inequality J;Q dt � 80, what modification(s) must be made to the result of the analysis? 9By differentiating (6.62) with respect to Q, and ( dMR - dQ = ) ( P'(Q) letting the derivative be negative, we have 1 1 - - E ) + P-E'(Q) 1 E2 < 0 Multiplying both sides of this inequality by E2QjP, and making use of the fact that = - 1/E, we can transform the inequality to the form -(E E'(Q)Q < E- 1 - 1) + E'(Q)Q < 0, or P'(Q)QjP In the latter inequality, the left-hand side is positive by virtue of our assumption that E'(Q) > 0; the right-hand side is positive because (6.51) requires MR to be positive, and, in view of (6.62), the positivity of MR in turn dictates that positive fraction, and so is the denominator in (6.68). E > 1. Hence, E'(Q)Qj(E - 1) is a CHAPTER 6: CONSTRAINED PROBLEMS 157 3 If the transversality conditions (5.5) and (5.7) are applied to the Lagrangian integrands .r for social optimum and monopoly, respectively, what terminal conditions will emerge? Are these conditions economically reasonable? 4 Compare rQ• and rQm in (6.60) and (6.64), assuming that (1) MC = rMc 0, (2) the elasticity of demand is output-invariant, and (3) the demand becomes more elastic over time because of the emergence of substitutes. 5 Assume that (1) the demand elasticity is output-invariant and time­ invariant, and (2) the MC is positive and output-invariant at any given time, but increases over time. Write out the appropriate expressions for rq, and rQ m • find the difference rQ m - rq , , study its algebraic sign, and indicate under what conditions the monopolist will be more conservationistic and less conservationistic than the social optimizer. 6 Let q = the cumulative extraction of an exhaustible resource: q(t) = J� Q(t) dt. Verify that the current extraction rate Q is related to q by the equation Q(t) = q'(t). (a) Assume that the cost of extraction C is a function of the cumulative extraction q as well as the current rate of extraction q'. How must the N(Q) expression in (6.44) be modified? (b) Reformulate the social optimizer's problem (6.45), expressing it in terms of the variables q and q'(t). (c) Write the Lagrangian integrand, and find the Euler-Lagrange equation. 7 Using the definition of q in the preceding problem, and retaining the assumption that C = C(q, q'), reformulate the competitive firm's problem (6.48) in terms of the variables q and q'(t). Show that the Euler-Lagrange equation for the competitive firm is consistent with that of the social optimizer. "" � . · ··, . - ·::: ' OPTIMAL CONTROL THEORY CHAPTER 7 OPTIMAL CONTROL: THE MAXIMUM PRINCIPLE The calculus of variations, the classical method for tackling problems of dynamic optimization, like the ordinary calculus, requires for its applicabil­ ity the differentiability of the functions that enter in the problem. More importantly, only interior solutions can be handled. A more modern devel­ opment that can deal with nonclassical features such as corner solutions, is found in optimal control theory. As its name implies, the optimal-control formulation of a dynamic optimization problem focuses upon one or more control variables that serve as the instrument of optimization. Unlike the calculus of variations, therefore, where our goal is to find the optimal time path for a state variable y, optimal control theory has as its foremost aim the determination of the optimal time path for a control variable, u . Of course, once the optimal control path, u*(t), has been found, we can also find the optimal state path, y*(t), that corresponds to it. In fact, the optimal u*(t) and y*(t) paths are usually found in the same process. But the presence of a control variable at center stage does alter the basic orientation of the dynamic optimization problem. A couple of questions immediately suggest themselves. What makes a variable a "control" variable? And how does it fit into a dynamic optimiza­ tion problem? To answer these questions, let us consider a simple economic illustration. Suppose there is in an economy a finite stock of an exhaustible · 161 PART 3: OPTIMAL CONTROL THEORY 162 S (such as coal or oil), as discussed in the Hotelling model, with S(O) = S0. As this resource is being extracted (and used up), the resource resource stock will be reduced according to the relation - dS(t) dt -- = - E( t) where E(t) denotes the rate of extraction of the resource at time t. The E( t ) variable qualifies as a control variable because it possesses the following two properties. First, it is something that is subject to our discretionary choice. Second, our choice of E(t) impinges upon the variable which indicates the state of the resource at every point of time. Consequently, the E(t) variable is like a steering mechanism which we can maneuver so as to " drive" the state variable S(t) to various positions at any time t via the differential equation dSjdt = -E(t). By the judicious steering of such a control variable, we can therefore aim to optimize some performance crite­ rion as expressed by the objective functional. For the present example, we may postulate that society wants to maximize the total utility derived from using the exhaustible resource over a given time period [0, T]. If the terminal stock is not restricted, the dynamic optimization problem may take the following form: S(t) Maximize subject to and JTU( E)e-P1 dt 0 dS dt S(O ) - = = -E(t) S0 S( T ) free ( S0 , T given) In this formulation, it happens that only the control variable E enters into the objective functional. More generally, the objective functional can be expected to depend on the state variable(s) as well as the control variable(s). Similarly, it is fortuitous that in this example the movement of the state variable S depends only on the control variable E. In general, the coprse of movement of a state variable over time may be affected by the state variable(s) as well as the control variable(s), and indeed even by the variable itself. With this background, we now proceed to the discussion of the method of optimal control. t 7.1 THE SIMPLEST PROBLEM OF OPTDKAL CONTROL To keep the introductory framework simple, we first consider a problem with a single state variable y and a single control variable u. As suggested earlier, the control variable is a policy instrument that enables us to 163 CHAPTER 7: OPTIMUM CONTROL: THE MAXIMUM PRINCIPLE u � c a 0 I I I Control path v--1 I I t, . . A y c I I 0 (a) t, T t2 (b) FIGURE 7.1 influence the state variable. Thus any chosen control path u(t) will imply an associated state path y(t). Our task is to choose the optimal admissible control path u*(t) which, along with the associated optimal admissible state path y*(t), will optimize the objective functional over a given time interval [0, T). Special Features of Optimal Control Problems It is a noteworthy feature of optimal control theory that a control path does not have to be continuous in order to become admissible; it only needs to be piecewise continuous. This means that it is allowed to contain jump discon­ tinuities, as illustrated in Fig. 7.1a, although we cannot permit any discon­ tinuities that involve an infinite value of u . A good illustration of piecewise-continuous control in daily life is the on-off switch on the com­ puter or lighting fixture. Whenever we turn the switch on ( u 1) and off (u = 0), the control path experiences a jump. The state path y(t), on the other hand, does have to be continuous throughout the time period [0, T ]. But, as illustrated in Fig. 7.1b, it is permissible to have a finite number of sharp points, or corners. That is to say, to be admissible, a state path only needs to be piecewise differentiable. 1 Note that each sharp point on the state path occurs at the time the control path makes a jump. The reason for this timing coincidence lies in the procedure by which the solution to the problem is obtained. Once we have = 1 Sharp points on a state path can also be accommodated in the calculus of variations via the Weierstrass-Erdmann conditions. We have not discussed the subject in this book, because of the relative infrequency of its application in economics. The interested reader can consult any book on the calculus of variations. 164 PART 3: OPTIMAL CONTROL THEORY found that the segment of the optimal control path for the time interval [0, is, say, the curve in Fig. 7 . 1 a , we then try to determine the t1) ab corresponding segment of the optimal state path. This may turn out to be, say, the curve AB in Fig. whose initial point satisfies the given initial condition. For the next time interval, we again determine the 7.1b, [t1, t2), optimal state-path segment on the basis of the pre-found optimal control, curve cd, but this time we must take point B as the " initial point" of the optimal state-path segment. Therefore, point B serves at once as the terminal point of the first segment, and the initial point of the second segment, of the optimal state path. For this . reason, there can be no discontinuity at point B, although it may very well emerge as a sharp point. Like admissible control paths, admissible state paths must have a finite y value for every t in the time interval [0, T]. Another feature o f importance i s that optimal control theory is capable u , such as the '1& for all t E [0 , T ], where '1& denotes some bounded control set. The control set can in fact be a closed, convex set, such as u(t) E [0 , 1]. The fact that '1& can be a closed set means that corner solutions (boundary solutions) can be admitted, which injects an important nonclassical feature into the framework. When this feature is combined with the possibility of jump discontinuities on the control path, an interest­ ing phenomenon called a bang-bang solution may result. Assuming the control set to be '1& = [0, 1], if for instance the optimal control path turns out to jump as follows: of directly handling a constraint on the control variable restriction u(t) E u*(t) u*(t) u* (t) = = = 1 for 0 for t E ( 0 , t1) t E (t1, t2) for t E [ t2, T ] 1 then we are " banging" against one boundary of the set 9&, and then against the other, in succession; hence, the name " bang-bang." Finally, we point out that the simplest problem in optimal control theory, unlike in the calculus of variations, has a free terminal state (vertical terminal line) rather than a fixed terminal point. The primary reason for this is as follows: In the development of the fundamental first-order condition known as notion of an arbitrary .iu. the maximum principle, we shall invoke the tl.u must, however, imply an Any arbitrary associated .iy. If the problem has a fixed terminal state, we need to pay heed to whether the associated tl.y will lead ultimately to the designated terminal state. Hence, the choice of .iu may not be fully and truly arbitrary. If the problem has a free terminal state (vertical terminal line), on the other hand, then we can let the arbitrary .iu lead to wherever it may without having to worry about the final destination of y. And that simplifies the problem. CHAPTER 7: OPTIMUM CONTROL: THE MAXIMUM PRINCIPLE 165 The Simplest Problem Based on the preceding discussion, we may state the simplest problem of optimal control as (7.1) {0 F( t, y, u ) dt Maximize V= subject to y = f( t, y, u ) y(O) = A y( T ) free ( A, T given) u(t) E � for all t E [0, T ] and Here, as in the subsequent discussion, we shall deal exclusively with the problem. This way, the necessary conditions for optimization can be stated with more specificity and less confusion. When a minimization problem is encountered, we can always reformulate it as a maximization problem by simply attaching a minus sign to the objective functional. For example, minimizing J F(t,y, u ) dt is equivalent to maximizing JJ ­ maximization F(t, y, u) dt. J In (7. 1), the objective functional still takes the form of a definite integral, but the integrand function F no longer contains a y' argument as in the calculus of variations. Instead, there is a new argument u . The presence of the control variable u necessitates a linkage between u and y, to tell us how u will specifically affect the course taken by the state variable y. This information is provided by the equation y = f(t, y, u), where the dotted symbol y, denoting the time derivative dy /dt, is an alternative notation to the y' symbol used heretofore. 2 At the initial time, the first two arguments in the f function must take the given value t = 0 and y(O) = A, so only the third argument is up to us to choose. For some chosen policy at t = 0, say, u 1(0), this equation will yield a specific value for y,. say y 1(0), which entails a specific direction the y variable is to move. A different policy, u (0), will in general give us a different value, yiO), via the f 2 function . And a similar argument should apply to other points of time. What this equation does, therefore is to provide the mechanism whereby our choice of the control u can be translated into a specific pattern of movement of the state variable y. For this reason, this equation is referred to as the equation of motion for the state variable (or the state equation for short). Normally, the linkage between u and y can be adequately described by a first-order differential equation y = f(t, y, u ). However, if it happens that the pattern of change of the state variable y cannot be captured by the first a 2 Even though the y and y' symbols are interchange ble, we shall exclusively use y in the context of optimal control theory, to make a visual distinction from the calculus·of-variations context. PART 3: OPTIMAL CONTROL THEORY 166 derivative y but requires the use of the second derivative y """ d 2y1dt 2, then the state equation will take the form of a second-.order differential equation, which we must transform into a pair of first-order differential equations. The complication is that, in the process, an additional state variable must be introduced into the problem. An example of such a situation can be found in Sec. 8 . 4. We shall consistently use the lowercase letter f as the function symbol in the equation of motion, and reserve the capital-letter F for the integrand function in the objective functional. Both the F and f functions are assumed to be continuous in all their arguments, and possess continuous first-order partial derivatives with respect to t and y, but not necessarily with respect to u . The rest o f problem (7.1) consists o f the specifications regarding the boundaries and control set. While the vertical-terminal-line case is the simplest, other terminal-point specifications can be accommodated, too. As to the control set, the simplest case is for � to be the open set � = ( - oo, + oo). If so, the choice of u will in effect be unconstrained, in which case we can omit the statement u ( t ) E � from the problem altogether. A Special Case As a special case, consider the problem where the choice of u is uncon­ strained, and where the equation of motion takes the particularly simple form y=u Then the optimal control problem becomes (7.2) fTF(t, y, u) dt Maximize V= subject to y=u y(O) = A and 0 y( T ) free F . .• ( A, T given) By substituting the equation of motion into the integrand function, ever, we can eliminate y , and rewrite the problem as ( 7.2' ) fTF(t, y, y) dt Maximize V subject to y(O) = A = 0 y( T ) free how­ " l.' ( A, T given) This is precisely the problem of the calculus of variations with a vertical terminal line. The fundamental link between the calculus of variations and CHAPTER 7: OPTIMUM GONTROL: THE MAXIMUM PRINCIPLE 167 optimal control theory is thus apparent. But the equations of motion encountered in optimal control problems are generally much more compli­ cated than in (7.2). 7.2 THE MAXIMUM PRINCIPLE The most important result in optimal control theory-a first-order neces­ condition-is known as the maximum principle. This term was coined by the Russian mathematician L. S. Pontryagin and his associates. 3 As mentioned earlier in Sec. 1.4, however, the same technique was indepen­ dently discovered by Magnus Hestenes, a mathematician at the University of California, Los Angeles, who later also extended Pontryagin's results. The statement of the maximum principle involves the concepts of the Hamiltonian function and costate variable. We must therefore first explain these concepts. sary The Costate Variable and the Hamiltonian Function Three types of variables were already presented in the problem statement (7.1): t (time), y (state), and u (control). It turns out that in the solution process, yet another type of variable will emerge. It is called the costate variable (or auxiliary variable), to be denoted by A . As we shall see, a costate variable is akin to a Lagrange multiplier and, as such, it is in the nature of a valuation variable, measuring the shadow price of an associated state variable. Like y and u , the variable A can take different values at different points of time. Thus the symbol A is really a short version of A(t). The vehicle through which the costate variable gains entry into the optimal control problem is the Hamiltonian function , or simply the Ham il­ tonian , which figures very prominently in the solution process. Denoted by H, the Hamiltonian is defined as ( 7 .3) H( t , y , u , A) = F( t , y , u) + A( t ) f( t , y, u ) Since H consists of the integrand function F plus the product of the costate variable and the function f, it itself should naturally be a function with four arguments: t, y, u as well as A. Note that, in (7.3), we have assigned a unitary coefficient to F, which is in contrast to the yet undetermined A(t) 3L. S. Pontryagin, V. G. Boltyanskii, R. V. Gamkrelidze, and E. F. Mishchenko, The Mathemal· ical Theory of Optimal Processes, translated from the Russian by K. N. Trirogoff, Interscience, New York, 1962. This book won the 1962 Lenin Prize for Science and Technology. . .., 168 PART 3, OPTIMAL CONTROL THEORY coefficient for f. Strictly speaking, the Hamiltonian should have been written as ( 7 .4) H "' >. 0F( t , y , u ) + A( t) f( t , y , u) where >. 0 is a nonnegative constant, also yet undetermined. For the vertical-terminal-line problem (7.1), it turns out that the constant A0 is always nonzero (strictly positive); thus, it can be normalized to a unit value, thereby reducing (7.4) to (7.3). The fact that A0 * 0 in the simplest problem is due to two stipulations of the maximum principle. First, the multipliers A 0 and >.( t ) cannot vanish simultaneously at any point of time. Second, the solution to the vertical-terminal-line problem must satisfy the transversality condition A(T ) = 0, to be explained in the ensuing discussion. The condition A( T ) = 0 would require a nonzero value for >.0 at t = T. But since A0 is a nonnegative constant, we can conclude that · A 0 is a positive constant, which can then be normalized to unity. For formulations of the optimal control problem other than (7. 1), on the other hand, A 0 may turn out to be zero, thereby invalidating the Hamiltonian in (7.3). The purist would therefore insist on checking in every problem that A 0 is indeed positive, before using the Hamiltonian (7.3). The checking process would involve a demonstration that >.0 = 0 would lead to a contradiction and violate the aforementioned stipulation that A 0 and A(t) cannot vanish simultaneously.4 In reality, however, the eventuality of a zero A 0 occurs only in certain unusual (some say " pathological") situations where the solution of the problem is actually independent of the integrand function F, that is, where the F function does not matter in the solution process.5 This is, of course, why the coefficient A0 should be set equal to zero, so as to drive out the F function from the Hamiltonian. Since most of the problems encountered in economics are those where the F function does matter, the prevalent practice among economists is simply to assume A 0 > 0, theri normalize it to unity and use the Hamiltonian (7.3), even when the problem is not one with a vertical terminal line. This is the practice we shall follow. The Maximum Principle In contrast to the Euler equation which is a single second-order differential equation in the state variable y, the maximum principle involves two 4 For specific examples of the checking process, see Akira Takayama, Mathematical Economics, 2d ed., Cambridge University Press, Cambridge, 1985, pp. 617-618, 674-675, and 679-680. 5An example of such a problem can be found in Morton I. Kamien and Nancy L. Schwartz, Dynamic Optimization: The Calculus of Variations and Optimal Control in Economics and Management, 2d ed. Elsevier, New York, 1991, p. 149. CHAPTER 7: OPTIMUM CONTROL: THE MAXIMUM PRINCIPLE 169 first-order differential equations in the state variable y and the costate variable A. Besides, there is a requirement that the Hamiltonian be maxi­ mized with respect to the control variable u at every point of time. For pedagogical effectiveness, we shall first state and discuss the conditions involved, before providing the rationale for the maximum principle. For the problem in (7.1), and with the Hamiltonian defined in (7.3), the maximum principle conditions are Max H( t,y, u , A ) u (7. 5) A(T) aH ay = [O, T ] [ equation of motion for y ] y= A = E •. " · . aH ()). for all t , ; ·' · [ equation of motion for A] [ transversality condition ] 0 The symbol Max H means that the Hamiltonian is to be maximized u with respect to u alone as the choice variable. An equivalent way of expressing this condition is ( 7 .6) H( t, y , u* , A) ' · '2::. for all t H( t , y , u , A ) E [0, T) where u* is the optimal control, and u i s any other control value. In the following discussion, we shall, for simplicity, sometimes use the shorter notation " Max H " to indicate this requirement without explicitly mention­ . ing u . The reader will note that it is this requirement of maximizing H with respect to u that gives rise to the name " the maximum principle." It might appear on first thought that the requirement in (7.6) could have been more succinctly embodied in the first-order condition iJHjiJu = 0 (properly supported by an appropriate second-order condition). The truth, however, is that the requirement Max H is a much broader statement of u the requirement. In Fig. 7.2, we have drawn three curves, each indicating a possible plot of the Hamiltonian H against the control variable u at a specific point of time, for specific values of y and A. The control region is assumed to be the closed interval [a, c). For curve 1, which is differentiable with respect to u , the maximum of H occurs at u = b, an interior point of the control region �; in this case, the equation iJHjiJu 0 could indeed serve to identify the optimal control at that point of time. But if curve 2 is the relevant curve, then the control in � that maximizes H is u = c, a boundary point of �- Thus the condition iiHjiJu = 0 does not apply, even though the curve is differentiable. And in the case of curve 3, with the Hamiltonian linear in u , the maximum of H occurs at u = a, another boundary point, and the condition aH;au 0 is again inapplicable because = = PART 3: OP'I'IMAL OONTIIOL THBORY 170 H �..;:; I I I I : --�----------- -----4·a 0 b ' I --L- a - c FIGURE 7.2 nowhere is that derivative equal to zero. In short, while the condition may serve the purpose when the Hamiltonian is differentiable with respect to u and yields an interior solution, the fact that the control region may be a closed set, with possible boundary solutions, necessitates the broader statement Max H. In fact, under the maximum principle the u Hamiltonian is not even required to be differentiable with respect to u . The case where the Hamiltonian is linear in u is of special interest. For one thing, it is an especially simple situation to handle when H plots against u as either a positively sloped or a negatively sloped straight line, since the optimal control is then always to be found at a boundary of u . The only task is to determine which boundary. (If H plots against u as a horizontal straight line, then there is no unique optimal control.) More importantly, this case serves to highlight how a thorny situation in the calculus of variations has now become easily manageable in optimal co.ntrol theory. In the calculus of variations, whenever the integrand function is linear in y', resulting in Fy'y' = 0, the Euler equation may not yield a solution satisfying the given boundary conditions. In optimal control theory, in contrast, this case poses no problem at all. Moving on to the other parts of (7 .5), we note that the condition y = oHjiJA is nothing but a restatement of the equation of motion for the state variable originally specified in (7 .1). To reexpress y as a partial derivative of H with respect to the costate variable A is solely for the sake of showing the symmetry between this equation of motion and that for the costate variable. Note, however, that in the latter equation of motion, A is the negatiue of the partial derivative of H with respect to the state variable y. Together, the two equations of motion are referred to collectively as the Hamiltonian system, or the canonical system (meaning the "standard" oHjo u = 0 CHAPTER 7: OPTIMUM CONTROL: THE MAXIMUM PRINCIPLE 171 system of differential equations) for the given problem. Although we have more than one differential equation to deal with in optimal control theory-one for every state variable and every costate variable-each dif­ ferential equation is of the first order only. Since the control variable never appears in the derivative form, there is no differential equation for u in the Hamiltonian system. But from the basic solution of (7.5) one can, if desired, derive a differential equation for the control variable. And, in some models, it may turn out to be more convenient to deal with a dynamic system in the variables (y, u ) in place of the canonical system in the variables (y, A ). The last condition in (7.5) is the transversality condition for the free-terminal-state problem-one with a vertical terminal line. As we would expect, such a condition only concerns what should happen at the terminal time T. Find the curve with·the shortest distance from a given point a given straight line L. We have encountered this problem before in the calculus of variations. To reformulate it as an optimal control problem, let point P be (0, A), and assume, without loss of generality, that the line L is a vertical line. (If the given position of line L is not vertical, it can always be made so by an appropriate rotation of the axes.) The previously used F function, (1 + y' 2 ) I /Z can be rewritten as (1 + u 2 ) I12, provided we let y' = u , or j = u . Also, to convert the distance-minimization problem to one of maximization, we must attach a minus sign to the old integrand. Then our problem is to EXAMPLE 1 P to ( 7 . 7) {T o J Maximize V= subject to j=u and y(O) = A - 1 /2 ( 1 + u 2 ) dt y( T ) free ( A , T given) Note that the control variable is not constrained, so the optimal control will be an interior solution. Step i We begin by writing the Hamiltonian function 1/ H = - ( 1 + u 2 ) 2 + Au ( 7.8) Obsening that H is differentiable and nonlinear, we can apply the first-order condition iJHjiJu = 0 to get aH au ,, ;. . 172 PART 3o OPTIMAL CONTROL THEORY This yields the solution6 (7.9) = u ( t) Further differentiation of / A ( l - A2} - l 2 aH;au using the product rule yields a 2H au -2 = - (1 + u2) - 3;2 <0 Thus the result in (7.9) does maximize H. Since (7.9) expresses of A , however, we must now look for a solution for A. u in terms Step ii T o do that, we resort t o the equation o f motion for the costate variable A = -aH;ay in (7.5). But since (7.8) shows that H is independent of y, we have ( 7 .10 } aH ay - A = -- =0 A ( t) = constant Conveniently, the transversality condition A(T) = 0 in (7.5) is sufficient for definitizing the constant. For if A is a constant, then its value at t T is also its value for all t. Thus, = (7.10') A* ( t ) = 0 t for all E [0, T ] Looking back to (7.9), we can now also conclude that ( 7.11} u*(t) = 0 t for all E [0, T ] Step iii From the equation of motion y = u , we are now able to write ( 7.12) y( t) = constant - y=O Moreover, the initial condition stant and write y(O) = A enables us to definitize this con­ (7.12') y* ( t ) = A 6 ·�·· The aH;au = 0 equation can be written as u(l + u2)- 112 = A Squaring both sides, multiplying through by u2 ( 1 (1 - + .\2 ) u2), and collecting terms, we get = .\2 This result implies that A.2 * 1, for otherwise the equation would become 0 = 1, which is impossible. Dividing both sides by the nonzero quantity (1 - A2), and taking the square root, we finally arrive at (7.9). :."'� . ' 1 73 CHAPTER 7: OPTIMUM CONTROL: THE MAXIMUM PRINCIPLE y y'(t) = A A t-----------... B 0 t T FIGURE 7.3 This y* path, illustrated in Fig. 7.3, is a horizontal straight line. Alterna­ tively, it may be viewed as a path orthogonal to the vertical terminal line. EXAMPLE 2 ( 7. 13) '· Find the optimal control that will = fo2(2y - 3u) dt Maximize V subject to j=y + u y(O) = 4 y(2) free u(t) E � = [0, 2] and Since this problem is characterized by linearity in set, we can expect boundary solutions to occur. Step i u and a closed control The Hamiltonian of (7.13), namely, H = 2y - 3 u + A(Y + u ) = (2 + A)y + ( A - 3)u is linear in u , with slope aH;au = A - 3. If at a given point o f time, we find A > 3, then an upward-sloping curve like curve 1 in Fig. 7.4 will prevail; to maximize H, we have to choose u* = 2. If A < 3, on the other hand, then curve 2 will prevail, and we must choose u* = 0 instead. In short, ( 7 . 14) u* (t) = {�} Both u* = 2 and u* = 0 are, of course, boundary solutions. Note that, because H is linear in u , the usual first-order condition aH;au = 0 is inapplicable in our search for u*. Step i i It is our next task to determine A(t), as needed in (7. 14). From the equation of motion for A, we have the differential equation aH A = - -. = - 2 - A ay or A + A = -2 PART 3: O PTIMAL CONTROL THEORY 174 H Max H Fl� 'f.4 Its general solution is7 A(t) = ( k arbitrary) ke-1 - 2 Since the arbitrary constant k can be definitized to k 2e 2 by using the transversality condition A(T) = A(2) = 0, we can write the definite solution as = ( 7 .15) Note that A*(t) is a decreasing function, falling steadily from an initial value A*(O) = 2 e 2 - 2 "' 12.778 to a terminal value A*(2) 2 - 2 0. Thus, A* exceeds 3 at first, but eventually falls below 3. The critical point of time, when A* = 3 and when the optimal control should be switched from u* 2 to u* = 0, can be found by setting A*(t) 3 in (7.15) and solving for t. Denoting that particular t by the Greek letter T, we have = = ( 7 .1 6) = = T = 2 - ln 2.5 "' 1 .084 Consequently, the optimal control in (7.14) can be restated more specifically in two phases: ( 7 . 17) = Phase I : u*1 Phase II: u*11 = u*[T, 2] u*[O, T) = = 2 0 7 First-order linear differential equations of this type are explained in Sec. 14.1 of Alpha C. Chiang, Fundamental Methods of Mathematical Economics, 3d ed., McGraw-Hill, New York, 1984. . � CHAJ'TER 7: 175 OPTDWJI coNTRoL: THE MAXIIfUM PIUNCIPLE u• Phase I (•J :o 2 Phase II 1-----___,j 0 �--------_.._________��2 1 ' = 1 .083 y• 39.339 (b) 15.739 I I I - - - - - - - - - -�- - - - - - - - I i I I 4.000 0 =-----"--..-.t�--o ---�2.__ t 1 ' = 1.083 As graphically depicted in Fig. simple variety of bang-bang. FIGURE 7.5 7.5a, this optimal control exemplifies a Step iii Even though the problem only asks for the optimal control path, we can also find the optimal state path, in two phases. In phase I, the equation of motion for y is j = y + u = y + 2, or with initial value y(O) = 4 j -y = 2 Its solution is y*1 ( 7.18) = y*[O, r) = 2(3e1 - 1) In phase II, the equation of motion for y is y = y + 0; � y -y = O with general solution ( 7 . 19) y *11 = y*[r, 2] = ce1 ( c arbitrary) Note that the constant c cannot be definitized by the initial condition 4 given in (7.13) because we are already in phase II, beyond t = 0. Nor can it be definitized by any terminal condition because the terminal state is free. However, the reader will recall that the optimal y path is required to be continuous, as illustrated in Fig. 7.1b. Consequently, the y(O) = PART 3, OPTIMAL CONTROL THEORY 176 initial value of y *11 must be set equal to the value of y'"r evaluated at Inasmuch as r. (by ( 7.18)] and y *11( r) = ce• [by (7 .19) ] we find, by equating these two expressions and solving for c, that c e-•), so that the optimal y path in phase II is = 2(3 - ( 7.19' ) The value of y* at the switching time T is approximately 2(3eL096 1) 15.739. By joining the two paths (7.18) and (7.19'), we obtain the complete y* path for the time interval [0, 2], as shown in Fig. 7.5b. In this particular example, the joined path happens to look like one single exponential curve, but the two segments are in fact parts of two separate exponential func­ tions. - = EXERCISE 7.2 1 2 2, A(t) 3 at only A(t) = 3 for In Example is a decreasing function, and attains the value one point of time, T. What would happen if it turned out that all t? Find the optimal paths o f the control, state, and costate variables to Maximize subject to and f0' 3ydt y =y + u y(O) = 5 u(t) E (0, 2] Be sure to check that the Hamiltonian is maximized rather than minimized. 3 Find the optimal paths of the control, state, and costate variables to Maximize subject to and , t(Y 0 - u2) dt y=u y(O) = 0 y(2) free u ( t) unconstrained Make sure that the Hamiltonian is maximized rather than minimized. 177 CHAPTER 7 , OPTIMUM CONTROL: THE MAXIMUM PRINCIPLE 4 Find the optimal paths of the control, state, and costate variables to Maximize il � subject to y = u -y and y(O) - = + u2) dt (y2 1 y( l ) free u ( t) unconstrained Make sure that the Hamiltonian is maximized rather than minimized. [ Hint: The two equations of motion should be solved simultaneously. Review the material on simultaneous differential equations in Alpha C. Chiang, Fundamental Methods of Mathematical Economics, 3d ed., McGraw-Hill, New York, Sec. 1984, 18.2.] 7.3 THE RATIONALE OF THE MAXIMUM PRINCIPLE We shall now explain the rationale underlying the maximum principle. What we plan to do is not to give a detailed proof-the complete proof given by Pontryagin and his associates (Chap. 2 of their book) runs for as many as 40 pages-but to present a variational view of the problem, to make the maximum principle plausible. This will later be reinforced by a comparison of the conditions of the maximum principle with the Euler equation and the other conditions of the calculus of variations. A Variational View of the Control Problem To make things simple, it is assumed here that the control variable u is unconstrained, so that u* is an interior solution. Moreover, the Hamilto­ nian function is assumed to be differentiable with respect to u , and the aH 1a u 0 condition can be invoked in place of the condition " Max H." As u usual, we take the initial point to be fixed, but the terminal point is allowed to vary. This will enable us to derive certain transversality conditions in the process of the discussion. The problem is then to = (7.20) Maximize v= subject to y and y(O) = f0TF( t, y, u ) dt f( t , y, u ) = Yo (given) 178 PART 3, OPTIMAL CONTROL THEORY Step i As the first step in the development of the maximum principle, let us incorporate the equation of motion into the objective functional, and then reexpress the functional in terms of the Hamiltonian. The reader will observe that, if the variable always obeys the equation of motion, then the quantity [ - j] will assuredly take a zero value for all in the interval [0, T ]. Thus, using the notion of Lagrange multipliers, we can form an expression for each value of and still get a zero value. Although there is an infinite number of values of t in the interval [0, T ) summing - y] over in the period [0 , T ) would still yield a total value of zero: y f(t, y, u ) A(t)( f(t, y, u ) - y] A(t)( f(t, y, u) t t t, , jTA (t ) [ f(t, y, u) - yj dt = 0 0 ( 7.2 1) For this reason, we can augment the old objective functional by the integral in (7.21) without affecting the solution. That is, we can work with the new objective functional (7.22) r= V + J0TA(t) [ f(t,y, u ) - y]. dt = t, y, u) + A(t ) [ f(t , y , u ) - y ] } d' t ({F( 0 V, confident that r will have the same value as as long as the eq� of motion in (7.20) is adhered to at all times. Previously, we have defined the Hamiltonian function as H(t, y, u , A ) = F(t, y, u) + A(t ) f(t, y, u ) substitution of the H function into (7.22) can simplify The functional to the form ( 7 .22' ) i/= = the new jT [H(t,y, u , >.) - >.(t) Y ] dt 0 t,y , u , A) dt - (>.( t)ydt (H( 0 0 It is important to distinguish clearly between the second term in the Hamiltonian, on the one hand, and the Lagrange-multiplier expression, )[ ), on the other. The latter explicitly contains whereas the former does not. When the last integral in (7.22') is integrated A(t) f(t,y, u), A(t f(t, y, u ) - y y, 179 CHAPTER 7: OPTIMUM CONTROL: THE MAXIMUM PRINCIPLE by parts,8 we find that - f TA.( t) y dt = - A. ( T ) Yr + .A(O)y0 + f Ty(t)A. dt 0 0 · Hence, through substitution of this result, the new objective functional can be further rewritten as { 7 .22" ) 'Y= H( t, y, u , .A ) + y(t) ,q dt - A.(T)yr fr[ 0 + .A{ O)y0 --------�--� ------- � The r expression comprises three additive component terms, !1 1, fl2, and !13. Note that while the 01 term, an integral, spans the entire planning period [0, T ], the !12 term is exclusively concerned with the terminal time T, and !13 is concerned only with the initial time. Step ii The value of r depends on the time paths chosen for the three variables y, u , and .A, as well as the values chosen for T and Y r · In the present step, we shall focus on .A . The .A variable, being a Lagrange multiplier, differs fundamentally from u and y , for the choice of the .A(t) path will produce no effect on the value of r, so long as the equation of motion y = f( t, y, u ) is strictly adhered to, that is, so long as ( 7 .23) aH a.A y= ­ for all t e [O , T ] So, to relieve us from further worries about the effect of A(t) on r, we simply impose ( 7 .23) as a necessary condition for the maximization of r. This accounts for one of the three conditions of the maximum principle. This, of course, is hardly an earth-shaking step, since the equation of motion is actually given as a part of the problem itself. 8The formula for integrating a definite integral by parts has been given in (2.15). �. we replace the symbol u in (2.15) by w, because u is now used to denote � Control v�. Let u A ( l) w = y(t) = Then, since ( implying that du = A dt) (implying that dw = jdt) A(t)jdt = u dw, we have - tA(t)jdt - - [ A (t)y(t)]� + t y(t)A dt 0 0 wbidl leada to the J"811\ltl in the text. 180 PART 3: 0PTDMAL CONTROL T»EORY Step iii We now turn to the u(t) path and its effect on the ( t ) path. If we have a known u*(t) path, and if we perturb the u*(t) path with a perturb­ ing curve p(t), we can generate " neighboring" control paths y u ( t) = u*(t) + £p(t) ( 7 .24) one for each value o f £ . But, i n accordance with the equation o f motion y = f(t, y, u), there will then occur for each £ a corresponding perturbation in the y* ( t ) path. The neighboring paths can be written as y y(t) = y* ( t ) + £q(t) ( 7.25) T and Yr are variable, we also have T = T* + 6.T and Yr = y/ + AYr dT dyr implying 6.T and d'; = tlyr d£ = Furthermore, if � ( 7.26) � ( ) In view of the u and y expressions in (7.24) and (7.25), we can express Y in terms of £, so that we can again apply the first-order condition dYjd£ = 0. The new version of Y is · ( 7 .27) Y= H [ t,y* + £q( t) , u* + £p(t) , A] + A[y* + £q(t)] } dt r(•){ 0 - A( T) Yr + A{O)y0 Step iv We now apply the condition d'Yjd£ 0. In the differentiation process, the integral term yields, by formula (2.11), the derivative = ( 7 .28) � T<• l {[:� q(t) + :�p(t) 1 + Aq(t) } dt + [ H + A L_ r �: y h� And the derivative of the second term in (7.27) with respect to £ is, from t product rule, ( 7 .29) dy d A( T) dT . - A( T) der - Yr � --;[; = -A( T ) Ayr - YrA( T) 6.T [ by (7.26)] On the other hand, the A(O)y0 term in (7.27) drops out in differentiation. Thus dYjd� is the sum of (7.28) and (7.29). In adding these two expres­ sions, however, we note that one component of (7.28) can be rewritten as follows: [. ] Ay t-T dT d£ . = A( T)yr t:. T [ by ( 7.26)] Thus, when the sum of (7.28) and (7.29) is set equal to zero, the first-order CHAPTER 7 : OPTIMUM CONTROL: THE MAXIMUM PRINCIPLE 181 condition emerges (after rearrangement) as r �� = l [(:: + A ) q ( t) + :�p(t) ] dt + [ H ] t =T LlT - A ( T ) dJr = 0 (7.30) 0 . The three components of this derivative relate to different arbitrary things: The integral contains arbitrary perturbing curves and q(t), whereas the other two involve arbitrary LlT and Llyr, respectively. Conse­ quently, each of the three must individually be set equal to zero in order to satisfy By setting the integral component equal to zero, we can deduce two conditions: p(t) (7.30). iJH iJy aH =0 au and The first gives us the equation of motion for the costate variable A (or the costate equation for short). And the second represents a weaker version of the condition-weaker in the sense that it is predicated on the u assumption that is differentiable with respect to u and there is an interior solution. Since the simplest problem has a fixed T and free Yr, the LlT term in is automatically equal to zero, but is not. In order to make the - A(T) expression vanish, we must impose the restriction "MaxH" H (7.30) ilyr !:!.yr A( T ) = 0 (7.5). This explains the transversality condition in Note that although the A(t) path was earlier, in Step ii, brushed aside as having no effect on the value of the objective functional, it has now made an impressive comeback. We see that, in order for the maximum principle to work, the A(t) path is not to be arbitrarily chosen, but is required to follow a prescribed equation of motion, and it must end with a terminal value of zero if the problem has a free terminal state. 7.4 ALTERNATIVE TERMINAL CONDITIONS What will happen to the maximum principle when the terminal condition specifies something other than a vertical terminal line? The general answer is that the first three conditions in will still hold, but the transversality condition will assume some alternative form. (7.5) Fixed Terminal Point The reason why the problem with a fixed terminal point (with both the terminal state and the terminal time fixed) does not qualify as the " sim- 182 PART 3o OPTIMAL CONTROL THEORY plest" problem in optimal control theory is that the specification of a fixed terminal point entails a complication in the notion of an " arbitrary" perturbing curve p(t) for the control variable u . If the perturbation of the u*(t) path is supposed to generate through the equation of motion j = f(t, y, u ) a corresponding perturbation in the y*(t) path that has to end at a preset terminal state, then the choice of the perturbing curve p(t) is not truly arbitrary. The question then a,rises as to whether we can still legiti­ mately deduce the condition oHjou = 0 from (7.30). Fortunately, the validity of the maximum principle is not affected by this compromise in the arbitrariness of p(t). For simplicity, however, we shall not go into details to demonstrate this point. For our purposes, it suffices to state that, with a fixed terminal point, the transversality is replaced by the condition y ( T ) = Yr ( T , yr given) Horizontal Terminal Line (Fixed­ Endpoint Problem) If the problem has a horizontal terminal line (with a free terminal time but a fixed "endpoint," meaning a fixed terminal state), then Y r is fixed ( Llyr = 0), but T is not (LlT is arbitrary). From the second and third component terms in (7.30), it is easy to see that the transversality condition for this case is ( 7 .31 ) The Hamiltonian function must attain a zero value at the optimal terminal time. But there is no restriction on the value of A. at time T. Terminal Curve In case a terminal curve Yr = cf>(T) governs the selection of the terminal point, then LlT and Llyr are not both arbitrary, but are linked to each other by the relation Llyr = cf>'(T) LlT. Using this to eliminate Llyr, we can combine the last two terms in (7.30) into a single expression involving LlT only: [ H ]1= r tl T - A.( T ) cf>' ( T ) LlT = [ H - A. c/>' ]1 - r tl T It follows that, for an arbitrary LlT, the transversality condition should be ( 7 .32) Truncated Vertical Terminal Line Now consider the problem in which the terminal time T is fixed, but the terminal state is free to vary, only subject to Yr � Ymin> where Ymin denotes a given minimum permissible level of y. CHAPTER 7: OPTIMUM CONTROL: THE MAXIMUM PRINCIPLE 183 Only two types of outcome are possible in the optimal solution: y/ > = Ymin · In the former outcome, the terminal restriction is automatically satisfied. Thus, the transversality condition for the problem with a regular vertical terminal line would apply: Ymin ' or Yr* ( 7 .33) A( T ) = 0 for Y / > Ymin In the other outcome, y/ = Ymin• since the terminal restriction is binding, the admissible neighboring y paths consist only of those that have terminal states Yr � Ymin · If we evaluate (7.25) at t = T and let y/ Ymin• we obtain = Yr = Ymin + Eq( T ) Assuming that q(T) > 0 on the perturbing curve q(t),9 the requirement � 0. But, by the Kuhn-Tucker conditions, the nonnegativity of E would alter the first-order condition d'YjdE = 0 to d'YjdE � 0 for our maximization problem.10 It follows that (7.30) would now yield an inequality transversality condition Yr � Ymin would dictate that E At the same time, we can see from (7.26) that, given E � 0, the requirement of Yr � Y min-which is the same as Yr � y/ in the present context-im­ plies t:.yr � 0. Thus the preceding inequality transversality condition re­ duces to ( 7.34) A(T) � 0 for y/ = Ymin Combining (7.33) and (7.34) and omitting the * symbol, we can finally write a single summary statement of the transversality condition as follows: ( 7 .35) ( Yr - Ymin )A(T) = 0 Note that the last part of this statement represents the familiar complemen­ tary-slackness condition from the Kuhn-Tucker conditions. As in the similar problem with a truncated vertical terminal line in the calculus of variations, the practical application of (7.35) is not as complicated as the condition may appear. We can always try the A(T ) = 0 condition first, and check whether the resulting y/ value satisfies the terminal restriction y/ � Y min· If it does, the problem is solved. If not, we then set Yr* = Ymin in order to satisfy the complementary-slackness condition, and treat the problem as one with a given terminal point. 9 This assumption does not bias the final result of the deductive process here. 10 The Kuhn-Tucker conditions are explained in Alpha C. Chiang, Fundamental Mathematical Economics, 3d ed., McGraw-Hill, New York, 1984, Sec. 21.2. Methods Of ,--- PART 3, 0PTndAL CONTROL THEORY 184 Truncated Horizontal Terminal Line Let the terminal state be fixed, but the terminal time T be allowed to vary subject to the restriction that T * ::s; Tmax• where Tmax is the maximum permissible value of T-a preset deadline. Then we either have T* < Tmax or T* = Tmax in the optimal solution. In the former outcome, the terminal restriction turns out to be nonbinding, and the transversality condition for the problem with a regular horizontal terminal line would still hold: [ H ]t-T = 0 ( 7.36) for T * < Tmax But if T* = Tmax• then by implication all the admissible neighboring y paths must have terminal time T ::s; Tmwc By analogous reasoning to that leading to the result (7.34) for the truncated vertical terminal line, it is possible to establish the transversality condition [ H )t-T � O ( 7 .37) for T * = Tmax By combining (7.36) and (7.37) and omitting the * symbol, we then obtain the following summary statement of the transversality condition: ( 7.38) EXAMPLE 1 Maximize subject to y =y and y( O ) = 1 + u y(l) = 0 With fixed endpoints, we need no transversality condition in this problem. Step i Since the Hamiltonian function is nonlinear: and since H = - u 2 + A (y + u ) u is unconstrained, we can apply the first-order aH - = -2u au This yields the solution ( 7 .39) u = A/2 + ,\ condition =0 or, more accurately, u ( t) = !A( t) Since iPH;au2 = - 2 i s negative, this u(t) solution does maximize rather than minimize H. But since this solution is expressed in terms of A(t), we must find the latter path before u(t) becomes determinate. 185 CHAPTER 7: OPTIMUM CONTROL: THE MAXIMUM PRINCIPLE Step ii From the costate equation of motion aH ay -A we get the general solution A(t) ( 7.40) = ke-1 ( k arbitrary) . , ·· , _ To definitize the arbitrary constant, we try to resort to the boundary conditions, but, unfortunately, for the fixed-terminal-point problem these conditions are linked to the variable y instead of A. Thus, it now becomes necessary first to look into the solution path for Step iii y. The equation of motion for y is y y = (7.40), however, we can rewrite this equation as y = + u . Using (7.39) and + y tke - 1, or This is a first-order linear differential equation with a variable coefficient and a variable term, of the type = w(t)-here with u(t) = - 1 and w(t) tke - 1 • Via a standard formula, its solution can be found as followsY dyjdt + u(t) y = y ( t ) = e- f - l dt ( c + I � ke- le f-ldt dt ) ( 7 . 4 1) •, = e1 ( c + J �ke-1e-1 dt ) = e1( c - �ke-21) 1 ( c arbitrary) -1 ce1 - -ke 4 = u See Alpha C. Chiang, Fundamental Methods of Mathematical Economics, 3d ed., McGraw­ Hill, New York, 1984, Sec. 14.3. In performing the integrations involved in the application of the formula, we have omitted the constants of integration whenever they can be subsumed under other constants. Alternatively, we can find the complementary function and the particu­ lar integral separately and then combine them. With a variable term in the differential equation, we can obtain the particular integral by the method of undetermined coefficients (ibid., Sec. 15.6) 186 PART 3: OPTIMAL CONTROL THEORY Step iv Now the boundary conditions y(O) = 1 and y(1) = 0 are directly applicable, and they give us the following definite values for c and k: c = 1 -1 e 2 Consequently, substituting-these into (7.41), (7.40), and (7.39), we havttthe following definite solutions for the three optimal paths: 1 y* ( t) = 1 - e 2 et - e2 1 - e2 e-t and The search for the u*(t), y*(t), and A*(t) paths in the present problem turns out to be an intertwined process. This is because, unlike the simplest problem of optimal control, where the transversality condition A(T) = 0 may enable us to get a definite solution of the costate path A*(t) at an early stage of the game, the fixed-terminal-point problem does not allow the application of the boundary conditions on y(O) and y(T) until the final stage of the solution process. EXAMPLE condition 2 Let us reconsider the preceding example, with the terminal y(1) = 0 replaced by the restriction T= 1 y(1) ;e: 3 The problem is then one with a truncated vertical terminal line, and the appropriate transversality condition is (7.35). We shall first try to solve this problem as if its vertical terminal line is not truncated. If y*(1) turns out to be ;e: 3, then the problem is solved; otherwise, we shall then redo the problem by setting y(1) = 3. Step i The Hamiltonian remains the same solution for the control variable is still (7.42 ) u ( t) = �.\(t) as in -� 1, .and the [ from (7 .39)] Step ii Although the general solution for A is still (7.43) .\(t) = ke- t [ from (7.40)] we now can use the transversality condition .\(T) = 0 or A(l) - 0 to defini.. . tize the arbitrary constant. The result is k = 0, so that ( 7.43') A*(t) = 0 187 CHAPTER 7, OPTIMUM CONTROL, THE MAXIMUM PRINCIPLB . It then follows from (7.42) that ( 7 .44) u* ( t ) Step iii =0 y, we find [ by (7.44)] From the equation of motion for y =y + u =y The general solution o f this differential equation is y ( t ) = ce1 , where the constant c can be definitized to y(O) = 1. Thus the optimal state path is · (7 .45) c =1 by the initial condition y* (t) = e1 Step iv It remains to check (7.45) against the terminal restriction. At the fixed terminal time T = 1, (7.45) gives us y*(l) = e . This, unfortunately, violates the terminal restriction y(1) ;;:.: 3. Thus, in order to satisfy the transversality condition (7 35), we now have to set y(1) = 3 and resolve the problem as one with a fixed terminal point. Note that had the terminal restriction been T = 1, y(1) ;;:>:: 2, then (7.45) would have been an acceptable solution. . · - ' , EXAMPL:& Jt. V= Maximize ' - -. :'_,_, ·. � ,-,.... fT - 1 dt 0 y =y + u y(O) = 5 y( T ) = 11 u(t) E [ - 1, 1] subject to and T free This example illustrates the problem with a horizontal terminal line. More­ over, it illustrates the type of problem known as a time-optimal problem, , whose objective is to reach some preset target in the minimum amount of time. The time-optimal nature of the problem is conveyed by the objective functional: Clearly, to Step maximize fT - 1 dt = [ t T - 0 this integral is to ]0 = -T minimize T. i To begin with, form the Hamiltonian - ( 7 .46) H = -1 + A(y + u ) 188 PART 3: 0PT� CONTROL THEORY Because this H function is linear in u , the iJHjiJu = 0 condition is inappli­ cable. And, with the control variable confined to the closed interval [ - 1, 1], the optimal value of u at any point of time is expected to be a boundary value, either - 1 or 1. Specifically, if A > 0 (H is an increasing function of u ), then u* 1 (upper bound); but if A < 0, then u* = - 1. As a third possibility, if A = 0 at some value of t, then the Hamiltonian will plot as a horizontal line against u , and u* will become indeterminate at that point of time. This relationship between u* and A �n be succinctly captured in the so-called signum function, denoted by the symbol sgn, and defined as follows: = ( 7.47) y= y = sgn x { �7 } indete inate if x {;} 0 Note that if y is a signum function of x, then y (if determinate) can only take one of two values, and the value of y depends on the sign (not magnitude) of x. Applied to the present problem, this function becomes ( 7 .48) u* = sgn A or Once more, we find that a knowledge of determined. A is necessary before u can be Step ii The equation of motion for the costate variable is, from (7.46), iJH ay -A which integrates to ( 7 .49) A(t) = ke -t ( k arbitrary) In this result, A(t), being exponential, can take only a single algebraic sign-the sign of the constant k. Consequently, barring the eventuality of k = 0 so that A(t) 0 for all t (which eventuality in fact does not occur here), u* must be determinate and adhere to a single sign-nay, a single constant value-in accordance with the signum function. For this reason, even though the linearity of the Hamiltonian in the control variable u results in a boundary solution in the present example, it produces no bang-bang phenomenon. It turns out that the clue to the sign of k lies in the transversality condition [H]1_r 0. Using the H in (7.46), the A in (7.49), and the terminal condition y(T) 11, we can write the transversality condition as = = = - 1 + ke- r ( l l + u*) = 0 189 CHAPTER 7: OPTIMUM CONTROL: THE MAXIMUM PRINCIPLE Since u* is either 1 or - 1, the quantity (11 e - r. Therefore, + u*) must be positive, as is k must be positive in order to satisfy this condition. It then follows that A(t) > 0 for all t, and that u* (t) ( 7.50) = 1 Step iii With u * = 1 for all t, we can expre8s 'the equation of motion of the state variable, y = y + u , as y -y = 1 This fits the format of the first-order linear differential equation with a constant coefficient and a constant term, dyjdt + ay = b-here with a = - 1 and b = 1. Its definite solution gives us the optimal y path12 y*(t) ( 7.51) = = [y(O) - � ]e-at + � e 6 1- 1 = 5) [y(O) Step iv Having obtained the optimal control and state paths u*(t) and y*(t), we next look for A*(t). For this purpose, we first return to the transversality condition [ H lt - r 0 to fix the value of the constant k. In view of (7.50) and (7.51), the transversality condition now reduces to = or Thus k path = 6k = 1 i. Substituting this result in (7.49) then yields the optimal A*( t) ( 7 .52) A = ie- 1 Step v The three optimal paths in (7.50), (7.51), and (7.52) portray the complete solution to the present problem except for the value of T*. To calculate that, recall that the terminal state value is stipulated to be y(T) = 1 1 . This, in conjunction with the y*(t) path obtained earlier, tells us that 11 = 6e r - 1, or eT = 2. Hence, T* = ln 2 ( == 0 .6931) The optimal paths for the various variables are easily graphed. We shall leave this to the reader. 12 This solution formula is derived in Alpha C. Chiang, Fundamental Methods of Mathematical Economics, 3d ed., McGraw-Hill, New York, 1984, Sec. 14. 1. 190 PART 3: OPTIMAL CONTROL THEORY The Constancy of the Hamiltonian in Autonomous Problems All the examples discussed previously share the common feature that the problems are "autonomous;" that is, the functions in the integrand and f in the equation of motion. do not contain t as an explicit argument. An important consequence of this feature is that the optimal Hamiltonian-the Hamiltonian evaluated along the optimal paths of y, u , and A-will have a constant value over time. To see this, let us first examine the time derivative of the Hamiltonian H(t, y, u , A) in the general case: dH - dt oH = - at + oH -.Y ay aH + -u + au oH . -A aA When H is maximized, we have iJHjou 0 (for an interior solution) or u = 0 (for a boundary solution). Thus the third term on the right drops out. Moreover, the maximum principle also stipulates that y = oHjoA and A = -aHjoy. So the second and fourth terms on the right exactly cancel out. The net result is that H*, the Hamiltonian evaluated along the optimal paths of all variables, satisfies the equation = ( 7 .53) dH* aH* dt at This result holds generally, for both autonomous and nonautonomous problems. In the special case of an autonomous problem, since t is absent from the F and f functions as an explicit argument, the Hamiltonian must not contain the t argument either. Consequently, we have oH* jot = 0, so that ( 7 .54) dH* -- dt =0 or H* = constant [ for autonomous problems ] This result is of practical use in an autonomous problem with a horizontal terminal line. The transversality condition [ H ]1 � r 0 is nor­ mally expected to hold at the terminal time only. But if the Hamiltonian is a constant in the optimal solution, then it must be zero for all t, and the transversality condition can be applied at any point of time. In Example 3, for instance, we find that = H* = - 1 + A* ( y* + u* ) = - 1 + �e-1(6 e1 - 1 + 1} = 0 This zero value of H* prevails regardless of the value of t, which shows that the transversality condition is indeed satisfied at all t. 191 CHAPTER 7: OPTIMUM CONTROL: THE MAXIMUM PRINCIPLE EXERCISE 7.4 1 Find the optimal paths of the control, state, and costate variables to Maximize 2 foT - (t2 + u2) dt u subject to y and y(O) = = y( T ) 4 T free Find the optimal paths of the control, state, and costate variables to Maximize subject to fo43y dt y = y+u y ( O) and 3 = 5 = 5 0 :S u(t) y(4) :S � 300 2 We wish to move from the initial point (0, in the ty plane to achieve the terminal state value y(T) = 0 as soon as possible. Formulate and solve the problem, assuming that dy1dt = 2 u , and that the control set is the closed interval [ - 1, 1). 8) 4 Find the optimal control path and the corresponding optimal state path that minimize the distance between the point of origin (0, 0) and a terminal curve y(T ) = 10 - T 2, T > 0. Graph the terminal curve and the y * ( t) path. 5 (7.37) for the Demonstrate the validity of the transversality condition problem with a truncated horizontal terminal line. 7.5 THE CALCULUS OF VARIATIONS AND OPTIMAL CONTROL THEORY COMPARED We have shown earlier in (7.2) and (7.2') that a simple optimal control problem can be translated into an equivalent problem of the calculus of variations. One may wonder whether, in such a problem, the optimality conditions required by the maximum principle are also equivalent to those of the calculus of variations. The answer is that they indeed are. For problem (7.2), the Hamiltonian function is ( 7 .55) H = F( t,y, u ) + Au Assuming this function to he differentiable with � tG u; We ; �y list :::. �"' : . PART 3: OPTIMAL CONTROL THEORY 192 the following conditions by the maximum principle: aH au - =F + A = O (7.56) u aH aA aH ay y= - =u A= A( T ) = 0 The first equation in (7.56) can be rewritten as A - Fu . But in view of the second equation therein, it can be further rewritten as = A = - F:r ( 7 .57) Differentiation of (7.57) with respect to t yields d . A = - - F. dt y However, the third equation in (7.56) gives another expression for equating the two expressions, we end up with the single condition A. By d F - - F. = 0 y dt y which is identical with the Euler equation (2. 18). When the Hamiltonian is maximized with respect to u, the condition aH;au should be accompanied by the second-order necessary condition a 2Hjau 2 � 0. Further differentiation of the aH;au expression in (7.56) yields a 2H F F -au 2 = u u = yy � 0 This, of course, is the Legendre necessary condition. Thus the maximum principle is perfectly consistent with the conditions of the calculus of variations. Now, let us take a look at the transversality conditions. For a control problem with a vertical terminal line, the transversality condition is A(T ) = 0. By (7.57), however, this may be written as [ - F:rlt - T = 0, or, equivalently, { F:rLr = o Again, this is precisely the transversality condition in the calculus of variations presented in (3.10). CHAPTER 7: OPTIMUM CONTROL: THE MAXIMUM PRINCIPLE 193 For the problem with a horizontal terminal line, the optimal-control transversality condition is [ H l1_r = 0 . In view of (7.55), this means [ F + A u lt-T = 0. Using (7.56) again and after substituting y for u , how­ ever, we can transform this condition into [ F - F-:Y ] Y t-T = 0 Except for the slight difference in symbology, this is precisely the transver­ sality condition under the calculus of variations given in (3. 1 1). It can also be shown that the transversality condition for the problem with a terminal curve Yr = cf>(T ) under optimal control theory can be translated into the corresponding condition under the calculus of variations, and vice versa. The details of the demonstration are, however, left to the reader. 7.6 THE POLITICAL BUSINESS CYCLE Applications of the maximum principle to problems of economics mush­ roomed between 1965 and 1975, and the technique has become fairly common. Its applications range from the more standard areas in macro- and microeconomics all the way to such topics as fishery, city planning, and pollution control. In the present section, we introduce an interesting model of William Nordhaus,l3 which shows that, in a democracy, attempts by an incumbent political party to prevent its rival party (or parties) from ousting it from office may encourage the pursuit of economic policies that result in a particular time profile for the rate of unemployment and the rate of inflation within each electoral period. Over successive electoral periods, the repetition of that pattern will then manifest itself as a series of business cycles rooted solely in the play of politics. The Vote Function and the Phillips Tradeoff The incumbent party, in control of the national government, is obliged in a democracy to pursue policies that appeal to a majority of the voters in order to keep on winning at the polls. In the present model, attention is focused on economic policies only, and in fact on only two economic variables: U (the rate of unemployment) and p (the rate of inflation). Since the ill effects of unemployment and inflation seem to have been the primary economic concerns of the electorate, this choice of focus is certainly reasonable. The reaction of the voters to any realized values of U and p is assumed to be 13 William D. Nordhaus, " The Political Business Cycle," Review of Economic Studies, April 1975, pp. 169-190. � .. ,. ' . �:� ' PAJtT s, OPTIMAL CONTRoL THEORY 194 p(%) vote function embodied in an (aggregate) u = u( U, p) ( 7 .58) where u is a measure of the vote-getting power of the incumbent party. The partial derivatives of u with respect to each argument are negative, because high values of U and p are both conducive to vote loss. This fact is reflected in Fig. 7.6, where, out of the three isouote curves illustrated, the highest one is associated with the lowest u. The notion of the isovote curve underscores the fact that, on the political side, there is a tradeoff between the two variables U and p. If the incumbent party displeases the voters by producing higher inflation, it can hope to recoup the vote loss via a sufficient reduction in the rate of unemployment. Aside from the political tradeoff, the two variables under consideration are also linked to each other in an economic tradeoff via the expectations­ augmented Phillips relation ( 7 .59) p = cp ( U ) + ( q/ < 0, 0 arr < a ::S; 1) where rr denotes the expected rate of inflation. Expectations are assumed to be formed adaptively, according to the differential equation ( 7 .60) 7T = b( p - rr ) ( b > 0) All in all, we now have three variables, U, p, and rr. But which of these should be considered as state variables and which as control variables? For a variable to qualify as a state variable, it must come with a given equation of motion in the problem. Since (7.60) constitutes an equation of motion for rr, we can take rr as a state variable. The variable U, on the other hand, does not come with an equation of motion. But since U can affect p via (7.59) and then dynamically drive rr via (7.60), we can use it as a control variable. To use U as a control variable, however, requires the implicit assumption that the government in power does have the ability to implement any target rate of unemployment it chooses at any point of time. As to the remaining · 195 CHAPTER 7: OPTIMUM CONTROL: THE MAXIMUM PRINCIPLE variable, p, (7.59) shows that its value at any time t will become determi­ nate once the values of the state and control variables are known. We may thus view it as neither a state variable nor a control variable, but, like just a function of the other two variables. v, The Optimal Control Problem Suppose that a party has just won the election at t 0, and the next election is to be held T years later at t T. The incumbent party then has a total of T years in which to impress the voters with its accomplishments (or outward appearances thereoD in order to win their votes. At any time in the period [0, T ], the pair of realized values of U and p will determine a Such values of v for different points of time must all specific value of enter into the objective functional of the incumbent party. But the various values may have to be weighted differently, depending on the time of occurrence. If the voters have a short collective memory and are influenced more by the events occurring near election time, then the v values of the later part of the period [0, T ] should be assigned heavier weights than those that come earlier. We may then formulate the optimal control problem of the incumbent party as follows: = = v. Maximize ( 7.61) subject to and j rv ( U, p ) 0 e '1 dt p = ¢( U ) + a1r 7T = b( p - 7r ) ' ,; 1 .: 1r(T ) free A few comments on (7.61) may be in order. First, the weighting system for the values pertaining to different points of time has been given the specific form of the exponential function e '1, where r > 0 denotes the rate of decay of memory. This function shows that the values at later points of time are weighted more heavily. Note that, in contrast to the expression - pt e , what we have here is not a discount factor, but its reverse. Second, we have retained the expectations-augmented Phillips relation in the problem statement. At the moment, however, we are not yet equipped to deal with such a constraint. Fortunately, the variable p can easily be eliminated by substituting that equation into the vote function and the equation of motion. Then the p equation will disappear as a separate constraint. Third, as the boundary conditions indicate, the incumbent party faces a vertical terminal line, since T (election time) is predetermined. Fourth, although the rate of unemployment is perforce nonnegative, no nonnegativity restriction has actually been placed on the control variable U. The plan-and this is an oft-used strategy-is to impose no restriction and just let the solution fall v v 196 PART 3: OPTIMAL CONTROL THEORY out however it may. If U*(t) turns out to have economically acceptable values for all t, then there is no need to worry; if not, and only if not, we shall have to amend the problem formulation. As stated in (7.61), the problem contains general functions, and thus cannot yield a quantitative solution. To solve the problem quantitatively, Nordhaus assumes the following specific function forms: ( 7.62) ( 7.63) v( U, p) = - U2 - hp p = (j - kU) + arr ( h > 0) (j, k > 0, 0 < a � 1) From (7.62), it can be seen that the partial derivatives of v are indeed negative. In (7.63), we find that the Phillips relation l/>(U) has been lin­ earized. Using these specific functions, and after substituting (7.63) into (7 .62), we now have the specific problem: Maximize ( 7.64) subject to and {( - U2 - hj 0 + hkU - harr)e'' dt 1T = b(j - kU - ( 1 - a )rr] rr( T ) free rr(O) = rr0 Maximizing the Hamiltonian .. · The Hamiltonian is ( 7 .65) H = ( - U2 - hj + hkU - ha1T)e'' + Ab [ j - kU - ( 1 - a)rr] Maximizing H with respect to the control variable i U, we have the equation aH ( - 2 U + hk )e'' - ltbk = o au = This implies the control path ( 7 .66) U( t) = �k(h - Abe-'') Since a2HjaU2 = - 2e71 < 0, the control path in (7.66) indeed maximizes the Hamiltonian at every point of time, as the maximum principle requires. The presence of A in the U(t) solution now necessitates a search for the A(t) path. The Optimal Costate Path The search for the costate path begins with the equation of motion aH . ll = - - = hae'1 + ..\b(l - a ) arr 197 CHAPTER 7: OPTIMUM CONTROL: THE MAXIMUM PRINCIPLE When rewritten into the form A - b(l - a ) A = hae rt the equation is readily recognized as a first-order linear differential equation with a constant coefficient but a variable term. Employing the standard methods of solution,14 we can find the complementary function Ac and the particular integral X to be, respectively, ( A arbitrary) A c = Ae b( l - a )t _ A= ha - e rt It follows that the general solution for ( 7 .67) A(t) = r - (B B A is _ = b + ab ) Ac + A = Aeh<l -a)t + ha ert B Note that the two constants A and B are fundamentally different in nature; B is merely a shorthand symbol we have chosen in order to simplify the notation, but A is an arbitrary constant to be definitized. To definitize A, we can make use of the transversality condition for the vertical-terminal-line problem, A( T ) 0. Letting t = T in (7.67), apply­ ing the transversality condition, and solving for A, we find that A ( - hajB)e 8 T. It follows that the definite solution-the optimal costate path -is = = ( 7.67' ) A* ( t ) = ha - [ e rl - e B T + b(l - a )l ] B The Optimal Control Path Now that we have found A*(t), all it takes is to substitute (7.67' ) into (7.66) to derive the optimal control path. The result is, upon simplification, ( 7 . 68) U*( t ) kh = - (( r 2B b ) + bae B(T -1)] It is this control path that the incumbent party should follow in the interest of its reelection in year T. 14See Alpha C. Chiang, Fundamental Methods of Mathematical Economics, 3d ed., McGraw­ Hill, New York, 1984, Sec. 14.1 (for the complementary function) and Sec. 15.6 (for the particular integral). 198 PART 3o OPTIMAL CONTROL THEORY What are the economic implications of this path? First, we note that U* is a decreasing function of t. Specifically, we have dU* dt ( 7 69) -- . = 1 - - khbae 8<T -t) 2 < 0 because k, h, b, and a are all positive, as is the exponential expression. The vote-maximizing economic policy is, accordingly, to set a high unemploy­ ment level immediately upon winning the election at t 0, and then let the rate of unemployment fall steadily throughout the electoral period [0, T ]. In fact, the optimal levels of unemployment at time 0 and time T can be exactly determined. They are = U * ( O) = U* ( T ) = ! ' •, -; ·.}· . . ... ,. /.•:::. ' , , kh - [( r - b) + bae 8T] 2B kh kh [ ( r - b) + ba ] = 2 2B Note that the terminal unemployment level, khj2, is a positive quantity. And since U*(T) represents the lowest point on the entire U*(t) path, the U* values at all values of t in [0, T ] must uniformly be positive. This means that the strategy of deliberately not imposing any restriction on the variable U does not cause any trouble regarding the sign of U in the present case. However, to be economically meaningful, U*(O) must be less than unity or, more realistically, less than some maximum tolerable unemployment rate Umax < 1. Unless the parameter values are such that U*(O) ::::;; Umax• the model needs to be reformulated by including the constraint U(t) E [0, umaxl· The typical optimal unemployment path, U* (t), is illustrated in Fig. 7.7, where we also show the repetition of similar U*(t) patterns over successive electoral periods generates political business cycles. However, the curvature of the U*(t) path does not always have to be concave as in Fig. 7. 7. For, by differentiating (7.69) with respect to t, we can see that d 2 U* dt 2 ( 7.70) -- = 1 0 Bkhbae B<T - t ) :2: < . 2 - 88 ;'' T 2T 3T �: FIGtJIU!: 7.7 CHAPTER 7: OPTIMUM CONTROL : THE MAXIMUM PRINCIPLE r = 0.03, b 0.30, and a = 0.70, then B = r - b + ab 0.06, and the U * (t) path will be concave. But a positive value of B will If, for illustration, = - 199 = turn the curvature the other way. And, with parameter changes in different electoral periods, the position as well as the curvature of the U * (t) paths in succeeding electoral periods may very well change. Nevertheless, the politi­ cal business cycles will tend to persist. The Optimal State Path The politically inspired cyclical tendency in the control variable spill over to the state variable 7T, U must also and hence also to the actual rate of inflation p. The general pattern would be for the optimal rate of inflation to be relatively low at the beginning of each electoral period, but undergo a steady climb. In other words, the time profile of p* tends to be the opposite of that of U * . But we shall not go into the actual derivation of the optimal inflation rate path here. The reader is reminded that the conclusions of the present model-like those of any other model-are intimately tied to the assumptions adopted. In particular, the specific forms chosen for the vote function in (7.62) and the expectations-augmented Phillips relation (7.63) undoubtedly exert an important influence upon the final solutions. Alternative assumptions-such as changing the linear term - hp in (7.62) to hp are likely to modify significantly the U *(t) solution as - 2 - well as the p * ( t ) solution. But the problem formulation is also likely to become much more complicated . EXERCISE 7.6 1 (a) What would happen in the Nordhaus model if the optimal control path were characterized by dU* jdt = 0 for all t? (b) What values of the various parameters would cause dU* jdt to become zero? (c) Interpret economically the parameter values you have indicated in part (b). 2 What parameter values would, aside from causing dU*jdt 0, also cause U *(t) = 0 for all t? Explain the economic implications and rationale for = ,_ . , such an outcome. 3 How does a change in the value of parameter r (the rate of decay of voter memory) affect the slope of the U*(t) path? Discuss the economic implications of your result. [ Note: B = r - b + ab.] 4 Eliminate the e '1 term in the objective function in (7.64) and write out the new problem. (a) Solve the new problem by carrying out the same steps illustrated in the text for the original problem. as those PART 3, OPTIMAL CONTROL THEORY 200 (b) 7.7 Check your results by setting r = 0 in the results of the original model, and (7.69). especially (7.68) ENERGY USE AND ENVIRONMENTAL QUALITY When an economy is faced with an essential resource that is exhaustible, say, fossil fuel, it certainly behooves its citizenry to be concerned about the question of how the limited supply of the resource is best to be allocated for use over time. We have discussed some of the issues involved in Sec. 6.3 with the method of the calculus of variations. But the citizens of the present-day world are also intensely concerned about the quality of the environment in which they live. If the use of the exhaustible fuel generates pollution as a by-product, then what is the optimal time path for energy use? We shall illustrate how such a question can be tackled by optimal control theory with a model of Bruce A. Forster.15 The Social Utility Function Let S(t) denote the stock of the fuel and E(t) the rate of extraction of fuel t. Then we have (and energy use) at any time S ( 7 . 71) = -E Energy use, E , makes possible the production o f goods and services for consumption, C, which creates utility, but also generates a flow of pollution, P, which creates disutility. Instead of writing a simple utility function U( E ) a s we did i n the introductory section o f this chapter, therefore, our new utility function should contain two arguments, C(E) and P ( E ). Forster specifies the consumption function and the pollution function as follows: ( 7 . 72 ) C ( 7 .73) P = = C( E ) ( C' 0, C " < 0) P( E ) ( P' > 0, P" > 0) > While energy use raises consumption at a decreasing rate, it generates pollution at an increasing rate. In this particular model, pollution is as­ sumed for simplicity to be nonaccumulating; that is, it is a flow that 16 Bruce A. Forster, "Optimal Energy Use in a Polluted Environment," Journal of Environ­ mental Economics and Management, 1980, pp. 321-333. While this paper presents three different models, here we shall confine our attention exclusively to the first one, which assumes a single energy source producing a nonaccumulating pollutant. Another model, treating pollution as a stock variable and involving two state variables, will be discussed in Sec. 8.5. CHAPTER 7, OPTIMUM CONTROL, THE MAXIMUM PRINCIPLE 201 dissipates and does not build up into a stock. This is exemplified by the auto-emission type of pollution. The social utility function depends on consumption and pollution, with derivatives as follows: ( 7 . 74) U = U(C, P) ( Uc > 0 , Up < 0, Ucc < 0, Upp < 0 , Ucp = 0) The specification of Uc > 0 and Ucc < 0 shows that the marginal utility of consumption is positive but diminishing. In contrast, the specification of Up < 0 and Upp < 0 reveals that the marginal utility of pollution is negative and diminishing (given a particular increase in P, Up may decrease from, say, - 2 to - 3). In terms of the marginal dis utility of P( = - Up), there­ fore, Upp < 0 signifies increasing marginal disutility. Since both C and P in turn depend on the social utility hinges, in the final analysis, on energy use alone-positively via consumption and negatively via pollution. This means that C and can both be substituted out, leaving E as the prime candidate for the control variable. The only other variable in the model, S , appears in (7.71) in the derivative form. Since it is a variable dynamically driven by the control variable it is clear that S plays the role of a state variable here. E, P E, The Optimal Control Problem If an Energy Board is appointed to plan and chart the optimal time path of the energy-use variable over a specified time period [0, T ], the dynamic optimization problem it must solve may take the form E subject to fu[C(E), P(E)] dt 0 B = -E and S(O) Maximize ( 7.75) = S0 S( T ) ;:::.:. 0 ( S0, T given) This particular formulation allows no discount factor in the integrand, a practice in the Ramsey tradition. And it grants the Energy Board the discretion of selecting the terminal stock S(T), subject only to the nature­ imposed restriction that it be nonnegative. Since the terminal time is fixed, a truncated vertical terminal line characterizes the problem. With a single control variable E and a single state variable S, the problem can be solved with relative ease. Maximizing the Hamiltonian The Hamiltonian function ( 7 .76) H = U(C( E ), P(E)] - A.E 202 PART 3: OPTIMAL CONTROL THEORY involves nonlinear differentiable functions U, C, and P. Thus we may maximize H with respect to the control variable simply by setting its first derivative equal to zero: aH aE (7.77) = Uc C'( E) + UpP'( E) - A = 0 When solved, this equation expresses E in terms of A. To make sure that (7.77) maximizes rather than minimizes the Hamil­ tonian, we check the sign of a 2HjaE 2 . Since Uc and Up are, like U, dependent on E, the second derivative is a 2H -2 = U.cc C' 2 + U.c C" + UPP P'2 aE UP P" < 0 (by (7.72) , (7.73) ; and (7.74)] + Its negative sign guarantees that (7.77) does maximize H. The Optimal Costate and Control Paths To elicit more information about E from (7.77), however, we need to look into the time path of A. The maximum principle tells us that the equation of motion for A is (7.78) aH . A = -- = 0 as implying A(t) = c ( constant) To definitize the constant c, we can resort to the transversality condition. For the problem at hand, with a truncated vertical terminal line, the condition takes the form (7.79) A( T ) ;:::: 0 A( T ) S ( T ) = 0 S ( T ) ;:::: 0 ( by (7.35)] In practical applications of this type of condition, the standard initial step is to set A(T) = 0 (as if the terminal line is not truncated) to see whether the solution will work. Since A(t) is a constant by (7. 78), to set A(T) = 0 is in effect to set A(t) = 0 for all t. With A(t) = 0, (7.77) reduces to an equation in the single variable E, (7.80 ) Uc C'( E) + UpP'( E) = 0 which, in principle, can be solved for the optimal control path. Since this equation is independent of the variable t, its solution is constant over time: (7 .81 ) E* ( t) = E* ( a specific constant) ( if A*(t) = 0] Whether this solution is acceptable from the standpoint of the S(T) � restriction is, however, still a matter to be settled. 0 ' j ' ' l j 203 CHAPTER 7: OPTrMUM CONTROL: THE MAXIMUM PRINCIPLE Meanwhile, it is useful to examine the economic meaning of (7.80). The first term, UcC'( E), measures the effect of a change in E on U via C. That is, it represents the marginal utility of energy use through its contri­ bution to consumption. Similarly, the Up P'( E) term expresses the marginal disutility of energy use through its pollution effect. What (7.80) does is, therefore, to direct the Energy Board to select a E* value that balances the marginal utility and disutility of energy use, much as the familiar MC = MR rule requires a firm to balance the cost and revenue effects of production. The Optimal State Path It remains to check whether the E* solution in (7.81) can satisfy the S ( T ) � 0 restriction. For this purpose, we must find the state path S(t). With constant energy use, the equation of motion S - E can be readily integrated to yield = S(t) = - Et + k ( k arbitrary) Moreover, by setting t 0 in this result, it is easily seen that k represents the initial fuel stock S0• Thus the optimal state path can be written as = ( 7.82) S*(t) = S0 - E*t The value of S*(t) at any time clearly hinges on the magnitude of E*. Since the functions we have been working with-U(C, P), C(E), and P(E ) -are all general functions, E * cannot be assigned a specific numerical value or parametric expression. Nonetheless, we can examine the S(T) � 0 restriction qualitatively. Consider the three illustrative values of E* in Fig. 7.8, where E* 1 < E* 2 < E* 3 • When the low rate of energy use E\ is in effect, the optimal stock S* (t) appears as a gently downward-sloping straight line, such that S * ( T ) is positive. With the higher rate of energy use E* 2, on the other hand, the fuel stock is brought down to zero at time T. Even so, the Energy Board would still be within the bounds of its given authority. But the other case, E * 3 , entailing the premature exhaustion of the fuel endowment, would patently violate the S(T) � 0 stipulation. Thus, if our solution E* in (7.81) s•<tl S*(tl 80 - E"t (E•1 < E•2 < E•3) = 204 PART 3 : OPTIMAL CONTROL THEORY turns out to be like E* 1 or E* 2, then the transversality condition (7. 79) is met, and the problem solved. But if it is like E* 3, then we must set S(T) = 0, and solve the problem as one with a given ter.minal point. In that event, the E* value can be directly found from (7.82) by setting t = T and S(T) = 0: ( 7 .83) S E* = o T [ if ( 7 .81) violates S * ( T ) � 0] This new E* can be illustrated by E* 2 in Fig. 7.8. It is a notable feature of this model that E*, the optimal rate of energy use, is constant over time. This constancy of E * prevails whether the terminal-stock restriction, S(T) � 0, is binding [as in (7.83)] or nonbinding [as in (7.81)]. What assumption in the model is responsible for this particu­ lar result? The answer lies in the absence of a discount factor. If a discount factor is introduced [see Prob. 3 in Exercise 7.7], the E* path then will turn out to be decreasing over time, provided that A*(t) > 0. However, in the other case where A*(t) = 0, E* will still be constant. EXERCISE 7. 7 1 Suppose that the solution of (7.80) turns out to be E*3, which fails to satisfy the S(T) 2: 0 restriction, and consequently the Energy Board is forced to select the lower rate of energy use, E* 2, instead. (a) Does E* 3 satisfy the " marginal utility = marginal disutility" rule? (b) Does E*2 satisfy that rule? If not, is E*2 characterized by " marginal utility < marginal disutility" or " marginal utility > marginal disutility"? Explain. 2 Let the terminal condition in the Forster model be changed from S ( T ) 2: 0 to S(T) 2: Smin > 0. How should Fig. 7.8 be modified to show that E*1 results in S(T) > Smin• E*2 results in S(T) = Smin• and E*3 results in S(T) < Smin? 3 Suppose that the Energy Board decides to incorporate a discount factor e - pt into the objective functional. (a) Write the new Hamiltonian, and find the condition that will maximize the new Hamiltonian. (b) Examine the optimal costate path. Do you still obtain a constant A path as in (7. 78)? (c) If the transversality condition A(T) = 0 applies, what will become of the H-maximizing condition in part (a)? Can the condition be simplified to (7.80)? What can you conclude about E* for this case? (d) If the transversality condition is A(T) > 0 and S(T ) = 0 instead, what will become of the H-maximizing condition in part (a)? Find the derivative dE jdt and deduce the time profile of the E*(t} path for this case. CHAPTER 8 � •· ��'. > .·.; MORE ON OPTIMAL CONTROL For a better grasp of mathematical methods and results, it is always helpful to understand the intuitive economic meanings behind them. For this reason, we shall now seek to buttress our mathematical understanding of the maximum principle by an economic interpretation of the various condi­ tions it imposes. Mter that is accomplished, we shall then move on to some other aspects of optimal control theory such as the current-value Hamilto­ nian, concavity sufficient conditions, and problems with multiple state and control variables. 8.1 ECONOMIC INTERPRETATION OF THE MAXIMUM PRINCIPLE AN In a remarkable article, Robert Dorfman shows that each part of the maximum principle can be assigned an intuitively appealing economic mean­ ing, and that each condition therein can be made plausible from a common­ sense point of view.1 We shall draw heavily from that article in this section. 1 Robert Dorfman, Economic Review, "An Economic Interpretation of Optimal Control Theory," American December 1969, pp. 817-831. 205 206 PART 3, OPTIMAL CONTROL THEORY Consider a firm that seeks to maximize its profits over the time interval [0, There is a single state variable, capital stock And there is a single control variable representing some business decision the firm has T]. K. u, to make at each moment of time (such as its advertising budget or inventory policy). The firm starts out at time 0 with capital but the terminal capital stock is left op�m. At any moment of time, the profit of the firm K0, depends on the amount of capital it currently holds · as well as on the policy u it currently selects. Thus, its profit function is 7T(t, K, u). But the policy selection u also bears upon the rate at which capital K changes over time; that is, K is affected by u. It follows that the optimal control problem is to n = iT7T(t, K, u) dt 0 Maximize (8.1) '' ,_� �. .· · subject to K = f(t, K, u ) K(O) K0 and = The Costate Variable as K( T) free ( K0, T given) a Shadow Price The maximum principle places conditions on three types of variables: u K control, state, and costate. The control variable and the state variable have already been assigned their economic meanings. What about the costate variable >.? As intimated in an earlier section, >. is in the nature of a Lagrange multiplier and, as such, it should have the connotation of a shadow price. To confirm this, let us adapt (7 .22") to the present context, and plug in the optimal paths or values for all variables, to get n* = f[ H( t, K * , u* , A*) + K* (t)A* j dt - A* (T)K*(T) + A* (O)K0 0 n* Partial differentiation of with respect to the (given) initial capital and the (optimal) terminal capital yields (8.2) an * aK0 = A*(O) and an* aK*(T) = - >.*(T) Thus, >.*(0), the optimally determined initial costate value, i s a measure of the sensitivity of the optimal total profit to the given initial capital. If we had one more (infinitesimal) unit of capital initially, ll * would be larger n• by the amount >.*(0). Therefore, the latter expression can be taken as the imputed value or shadow price of a unit of initial capital. In the other partial derivative in (8.2), the terminal value of the optimal costate path, 207 CHAPTER 8: MORE ON OPTIMAL CONTROL A*(T), is seen to be the negative of the rate of change of IT* with respect to the optimal terminal capital stock. If we wished to preserve one more unit (use up one less unit) of capital stock at the end of the planning period, then we would have to sacrifice our total profit by the amount A*(T). So, again, the A* value at time T measures the shadow price of a unit of the terminal capital stock. In general, then, A* (t) for any t is the shadow price of capital at that particular point of time. For this interpretation to be valid, however, we have to write the Lagrange-multiplier expression as A(t)[ f(t, K, u ) - K } rather than A(t)[ K f(t, K, u )]. Otherwise, A* would become instead the negative of the shadow price of K. - The Hamiltonian and the Profit Prospect The Hamiltonian of problem ( 8.1) is ( 8.3) H = 1r(t, K, u ) + A ( t ) f( t, K , u ) The first component o n the right is simply the profit function at time t, based on the current capital and the current policy decision taken at that time. We may think of it as representing the "current profit corresponding to policy u . " In the second component of (8.3), the f(t, K, u ) function indicates the rate of change of (physical) capital, K, corresponding to policy u , but when the f function is multiplied by the shadow price ,.\(t), it is converted to a monetary value. Hence, the second component of the Hamil­ tonian represents the " rate of change of capital value corresponding to policy u . " Unlike the first term, which relates to the current-profit effect of u, the second term can be viewed as the future-profit effect of u, since the objective of capital accumulation is to pave the way for the production of profits for the firm in the future. These two effects are in general competing in nature: If a particular policy decision u is favorable to the current profit, then it will normally involve a sacrifice in the future profit. In sum, then, the Hamiltonian represents the overall profit prospect of the various policy decisions, with both the immediate and the future effects taken into ac­ count.2 The maximum principle requires the maximization of the Hamiltonian with respect to u . What this means is that the firm must try at each point 2 Note that if the Hamiltonian is written out more completely as H A(t)f(t, K, u) as in (7.4), = A01r(t, K, u) + and if it turns out that A0 = 0 , then this would mean economically that the current-profit effect of u somehow does not matter to the firm, and does not have to be weighed against the future-profit effect. Such an outcome is intuitively not appealing. In a meaningful economic model, we would expect normalized to one. A0 to be nonzero, in which case A 0 can be 208 PART 3: OPTIMAL CONTROL THEORY of time to maximize the overall profit prospect by the proper choice of u . Specifically, this would require the balancing of prospective gains in the current profit against prospective losses in the future profit. To see this more clearly, examine the weak version of the " Max H " condition: iJH iJ7T of - = - + A(t) - = 0 iJu ilu au When it is rewritten into the form ·' �- � - . iJ f -A(t) ­ au (8.4) this condition shows that the optimal choice u * must balance any marginal increase in the current profit made possible by the policy [the left-hand-side expression in (8.4)} against the marginal decrease in the future profit that the policy will induce via the change in the capital stock [the right-hand-side expression in (8.4)}. The Equations of Motion The maximum principle involves two equations of motion. The one for the state variable K, included as part of the problem statement (8.1) itself, merely specifies the way the policy decision of the firm will affect the rate of change of capital. The equation of motion for the costate variable is iJ f iJ7T iJH . A = - - = - - - A(t) aK aK iJK or, after multiplying through by (8.5) - 1 , ' . •'/:/ 07T iJ f - A = - + A( t ) aK iJK • The left-hand-side of (8.5) denotes the rate of decrease of the shadow price over time, or the rate of depreciation of the shadow price. The equation of motion requires this rate to be equal in magnitude to the sum of the two terms on the right-hand side of (8.5). The first of these, iJ1r jiJK, represents the marginal contribution of capital to current profit, and the second, A(iJ fjiJK), represents the marginal contribution of capital to the enhance­ ment of capital value. What the maximum principle requires is that the shadow price of capital depreciate at the rate at which capital is contributing to the current and future profits of the firm. 209 CHAPTER 8: MORE ON OPTIMAL CONTROL Transversality Conditions What about transversality conditions? With a free terminal state K(T ) at a fixed terminal time T (vertical terminal line), that condition is A( T ) = 0 This means that the shadow price of capital should be driven to zero at the terminal time. The reason for this is that the valuableness of capital to the firm emanates solely from its potential for producing profits. Given the rigid planning horizon T, the tacit understanding is that only the profits made within the period [0, T] would matter, and that whatever capital stock that still exists at time T, being too late to be put to use, would have no economic value to the firm. Hence, it is only natural that the shadow price of capital at time T should be set equal to zero. In view of this, we would not expect the firm to engage in capital accumulation in earnest toward the end of the planning period. Rather, it should be trying to use up most of its capital by time T. The situation is not unlike that of a pure egotist-a sort of Mr. Scrooge-who places no value on any material possessions that he himself cannot enjoy and must leave behind upon his demise. For a firm that intends to continue its existence beyond the planning period [0, T ], it may be reasonable to stipulate some minimum acceptable level for the terminal capital, say, K min · In that case, of course, we would have a truncated vertical terminal line instead. The transversality condition now stipulates that A( T ) � 0 and [ K * ( T ) - K min ] A ( T ) = 0 [ from ( 7 .35)] If K*(T ) turns out to exceed K min • then the restriction placed upon the terminal capital stock proves to be nonbinding. The outcome is the same as if there is no restriction, and the old condition A(T ) = 0 will still apply. But if the terminal shadow price A(T ) is optimally nonzero (positive), then the restriction K min indeed is binding, in the sense that it is preventing the firm from using up as much of its capital toward the end of the period as it would otherwise do. The amount of terminal capital actually left by the firm will therefore be exactly at the minimum required level, K min · Finally, consider the case of a horizontal terminal line. In that case, the firm has a prespecified terminal capital level, say K ro• but is free to choose the time to reach the target. The transversality condition simply means that, at the (chosen) terminal time, the sum of the current and future profits pertaining to that point of time must be zero. In other words, the firm should not attain K ro at a time when the sum of immediate and future profits (the value of H) is still positive; rather, it should-after 210 PART 3, 0PTUMAL CONTROL THEORY having taken advantage of all possible profit opportunities-attain Kro at a time when the sum of immediate and future profits has been squeezed down to zero. 8.2 THE CURRENT-VALUE HAMILTONIAN ,,; In economic applications of optimal control theory, the integrand function F often contains a discount factor e-pt. Such an F function can in general be expressed as F( t, y, u) = G(t, y, u ) e - p1 (8.6) so that the QPti:mal control problem is to ·, ' Maximize (8.7) subject to :Y = f( t, y, u ) and boundary conditions By the standard definition, the Hamiltonian function takes the form H = G(t,y, u )e-p1 + A {(t,y, u ) (8.8) ·" But since the maximum principle calls for the differentiation of H with respect to u and y, and since the presence of the discount factor adds complexity to the derivatives, it may be desirable to define a new Hamilto­ nian that is free of the discount factor. Such a Hamiltonian is called the current-value Hamiltonian, where the term "current-value" (as against " present-value") serves to convey the "undiscounted" nature of the new Hamiltonian. The concept of the current-value Hamiltonian necessitates the com­ panion concept of the current-value Lagrange multiplier. Let us therefore first define a new (current-value) Lagrange multiplier m : (implying A = me-P1) {8.9) Then the current-value Hamiltonian, denoted by (8.10) He = He P1 = G(t, y, u ) + mf(t,y, u ) As intended, (8.10') He, can be written as [ by (8.8) and {8.9)] He is now free of the discount factor. Note that (8.10) implies 'f 211 CHAPTER 8: MORE ON OPTIMAL CONTROL The Maximum Principle Revised If we choose to work with H. H, then all the conditions of the instead of maximum principle should be reexamined to see whether any revision is needed. The first condition in the maximum principle is to maximize respect to u H with at every point of time. When we switch to the current-value H. = HeP1, the condition is essentially unchanged except for H. for H. This is because the exponential term eP1 is a constant for any given t. The particular that maximizes H will therefore also maximize H Thus the revised condition is simply for all t E [0, T ] (8.11) Hamiltonian the substitution of u •. The equation of motion for the state variable originally appears in the canonical system as j = iJHjiJA. (8.8) and (8.10)], this equation (8.12) Since iJHjiJA = f(t, y, u ) iJH0jiJm = [by should now be revised to aH. y. = am [ equation o f motion for y] T o revise the equation o f motion for the costate variable, ,\ = -aHjoy, we shall first transform each side of this equation into an expression m, involving the new Lagrange multiplier and then equate the two results. For the left-hand side, we have, by differentiating (8.9), H in (8.10'), we can rewrite the right-hand side as aH aH - __c e -pt iJy y iJ Upon equating these two results and canceling the common factor e-pt, we Using the definition of arrive at the following revised equation of motion: ( 8.13) aH. m. = - -- + pm ay [ equation of motion for m] A, Note that, compared with the original equation of motion for the new one for m involves an extra term, It remains to examine the transversality conditions. We shall d o this pm. for the vertical terminal line and the horizontal terminal line only. For the former, we can deduce that (8.14) A{T) = 0 [ me -p1]1-T = 0 m ( T ) e -pT = 0 [ by ( 8.9) } PART 8: OPTIMAL CONTROL THEORY 212 For the latter, similar reasoning shows that [ He e-P1] t - T = 0 (8.15) [ by (8.10')] Autonomous Problems As a special case of problem (8. 7), both the G and f functions may contain no t argument; that is, they may take the forms .. , . G = G(y, u ) and · Then the problem becomes f = f(y, u ) Maximize ( 8.16) subject to :Y = f(y, u ) and boundary conditions , ;.. __ Since the integrand G(y, u)e -P1 still explicitly contains t, the problem is, strictly speaking, nonautonomous. However, by using the current-value Hamiltonian, we can in effect take the discount factor e -pt out of considera­ tion. It is for this reason that economists tend to view problem (8.16) as an autonomous problem-in the special sense that the t argument explicitly enters in the problem via the discount factor only. The current-value Hamiltonian is, of course, as applicable to problem (8.16) as it is to (8.7). And all the revised maximum-principle conditions (8.11) to (8.15) still hold. But the current-value Hamiltonian of the au­ tonomous problem (8.16) has an additional property not available in prob­ lem (8.7). Since He now specializes to the form He = G(y, u ) + mf (y, u ) which is free of the t argument, its value evaluated along the optimal paths 'of all variables must be constant over time. That is, (8.17) dH* c =0 dt __ or He* = constant [autonomous problem] This result is, of course, nothing but a replay of (7.54). Another View of the Eisner-Strotz Model Consider the following autonomous control problem adapted from the Eisner-Strotz model, originally studied as a calculus-of-variations problem CHAPTER s, MORE ON OPTIMAL CONTROL 2 13 with an infinite horizon in Sec. 5.2: Maximize subject to ( 8.18) and for[1r(K)' .:... C( l)]e-P' dt K=I boundary conditions The meanings of the symbols are: 'll" profit, capital stock, = adjustment cost, and net investment. The only state variable is and the only control variable is The and functions have derivatives I= I. C'(I) > 0 1r"(K) < 0 1r and = K= C C"(I) > 0 K, C [see Fig. 5 . 1] From the Hamiltonian function H = [1r (K) - C(I)]e-pt + AI ' ' there emerge the maximum-principle conditions (not including the transversality condition) · ::o··· aH aI = -C'(I)e-p1 + A = 0 aH = I K. = aA aH = -1r'(K)e-pt .A = - aK If we decide to work with the current-value Hamiltonian, however, we have ( 8 . 19) He = 1r(K) - C(I) + mi [ by ( 8 . 1 0) ] with equivalent maximum-principle conditions ( 8 .20) ( 8.21) ( 8 .22 ) aHe = - C'(I) + m = 0 BI [by ( 8 . 1 1 ) ] K. = aHc am = I aHe + pm = -1r'(K) + pm m= aK - - [by ( 8 . 1 2) ] [by ( 8. 13 ) ] The latter version is simpler because it is free of the discount factor e -�'1• From (8.20), we see that m = C'(I) > O dm = C"(l) > 0 di 214 PART 3 , OPTIMAL CONTROL THEORY that is, m is a monotonically increasing function of I. Accordingly, we should be able to write an inverse function 1/1 = C'-1: I = 1/J ( m ) (8.23} ( 1/1' > 0) Substituting (8.23) into (8.21), we can express K as K (8.24} = 1/!( m ) Paired together, (8.24) and (8.22) constitute a system of two simultaneous equations in the variables K and m. After solving these for the K*(t) and m*(t) paths, and definitizing the arbitrary constants by the boundary and transversality conditions, we can find the optimal control path I*(t) via (8.23). EXERCISE 8.2 Find the revised transversality conditions stated in terms Hamiltonian for the following: 1 A problem with terminal curve Yr = tf>(T). or• �� · i ' · ··, · · '·· 2 A problem with a truncated vertical terminal line. S A problem with a truncated horizontal terminal line. 8.3 SUFFICIENT CONDITIONS The maximum principle furnishes a set of necessary conditions for optimal control. In general, these conditions are not sufficient. However, when certain concavity conditions are satisfied, then the conditions stipulated by the maximum principle are sufficient for maximization. We shall present here only two such sufficiency theorems, those of Mangasarian and Arrow. The Mangasarian Sufficiency Theorem A basic sufficiency theorem due to 0. L. Mangasarian3 � optimal control problem Maximize (8.25) subject to and uud for the u) dt V = fTF(t,y, 0 y = f(t,y , u ) y( O) = Yo ( y0 , T given) 30. L. Mangasarian, "Sufficient Conditions for the Optimal Control oC NonliDear �." SIAM JtJUrnal on Control, Vol. 4, February 1966, pp. 139-152. \ 2 15 CHAPTER 8: MORE ON OPTIMAL CONTROL the necessary conditions of the maximum principle are also sufficient for the global maximization of V, if (1) both the F and f functions are differen­ tiable and concave in the variables (y, u) jointly, and (2) in the optimal solution it is true that (8.26) for all A(t) ;::: 0 t E [0, T ] if f is nonlinear in y or in u [If f is linear in y and in u, then A(t) needs no sign restriction.] As a preliminary to demonstrating the validity of this theorem, let us first remind ourselves that with the Hamiltonian H = F( t,y, u ) + A f( t , y, u ) the optimal control path u* (t)-along with the associated paths-must satisfy the maximum principle, so that aH 0U - This implies that Iu* = Fu ( t , y* , u* ) + A*fu ( t , y* , u* ) = y*(t) and A*(t) 0 Fu ( t, y* , u* ) = - A*fu(t,y*, u* ) (8.27) Moreover, from the costate equation of motion, have A = -iJHjily, we should A* = - Fy( t, y* , u* ) - A*fy( t , y*, u* ) which implies that (8.28) Fy( t, y* , u* ) = -A* - A*fy( t, y*, u* ) Finally, assuming for the time being that the problem has a vertical terminal line, the initial condition and the transversality condition should give us (8.29) Yo* = Yo (given ) and A*( T ) = 0 These relations will prove instrumental in the following development. Now let both the F and f functions be concave in (y, u ). Then, for two distinct points (t, y*, u*) and (t, y, u ) in the domain, we have (8.30) (8.30' ) f( t , y, u ) - :$; Fy(t, y* , u* ) ( Y - y* ) + Fu ( t , y* , u*) ( u - u* ) f( t , y * , u* ) 5, fy(t,y*, u*)(y - y * ) + fu ( t , y * , u* ) ( u - u* ) F(t,y, u ) - F(t , y* , u* ) [ cf. (4.4)] PART 3 : OPTIMAL CONTROL THEORY 216 Upon integrating both sides of (8.30) over [0, T ], that inequality becomes (8.3 1) V - V* � JT [ Fy (t,y* , u* ) (y - y* ) 0 = + Fu(t, y*, u* ) ( u - u* )] dt fT[ -A*(y - y*) - A.*{y(t, y* , u*)(y - y* ) 0 (by (8.28) and ( 8.27)] -A.*fu(t, y* , u* ) ( u - u*)j dt The first component of the last integral, relating to the expression y*), can be integrated by parts to yield4 JT0 - A*(y - y*) dt = {0 A.* [ f( t, y, u) - f( t,y* , u* ) ] dt This result enables us, upon substitution into (8.3f ) (8.31), - A* (y . I� . to write V - V* � {A.* [ f(t, y , u) - f(t,y*, u*) - {y (t,y* , u*)(y - y* ) 0 -fu(t, y*, u* ) ( u - u*)] dt �0 The last inequality follows from the assumption of ;:::.: 0 in fact that the bracketed expression in the integrand is Consequently, the final result is )..* (8.26), and the by (8.30'). �0 v � V* (8.3f' ) V* which establishes to be a (global) maximum, as claimed in the theorem. Note that if the function is linear in (y, then (8.30') becomes a strict equality. In that case, the bracketed expression in the integrand in f u), 4Let u = - A* and v = y - y*. Then du = -A* dt and dv = (y - y * ) dt. So, f - A* (y - y* ) dt ( = j0Tvdu ) = ( -A*(y - y* )]� - r - A* (y - y*) .. :· dt = {"-* (:Y - :Y* ) dt [by (8.29)] - {A* [ f( t , y , u ) - f(t , y* , u* )] dt 0 [by (8.25)] 0 CHAPTER 8: MORE ON OPTIMAL CONTROL 217 (8.3Y) is zero, and we can have the desired result V - V * � 0 regardless of the sign of A*, so the restriction in (8.26) can be omitted. The above theorem is based on the F and f functions being concave. If those functions are strictly concave, the weak inequalities in (8.30) and (8.30' ) will become strict inequalities, as will the inequalities in (8.31), (8.3Y), and (8.3f' ). The maximum principle will then be sufficient for a unique global maximum of V. Although the proof of the theorem has proceeded on the assumption of a vertical terminal line, the theorem is also valid for other problems with a fixed T (fixed terminal point or truncated vertical terminal line). To see this, recall that in the proof of this theorem the transversality condition A*(T) 0 in (8.29) is applied in the integration-by-parts process (see the associated footnote) to make the expression - A*(T)(yr - Yr* ) vanish. But the latter expression will also vanish if the problem has a fixed terminal state Yro• for then the said expression will become - A*(T)(Yro - Yr* ), and it must be zero because Yr* has to be equal to Yr o· Moreover, if the problem has a truncated vertical terminal line, then either the transversality condi­ tion A* ( T ) 0 is satisfied (if the truncation point is nonbinding) or we must treat the problem as one with a fixed terminal state at the truncation point. In either event, the said expression will vanish. Thus the Mangasarian theorem is applicable as long as T is fixed. In the application of this theorem, it is possible to combine Mangasar­ ian's conditions (1) and (2) into a single concavity condition on the Hamilto­ nian. If the F and f functions are both concave in (y, u ), and if A is nonnegative, then the Hamiltonian, H = F + A {, being the sum of two concave functions, must be concave in (y, u ), too. Hence, the theorem can be restated in terms of the concavity of H. = = The Arrow Sufficiency Theorem Another sufficiency theorem, due to Kenneth J. Arrow,5 uses a weaker condition than Mangasarian's theorem, and can be considered as a general­ ization of the latter. Here, we shall describe its essence without reproducing the proof. At any point of time, given the values of the state and costate variables y and A, the Hamiltonian function is maximized by a particular u , u*, 6 This theorem appears without proof as Proposition 5 in Kenneth J . Arrow, " Applications of Control Theory to Economic Growth, " in George B. Dantzig and Arthur F. Veinott, Jr., eds., Mathematics of the Decision Sciences, Part 2, American Mathematical Society, Providence, RI, 1968, p. 92. It also appears with proof as Proposition 6 in Kenneth J. Arrow and Mordecai Kurz, Public Inue�tment, The Rate of Return, and Optimal Fiscal Policy, published for Resources for the Future, Inc., by the Johns Hopkins Press, Baltimore, MD, 1970, p. 45. A proof of the theorem is also provided in Morton I. Kamien and Nancy L. Schwartz, "Sufficient Conditions in Optimal Control Theory," Journal of Economic Theory, Vol. 3, 1971, pp. 207-�14. PAltT S: OPTIMAL CONTROL THEORY 218 which depends on (8.32) t, y, and A: u* = u*(t,y, A) When (8.32) is substituted into the Hamiltonian, we obtain what is referred to as the maximized Hamiltonian function H0(t, y, A) = F( t , y , u* ) + A f(t,y , u*) Note that the concept of H0 is different from that of the optimal Hamilto­ nian H* encountered in (7.53) and (7.54). Since H* denotes the Hamilto­ nian evaluated along all the optimal paths, that is, evaluated at y*(t), u*(t), and A*(t) for every point of time, the y, u, and A arguments can all be substituted out, leaving H* as a function of t alone: H* H*(t). In contrast, H0 is evaluated along u*(t) only; thus, while the u argument is substituted out, the other arguments remain, so that H0 = H0(t, y, A) is ( 8.33) = still a function with three arguments. The Arrow theorem states that, in the optimal control problem (8.25), the conditions of the maximum principle are sufficient for the global maxi­ mization of V, if the maximized Hamiltonian function H0 defined in (8.33) is concave in the variable y for all t in the time interval [0, T], for given A . The reason why the Arrow theorem can b e considered as a generaliza­ tion of the Mangasarian theorem-or the latter as a special case of the former-is as follows: If both the F and f functions are concave in (y, u) and A � 0, as stipulated by Mangasarian, then H = F + A f i s also concave in (y, u), and from this it follows that H0 is concave in y, as stipulated by Arrow. But H0 can be concave in y even if F and f are not concave in (y, u), which makes the Arrow condition a weaker requirement. Like the Mangasarian theorem, the validity of the Arrow theorem actually extends to problems with other types of terminal conditions as long as T is fixed. Also, although the theorem has been couched in terms of the regular Hamiltonian H and its " maximized" version H0, it can also be rephrased using the current-value Hamiltonian He and its " maximized" version Hc0, which differ from H and H0, respectively, only by a discount factor e-pt. EXAMPLE 1 problem Maximize subject to and In Sec. 7 .2, Example 1, we discussed the Bh�ce [T 2V = ), - ( 1 + 0 j=u y(O) = A u 2 ) 1/2 y(T ) free Let us apply both sufficiency theorems. dt ( A, T given) [ from ( 7.7)] 219 CHAPTER 8: MORE ON OPTIMAL CONTROL For the Mangasarian theorem, we note that neither the F function nor the f function depends on y , so the concavity condition relates to u alone. From the F function, we obtain -1 and Fu = u ( 1 + U 2 ) /2 Fuu = - ( l + u 2 ) -3/2 < 0 - [by the product rule] Thus F is concave in u . As to the f function, f = u , since it is linear in u , it is automatically concave in u . Besides, the fact that f is linear makes condition (8.26) irrelevant. Consequently, the conditions of Mangasarian are satisfied, and the optimal solution found earlier does maximize V (and minimize the distance) globally. Once the Mangasarian conditions are satisfied, it is no longer neces­ sary to check the Arrow condition. But if we do wish to apply the Arrow theorem, we can proceed to check whether the maximized Hamiltonian H 0 is concave in y. In the present example, the Hamiltonian is H = -(1 + 1 u 2 ) 12 + A u When the optimal control u (t) = 1 A ( 1 - A2 ) - 12 [from (7.9)] is substituted into H to eliminate u , the resulting H 0 expression contains A alone, with no y. Thus H 0 is linear and hence concave in y fpr given A, and it satisfies the Arrow sufficient condition. · EXAMPLE 2 In the problem of Example 2 in Sec. . ··· ··; Maximize .·.· ,,.> · subject to and 7.2: V = J \2y - 3 u ) dt 0 y =y + u y(O) = 4 y(2) free [from (7.13)] u(t) E [0, 2] both the F and f functions are linear in (y, u ). As a result, all the second-order partial derivatives of F and f are zero. With reference to the test for sign semidefiniteness in (4. 12), we have here ID1 1 ID2 1 0, which establishes the negative semidefiniteness of the quadratic form in­ volved. Hence, both F and f are concave in (y, u ). Moreover, as in Example 1, the stipulation in (8.26) is irrelevant. The conditions of the Mangasarian theorem are therefore satisfied. As to the Arrow condition, we recall from (7.14) that the optimal control depends on A, and that u * takes the boundary values 2 and 0 in the = = 220 PART S: OPT04AL CONTROL THEORY two phases of the solution as follows: u*11 and =0 [ from (7 . 17)] Therefore, we should have two maximized Hamiltonian functions, one for each phase. From the Hamiltonian H = 2y - 3u + A(y + u ) we obtain, upon eliminating u , H 01 = 2y - 6 + A ( y + 2 ) = (2 + A )y - 6 + 2A H 0 11 = 2y + Ay = (2 + A )y In either phase, H 0 is linear in y for given A, so the Arrow condition satisfied. iiJ Let us now check whether the maximum principle is suffi­ cient for the control problem adapted from the Eisner-Strotz model (Sec. 8.2, Example 1): EXAMPLE 3 Maximize {[77'(K) - C ( I ) ] e-P1 dt 0 subject to and boundary conditions [ from (8. 18)] 7T"(K) where < 0, C '(I) > 0, and C"(l) > 0. For the Mangasarian conditions, we can immediately see that the f function, f = I, is linear and concave in I. To check the concavity of the F function in the variables 1 ), we need the second-order partial derivatives of F = C(l)}e -P1• These are found to be [7T(K) - (K, Fxx = 1r" ( K ) e-pt < 0 Fxr = FIK = 0 Fu = - C"( I )e-P1 < 0 Thus, with reference to the test for sign definiteness in (4.8) and (4.9), we find here ID11 < 0 and ID2 1 > 0, indicating that F is strictly concave in 1). Since condition (8.26) is again irrelevant, the Mangasarian condi­ tions are fully met. To check the Arrow sufficient condition, we recall from (8.20) and (8.23) that, with the current-value Hamiltonian (K, He = 77'(K) - C( I) + ml [ from (8.19)] m in terms of I, we can, by setting aHjai = 0, solving for and then writing I as an inverse function of m, express the optimal control the in ,. .,,,,;;<·'··; 221 CHAPTER 8: MORE ON OPTIMAL CONTROL form I = 1/J ( m ) [ from ( 8 .23)] Substitution o f this optimal control into He yields the value Hamiltonian maxithize(feurrent- '' Hc0 = rr ( K) - C (l/f( m )] + m l/f( m ) Since aHc0faK = rr'(K) and a2Hc0faK 2 = rr"(K) < 0, the maximized cur­ rent-value Hamiltonian is strictly concave in the state variable K. The Arrow sufficient condition is thus satisfied. The checking of concavity in the present example is slightly more laborious than in the preceding two examples, because even though the model is simple, it involves general functions. For more elaborate general-function models, checking concavity-especially for the maximized Hamiltonian-can be a tedious process.6 Frequently, therefore, models are constructed with the appropriate concavity;convexity properties incorpo­ rated or assumed at the outset, thereby obviating the need to check suffi­ cient conditions. EXERCISE 8.3 Check the Mangasarian and Arrow sufficient conditions for the following: 1 Example 1 in Sec. 7.4. 2 Example 3 in Sec. 7.4. � ' .' ' . 3 Problem 1 in Exercise 7 .4. 4 Problem 3 in Exercise 7.4. 5 The Nordhaus model in Sec. 7.6. 6 The Forster model in Sec. 7.7. 8.4 PROBLEMS WITH SEVERAL STATE AND CONTROL VARIABLES For simplicity, we have so far confined our attention to problems with one state variable and one control variable. The generalization to problems with multiple state variables and control variables is, in principle, very straight­ forward. But the solution procedure naturally becomes more complicated. 6 For more examples of checking the Arrow sufficient condition, see Morton I. Kamien and Nancy L. Schwartz, Dynamic Optimization: The Calculus of Variations and Optimal Control in Economics and Management, 2d ed., Elsevier, New York, 1991, pp. 222-225; Atle Seierstad and Knut Sydsreter, Optimal Control Theory with Economic Applications, Elsevier, New York, 1987, pp. 112-129. 222 PART 3: OPTIMAL CONTROL THEORY The Problem be n state variables, y1 , . . . , y,. , and control variables, u 1, . . . , u m. The two numbers n and are not restricted in relative magnitudes; we can have either n < m , n = m, or n > m. Each state variable yi must come witp an equation of motion, describing how yi specifically depends on t, on the values of Yi and the other state variables at time t, as well on the values of each control variable u chosen at time t. Let there · · m m as Thus, there should be in the problem n i differential equations in the form y There should be n initial conditions on the variables. Assuming the problem to be one with fixed terminal points, we should also have n given terminal conditions on the state variables. In contrast, the control variables are not accompanied by equations of motion, but each control variable may be subject to the restriction of a control region Assuming fixed initial and terminal points for the time being, the optimal control problem can be expressed as ui �i . Maximize subject to (8.34) Y 1( 0) = Yw. , y,. (O) = Y,.o y,(T) = y1T , . . . , y,. (T) = y,. T u 1(t} �1, . . . , um(t) �m · · · and E E This rather lengthy statement of the problem can be shortened some­ what by the use of index subscripts. Using the index j for the state variables and the index i for the control variables, we may restate the problem as Maximize ( 8.34') subject to and V = /,T0 F(t, y1, . . . , y ,., u 1, . . . , u m ) dt i um ) Yi = f ( t, yl , . . . , y,., ul, yj{ O) = Yio yj(T ) = YiT ui( t) E �i ( i = l, . . . , m , j = l, . . , n ) . . . • . 223 CHAPTER 8: MORE ON OPTIMAL CONTROL To have an even more compact statement of the problem, we may employ vector notation. Define a y vector and a u vector Then the integrand function F in (8.34') can be abbreviated to F(t, y, u ) By the same token, the arguments in the fi functions can be simplified to (t, y, u ). If we define two more vectors . then the equations of motion can be collectively written in a single vector equation: :Y = f(t, y, u ). Obvious extensions of the idea to the boundary conditions would enable us to write the vector equations y(O) = y 0 and y(T ) = Y T• where y(O), y0, y(T ), and Y r are all n X 1 vectors. And, finally, we can express the control-region restrictions by the vector statement u(t) E �. where u(t) and � are both m X 1 vectors. In vector notation, therefore, the statement of the optimal control problem is simply Maximize (S.M"} subject to and v= fTF( t, y' u ) dt 0 :Y = f( t, y , u ) y(O) = Y o y(T ) = Yr u(t) E � which, in appearance, is not at all different from the problem with one state variable and one control variable. The only difference is that, in (8.34"), several symbols represent vectors, and therefore involve much more than what is visible on the surface. Note, however, that even in the vector statement of the problem, not all the symbols represent vectors. The symbol V is clearly a scalar, and so are the symbols t and T. The Maximum Principle The extension of the maximum principle to the multivariable case is compa­ rably straightforward. First of all, to form the Hamiltonian, we introduce a 224 PART 3, OPTIMAL CONTROL THEORY Lagrange multiplier for every ( 8.35) {J function in the problem. Thus, we have H = F( t' y' u ) + n L Aj fj ( t' y ' u ) j- 1 where y and u are the state- and costate vectors defined earlier. The summation term on the right can, if desired, also be written in vector notation. Once we define another n X 1 vector with transpose X = [ A1( t) the Hamiltonian can be rewritten as ( 8 .35') H( t , y , u , A) = F( t , y , u ) + X f( t , y , u ) which looks strikingly similar to the Hamiltonian i n a one-variable case, except that, here, the last term is a scalar product-the product of the row vector X and the column vector f(t, y, u ). Note that the Hamiltonian itself is also a scalar. The requirement that the Hamiltonian be maximized at every point of time with respect to the control variables stands as before. We can thus still write the condition Max H (8 .36) u But u is now an m-vector, so we must at each point of time make a choice of m control values, u t, . . . , u m* · As to the equations of motion, those for the state variables always come with the problem; they are just the n differential equations appearing in (8.34). Moreover, these n equations are still expressible in terms of the derivatives of the Hamiltonian: (8 .37) ilH yj = �J (j = l , . . . , n ) Similarly, the equations of motion for the costate variables are simply ( 8.38) iJH . AJ· = - iJy j (j = l, . . . , n ) Together, (8.37) and (8.38)-representing a total of 2n differential equa­ tions-constitute the canonical system of the present problem. The 2n arbitrary constants that are expected to arise from these differential equa­ tions can be definitized by using the 2n boundary conditions. 225 CHAPTER 8: MORE ON OPTIMAL CONTROL If desired, the canonical system can also be translated into vector notation. The equivalent statement for (8.37) is (8.37' ) Y= aH ax [X = transpose of A ] The noteworthy thing about this is that H is differentiated with respect to (a row vector), not A (a column vector). A look at (8.35') would make it clear why this should be the case. More to the point, however, is the fact that (8.37') is in line with the mathematical rule that the derivative of a scalar with respect to a row (column ) vector is a column (row) vector. Accordingly, the vector-notation translation of (8.38) should be X iJH iJy (8 .38') [ X : transpose of A] Transversality Conditions Problem (8.34) assumes fixed endpoints. I f the terminal point i s variable, we again need appropriate transversality conditions. Referring back to (7.30), we see that, for the one-state-variable prob­ lem, the transversality conditions are derived from the last two terms of the dVjd E expression set equal to zero: [H ]1_ r tlT - A(T) tlyr 0. When there are n state variables in the problem, those two terms will expand to an expression with (n + 1) terms: = From this, the following two basic transversality conditions arise: ( 8.40) [ H Jt- T ( 8.41) A1( T ) =0 = [ if T is free ] [ if YJ T is free) 0 Clearly, these conditions are essentially no different from the transver­ sality conditions for the one-state-variable case. The only differences are that (1) the Hamiltonian H in (8.40) for the present n-variable problem contains more terms than its one-variable counterpart, and (2) there will be as many conditions of the A/T ) 0 type in (8.41) as there are state variables with free terminal states. The transversality conditions for the terminalccurve and truncated­ terminal-line problems are also essentially similar to those for the one­ state-variable case. Suppose that the terminal time is free, and two of the state variables, y1 and y2, are required to have their terminal values tied to the terminal time by the relations = ( 8.42 ) Y1r = f/>1( T ) and 226 PART 3: OPTIMAL CONTIIOL 1'HB()BY AT, we may expect the following to hold: Ay1r = cfJ/ ( T ) AT and Using these to eliminate Ay 1r and Ay2r in (8.39), we can rewrite the latter Then, for a small as The transversality condition that emerges for this terminal-curve problem is [ cf. ( 7.32) ] (8 .44) Condition (8.44)-which should replace (8.40)-together with the two equa­ tions in (8.42) provide three relations to determine the three unknowns T, y 1T , and y2r. The other state variables, (y3, . . . , Yn), assumed here to have free terminal values, are still subject to the type of transversality condition in (8.41 ). If the terminal time is fixed, and all the state variables have truncated vertical terminal lines, then the transversality condition is (YJT - YJ,min)..\J( T ) = 0, (8.45) (j = l , . , n ) . . [ cf. ( 7 .35) ] This condition serves as a replacement for (8.41). Finally, for the problem with a maximum permissible terminal time, Tmax• the transversality condition is (8.46) This condition is another type of replacement for (8.40). An Example from Pontryagin While the conceptual extension of the maximum principle to the multivari­ able problem is straightforward, the actual solution process can become much more involved. For this reason, economic applications with more than one state variable are not nearly as common as those with one state variable. A taste of the increasing complexity can be had from a purely mathematical example adapted from the work of Pontryagin et al.,? with two state variables and one control variable. This example actually starts out with a single state variable y, which we wish to move from an initial state y(O) = Yo to the terminal state 7L. S. Pontryagin et a!., York, 1962, pp. 23-27. ·TAe MatherraaliCGl T"-Y of Optimal �. In� N­ 227 CHAPTER 8: MORE ON OPTIMAL CONTROL y(T) = 0 in the shortest possible time. In other words, we have a time-opti­ mal problem. Instead of the usual equation of motion in the form of y = f(t,y, u), however, we have this time a second-order differential equa­ tion, ji = u , where ji = d 2yjdt 2, and where u is assumed to be constrained to the control region [ - 1, 1]. Since the second-order derivative ji is not admissible in the standard format of the optimal control problem, we have to translate this second-order differential equation into an equivalent pair of first-order differential equations. It is this translation process that causes the problem to become one with two state variables. Thus, this example also serves to illustrate how to handle a problem that contains a higher-order differential equation in the state equation of motion. To effect the translation, we introduce a second state variable z: z =j i = ji implying Then the equation ji = u can be rewritten as first-order equations of motion y=z �ne i=u for each state variable. The problem is then to Maximize ' and i = u . And we now have two ( . subject to (8.47) and .'·· -.. lT0 - 1 dt y =z i=u y(O) = Yo (given) z(O) = z0 u(t) E [ - 1, 1] y( T) -= 0 T free The boundary condition on the new variable z is equivalent to the specifi­ cation of an initial value for j. The specification of both y(O) and j(O) is a normal ingredient of a problem involving a second-order differential equa­ tion in y. Step i The Hamiltonian function is . ( 8.48) Being linear in u, this function leads to corner solutions for the given control region, the optimal control should be (8.49) u* = sgn A2 [ cf. ( 7 .48)] u. Jn view of 228 PART a, OPTIMAL CONTROL THEORY Step ii Since u * depends on A2, we must next turn to the equations of motion for the costate variables: aH A1 = - ay = o implying A1( t) = c1 (constant) aH . A2 = - - = - A l ilz = Thus, the time path for A2 is linear: ( 8 . 50 ) ( c 1 , c2 arbitrary) Excepting the special case of c 1 = 0, this path plots as a straight line that intersects the t axis. At that intersection, A2 switches its algebraic sign and, according to (8.49), the optimal control u* must make a concomitant switch from 1 to - 1 or the other way around, thereby yielding a bang-bang solution. However, since the linear function A it) can cross the t axis at most once, there cannot be more than one switch in the value of u* . , · Step iii Next, we examine the state variables. The variable z hae the equation of motion i = u. Inasmuch as u can optimally take only two possible values, u * = ± 1, there are only two possible forms that equation can take: i = 1 and i = - 1. Possibility 1 (8 .51) u * = 1 and i = 1. In this case, the time path for z is z(t) = t + c3 Then, since y = z = t + c3, straight integration yields the following quadratic time path for y: (8 .52) The initial condition y(O) = y0 tells us that c4 = y0• But c3 is not as easy to definitize from the terminal condition y( T ) = 0, because T is not yet known. Glancing at (8.51), however, we note that c3 represents the value of z at t = 0. If we denote z(O) by z0, then c3 = z0• Thus, (8.51) and (8.52) may be definitized to (8.53) (8 .54) z(t) = t + z0 2 y ( t) = �t + z0t + Yo The way these last two equations are written, y and z appear as two separate functions of time. But it is possible to condense them into . one single equation by eliminating t and expressing y in terms of z. This will enable us to depict the movement of y and z in a phase diagram in the yz 229 CHAPTER 8: MORE ON OPTIMAL CONTROL Possibility 1 u* = 1 FIGURE 8.1 . plane. Since 2 t 2 + 2z0 t z 2 ( t + z0) 2 we can solve this for t and rewrite (8.54) as = = + z02 ·, ( 8 .54' ) In the yz plane, this equation plots as a family of parabolas as illustrated in Fig. 8.1. Each parabola is associated with a specific value of k which is based on the values of Yo and z0• In particular, when k 0, we get the parabola that passes through the point of origin. Of course, once y0 and z0 are numerically specified, we can pinpoint not only the relevant parabola, but also the exact starting point on that parabola. Figure 8.1 is a one-variable phase diagram, with z = dy jdt on the vertical axis and y on the horizontal axis. Above the horizontal axis, with y > 0, the directional arrowheads attached to the curves should uniformly point eastward. The opposite is true below the horizontal axis. Note that, according to these arrowheads, the designated terminal state y(T ) = 0 is attainable only if we start at some point on the heavy arc of the parabola that passes through the point of origin. Note also that the terminal value of z is z(T) 0. = = Possibility 2 u* = - 1 and i - 1. Under the second possibility, the equation of motion for z integrates to = ( 8.55) z(t) = -t + c5 = -t + z0 [ cf. ( 8.53)] Since this result also represents the j path, further integration and defini- PAR',I' 8: OPTlMAL CONTROL THEORY 230 y = - 1 - z' Possibility u' = -1 2 2 y .� •. ' .. FIGURE 8.2 tization of the constant gives us (8 .56) y(t) = - it 2 + Zot + Yo [ cf. ( 8 .54)) Combining the last two equations, we obtain the single result which is comparable to (8.54') in nature. Note that the two shorthand symbols k and h are not the same. Like (8.54'), this new condensed equation plots as a family of parabolas in the yz plane. But because of the negative coefficient of the z2 term, the curves should all bend the other way, as shown in Fig. 8.2. Each parabola is uniquely associated with a value of h; for example, the one passing through the origin corresponds to h = 0. As before, the directional arrowheads uniformly point eastward above the horizontal axis, but westward below. Here, again, only the points located on the heavy arc can lead to the designated terminal state y(T ) = 0. The terminal value of z is, as may be expected, still z(T ) = 0. Although we have analyzed the two cases of u * = 1 and u * = - 1 as separate possibilities, both possibilities will, in a bang-bang solution, become realities, one sequentially after the other. It remains to establish the link between the two and to find out how and when the switch in the optimal control is carried out, if needed. Step iv This link can best be developed diagrammatically. Since the heavy arcs in Figs. 8.1 and 8.2 are the only routes that can lead ultimately to the desired terminal state, let us place them together in Fig. 8.3. Even though the two arcs appear to have been merged into a single curve, they should be CHAPTER 8: MORE ON OPTIMAL CONTROL 231 z (ey) I (u• = + 1) FIGURE 8.3 kept distinct in our thinking; the arrowheads go in opposite directions on the two arcs, and they call for different optimal controls-u* = + 1 for arc AO and u * = - 1 for arc B O. Should the actual initial point be located anywhere on arc AO or arc B 0, the selection of the proper control would start us off on a direct journey toward the origin. No switch in the control is needed. For an initial point located off the two heavy arcs, however, the journey to the origin would consist of two legs. The first leg is to take us from the initial point to arc AO or B 0, as the case may be, and the second leg is to take us to the origin along the latter arc. For this reason, there is no need to reproduce in Fig. 8.3 every part of every parabola in Figs. 8.1 and 8.2. All we need are the portions of the parabolas flowing toward and culminating in arc AO and arc B O, respectively. If, for instance, point C is the initial point, then the first leg is to follow a parabolic trajectory to point D. Since the parabola involved is taken from Fig. 8.2 under possibility 2, the control appropriate to the circumstance is u * - 1. Once point D is reached, however, we should detour onto arc AO. Since the latter comes from Fig. 8.1 under possibility 1, the control called for is u* = 1 . By observing these u* values, and only by so doing, we are able to proceed on course along the optimal path CDO. Other initial points off the heavy arcs (such as point E) can be similarly analyzed. For each such initial point, there is a unique optimal path leading to the designated terminal state, and that path must involve one (and only one) switch in the optimal control. Thus, in such cases, the optimal movement of y over time is never monotonic; an initial rise must be followed by a subsequent fall, and vice versa. The optimal movement of the control variable over time, on the other hand, traces out a step function composed of two linear segments. = �'.· . 232 PART 3: OPTIMAL CONTROL THEORY More about the Switching Having demonstrated the general pattern of the bang-bang solution, Pon­ tryagin et al. dropped the example at this point. However, we can proceed a step further with the discussion of the CD O path, to find the coordinates of the switching point D and..also to determine the exact time, r, at which the switch in the control should be made. Point D is the intersection of two curves. One of these, arc AO, being part of a parabola taken from Fig. 8.1, falls under possibility described by (8.54') with k = 0. Specifically, we have 1, and is 2, and is ( z < 0) ( 8 .57) 8.2 The other is a parabola taken from Fig. described instead by (8.56'). is positive. Hence, we have under possibility Since the intercept on the y axis is positive, · (h > ( 8 .58) At point D, both (8.57) and (8.58) are satisfied. simultaneously yields ble). Thus, y= 4h and Point (8 .59) And the switch in - /h . u* z= D = - 0) Solving these two equations fh (the positive root is inadmissi­ (�h , - /h ) should be made precisely when The calculation of the switching time r From the initial point C, the movement of = - t + z0. At time r, z y= th and z( = y) = can be undertaken from the knowledge that, at point D, and hence at time r, (8.55): z(t) h z attains the value - /h. follows the pattern given in its value should be z( r ) = - r + z0 Setting z( T) equal to - lh , (8 .60) we then find that T = z0 + /Ji This result shows that the larger the values of z0 and h, the longer it would take to reach switching time. A look at Fig. 8.3 will confirm the reasonable­ ness of this conclusion: A larger z0 and/or a larger would mean an initial point farther away from arc AO, and hence a longer journey to reach arc h AO, where the switching point is to be found. It should be realized, of course, that (8.60) is appropriate only for switching points located on arc AO, because the assumptions of a negative z and a positive h used in its derivation are applicable only to arc AO. For switching points located on arc B O, the expression for the same. T is different, although the procedure for finding it is 233 CHAPTER 8: MORE ON OPTIMAL CONTROL It is also not difficult to calculate the amount of time required to travel from point D to the final destination at the origin. On that second leg of the journey, the value of z goes from z( T) = - /h to zero. The relevant equation for that movement is (8.53), where, for the present purpose, we should consider T as the initial time and z( -r) = - lh as the initial value z0• In other words, the equation should be modified to� z(t) = t - /h. At the terminal time-which we shall denote for the time being as T'-the left-hand-side e_!Pression becomes z(T') 0, and the right-hand-side one becomes T' - -/h . Thus, we obtain = T' = {h ( 8 .61) The reason we use the symbol T ' instead of T is that the symbol T is supposed to represent the total time of travel from the initial point C. to the origin, whereas the quantity shown in (8.61) tells only the travel time from point D to the origin. To get the optimal value of T, (8.60) and (8.61) should be added together. That is, T* ( 8 .62) = -r + T' = z0 + 2/h The preceding discussion covers most aspects of the time-optimal problem under consideration. About the only thing not yet resolved is the matter of the arbitrary constants c1 and c2 in (8.50). The purpose of finding the values of c1 and c2 is only to enable us to definitize the A 2( t ) path, so as to know when A 2 will change sign and thereby trigger the switch in the optimal control. Since we have already found -r in (8.60) via another route, there is little point in worrying about c 1 and c2• For the interested reader, however, we mention here that c1 and c2 can be definitized by means of the following two relations: First, when t = T, A2 must be zero (at the juncture of a sign change); hence, from (8.50), we have - c1T + c2 0. Second, the transversality condition = ( H ]t=T = [ - 1 + A 1 z + A 2u ]1_T = 0 reduces to - 1 + A 2( T ) 0, since z(T) 0 and u(T) = 1. Substituting (8.50) into this simplified transversality condition yields another equation relating c1 to c2• Together, these two relations enable us to definitize c 1 and c2• = = EXERCISE 8.4 1 On the basis of the optimal path CDO in Fig. 8.3, plot the time path u*(t ). 2 On the basis of the optimal path CDO in Fig. 8.3, describe the way y* changes its values over time, bearing in mind that the value of z represents the rate of change of y. From your description, plot the time path y*(t). PART s, OPTIMAL CONTROL THEORY 234 3 For each of the following pairs of Yo and z0 values, indicate whether the resulting optimal path in Fig. 8.3 would contain a switch in the control, and explain why: (a) (y0 , z0) = (2, 1) (b) (y0, z0) = (2, 2) (c) (y0, z0) = ( - i , l) 4 Let point E in Fig. 8.3 be the initial point. (a) What is the optimal path in the figure? (b) Describe the time path u * (t ). (c) Describe the time path y*(t), taking into account the fact that z "' j. Plot the y*(t ) path. 5 Let point E in Fig. 8.3 be the initial point. (a) Find the coordinates of the switching point on arc B O. (b) Find the switching time T. (c) Find the optimal total time for traveling from point E to the destination at the origin. - _ 8.5 ANTIPOLLUTION POLICY In the Forster model on energy use and environmental quality in Sec. ·7 .7, pollution is taken to be a flow variable.8 This is exemplified by auto emission, which, while currently detrimental to the environment, dissipates quickly and does not accumulate into a long-lasting stock. But in other types of pollution, such as radioactive waste and oil spills, the pollutants do emerge as a stock and produce lasting effects. A formulation with pollution as a stock variable is considered by Forster in the same paper cited previously. This formulation contains two state variables and two control variables. The Pollution Stock As before, we use the symbol E to represent the extraction of fuel and energy use. But the symbol P will now denote the stock (rather· than flow) of pollution, with P as its flow. The use of energy generates a flow of pollution. If the amount of pollution flow is directly proportional to the amount of energy used, then we can write P = a E, (a > 0). Let A stand for the level of antipollution activities, and assume that A can reduce the pollution stock in a proportionate manner. Then, from this consideration, we have P = J3A, ((3 > 0). Further, if the pollution stock is subject to - 8Bruce A. Forster, " Optimal Energy Use in a Polluted ED"rironment," Journal ofEnvii'"01IIIWIItal Economics and Management, 1980, pp. 321-333. ' ,, �� . .-. ·. CHAPTER s, MORE ON OPTIMAL CONTROL 235 exponential decay at the rate > 0, then we have P = Combining these factors that affect we can write P S -8P. (8.63) P = P, aE - {3A - SP ( a , f3 > 0, 0 jP = -S, so that < S < 1) The Stock of Energy Resource A in itself requires the use of A causes a reduction in S, the stock of the energy resource. Assuming a proportional relationship between A and S, we can, by the proper choice of unit A, simply write S = -A. But since S is also reduced by the use of energy in other economic activities, we should also have S = -E [same as (7.71)]. Combining these considerations, we have The implementation of antipollution activities energy. That is, S ( 8 .64) = -A - E The Control Problem (8.63) and (8.64) delineate the dynamic changes of P S, respectively, these equations can evidently serve the equations of motion in this model. This then points to P (pollution stock) and S (fuel stock) as the state variables. A further look at (8.63) and (8.64) also reveals that E (energy use) and A (antipollution activities) should play the role of Since the relations in as and control variables in the present analysis. If we adhere to the same utility function used before: U = U[C(E) , P] (Uc > 0, Up < 0, Ucc < 0 , Upp < 0 , C' > 0 , C" < 0) [ from ( 7.74) and (7.72)] then the dynamic optimization problem may be stated as �·::� Maximize subject to (&.�}_ �- · l ·- and fTU[C( E), P) dt 0 P = aE - {3A - 8P S = -A - E P(O) = P0 > 0 P(T) � 0 free S(O) = 80 > 0 S(T) � 0 free E � O O�A�A (T given) Two aspects of this problem are comment·worthy. First, while the P T, initial values of and S are fixed, as is the terminal time the terminal and the energy resource stock S are values of both the pollution stock P 236 PART 3: OPTIMAL CONTROL THEORY left free, subject only to nonnegativity. This means that there is a truncated vertical terminal line for and another one for S. Second, both control variables and are confined to their respective control regions. For the control region is [0, oo). And for A, the control region is [0, where A denotes the maximum feasible level of antipollution activities. Since budget E A P, E, A], considerations and other factors are likely to preclude unlimited pursuit of environmental purification, the assumption of an upper bound seem unreasonable. A does not Maximizing the Hamiltonian As usual, we start the solution process by writing the Hamiltonian function: (8.66) H = U[C(E), P] + Ap(aE - {3 A - oP) - A s( A + E ) A where the subscript of each costate variable indicates the state variable associated with it. To maximize H with respect to the control variable E, E � 0, the Kuhn-Tucker condition is aH;aE � 0, with the comple­ mentary-slackness proviso that E(oHjoE) = 0. But inasmuch as we can rule out the extreme case of E = 0 (which would totally halt consumption production), we must postulate E > 0. It then follows from complementary where slackness that we must satisfy the condition aH = Uc C'( E ) + aAp - As = 0 aE Note that o 2HjoE 2 = UccC' 2 + UcC" < 0; so H indeed is maximized rather ( 8.67 ) than minimized. In addition, H should be maximized with respect to A . As shows, H is linear in the variable with A, (8.66) aH = -{3Ap - As oA A]. � Besides, E is restricted to the closed control set (0, Thus to maximize H, the left-hand-side boundary solution A* = 0 should be chosen if aH;aA is negative, and the right-hand-side boundary oHjoA is positive. That is, - (8.68) From (8.67), however, we see that A* = A* = A should be selected if {�} 237 CHAPTER 8: MORE ON OPTIMAL CONTROL Substitution of this into (8.68) results in the alternative condition A* = { 1} ( 8 .68' ) The optimal choice of A thus depends critically on A P· The Policy Choices Categorically, the optimal choice of antipollution activities A is either an interior solution or a boundary solution. Forster shows that it is not possible to have an interior solution in the present model. To see this, consider the equations of motion of the costate variables: ( 8.69) (8 .70) If v+• aH . Ap = = - Up + 8Ap iJP As = aH - - as = = 0 As = constant A* is an interior solution, then {3Ap + As = 0 [ by ( 8 .68)] Since As is a constant by (8.70), this last equatio n shows that Ap must also be a constant, which in turn implies that [by (8.69)] (8.71) But the constancy of A p requires Up to be constant, too. Since U is monotonic in P, there can only be one value of P that would make Up take any particular constant value. Thus P, too, must be a constant if is an interior solution. Given the initial pollution stock P0 > 0, to have a constant P is to fix the terminal pollution stock at P(T) = P0 > 0. For a problem with a truncated terminal line, the transversality condition includes the stipulation that A* ( 8 . 72) P ( T ) Ap( T ) = 0 With a positive P(T ), it is required that Ap(T) = 0, which, since Ap is a constantby (8. 71), means that for all t E {0 , T ] Ap ( t ) = 0 But the zero value for Ap would imply, by (8.68'), that UcC'(E) = 0, which contradicts the assumptions that Uc and C' are both positive. Conse­ quently, an interior solution for must be ruled out in the present model. A* PART 3: OPTIMAL CONTROL 'ftB!QilY 238 Boundary Solutions for Control Variable A The only viable policies are therefore the boundary solutions A* = 0 (jO attempt to fight pollution) or A* = A (abatement of pollution at the ma:IJ. mal feasible rate). These two policies share the common feature that [from ( 8 .67)] ( 8.73) Interpreted economically, the effect of energy use on utility through con­ sumption, UcC'(E), should under both policies be equated to the shadow value of the depletion of the energy resource, measured by A8, adjusted for the shadow value of pollution via the -aAp term. But the two policies differ in that ( 8.74) A* = {1} is associated with >.. 8 { � } - f3>.. p [by (8.68)] The economic meaning of the first line in (8. 74) is that a hands-off policy on pollution is suited to the situation where the shadow price of energy resource, As, exceeds that of pollution abatement, measured by -f3Ap. In this situation, it is not worthwhile to expend the resource on antipollution activities because the resource cost is greater than the benefit. But, as shown in the second line, fighting pollution at the maximal rate is justified in the opposite situation. In distinguishing between these two situations, the parameter /3, which measures the efficacy of antipollution activities, plays an important role. Let us look further into the implications of the A* = 0 case. Since the pollution stock is left alone in this case, the terminal stock of pollu­ tion is certainly positive. With P(T) > 0, the transversality stipulation P(T)Ap(T) = 0 in (8.72) mandates that Ap(T) = 0. This suggests that Ap, the shadow price of pollution, which is initially negative, should increase over time to a terminal value of zero to satisfy the complementary-slackness condition. That is, Ap > o Now if we differentiate (8. 73) totally with respect to t, we find (UccC' 2 + UcC")E = -aAp Since the expression in parentheses is negative, E sign. So we can conclude that and A p have the same The increasing use of energy over time will, in the present case, lead to the exhaustion of the energy resource. As (8.70) shows, As is a constant-a 239 CHAPTER 8: MORE ON OPTIMAL CONTROL positive constant because As denotes the shadow price of a valuable re­ source. The positivity of the As constant means that, in order to satisfy the transversality-condition stipulation S ( T ) As ( T ) = 0 [cf. ( 8.72)] S(T) must be zero, signifying the exhaustion of the energy resource stock at the terminal time T. Turning to the other policy, we have pollution abatement at the maximal feasible rate, A* = A. Despite the maximal effort, however, we do not expect the antipollution activities to run the pollution stock P all the way down to zero. Upon the assumptions that Up < 0 and Upp < 0, we know that, as P steadily diminishes as a result of antipollution activities, Up will steadily rise from a negative level toward zero. When the P stock becomes sufficiently small, Up will reach a tolerable level where we can no longer justify the resource cost of further pollution-abatement effort. There­ fore, we may expect the terminal pollution stock P(T ) to be positive. If so, then Ap(T) 0 by the complementary-slackness condition, and the rest of the story is qualitatively the same as the case of A* = 0. = EXERCISE 8.5 1 2 ;' ,f . - • S What will happen to the optimal solution if the control region in the Forster model for control variable A in (8.65) is changed to A ;;::; 0? Let a discount factor e -P•, (p > 0), be incorporated into problem (8.65). (a) Write the current-value Hamiltonian He for the new problem. (b) What are the conditions for maximizing He? (c) Rewrite these conditions in terms of Ap and A s (instead of m p and m8), and compare them with (8.67) and (8.68) to check whether the new conditions are equivalent to the old. In the new problem with the discount factor in Prob. 2, check whether it is possible to have an interior solution for the control variable A. In answering this question, use the regular (present-value) Hamiltonian H, and assume that there exists a tolerably low level of P below which the marginal resource cost of further pollution abatement effort does not compensate for the marginal environmental betterment. ' ;� ·. •• :� ;;.! CHAPTER 9 INFINITEHORIZON PROBLEMS In our discussion of problems of the calculus of variations, it was pointed out that the extension of the planning horizon to infinity entails certain mathematical complications. The same can be said about problems of optimal control. One main issue is the convergence of the objective func­ tional, which, in the infinite-horizon context, is an improper integral. Since we have discussed this problem in Sec. 5.1, there is no need to elaborate on the topic here. But the other major issue-whether finite-horizon transver­ sality conditions can be generalized to the infinite-horizon context-deserves some further discussion, because serious doubts have been cast upon their applicability by some alleged counterexamples in optimal control theory. We shall examine such counterexamples and then argue that the counterexamples may be specious. 9.1 TRANSVERSALITY CONDITIONS In their celebrated book, Pontryagin et al. concern themselves almost exclusively with problems with a finite planning horizon. The only digres­ sion into the case of an improper-integral functional is a three-page discus­ sion of an autonomous problem in which the " boundary condition at 240 CHAPTER 9: INFINITE-HORIZON PROBLEMS 241 infinity" is assumed to take the specific form1 lim y(t) = yoo ( 9 .1) t -><<> In other words, the infinite horizon is specified to be accompanied by a fixed terminal state. For that case, it is shown that the maximum-principle condition-including the transversality condition for what we have been referring to as the horizontal-terminal-line problem-apply as they do to a finite-horizon problem. For the latter, the transversality condition is [ H ]t-T = 0 When the problem is autonomous, moreover, the maximized value of the Hamiltonian is known to be constant over time, so the condition H = 0 can be checked not only at t = T, but at any point of time at all. Accordingly, the transversality condition pertaining to an infinite-horizon autonomous problem with the boundary condition lim ( 9 .2) t � oo (9.1) can be expressed not only as H=O but more broadly as H= 0 ( 9 .3) for all t E [ 0 , oo) The fact that the transversality condition for the horizontal-terminal­ line problem can be adapted from the finite-horizon to the infinite-horizon context makes it plausible to expect that a similar adaptation will work for other problems. If the terminal state is free as t -> oo, for instance, we may expect the transversality condition to take the form ( 9.4) lim t � oo A(t) = 0 [ transversality condition for free terminal state] Similarly, i n case the variable terminal state i s subject to a prespecified minimum level Ymin as t -> oo, we may expect the infinite-horizon transver­ sality condition to be lim (9 .5) t -+<>0 A(t) ;::-: 0 and lim t � oo A(t) [y( t) - Ymin] = 0 The Variational View That (9.4) from an finite-horizon is a reasonable transversality condition can be gathered adaptation of the procedure used in Sec. 7.3 to derive the 1L. S. Pontryagin et a!., The Mathematical Theory of Optimal Processes, Interscience, New York, 1962, pp. 189- 191. . 242 ',. , PART 3: OPTIMAL CONTROL THEORY transversality conditions for problem (7.20). There, we first combined-via a Lagrange multiplier-the original objective functional with the equa­ tion of motion for into a new functional y V = t0 H(t,y, u ,) .) dt - (A(t)ydt 0 V [from ( 7.22' ) ] Then, after integrating the second integral by parts, we rewrote V = ((H(t,y, u , A) + y(t),\) dt - A(T)yr + A(O)y0 0 V as [from ( 7.22'')] To generate neighboring paths for comparison with the optimal control path and the optimal state path, we adopted perturbing curves for and for to write p(t) u q (t ) y , u (t) = u* (t) + Ep (t) and y(t) = y*(t) + Eq(t) [from ( 7.24) and ( 7 .25)] Similarly, for variable T and we wrote [from ( 7 .26)] T = T* + E ilT and As a result, the functional V was transformed into a function �f E. The Yr• Jr = Yr* + E ilYr first-order condition for maximizing V then emerged as , !JJ �·I:; :: = �T[(:� + A) q(t) + :�p(t) ] dt + [H]1_r!lT - A(T) !lyr = O [from ( 7 .30)] When adapted to the infinite-horizon framework, this equation be·I f1· comes (9.6) aHP (t) dt + lim H!lT - lim A(t) !lyT aH + A } q(t) + &;; dV ®[ ( ay d E J0 J . t�® �} jl ·· · ; -· , '" =0 nl !1 2 t � oo !13 In order to satisfy this condition, each of the three component terms (01, !12, !13) must vanish individually. It is the vanishing of the last two terms, !12 and !13, that gives rise to transversality conditions. The Variability of T and Yr For an infinite-horizon problem, the terminal time T is" not fixed, so that !lT is nonzero. Thus, to make the fi2 term in (9.6) vanish, we must impose CHAPTER 9: INFINITE-HORIZON PROBLEMS 243 the condition ( 9 . 7) lim H = O [ infinite-horizon transversality condition] / -HX> This condition appears to be identical with (9.2). Unlike the latter, however, (9.7) plays the role of a general transversality condition for infinite-horizon problems, regardless of whether the terminal state is fixed. It therefore carries greater significance. We have based the justification of this condition on the simple fact that the terminal time is perforce not fixed in infinite-horizon problems. But a full-fledged proof for this condition has been provided by Michel.2 The proof is quite long, and it will not be reproduced here. Instead, we shall add a few words of economic interpretation to clarify its intuitive appeal. If, following Dorfman (see Sec. 8.1), we let the state variable represent the capital stock of a firm, the control variable the business policy decision, and the F function the profit function of the firm, then the Hamiltonian function sums up the overall (current plus future) profit prospect associated with each admissible business policy decision. So long as H remains positive, there is yet profit to be made by the appropriate choice of the control. To require H ---> 0 as t ---> oo means that the firm should see to it that all the profit opportunities will have been taken advantage of as t ---> oo . Intuitively, such a requirement is appropriate whether or not the firm's terminal state (capital stock) is fixed. With regard to the 03 term in (9.6), on the other hand, it does matter whether Y r is fixed or free. Suppose, first, that the terminal state is fixed. Then we have lim 1 - ., Yr = y.,, same as in (9. 1). Since this implies ll.yr = 0 at time infinite, the 03 term will vanish without requiring any restriction on the terminal value of A. In contrast, with a free terminal state, we no longer have ll.yr = 0 at time infinite. Consequently, to make 03 vanish, it is necessary to impose the condition lim 1 _., A(t) = 0, and this provides the rationale for the transversality condition (9.4). Despite the preceding considerations, many writers consider the ques­ tion of infinite-horizon transversality conditions to be in an unsettled state. The transversality condition (9. 7) is not in dispute. However, the transver­ sality condition (9.4)-lim1 _., A(t) 0-has been thrown into doubt by several writers who claim to have found counterexamples in which that condition is violated. If true, such counterexamples would, of course, dis­ qualify that condition as a necessary condition. We shall argue, however, that those are not genuine counterexamples, because the problems involved do not have a truly free terminal state, so that (9.4) ought not to have been applied in the first place. = 2 Phillipe Michel, "On the Transversality Econometrica, July 1982, pp. 975-985. �- a IJIAlaite Horiaon Optimal Prablema," ' . ' • ' · ".·.. 244 9.2 PART 3, OPTIMAL CONTROL THEORY SOME COUNTEREXAMPLES REEXAMINED The Halkin Counterexample "' Perhaps the most often-cited counterexample is the following autonomous control problem constructed by Halkin:3 subject to (9.8) and { \..> {(1 - y ) u dt Maximize 0 y = ( l - y) u y(O) = 0 u(t) E [0, 1] Inasmuch as the integrand function F is identical with the f function in the equation of motion, the objective function can be rewritten as ( 9.9) 1"'y dt = [y(t)] � = 0 lim y ( t ) t -HIO - y(O) = lim y(t) t -t oo ' ' Viewed in this light, the objective functional is seen to depend exclusively on the terminal state at the infinite horizon. For such a problem, known as a terminal control problem, the initial and intermediate state values attained on an optimal path do not matter at all, except for their role as stepping stones leading to the final state value. Hence, to maximize (9.9) is tanta­ mount to requiring the state variable to take one specific value-the upper bound of y-as its terminal value. To find the upper bound of y, let us first find the y(t) path. The equation of motion for y, which can be written as y + u (t)y = u (t) i s a first-order linear differential equation with a variable coefficient and a 3 Hubert Halkin, "Necessary Conditions for Optimal Control Problems with Infinite Horizons," Econometrica , March 1974, pp. 267-272, especially p. 271. A slightly different version of the Halkin counterexample is cited in Kenneth J. Arrow and Mordecai Kurz, the Rate of Return , and Optimal Fiscal Policy, Public Investment, Johns Hopkins Press, Baltimore, MD, 1970, p. 46 (footnote). The Halkin counterexample has also been publicized in Akira Takayama, Mathematical Economics, 2d ed., Cambridge University Press, Cambridge, 1985, p. 625; and Ngo Van Long and Neil Vousden, " Optimal Control Theorems," Essay 1 in John D. Pitchford and Stephen J. Turnovsky, eds., Applications of Control Theory to Economic Analysis, Holland, Amsterdam, 1977, p. 16. North­ 245 CHAPTER 9: INFINITE-HORIZON PROBLEMS variable term. Its general solution is4 y(t) (9.10) ce - fu dt + 1 = ke - f6 u d t + 1 ( c arbitrary) ( k arbitrary) = I �)111'•'•' ' ' Next, by setting t = 0 in the general solution and making use of the initial condition y(O) = 0, we find 0 so that (9.10') k = = y(O) = ke 0 + 1 = k + 1 - 1 . Therefore, the definite solution is y(t) = 1 e - Jb u dt = 1 e - Z(t ) _ _ where Z(t) is a shorthand symbol for the definite integral of u . Since u(t) E [0, 1], Z(t) is nonnegative. Hence, the value of e - Z<t> must lie in the interval (0, 1], and the value of y(t) must lie in the interval [0, 1). It follows that any u(t) path that makes Z(t) --> oo as t --> oo-thereby making e - Z<t > --> 0 and y(t) --> 1-will maximize the objective functional in (9.10') and be optimal. In other words, there is no unique optimal control. Halkin himself suggests the following as an optimal control: u* u* = = 0 for t 1 for E [0, 1 ] ( 1 , oo) tE This would work because J� 1 dt is divergent, resulting in an infinite Z(t). But, as Arrow and Kurz point out (op. cit . p. 46), any constant time path for u , , u* ( t ) = u0 for all t (O < u 0 < 1) would serve just as well. The fact that Arrow and Kurz chose u(t) = u 0, where u 0 is an interior point in the control region [0, 1], makes it possible to use the derivative condition aH;au = 0. Since the Hamiltonian for this problem is (9 .11) H = ( 1 - y)u + ..\(1 - y) u = (1 + ..\ ) ( 1 - y) u the condition aH;au = 0 means ( 1 + ..\ )(1 - y) 0, or more simply (1 + ..\ ) = 0, inasmuch as y i s always less than one. Hence, we find ,.\* (t) = - 1, , = 4 See Alpha C. Chiang, Fundamental Methods ofMathematical Economics, 3d ed., McGraw-Hill, New York, 1984, Sec. 14.3, for the standard formula for solving such a differential equation. The rationale behind the conversion of the indefinite integral Ju dt to the definite integral J�u dt in (9.10) is discussed in op. cit., Sec. 13.3 (last subsection). The constant k differs from the constant c, because it has absorbed another constant that arises in connection with t-he lower limit of integration of the definite integral. 246 PART 3; OPTIMAL CONTROL THEORY which contradicts the transversality condition (9.4), and seemingly makes this problem a counterexample. As pointed out earlier, however, the structure of the Halkin problem is such that it requires the choice of the upper bound of y as the terminal state, regardless of the y*(t) path chosen. Put differently, there is an implicit fixed terminal state, even though this is not openly declared in the problem statement. In view of this, the correct transversality condition is not lim 1 _, ,. ,\(t) = 0 as in (9.4), but rather lim 1 _, ,. H = 0 as given in (9.2). More specifically, we should use the condition H = 0 for all t as in (9.3), because the problem is autonomous. Using the transversality condition H = 0 for all we can approach the Halkin problem as follows. First, noting that the Hamiltonian is linear in u , we only need to consider the boundary values of the control region as t, 1. candidates for u*(t), namely, either u = 0 or u = The consequence of choosing u = 0 at any point of time, however, is to make y = 0 (by the equation of motion) at that point of time, thereby preventing y from growing. For this reason, we must avoid choosing u (t) = 0 for all t.5 Suppose, instead, we choose the control u*(t) = for all 1 t Then we have the equations of motion aH A = -ay y= whoee aH - o,\ ( 1 + ,\ ) u = = ( 1 - y) u = = 1+A ' · 1 -y � solutions are, respectively, ,\ ( t ) = Ae1 y(t) By the initial condition = Be - y(O) = -1 1 + 0, = ( '"····· .::•; .' .. .,. . . , :· · ( A arbitrary) 1 we (B can :lefinite solution of the state variable is y(t) : 1 - e- 1 arbitrary) determine that· B = - 1, · · with lim 1 ->«> y(t) = · 1!10 r ·.· the .- ' 1 fhis latter fact shows that the control w e have chosen, u*(t) = 1, can naximize the objective functional just as those of Halkin and Arrow and Kurz. What about the arbitrary constant A? To definitize A, we call upon 'The choice of u -0 for some limited initial period � � dOiee Jio· harm, ���ne·e ollly tbe IIIJIIIIItlded. by Halldn doel � erminal state counts in the present problem. The eohrtion he feature that u - 0 initially, for t E [0, 1]. ' CHAPTER 9: INFINITE-HORIZON PROBLEMS 247 the transversality condition H = 0 for all t. From (9.11), we see that, with u = 1 and y < 1 for all t, the only way to satisfy the condition H = 0 is to have ,.\ 1 for all t. Hence, A = 0. Even though our optimal control differs from that of Halkin and that of Arrow and Kurz, the conclusion that ,\*(t) = - 1 is the same. Yet, we do not view this as a violation of an expected transversality condition. Rather, it represents a perfectly acceptable conclusion drawn from the correct transversality condition. In this light, Halkin's problem does not constitute a valid counterexample. Some may be inclined to question this conclusion on grounds that the requirement on the terminal state, being based on the maximization of (9.9), is in the nature of an outcome of optimization rather than an implicit restriction on the terminal state. A moment's reflection will reveal, how­ ever, that such an objection is valid only for a static optimization problem, where the objective is merely to choose one particular value of y from its permissible range. But in the present context of dynamic optimization, the optimization process calls for the choice of, not a single value of a variable , but an entire path. I n such a context, any restriction o n the state variable at a single point of time (here, the terminal time) must, as a general rule, be viewed as a restriction on the problem itself. The fact that the problem is a special case, with the objective functional depending exclusively on the terminal state, in no way disqualifies it as a dynamic optimization problem; nor does it alter the general rule that any requirement placed on the terminal state should be viewed as a restriction on the problem. = - Other Counterexamples In another alleged counterexample, Karl Shell6 shows that the transversal­ ity condition (9.4) does not apply when the problem is to maximize an integral of per-capita consumption c(t) over an infinite period of time: J; c(t) dt. But we can again show that the problem has an implicit fixed terminal state, so that (9.4) should not have been expected to apply in the first place. The Shell problem is based on the neoclassical production function Y Y( K, L), which, on the assumption of linear homogeneity, can be rewritten in per-capita (per-worker) terms as = y= where y = YjL and q,(k ) with f/J' ( k ) > 0, f/J"( k ) <0 k = KjL . Let n denote the rate of growth of labor, and 6Kari Shell, "Applications of Pontryagin's Maximum Principle to Economics," in H. W. Kuhn and G. P. Szegii, eds., Mathematical Systems Theory and Economics, New York, 1969, pp. 241-292, especially pp. 273-275. Vol. 1, Springer-Verlag, 248 PART 3: OPTIMAL CONTROL THEORY let � denote the rate of depreciation of capital. Then the rate of growth of k, the capital-labor ratio, is (9.12) ,.,. k = cP( k ) - c - ( n + 8)k which is similar to the funoamental differential equation of the Solow growth model.7 In a steady state, we should have k = 0. By setting k = 0 and transposing, we obtain the following expression for the steady-state per-capita consumption: (9.13) c = q,( k ) - (n + �)k [steady-state value of c ] Note that the steady-state value of c is a function of the steady-state value of k. The highest possible steady-state c is attained when dcjdk = q,' (k) ­ > ( n + �) = 0, that is, when (9.14) q,'( k ) = n + � The condition given in (9.14) is known as the golden rule of capital .>·d•· .. accumulation. 8 Since q,(k) is monotonic, there is only one value of k that · · will satisfy (9.14). Let that particular value of k-the golden-rule value of k-be denoted by k , and let the corresponding golden-rule value of c be denoted by c. Then (9.15) c = q,( k ) - ( n + �)k [by (9.13)] This value of c represents the maximum sustainable per-capita consump­ tion stream, since it is derived from the maximization condition dcjdk = 0. Now consider the problem of maximizing J; c(t) dt. Since this integral obviously does not converge for c(t) ;;::, 0, Shell adopts the Ramsey device of rewriting the integrand as the deviation from " Bliss"-with the " Bliss" identified in this context with c. In other words, the problem is to (9.16) Maximize {0 < c - c) dt subject to k = q,( k ) - c - ( n + �) k k(O) = k0 0 :o; c(t) :S q, [ k ( t)] and 7 For details of the derivation, see the ensuing discussion leading to (9.25). 8See E. S. Phelps, " The Golden Rule of Capital Accumulation: A Fable for Growthmen," American Economic Review, September 1961, pp. 638-643. This rule is christened the "golden rule" because it is predicated upon the premise that each generation is willing to abide by the golden rule of conduct ("Do unto others as you would have uniform savings ratio to be adhered to by all generations. others do unto you.") and adopt a 249 CHAPTER 9: INFINITE-HORIZON PROBLEMS Although the state variable, k , has a given initial value k0, no terminal value is explicitly mentioned. The control variable, is confined at every point of time to the control region [0, <f>(k)], which is just another way of saying that the marginal propensity to consume is restricted to the interval [0, 1] . For this autonomous problem, the Hamiltonian is c, H = c - c + -\(<f>( k ) - c - ( n + o)k] ( 9 . 17) Shell shows that the optimal path o f the costate variable has the limit value 1 as t -+ oo, and the optimal state path is characterized by lim 1 � ., k(t) = k . A(t) optimally tends to a unit value instead of zero leads Shell The fact that to consider this a counterexample against the transversality condition (9.4). However, the Shell problem in fact also contains an implicit fixed terminal state, so that the appropriate transversality condition is all t, rather than lim1 A(t) 0. H = 0 for To see this, recall that the very reason for rewriting the objective functional as in (9.16) is to ensure convergence. Since the integrand expression - c), a continuous function of time, maintains the same sign for all and offers no possibility of cancellation _ ., = (c t between positive and negative terms associated with different points of time, the only way for the integral to converge is to have - c) -> 0 as -> oo.9 Hence, must tend to c and, in line with (9. 15), must tend to k , as -> oo. What this does is to fix the terminal state, if only implicitly, in the (c k(t) c(t) t t form of lim 1 � ,., k, thereby disqualifying (9.4) as a transversality condition. The important point is that this terminal state does not come k(t) = about as the result of optimization; rather, it is a condition dictated by the convergence requirement of the objective functional. We can easily check the correct transversality condition 0 for t -> oo. As t becomes infinite, c and k will take the values H= c and k, respectively; moreover, c and k will be related to each other by (9_15). Hence, from (9.17), it i s clear that will tend t o zero. Since, for this H H autonomous problem, should be zero for all by setting (9. 17) equal to zero, to get t, we can actually solve for A c - c(t) A( t) -- ---;-:-....,.,--.--; ----:-:--:4> [ k ( t)] - c(t)- (n +---:o)k(t) ( 9 . 18) By taking the limit of this expression, we can verify Shell's result that 1 as -> oo. 1° From our perspective, however, this result by no A(t) -> t 9 See the discussion of Condition II in Sec. 5.1. 10 To do so, we can resort to L'Hopital's rule, since both the numerator and the denominator in (9.18) approach zero as lim t - oo t becomes infinite. Recalling that A ( t) = lim t - oo . - i: f'(k)k - i: - ( n + (9.14) is· satisfied as t -+ oo, we have a) k . = - i: lim -;- = 1 t - oo - c PART 3: OPTIMAL CONTROL THEORY 250 means contradicts any applicable transversality condition. Rather, it is perfectly consistent with, and indeed follows from, the correct transversality condition, H = for all t. Another intended counterexample is found in the regional investment model of John Pitchford.U Suppressing population growth, he considers the problem of choosing and 12, the rate of investment in each of two regions, so as to 0 11 Maximize subject to {9.19) {lc(t ) - c] dt K l = [1 - 5K l /(2 = 12 - 5Kz c �1( K1) + �2(K2) - l1 - l2 � 0 0 · ·£ " = and K1(0) > 0 K2 (0) > 0 11 � 0 12 � 0 The symbol c again denotes the maximum sustainable level of consumption. The model is essentially similar to the Shell counterexample, ·except that there are here two control variables, and 12 , and two state variables, K1 and as well as certain inequality constraints (a subject to be discussed in the next chapter). Pitchford also assumes implicitly that the terminal states are constrained to be nonnegative. The solution to this problem involves the terminal states > and > defined by the relations [cf. (9.14); here, n = Since the terminal value of each of the costate variables is found to be one, the transversality condition implied by (9.5) for nonnegative termi­ nal capital values, namely, lim , - � is not satisfied (Pitchford, op . p. 143). This, of course, would not have come as a surprise, if the objective functional had been required to converge. For then c(t) must tend to c, and must tend to thereby making the problem one with fixed terminal states, to which a different type of transversality condition should apply. However, not wanting to rule out the possible equilibria and a priori, Pitchford decides to proceed on the assumption that the integral is not required to converge-which, however, makes one wonder why he would then still employ the Ramsey device of expressing the integrand as c(t) - c, a device explicitly purported to ensure convergence. Because of this assumption, it takes a rather lengthy excursion before he is [1 K2 , K2 0, 0]. cit., (0, K2 ), �1'(K1) �2'(K2) = = 5 K1 0 Ai(t)Ki(t) 0, = Ki (K11 0) Ki, (0, 0), � ._d 11John D. Pitchford, " Two State Variable Problems," Essay 6 in John D. Stephen J. Turnovsky, eds. , Applications of Control Theory to Economic An8l,ai., North­ Holland, Amsterdam, 1977, pp. 127-154, especially pp. 134-143. 251 CHAPTER 9: INFINITE-HORIZON PROBLEMS led back to the conclusion that the objective functional has to converge after all. The upshot is then the same: The terminal states are implicitly fixed. And the model is not a genuine counterexample. The Role of Time Discounting In a noteworthy observation, Pitchford points out (op. cit., p. 128, footnote) that all the known counterexamples apparently share one common feature: No time discounting is present in the objective functional. While Pitchford offers no reason for this phenomenon, he reports a finding of Weitzman12 that, for discrete-time problems, the transversality condition involving the costate variables indeed becomes necessary when there is time discounting and the objective functional converges. Moreover, Pitchford conjectures that a similar result would hold for the continuous-time case. The relevant work on the continuous-time case comes later in a paper by Benveniste and Scheinkman. 13 What our foregoing discussion can offer is a simple intuitive explana­ tion of the link between the absence (presence) of time discounting on the one hand and the failure (applicability) of the transversality condition (9.4) or (9.5) on the other. Consider the case of maximizing J;[c(t) - c] dt. If this functional is required to converge, and given that no possibility exists for cancellation between positive and negative terms, the integrand [c(t) - c] must tend to zero as t � oo. Since the only way for this to occur is for c(t) to approach c, the terminal state for the problem is in effect fixed, whether or not explicitly acknowledged in the problem statement. But if time discount­ ing is admitted, so that the integral is, say, J;[c(t) - c]e-pl dt, then conver­ gence will merely call for the vanishing of [c(t) - c]e-p1 as t -> oo, which can be achieved as long as [ c(t) - c] tends to some finite number, not necessar­ ily zero. Inasmuch as c(t) is no longer obliged to approach a unique value c for convergence, and nor is K(t) obliged to approach a unique value K, the problem with time discounting would indeed be one with a free terminal state, with (9.4) or (9.5) as its expected transversality condition. The Shell and Pitchford models omit time discounting, and thus do not fall into this category. The case where the objective functional does contain a discount factor is illustrated by the neoclassical optimal growth model which we shall discuss in Sec. 9.3. 12 M. L. Weitzman, " Duality Theory for Infinite Horizon Convex Models," Science, 1973, pp. Management 783-789. 13L. M . Benveniste and J. A. Scheinkman, " Duality Theory for Dynamic Optimization Models of Economics: The Continuous Time Case," Journal of Economic Theory, 1982, pp. 1-19. PART 3o OPTIMAL CONTROL THEORY 252 The Transversality Condition as Part of a Sufficient Condition Although the transversality condition lim, A(t) = 0 is not universally accepted as a necessary condition for infinite-horizon problems, a limit condition involving A does enter as an integral part of a concavity sufficient condition for such problems. In an amalgamated form-combining the concavity provisions of Mangasarian as well as those of Arrow-the suffi­ ciency theorem states that the conditions in the maximum principle are sufficient for the global maximization of V in the infinite-horizon problem �oo {F(t , y, u ) dt Maximize V= subject to y = f( t , y, u ) and 0 y(O) = Yo ( Yo given) provided ( 9.20) either H = F(t, y, u) + A f(t, y, u) is concave in (y, u ) for all t E [0, T], or H 0 = F(t, y, u*) + A f(t, y, u*) is concave in y for all t, for given A and ( 9 .21 ) lim 1 -><0 A(t) [y(t) - y* (t)] � 0 where y*(t) denotes the optimal state path and y(t) i s any other admissible state path. The terminal state can be either fixed or free. Note that in (9.20) we have merged Mangasarian's concavity condi­ tions on the F and f functions and nonnegativity condition on A(t) into a single concavity condition on the Hamiltonian H. The concavity of H is in (y, u), jointly. In contrast, Arrow's condition is that H 0 be concave in the y variable alone. The limit expression in (9.21) is the infinite-horizon counterpart of the expression A*(TXyr - Yr *) which we have encountered earlier in the proof of the Mangasarian theorem [see footnote related to (8.31) and (8.3Y)]. In that proof, the expression just cited is shown to vanish, so that the result V s; V* emerges, establishing V* to be a global maximum. But, to establish the maximality of V*, it is in fact also acceptable to have A*(T )(yr - Yr *) positive. It is along the same line of reasoning that condition (9.21) restricts the limit of A(t)[y(t) - y*(t)] to be either positive or zero as t ...... oo. The reader may also find it of interest to compare (9.21) with (5.44), the corresponding condition in the calculus-of-variations context. At first glance, condition (5.44) may strike one as entirely different from (9.21) because it shows the " s; " rather than the " � " inequality. However, from (7.57) we CHAPTER 9: INFINITE-HORIZON PROBLEMS 253 recall that A - FY Since multiplying an inequality by a negative number reverses the sense of inequality, it � ·� that (9;tl) aMI (1t44) are ;.. -- · :· · ,;,'· ·· ::.,. ... ;_, · •,-,:·. ·y ; . . , .. , indeed exactly the same. = •. ,_ . 9.3 THE NEOCLASSICAL THEORY OF OPTIMAL GROWTH The Ramsey model of saving behavior (Sec. 5.3), which deals with the important issue of intertemporal resource allocation, has exerted a strong influence on economic thinking, although this influence did not come into being until after World War II, when that model was "rediscovered" by growth theorists long after its publication. In a more recent development, the same basic issue is formulated as a problem of optimal control rather than the calculus of variations. Moreover, the new treatment-labeled as " the neoclassical theory of optimal growth" -extends the Ramsey model in two major respects: (1) The labor force (identified with the population) is assumed to be growing at an exogenous constant rate n > 0 (the Ramsey model has n = 0), and ( 2) the social utility is assumed to be subject to time discounting at a constant rate p > 0 (the Ramsey model has p 0). Our discussion of this subject will be based primarily on a classic paper by David Cass.14 = The Model This theory is labeled a " neoclassical" theory, because its analytical frame­ work revolves around the neoclassical production function Y Y( K, L), assumed to be characterized by constant returns to scale, positive marginal products, and diminishing returns to each input. 15 Such a production function, being linearly homogeneous, can be rewritten in per-worker terms -or per-capita terms, as we shall draw no distinction between the popula­ tion and the labor force. Letting the lowercases of Y and K be defined, respectively, as = (average product of labor) ( capital-labor ratio) 14 David Cass, " Optimum Growth in an Aggregate Model of Capital Accumulation," Review of 1965, pp. 233-240. 15 In discussing the Ramsey model, we denoted output by Q. Here we use instead the notation Y, which is another commonly used symbol for income and output, especially at the macro leveL Economic Studies, July PART 3: OPTIMAL C9NTROL THEORY 254 0 .______ ___ __ k = !£ L FIGURE 9.1 we can express the production function by (9.22) y = cf>( k ) for all k > 0 with cf>'( k ) > 0 and cf>"( k ) < 0 Additionally, it is stipulated that lim cf>' ( k ) k --+ 0 = oo and lim cf>'( k) k .-+ CO = 0 The graph of cf>(k)-the APPL curve plotted against the capital-labor ratio -has the general shape illustrated in Fig. 9.1. The total output Y is allocated either to consumption C or gross investment Ig. Therefore, net investment, I = K, can be expressed as K= Ig - oK = Y- C - [ o = depreciation rate] oK Dividing through by L, and using the symbol c = CjL for per-capita consumption, we have (9.23) 1 . -K = y L - c - ok = cf>( k ) - c - ak The right-hand side now contains per-capita variables only, but the left-hand side does not. To unify the two sides, we make use of the relation . dK d dL dk K = dt = ( kL) = k dt + L dt dt = knL + Lk [product rule] [ n dLidt ] = = L( kn + k ) By substituting this last result into (9.23) and rearranging, we 6nally obtain an equation involving per-capita variables only: · (9. 24) k = cf>( k ) - c - ( n + o ) k · · '·. ' · 255 CHAPTER 9: INFINITE-HORIZON PROBLEMS This equation, describing how the capital-labor ratio k varies over time, is the fundamental differential equation of neoclassical growth theory which we have already encountered in (9.12). The level of per-capita consumption, c, is what determines the utility or welfare of society at any time. The social utility index function, U(c), is assumed to possess the following properties: U"( c ) < 0 U ( c) > 0 and lim U'(c) = oo c>0 lim U'(c) 0 for all ' ( 9.25) = c --> oo c --+ 0 The U(c) function is, of course, to be summed over time in the dynamic optimization problem. But, since the population (labor force) grows at the rate n, Cass decides that the social utility attained at any point of time should be weighted by the population size at that time before summing. Hence, with a discount rate p, the objective function takes the form ...... ,.: ( 9 .26 ) = L0 j"" U( c ) e-<p-n)t dt 0 To ensure convergence, Cass stipulates that p - n > 0. It might be pointed out, however, that this is equivalent to stipulating a single positive discount rate r, where r = p - n . If, further, we let L 0 1 by choice of unit, then the functional will reduce to the simple form = (r = p ( 9 .26' ) - n > 0) · In other words, weighting social utility by population size and simultane­ ously requiring the discount rate p to exceed the rate of population growth n, is mathematically no different from the alternative of not using popula­ tion weights at all but adopting a new, positive discount rate r. For this reason, we shall proceed on the basis of the simpler alternative (9.26' ). The problem of optimal growth is thus simply to _, _ ,,.,. : ' , .. • ::_ ; . : , ,_ , Maximize subject to · , . and k = 4>( k ) - c - ( n k(O) = k0 0 ::::; c( t) + 6) k .; '� ::::; <i> [ k ( t)] This resembles the Shell problem (9.16), but here the integrand function is not (c - c) (the deviation of c from the golden-rule level), but a utility index 256 PART 3, 0PTOlAL CONTROL THEORY H (and components) H \','· ''- :; - U(c)e-rt '·'''' FIGURE 9.2 function of per-capita consumption, discounted at the rate one state variable, k, and only one control variable, c. r. There is only The Maximum Principle The Hamiltonian for this problem, (9.28) H = U( c)e-rt + A [ 4>( k ) - c - ( n + o)k] is nonlinear in c. More specifically, at any given point of time, the first additive component of H would plot against c as the illustrative broken curve in Fig. 9.2, and the second component would appear as the broken straight line. 16 Their sum, H, is seen to contain a hump, with its peak occurring at a c value between c = 0 and c = c1. Since c 1 is the solution of the equation [4>(k) - c - (n + o)k] = 0, it follows that cl = cjJ(k) ­ (n + o)k, so that c 1 < <f>(k). Hence, the maximum of H corresponds to a value of c in the interior of the control region [0, <f>(k)]. We can accordingly find the maximum of H by setting aH iJc - = U'(c)e-rt - A = 0 From this, we obtain the condition (9.29) U'(c) = Aert which states that, optimally, the marginal utility of per-capita consumption should be equal to the shadow price of capital amplified by the exponential 16 The shape of the broken curve is based on the specification for U(c) in (9.25). The broken straight line has vertical intercept A[<f>(k) - (n + 8)k) and slope - A . We take A to be positive at any point of time, since it represents the shadow price (measured in utility) of capital at that time; hence, the negative slope of the broken straight line. 257 CHAPTER 9: INFINITE-f!ORIZON PROBLEMS term er t . Since a 2Hjac 2 = U"(c)e - r t is negative by (9.25), H is indeed maximized. The maximum principle calls for two equations of motion. One of these, A = - aH;ak, entails for the present model the differential equation A = - A [ cP' ( k ) - ( n + 8)] ( 9.30) And the other, ( 9.31) k = aH;aA, simply restates the constraint h = cP( k ) - c - ( n + o ) k The three equations (9.29) through (9.31) should in principle enable us to solve for the three variables c, A, and k. Without knowledge of the specific forms of the U(c) and cP(k ) functions, however, we can only undertake a qualitative analysis of the model. It is, of course, also possible to work with the current-value Hamiltonian (9.32) He = U( c) + m [ cP( k ) - c - ( n + 8 ) k ] In this case, the maximum principle requires that or (9.33) [ by (8.10) ] aHjac = U'(c) - m = 0, m = U'( c) This condition does maximize He, because a 2Hjac 2 = U"(c) < 0 by (9.25). The equation of motion for the state variable k can be read directly from the second line of (9.27), but it can also be derived as (9.34) . aHe k = - = cP( k ) - c - ( n + o ) k am And the equation of motion for the current-value multiplier ( 9.35) a He m= + rm = - m [ cP' ( k ) - ( n + o ) ] + rm ak - m [ cP' ( k ) - ( n + a + r ) ] m is [ by (8.13)] The ensuing discussion will be based on the current-value maximum­ principle conditions (9.33) through (9.35). Since there is no explicit t argument in these, we now have an autonomous system. This makes possible a qualitative analysis by a phase diagra� , as we did in Sec. 5.4. Constructing the Phase Diagram Since the two differential equations (9.34) and (9.35) involve the variables k and m , the normal phase diagram would be in the km space. To use such a diagram, it would be necessary first to eliminate the other variable, c. But PART So OPTIMAL CONTROL THEORY 258 (9.33), function since the other equation, contains a of c-to wit, U '(c)- instead of the plain c itself, the task of eliminating c is more complicated than that of eliminating m . We shall, therefore, depart from Cass's procedure and seek instead to eliminate the m variable. In so doing, we shall create a differential equation in the variable as a by-product. The analysis can then be carried out with a phase diagram in the kc space. We begin by differentiating with respect to to obtain an expression for rh : c (9.33) m t, = U"( c)c This, together with the equation m enables u s t o rh and m expressions. The result, after rearrangement, is = U'(c), rid (9.35) of the which is a differential equation in the variab le c. Consequently, we now can work with the differential equation system k = cf>( k ) - c - ( n + 8)k U' ( c) c = U" [cf>'(,lt) - ( n + 8 + r)] ( 9.36) - (c) T o construct the phase diagram, we first draw the c = 0 curve. These are defined by the two equations (9.37) c = cf>( k) - ( n + 8)k [ (9.38) cf>'( k ) = n + 8 + r [equation for equation for k k = = 0 curve and tJie 0 curve] c = 0 curve] 17 The k 0 curve shows up in the space as a concave curve, as illustrated in Fig. As indicates, this curve expresses c as the difference between two functions of The graph of already encountered earlier in Fig. is reproduced in Fig. And the term simply gives us an upward-sloping straight line. Plotting the difference between these two curves then yields the desired k curve in Fig. As to the curve, requires that the slope of the l/J(k) curve assume the specific value The curve being monotonic, this requirement can be satisfied only at a single point on that curve, point B, = 9.3b. c=0 17The kc k: cf>(k) and (n + 8)k. 9.1, (9.38) n + 8 + r. 9.3a. l/J(k), (n + 8)k =0 : ,·f· ':_. 9.3b. cf>(k) U'(c)/U"(c) term is always positive, by (9.25), and can never be zero . to vanish is to let the bracketed expression in (9.36) be zero. only way for c - (9.37) �� · · · CHAPTER 9: INFINITE-HORIZON PROBLEMS 259 (n + li)k �-""?r"- 1/J(k) (a) 0 k c " k /,c=O I I I I 1\ c 1/J'(k) D (b) c 0 (Modified golden rule) (Golden rule) = n + li + r /h=O c = 1/J(k) - (n I I I I I I I I I I I I I I k + li)k k FIGURE 9.3 which corresponds to a unique k value, k. Thus the c = 0 curve must plot as a vertical straight line in Fig. 9.3b, with horizontal intercept k. The intersection of the two curves in Fig. 9.3b, at point E, determines the steady-state values of k and c. Denoted by k and c, respectively, these values are referred to in the literature as the modified-golden-rule values of capital-labor ratio and per-capita consumption, as distinct from the golden­ rule values k and c discussed earlier in the Shell counterexample. As indicated in (9.14), k is defined by 4>'<fl) = n + B. Thus it corresponds to point A on the 4>(k) curve in Fig. 9.3a , where the tangent to that curve is parallel to the (n + B)k line. In contrast, k is defined by 4>'(k ) = n + B + r , by (9.38), which involves a larger slope of 4>(k). This is why point B (for k ) must b e located to the left of point A (for k ). I n other words, the modified-golden-rule value of k (based on time discounting at rate r) must be less than the golden-rule value of k (with no time discounting). By the same token, the modified-golden-rule value of c-the height of point E in Fig. 9.3b-must be less than the golden-rule value-the height of point D. PART 3: OPTIMAL CONTROL THEORY 260 Analyzing the Phase Diagram To prepare for the analysis of the phase diagram, we have, in Fig. 9.4, added vertical sketching bars to the k = 0 curve, and horizontal ones to the c = 0 curve. These sketching bars are useful in guiding the drawing of the streamlines-to remind us that the streamlines must cross the k = 0 curve with infinite slope, and cross the c = 0 curve with zero slope. For clues about the general directions the streamlines should take, we partially differentiate the two differential equations in (9.36), to find that (9.39) ak ac ac -1 < 0 U'( c) 4>"( k ) < 0 (by (9 .25) and (9.22)] ak - U"(c) According to (9.39), as c increases (going northward), k should follow the ( + , 0, - ) sign sequence. So, the k-arrowheads must point eastward below the k = 0 curve, and westward above it. Similarly, (9.40) indicates that c should follow the ( + , 0, -) sign sequence as k increases (going eastward). Hence, the c-arrowheads should point upward to the left of the c 0 curve, (9.40) = = and downward to the right of it. The streamlines drawn in accordance with such arrowheads Yield a saddle-point equilibrium at point E, (k, c), where k and c denote the intertemporal equilibrium values of k and c, respectively. The constancy of k implies that y = cf>(k) is constant, too. Since k = KjL and y = YjL, the simultaneous constancy of k and y means that, at E, the variables Y, K, and L all grow at the same rate. The fact that Y and K share a common growth rate is especially important as a sine qua non of a steady state or c FIGURE 9.4 261 CHAPTER 9: INFINITE-HORIZON PROBLEMS growth equilibrium. Given the configurations of the k = 0 and c = 0 curves in this problem, the steady state is unique. Note that the only way the economy can ever move toward the steady state is to get onto one of the stable branches-the "yellow brick road"-leading to point E. This means that, given an initial capital-labor ratio k0, it must choose an initial per-capita consumption level c0 such that the ordered pair (k0, c 0 )- not shown in the graph-lies on a stable branch. Otherwise, the dynamic forces of the model will lead us into a situation of either (1) ever-increasing k accompanied by ever-decreasing c (along the streamlines that point toward the southeast), or (2) ever-increasing c accompanied by ever-decreasing k (along the streamlines that point toward the northwest). Situation (1) implies progressively severe belt-tightening, culminating in eventual starvation, whereas situation (2) implies growing overindulgence, leading to eventual capital exhaustion. Since neither of these is a viable long-run alternative, the steady state at E, with a sustain­ able constant level of per-capita consumption, is the only meaningful long­ run target for the economy depicted in the present model. It may be noted that even at E, the per-capita consumption becomes constant, and its level cannot be raised further over time. This is because a static production function, Y = Y(K, L ), is assumed in the model. To make possible a rising per-capita consumption, technological progress must be introduced. This will be discussed in the next section. Transversality Conditions To select a stable branch from the family of streamlines is tantamount to choosing a particular solution from a family of general solutions by defini­ tizing an arbitrary constant. This is to be done with the help of some boundary conditions. The requirement that we choose a specific initial c0 such that the ordered pair (k0, c0) sits on the stable branch is one way of doing it. An alternative is to look at an appropriate transversality condition. Since we are dealing with general functions and not working with quantita­ tive solutions of differential equations, however, it is difficult to illustrate the use of a transversality condition to definitize an arbitrary constant. Nevertheless, we can, in light of the phase-diagram analysis, verify that the steady-state solution indeed satisfies the expected transversality conditions. One transversality condition we may expect the steady-state solution to satisfy is that (9.41) A -> 0 as t -> oo [by (9.4)] This is because the objective functional does contain a discount factor and the terminal !)tate is free. Since the solution path for A in the Cass model is A* = U'(c* )e-rt [ by (9.29)] 262 PART 3: 0PTDMAL CONTROL THEORY and since the limit of U ' (c*) is finite as t --+ oo/8 this expression does satisfy the transversality condition (9.41). Another condition we may expect is that, in the solution, H --+ 0 (9.42) (by (9.7)] as t -+ oo For the present problem, the solution path for H takes the form (by (9.28)] H* = U( c* )e -'1 + A* ( <f>( k * ) - c* - ( n + B ) k * ] Since U(c*) is finite as t -+ oo, the exponential term U(c*)e -'1 tends to zero as t becomes infinite. In the remaining term, we already know from (9.41) that A* tends to zero; moreover, the bracketed expression, representing k by (9.36), is equal to zero by definition of steady state. Therefore, the transversality condition (9.42) is also satisfied. Checking the Saddle Point by Characteristic Roots In Fig. 9.4, the pattern of the streamlines drawn leads us to conclude that the equilibrium at E is a saddle point. Since the sketching of the stream­ lines is done on a qualitative basis with considerable latitude in the position­ ing of the curves, it may be desirable to check the validity of the conclusion by another means. We can do this by examining the characteristic roots of the linearization of the (nonlinear) differential-equation system of the model. 19 On the basis of the two-equation system (9.36): L - k = 4>( k ) - c - ( n + o) k U ' ( c) c = - " [ c/l' ( k ) - ( n + 5 + r)] U (c) l ] we first form the Jacobian matrix and evaluate it at the steady-state point E, or (k, c), ( 9 .�) 18 ak ak JE = ac ak ! · ·:':. From (9.25), we see that does not tend to zero as t to have U'(c) ...., ...., oo, ak ac ac ac ;• I ' O< , c> we must have oo; thus the limit of i' <.i· ;· U'(c*) is c ...., finite 0. In the present problem, c* as t ...., oo. 19 For more details of the procedure of linearization of a nonlinear system, see Alpha C. Chiang, Fundamental Methods of Mathematical Economics, Sec. 18.6. 3d ed., McGraw-Hill, New York, 1984, ' 263 CHAPTER 9, INFINITE-HORIZON PROBLEMS The four partial derivatives, when evaluated at E, where turn out to be ::IE = IE cP'(k) - ( n + ll} ak = ac ac ak ac - ac I I E E r > 0 '·· ·· · · --ll.·h n +S + r, ":·, ·_ , , ,�, -,��,·� - -1 < o U'( c) cP"( k ) < 0 U"( c) - [ U" ( c) ] 2 + U "' ( c) U'(c) = = = 4''(k) = _ ( U"(c)] 2 ( cP' ( k ) - ( n + S + 1')] = 0 _ It follows that the Jacobian matrix takes the form (9 .43') The qualitative information we need about the characteristic roots r1 and r2 to confirm the saddle point is conveyed by that result that (9 .44) This implies that the two roots have opposite signs, which establishes the steady state to be locally a saddle point. EXERCISE 1 9.3 . '· A b 2 . . �-· Let the production function and the utility function be 1 b U = U - -c - . -� •, ., {0 < a < 1) ( b > O) .. ( a ) Find the y = ,P(k ) function and the U'(c) function. Check whether these conform to the specifications in (9.22) and (9.25). (b) Write the specific optimal control problem. (c) Apply the maximum principle, using the current-value Hamiltonian. (d) Derive the differential-equation system in the variables k and c. Solve for the steady-state values Oi, c). In the phase-diagram analysis of the Cass model, directional arrowheads in Fig. 9.4 are determined from the signs of the partial derivatives ak joe and iJcjiJk in (9.39) and (9.40). 264 PART 3: OPTIMAL CONTROL THEORY (a) Is it possible andjor desirable to accomplish the same purpose by using \ okjiJk in lieu of ak;ac? Why? (b) Is it possible andjor desirable to consider the ·sign of iJcjiJc in place of ac;ak? Why? In Fig. 9.4, suppose that the economy is currently located on the lower stable branch and is moving toward point E. Let there be a parameter change such that the c 0 curve is shifted to the right, but the k 0 curve is unaffected. (a ) What specific parameter changes could cause such a shift? (b) What would happen to the position of the steady state? (c) Can we consider the preshift position of the economy-call it ( k 1, c1)-as the appropriate initial point on the new stable branch in the postshift phase diagram? Why? Check whether the Cass model (9.27) satisfies the Mangasarian sufficient conditions. = 4 9.4 = EXOGENOUS AND ENDOGENOUS TECHNOLOGICAL PROGRES S The neoclassical optimal growth model discussed i n the preceding section provides a steady state in which the per-capita consumption, c, stays constant at c, leaving no hope of any further improvement in the average standard of living. The culprit responsible for the rigid capping of per-capita consumption is the static nature of the production function, Y = Y(K, L ). Since the same production technology holds for all t, no upward shift over time can occur. Once technological progress is allowed, however, we can easily remove the cap on per-capita consumption. "-j! Dynamic Production Functions For a general representation of a dynamic production function, we can simply write (9.4 5 ) Y = Y( K, L , t) ( ¥; > 0 ) The positive sign of ¥; shows that technological progress does take place, but it offers no explanation of how the progress comes into being. This representation can thus be used only for exogenous technological progress. An alternative way of writing a dynamic production function is to introduce a technology variable explicitly into the function. Let A = A(t) represent the state of the art, with dAjdt > 0, then we can write (9.46) Y = Y( K, L , A) The advantage of (9.46) over (9.45) is that, with an explicit variable A, we 265 CHAPTER 9: INFINITE-HORIZON PROBLEMS now can either leave A as an exogenous variable or �e it endogenous by postulating how A is determined in the model.. . ·. :� Neutral Exogenous Technological Progress Technological progress is said to be " neutral" when it leaves a certain economic variable unaffected under a stipulated circumstance. Specifically, technological progress is Hicks-neutral if it leaves the marginal rate of technical substitution (MRTS = MPPLfMPPK) unchanged at the same K1L ratio. That is, if we hold KjL constant and examine the MRTS's before and after progress, the two MRTS 's will be identicaL To capture this feature, we can use the following special version of (9.46): Y = A( t ) Y( K, L ) ( 9 .47) [Hicks-neutral] where Y(K, L) is taken to be linearly homogeneous. The reason why the variable A will not disturb the ratio of the marginal products of K and L is that it lies outside of the Y(K, L) expression. In contrast, technological progress is Harrod-neutral if it leaves the output-capital ratio (YjK) unchanged at the same MPPK . The special version of (9.46) that displays this feature is Y = Y [ K, A( t)L] (9.48) [Harrod-neutral] where Y is taken to be linearly homogeneous in K and A(t)L. Since A(t) is attached excbsively to L, this type of technological progress is said to_ be purely labor-augmenting. Considering the way A(t) and L are combined, we may view technology and labor as perfect substitutes in the production process. It is the fact that A is totally detached from the K variable that explains how the YjK ratio can remain undisturbed at the same MPPK as technological progress occurs. A third type of neutrality, Solow neutrality, is the mirror image of Harrod neutrality, with the roles of K and L interchanged. The function can then be written as Y = Y [ A( t ) K , L] ( 9.49) where ·y [Solow-neutral] is taken to be linearly homogeneous in AK and L. Neoclassical Optimal Growth with Harrod-Neutral Progress In growth models with exogenous technological progress, Harrod neutrality is frequently assumed. The main reason for the popularity of Harrod neutrality is that it is perfectly consistent with the notion of a steady state; it can yield a dynamic equilibrium in which Y and K may grow apace with 266 PART 3: OPTIMAL CONTROL THEORY each other. As a bonus, it turns out that Harrod-neutral technological progress entails hardly any additional analytical complexity compared with the case where a static production function is used. Let us define efficiency labor, 71, by ( 9 .50) Then the production function (9.48) becomes Y = Y( K, 71 ) ( 9 .51) If we now consider efficiency labor 71 (rather than natural labor L) as the relevant labor-input variable, then (9.51) can be treated mathematically just as a static production function, despite that its forebear-(9.48)-is really dynamic in nature. By the assumption of linear homogeneity, we can rewrite (9.51) as where y ( 9 .5f ) 'I "" y - and k "" 71 'I K - 71 This is identical in nature with (9 . 22), and we shall indeed retain all the assumed features of cf> enumerated in connection with (9.22). The equation of motion for k'l, the new state variable, can be found by the same procedure used to derive (9.24). The result is ( 9 .52) c � A a=A where c" "" 71 t and n "" ­ L The optimal control problem is therefore Maximize (9.53) subject to k" = cf>(k,.J - c'l - (e� + n + 3)i., k'I(O) = k'l0 and The similarity between this new formulation and the old problem (9.27) should be patently clear. Since the current-value Hamiltonian in the new problem is . '· ,-:. ( 9.54) where m' is the new current-value costate variable, the maximum principle 267 CHAPTER 9: INFINITE-HORIZON PROBLEMS calls for the following conditions: ( 9 .55) (9.56) ( 9.57) m' = U'( c..,) [cf. ( 9 .33) ] k.., = c/J( k..,) - c.., - (a + n + S ) k .., rh' = - m' [ c/J ( k ..,) ' - [ cf. (9 .34)] (a + n + a + r)j [cf. ( 9 .35)] By eliminating the m' variable, the last three equations can be condensed into the differential-equation system k.., = lf>( k..,) - c.., - ( a + n + S) k.., ( 9 .58) c.., = - U'( c.,) [c/J' ( k .., ) - ( a + n + a + r)j " U ( c.., ) [cf. ( 9 .36)] These then give rise to the following pair of equations for the new phase diagram: (9.59) c.., = lf>(k..,) - (a + n + S)k.., ( 9.60) c/J'(k..,) = a + n + a + r ( equation for k.., = 0 curve] [ equation for c.., = 0 curve] Since the phase-diagram analysis is qualitatively the same as that of Fig. 9.4, there is no need to repeat it. Suffice it to say that the intersection of the k.., = 0 curve and the c.., = 0 curve delineates a steady state in which Y, K, and 7J all grow at the same rate. But one major difference should be noted about the new steady state. Previously, we had c = CjL constant in the steady state. Now we have instead (9.61) c c c.., = - = 7J = AL c L . :.. �.1 . which implies that (9.6f) ;_.":· ·,,�,.,.,, conllt.ut' - = c .., A . J � j>' --.-. .. Thus, as long as A increases as a result of technological progress, per-capita consumption C1L will rise over time pari passu . The cap on the average standard of living has been removed. Endogenous Technological Progress While exogenous technological progress has the great merit of simplicity, it shirks the task of explaining the origin of progress. To rectify this shortcom­ ing, it is necessary to endogenize technological progress. As early as the 1960s, economists were already exploring the economics of knowledge ere- PART 3o OPTIMAL CONTROL THEORY 268 ation. The well-known " learning-by-doing" model of Kenneth Arrow,20 for example, regards gross investment as the measure of the amount of " doing" that results in " learning" (technological progress). As another example, a model of Karl Shell,21 which we shall now briefly discuss, makes the accumulation of knowledge explicitly dependent on the amount of resources devoted to inventive activity. The production function in the Shell model, Y(t ) = Y [ K( t ) , L ( t ) , A(t ) ] is the same as (9.46). But now the variable A(t), denoting the stock knowledge, is given the specific pattern of change (9.62) of (0 < u s 1 , 0 s a s 1 , {3 ;;:: 0) A(t) = ua( t) Y( t ) - {3A( t ) where u is the research success coefficient, a(t) denotes the fraction of output channeled toward inventive activity at time t, and f3 is the rate of decay of technical knowledge. Out of the remaining resources, a part will be saved (and invested). The K(t) variable thus changes over time according to (9.63) K(t) = s ( t ) [ 1 - a(t) ] Y( t) - 8K(t) where s denotes the propensity to save and 8 denotes the rate of deprecia­ tion. In order to focus attention on the accumulation of knowledge and capital, we assume that L is constant, and (by choice of unit) set it equal to one. In a decentralized economy, the dynamics of accumulation can be traced from the equation system (9.62) and (9.63) in the two variables A and K. If a government authority seeks to maximize social utility, on the other hand, there then arises the optimal control problem (9.64) Maximize ['u[ ( 1 - s ) ( l - a ) Y ] e -p' dt 0 subject to A = uaY( K, A ) - {3A K = s( 1 - a ) Y( K, A) - 8K and A(O) = A0 K(O) = K0 The integrand in the objective functional may look unfamiliar, but it is merely another way of expressing U(C)e -P1, because C is the residual of Y 20 Kenneth J. Arrow, " The Economic Implications of Learning by Doing," Studies, June 1 962, 21 pp. 155-173. Karl Shell, " Towards a Theory of Inventive Activity Economic Review, May 1966, pp. 62-68. · and Review of Economic Capital Accumulation," American CHAPTER 9: INFINITE-HORIZON PROBLEMS 269 after deducting what goes to inventive activity and capital accumulation: C = Y - aY - s ( l - a ) Y = ( 1 - s){l - a)Y This problem contains two state variables ( A and K ) and two control variables (a and s). The maximum principle is, of course, directly applicable to this prob­ lem. But, with two state variables, the solution process is not simple. Shell presents no explicit solution of the dynamics of the model, but shows that A and K will approach specific constant limiting values A and K . There is no need to enumerate these limiting values here, for their significance lies not in their exact magnitudes, but in the fact that they are constant. Since A _. A and K _. K as t _. oo, the specter of a capped standard of living is here to visit us again. One may wonder whether allowing a growing population and making the social utility a function of per-capita consumption will change the outcome. The answer is no. The difficulty, as Kazuo Sato (a discussant of the Shell paper) points out,22 may lie in the specification of A in (9.62), which ties A to Y. If (9. 62) is changed to (9.65) A = ua A - {3A or A - = ua A {3 then no cap on the growth of knowledge will exist so long as ua - {3 > 0. Interestingly, a similar idea is featured in a recent model of endogenous technological progress by Paul Romer to ensure unbounded progress. Endogenous Technology a Ia Romer Knowledge, according to the Romer model,23 can be classified into two major components. The first component, which may be broadly labeled as human capital, is person-specific. It constitutes a " rival good" in the sense that its use by one firm precludes its use by another. The other component, to be referred to as technology, is generally available to the public. It is a " nonrival good" in the sense that its use by one firm does not limit its use by others. Because of the rivalous nature of human capital, the person who invests h,l the accumulation of human capital receives the rewards arising therefrom. In contrast, the nonrivalry feature of technology implies knowl­ edge spillovers, such that the discoverer of new technology will not be the 22American Economic Review, May 1966, p. 79. 23 Paul M. Romer, " Endogenous Technical Change," Journal of Political Economy, No. 5, Part 2, October 1990, pp. 871-8102. Vol. 98, 270 PART 3, OPTIMAL CONTROL THEORY sole beneficiary of that discovery. The inability of the discoverer to reap all the benefits creates an economic externality that causes private efforts at technological improvement to fall short of what is socially optimal. Both human capital and technology are created by conscious action. To reduce the number of state variables, however, human capital is simply assumed to be fixed and inelastically supplied, although its allocation to different uses is still to be endogenously determined. Romer denotes human capital by H, but to avoid confusion with the symbol for the Hamiltonian, we shall use S (skill or skilled labor) instead, with S 0 as its fixed total. Since S can be used for the production of the final good, Y, or for the improvement of technology, A (state of the art), we have (9.66) Technology A, on the other hand, is not fixed. It can be created by engaging human capital SA in research and applying the existing tecliliology A u follows: (9.67) where u is the research success parameter. This equation is closely similar to (9.65), except that, here, human capital SA has replaced a (the fraction of output channeled to research), and /3 = 0 (technology does not depreci­ ate). Note that A;A = uSA > 0 as long as both u and SA are positive. Thus technology can grow without bound. Note also that research activity is assumed to be human-capital-intensive and technology-intensive, with no capital (K) and ordinary unskilled labor ( L) engaged in that activity. In the production of the final good Y, however, K and L do enter as inputs along with human capital Sy and technology A. The production function for Y is developed in several steps. First, think of technology as made up of an infinite set of "designs" for capital, including those that have not yet been invented at the present time. If we let x, = 0 for i � A, then A can serve as the index of the current technol­ ogy. The production function for the final good is assumed to be of the Cobb-Douglas type: where L0 indicates a fixed and inelastically supplied amount of ordinary labor. For simplicity, Romer allows all the designs x, to enter in an additively separable manner. As the second step, let the design index i be 271 CHAPTER 9: INFINITE-HORIZON PROBLEMS turned into a continuous variable. The production function then becomes (9.68) Since all x(i) enter symmetrically in the integrand, we may deduce that there is a common level of use, x, of all x(i). It follows that )[AX ( t" ) l -a-f3 dt· ,0 = = Therefore, (9.68) can be simplified to [A-l-a-f3 d" ) , 0 t X ;xt-a-{3 iA di 0 = AXJ -a-{3 (9.68') Next, assume that (1) capital goods are nothing but foregone consumption goods, both types of goods being subject to the same final-good production function, and (2) it takes -y units of capital goods to produce one unit of any type of design. Then the amount of capital actually used will be K = -yAX => K x= -yA Substituting"this into (9.68') results in ( 9.68" ) Y = Sy"'L /A = = ··· ,. ( K ) l -a - {3 -yA -J SyaL /Aa + f3K l - a-f3 -y a +{3 a + f3 - ! ( S yA ) ( L o A ) f3 K l -a -f3-ya In this last function, all four types of inputs are present: human capital Sy, technology A , labor L 0 , and capital K. More importantly, we observe from the (SyA) and ( L 0 A) expressions that technology can be viewed as human­ capital-augmenting as well as labor-augmenting, but it is detached from K. In other words, it is characterized by Harrod neutrality-here endogenously introduced rather than exogenously imposed. Our experience with Harrod neutrality suggests that this production function is consistent with a steady state in which technology-and, along with it, output, capital, and con­ sumption-can all grow without bound. In this model, consumption C does not need to be expressed in per-capita terms because L 0 (identified with population and measured by head count) is constant. By the same token, K rather than K1L can ' PART s, OPTIMAL CONTROL THEORY 272 appropriately serve as the variable for analysis. Net investment, as usual, is just output not consumed. Thus, recalling that Sy = 80 - SA, we have (9.69) K= Y- C The Optimal Control Problem Facing this background, society may consider an optimal control problem with two state variables, A and K-with (9.67) and (9.69) as their equa­ tions of motion-and two control variables, C and SA . Adopting the specific constant-elasticity utility function U( C ) = c1-a (0 < 8 < 1) 1-8 the control problem takes the form (9.70) "" c l - e Maximize f --e -pt dt 1-8 subject to A = uSA A 0 " K = y"+P - !A<> +il ( S0 - SA) L rfKl -a-13 - C A(O) and = K(O) = K0 A0 For convenience, define the shorthand symbol (9.71) Then we have the current-value Hamiltonian He = c1-e 1 _ 8 + AA( uSAA ) + AK(d - C ) where AA and AK represent the shadow prices of A and K , respectively. From this, we get the conditions (9.72) (9 .73) oHc ac - = c - e - ,\K = 0 = CHAPTER 9: INFINITE-HORIZON PROBLEMS 273 In addition to the A and K equations given in the problem statement, the maximum principle requires the following equations of motion for the costate variables: . (9.74) aHc aA aHc AA = - - + pAA = -AAuSA - A K ( a + I') A - 1 A + pA A . Ax = - aK + pAx = - A x ( l - a - 13 ) K - 1 A + pAx The Steady State Since there are four differential equations, the system cannot be analyzed with a phase diagram. And solving the system explicitly for its dynamics is not simple. Romer elects to focus the discussion on the properties of the balanc�d growth equilibrium inherent in the model: a steady state with Harrod-neutral technological progress. Questions of interest include: What is the rate of growth in that steady state? How is the growth rate affected by the various parameters? What economic policies can be pursued to promote growth? The basic feature of such a steady state is that the variables Y, K, A, and C all grow at the same rate. We thus have in the steady state ( 9.75) ¥ K 6 A - = - = - = - = uSA y K c A ( by (9.67)] But we want to express the growth rate in terms of the parameters only, with SA substituted out. Since Ax = c-8 [by (9.72)], we can calculate (9.76) - oc-8- 1 6 c 8 6 c = -0- = - OuSA If we get another Ax/Ax expression from the second equation in (9.74), then we can equate it to (9. 76) and solve for SA' But that equation is not convenient to work with. Instead, Romer first calculates AA /AA from the first equation in (9.74), then takes advantage of the relation ix/Ax = >.AlA A that characterizes the steady state. By dividing the AA expression in (9.74) through by AA and simplifying, it is found that (9.77) Equating (9.76) and (9. 77) and solving for SA, we then ascertain that SA PAR"r S: OJ"'''MAL COHTROL THBOKY 274 attains in steady state the constant value (9 .78) SA = a ( a + {3) S0 - ap ------- a ( a8 + {3 ) I t follows that the parametriC&llY expressed steady-state growth rate i s Y K C A - =- = - = - = (9.79) Y K C a ( a + {3 ) S0 - ap A aO + {3 The result in (9.79) shows not only the growth rate, but also how the various parameters affect that rate. Visual inspection is sufficient to estab­ lish that human capital S 0 has a positive effect on the growth rate, as does the research success parameter a, but a negative effect is exerted by the discount rate p. More formally, the effects of the various parameters can be found by taking the partial derivatives of (9.79). To conclude this discussion, we should point out a complication regard­ ing the rate-of-growth expression in (9.79): A rate of growth should be a pure number, and economic commonsense would lead us to expect it to be a positive fraction. Yet the expression in (9.79) comprises not only the pure­ number parameters a, a, {3, p, and 8, but also the parameter S0 which has a physical unit. The magnitude of the expression thus becomes problematic. The problem may have started at an early stage with the specification of (9.67), where SA• a variable with a physical unit, enters into the rate of growth A;A. If so, the remedy would be to replace SA with a suitable ratio figure in (9.67). EXERCISE 9.4 Y = A(t)K"LfJ, the 1 Show that, in the Cobb-Douglas production function 2 Derive the equation of motion (9.52) by following the same procedure used 3 4 5 6 \·. . technological progress can be considered to be either ( a ) Hicks-neutral, ( b ) Harrod-neutral, o r ( c ) Solow-neutral. to derive (9.24). Verify the validity of the maximum-principle conditions in (9.55), (9.56), and (9.57). Verify that the differential-equation system (9.58) is equivalent to the set of equations (9.55), (9.56), and (9.57). What would happen if (9.67) were changed to concave function? A = liSAI{l( A ), where 1{1 is a In the Romer model, how is the steady-state growth rate specifically affected by the following parameters? (a} a ( b ) {3 (c) IJ ! ' 1!1 �1 o il n �.�{ �� �� CHAPTER 10 OPTIMAL CONTROL WITH CONSTRAINTS Constraints have earlier been encountered in Sec. 7.4, where we discussed various types of terminal lines, including truncated ones. Those constraints are only concerned with what happens at the endpoint of the path, and the conditions that are developed to deal with them are in the nature of transversality conditions. In the present chapter, we turn to constraints that apply throughout the planning period [0, T ]. As in the calculus of variations, the treatment of constraints in optimal control theory relies heavily on the Lagrange-multiplier technique. But since optimal control problems contain not only state variables, but also control variables, it is necessary to distinguish between two major categories of constraints. In the first category, control variables are present in the constraints, either with or without the state variables alongside. In the second category, control variables are absent, so that the constraints only affect the state variables. '1\s we shall see, the methods of treatment for the two categories are different. 10. 1 CONSTRAINTS INVOLVING CONTROL VARIABLES Within the first category of constraints-those with control variables pres­ ent-four basic types can be considered: equality constraints, inequality constraints, equality integral constraints, and inequality integral con275 PART 3, OPTIMAL CONTROL THEORY 276 straints. We shall in general include the state variables alongside the control variables in the constraints, but the methods of treatment discussed in this section are applicable even when the state variables do not appear. Equality Constraints in Let there be two control variables required to satisfy the condition u 1 and u2, that a problem, · :.,. • are >) We shall refer to the g function as the constraint function, and the constant c as the constraint constant. The control problem may then be stated as ( 10.1) ,. Maximize j F( t, y, u 1, u 2) dt subject to Y T 0 = f( t, y , u1, u 2) g( t, y, u 1, u2) = c and boundary conditions This is a simple version of the problem with m control variables and q equality constraints, where it is required that q < m . The maximum principle calls for the maximization o f the Hamiltonian ( 10.2) for every t E [0, T ]. But this time the maximization of H is subject to the constraint g(t, y, u 1 , u 2) = c . Accordingly, we form the Lagrangian expres­ sion J= H + 8(t) [ c - g(t , y, u 1 , u2) } ( 10.3) = F( t , y , u 1 , u 2) + >. ( t) f( t, y , u 1 , u 2) + 8(t) [ c - g(t , y , u 1 , u2 )} where the Lagrange multiplier 8 is made dynamic, as a function of t. This is necessitated by the fact that the g constraint must be satisfied at every t in the planning period. Assuming an interior solution for each u J • we require that ( 10.4) iJ../ = iJu1 iJg iJF iJ f + >. - - 8 - = 0 iJ u i iJ u 1 iJ u i for all t E [O, T ] ( j = 1, 2) CHAPTER 10: OPTIMAL CONTROL WITH CONSTRAINTS 277 Simultaneously, we must also set i)..£' aiJ ( 10.5) = c - g( t , y , u 1, u 2 ) = 0 for all t E [0, T ] to ensure that the constraint will always be in force. Together, (10.4) and (10.5) constitute the first-order condition for the constrained maximization of H. This must be supported, of course, by a proper second-order condition or a suitable concavity condition. The rest of the maximum-principle conditions includes: ( 10.6 ) and ( 10.7) y = ( = ilH ) a../ ilA [equation of motion for y] aA ( ). = _ a../ = _ ilH ay ay + 8 ag ay ) [equation of motion for A ] plus an appropriate transversality condition. Note that the equation of motion for y, (10.6), would turn out the same whether we differentiate the Lagrangian function (the augmented Hamiltonian) or the original Hamilto­ nian function with respect to A . On the other hand, it would make a difference in the equation of motion for A, (10. 7), whether we differentiate .../" or H with respect to y. The correct choice is the Lagrangian expression ..£'. This is because, as the constraint in problem (10.1) specifically pre­ scribes, the y variable impinges upon the range of choice of the control variables, and such effects must be taken into account in determining the path for the costate variable A . While it i s feasible to solve a problem with equality constraints in the manner outlined above, it is usually simpler in actual practice to use substitution to reduce the number of variables we have to deal with. Substitution is therefore recommended whenever it is feasible. Inequality Constraints \ The substitution method is not as easily applicable to problems with in­ equality constraints, so it is desirable to have an alternative procedure for such a situation. We first remark that when the g constraints are in the ineq uality form, there is no need to insist that the number of control variables exceed the number of constraints. This is because inequality constraints allow us much more latitude in our choices than constraints in the form of rigid equalities. For simplicity, we shall illustrate this type of problem with two PART 8: OPl'IMAL CONTROL TJJE<)RY 278 control variables and two inequality constraints: Maximize J TF( t,y, u1, u 2 ) dt subject (o Y and boundary conditions ( 10.8) 0 = f(t,y, U v u 2 ) gl(t,y, u l, u 2 ) :;; cl g 2 ( t, y, ul> u 2 ) :;; c 2 The Hamiltonian defined in (10.2) is still valid for the present problem. But since the Hamiltonian is now to be maximized with respect to u 1 and u 2 subject to the two inequality constraints, we need to invoke the Kuhn­ Tucker conditions. Besides, for these conditions to be necessary, a con­ straint qualification must be satisfied. According to a theorem of Arrow, Hurwicz, and Uzawa, any of the following conditions will satisfy the con­ straint qualification1: All the constraint functions gi are concave in the control variables uJ [here, concave in (u1, u 2 )]. (2) All the constraint functions gi are linear in the control variables u J [here, linear in (u 1 , u 2)]-a special case of (1). (3) All the constraint functions gi are convex in the control variables u 1. And, in addition, there exists a point in the control region u 0 E U [here, u 0 is a point (u 10, u 20)] such that, when evaluated at u0, all constraints gi are strictly < c ; . (That is, the constraint set has a nonempty interior.) (1) {4) The gi functions satisfy the rank condition: Taking only those con­ straints that turn out to be effective or binding (satisfied as strict equalities), form the matrix of partial derivative [agijau ). (where e indicates "effective constraints only"), and evaluate the partial deriva­ tives at the optimal values of the y and u variables. The rank condition is that the rank of this matrix be equal to the number of effective constraints. .... . 1K. J. Arrow, L. Hurwicz, and H. Uzawa, " Constraint Qualifications in Nonlinear Program­ ming," Naval Research Logistics Quarterly, January 1961. The summary version here is adapted from Akira Takayama, Mathematical Economics, 2d ed., Cambridge University Press, Cambridge, 1985, p. 648. Note that our constraints are written as g :s; c (rather than g ;;,; c as in Takayama). ;;; CHAPTER 10: OPTIMAL CONTROL WITH CONSTRAINTS 279 We now augment the Hamiltonian into a Lagrangian function: J= F( t, y, u 1 , u 2 ) ( 10.9) + A(t) f( t, y , u 1 , u 2 ) + 0 1( t) [ c1 - g 1 (t, y, u 1 , u 2 )] + 02 (t) [ c2 - g 2 ( t , y, u 1 , u 2 ) ] The essence of _/ may become more transparent if we suppress all the arguments and simply write ( 10.9') The first-order condition for maximizing _/ calls for, assuming interior solutions, ( 10.10) as well as ( 10.11) iJJ j)_/ . - = C·' - g ' � 0 iJO; ( i = 1 , 2 and j = =0 0' iJO; 1 , 2) for all t E [ 0, T ] Condition (10.1 1) differs from (10.5) because the constraints in the present problem are inequalities. The iJ.LjiJO; � 0 condition merely restates the i th constraint, and the complementary-slackness condition O;(iJd'jiJO) = 0 en­ sures that those terms in (10.9) involving 0; will vanish in the solutio·n, so that the value of _/ will be identical with that of H = F + A f after maximization. Note that, unlike in nonlinear programming, we have in (10. 10) the first-order conditions iJJjiJ u i = 0, not iJd'jiJ u i � 0. This is because the u i variables are not restricted to nonnegative values in problem (10.8). If the latter problem contains additional nonnegativity restrictions ui ( t) � O then, by tht;, . �n-Tucker conditions, we should replace the /Jd'fiu i conditions b{ ('Ib.10) with . =0 ' ( 10.12) It should be pointed out that the symbol .L' in (10.12) denotes the same Lagrangian as defined in (10.9), without separate O(t) type of multiplier terms appended on account of the additional constraints u 1(t) � 0. This procedure is directly comparable to that used in nonlinear programming. The alternative approach of adding a new multiplier for each nonnegativity restriction u i(t) � 0 is explored in a problem in Exercise 10.1. 280 PART 3, OPTIMAL CONTROL THEORY Other maximum-principle conditions include the equations of motion for y and A. These are the same as in (10.6) and (10. 7): ad' y= ­ aA ( 1 0 . 13) ad' . A = -­ ay and Wherever appropriate, of course, transversality conditions must be added, too. Isoperimetric Problem When an equality integral constraint is present, the control problem is known as an isoperimetric problem. Two features of such a problem are worth noting. First, the costate variable associated With the integral con­ straint is, as in the calculus of variations, constant over time. Second, although the constraint is in the nature of a strict equality, the integral aspect of it obviates the need to restrict the number of constraints relative to the number of control variables. We shall illustrate the solution method with a problem that contains orie state variable, one control variable, and one integral constraint: ( 10.14) Maximize tF( t , y , u ) dt subject to y = f(t , y , u ) 0 {G(t, y, u ) dt = k 0 and y(O) y(T) free = Yo ( k given) (y0, T given) The approach to be used here is to introduce a new state variable r(t) into the problem such that the integral constraint can be replaced by a condition in terms of f(t). To this end, let us define f(t) ( 10 . 15) = - {G(t, y, u ) dt 0 where, the reader will note, the upper limit of integration is the variable t, not the terminal time T. The derivative of this variable is ( 10 . 16) t= [equation of motion for r 1 - G(t,y, u ) and the initial and terminal values of ( 10 . 17 ) f(O) = - r(t) in the planning period are ta(t,y, u ) dt = 0 0 28 1 CHAPTER 10: OPTIMAL CONTROL WITH CONSTRAINTS and ( 10 . 18) f( T ) = - jTG(t , y , u ) dt = - k 0 [by ( 10 . 14) ] From (10. 18), it is clear that we can replace the given integral constraint by a terminal condition on the r variable. By incorporating r into the problem as a new state variable, we can restate (10. 14) as Maximize jTF(t, y, u ) dt subject to y (10.19) and 0 = f(t, y, u ) f = - G(t, y, u ) y(O) = Yo y( T ) free f( T ) - k f(O) = 0 = (y0, T given) ( k given) This new problem is an unconstrained problem with two state variables, y and r. While the y variable has a vertical terminal line, the new r variable has a fixed terminal point. Inasmuch as this problem is now an uncon­ strained problem, we can work with the Hamiltonian without first expand­ ing it into a Lagrangian function. Note that this procedure of substituting out the constraint can be repeatedly applied to additional integral constraints, with each application resulting in a new state variable for the problem. This is why there is no need to limit the number of integral constraints. Defining the Hamiltonian as ( 10 .20) H = F( t , y , u ) + >.. f(t,y, u ) - p. G(t, y , u ) we have the following conditions from the maximum principle: Max H u aH a>.. aH A = -ay aH f= aiL aH it = - - y= - ( 10.21) ar >.. ( T ) = 0 for all t E [ O, T ] ' [equation of motion for y ] [equation of motion for >.. ] [equation of motion for r ] [equation of motion for IL ] [ transversality condition] What distinguishes (10.21) from the conditions for the usual unconstrained PART 3: OPTIMAL CONTROL THEORY 282 problem is the presence of the pair of equations of motion for r and J.L . Since the r variable i s an artifact whose mission i s only to guide u s to add the -J.LG(t, y, u) term to the Hamiltonian-a mission that has already been accomplished-and whose time path is of no direct interest, we can safely omit its equation of motion from (10.21) at no loss. On the other hand, the equation of motion for J.L does impart a significant piece of information. Since the r variable does not appear in the Hamiltonian, it follows that aH - Jl = - ar- = 0 ( 10.22) J.L(t) = constant This validates our earlier assertion that the costate variable associated with the integral constraint is constant over time. But as long as we remember that the J.L multiplier is a constant, we may omit its equation of motion from (10.21) as well. Inequality Integral Constraint Finally, we consider the case where the integral constraint enters problem as an inequality, say, 1TF(t,y, u) dt y = f(t,y, u) Maximize .( 10.23 ) tile 0 subject to jTG(t,y, u) dt :::;; k 0 y(O) = Yo and y(T) free (k given) (y0, T given) Taking a cue from the isoperimetric problem, we can again dispose of the inequality integral constraint by making a substitution. Define a new state variable r the same as in (10.15): r(t) = - J'G(t,y, u) dt 0 where the upper limit of integration is simply ( 10 .24) t = -G(t, y, u) t (not T). The derivative of [ equation of motion for r ] and its initial and terminal values are ( 10.25) . f(O) = - ta(t,y, u ) dt 0 f(T) = - = (a(t,y, u) dt � 0 0 -k (by ( 10 .23)] r is CHAPTER 10: 283 OPTIMAL CONTROL WITH CONSTRAINTS Using (10.24) and (10.25), we can restate problem (10.23) as Maximize fTF( t, y, u ) dt subject to j = f( t, y , u ) 0 t = - G(t, y, u ) y(O) = Yo y( T ) free f(T) � -k f(O) = 0 ( 10.26) and (y0, T given) (k given) Like the problem in (10.19), this is an unconstrained problem with two state variables. But, unlike (10.19), the new variable r in (10.26) has a truncated vertical terminal line. The Hamiltonian of problem (10.26) is simply H= ( 10.27) F(t,y, u) + A [(t,y, u ) - J.L G(t,y, u ) If the constraint qualification is satisfied, then the maximum principle requires that :. ' }'• Max H u aH . aH [ equation o f motion for A] ay aH f=­ OJ.L aH = - ar >.. ( T ) = 0 J.L ( T ) � 0 iL . , ·· t E [0, T ] [ equation of motion for y] y=o>.. A= for all .. [ equation of motion for r] [ equation of motion for J.L] [ transversality condition for y] f(T ) + k � 0 J.L( T ) [ f( T ) + k ] = 0 [transversality condition for r} Note, again, that because the Hamiltonian i s independent o f f , we have aH P- = - - = 0 ar J.L(t) = constant It becomes clear, therefore, that the multiplier associated with any integral constraint, whether equality or inequality, is constant over time. In the present problem, moreover, we can deduce from the transversality condition 284 PART 3, OPTIMAL CONTROL THEORY that the constant value of JL is nonnegative. But, as in fact omit from (10.28) the equations of motion for r ( 10.21), and JL, we can in as long as we bear in mind that JL is a nonnegative constant. In contrast, the transversal­ ity condition for r -involving complementary slackness-should be re­ tained to reflect the inequality nature of the constraint. In sum, the conditions in (10.28) can be restated without reference to f as follows: Max H for all u y= ii . E [0, T ] aH aA A = - (10.28' ) t ,_ j aH ay ­ ) A( T = 0 : ·-.. . T k - J0 G(t,y, u) dt � O JL [ k - �TG(t,y, u ) dt ] 0 [ ( 10 25 ) ] JL = constant � O by = and . In the preceding discussion, the four types of constraints have been explained separately. But in case they appear simultaneously in a problem, we can still accommodate them by combining the procedures appropriate to each type of constraint present. For every constraint in the form of g(t,y, u) = 0 or g(t,y, u) � c, we append a new 8-multiplier term to the Hamiltonian, which is thereby expanded into a Lagrangian. And for every integral constraint, whether equality or inequality, we introduce a new (suppressible) state variable r, whose costate variable JL -a constant-is reflected in a - JLG(t, y, u) term in the Hamiltonian. If a constraint is an inequality, moreover, there will be a complementary-slackness condition on the multiplier 8 or JL, as the case may be. EXAMPLE 1 The political-business-cycle model (7.61): Maximize ( 10. 29) subject to p = <f>( U } + a7T ir = b(p - 7T) and 7T(O) = 7To 7T( T ) contains an equality constraint p = <f>(U) free + a7T. ( 7T0, T given) But we solved the problem as an unconstrained problem by first substituting out the constraint equa­ tion. Now we show how to deal with it directly as a constrained problem. CHAPTER 10: OPTIMAL CONTROL WITH CONSTRAINTS 285 In this problem, 7T is the state variable; it has an equation of motion. Earlier, since p was substituted out, U was the only control variable. Now that p is retained in the model, however, it ought to be taken as another control variable. Thus the constraint equation p - 4>( U ) - a7T = 0 is in line with the general format of g(t, y, u 1, u 2 ) there is no explicit t argument in it. By (10.3), we can write the Lagrangian ( 10.30) = c in (10.1), although ./ = v( U, p ) e '1 + Ab( p - 7T) + O [ cf>( U ) + a 7T - p] If the following specific functions are adopted: v( U, p) = - U 2 - hp cf>( U ) = j - kU (h > 0) [from (7 .62)] ( j , k > O) [from ( 7 .63)] then the Lagrangian becomes ( 10.30' ) ..;/' = ( - U 2 - hp )er1 + Ab(p - 7T) + O[j - kU + a 7T - p ] Accordingly, the maximum principle calls for the conditions a./ - = - 2 Ue'1 - Ok au a./ - = - he'1 >.b ap ( 10.31) + ( 10.32) = 8 0 = a./ - = j - k U + a7T - p ao a./ 7T = - = b( p - 7T) a>. a./ A = - - = >.b - Oa a7T ( 10 .33) ( 10.34) ( 10.35) [by ( 10. 4 ) ] [ by ( 10.4) ] 0 = 0 [by ( 10.5)] [by ( 10.6)] [by ( 10 .7)] plus a transversality condition. These should, of course, be equivalent to. the conditions derived earlier in Sec. 7.6 by a different approach. To verify the equivalence of the two approaches, let us reproduce here for coml;larison the major conditions from Sec. 7.6: ( 10.36) aH au = ( - 2U + hk )ert - >.bk = 0 [maximizing the Hamiltonian] ( 10.37) 7T = b[j - k U - (1 - a )7T] [state equation of motion] ( 10.38) A = haert + A b(1 - a ) [costate equation of motion] PART 3o OPTIMAL CONTROL THEORY 286 We shall first show that (10.31) and (10.32)-conditions on the two control variables U and p-together ate equivalent to (10.36). Solving (10.32) for 8, we find fJ ( 10.39 ) = - he rt + >.. b - Substituting this into (10.31) results in ad' - = - 2 Ue rt au + hkert - >.. bk = 0 which is identical with (10.36). Next, solving (10.33) for p = j - kU + a rr p, we get which, of course, merely restates the constraint. This enables u s to rewrite (10.34) as 7T = b(j - k U + arr - rr) = b[ j - kU - ( 1 - a ) rr] which is the same as (10.37). Finally, using the () expression in (10.39), we may rewrite (10.35) as A = >.. b + haert - Aba = Ab(1 - a ) + haert which is identical with (10.38). Hence, the two approaches are equivalent. EXAMPLE 2 In Sec. 7.4, Example 3, we encountered a time-optimal problem '(ltJ.40) Maximize t - 1 dt subject to y =y + u o�· �- 1 0 y( T ) y( O) = 5 u(t) E ( - 1, 1 ] and = 11 T free with a constrained control variable. We solved it by resorting to the signum function (7.48) in choosing the optimal control. But it is also possible to view the control set as made up of two inequality constraints ( 10.41 ) -1 :o; u (t) and u ( t) :$ 1 and solve the problem accordingly. Note that the constraint qualification is satisfied here since each constraint function in (10.41) is linear in u . First, w e augment the Hamiltonian o f this problem, H = - 1 + A(y + u) CHAPTER lOo OPTIMAL CONTROL WITH CONSTRAINTS 287 into a Lagrangian by taking into account the two constraints in (10.41). The result is J= ( 10 .42) - 1 + A(y + u ) + 0 1( u + 1) + 02 ( 1 - u ) [by ( 10.9)] To satisfy the maximum principle, we must have: for all t ( 10.43) E [ 0, T ] ( 10.44) ( 10.45) ( 10 .46) ( 10 .47) { · ·· a..I" - = 1 - u >- O ao 2 a..I" :Y = - = y + u aA . A = a..l" - - ay = - 02( 1 - u ) = O A plus a transversality condition. Even though the two equations of motion (10.46) and (10.47) here are derived from ..1" instead of H, they are the same as those derived from H in Sec. 7.4. So nothing more needs to be said here about these. What needs to be verified is that conditions (10.43) through (10.45) would lead us to the same choice of control as the signum function: ( 10.48) u* = 1 if A > 0 u* = - 1 if A < 0 We shall demonstrate here that A > 0 indeed implies u* = 1. Let A > 0. Then, to satisfy (10.43), we must have 01 - 0 2 < 0 or 0 1 < 0 2 . Since both 0 1 and 0 2 are nonnegative, the condition 0 1 < 0 2 implies that 02 > 0. Thus, by the complementary-slackness condition in (10.45), it follows that 1 - u = 0, so the optimal control is u* = 1. This completes the intended demonstra­ tion. The other case (A < 0) is similar, and will be left to the reader as an exercise. Current-Value Hamiltonian and Lagrangian When the constrained problem involves a discount factor, it is possible to use the current-value Hamiltonian He in lieu of H. In that case, the Lagrangian 1 should be replaced by the current-value Lagrangian �- 288 PART 3, OPTIMAL CONTROL THEORY Consider the inequality-constraint problem (10�'l9 ) '� ( Maximize J T<t>(t, y, u ) e-P1 dt subject to - y = f( t, y, u ) 0 g(t , y, u ) � c boundary conditions •. . and The regular Hamiltonian and Lagrangian are ( 10.50) H = <P(t, y , u ) e -pt + A f( t , y, u ) .L' = <P(t, y, u )e-pt + A f( t , y , u ) + 9 ( c - g ( t, y , u ) ] And the maximum principle calls for (assuming interior solution)2: ( 10.51) ( 10 .52) ( 10.53) ( 10 .54) ad' for all t E [0, T ] au ad' 9�0 - = c - g(t ' y u ) > 0 a9 ad' :Y = ­ aA ad' . A = -­ ay -=0 ' ad' 9- = 0 a9 plus an appropriate transversality condition. By introducing new multipliers ( implying A = me-P1) ( implying 9 ne-P1) ( 10.55) = we can introduce the current-value versions of H and ( 10.56) 2 He = HeP1 = <P(t, y , u) + mf( t , y , u ) ..zf as follows: ..£; = .L'eP1 = <P(t, y, u ) + mf( t, y, u ) + n ( c - g( t , y, u ) ] If the control variable is constrained by a nonnegativity requirement, to u�O ad' u- = 0 au [cf. (10.51) must be changed ( 10.12)] CHAPTER 10: OPTIMAL CONTROL WITH CONSTRAINTS 289 Tt can readily be verified that a..£;, au -- = a..z!' Pt e au - an and ao Therefore, conditions (10.51), (10.52), and (10.53) can be equivalently ex­ pressed with ..£;, and the hew multipliers m and n as follows: a..£;, 0 au a..£;, >0 an a..£;, j=am ( 10 .57) - ( 10.58) ( 10.59) = for all t E [0, T ] n ;::: O n a..£;, =0 an - The only major modification required when we use ..£;, i s found in the equation of motion for the costate variable, (10.54). The equivalent new statement, using the new variable m , is m. ( 10.60) = - a..£;, ay - + pm To verify this, first differentiate the A expression in (10.55) with respect to t, to obtain Then differentiate ..z!' with respect to y to get iJ..z!' - - iJy = - <1>y e - pt - A {y + (Jgy Equating these two expressions in line with (10.54), and multiplying through by e P 1, we find that = - <I>Y - mfy + ngY Since the right-hand-side expression is equal to - il.L';,jily [from (10.56)], the rit - pm result in ( 10.60) immediately follows. For problems with integral constraints only, no Lagrangian function is needed, or, to put it differently, the Lagrangian-r-educes to the Hamiltonian. This is because we absorb each integral constraint into the problem via a new state variable-such as f in (10.19) and (10.26). The maximum-princi­ ple conditions can, accordingly, be stated in terms of the Hamiltonian. If we decide to use He in place of H, the procedure outlined in Sec. 8.2 can be applied in a straightforward manner. The major modification is, again, to be found in the costate equation of motion, namely, replacing the condition 290 PART a, OPTIMAL CONTROL THEORY rit -aHcfiiy + p m . With a newly introduced state variable f, the problem will have a new costate variable J.L. Since its equation of motion is jL -iJHjiJf 0, J.L is a constant. A = -iJHjoy by the condition = = r, = Sufficient Conditions The Mangasarian and Arrow sufficient conditions, previously discussed in the context of unconstrained problems, turn out to be valid also for con­ strained problems when the terminal time T is fixed. This would include cases with a fixed terminal point, a vertical terminal line, or a truncated vertical terminal line. Let us use the symbol u to represent the vector of control variables. For problems (10.1) and (10.8), for example, we have u = ( u 1, u 2). As before, let H 0 denote the maximized Hamiltonian-the Hamiltonian evalu­ ated along the u*(t) path. But, in the present context, the. Hamiltonian is understood to be maximized subject to all the constraints of the g(t,y, u) = c form or the g(t, y, u) � c form present in the problem. Besides, since every integral constraint present in the problem is reflected in H via the new costate variable J.L, it must also be similarly reflected in H 0 • For simplicity, we can consolidate the Mangasarian and Arrow suffi­ cient conditions into a single statement3: The maximum-principle condi­ tions are sufficient for the global maximization of the objective functional if { 10.61) either _£' is concave in (y, u ) for all t E [0, T ] ; or H 0 is concave in y for all t E [0, T ] , for given A These conditions are also applicable to infinite-horizon problems, but in the latter case, (10.61) is to be supplemented by a transversality condition { 10.62) lim A { t) [y(t) - y* ( t)J t-oo � 0 [cf. (9.21)] A few comments about (10.61) may be added here. First, as pointed out before, the concavity of d' in (y, u) means concavity in the variables y and u jointly, not separately in y and in u. Second, since H and ../ are ·�.I.�-� ._ For a more comprehensive statement of sufficient conditions, see Ngo Van Long and Neil Vousden, "Optimal Control Theorems," Essay 1 in John D. Pitchford and Stephen J. Turnovsky, eds., Applications of Control Theory to Economic Analysis, North-Holland. Ams­ terdam, 1977, pp. 1 1-34 (especially pp. 25-28, Theorems 6 and 7). 3 ·:.:· · � ' ,i:l i 291 CHAPTER 10: OPTIMAL CONTROL WITH CONSTRAINTS composed of the H = F, {, g, and G functions as follows: F+ A ( - J.LG and ./ = H + 0[ c - g] it is clear that (10.61) will b e satisfied i f the following are simultaneously true: F is concave in (y, u ) A f is concave in (y, u ) J,LG is convex in (y, u ) and O g is convex in (y, u ) for all tE [0, T ] In the case of an inequality integral constraint, however, where J.L is a nonnegative constant [by ( 10.28)], the convexity of J,LG is ensured by the convexity of G itself. Similarly, in the case of an inequality constraint, where 0 2:: 0 [by (10. 1 1)], the convexity of Og is ensured by the convexity of g itself. Finally, if the current-value Hamiltonian and Lagrangian are used, 0 ( 10.61) can be easily adapted by replacing ./ by _£,:, and H 0 by Hc . EXERCISE 10.1 1 In Example 1, p is taken to be an additional control variable. Why is it not taken to be a new state variable? In Example 2, it is demonstrated that if A > 0, then the optimal control is * 1. By analogous reasoning, demonstrate that, if A < 0, then the optimal control is u * - 1. 3 An individual's productive hours per day (24 hours less sleeping and leisure time-normalized to one) can be spent on either work or study. Work results in immediate earnings; study contributes to human capital K(t) (knowledge) and improves future earnings. Let the proportion of productive hours spent on study be denoted by s. The rate of change of human capital, K, is assumed to have a constant elasticity a (0 < a < 1) with respect to sK. Current earnings are determined by the level of human capital multiplied by the time spent on work. (a ) Formulate the individual's problem of work-study decision for a planning period [0, T ] with a discount rate p (0 < p < 1). (b) Name the state and control variables. (c) What boundary conditions are appropriate for the state variable? (d) Is this problem a constrained control problem? Which type of constraints does it have? (e) How does this problem resemble the Dorfman model of Sec. 8.1? How does it differ from the Dorfman model? 2 u = = '. ! PART 3, OPTIMAL CONTROL THEORY 292 4 Consider the problem Maximize subject to jrF(t , y , u ) dt 0 j = f( t , y , u ) g(t,y, u ) � c y ( O) = Yo and l' i. ... ·-· 0 � y( T ) free (y0 , T given) u(t) We can, as we did in (10. 12), apply the Kuhn-Tucker conditions to the nonnegativity restriction on u(t) without introducing a specific multiplier. Alternatively, we may treat the nonnegativity restriction as an additional constraint of the g(t, y, u ) � c type. Write out the maximum-principle conditions under both approaches, and compare the results. [Use 6' do denote the new multiplier for the 0 � u(t) constraint and -t!'' to denote the corresponding new Lagrangian.] 6 In a maximization problem let there be two state variables (y1, y2), two control variables ( u 1 , u 2), one inequality constraint, and one inequality integral constraint. The initial states are fixed, but the ternrinal states are free at a fixed T. (a) Write the problem statement. (b) Define the Hamiltonian and the Lagrangian. (c) List the maximum-principle conditions, assuming interior solutions. ' 10.2 THE DYNAMICS OF A REVENUE· MAXIMIZING FIRM In the Evans model of the dynamic monopolist (Sec. 2.4), the stated objective is to maximize the total profit. This, of course, is nothing extraor­ dinary because profit maximization has long been the accepted hypothesis among economists. A well-known model by Baumol takes the view, however, that instead of maximizing profits, modern firms may actually try to maximize sales revenue, subject to a minimum requirement on the rate of return.4 The primary basis for this view is the separation of ownership and management in the typical corporate form of the firm, where the managers may, in their own interest, seek to maximize sales, despite the stockholders' � William J. Baumol, "On the Theory of Oligopoly," See also his Business Behavior, Value and Growth, Econometrica , August 1958, pp. 187-198. revised edition, Harcourt, Brace & World, New York, 1967. This model is discussed as an example of nonlinear programming in Alpha Chiang, Fundamental Methods of Mathematical Economics, 1984, Sec. 21.6. C. 3d ed., McGraw-Hill, New York, CHAPTER 1 0 : OPTIMAL CONTROL WITH CONSTRAINTS 293 objective of maximizing profit instead. To placate the stockholders, however, the managers must attempt to attain at least a minimum acceptable rate of return. Since the Baumol model is static, its validity has been questioned in the dynamic context. In particular, since profits serve as the vehicle of growth, and growth makes possible greater sales, it would seem that in the dynamic context profit maximization may be a prerequisite for sales maxi­ mization. To address the issue of whether the Baumol proposition retains its validity in a dynamic world, Hayne Leland has developed an optimal control model.5 The Model Consider a firm that produces a single good with a neoclassical production function Q = Q(K, L), which is linearly homogeneous and strictly quasicon­ cave. The usual properties listed below are supposed to hold: ( 10 .63) Q - = 4> ( k ) L 4>(0) = 0 QK = cf>' ( k ) QK > O QL = lf>( k ) - klf>' ( k ) All prices-including the wage rate W and the price of capital goods-are assumed to be constant, with the price of the firm's product normalized to one. Thus the revenue and (gross) profits of the firm are, respectively, R = Q( K, L ) 1r =R - · 1 = Q( K, L ) WL = Q ( K , L ) - WL To satisfy the stockholders, the managers must keep in mind a minimum acceptable rate of return on capital, r0 • This means that the managerial behavior is constrained by the inequality ( 10.64) or Q(K, L ) - WL - r0 K ?:! 0 It remains to specify the decision on investment and capital accumula­ tion. For simplicity, it is postulated that the firm always reinvests a fixed proportion of its profits 1r. Define a as the fraction of profits reinvested, 5 Hayne E. Leland, "The Dynamics of a Revenue Maximizing Firm," Review, June 1972, pp. 376-385. International Economic PART 3, OPTIMAL CONTIIOL � 294 divided by the constant price of capital goods. Then we have K( = l ) = a'7T = a[Q( K, L ) - WL] K(O) = K 0• ' ( 10.65) . with initial capital Drawing together the variou� considerations, we � state the problem of this revenue-maximizing firm as one with an inequs,ijty 1:9nstraint: ·' ·.... .. . · ·' Maximize ( 10.66) subject to and K = a[Q( K, L ) - WL ] WL + r0 K - Q ( K, L) s; 0 K(O) K0 K( T ) free = ( K0 , T given) There are only two variables in this model, K and L. The presence of an equation of motion for K identifies K as a state variable; this leaves L as the sole control variable. . Note that the constraint function WL + r0K - Q(K, L) is convex in the control variable L, and the constraint set does contain a point such that the strict inequality holds (the rate of return exceeds the r0 level). Thus the constraint qualification is satisfied. ·---� ! The Maximum Principle The current-value Hamiltonian of this problem is Hc = Q( K , L ) + m a [ Q(K, L ) - WL} · By augmenting the Hamiltonian with the information in the inequality constraint, we can write the current-value Lagrangian [see (10.56)] as ..£;; = Q(K, L) + ma [ Q( K, L) - WL ] + n [Q( K, L) - WL - r0K ] From (10.57) and (10.58), we then have the following first-order conditions for maxim�ing ..£;; : ( 10.67) ( 10.68) il..£;; iJL = ( 1 + ma + n ) QL - ( m a + n)W = 0 for all t E [O, T ) il..£;; - = Q( K , L) - WL - r0K ;;>: O n ;;>: O iJn and n [ Q(K, L ) - WL - r0K ] = '· 0 To determine·ttte dynamics of the system, we rely on the two equations of motion . K = iJ..£;; iJ m = a(Q( K, L) - WL ] CHAPTD 10: OPTIMAL CON'I'BOL WITH CONSTRAINTS 295 and a.£;, . m = - - + p m = - ( 1 + m a + n ) QK -t nro f pm K a [by ( 1 0 .60 )] Finally, the transversality condition m(T) should be imposed since K ( T ) is free. = 0 Qualitative Analysis Rather than employ a specific production function to obtain a quantitative solution, Leland carries out the analysis of the model qualitatively. Also, instead of working with the original input variables K and L, he uses the properties of the production function in ( 10.63) to convert the maximum­ principle conditions into terms of the capital-labor ratio, k. Accordingly, condition (10.67) is restated as ( 10.69) a.£;, = ( 1 + m a + n ) [ �j� ( k ) - k ljl'( k ) ] - ( m a + n ) W = 0 aL By dividing the first inequality in (10.68) by L write a new version of the condition: ( 10 .70) 1/l ( k ) - W - r0k � 0 * 0 and using ( 10.63), we can n [ 1/1( k ) - W - r0k ] = 0 n�O And the two equations of motion become ' ' ( 10.71) K = a L [ I/I ( k ) - W ] ( 10 .72) m = - ( 1 + rria + n )ljl'( k ) + nr0 + pm The k variable now occupies the center stage except in (10.71), where the original variables K imd L still appear. Certain specific levels of the capital-labor ratio k have special roles to play in the model. Two of these are: k k 0 = the profit-maximizing level of k = the k that yields the rate of return r0 [ k satisfies QL = W or ljl(k) [ k 0 satisfies ; = r0 - k ljl' ( k ) = W-hy ( 10 .63) } or 1/1( k) - W = r0k-by ( 10 .64) ) In Fig. 10.1, we plot the APPL (QjL ) and the MPPL (QL) curves against k. The APPL curve has been encountered previously in Figs. 9. 1 and 9.3a; the MPPL curve should lie below the APPL curve by the vertical distance 296 PART 3: OPTIMAL CONTROL THEORY !{ = L ,P(k ) w . ' ' �-, '.-' FIGURE 10.1 k cp'(k). The intersection of the QL curve and the W line detennines the ." magnitude of k. While the position of k 0 can only be arbitrarily indicated, it is reasonable to expect k 0 to be located to the left of k. Since it is not permissible to choose a k < k0, Leland concentrates on the situation of k > k0, the case where the rate of return exceeds the minimum acceptable level r • The complementary-slackness condition in 0 (10.70) then leads to n = 0. This will simplify (10.69) to ( 10 .73) a.£;, a J L n-O = (1 + ma) [cp( k ) - k cp'( k )] - m aW = 0 This equation can be plotted as a curve in the km space i f we can find an expression for its slope so that we can ascertain its general configuration. Such a slope expression is available by invoking the implicit-function rule. Call the expression in (10.73) between the two equals signs F( m k). Then the slope of the curve is , ( 10 .74) ( ) dm a.£;, aF;ak for 0 BL = = aF;am dk = ( 1 + m a ) k <P"( k ) a ( cp ( k ) - k cp' ( k ) - W ] The following observations may be gathered from this result: (1) The numerator is negative, because cp"(k) < 0, as shown by the curva­ ture of the APPL curve in Fig. 10. 1. (2) For any k < k, the denominator is also negative. This can also be seen in Fig. 10. 1 since the QL curve lies below the W line to the left of k. Thus, for any k E (k0, k), the a.L;,;aL = 0 curve is positively sloped. (3) As k approaches k from the left, the denominator tends to zero, so dmjdk -+ oo. (4) As k approaches zero, the numerator tends to zero, so dmjdk -+ 0. CHAPTER m 1 0: 297 OPTIMAL CONTROL WITH CONSTRAINTS m. = 0 1 /. m I I I I I I I I > 0 . (·: ' a;ec = 0 aL � ... �:· I 1\ ks FIGURE 10.2 k These observations enable us to sketch the iJ..zf',jiJL = 0 curve the way it is shown in Fig. 10.2. By concentrating on k > k 0, which implies n = 0, we also get a simpler version of m from (10. 72). If we set that m equal to zero, then we have the equation ( 10.75) m ln- 0 = - ( 1 + m a ) 4>' ( k ) + p m = 0 which can also be graphed in the km space. To use the implicit-function rule again, let us call the expression in (10. 75) between the two equals signs G(m, k). Then the slope of the m 0 curve is = ( 10 .76) dm - ( for m = 0) dk = iJGjiJk ( 1 + m a ) 4>" ( k ) - --- = -----iJGjiJm - a4>' ( k ) + p While the numerator is negative, the denominator can take either sign. Let k + = the k value that satisfies p = a 4>' ( k ) Then we see, first, that since 4>'(k ) decreases with k (see Fig. 10.1), for any > k + the denominator in (10.76) is positive, so that dmjdk is negative and the m = 0 curve is downward sloping. Second, as k -> k + from the right, the slope of the m = 0 curve tends to infinity. These explain the general shape of the m = 0 curV-e in Fig. 10.2. The two curves intersect at k k '. Thus, k = k ' = the k level that satisfies iJ..zf,jiJL = m = 0, when n = 0 Such an intersection exists if and only if k + occurs to the left of k. The m = 0 curve is where the multiplier m is stationary. What happens to m at points off the m = 0 curve? The answer is provided by the derivative a m. ak = - ( 1 + m a ) 4>" ( k ) > 0 298 PART 3: OPTIMAL CONTROL THEORY The message here is that as k increases, m will increase, too . Consequently, we have m < 0 to the left of the m = 0 curve, and m > 0 to its right. This is why we have made the arrowheads on the a.L;,;aL = 0 curve point southward to the left, and northward to the right, of the m 0 curve. From Fig. 10.2, it should be clear that the stationary point formed by the intersection of the two curves-may not occur at k , the profit-maximizing level of k . That is, placing the revenue-maximizing firm in the dynamic context does not force it to abandon its revenue orientation in favor of the profit orientation. In fact, Leland shows by further analysis that there is an eventual tendency for the revenue-maximizing firm to gravitate toward k 0, where the rate of return is at the minimum acceptable level r0• The only circumstance under which the firm would settle at the profit-maximizing level k is when k + exceeds k . The reader may have noticed that condition (10.7 1 ) has played no role in the preceding analysis. This is because the analysis has been couched in terms of the capital-labor ratio k, whereas (10.71) involves K and L as well as k . A simple transformation of (10.71) will reveal what function it can perform in the model. Since k "" K/L, we have K = kL, so that = K = kL + ki Equating this with the K expression in ( 10.71) and rearranging, that t a [ cf> ( k ) k L we find j't . W] - k Once we have found an optimal time path for k , then this equation can yield a corresponding optimal path for the control variable L. 10.3 STATE-SPACE CONSTRAINTS The second category of constraints consists of those in which no control variables appear. What such constraints do is to place restrictions on the state space, and demarcate the permissible area of movement for the variable y. A simple example of this type of constraints relevant to many economic problems is the nonnegativity restriction y(t) � 0 or -y( t ) � 0 for all t E [0, T J But more generally, the constraint may take the form h(t,y) �c for all t E [0, T ] In either case, the control variable u is absent in the constraint function. By coincidence, it may happen that when we ignore the state-space constraint and solve the �ven problem as an unconstrained one, the optimal path y*(t) lies entirely in the permissible area. In that event, the constraint CHAPTER 10: OPTIMAL CONTROL W1'l'H C01IIliiTRAll'ITS 299 y y y*(t ) 0 0 �-------��/��--/' ' , Unconstrained solution (a) iT 1 '1 r I y•co '\-�- / ' ' ' - - - _... 1 I I I 7--..L---/"-. T . t Unconstrained solution (b) FIGURE 10.3 is trivial. With a meaningful constraint, however, we would expect the unconstrained optimal path to violate the constraint, such as the broken curve in Fig. 10.3a which fails the nonnegativity restriction. The true optimal solution, exemplified by the solid curve, can contain one or more segments lying on the constraint boundary itself. That is, there may be one or more "constrained intervals" (or " blocked intervals") within [0, T ] when the constraint is binding (effective), with h (t, y) c. Sometimes the true optimal path may have some segment(s) in common with the unconstrained solution path; but the two paths can also be altogether different, such as illustrated in Fig. 10.3b. Although we cannot in general expect the unconstrained solution to work, it is not a bad idea to try it anyway. Should that solution turn out to satisfy the constraint, then the problem would be solved. Even if not, useful clues will usually emerge regarding the nature of the true solution. = Dealing with State-Space Constraints Let the problem be ( 10. 77) Maximize JTF( t, y, u ) dt subject to y = f( t, y, u ) ... 0 h(t, y ) _ ,_ _ _ ,_ and $ c . ·· s . boundary conditions Instinctively, we may expect the method of Sec. 10. 1 to apply to this . problem. If so, we have the Lagrangian ( 10 .78) ./ = F(t, y , u) + A f( t , y , u) + 8 [ c - h ( t , y)] 300 PART 3, 0PT� CONTROL THEORY with maximum-principle conditions (assuming interior solution) aJ - = F + A {, = O au u u - j)_/ -ao = c - h(t, y ) � o j)_/ y = aA = f(t, y , u) -, j)_/ '1· -: '' aJ 6- = 0 a6 6�0 . ·· ·.-., . ay plus a transversality condition if needed Why cannot these conditions be used as we did before? For one thing, in previously discussed constrained problems, the solu­ tion is predicated upon the continuity of the y and A variables, so that only the control variables are allowed to jump (e.g., bang-bang). But here, with pure state constraints, the costate variable A can also experience jumps at the junction points where the constraint h(t, y) $ c turns from inactive (nonbinding) to active (binding) status, or vice versa. Specifically, if -r is a junction point between an unconstrained interval and a constrained interval (or the other way around), and if we denote by A -(-r) and A +(-r) the value of A just before and just after the jump, respectively, then the jump condition is6 ( 10.80) ( b � 0) Since the value of b is indeterminate, however, this condition can only help in ascertaining the direction of the jump. Note that it is possible for b to be zero, which means that A may not be discontinuous at a junction point. This latter situation can occur when the constraint function h( t, y) has a sharp point at -r, that is, when h is discontinuous at -r. At any rate, the solution procedure requires modification. We now need to watch out for junction points, and make sure not to step into the forbidden part of the state space. An Alternative Approach While it is possible to proceed on the basis of the conditions in (10.79), it would be easier to take an alternative approach in which the change in the status of the constraint h(t, y) s c at junction points is taken into account 6 For a more detailed cliscussion of jump conditions, see Atle Seierstad and Knut Sydsreter, Optimal Control Theory with Economic Applications, Elsevier, New York, 1987, Chap. 5, especially pp. 317-319. 301 CHAPTER 10, OPTIMAL CONTROL WITH CONSTRAINTS in a more explicit way.7 The major consideration of this latter approach is that since h(t, y) is not allowed to exceed c, then whenever h(t, y) = c (the constraint becomes binding) we must forbid h(t, y) to increase. This can be accomplished simply by imposing the condition dh whenever h ( t, y) dt Note that the derivative dhjdt (a total derivative), unlike h itself, is a function of t and y, as well as u , because d ah ah dy . dt h(t, y ) = dt = h1 + h y f( t,y, u) = h ( t , y, u ) at + ay Therefore, the new constraint dhjdt ::; 0 whenever h(t,y) or, more -<0 = c = c, explicitly, ( 10.81) h (t,y, u ) = h 1 + h J( t, y, u ) ::; 0 whenever h (t,y) fits nicely into the g(t, y, u) ::; category discussed in Sec. 10.1. The only difference is that (10.81) is not meant for all t E [0, T ], but needs to be put in force only when h(t, y) = = c c c. With this new constraint, the problem statement now takes the form Maximize J TF(t, y, u ) dt subject to :Y = f( t , y, u ) h( t, y, u ) = h 1 + h y {( t, y, u ) ::; 0 whenever h(t, y) ( 10.82) 0 = c and boundary conditions In order to make possible a comparison of the new maximum-principle conditions with those in (10.79), we shall adopt distinct multiplier symbols A. and 0 here. Let the Lagrangian be written as ( 10 .83) ..£'' = F(t,y, u ) + A {( t, y , u ) - 0h Then, as part of the maximum-principle conditions, we require (assuming that the' constraint qualification is satisfied) that a.L"' = + - 0h �" = 0 yI au F A f u 7See Magnus R. Hestenes, Calculus York, 1 966, Chap. 8, Theorem 2 . 1 . u u of Variations and Optimal Control Theory, Wiley, New 302 PART 3: OPTIMAL CONTROL THEORY and a1'' ae 0- = 0 While the set of conditions on 0 seems to serve the purpose of restating the constraint, it does not make clear that this set applies only when h(t, y) = c. To remedy this, we append the complementary-slackness condition h ( t , y) � c e[c - h (t,y)] = 0 Then, h(t, y) < c (constraint not binding) would mean 0 = 0, which would cause the last term in _/' to drop out, and thereby nullify the conditions regarding a1'';ae. Conversely, when h(t, y) = 0 (constraint binding), we intend the complementary-slackness condition to imply 0 > 0. Thus this is a stronger form of complementary slackness. [In the normal interpretation of complementary slackness, h(t, y) = 0 is consistent with 0 = 0 as well as e > o .J In addition to the preceding, the maximum principle for the present approach also places a restriction on the way the 0 multiplier changes over time: At points where 0 is differentiable, 0 must be nonpositive whenever h(t, y) c . This restriction will be explained later. Collecting all the results together, we have (assuming interior solution) the following set of conditions: = a1'' au = Fu + A fu - ® h y fu = 0 . . a1'' . = - h = - [ h 1 + h y {( t, y, u ) j :2: 0 ae e[c - h (t,y)] = 0 h ( t, y ) � c ( 10.84) "' . \, 0 0 [ = 0 when h ( t, y) < a1'' y = aA = f( t, y, u ) c] r�'f· � t� a1'' A = - ----ay = - FY - A {y + 8 [ h y1 + h y {y + h yy f j . : ..,_ � a1'' ae 8 - == 0 plus a transversality condition if needed Note that, after deriving the h expression from the constraint function h(t, y ), we can directly apply the conditions in (10.84) without first trans­ forming problem (10. 77) into the form of ( 10.82). Here, as in (10.79), we assume that the maximization of the La­ grangian with respect to· u yields an interior solution. If the control variable is itself subject to a nonnegativity restriction u (t ) � 0, then the a../'jau = 0 CHAPTER 10: OPTIMAL CONTROL WITH CONSTRAINTS 303 condition should be replaced by the Kuhn-Tucker conditions a../' -- < 0 au - u 2: a../' u -- 0 = au 0 These conditions allow for the possibility of a boundary solution. Boundary solutions can also occur, of course, if there is a closed control region for u . A distinguishing feature of this approach is that the f(t, y , u) function enters into the Lagrangian ../' twice, with two different multipliers, A (a costate variable) and e (a Lagrange multiplier for a constrained problem). Accordingly ' the partial derivative f" also appears twice in the a..L" I au = 0 condition-the first line in (10.84)-once with A and once with 0. In contrast, the o../j ou = 0 condition in ( 10.79) does not contain the (} multi­ plier. Thus, under the new approach, the behavior of 0 and its effects on the system are more explicitly depicted. When the state-space constraint is nonbinding, with h(t, y ) < c, e takes a zero value, and (10.84) reduces to the regular maximum-principle conditions. But when the constraint changes its status from nonbinding (inactive) to binding (active), the conditions in (10 .84) will become fully operative, prescribing how the 0 multiplier affects the system, and how 0 itself must change over time. While the approach summarized in (10.84), with more explicit informa­ tion about junction points, is easier to use than the approach in (10. 79), the two approaches are in fact equivalent. The reader is asked to demonstrate their equivalence by following the set of steps outlined in Exercise 10.3, Prob. 1. In the process, it will become clear why the 0 � 0 condition is needed. In another exercise problem, the reader is asked to write out the maximum-principle conditions for the special case of nonnegativity con­ straint y(t) 2:: 0. An Example Let us consider the problem8 ' �:.- Maximize f\4 - t ) u � .. subject to j= u ( 10.85) 0 y - t�1 y(O) and U E = 0 ( 0, 2 ] 8 This problem is given as an exercise in Atle � provide and discuss here its complete solution. . · .< . ,.. 8114 :IC.mat�;'t'lp. eit. P- 329. We 304 PART a, OPTIMAL CONTROL THEORY y y=t+ 1 3 2 ,, ,, 'i <J, . ,. 0 2 which contains a state-space constraint, constraint and form the Hamiltonian y - t ;s; 1. If we disregard this H = (4 - t) u + Au we see that" H is linear in u , with slope aH -=4-t+A au Given the closed control region [0, 2], we may expect a boundary solution to arise. Specifically, from the integrand function we gather that in order to maximize the objective functional, we should let u be as high as possible. Thus a reasonable conjecture about the unconstrained solution is u(t) = 2. This would imply that y = u = 2 and y(t) = 2t + k. After using the initial condition y(O) = 0, we obtain the simple linear path y(t) = 2t This path is the one that allows the fastest rise in y, because it corresponds to the highest value of u , and hence the highest value of y . As Fig. 10.4 shows, however, this path, steadily rising from the point of origin, can stay below the constraint boundary y = t + 1 only up to the point (1, 2), that is, up to t = 1, at which time the constraint becomes binding. Then we have to change course. Let us now turn to (10.84) for guidance as to what new course to take. First, the state-space constraint y t ;s; 1 implies that - h(t,y) =" y - t c=1 h =y - 1 = u - 1 CHAPTER 1 0: OPTIMAL CONTROL WITH CONSTRAINTS 305 Since the constraint function is linear in u (it does not oontain constraint qualification is satisfied. The Lagrangian is, by �10.13� u ), the ../' = ( 4 - t) u + Au - 0( u - 1) which i s linear i n u . The maximum principle requires, by (10.84), that ../' be maximized with respect to a..I'' u [corner solution] -- = 1 - u >O 0( 1 - u ) 0 0�0 a0 y - t 5: 1 0(1 - y + t) = 0 0 5: 0 ( = 0 when constraint nonbinding) y=u = = A = constant No transversality condition is needed because the endpoints are fixed. Taking the clue from the earlier-discussed unconstrained solution, we initially adopt for the first time interval [0, 1) the control u = 2, which implies the state path y = 2t. Thus, we have u*[O, 1) = 2 ( 10.86) the y*[0, 1 ) = 2t As soon as we hit the constraint boundary, condition tells us to set a../'ja0 u=1 0 becomes positive, and [by complementary slackness] This means that the equation of motion becomes y = 1, and yields the path To definitize the arbitrary constant kl> we reason from the continuity of y that this second segment must start from the point (1, 2) where the first segment ends. This fact enables us to definitize the path to y(t) = t + k1• y(t) = t + 1 [same as the constraint boundary] Thus, as shown in Fig. 10.4, the y path now begins to crawl along the boundary. But this path obviously cannot take us to the destination point (3, 3), so we need at least one more new segment in order to complete the journey. To determine the third segment, we first try to ascertain the proper control, and then use the equation y = u. For the first segment, we chose u* 2 to ensure the fastest rise in y, and adhered to that u value for as long as we could, until the boundary constraint forced us to change course. The constraint boundary later be­ comes the new best path, and we should again stay on that path for as long = 306 PART 3, QPT�L CONTROL THEORY as feasible, until we are forced by the consideration of the destination location to veer off toward the point (3, 3). Since u( y) cannot be negative, it is not feasible to overshoot the y = 3 level and then back down. So the second segment should aim at y = 3 as its target, terminating at the point (2, 3). Thus we have = u* [ 1 , 2) ( 10.87) slope = 1 y* [ 1 , 2) = t+ 1 It then follows that the third segment is a horizontal str,right line with . u = 0: u*[2, 3] ( 10.88) = 0 y* [2, 3 ] = 3 By piecing together (10.86), (10. 88), and ( 10.89), we finally obtain the complete picture of the optimal control and state paths. In the present example, the first segment of the optimal path coincides with that of the unconstrained problem. But this is not always the case. In problems where nonlinearity exists, the true optimal path may have no shared segment with the unconstrained path. This type of outcome is illustrated in Fig. 10.3b.9 EXERCISE 1 2 10.3 Establish the equivalence of (10.79) and (10.84) by the following procedure: 0), and solve for A. (b) Differentiate A totally with respect to t, to get an expression for A. ( c) Expand the expression obtained in part (b) by using the A expression in (10.79). (d) Substitute the result from part (a) into the A expression in (10.84). (e) Equate the two A expressions in parts (c) and (d), then simplify, and deduce that e � 0. (a) Equate iJ.L'fiJu and iJ./'jiJu (both = (a) Write out the maximum-principle conditions under each of the two approaches when the state-space constraint is y(t) � 0. (b) In a problem with y(t) � 0, assume that the terminal time T is fixed. Is there need for a transversality condition? If so, what kind of · transversality condition is appropriate? I. Kamien and Nancy 'L� �i Dynamic �::n.e Calculus of Variations and Optimal Control in ECOitOfrliu and Management, lkl ed., Elaeviar, New York, 1991, pp. 231-234. ,· . 9For an example, see Morton 307 CHAPTER 10: OPTIMAL CONTROL WITH CONSTRAINTS 10.4 ECONOMIC EXAMPLES OF STATE-SPACE CONSTRAINTS In this section, we present two economic examples in which a nonnegativity state-space constraint appears. Inventory and Production The first example is concerned with a firm's decision regarding inventory and production.1 0 To begin with, let the demand for the firm's product be given generally as D(t) > 0. To meet this demand, the firm can either draw down its inventory, X, or produce the demanded quantity as current output, Q, or use a combination thereof. Production cost per unit is c, and storage cost for inventory is s per unit, both assumed to be constant over time. The firm's objective is to minimize the total cost f0T(cQ + sX) dt over a given period of time [0, T ]. It is obvious that output cannot be negative, and the same is true of inventory. Thus, Q(t) ;;:: 0 and X(t) ;;:: 0. These two variables are related to each other in that inventory accumulation (decumulation) occurs whenever output exceeds (falls short oO the demanded quantity. Thus, X( t) = Q(t) - D(t) This suggests that Q can be taken a s a control variable that drives the state variable X. Assuming a given initial inventory X0 , we can express the problem as Maximize jT( - cQ - sX ) dt subject to X = Q - D(t) ( 10.89) 0 -X � X(O) and 0 = X0 Q E [0, oo) X(T) <!: . o free ( X0, T given) Note that minimization of total cost has been tranaJated into maximization 10 ,.,;_,� _ to Applied Optimal Control, This problem is discussed in Greg Knowles, An l . ' ,. ·, Academic, New York, 1981, pp. 135-137. 308 PART 3: 0PTUMAL CONTROL THEORY of the negative of total cost. Note also that the nonnegativity state-space constraint automatically necessitates the truncation of the vertical terminal line. = The unconstrained view of this problem shows that the Hamiltonian H - cQ :_ sX + A(Q - D) i s linear i n the control variable Q, with - = -c aH aQ The rule for the choice of +A Q is therefore Q* {= 0 is indeterminate is unbounded } Q) is not feasible, whereas the second Q) is not helpful. Observe, however, that even in the case of A we can still select Q* 0, as in the case where A In fact, Q* 0 would make a lot of sense since Q enters into the objective functional negatively, so that choosing the minimum admissible value of Q = c, < c. The third possibility (unbounded possibility (indeterminate = = serves the purpose of the problem best. However, then the equation of motion becomes X = -D(t) [when < 0 Q = 0) and the firm's inventory is bound to be exhausted sooner or later. When the = 0, it must reorient the control. firm runs into the constraint boundary Before applying conditions (10.84), we first verify that the constraint X qualification is satisfied. This is indeed so, because the constraint function -X is linear in Q (it contains no Q). From the constraint -X ;:<;; 0, we readily find that h = -X c = O =c and h = -X = - (Q - D) Thus, by (10.83), the Lagrangian is d'' - Q - sX + A(Q - D) + 0(Q - D) which i s linear in the control variable Q. The conditions i n (10.84) require CHAPTER 10: OPTIMAL CONTROL WITH CONSTRAINTS 309 that ../'' be maximized with respect to a../'' ae Q [comer solution] 8(Q - D ) = 0 -=Q-D>O 0X = 0 [ 0 when X > 0] = a../'' =Q -D aA B../'' A = - -- = s ax A( T ) ;;;: 0 X( T ) ;;;: 0 X= A( T ) X( T ) = 0 Acting on the clue from the unconstrained view of the problem, first choose we Q=O which simplifies the state equation of motion to X= -D This result indicates that the firm should produce nothing and meet the demand exclusively by inventory decumulation. While following this rule, the inventory existing at any time will be But such a policy can continue only up to the time t = r, when the given inventory X0 is exhausted. The value of -r, the junction point, can be found from the equation {0 n(t) dt = X0 [exhaustion of inventory] If we assume that -r < T, then at t = r we need to revise the zero-produc­ tion policy. The rest of the story is quite simple. Having no inventory left, the firm must start producing. In terms of the maximum principle, the activation of the constraint makes the 8 multiplier positive, so that a../'';ae 0, or = Q=D for t E [ -r, T ] PART 3, 0PTU4AL CONTROLTHEORY 310 Under this new rule,. the firm should meet the entire demand from current production. Inasmuch as X = Q - D, it follows that X= O for t E [ 'T, T ] meaning that no change in invento-ry should be contemplated. Since 0, inventory should remain at the zero level from the point complete optimal control and state paths are therefore ( 10.90) Q*(O, T) = 0 Q* (T, T ] = D( t) 'T X(T ) = on. The X*[O, T) = X0 - fD(t) dt 0 X* ( T, T ] = 0 Capital Accumulation under Financial Constraint William Schworm has analyzed a firm that has no borrowing facilities, and whose investment must be financed wholly by its own retained earnings.U This model can serve as another illustration of a state-space constrained problem. The firm is assumed to have gross profit expenditures J(t). 7T(t, K ) and investment J(t) � 0. Besides, it It cannot sell used capital. Hence, cannot borrow funds. Thus its cash flow c/l(t) is dependent only on the profit proceeds and the investment outlay: cJ!(t) = 7T(t, K ) I t is the .,.I of the firm t o maximize - I(t) its present [0 'c/1( t)e-P1 dt where the symbol value . p denotes the rate of return the stockholders can earn on alternative investments as well as the rate of return the firm can earn on its retained earnings. R(t) p) on R, and by Starting from a given initial level, the firm's retained earnings can be augmented only by any returns received (at the rate any positive cash flow. Consequently, we have R (t) = pR(t) + c/l (t) = pR(t) + 7T(t, K) - l( t) 11William E. Schworm, " Financial Constraints and Capital Accumulation," International Economic Reu�w, October 1980, pp. 643-660. \ Assuming no d �preciation, we also have CHAPTER 10: OPTI CONTROL W1TH CONSTRAINTS K(t) = 311 I(t) These two differential equations suggest that R and K can serve as state variables of the model, whereas I can play the role of the control variable. R and K must, of course, be nonnegative. But whereas automatically takes care of the nonnegativity of the assumption of I(t) K, the nonnegativity of retained earnings, R, needs to be explicitly pre­ scribed in the problem. That is why the problem features a state-space Both state variables �0 constraint. Collecting all the elements mentioned above, we can state the problem as follows: Maximize ([ rr ( t, K ) - I( t)] e-P1 dt subject to R(t) = pR(t) + rr( t, K) - l(t) K( t) = I( t) ( 10.91) 0 � -R(t) R(O) and I( t) = E 0 R0 [0, K(O) = K0 oo) Being given in the general-function form, this model can only be analyzed qualitatively. Actually, Schworm first transforms the model into one where R(t) is a constrained control (rather than state) variable. But we shall treat it as one with a state-space constraint and derive some of Schworm's main results by using the maximum-principle conditions in (10.84). Again, we can verify that the constraint qualification is satisfied, because the constraint function, - R(t), is linear in the control variable I. The constraint function also supplies the information that h = -R h = -R = - ( pR + 7r(t, K ) - I ] Thus, by (10.83), we have the Lagrangian ...£'' = [ rr(t , K ) - I ] e -pt + A n ( p R "•'.•, : . " + rr(t , K ) - I ] + A K I + E> ( p R + rr(t, K ) - I] • • ; where AR and A K are the costate variables for R and x; reepective)y. The 312 PART 3: OPTIMAL CONTROL THEORY conditions in ( 10.92) ( 10.93) ( 10.94) { 10.95) (10.84) stipulate that iJ.£" -= - e -P1 - A. R + A. X - 0 -< 0 iJI iJ./' iJ0 = pR + 1r( t, K ) - I � 0 R(t) � 0 0R(t) = 0 . ( = 0 when R > 0) iJ ./' . R = iJ = pR + 1r(t, K ) - I A. R iJ./' K= iJA.x = I iJ./' . A. R = - iJ = -p( A. R + 0) R . ()..£'' I�O iJ./' IiJ I = 0 iJ./' 0- = 0 a0 ·.· i i Ax = - iJ = - TTx(e-P' + A. R + 8) K plus transversality conditions Note that, by virtue of the nonnegativity restriction on the control variable I, the Kuhn-Tucker conditions are applied in (10.92). While Schwarm discusses both the cases of I(t) > 0 and I(t) = 0, we shall consider here only the optimization behavior for the case of I(t) > 0. With I positive, complementary slackness mandates that iJ./ 'jiJ I = 0, or [ by ( 10.92)] Differentiating this equation with respect to Ax = -pe-p1 + A R + 0 t, we get = -pe-P' - p( A. R + 0) + 0 [by ( 10.94)] = -p ( e -pt + A. R + 0) + 0 Moreover, by equating this Ax expression with (10.95), we find that ( 10.96 ) This result embodies the optimization rule for the firm when I > 0. It is of interest that, except for the 0 term on the right, the two sides of (10.96) are identical in structure. From (10.93) we know that 0 = 0 when R > 0. We can thus conclude from (10.96) that whenever R > 0, the firm 313 CHAPTER 10: OPTIMAL CONTROL WITH CONSTRAINTS should see to it that ( 10 .97) 7rx = P [investment rule when R > OJ That is, when retained earnings are positive (financial constraint nonbind­ ing), the firm should simply follow the usual myopic rule that the marginal profitability of capital be equal to the market rate of return. If, on the other hand, R = 0 (financial constraint binding) in some time interval (t1, t 2 ), then it follows that R 0 and R 0 in that interval. Hence, the R(t) equation in (10.91) will reduce to = ( 10 .98) l(t) = 1r(t, K ) [investment rule when R = = R = 0] This means that in a constrained interval, the firm should invest its current profit. We see, therefore, that when the R constraint is binding, investment becomes further constrained by the availability of current profit. Rules (10.97) and (10.98), though separate, are to be used in combina­ tion, with the firm switching from one rule to the other as the status of the R constraint changes. By so doing, the firm will be able to get as close as possible to the unconstrained optimal capital path. Schworm also derives other analytical conclusions. For those, the reader is referred to the original paper. EXERCISE 10.4 1 2 Draw appropriate graphs to depict the Q* and X* paths given in In problem (10.89), the demand is generally expressed different specifications of D(t) affect the following? as (10.90). D(t). Would (a) r (b) Q*[O, r) (c) Q*[r, TJ (d) X*[O, r) (e) 10.5 X*[r, T) LIMITATIONS OF DYNAMIC OPTIMIZATION While by no means exhaustive, the present volume has attempted to intro­ duce-in a readable way, we hope-most of the basic topics in the calculus of variations and optimal control theory. The reader has undoubtedly noticed that even in fairly simple problems, the solution and analysis procedure may be quite lengthy and tedious. It is for this reason that simple specific functions are often invoked in economic models to render the solution more tractable, even though such specific functions may not be totally satisfactory from an economic point of view. 314 PART s, OPTIMAL CONTROL THEORY It is for the same reason that writers often assume that the parame­ ters in the problem, including the discount rate, remain constant through­ out the planning period. Although there do exist in real life some economic parameters that are remarkably stable over time, certainly not all of them are. The constancy assumption becomes especially problematic in infinite­ horizon problems, where the parameters are supposed to remain at the same levels from here to eternity. Yet the cost-in terms of analytical complexity-of relaxing this and other simplifying assumptions can be extremely high. Thus we have here a real dilemma. Of course, in practical applications involving specific parameter values, the simple way out of the dilemma is to reformulate the problem whenever there occur significant changes in parameter values. But the constancy assumption may have to be retained in the new formulation. Owing to the complexity of multiple differential equations, economic models also tend to limit the number of variables to be considered. With the advent of powerful computer programs for solving mathematical problems, though, this limitation may be largely overcome in due time. After the reader has spent so much time and effort to master the various facets of the dynamic-optimization tool, we really ought not to end on a negative note. So by all means go ahead and have fun playing with Euler equations·, Hamiltonians, transversality conditions, and phase dia­ grams to your heart's content. But do please bear in mind what they can and cannot do for you. ·.t ANSWERS TO SELECTED EXERCISE PROBLEMS _,' ,, 1.2 EXERCISE 3 The terminal curve should be upward sloping. 1.3 EXERCISE 1 F[t, y(t), y'(t)] 2 (b) and (d) = 1 1.4 EXERCISE V*(D) 8; optimal path DZ i s DGIZ. V*(E) 8; optimal path EZ is EHJZ. V*(F) 10; optimal path FZ is FHJZ. V*(K) 4 V*(I) = 3; IZ V*(J) 1; JZ V*(A) 23; ACFHKZ 1 = = = = = = 2; �. 2.1 EXERCISE 2 dljdx 4x3(b - a) 4 dljdx 2e2x 6 y*(t) -l.t3 + �t + 1 = ,, · - . = 8 y*(t) = = e' + e-• + !te' 315 316 ANSWERS TO SELECTED EXERCISE PROBLEMS 2.2 EXERCISE y*(t) 2t 3 y*(t) = !t 2 + it + 2 5 y*(t) e< I + t)/2 + e <I - t)/2 1 = = 2.3 EXERCISE 1 y*(t) = t 3 y* (t) z* (t) = 4 e ""/2 - e .-;2 ----, ;;;-;;;­ 2.4 EXERCISE 1 - a + 2aab + {3b P = P, = static monopoly price O The lowest point of the price curve occurs at P. = 2b(l + ab) t0 = > (In A 2 - In A 1 )j2r � 0 Only case (c) can possibly (but not necessarily) involve a price reversal the time interval [0, T ]. The other cases have rising price paths. 2.5 EXERCISE 2 3 (a) in Not different. (b) A1 � 0, A 2 > 0 3.2 EXERCISE 1 (a) y*(t) = 4 3 (a) y*(t) = t + 4, T* EXERCISE =1 3.3 1 y*(t) !<t 2 - 3t + 1) 2" (a) No. (b) y*(t) = l t2 - tt + 1 3 (a) Only one condition is needed, to determine T*. = EXERCISE 1 (a) 3.4 'fT'(Lr) 2p/bk > 0. Thus Lr* is located such that the slope of the 'fT(L) curve is 2p/bk, on the positively sloped segment. = ' 1 317 ANSWERS TO SELECTED EXERCISE PROBLEMS 2 (b) An increase in p (or b, or m (a) L/ = n - !!_ /bk n (b) 7T'( L) = 0 when L k) pushes the location of L/ to the left. m = n - ( c) L T* is located to the left of mjn, such that the 7T( L) curve has a positive slope. EXERCISE 4.2 1 3 (a) There is no y term in F, and F is strictly convex in y'. (c ) Sufficient for a unique minimum. (a) The determinantal test (4.9) fails because ID21 = 0. (b) The determinantal test (4.12) is satisfied for positive semidefiniteness because I D 1 1 = 8 and 2, and ID21 = 0. The characteristic roots are r1 10 and r2 = 0. (c) Sufficient for a minimum. = EXERCISE 4.3 1 Problem 6: Fy'y' = 4 for all t; satisfies the Legendre condition for a minimum. Problem 9: Fy'y' = 0 for all t; satisfies the Legendre condition for a maximum as well as for a minimum. 3 F.,.. .,.• = 2 ( ;2;: ) 1 2 e -pt > 0 for all t; satisfies the Legendre condition for a minimum. EXERCISE 5.2 1 K * (t ) 3 Cg•g•K" - pCg• + 'TI'g 4 K * (t) = = ( K0 - 25 + p) exp ( = (K0 - 25) exp 0 ( P - P - �) t �) t + 25 - p + 25 EXERCISE 5.3 1 (a) (b) C * ( t) K * ' (t = A - 1/(b + 1 )e rt/(b+ 1) )= B - U+ (C = C * 0 e rt/Cb+ ( C * ) - bjb * ) - <b + U � 1 -C* b 1) [since B = U) 3UJ r (c) K*( t) = __ K e r t /(b+ !) b+ 1 ° rtf(b + IJ (t) = K d) ( K* oe EXERCISE 2 5.4 ( a ) No, (5.40) is valid with or without saturation. (b) Equation (5.41) is unaffected, but (5.42) should be changed to Q'(K) = 0 because now IL can never be zero. 3 The new equilibrium is still a saddle point, but it involves a positive (rather 5 than zero) marginal utility in equilibrium. (a) Only the streamlines lying above the stable branch remain relevant; the others cannot take us to the new level of K T· (c) Yes. EXERCISE 6.1 A 2 3 y*(t) = - 4 t 4 y*(t) 5 = + c1t + c2 A1e1 + A2e-1 + c1 The general solution is the same as in Example 2. EXERCISE 6.2 1 5'= B - U(C) + D(L) + A[ - C + Q(K, L) - K'] - U'(C) - A = 0 A = - U'(C) = - J.L 2 (a) .J;,:;i �-·,·;,_ Three variables and two constraints. (b) 5'= (1T - C)e-pt + A1[ - 1T (c) For 1r: e -pt - A1 = 0 For C: - e -P1 - A2 = 0 d + a K - {3K 2 ] + A 2[ - C + aK'2 + bK' ] = = A1 = e -pt A 2 = - e-P' For K: A 1(a - 2{3K) - d [A2(2aK' + b)] -"" 0 t dA 2 A1(a - 2{3K) - dt(2aK' + b) - A 2(2aK") = 0 = EXERCISE 6.3 2 4 Similarly to (6.10), A may now be condition will come into play. rQ • = g - Ep rQm - rQ • + rE E- 1 > <-: 0. A complementary-slackness rQ • [ E > 1 for positive MR] AHSWERS TO SELECTED EXERCISE PROBLEMS 6 (b) 319 {'N(q, q')e -P' dt Maximize 0 f0ooq' dt = So subject to EXERCISE 7.2 2 A*(t) = 3 e4 - 1 - 3 u* ( t) = 2 4 u*(t) = A*(t) = A 1 e li• y *(t) = 7e1 - 2 + A2e- fi• y* (t) = ( /2 - 1)A 1 e .f2• - (/2 + 1)A2 e - fit where A1 = ' ,. A2 = - e - 2/i r.; ( 1 - /2 ) e - 2V2 - ( /2 + 1 ) (1 - 1 --:= - - /2 ) e - 2 12 - ( /2 + 1 ) EXERCISE 7.4 A*(t) = 2 u*(t) = 1 y*(t) = t + 4 u* = - 1 A* = - ! T* y *(t) = - 2t + 8 4 u* = Jl/38 y*(t) = Jl/38 t A* = Jl/39 1 3 = 4 ( EXERCISE 7.6 ( ) 1 (a) The cyclical pattern will be eliminated. 3 - a or dU* -- dt 1 = - (T - t)-khbae8<T-o < 0 for all t (except t = T ) 2 EXERCISE 7. 7 , 1 (a) Yes. 3 (a) H = U[C(E), P(E))e -P1 - A E (b) ' ---\. No, E* 2 is characterized by " marginal utility > ma.rgiDal di8utilit;y." aH ·· ::: aE = [ UcC'(E) + UpP'( E ))e -pt - A = 0 (b) The A path is still a constant (zero) path. E*(t) E * as in (7.81). (d) The condition in part (a ) becomes (c) = · · . · ;· . ANSWERS TO SELECTED EXERCISE PROBLEMS 320 so dE - dt = -a'lt ;at --- a'lt;aE = peeP' -;;--;;-< - - UccC' 2 + UcC" + UppP' 2 + Up P" 0 EXERCISE 8.2 1 ( He - m <f>'J,_re-pT 0 T* :5: Tmax (Hclt� T � 0 = 3 EXERCISE 8.3 1 The Mangasarian conditions are satisfied, based on F being strictly concave in u (Fuu - 2), f being linear in (y, u ), and (8.26) being irrelevant. The Arrow condition is satisfied because H 0 Ay + !A2, which is linear in y for given A. 4 The Mangasarian and the Arrow conditions are both satisfied. (Here, H 0 is independent of y, and hence linear in y for given A.) 6 Both the Mangasarian and Arrow conditions are satisfied. = = EXERCISE 8.4 2 For a positive y0 , the y * value at first increases at a decreasing rate, to reach a peak when arc CD crosses the y axis. It then decreases at a decreasing rate until -r is reached, and then decreases at an increasing rate, to reach zero at t T . 3 (a) The initial point cannot be on arc AO or BO; there must be a switch. (b) The initial point lies on arc AO; no switch. (c) The initial point lies on arc B O; no switch. 5 (a) The switching point, F, has to satisfy simultaneously = y= and y = - !z2 !z2 + k (z > 0) [equation for arc B 0] (k [equation for parabola containing arc EF ] < 0) The solution yields y inadmissible). ( b ) 7' Fk - zo (c) The optimal total time = �k and = = 2Fk - z = Fk (the negative root is z0• EXERCISE 8.5 1 An unbounded solution .can occur when A* * 0. 321 ANSWERS TO SELECTED EXERCISE PROBLEMS 3 It is not possible to have an interior solution for the control variable A. [ Hint: An interior solution A* would mean that {3Ap + A s = 0 , which would imply Ap = constant, and Upe-p1 = SA p from the costate equation of motion. The existence of a tolerably low level of P would imply A p = 0 for all t, which would lead to a contradiction.] EXERCISE 9.3 1 (a) tf>(k) (c) He = = . A AK" 1 k = U '( c ) = c -< 1 +bl be- + m[Ak " - c - U- c - (d) k = AP - b (n + fi)k c= ( n + /3 + r ) 1/(a - 1) ( (n c c =A Aa B)k] 1+b -- Aa ( n + {i + r ) " /( a - 1 ) + - (n + [i ) [ A a k " - 1 - (n + {i + . r)] ( n + {i + r ) 1/(a - 1) Aa 3 ( b) E will give way t o a new steady state, E', o n the k = 0 curve to the right of E, with a higher k. (c) No. EXERCISE 9.4 as 1 (b) To write it 5 The marginal productivity of human capital in research activities could not grow in proportion to A. Y = K"[ A(t)Lil] will conform to the format of (9.48). EXERCISE 10.1 3 (a) Maximize subject to JT(l - s)Ke -p1 dt 0 K = A(sK)" K(O) = K0 (A > 0) K(T ) free O�s � 1 and EXERCISE 10.3 1 (a) A = A + <b> A - ;. + Shy my + 9(h,y + hyy n (K0, T given) ANSWERS TO SELECTED EXEBCIM PROJILBIIIII!I S22 (c) A = (e) ( () + -Fy - A {y + ()hy + ehy + ®( h,y + hyy f) e )hy = 0 e = - () :;;; 0 (since () ;;:: 0] = EXERCISE 10.4 2 (a) Higher Oower) demand would lower (raise) -r. But it is also conceivable that a mere rearrangement of the time profile of D(t) may leave -r unchanged. (c) Yes, because Q*[-r, T] is identical with D(t). · (e) X*[-r, T] 0 regardless of D(t), but the length of the' )leriod (-r, T ] would be affected by any change in T. ·:� = '\ ' ; �· ' IND EX . ,. Adaptive expectations, Antipollution policy, Arc, 55, 194 234-239 4-5 · �"' Arc length, 41 5, 13 Arrow, K. J., 217, 244n., 245-247, 268, 278 Arrow sufficient condition, 217-218, 252 applied, 219-221 for constrained problems, 290-291 Autonomous problem, 103, 212 Arc value, constancy of optimal Hamiltonian examples of, in, 190 106, 113, 248-249, 257 Auxiliary variable, 167 164, 1 76, • * J., 292-293 Bellman, R. E., 20, 22 ·"-'"" Benveniste, L. M., 251 Bernoulli, J., 17 Beyer, W. H., 42n. Bliss, 1 13, 123, 132, 248 Blocked interval, 299 Boltyanskii, V. G., 19n., 167 Bolza, problem of, 15-16, 76 Boundary solution, 164, 173, 237-239 Burness, H. S., 152-153, 155 Bang-bang solution, Baumol, W. Capital accumulation: of a firm (see Jorgenson model; Eisner-Strotz model) under financial constraint, golden rule of, modified golden rule of, of society 310-313 248, 259 259 (see Ramsey model; Neoclassical theory of optimal growth) Capital saturation, 121, 125-126 104 Cass, D., 253, 263-264 Catenary, 43 Catenoid, 43 Chakravarty, S., 99n., lOOn. Characteristic-root test, 85-86 Chiang, A. C., 38n., 40n., 50n., 65n., 86n., 88n., 1 17n., 153n., 174n., 183n., 185n., 189n., 197n., 245n., 262n., 292n . Cobb-Douglas production function, 105, 270 Complementary slackness, 138, 183, 236, 238, 279, 287, 302, 312 Concavity, checking, 83-86, 88-90 Conservation, 151-156 Constrained interval, 299 Constraint qualification, 278 applied, 294, 305, 308, 311 Capital stock, Constraints in calculus o f variations: Calculus of variations, differential-equation, 17-18 fundamental (simplest) problem of, link with optimal control theory, 191-193 Canonical system, 170-171 18, 27 166-167, 137 equality, 134-136 inequality, 137 integral, 138-141 isoperimetric problem, 138-141 323 J24 INDEX Embedding of problem, 20-22 Endogenous technological progress, 267-274 �onstraints in optimal control theory: equality, 276-277 inequality, 277-280 inequality integral, 282-284 integral, 280, 282 isoperimetric problem, 280-282 Energy conservation, 151-156 Environmental quality, 200 Equation of motion, 18, 165-166, 169, 181 with second-order derivative, 227 state-space, 298-303 �onsumption saturation, 121, 125-126 �antral set, 19-20, 164 �antral variable, 161-162 �onvergence of functional, 99-101, 1 06, �onvexity, checking, 83-86, 88-90 �orner solution (see Boundary solution) �ostate equation, 181 �ostate variable, 167, 280 constancy of, 280 as shadow price, Euler equation, 28, 33, 61, 192 113 development of, 32-34 expanded form of, 34 for higher derivatives, 4 7-48 integral form of, 33 for several variables, 45-46 special cases of, 3 7-45 Euler-Lagrange equation, 135-138, 141-142 Euler-Poisson equation, 48 Evans, G. C., 49, 69, 70 n . Exhaustible resource, 148-156, 200 206-208 �ounterexamples against infinite-horizon transversality condition: Halkin, 244-247 Expectations-augmented Phillips relation, 55, 194 Extremal, 27 Pitchford, 250-251 Shell, 247-251 �ourant, R., lOOn. �urrent-value Hamiltonian, 210, 213, 257, First variation, 95-97 Fixed-endpoint problem (see Horizontal­ terminal-line problem) �urrent-value Lagrangian, 287-289, 291, 294 Fixed-time (-horizon) problem terminal-line problem) Flexible accelerator, 105, 1 1 0 )antzig, G. B., 217n . Fulks, W., lOOn. )eterminantal test: Functional, 7-8 287-290, 291, 294 ::urrent-value Lagrange multiplier, 210 for sign definiteness, 84-85 (see Vertical- Forster, B . A., 200, 204, 234 Gamkrelidze, R. V., 19n., 167 for sign semidefiniteness, 85 )ifferential equation: with constant coefficient and constant term, 50, 189 with constant coefficient and variable term, 197 with separable variables, 143 with variable coefficient and variable term, 185, 244-245 >ifferentiation of definite integral, 29-31 >orfman, R., 205, 243, 291 )ynamic monopolist, 49-53, 69-70, 87-88, 94 )ynamic optimization, 3-6 Golden rule, 248, 259 modified, 259 Gould, J. P., 105n. H (see Hamiltonian function) He (see Current-value Hamiltonian) H 0 (see Maximized Hamiltonian) Hadley, G., lOOn., 130n. Halkin, H., 244-247 Hamermesh, D. S., 75 Hamiltonian function, 167-168 current-value, 210, 213, 257, 287-291, 294 economic interpretation of, 207 linear )ynamic production function, 264-265 (in u ), 170 maximized, 218, 252, 290 )ynamic programming, 20-22 optimal, 190 Hamiltonian system, 170-1 7 1 �fficiency labor, 266 �isner (see Eisner-Strotz model) Harrod neutrality, 265, 2 7 1 , 273 Hestenes, M. R., 19, 167, 301n. Hicks neutrality, 265 •:isner-Strotz model, 106-110, 124, 148 as a control problem, 212, 220 Horizontal-terminal-line problem, 111-11, 63, phase-diagram analysis of, 1 1 7-120 sufficient conditions applied to, 130, 220 182, 187-190 ,· , INDEX Hotelling, H., Lewis, T. R., Human capital, L'Hopital's rule, Hurwicz, 148, 149n., 152, 162 269 L., 278 Implicit-function rule, 296-297 Infinite horizon: · ,,,;,. · · . .._, .: ···· convergence of functional, 99-101, 106, 113 98-103 transversality conditions, 101-103, 115, 240-243, 261-262 counterexamples against, 244-251 Inflation-unemployment tradeoff, 54-58, 70-72 as a constrained problem, 145-14 7 Integration by parts, 32, 82, 216 Intertemporal resource allocation, 111 Inventory problem, 307-310 methodological issues of, Investment Isochrone, ( see Capital accumulation) 139-143, 149, 150, 262 244n., 290 n. Mangasarian, 0. L., 214 Mangasarian sufficient conditions, 214-217, 252 applied, 219-221 for constrained problems, 290-291 152-153, 155 Maximized Hamiltonian, 218, 252, 290 Maximum area under a curve, 141-143 Maximum principle, 19, 167-171 Matthews, S. A., (see Constraints 21 1-212 205-210 economic interpretation of, rationale of, 177-181 for several variables, 134, 137n . Jacobian matrix, 262-263 Jorgenson model, 103-105 Jump discontinuity, 164, 300 Junction point, 300, 309 223-226 15-16, 76 lOOn. Mayer, problem of, McShane, E. J., Michel, P., 1., 168n., 217n., 221n., 306n. Kemp, M. C., lOOn., l30 n ., 307n. Kuhn, H. W., 247n. Kuhn-Tucker conditions, 65, 102, 183, 236, 278-279, 303 Kurz, M., 217n., 244n., 245-247 Knowles, G., .f (see Lagrangian function) (see Current-value Lagrangian) distance problem) Minimum principle, 265 75-78 Lagrange multipliers, 135, 178 constancy of, 139-140 current-value, 210 Lagrangian function (augmented 276, 279 current-value, 287-289, 291, 294 Lagrangian integrand function, 134-135, 138, 140 Learning by doing, 268 Legendre necessary condition, 91-94, 192 Leibniz's rule, 29-30 applied, 31, 32, 80 Leland, H. E., 293, 295, 296 19n. Minimum surface o f revolution, 40-41, 93 Mishchenko, E. F., Modified golden 19n., 167 rule, 259 Monopoly: consequences of, Multistage Labor-demand adjustment, 243 Minimum-distance problem (see Shortest- dynamic, .L', Hamiltonian), equations, Long, N. V., for current-value Hamiltonian, Jacobian determinant, Labor-augmenting technological progress, Linearization of nonlinear differe�ial in optimal control theory) 280-282 Isovote curve, 194 Kamien, M. 152-153, 155 249 n. of F function, 35-36 for constrained problems 127-128 Isoperimetric problem, Linearity 325 150-156 49-53, 69-70, 87-88, 94 decision making, 4-6 Natural boundary condition, Negative definiteness (see Negative semidefiniteness 63 Quadratic form) ( see Quadratic form) Neoclassical production function, 103, 247, 253-254, 293 Neoclassical theory of optimal growth, 253-262 with Harrod-neutral technological progress, 265-267 Neutral technological progress, Newton, 1., 17 Nonnegativity restriction, Nordhaus, 265 298, 308 W. D., 193, 196, 199 J26 INDEX )bjective functional, 12-16 Research success coefficient, 268, 270 )ptimal control theory, 18-20 Revenue-maximizing firm, 292-298 features of, 163-164 Rival good, 269 link with calculus of variations, 166-167, Romer, P. M., 269-274 191-193 with several state and control variables, 221-226 example from Pontryagin, 226-233 simplest problem of, 162, 164-166 )ptimal policy functional, 21 )ptimal value function, 20-21 Saddlepoint equilibrium, 120, 123-124, 262 Samuelson, P. A., 126 n . Sato, K., 269 Scheinkman, J. A., 2 5 1 Schwartz, N. Schworm, W. L., 1 6 8 n . , 2 1 7 n . , 2 2 1 n . , 306n. E., 310-313 Second-order conditions, 79-81, 91-94 Second variation, 95-97 'ath: admissible, 6-7 Seierstad, optimal, 6 sgn (see A., 99n . , 221n., 300n., 303n., Signum function) 'ath value, 6-7 Shadow price, 206-209 'erturbing curve, 28, 180-182 Shell, K., 247-251, 255, 268 'hase diagram: Shortest-distance problem: one-variable, 38-39, 229-231 two-variable, 1 1 7-129, 260-262 in calculus of variations: between a point and a line, 66-67 'helps, E. S., 248n. between two given points, 44 'iecewise continuity, 163 between two points on a surface, 136-137 'iecewise differentiability, 163 'itchford, J. D., 244n, 250, 251, 290 n . 'olitical business cycle, 193-199, 284-286 second-order condition, 86-87, 93 in optimal control theory: between a point and a line, 1 7 1 - 1 73 sufficient conditions, 218-219 'ollution, 200 as a flow, 200 Signum function, 188, 227, 286 as a stock, 234-235 Social loss function, 54-55 'ontryagin, 241n. L. S., 19, 167, 177, 226, 232, 240, 'ositive definiteness (see Quadratic form) 'ositive semidefiniteness Social utility function, 255 Solow, R., 248 ( see Quadratic form) Solow neutrality, 265 Stable branches, 1 19, 120, 124, 260-261 'rinciple of optimality, 22 Stage variable, 4 'rinciple of reciprocity, 143 State equation, 18, 165 'roduction function: State-space constraints, 298-303 Cobb-Douglas, 105, 270 economic examples of, 307-313 dynamic, 264-265 State variable, 4 neoclassical, 103, 247, 253-254, 293 Steady state, 1 18, 248, 260 'ure competition and social optimum, 150-151 'ythagoras' theorem, 40 and Harrod neutrality, 265-266, 273 Stiglitz, J. E., 151-155 Strotz, R. H. (see Eisner-Strotz model) Sufficient conditions: !uadratic form, 83-86, 88-90 in calculus of variations, 81-83 '.amsey, F. P., 1 1 1 , 128 in optimal control theory, 214-218, 252 :amsey device, 1 13, 248, 250 :amsey model, 1 1 1-116 as a constrained problem, 144-145 phase-diagram analysis of, 1 18-120, 122-126 simplified, 120-126 sufficient condition applied to, 131-132 for infinite horizon, 130-132 applied, 219-221 for constrained problems, 290-291 Switching of control, 174-176, 230-233 Sydsreter, K., 99n., 221n., 300n., 303n. Szegii, G. P., 247n. Takayama, A., 81n., 168 n ., 244n . , 278n. :amsey rule, 115, 1 1 6 Taylor, D., 54 ID1 k condition, 278 Taylor series, 96 INDEX Technological progress, 264 endogenous, 267-274 neutrru exogenous, 265 labor-augmenting, 265 Terminru-control problem, 15, 244 Terminal-curve problem, 11, 64, 182 Time-optimal problem, 11, 187-189, 227, 286-287 Tintner, G., 52 Tradeoff between inflation and unemployment (see Inflation-unemployment tradeoff) Trajectory, 27 Transition equation, 18 Transversality conditions: in calculus of variations, 11-12, 60-63, 73-74 with horizontal terminal line, 63-64, 193 with infinite horizon, 101-103, 115 with terminal curve, 64-65 with truncated horizontal terminal line, 66 with truncated vertical terminal line, 65-66 with vertical terminal line, 63, 192 in optimal control theory: economic interpretation of, 209-210 with horizontru terminal line, 182, 190, 193 with infinite horizon, 240-243, 247-250 327 with terminal curve, 182 with truncated horizontru terminal line, 184 with truncated vertical terminal line, 182-183 with vertical terminal line, 169, 181, 192 Trirogolf, K. N., 19n., 167n. Truncated-horizontal-terminal-line problem, 66, 72 Truncated-vertical-terminru-line problem, 9-10, 65-66, 182-183, 186-187, 200-204, 235-236, 308 Turnovsky, S. J., 244n., 250n., 290 n . Turnpike behavior, 126-129 Uzawa, H., 278 Variable endpoint, types of, 9-11 Variations, first and second, 95-97 Veinott, A. F., Jr., 217n. Vertical-terminal-line problem, 9, 63, 164, 171, 181, 192, 195 Vote function, 193-194 Vousden, N., 244n., 290 n . Weierstrass-Erdmann corner condition, 163n. Weitzman, M . L., 251 Yellow brick road, 1 20, 124, 261