Time series analysis

Time Series
Analysis

C5967_FM.indd 1

10/22/07 10:39:05 AM


CHAPMAN & HALL/CRC
Texts in Statistical Science Series

Series Editors
Bradley P. Carlin, University of Minnesota, USA
Julian J. Faraway, University of Bath, UK
Martin Tanner, Northwestern University, USA
Jim Zidek, University of British Columbia, Canada
Analysis of Failure and Survival Data
P. J. Smith

The Analysis and Interpretation of
Multivariate Data for Social Scientists
D.J. Bartholomew, F. Steele,
I. Moustaki, and J. Galbraith
The Analysis of Time Series —
An Introduction, Sixth Edition
C. Chatfield

Applied Bayesian Forecasting and Time Series
Analysis
A. Pole, M. West and J. Harrison

Applied Nonparametric Statistical Methods,
Fourth Edition
P. Sprent and N.C. Smeeton

Applied Statistics — Handbook of GENSTAT
Analysis
E.J. Snell and H. Simpson
Applied Statistics — Principles and Examples
D.R. Cox and E.J. Snell
Bayes and Empirical Bayes Methods for Data
Analysis, Second Edition
B.P. Carlin and T.A. Louis
Bayesian Data Analysis, Second Edition
A. Gelman, J.B. Carlin, H.S. Stern
and D.B. Rubin
Beyond ANOVA — Basics of Applied
Statistics
R.G. Miller, Jr.

Computer-Aided Multivariate Analysis,
Fourth Edition
A.A. Afifi and V.A. Clark
A Course in Categorical Data Analysis
T. Leonard
A Course in Large Sample Theory
T.S. Ferguson
Data Driven Statistical Methods
P. Sprent

Decision Analysis — A Bayesian Approach

J.Q. Smith
Elementary Applications of Probability
Theory, Second Edition
H.C. Tuckwell

C5967_FM.indd 2

Elements of Simulation
B.J.T. Morgan

Epidemiology — Study Design and
Data Analysis, Second Edition
M. Woodward

Essential Statistics, Fourth Edition
D.A.G. Rees

Extending the Linear Model with R:
Generalized Linear, Mixed Effects and
Nonparametric Regression Models
J.J. Faraway

A First Course in Linear Model Theory
N. Ravishanker and D.K. Dey
Generalized Additive Models:
An Introduction with R
S. Wood

Interpreting Data — A First Course
in Statistics

A.J.B. Anderson
An Introduction to Generalized
Linear Models, Second Edition
A.J. Dobson

Introduction to Multivariate Analysis
C. Chatfield and A.J. Collins

Introduction to Optimization Methods and
Their Applications in Statistics
B.S. Everitt
Introduction to Randomized Controlled
Clinical Trials, Second Edition
J.N.S. Matthews
Introduction to Statistical Methods for
Clinical Trials
Thomas D. Cook and David L. DeMets
Large Sample Methods in Statistics
P.K. Sen and J. da Motta Singer
Linear Models with R
J.J. Faraway

Markov Chain Monte Carlo — Stochastic
Simulation for Bayesian Inference, Second Edition
D. Gamerman and H.F. Lopes
Mathematical Statistics
K. Knight

Modeling and Analysis of Stochastic Systems
V. Kulkarni

10/22/07 10:39:05 AM


Modelling Binary Data, Second Edition
D. Collett

Modelling Survival Data in Medical Research,
Second Edition
D. Collett

Multivariate Analysis of Variance and Repeated
Measures — A Practical Approach for
Behavioural Scientists
D.J. Hand and C.C. Taylor
Multivariate Statistics — A Practical Approach
B. Flury and H. Riedwyl
Practical Data Analysis for Designed
Experiments
B.S. Yandell

Practical Longitudinal Data Analysis
D.J. Hand and M. Crowder

Practical Statistics for Medical Research
D.G. Altman

Probability — Methods and Measurement
A. O’Hagan

Problem Solving — A Statistician’s Guide,
Second Edition
C. Chatfield
Randomization, Bootstrap and
Monte Carlo Methods in Biology,
Third Edition
B.F.J. Manly
Readings in Decision Analysis
S. French

Sampling Methodologies with Applications
P.S.R.S. Rao
Statistical Analysis of Reliability Data
M.J. Crowder, A.C. Kimber,
T.J. Sweeting, and R.L. Smith

C5967_FM.indd 3

Statistical Methods for Spatial Data Analysis
O. Schabenberger and C.A. Gotway
Statistical Methods for SPC and TQM
D. Bissell

Statistical Methods in Agriculture and
Experimental Biology, Second Edition
R. Mead, R.N. Curnow, and A.M. Hasted
Statistical Process Control — Theory
and Practice, Third Edition
G.B. Wetherill and D.W. Brown
Statistical Theory, Fourth Edition

B.W. Lindgren
Statistics for Accountants
S. Letchford

Statistics for Epidemiology
N.P. Jewell

Statistics for Technology — A Course in
Applied Statistics, Third Edition
C. Chatfield
Statistics in Engineering —
A Practical Approach
A.V. Metcalfe

Statistics in Research and Development,
Second Edition
R. Caulcutt
Survival Analysis Using S —Analysis of
Time-to-Event Data
M. Tableman and J.S. Kim
Time Series Analysis
H. Madsen

The Theory of Linear Models
B. Jørgensen

10/22/07 10:39:05 AM


C5967_FM.indd 4

10/22/07 10:39:05 AM


Texts in Statistical Science

Time Series
Analysis

Henrik Madsen
Technical University of Denmark

Boca Raton London New York

Chapman & Hall/CRC is an imprint of the
Taylor & Francis Group, an informa business

C5967_FM.indd 5

10/22/07 10:39:05 AM


CRC Press
Taylor & Francis Group
6000 Broken Sound Parkway NW, Suite 300
Boca Raton, FL 33487-2742
© 2007 by Taylor & Francis Group, LLC
CRC Press is an imprint of Taylor & Francis Group, an Informa business
No claim to original U.S. Government works
Version Date: 20140313

International Standard Book Number-13: 978-1-4200-5968-7 (eBook - PDF)
This book contains information obtained from authentic and highly regarded sources. Reasonable efforts
have been made to publish reliable data and information, but the author and publisher cannot assume
responsibility for the validity of all materials or the consequences of their use. The authors and publishers
have attempted to trace the copyright holders of all material reproduced in this publication and apologize to
copyright holders if permission to publish in this form has not been obtained. If any copyright material has
not been acknowledged please write and let us know so we may rectify in any future reprint.
Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmitted, or utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented,
including photocopying, microfilming, and recording, or in any information storage or retrieval system,
without written permission from the publishers.
For permission to photocopy or use material electronically from this work, please access www.copyright.
com ( or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood
Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization that provides licenses and
registration for a variety of users. For organizations that have been granted a photocopy license by the CCC,
a separate system of payment has been arranged.
Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used
only for identification and explanation without intent to infringe.
Visit the Taylor & Francis Web site at

and the CRC Press Web site at



“C5967 Final 1st Edition” — 2007/10/24 — 14:13 — page vii — #1

Contents

Preface

xiii

Notation
1 Introduction
1.1
Examples of time series . . . . . . . . . . . .
1.1.1 Dollar to Euro exchange rate . . . . .
1.1.2 Number of monthly airline passengers
1.1.3 Heat dynamics of a building . . . . .
1.1.4 Predator-prey relationship . . . . . .
1.2
A first crash course . . . . . . . . . . . . . .
1.3
Contents and scope of the book . . . . . . .

xv

.
.
.
.
.
.
.

.
.
.
.
.
.

.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.

.
.
.

1
2
2
2
3
4
5
7

2 Multivariate random variables
2.1
Joint and marginal densities . . . . . . . . . . . .
2.2
Conditional distributions . . . . . . . . . . . . . .
2.3
Expectations and moments . . . . . . . . . . . . .
2.4
Moments of multivariate random variables . . . .
2.5
Conditional expectation . . . . . . . . . . . . . . .
2.6
The multivariate normal distribution . . . . . . . .
2.7
Distributions derived from the normal distribution
2.8
Linear projections . . . . . . . . . . . . . . . . . .

2.9
Problems . . . . . . . . . . . . . . . . . . . . . . .

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

13
13
14
15
17
20
22

23
24
29

3 Regression-based methods
3.1
The regression model . . . . . . . . . . . . . .
3.2
The general linear model (GLM) . . . . . . . .
3.2.1 Least squares (LS) estimates . . . . . .
3.2.2 Maximum likelihood (ML) estimates . .
3.3
Prediction . . . . . . . . . . . . . . . . . . . . .
3.3.1 Prediction in the general linear model .
3.4
Regression and exponential smoothing . . . . .
3.4.1 Predictions in the constant mean model

.
.
.
.
.
.
.
.

.
.
.

.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.

.
.

31
31
33
34
40
44
45
47
48

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.

.
.
.
.
.
.
.

.
.
.
.
.
.
.
.


“C5967 Final 1st Edition” — 2007/10/24 — 14:13 — page viii — #2

3.4.2

3.5
3.6
3.7

Locally constant mean model and simple
exponential smoothing . . . . . . . . . .
3.4.3 Prediction in trend models . . . . . . . .
3.4.4 Local trend and exponential smoothing .

Time series with seasonal variations . . . . . . .
3.5.1 The classical decomposition . . . . . . . .
3.5.2 Holt-Winters procedure . . . . . . . . . .
Global and local trend model—an example . . .
Problems . . . . . . . . . . . . . . . . . . . . . .

4 Linear dynamic systems
4.1
Linear systems in the time domain . . .
4.2
Linear systems in the frequency domain
4.3
Sampling . . . . . . . . . . . . . . . . .
4.4
The z-transform . . . . . . . . . . . . .
4.5
Frequently used operators . . . . . . . .
4.6
The Laplace transform . . . . . . . . .
4.7
A comparison between transformations
4.8
Problems . . . . . . . . . . . . . . . . .

.
.
.
.
.
.

.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

50
52
56
59
60
61
62
65

.
.
.

.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.

.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

69
70
73
78
80
87
90
94
96

5 Stochastic processes
5.1
Introduction . . . . . . . . . . . . . . . . . . .
5.2
Stochastic processes and their moments . . . .
5.2.1 Characteristics for stochastic processes
5.2.2 Covariance and correlation functions . .
5.3
Linear processes . . . . . . . . . . . . . . . . .
5.3.1 Processes in discrete time . . . . . . . .
5.3.2 Processes in continuous time . . . . . .

5.4
Stationary processes in the frequency domain .
5.5
Commonly used linear processes . . . . . . . .
5.5.1 The MA process . . . . . . . . . . . . .
5.5.2 The AR process . . . . . . . . . . . . .
5.5.3 The ARMA process . . . . . . . . . . .
5.6
Non-stationary models . . . . . . . . . . . . . .
5.6.1 The ARIMA process . . . . . . . . . . .
5.6.2 Seasonal models . . . . . . . . . . . . .
5.6.3 Models with covariates . . . . . . . . .
5.6.4 Models with time-varying mean values .
5.6.5 Models with time-varying coefficients .
5.7
Optimal prediction of stochastic processes . . .
5.7.1 Prediction in the ARIMA process . . .
5.8
Problems . . . . . . . . . . . . . . . . . . . . .

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

97
97
97
99
103
107

107
111
113
117
117
119
125
130
130
132
134
134
135
135
137
140

.
.
.
.
.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.
.
.
.
.


“C5967 Final 1st Edition” — 2007/10/24 — 14:13 — page ix — #3

6 Identification, estimation, and model checking
6.1
Introduction . . . . . . . . . . . . . . . . . . . . . . . .
6.2
Estimation of covariance and correlation functions . . .
6.2.1 Autocovariance and autocorrelation functions . .
6.2.2 Cross-covariance and cross-correlation functions
6.3
Identification . . . . . . . . . . . . . . . . . . . . . . . .
6.3.1 Identification of the degree of differencing . . . .
6.3.2 Identification of the ARMA part . . . . . . . . .
6.3.3 Cointegration . . . . . . . . . . . . . . . . . . . .
6.4

Estimation of parameters in standard models . . . . . .
6.4.1 Moment estimates . . . . . . . . . . . . . . . . .
6.4.2 The LS estimator for linear dynamic models . .
6.4.3 The prediction error method . . . . . . . . . . .
6.4.4 The ML method for dynamic models . . . . . .
6.5
Selection of the model order . . . . . . . . . . . . . . .
6.5.1 The autocorrelation functions . . . . . . . . . .
6.5.2 Testing the model . . . . . . . . . . . . . . . . .
6.5.3 Information criteria . . . . . . . . . . . . . . . .
6.6
Model checking . . . . . . . . . . . . . . . . . . . . . . .
6.6.1 Cross-validation . . . . . . . . . . . . . . . . . .
6.6.2 Residual analysis . . . . . . . . . . . . . . . . . .
6.7
Case study: Electricity consumption . . . . . . . . . . .
6.8
Problems . . . . . . . . . . . . . . . . . . . . . . . . . .

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

145
145
146
146
150
152
153
154
156
157
157
159
163
166
170
171
171
174

174
175
175
179
182

.
.
.
.
.
.
.
.
.

187
187
189
190
194
195
196
200
203
206

.
.
.

206
209
210

8 Linear systems and stochastic processes
8.1
Relationship between input and output processes . . . . . .
8.1.1 Moment relations . . . . . . . . . . . . . . . . . . .

215
215
216

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.

7 Spectral analysis
7.1
The periodogram . . . . . . . . . . . . . . . . . . . . . . .
7.1.1 Harmonic analysis . . . . . . . . . . . . . . . . . .
7.1.2 Properties of the periodogram . . . . . . . . . . .
7.2
Consistent estimates of the spectrum . . . . . . . . . . .
7.2.1 The truncated periodogram . . . . . . . . . . . . .
7.2.2 Lag- and spectral windows . . . . . . . . . . . . .
7.2.3 Approximative distributions for spectral estimates
7.3
The cross-spectrum . . . . . . . . . . . . . . . . . . . . .
7.3.1 The co-spectrum and the quadrature spectrum . .
7.3.2 Cross-amplitude spectrum, phase spectrum,
coherence spectrum, gain spectrum . . . . . . . .
7.4
Estimation of the cross-spectrum . . . . . . . . . . . . . .
7.5
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . .


“C5967 Final 1st Edition” — 2007/10/24 — 14:13 — page x — #4

8.2
8.3

8.4
8.5

8.6

8.7
8.8
8.9
8.10

8.1.2 Spectral relations . . . . . . . . . . . .
Systems with measurement noise . . . . . . . .
Input-output models . . . . . . . . . . . . . . .
8.3.1 Transfer function models . . . . . . . .
8.3.2 Difference equation models . . . . . . .
8.3.3 Output error models . . . . . . . . . . .
Identification of transfer function models . . .
Multiple-input models . . . . . . . . . . . . . .
8.5.1 Moment relations . . . . . . . . . . . .
8.5.2 Spectral relations . . . . . . . . . . . .
8.5.3 Identification of multiple-input models .
Estimation . . . . . . . . . . . . . . . . . . . .
8.6.1 Moment estimates . . . . . . . . . . . .
8.6.2 LS estimates . . . . . . . . . . . . . . .
8.6.3 Prediction error method . . . . . . . . .
8.6.4 ML estimates . . . . . . . . . . . . . . .
8.6.5 Output error method . . . . . . . . . .

Model checking . . . . . . . . . . . . . . . . . .
Prediction in transfer function models . . . . .
8.8.1 Minimum variance controller . . . . . .
Intervention models . . . . . . . . . . . . . . .
Problems . . . . . . . . . . . . . . . . . . . . .

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

9 Multivariate time series
9.1
Stationary stochastic processes and their moments . . . .
9.2
Linear processes . . . . . . . . . . . . . . . . . . . . . . .
9.3

The multivariate ARMA process . . . . . . . . . . . . . .
9.3.1 Theoretical covariance matrix functions . . . . . .
9.3.2 Partial correlation matrix . . . . . . . . . . . . . .
9.3.3 q-conditioned partial correlation matrix . . . . . .
9.3.4 VAR representation . . . . . . . . . . . . . . . . .
9.4
Non-stationary models . . . . . . . . . . . . . . . . . . . .
9.4.1 The multivariate ARIMA process . . . . . . . . .
9.4.2 The multivariate seasonal model . . . . . . . . . .
9.4.3 Time-varying models . . . . . . . . . . . . . . . .
9.5
Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . .
9.5.1 Missing values for some signals . . . . . . . . . . .
9.6
Identification of multivariate models . . . . . . . . . . . .
9.6.1 Identification using pre-whitening . . . . . . . . .
9.7
Estimation of parameters . . . . . . . . . . . . . . . . . .
9.7.1 Least squares estimation . . . . . . . . . . . . . .
9.7.2 An extended LS method for multivariate ARMAX
models (the Spliid method) . . . . . . . . . . . . .
9.7.3 ML estimates . . . . . . . . . . . . . . . . . . . . .

.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

218
220
222
222
223
223
223
226
226
227
227
228
229
229

229
229
230
230
233
238
241
244

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

247
249
251

254
255
259
260
260
261
261
261
262
262
265
267
269
269
270

.
.

271
271


“C5967 Final 1st Edition” — 2007/10/24 — 14:13 — page xi — #5

9.8
9.9

Model checking . . . . . . . . . . . . . . . . . . . . . . . . .
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . .

274
278

space models of dynamic systems
The linear stochastic state space model . . . . . . . . . . .
Transfer function and state space formulations . . . . . . .
Interpolation, reconstruction, and prediction . . . . . . . .
10.3.1 The Kalman filter . . . . . . . . . . . . . . . . . . .
10.3.2 k-step predictions in state space models . . . . . . .
10.3.3 Empirical Bayesian description of the Kalman filter
Some common models in state space form . . . . . . . . . .
10.4.1 Signal extraction . . . . . . . . . . . . . . . . . . . .
Time series with missing observations . . . . . . . . . . . .
10.5.1 Estimation of autocorrelation functions . . . . . . .
ML estimates of state space models . . . . . . . . . . . . .
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . .

283
284
286
288
289
296
296
299
301
307
307
307

310

11 Recursive estimation
11.1 Recursive LS . . . . . . . . . . . . . . . . . . . . . . . . . .
11.1.1 Recursive LS with forgetting . . . . . . . . . . . . .
11.2 Recursive pseudo-linear regression (RPLR) . . . . . . . . .
11.3 Recursive prediction error methods (RPEM) . . . . . . . .
11.4 Model-based adaptive estimation . . . . . . . . . . . . . . .
11.5 Models with time-varying parameters . . . . . . . . . . . .
11.5.1 The regression model with time-varying parameters
11.5.2 Dynamic models with time-varying parameters . . .

313
313
316
319
321
324
325
325
326

12 Real life inspired problems
12.1 Prediction of wind power production . . . . . .
12.2 Prediction of the consumption of medicine . .
12.3 Effect of chewing gum . . . . . . . . . . . . . .
12.4 Prediction of stock prices . . . . . . . . . . . .
12.5 Wastewater treatment: Using root zone plants
12.6 Scheduling system for oil delivery . . . . . . .
12.7 Warning system for slippery roads . . . . . . .

12.8 Statistical quality control . . . . . . . . . . . .
12.9 Wastewater treatment: Modeling and control .
12.10 Sales numbers . . . . . . . . . . . . . . . . . .
12.11 Modeling and prediction of stock prices . . . .
12.12 Adaptive modeling of interest rates . . . . . .

331
333
334
336
338
340
341
344
345
347
350
352
353

10 State
10.1
10.2
10.3

10.4
10.5
10.6
10.7

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

Appendix A The solution to difference equations

355

Appendix B Partial autocorrelations

357


“C5967 Final 1st Edition” — 2007/10/24 — 14:13 — page xii — #6

Appendix C Some results from trigonometry

361

Appendix D List of acronyms

363

Appendix E List of symbols

365

Bibliography

367

Index

373


“C5967 Final 1st Edition” — 2007/10/24 — 14:13 — page xiii — #7

Preface

The aim of this book is to give an introduction to time series analysis. The
emphasis is on methods for modeling of linear stochastic systems. Both time
domain and frequency domain descriptions will be given; however, emphasis
is on the time domain description. Due to the highly different mathematical
approaches needed for linear and non-linear systems, it is instructive to deal
with them in seperate textbooks, which is why non-linear time series analysis
is not a topic in this book—instead the reader is referred to Madsen, Holst,
and Lindström (2007).
Theorems are used to emphasize the most important results. Proofs are
given only when they clarify the results. Small problems are included at the
end of most chapters, and a separate chapter with real-life problems is included
as the final chapter of the book. This also serves as a demonstration of the
many possible applications of time series analysis in areas such as physics,
engineering, and econometrics.
During the sequence of chapters, more advanced stochastic models are
gradually introduced; with this approach, the family of linear time series
models and methods is put into a clear relationship. Following an initial
chapter covering static models and methods such as the use of the general
linear model for time series data, the rest of the book is devoted to stochastic
dynamic models which are mostly formulated as difference equations, as in the
famous ARMA or vector ARMA processes. It will be obvious to the reader
of this book that even knowing how to solve difference equations becomes
important for understanding the behavior of important aspects such as the

autocovariance functions and the nature of the optimal predictions.
The important concept of time-varying systems is dealt with using a
state space approach and the Kalman filter. However, the strength of also
using adaptive estimation methods for on-line forecasting and control is often
not adequately recognized. For instance, in finance the classical methods
for forecasting are often not very useful, but, by using adaptive techniques,
interesting results are often obtained.
The last chapter of this book is devoted to problems inspired by real
life. Solutions to the problems are found at />time.series.analysis. This home page also contains additional exercises,
called assignments, intended for being solved using a computer with dedicated


“C5967 Final 1st Edition” — 2007/10/24 — 14:13 — page xiv — #8

software for time series analysis.
I am grateful to all who have contributed with useful comments and
suggestions for improvement. Especially, I would like to thank my colleagues
Jan Holst, Henrik Spliid, Leif Mejlbro, Niels Kjølstad Poulsen, and Henrik
Aalborg Nielsen for their valuable comments and suggestions. Furthermore, I
would like to thank former students Morten Høier Olsen, Rasmus Tamstorf,
and Jan Nygaard Nielsen for their great effort in proofreading and improving
the first manuscript in Danish. For this 2007 edition in English, I would
like to thank Devon Yates, Stig Mortensen, and Fannar Örn Thordarson for
proofreading and their very useful suggestions. In particular, I am grateful to
Anna Helga Jónsdóttir for her assistance with figures and examples. Finally, I
would like to thank Morten Høgholm for both proofreading and for proposing
and creating a new layout in LATEX.

Lyngby, Denmark

Henrik Madsen


“C5967 Final 1st Edition” — 2007/10/24 — 14:13 — page xv — #9

Notation

All vectors are column vectors. Vectors and matrices are emphasized using
a bold font. Lowercase letters are used for vectors and uppercase letters are
used for matrices. Transposing is denoted with the upper index T .
Random variables are always written using uppercase letters. Thus, it is
not possible to distinguish between a multivariate random variable (random
vector) and a matrix. However, random variables are assigned to letters from
the last part of the alphabet (X, Y, Z, U, V, . . . ), while deterministic terms are
assigned to letters from the first part of the alphabet (a, b, c, d, . . . ). Thus, it
should be possible to distinguish between a matrix and a random vector.


“C5967 Final 1st Edition” — 2007/10/24 — 14:13 — page xvi — #10


“C5967 Final 1st Edition” — 2007/10/24 — 14:13 — page 1 — #11

CHAPTER 1

Introduction

Time series analysis deals with statistical methods for analyzing and modeling
an ordered sequence of observations. This modeling results in a stochastic
process model for the system which generated the data. The ordering of

observations is most often, but not always, through time, particularly in terms
of equally spaced time intervals. In some applied literature, time series are
often called signals. In more theoretical literature a time series is just an
observed or measured realization of a stochastic process.
This book on time series analysis focuses on modeling using linear models.
During the sequence of chapters more and more advanced models for dynamic
systems are introduced; by this approach the family of linear time series models
and methods are placed in a structured relationship. In a subsequent book,
non-linear time series models will be considered.
At the same time the book intends to provide the reader with an understanding of the mathematical and statistical background for time series
analysis and modeling. In general the theory in this book is kept in a second
order theory framework, focussing on the second order characteristics of the
persistence in time as measured by the autocovariance and autocorrelation
functions.
The separation of linear and non-linear time series analysis into two books
facilitates a clear demonstration of the highly different mathematical approaches that are needed in each of these two cases. In linear time series
analysis some of the most important approaches are linked to the fact that
superposition is valid, and that classical frequency domain approaches are
directly usable. For non-linear time series superposition is not valid and
frequency domain approaches are in general not very useful.
The book can be seen as a text for graduates in engineering or science
departments, but also for statisticians who want to understand the link between models and methods for linear dynamical systems and linear stochastic
processes. The intention of the approach taken in this book is to bridge the
gap between scientists or engineers, who often have a good understanding of
methods for describing dynamical systems, and statisticians, who have a good
understanding of statistical theory such as likelihood-based approaches.
In classical statistical analysis the correlation of data in time is often
disregarded. For instance in regression analysis the assumption about serial

“C5967 Final 1st Edition” — 2007/10/24 — 14:13 — page 2 — #12

2

Introduction

uncorrelated residuals is often violated in practice. In this book it will be
demonstrated that it is crucial to take this autocorrelation into account in the
modeling procedure. Also for applications such as simulations and forecasting,
we will most often be able to provide much more reasonable and realistic
results by taking the autocorrelation into account.
On the other hand adequate methods and models for time series analysis
can often be seen as a simple extension of linear regression analysis where
previous observations of the dependent variable are included as explanatory
variables in a simple linear regression type of model. This facilitates a rather
easy approach for understanding many methods for time series analysis, as
demonstrated in various chapters of this book.
There are a number of reasons for studying time series. These include a
characterization of time series (or signals), understanding and modeling the
data generating system, forecasting of future values, and optimal control of a
system.
In the rest of this chapter we will first consider some typical time series and
briefly mention the reasons for studying them and the methods to use in each
case. Then some of the important methodologies and models are introduced
with the help of an example where we wish to predict the monthly wheat
prices. Finally the contents of the book is outlined while focusing on the model
structures and their basic relations.

1.1

Examples of time series

In this section we will show examples of time series, and at the same time
indicate possible applications of time series analysis. The examples contain
both typical examples from economic studies and more technical applications.

1.1.1

Dollar to Euro exchange rate

The first example is the daily US dollar to Euro interbank exchange rate
shown in Figure 1.1. This is a typical economic time series where time series
analysis could be used to formulate a model for forecasting future values of
the exchange rate. The analysis of such a problem relates to the models and
methods described in Chapters 3, 5, and 6.

1.1.2

Number of monthly airline passengers

Next we consider the number of monthly airline passengers in the US shown
in Figure 1.2. For this series a clear annual variation is seen. Again it might
be useful to construct a model for making forecasts of the future number of
airline passengers. Models and methods for analyzing time series with seasonal
variation are described in Chapters 3, 5, and 6.


“C5967 Final 1st Edition” — 2007/10/24 — 14:13 — page 3 — #13

Examples of time series

3

0.80
0.75

$US to €

0.85

1.1

Q3 Q4
2004

Q1

Q2 Q3
2005

Q4

Q1

Q2 Q3
2006

Q4

Q1 Q2

2007

50000
40000

Airline passengers

60000

Figure 1.1: Daily US dollar to Euro interbank exchange rate.

1995

1996

1997

1998

1999

2000

2001

2002

Figure 1.2: Number of monthly airline passengers in the US. A clear annual variation
can be seen in the series.

1.1.3

Heat dynamics of a building

Now let us consider a more technical example. Figure 1.3 on the following
page shows measurements from an unoccupied test building. The data on
the lower plot show the indoor air temperature, while on the upper plot the
ambient air temperature, the heat supply, and the solar radiation are shown.
For this example it might be interesting to characterize the thermal behavior
of the building. As a part of that the so-called resistance against heat flux from
inside to outside can be estimated. The resistance characterizes the insulation
of the building. It might also be useful to establish a dynamic model for the
building and to estimate the time constants. Knowledge of the time constants
can be used for designing optimal controllers for the heat supply.


“C5967 Final 1st Edition” — 2007/10/24 — 14:13 — page 4 — #14

4

Introduction

0.2
0.4
0.6
Solar radiation (1)

(3)

(2)

(1)

0.0

Ambient air temp. (2)
8 10 12 14 16

Input variables

0:00

12:00

Oct 11 1983

0:00

12:00

Oct 12 1983

0:00

12:00

Oct 13 1983

0:00
Oct 14 1983

25

Indoor air temp.
26 27 28 29

Output variable

0:00

12:00

Oct 11 1983

0:00

12:00

Oct 12 1983

0:00

12:00

Oct 13 1983

0:00
Oct 14 1983

Figure 1.3: Measurements from an unoccupied test building. The input variables are

(1) solar radiation, (2) ambient air temperature, and (3) heat input. The output
variable is the indoor air temperature.

For this case methods for transfer function modeling as described in Chapter 8 can be used, where the input (explanatory) variables are the solar
radiation, heat input, and outdoor air temperature, while the output (dependent) variable is the indoor air temperature. For the methods in Chapter 8 it
is crucial that all the signals can be classified as either input or output series
related to the system considered.

1.1.4

Predator-prey relationship

This example illustrates a typical multivariate time series, since it is not
possible to classify one of the series as input and the other series as output.
Figure 1.4 shows a widely studied predator-prey case, namely the series of
annually traded skins of muskrat and mink by the Hudson’s Bay Company


“C5967 Final 1st Edition” — 2007/10/24 — 14:13 — page 5 — #15

A first crash course

5

13
12

Muskrat

11

Mink

10

log(skins traded)

14

1.2

1850

1860

1870

1880

1890

1900

1910

Figure 1.4: Annually traded skins of muskrat and mink by the Hudson’s Bay Company
after logarithmic transformation. It is not possible to classify one of the series as
input and the other series as output.

during the 62 year period 1850–1911. In fact the population of muskrats

depends on the population of mink, and the population of mink depends on
the number of muskrats. In such cases both series must be included in a
multivariate time series. This series has been considered in many texts on time
series analysis, and the purpose is to describe in general the relation between
populations of muskrat and mink. Methods for analyzing such multivariate
series are considered in Chapter 9.

1.2

A first crash course

Let us introduce some of the most important concepts of time series analysis
by considering an example where we look for simple models for predicting the
monthly prices of wheat.
In the following, let Pt denote the price of wheat at time (month) t. The
first naive guess would be to say that the price next month is the same as in
this month. Hence, the predictor is
Pt+1|t = Pt .

(1.1)

This predictor is called the naive predictor or the persistent predictor. The
syntax used is short for a prediction (or estimate) of Pt+1 given the observations
Pt , Pt−1 , . . ..
Next month, i.e., at time t + 1, the actual price is Pt+1 . This means that
the prediction error or innovation may be computed as
εt+1 = Pt+1 − Pt+1|t .

(1.2)

“C5967 Final 1st Edition” — 2007/10/24 — 14:13 — page 6 — #16

6

Introduction

By combining Equations (1.1) and (1.2) we obtain the stochastic model for
the wheat price
Pt = Pt−1 + εt

(1.3)

If {εt } is a sequence of uncorrelated zero mean random variables (white noise),
the process (1.3) is called a random walk. The random walk model is very
often seen in finance and econometrics. For this model the optimal predictor
is the naive predictor (1.1).
The random walk can be rewritten as
Pt = εt + εt−1 + · · ·

(1.4)

which shows that the random walk is an integration of the noise, and that the
variance of Pt is unbounded; therefore, no stationary distribution exists. This
is an example of a non-stationary process.
However, it is obvious to try to consider the more general model
Pt = ϕPt−1 + εt

(1.5)

called the AR(1) model (the autoregressive first order model). For this process
a stationary distribution exists for |ϕ| < 1. Notice that the random walk is
obtained for ϕ = 1.
Another candidate for a model for wheat prices is
Pt = ψPt−12 + εt

(1.6)

which assumes that the price this month is explained by the price in the same
month last year. This seems to be a reasonable guess for a simple model, since
it is well known that wheat price exhibits a seasonal variation. (The noise
processes in (1.5) and (1.6) are, despite the notation used, of course, not the
same).
For wheat prices it is obvious that both the actual price and the price in
the same month in the previous year might be used in a description of the
expected price next month. Such a model is obtained if we assume that the
innovation εt in model (1.5) shows an annual variation, i.e., the combined
model is
(Pt − ϕPt−1 ) − ψ(Pt−12 − ϕPt−13 ) = εt .
(1.7)
Models such as (1.6) and (1.7) are called seasonal models, and they are used
very often in econometrics.
Notice, that for ψ = 0 we obtain the AR(1) model (1.5), while for ϕ = 0
the most simple seasonal model in (1.6) is obtained.
By introducing the backward shift operator B by
Bk Pt = Pt−k

(1.8)

“C5967 Final 1st Edition” — 2007/10/24 — 14:13 — page 7 — #17

1.3

Contents and scope of the book

7

the models can be written in a more compact form. The AR(1) model can be
written as (1 − ϕ B)Pt = εt , and the seasonal model in (1.7) as
(1 − ϕ B)(1 − ψ B12 )Pt = εt

(1.9)

If we furthermore introduce the difference operator
∇ = (1 − B)

(1.10)

then the random walk can be written ∇Pt = εt using a very compact notation.
In this book these kinds of notations will be widely used in order to obtain
compact equations.
Given a time series of observed monthly wheat prices, P1 , P2 , . . . , PN , the
model structure can be identified, and, for a given model, the time series can
be used for parameter estimation.
The model identification is most often based on the estimated autocorrelation function, since, as it will be shown in Chapter 6, the autocorrelation
function fulfils the same difference equation as the model. The autocorrelation
function shows how the price is correlated to previous prices; more specifically
the autocorrelation in lag k, called ρ(k), is simply the correlation between Pt
and Pt−k for stationary processes. For the monthly values of the wheat price we

might expect a dominant annual variation and, hence, that the autocorrelation
in lag 12, i.e., ρ(12) is high.
The models above will, of course, be generalized in the book. It is important
to notice that these processes all belong to the more general class of linear
processes, which again is strongly related to the theory of linear systems as
demonstrated in the book.

1.3

Contents and scope of the book

As mentioned previously, this book will concentrate on analyzing and modeling
dynamical systems using statistical methods. The approach taken will focus
on the formulation of appropriate models, their theoretical characteristics,
and on links between the members of the class of stochastic dynamic models
considered. In general, the models considered are all linear and formulated in
discrete time. However, some results related to continuous time models are
provided.
This section describes the contents of the subsequent chapters. In order to
illustrate the relation between various models, some fundamental examples
of the considered models are outlined in the following section. However, for
more rigorous descriptions of the details related to the models we refer to the
following chapters.
In Chapter 2 the concept of multivariate random variables is introduced.
This chapter also introduces necessary fundamental concepts such as the


“C5967 Final 1st Edition” — 2007/10/24 — 14:13 — page 8 — #18

8

Introduction

conditional mean and the linear projection. In general, the chapter provides the
formulas and methods for adapting a second order approach for characterising
random variables. The second order approach limits the attention to first and
second order central moments of the density related to the random variable.
This approach links closely to the very important second order characterisation
of stochastic processes by the autocovariance function in subsequent chapters.
Although time series are realizations of dynamical phenomena, non-dynamical methods are often used. Chapter 3 is devoted to describing static models
applied for time series analysis. However, in the rest of the book dynamical
models will be considered. The methods introduced in Chapter 3 are all linked
to the class of regression models, of which the general linear model is the most
important member. A brief description of the general linear model follows
here.
In the following, let Yt denote the dependent variable and xt = (x1t , x2t , . . . ,
xpt )T a known vector of p explanatory (or independent) variables indexed by
the time t. The general linear model (GLM) is a linear relation between the
variables which can be written
p

Yt =

xkt θk + εt

(1.11)

k=1

where εt is a zero mean random variable, and θ = (θ1 , θ2 , . . . , θp )T is a vector

of the p parameters of the model. Notice that the model (1.11) is a static
model since all the variables refer to the same point in time.
On-line and recursive methods are very important for time series analysis.
These methods provide us with the possibility of always using the most
recent data, e.g., for on-line predictions. Furthermore, changes in time of the
considered phenomena calls for adaptive models, where the parameters typically
are allowed to vary slowly in time. For on-line predictions and control, adaptive
estimation of parameters in relatively simple models is often to be preferred,
since the alternative is a rather complicated model with explicit time-varying
parameters. Adaptive methods for estimating parameters in the general linear
model are considered in Chapter 3. This approach introduces exponential
smoothing, the Holt-Winter procedure, and trend models as important special
cases.
The remaining chapters of the book consider linear systems and appropriate
related dynamical models. A linear system converts an input series to an output
series as illustrated in Figure 1.5.
In Chapter 4 we introduce linear dynamic deterministic systems. In this
chapter, one should note that for random variables capital letters are used
whereas for deterministic variables we use lower case letters.
As a background for Chapter 4, one should be aware that for linear and
time-invariant systems the fundamental relation between the deterministic