advanced engineering mathematics – mathematics

<span class='text_page_counter'>(1)</span><div class='page_container' data-page=1>

<b>Introduction to Time Series Analysis. Lecture 13.</b>

<b>Peter Bartlett</b>

Last lecture:

1. Yule-Walker estimation

</div>
<span class='text_page_counter'>(2)</span><div class='page_container' data-page=2>

<b>Introduction to Time Series Analysis. Lecture 13.</b>

1. Review: Maximum likelihood estimation

2. Computational simplifications: un/conditional least squares
3. Diagnostics

4. Model selection

</div>
<span class='text_page_counter'>(3)</span><div class='page_container' data-page=3>

<b>Review: Maximum likelihood estimator</b>

Suppose that X₁, X₂, . . . , X_n is drawn from a zero mean Gaussian
ARMA(p,q) process. The likelihood of parameters φ ∈ _Rp, θ ∈ _Rq,

σ_w2 ∈ _R+ is defined as the density of X = (X1, X2, . . . , Xn)′ under the
Gaussian model with those parameters:

L(φ, θ, σ_w2 ) = 1
(2π)n/2 _|_Γ

n|1/2

exp

−1

2X

′_Γ−1

n X

,

where |A| denotes the determinant of a matrix A, and Γn is the
variance/covariance matrix of X with the given parameter values.

</div>
<span class='text_page_counter'>(4)</span><div class='page_container' data-page=4>

<b>Maximum likelihood estimation: Simplifications</b>

<i>We can simplify the likelihood by expressing it in terms of the innovations.</i>
Since the innovations are linear in previous and current values, we can write






X₁
..
.
X_n







| {z }

X
= C






X₁ − X₁0

..
.

X_n − Xn−1

n






| {z }

U

where C is a lower triangular matrix with ones on the diagonal.
Take the variance/covariance of both sides to see that

Γ_n = CDC′ _where _D _{= diag(P}0

</div>
<span class='text_page_counter'>(5)</span><div class='page_container' data-page=5>

<b>Maximum likelihood estimation</b>

Thus, |Γ_n| = |C|2P₁0 · · ·P_nn−1 = P₁0 · · ·P_nn−1 and

X′_Γ−1

n X = U′C′Γn−1CU = U′C′C−TD−1C−1CU = U′D−1U.
So we can rewrite the likelihood as

L(φ, θ, σ_w2 ) = 1
(2π)n_P0

1 · · ·Pnn−1

1/2 exp −
1
2

n

X

i=1

(Xi − X_ii−1)2/P_ii−1

!

= 1

(2πσ_w2 )nr₁0 · · ·rnn−11

/2 exp

−S(φ, θ)

2σ2

w

,

where ri−1

i = Pii−1/σw2 and

S(φ, θ) =

n

X

i=1

X_i − Xi−1

i

2
ri−1

i

</div>
<span class='text_page_counter'>(6)</span><div class='page_container' data-page=6>

<b>Maximum likelihood estimation</b>

The log likelihood of φ, θ, σ_w2 is

l(φ, θ, σ_w2 ) = log(L(φ, θ, σ_w2 ))
= −n

2 log(2πσ
2

w) −

1
2

n

X

i=1

logri−1

i −

S(φ, θ)
2σ2

w

.

Differentiating with respect to σ_w2 shows that the MLE ( ˆφ,θ,ˆ σˆ_w2 ) satisfies

n
2ˆσ2

w

= S( ˆφ,θ)ˆ
2ˆσ4

w

⇔ σˆ_w2 = S( ˆφ, θ)ˆ

n ,

and φ,ˆ θˆminimize log S( ˆφ,θ)ˆ
n
!
+ 1
n
n
X
i=1

logri−1

</div>
<span class='text_page_counter'>(7)</span><div class='page_container' data-page=7>

<b>Summary: Maximum likelihood estimation</b>

The MLE ( ˆφ,θ,ˆ σˆ_w2 ) satisfies

ˆ

σ_w2 = S( ˆφ, θ)ˆ

n ,

and φ,ˆ θˆminimize log S( ˆφ,θ)ˆ
n

!

+ 1
n

n

X

i=1

logri−1

i ,

where ri−1

i = Pii−1/σw2 and

S(φ, θ) =

n

X

i=1

X_i − Xi−1

i

2
ri−1

i

</div>
<span class='text_page_counter'>(8)</span><div class='page_container' data-page=8>

<b>Maximum likelihood estimation</b>

Minimization is done numerically (e.g., Newton-Raphson).
Computational simplifications:

• <i>Unconditional least squares. Drop the</i> log r_ii−1 terms.

• <i>Conditional least squares. Also approximate the computation of</i> xi−1

i by
dropping initial terms in S. e.g., for AR(2), all but the first two terms in S

depend linearly on φ₁, φ₂, so we have a least squares problem.
The differences diminish as sample size increases. For example,

Pt−1

t → σw2 so rtt−1 → 1, and thus n−1

P

</div>
<span class='text_page_counter'>(9)</span><div class='page_container' data-page=9>

<b>Review: Maximum likelihood estimation</b>

For an ARMA(p,q) process, the MLE and un/conditional least
squares estimators satisfy



ˆ

φ
ˆ
θ

 −

 φ
θ


 ∼ AN



0,

σ_w2
n

 Γφφ Γφθ
Γ_θφ Γ_θθ,



−1


,
where


 Γφφ Γφθ
Γ_θφ Γ_θθ,


 = Cov((X, Y ),(X, Y )),
X = (X₁, . . . , X_p)′ _φ(B_)X

t = Wt,

Y = (Y₁, . . . , Y_p)′ _θ(B_)Y

</div>
<span class='text_page_counter'>(10)</span><div class='page_container' data-page=10>

<b>Introduction to Time Series Analysis. Lecture 13.</b>

1. Review: Maximum likelihood estimation

2. Computational simplifications: un/conditional least squares
3. Diagnostics

4. Model selection

</div>
<span class='text_page_counter'>(11)</span><div class='page_container' data-page=11>

<b>Building ARMA models</b>

1. Plot the time series.

Look for trends, seasonal components, step changes, outliers.
2. Nonlinearly transform data, if necessary

3. Identify preliminary values of p, and q.
4. Estimate parameters.

5. Use diagnostics to confirm residuals are white/iid/normal.

</div>
<span class='text_page_counter'>(12)</span><div class='page_container' data-page=12>

<b>Diagnostics</b>

How do we check that a model fits well?
The residuals (innovations, x_t − xt−1

t ) should be white.
<i>Consider the standardized innovations,</i>

e_t = xt − xˆ

t−1

t

q
ˆ
Pt−1

t

.

This should behave like a mean-zero, unit variance, iid sequence.
• Check a time plot

</div>
<span class='text_page_counter'>(13)</span><div class='page_container' data-page=13>

<b>Testing i.i.d.: Turning point test</b>

{X_t} i.i.d. implies that X_t, X_t₊₁ and X_t₊₂ are equally likely to occur in
any of six possible orders:

0 5 10 15 20

0.5
1
1.5
2
2.5
3
3.5

(provided Xt, Xt+1, Xt+2 are distinct).

</div>
<span class='text_page_counter'>(14)</span><div class='page_container' data-page=14>

<b>Testing i.i.d.: Turning point test</b>

Define T = |{t : Xt, Xt+1, Xt+2 is a turning point}|.

ET = (n − 2)2/3.

Can show T ∼ AN(2n/3,8n/45).

Reject (at 5% level) the hypothesis that the series is i.i.d. if

T −
2n

3

> 1.96
r

8n
45.

</div>
<span class='text_page_counter'>(15)</span><div class='page_container' data-page=15>

<b>Testing i.i.d.: Difference-sign test</b>

S = |{i : X_i > X_i₋₁}| = |{i : (∇X)_i > 0}|.

ES = n − 1
2 .

Can show S ∼ AN(n/2, n/12).

Reject (at 5% level) the hypothesis that the series is i.i.d. if

S −
n

2

> 1.96
r

n
12.

Tests for trend.

</div>
<span class='text_page_counter'>(16)</span><div class='page_container' data-page=16>

<b>Testing i.i.d.: Rank test</b>

N = |{(i, j) : X_i > X_j and i > j}|.

EN = n(n − 1)
4 .

Can show N ∼ AN(n2/4, n3/36).

Reject (at 5% level) the hypothesis that the series is i.i.d. if

N −
n2

4

> 1.96
r

n3
36.

</div>
<span class='text_page_counter'>(17)</span><div class='page_container' data-page=17>

<b>Testing if an i.i.d. sequence is Gaussian: qq plot</b>

Plot the pairs (m₁, X₍₁₎), . . . ,(m_n, X₍_n₎),
where mj = EZ(j),

Z₍₁₎ < · · · < Z₍_n₎ are order statistics from N(0,1) sample of size n, and

X₍₁₎ < · · · < X₍_n₎ are order statistics of the series X1, . . . , Xn.
<i>Idea: If</i> Xi ∼ N(µ, σ2), then

EX₍_j₎ = µ + σmj,
so (m_j, X₍_j₎) <i>should be linear.</i>

</div>
<span class='text_page_counter'>(18)</span><div class='page_container' data-page=18>

<b>Introduction to Time Series Analysis. Lecture 13.</b>

1. Review: Maximum likelihood estimation

2. Computational simplifications: un/conditional least squares
3. Diagnostics

4. Model selection

</div>

<!--links-->