Lecture Undergraduate econometrics - Chapter 5: Inference in the simple regression model: Interval estimation, hypothesis testing, and prediction

Chapter 5
Inference in the Simple Regression Model: Interval Estimation, Hypothesis Testing,
and Prediction

Assumptions of the Simple Linear Regression Model
SR1.

yt = β 1 + β 2 xt + et

SR2.

E(et) = 0 ⇔ E[yt] = β1 + β2xt

SR3.

var(et) = σ2 = var(yt)

SR4.

cov(ei, ej) = cov(yi, yj) = 0

SR5.

xt is not random and takes at least two different values

SR6.

et ~ N(0, σ2) ⇔ yt ~ N[(β1 + β2xt), σ2] (optional)

Slide 5.1
Undergraduate Econometrics, 2nd Edition –Chapter 5

If all the above-mentioned assumptions are correct, then the least squares estimators b1
and b2 are normally distributed random variables and have, from Chapter 4.4, normal
distributions with means and variances as follows:


σ 2 ∑ xt2 
b1 ~ N  β1 ,
 T ∑ ( x − x ) 2 
t




σ2
b2 ~ N  β2 ,
 ∑ ( x − x ) 2 
t


From Chapter 4.5 we know that the unbiased estimator of the error variance is as follows:

σˆ

2

∑ eˆ
=

2
t

T −2
Slide 5.2

Undergraduate Econometrics, 2nd Edition –Chapter 5


By replacing the unknown parameter σ2 with this estimator we can estimate the variances
of the least squares estimators and their covariance.
In Chapter 4 you learned how to calculate point estimates of the regression
parameters β1 and β2 using the best, linear unbiased estimation procedure. The estimates
represent an inference about the regression function E(y) = β1 + β2x of the population
from which the sample data was drawn.
In this chapter we introduce the additional tools of statistical inference: interval

estimation, prediction, interval prediction, and hypothesis testing. A prediction is a
forecast of a future value of the dependent variable y, for creating ranges of values,
sometimes called confidence intervals, in which the unknown parameters, or the value
of y, are likely to be located. Hypothesis testing procedures are a means of comparing
conjecture that we as economists might have about the regression parameters to the
information about the parameters contained in a sample of data. Hypothesis tests allow

Slide 5.3
Undergraduate Econometrics, 2nd Edition –Chapter 5


us to say that the data are compatible, or are not compatible, with a particular conjecture,
or hypothesis.

The procedures for interval estimation, prediction, and hypothesis testing, depend
heavily on assumption SR6 of the simple linear regression model, and the resulting
normality of the least squares estimators. If assumption SR6 is not made, then the sample
size must be sufficiently large so that the least squares estimator’s distributions are
approximately normal, in which case the procedures we develop in this chapter are also
approximate. In developing the procedures in this chapter we will be using the normal
distribution, and distributions related to the normal, namely “Student’s” t-distribution and
the chi-square distribution.

Slide 5.4
Undergraduate Econometrics, 2nd Edition –Chapter 5


5.1

Interval Estimation

5.1.1 The Theory
A standard normal random variable that we will use to construct an interval estimator is
based on the normal distribution of the least squares estimator. Consider, for example,
the normal distribution of b2 the least squares estimator of β2, which we denote as


σ2
b2 ~ N  β2 ,
2 


(
x

−
x
)
∑ t


A standardized normal random variable is obtained from b2 by subtracting its mean and
dividing by its standard deviation:

Slide 5.5
Undergraduate Econometrics, 2nd Edition –Chapter 5


Z=

b2 − β2
~ N (0,1)
var(b2 )

(5.1.1)

That is, the standardized random variable Z is normally distributed with mean 0 and
variance 1.
5.5.1a The Chi-Square Distribution
• Chi-square random variables arise when standard normal, N(0,1), random variables are
squared.
If Z1, Z2, ..., Zm denote m independent N(0,1) random variables, then

V = Z12 + Z 22 + K + Z m2 ~ χ (2m )

(5.1.2)

Slide 5.6
Undergraduate Econometrics, 2nd Edition –Chapter 5


The notation V ~ χ (2m ) is read as: the random variable V has a chi-square distribution
with m degrees of freedom. The degrees of freedom parameter m indicates the
number of independent N(0,1) random variables that are squared and summed to form
V.
• The value of m determines the entire shape of the chi-square distribution, and its mean
and variance
E[V ] = E χ (2m )  = m
(5.1.3)
var[V ] = var χ(2m )  = 2m
In Figure 5.1, graphs of the chi-square distribution for various degrees of freedom, m,
are presented.
Slide 5.7
Undergraduate Econometrics, 2nd Edition –Chapter 5


• Since V is formed be squaring and summing m standardized normal [N(0,1)] random
variables, the value of V must be nonnegative, v ≥ 0.
• The distribution has a long tail, or is skewed, to the right.
• As the degrees of freedom m gets larger, the distribution becomes more symmetric and
“bell-shaped.”
• As m gets large, the chi-square distribution converges to, and essentially becomes, a
normal distribution.
5.5.1b The Probability Distribution of σˆ 2
• If SR6 holds, then the random error term et has a normal distribution, et ~ N(0,σ2).

• Standardize the random variable by dividing by its standard deviation so that , et/σ ~
N(0,1).
• The square of a standard normal random variable is a chi-square random variable with
2
.
one degree of freedom, so (et / σ) 2 ~ χ (1)
Slide 5.8
Undergraduate Econometrics, 2nd Edition –Chapter 5


• If all the random errors are independent then
2

2

2

2

e  e  e 
e 
∑t  σt  =  σ1  +  σ2  + L +  σT  ~ χ(2T )

(5.1.4)

Since the true random errors are unobservable we replace them by their sample
counterparts, the least squares residuals eˆt = yt − b1 − b2 xt to obtain

V=

∑ eˆ

2
t

t

σ

2

(T − 2)σˆ 2
=
σ2

(5.1.5)

• The random variable V in Equation (5.1.5) does not have a χ (2T ) distribution because
the least squares residuals are not independent random variables.
Slide 5.9
Undergraduate Econometrics, 2nd Edition –Chapter 5


• All T residuals eˆt = yt − b1 − b2 xt depend on the least squares estimators b1 and b2. It

can be shown that only T – 2 of the least squares residuals are independent in the
simple linear regression model. That is, when multiplied by the constant (T – 2)/σ2 the
random variable σˆ 2 has a chi-square distribution with T – 2 degrees of freedom,
(T − 2)σˆ 2
2

~
V=
χ
(
T − 2)
σ2

(5.1.6)

• We have not established the fact that the chi-square random variable V is statistically
independent of the least squares estimators b1 and b2, but it is. Now we turn our
attention to define a t-random variable.

Slide 5.10
Undergraduate Econometrics, 2nd Edition –Chapter 5


5.1.1c The t-Distribution
• A “t” random variable (no uppercase) is formed by dividing a standard normal, Z ~
N(0,1), random variable by the square root of an independent chi-square random
variable, V ~ χ (2m ) , that has been divided by its degrees of freedom, m.

If Z~N(0,1) and V ~ χ (2m ) , and if Z and V are independent, then

t=

Z
V

~ t( m )

(5.1.7)

m

• The shape of the t-distribution is completely determined by the degrees of freedom
parameter, m, and the distribution is symbolized by t(m).

Slide 5.11
Undergraduate Econometrics, 2nd Edition –Chapter 5


• Figure 5.2 shows a graph of the t-distribution with m = 3 degrees of freedom, relative
to the N(0,1). Note that the t-distribution is less “peaked,” and more spread out than
the N(0,1).
• The t-distribution is symmetric, with mean E[t(m)] = 0 and variance var[t(m)] = m/(m−2).
• As the degrees of freedom parameter m→∞, the t(m) distribution approaches the
standard normal N(0,1).

5.1.1d A Key Result
• From the two random variable V and Z we can form a t-random variable. Recall that a
t-random variable is formed by dividing a standard normal random variable, Z~N(0,1),
by the square root of an independent chi-square random variable, V ~ χ (2m ) , that has
been divided by its degrees of freedom, m. That is Equation (5.1.7).

Slide 5.12
Undergraduate Econometrics, 2nd Edition –Chapter 5


• The t-distribution’s shape is completely determined by the degrees of freedom

parameter, m, and the distribution is symbolized by t(m).

Using Z and V from

Equations (5.1.1) and (5.1.5), respectively, we have

b2 − β2
t=

Z
V

=

T −2

σ2
∑ ( xt − x )2
(T − 2)σ? 2
σ2

T −2

=

b2 − β2
σ2
∑ ( xt − x )2
(5.1.8)

=

b2 − β2
b − β2
~ t (T − 2)
= 2
se(
)
b
ˆ
var(b2 )
2

Slide 5.13
Undergraduate Econometrics, 2nd Edition –Chapter 5


5.1.2 Obtaining Interval Estimates
• If assumptions SR1-SR6 of the simple linear regression model hold, then

t=

bk − βk
~ t(T − 2) , k = 1, 2
se(bk )

(5.1.9)

The random variable t in Equation (5.1.9) will be the basis for interval estimation and
hypothesis testing in the simple linear regression model.

• Equation (5.1.9), for k = 2, is

t=

b2 − β2
~ t(T − 2)
se(b2 )

(5.1.10)

where
Slide 5.14
Undergraduate Econometrics, 2nd Edition –Chapter 5


σˆ 2
? b2 ) =
var(
∑ ( xt − x )2

and se(b2 ) = var(b2 )

• Using Table 2 inside the front cover of the book we can find critical values tc from a
t(m) distribution such that

P(t ≥ tc ) = P(t ≤ −tc ) =

α
2

where α is a probability value often taken to be α = .01 or α = .05. The values tc and tc are depicted in Figure 5.3. Each of the shaded “tail” areas contains α/2 of the
probability, so that 1-α of the probability is contained in the center portion.
Consequently, we can make the probability statement
Slide 5.15
Undergraduate Econometrics, 2nd Edition –Chapter 5


P ( −t c ≤ t ≤ t c ) = 1 − α

(5.1.11)

• Now, we put all these pieces together to create a procedure for interval estimation.
Substitute t from Equation (5.1.10) into Equation (5.1.11) to obtain

P[ −tc ≤

b2 − β2
≤ tc ] = 1 − α
se(b2 )

(5.1.12)

Simplify this expression to obtain
P[b2 − tcse(b2 ) ≤ β2 ≤ b2 + tcse(b2 )] = 1 − α

(5.1.13)

Slide 5.16
Undergraduate Econometrics, 2nd Edition –Chapter 5

In the interval endpoint, b2 – tcse(b2) and b2 + tcse(b2), both b2 and se(b2) are random
variables, since their value are not known until a sample of data is drawn.
• The probability endpoints of the interval define an interval estimator of β2. The
probability statement in Equation (5.1.13) says that the interval b2 ± tcse(b2 ) , with
random endpoints, has probability 1–α of containing the true but unknown parameter
β2. This interval estimation procedure and its properties are established based on
model assumptions SR1-SR6 and may be applied to any sample of data that we might
obtain. When b2 and se(b2) in Equation (5.1.13) are estimated values (numbers), based
on a sample of data, then b2 ± tcse(b2 ) is called a (1–α)×(100%) interval estimate of β2,
or, equivalently, it is called a (1–α)×(100%) confidence interval.
• The properties of the random interval estimator are based on the notion of repeating
sampling. If we were to select many random samples of size T, compute the least
squares estimate b2 and its standard error se(b2) for each sample, and then construct
Slide 5.17
Undergraduate Econometrics, 2nd Edition –Chapter 5


the interval estimate b2 ± tcse(b2 ) for each sample, then (1–α)×(100%) of all the
intervals constructed would contain the true parameter β2. This we know before any
data are actually collected.
• Any one interval estimate, based on one sample of data, may or may not contain the
true parameter β2, and since β2 is unknown, we will never know if it does or not.
When confidence intervals are discussed, remember that our confidence is in the
procedure used to construct the interval estimate; it is not in any one interval estimate
calculated from a sample of data.

Slide 5.18
Undergraduate Econometrics, 2nd Edition –Chapter 5

5.1.3 The Repeated Sampling Context
• Table 5.1, using the ten samples of data, reports the least squares estimates, the
estimates of σ2, and the estimated errors from each sample and Table 5.2 presents the
95 percent confidence interval estimates for the parameters β1 and β2.
• Sampling variability causes the center of each of the interval estimates to change with
the location of the least squares estimates, and it causes the widths of the intervals to
change with the standard errors.

Table 5.1 Least Squares Estimates from 10 Random Samples
n

b1

se(b1)

b2

se(b2)

σˆ 2

1

51.1314

27.4260

0.1442

0.0378

2193.4597

2

61.2045

24.9177

0.1286

0.0344

1810.5972

3

40.7882

17.6670

0.1417

0.0244

910.1835

Slide 5.19
Undergraduate Econometrics, 2nd Edition –Chapter 5

4

80.1396

23.8146

0.0886

0.0329

1653.8324

5

31.0110

22.8126

0.1669

0.0315

1517.5837

6

54.3099

26.9317

0.1086

0.0372

2115.1085

7

69.6749

19.2903

0.1003

0.0266

1085.1312

8

71.1541

26.1807

0.1009

0.0361

1998.7880

9

18.8290

22.4234

0.1758

0.0309

1466.2541

10

36.1433

23.5531

0.1626

0.0325

1617.7087

Table 5.2 Interval Estimates from 10 Random Samples
n
b1 − tcse(b1 )
b1 + tcse(b1 )

b2 − tcse(b2 )
b2 + tcse(b2 )
1
-4.3897 106.6524
0.0676
0.2207
2
10.7612 111.6479
0.0590
0.1982
3
5.0233
76.5531
0.0923
0.1910
4
31.9294 128.3498
0.0221
0.1551
5
77.1926
0.1032
0.2306
−15.1706
6
0.0334
0.1838
−0.2105 108.8303
7
30.6237 108.7261

0.0464
0.1542
Slide 5.20
Undergraduate Econometrics, 2nd Edition –Chapter 5


8
9
10

18.1541
−26.5649
−11.5374

124.1542
64.2229
83.8240

0.0278
0.1131
0.0968

0.1741
0.2384
0.2284

• Since 95 percent of all interval estimates constructed this way contain the true
parameter values, we would expect perhaps nine or ten of these intervals to contain the
true but unknown parameters.
• We have used the least squares estimators to obtain, from a sample of data, point

estimates that are “best guesses” of unknown parameters. The estimated variance
ˆ bk ) , for k = 1 or 2, and its square root var(
ˆ bk ) = se(bk ) , provide information
var(
about the sampling variability of the least squares estimator from one sample to
another.

• Interval estimators combine point estimation with estimation of sampling variability to
provide a range of values in which the unknown parameters might fall. Interval
estimates are a convenient way to inform others about the estimated location of the
Slide 5.21
Undergraduate Econometrics, 2nd Edition –Chapter 5


unknown parameter and also provide information about the sampling variability of the
least squares estimator, through se(bk), and the “level of confidence” 1–α.

• When the sampling variability of the least squares estimator is relatively small, then
the interval estimates will be relatively narrow, implying that the least squares
estimates are “reliable.” On the other hand, if the least squares estimators suffer from
large sampling variability, then the interval estimates will be wide, implying that the
least squares estimates are “unreliable.”

5.1.4 An Illustration

• For the food expenditure data in Table 3.1, where T = 40 and the degrees of freedom
are T – 2 = 38, if we let α = .05, Equation (5.3.13) becomes
P[b2 − 2.024se(b2 ) ≤ β2 ≤ b2 + 2.024se(b2 )] = .95

(5.1.14)

Slide 5.22
Undergraduate Econometrics, 2nd Edition –Chapter 5


• The critical value tc = 2.024, which is appropriate for α = .05 and 38 degrees of
freedom, can be found in the Table 2 at the end of the book.

• It can be computed exactly with a software package.
• To construct an interval estimate for β2 we use the least squares estimate b2 = .1283
which has the standard error
ˆ b2 ) = 0.0009326 = 0.0305
se(b2 ) = var(

• Substituting these values into Equation (5.1.14) we obtain a “95 percent confidence
interval estimate” for β2:
b2 ± tcse(b2 ) = .1283 ± 2.024(.0305) = [.0666, .1900]

Slide 5.23
Undergraduate Econometrics, 2nd Edition –Chapter 5


• Is β2 in the interval [.0666, .1900]? We do not know, and will never know. What we
do know is that when the procedure we used is applied to many random samples of
data from the same population, then 95 percent of all the interval estimates constructed
using this procedure will contain the true parameter.

The interval estimation

procedure “works” 95 percent of the time.

• All we can say about the interval estimate based on our one sample is that, given the
reliability of the procedure, we would be “surprised” if β2 was not in the interval
[.0666, .1900]. Since this interval estimate contains no random quantities we cannot
make probabilistic statements about it such as “There is a .95 probability that β2 is in
the interval [.0666, .1900].”
• What is the usefulness of an interval estimate of β2? When reporting regression results
we always give a point estimate, such as b2 = .1283. However, the point estimate
alone gives no sense of its reliability. Interval estimates incorporate both the point
estimate and the standard error of the estimate, which is a measure of the variability of
Slide 5.24
Undergraduate Econometrics, 2nd Edition –Chapter 5


the least squares estimator. If an interval estimate is wide (implying a large standard
error), there is not much information in the sample about β2. A narrow interval
estimate suggests that we have learned more about β2.

5.2

Hypothesis Testing

Slide 5.25
Undergraduate Econometrics, 2nd Edition –Chapter 5