Analysis of messy data volume III analysis of covariance

Analysis
of Messy Data
VOLUME III:
ANALYSIS OF COVARIANCE

George A. Milliken
Dallas E. Johnson

CHAPMAN & HALL/CRC
A CRC Pr ess Compan y
Boca Raton London Ne w York Washington, D.C.


C0317fm frame Page 4 Monday, July 16, 2001 7:52 AM

Library of Congress Cataloging-in-Publication Data
Milliken, George A., 1943–
Analysis of messy data / George A. Milliken, Dallas E. Johnson.
2 v. : ill. ; 24 cm.
Includes bibliographies and indexes.
Contents: v. 1. Designed experiments -- v. 2. Nonreplicated
experiments.
Vol. 2 has imprint: New York : Van Nostrand Reinhold.
ISBN 0-534-02713-X (v. 1) : $44.00 -- ISBN 0-442-24408-8 (v. 2)
1. Analysis of variance. 2. Experimental design. 3. Sampling
(Statistics) I. Johnson, Dallas E., 1938– . II. Title.
QA279 .M48 1984
519.5′352--dc19

84-000839

This book contains information obtained from authentic and highly regarded sources. Reprinted material
is quoted with permission, and sources are indicated. A wide variety of references are listed. Reasonable
efforts have been made to publish reliable data and information, but the author and the publisher cannot
assume responsibility for the validity of all materials or for the consequences of their use.
Apart from any fair dealing for the purpose of research or private study, or criticism or review, as permitted
under the UK Copyright Designs and Patents Act, 1988, this publication may not be reproduced, stored
or transmitted, in any form or by any means, electronic or mechanical, including photocopying, microfilming, and recording, or by any information storage or retrieval system, without the prior permission
in writing of the publishers, or in the case of reprographic reproduction only in accordance with the
terms of the licenses issued by the Copyright Licensing Agency in the UK, or in accordance with the
terms of the license issued by the appropriate Reproduction Rights Organization outside the UK.
The consent of CRC Press LLC does not extend to copying for general distribution, for promotion, for
creating new works, or for resale. Specific permission must be obtained in writing from CRC Press LLC
for such copying.
Direct all inquiries to CRC Press LLC, 2000 N.W. Corporate Blvd., Boca Raton, Florida 33431.
Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are
used only for identification and explanation, without intent to infringe.

Visit the CRC Press Web site at www.crcpress.com
© 2002 by Chapman & Hall/CRC
No claim to original U.S. Government works
International Standard Book Number 1-584-88083-X
Library of Congress Card Number 84-000839
Printed in the United States of America 1 2 3 4 5 6 7 8 9 0
Printed on acid-free paper


C0317fm frame Page 5 Monday, June 25, 2001 1:04 PM

Table of Contents
Chapter 1

Introduction to the Analysis of Covariance

1.1
Introduction
1.2
The Covariate Adjustment Process
1.3
A General AOC Model and the Basic Philosophy
References
Chapter 2

One-Way Analysis of Covariance — One Covariate in a
Completely Randomized Design Structure

2.1
2.2
2.3
2.4

The Model
Estimation
Strategy for Determining the Form of the Model
Comparing the Treatments or Regression Lines
2.4.1 Equal Slopes Model
2.4.2 Unequal Slopes Model-Covariate by Treatment Interaction
2.5
Confidence Bands about the Difference of Two Treatments
2.6
Summary of Strategies

2.7
Analysis of Covariance Computations via the SAS® System
2.7.1 Using PROC GLM and PROC MIXED
2.7.2 Using JMP®
2.8
Conclusions
References
Exercise
Chapter 3
3.1
3.2

3.3
3.4
3.5
3.6

Examples: One-Way Analysis of Covariance — One Covariate
in a Completely Randomized Design Structure

Introduction
Chocolate Candy — Equal Slopes
3.2.1 Analysis Using PROC GLM
3.2.2 Analysis Using PROC MIXED
3.2.3 Analysis Using JMP®
Exercise Programs and Initial Resting Heart Rate — Unequal Slopes
Effect of Diet on Cholesterol Level: An Exception to the Basic
Analysis of Covariance Strategy
Change from Base Line Analysis Using Effect of Diet on Cholesterol
Level Data

Shoe Tread Design Data for Exception to the Basic Strategy

© 2002 by CRC Press LLC


C0317fm frame Page 6 Monday, June 25, 2001 1:04 PM

3.7

Equal Slopes within Groups of Treatments and Unequal Slopes
between Groups
3.8
Unequal Slopes and Equal Intercepts — Part 1
3.9
Unequal Slopes and Equal Intercepts — Part 2
References
Exercises
Chapter 4

Multiple Covariates in a One-Way Treatment Structure in a
Completely Randomized Design Structure

4.1
4.2
4.3
4.4
4.5

Introduction
The Model

Estimation
Example: Driving A Golf Ball with Different Shafts
Example: Effect of Herbicides on the Yield of Soybeans — Three
Covariates
4.6 Example: Models That Are Quadratic Functions of the Covariate
4.7 Example: Comparing Response Surface Models
Reference
Exercises
Chapter 5

Two-Way Treatment Structure and Analysis of Covariance in
a Completely Randomized Design Structure

5.1
5.2
5.3

Introduction
The Model
Using the SAS® System
5.3.1 Using PROC GLM and PROC MIXED
5.3.2 Using JMP®
5.4
Example: Average Daily Gains and Birth Weight — Common Slope
5.5
Example: Energy from Wood of Different Types of Trees — Some
Unequal Slopes
5.6
Missing Treatment Combinations
5.7

Example: Two-Way Treatment Structure with Missing Cells
5.8
Extensions
Reference
Exercises
Chapter 6
6.1
6.2
6.3
6.4
6.5
6.6

Beta-Hat Models

Introduction
The Beta-Hat Model and Analysis
Testing Equality of Parameters
Complex Treatment Structures
Example: One-Way Treatment Structure
Example: Two-Way Treatment Structure

© 2002 by CRC Press LLC


C0317fm frame Page 7 Monday, June 25, 2001 1:04 PM

6.7
Summary
Exercises

Chapter 7

Variable Selection in the Analysis of Covariance Model

7.1
Introduction
7.2
Procedure for Equal Slopes
7.3
Example: One-Way Treatment Structure with Equal Slopes Model
7.4
Some Theory
7.5
When Slopes are Possibly Unequal
References
Exercises
Chapter 8

Comparing Models for Several Treatments

8.1
Introduction
8.2
Testing Equality of Models for a One-Way Treatment Structure
8.3
Comparing Models for a Two-Way Treatment Structure
8.4
Example: One-Way Treatment Structure with One Covariate
8.5
Example: One-Way Treatment Structure with Three Covariates

8.6
Example: Two-Way Treatment Structure with One Covariate
8.7
Discussion
References
Exercises
Chapter 9

Two Treatments in a Randomized Complete Block Design
Structure

9.1
9.2
9.3
9.4
9.5
9.6
9.7
9.8

Introduction
Complete Block Designs
Within Block Analysis
Between Block Analysis
Combining Within Block and Between Block Information
Determining the Form of the Model
Common Slope Model
Comparing the Treatments
9.8.1 Equal Slopes Models
9.8.2 Unequal Slopes Model

9.9
Confidence Intervals about Differences of Two Regression Lines
9.9.1 Within Block Analysis
9.9.2 Combined Within Block and Between Block Analysis
9.10 Computations for Model 9.1 Using the SAS® System
9.11 Example: Effect of Drugs on Heart Rate
9.12 Summary
References
Exercises

© 2002 by CRC Press LLC


C0317fm frame Page 8 Monday, June 25, 2001 1:04 PM

Chapter 10 More Than Two Treatments in a Blocked Design Structure
10.1
10.2
10.3

Introduction
RCB Design Structure — Within and Between Block Information
Incomplete Block Design Structure — Within and Between Block
Information
10.4 Combining Between Block and Within Block Information
10.5 Example: Five Treatments in RCB Design Structure
10.6 Example: Balanced Incomplete Block Design Structure with Four
Treatments
10.7 Example: Balanced Incomplete Block Design Structure with Four
Treatments Using JMP®

10.8 Summary
References
Exercises
Chapter 11 Covariate Measured on the Block in RCB and Incomplete
Block Design Structures
11.1
11.2
11.3
11.4
11.5
11.6

Introduction
The Within Block Model
The Between Block Model
Combining Within Block and Between Block Information
Common Slope Model
Adjusted Means and Comparing Treatments
11.6.1 Common Slope Model
11.6.2 Non-Parallel Lines Model
11.7 Example: Two Treatments
11.8 Example: Four Treatments in RCB
11.9 Example: Four Treatments in BIB
11.10 Summary
References
Exercises
Chapter 12 Random Effects Models with Covariates
12.1
12.2
12.3

12.4

Introduction
The Model
Estimation of the Variance Components
Changing Location of the Covariate Changes the Estimates of the
Variance Components
12.5 Example: Balanced One-Way Treatment Structure
12.6 Example: Unbalanced One-Way Treatment Structure
12.7 Example: Two-Way Treatment Structure
12.8 Summary
References
Exercises

© 2002 by CRC Press LLC


C0317fm frame Page 9 Monday, June 25, 2001 1:04 PM

Chapter 13 Mixed Models
13.1
13.2
13.3
13.4

Introduction
The Matrix Form of the Mixed Model
Fixed Effects Treatment Structure
Estimation of Fixed Effects and Some Small Sample Size
Approximations

13.5 Fixed Treatments and Locations Random
13.6 Example: Two-Way Mixed Effects Treatment Structure in a CRD
13.7 Example: Treatments are Fixed and Locations are Random with a
RCB at Each Location
References
Exercises
Chapter 14 Analysis of Covariance Models with Heterogeneous Errors
14.1
14.2
14.3

Introduction
The Unequal Variance Model
Tests for Homogeneity of Variances
14.3.1 Levene’s Test for Equal Variances
14.3.2 Hartley’s F-Max Test for Equal Variances
14.3.3 Bartlett’s Test for Equal Variances
14.3.4 Likelihood Ratio Test for Equal Variances
14.4 Estimating the Parameters of the Regression Model
14.4.1 Least Squares Estimation
14.4.2 Maximum Likelihood Methods
14.5 Determining the Form of the Model
14.6 Comparing the Models
14.6.1 Comparing the Nonparallel Lines Models
14.6.2 Comparing the Parallel Lines Models
14.7 Computational Issues
14.8 Example: One-Way Treatment Structure with Unequal Variances
14.9 Example: Two-Way Treatment Structure with Unequal Variances
14.10 Example: Treatments in Multi-location Trial
14.11 Summary

References
Exercises
Chapter 15 Analysis of Covariance for Split-Plot and Strip-Plot Design
Structures
15.1
15.2
15.3
15.4

Introduction
Some Concepts
Covariate Measured on the Whole Plot or Large Size of Experimental
Unit
Covariate is Measured on the Small Size of Experimental Unit

© 2002 by CRC Press LLC


C0317fm frame Page 10 Monday, June 25, 2001 1:04 PM

15.5

Covariate is Measured on the Large Size of Experimental Unit and a
Covariate is Measured on the Small Size of Experimental Unit
15.6 General Representation of the Covariate Part of the Model
15.6.1 Covariate Measured on Large Size of Experimental Unit
15.6.2 Covariate Measured on the Small Size of Experimental Units
15.6.3 Summary of General Representation
15.7 Example: Flour Milling Experiment — Covariate Measured on the
Whole Plot

15.8 Example: Cookie Baking
15.9 Example: Teaching Methods with One Covariate Measured on the
Large Size Experimental Unit and One Covariate Measured on the
Small Size Experimental Unit
15.10 Example: Comfort Study in a Strip-Plot Design with Three Sizes of
Experimental Units and Three Covariates
15.11 Conclusions
References
Exercises
Chapter 16 Analysis of Covariance for Repeated Measures Designs
16.1
16.2
16.3
16.4

Introduction
The Covariance Part of the Model — Selecting R
Covariance Structure of the Data
Specifying the Random and Repeated Statements for PROC MIXED
of the SAS® System
16.5 Selecting an Adequate Covariance Structure
16.6 Example: Systolic Blood Pressure Study with Covariate Measured
on the Large Size Experimental Unit
16.7 Example: Oxide Layer Development Experiment with Three Sizes
of Experimental Units Where the Repeated Measure is at the Middle
Size of Experimental Unit and the Covariate is Measured on the
Small Size Experimental Unit
16.8 Conclusions
References
Exercises

Chapter 17 Analysis of Covariance for Nonreplicated Experiments
17.1
17.2
17.3.
17.4
17.5
17.6
17.7
17.8

Introduction
Experiments with A Single Covariate
Experiments with Multiple Covariates
Selecting Non-null and Null Partitions
Estimating the Parameters
Example: Milling Flour Using Three Factors Each at Two Levels
Example: Baking Bread Using Four Factors Each at Two Levels
Example: Hamburger Patties with Four Factors Each at Two Levels

© 2002 by CRC Press LLC


C0317fm frame Page 11 Monday, June 25, 2001 1:04 PM

17.9

Example: Strength of Composite Material Coupons with Two
Covariates
17.10 Example: Effectiveness of Paint on Bricks with Unequal Slopes
17.11 Summary

References
Exercises
Chapter 18 Special Applications of Analysis of Covariance
18.1
18.2
18.3
18.4

Introduction
Blocking and Analysis of Covariance
Treatments Have Different Ranges of the Covariate
Nonparametric Analysis of Covariance
18.4.1 Heart Rate Data from Exercise Programs
18.4.2 Average Daily Gain Data from a Two-Way Treatment
Structure
18.5 Crossover Design with Covariates
18.6 Nonlinear Analysis of Covariance
18.7 Effect of Outliers
References
Exercises

© 2002 by CRC Press LLC


C0317fm frame Page 13 Monday, June 25, 2001 1:04 PM

Preface
Analysis of covariance is a statistical procedure that enables one to incorporate
information about concomitant variables into the analysis of a response variable.
Sometimes this is done in an attempt to reduce experimental error. Other times it is

done to better understand the phenomenon being studied. The approach used in this
book is that the analysis of covariance model is described as a method of comparing
a series of regression models — one for each of the levels of a factor or combinations
of levels of factors being studied. Since covariance models are regression models,
analysts can use all of the methods of regression analysis to deal with problems
such as lack of fit, outliers, etc. The strategies described in this book will enable the
reader to appropriately formulate and analyze various kinds of covariance models.
When covariates are measured and incorporated into the analysis of a response
variable, the main objective of analysis of covariance is to compare treatments or
treatment combinations at common values of the covariates. This is particularly true
when the experimental units assigned to each of the treatment combinations may
have differing values of the covariates. Comparing treatments is dependent on the
form of the covariance model and thus care must be taken so that mistakes are not
made when drawing conclusions.
The goal of this book is to present the structure and philosophy for using the
analysis of covariance by including descriptions of methodologies, illustrating the
methodologies by analyzing numerous data sets, and occasionally furnishing some
theory when required. Our aim is to provide data analysts with tools for analyzing
data with covariates and to enable them to appropriately interpret the results.
Some of the methods and techniques described in this book are not available in
other books, but two issues of Biometrics (1957, Volume 13, Number 3, and 1982,
Volume 38, Number 3) were dedicated to the topic of analysis of covariance. The
topics presented are among those that we, as consulting statisticians, have found to
be most helpful in analyzing data when covariates are available for possible inclusion
in the analysis.
Readers of this book will learn how to:
• Formulate appropriate analysis of covariance models
• Simplify analysis of covariance models
• Compare levels of a factor or of levels of combinations of factors when
the model involves covariates

• Construct and analyze a model with two or more factors in the treatment
structure
• Analyze two-way treatment structures with missing cells
• Compare models using the beta-hat model
• Perform variable selection within the analysis of covariance model

© 2002 by CRC Press LLC


C0317fm frame Page 14 Monday, June 25, 2001 1:04 PM

• Analyze models with blocking in the design structure and use combined
intra-block and inter-block information about the slopes of the regression
models
• Use random statements in PROC MIXED to specify random coefficient
regression models
• Carry out the analysis of covariance in a mixed model framework
• Incorporate unequal treatment variances into the analysis
• Specify the analysis of covariance models for split-plot, strip-plot and
repeated measures designs both in terms of the regression models and the
covariance structures of the repeated measures
• Incorporate covariates into the analysis of nonreplicated experiments, thus
extending some of the results in Analysis of Messy Data, Volume II
The last chapter consists of a collection of examples that deal with (1) using the
covariate to form blocks, (2) crossover designs, (3) nonparametric analysis of covariance, (4) using a nonlinear model for the covariate model, and (5) the process of
examining mixed analysis of covariance models for possible outliers.
The approach used in this book is similar to that used in the first two volumes.
Each topic is covered from a practical viewpoint, emphasizing the implementation
of the methods much more than the theory behind the methods. Some theory has
been presented for some of the newer methodologies. The book utilized the procedures of the SAS® system and JMP® software packages to carry out the computations

and few computing formulae are presented. Either SAS® system code or JMP® menus
are presented for the analysis of the data sets in the examples. The data in the
examples (except for those using chocolate chips) were generated to simulate real
world applications that we have encountered in our consulting experiences.
This book is intended for everyone who analyzes data. The reader should have
a knowledge of analysis of variance and regression analysis as well as basic statistical
ideas including randomization, confidence intervals, and hypothesis testing. The first
four chapters contain the information needed to form a basic philosophy for using
the analysis of covariance with a one-way treatment structure and should be read
by everyone. As one progresses through the book, the topics become more complex
by going from designs with blocking to split-plot and repeated measures designs.
Before reading about a particular topic in the later chapters, read the first four
chapters. Knowledge of Chapters 13 and 14 from Analysis of Messy Data, Volume I:
Designed Experiments would be useful for understanding the part of Chapter 5
involving missing cells. The information in Chapters 4 through 9 of Analysis of
Messy Data, Volume II: Nonreplicated Experiments is useful for comprehending the
topics discussed in Chapter 17.
This book is the culmination of more than 25 years of writing. The earlier
editions of this manuscript were slanted toward providing an appropriate analysis
of split-plot type designs by using fixed effects software such as PROC GLM of the
SAS® system. With the development of mixed models software, such as PROC
MIXED of the SAS® system and JMP®, the complications of the analysis of splitplot type designs disappeared and thus enabled the manuscript to be completed
without including the difficult computations that are required when using fixed

© 2002 by CRC Press LLC


C0317fm frame Page 15 Monday, June 25, 2001 1:04 PM

effects software. Over the years, several colleagues made important contributions.

Discussions with Shie-Shien Yang were invaluable for the development of the variable selection process described in Chapter 7. Vicki Landcaster and Marie Loughin
read some of the earlier versions and provided important feedback. Discussions with
James Schwenke, Kate Ash, Brian Fergen, Kevin Chartier, Veronica Taylor, and
Mike Butine were important for improving the chapters involving combining intraand inter-block information and the strategy for the analysis of repeated measures
designs. Finally, we cannot express enough our thanks to Jane Cox who typed many
of the initial versions of the chapters. If it were not for Jane’s skills with the word
processor, the task of finishing this book would have been much more difficult.
We dedicate this volume to all who have made important contributions to our
personal and professional lives. This includes our wives, Janet and Erma Jean, our
children, Scott and April and Kelly and Mark, and our parents and parents in-law
who made it possible for us to pursue our careers as statisticians. We were both
fortunate to study with Franklin Graybill and we thank him for making sure that we
were headed in the right direction when our careers began.

© 2002 by CRC Press LLC


C0317c01 frame Page 1 Sunday, June 24, 2001 1:46 PM

1

Introduction to the
Analysis of Covariance

1.1 INTRODUCTION
The statistical procedure termed analysis of covariance has been used in several
contexts. The most common description of analysis of covariance is to adjust the
analysis for variables that could not be controlled by the experimenter. For example,
if a researcher wishes to compare the effect that ten different chemical weed control
treatments have on yield of a specific wheat variety, the researcher may wish to

control for the differential effects of a fertility trend occurring in the field and for
the number of wheat plants per plot that happen to emerge after planting. The
differential effects of a fertility trend can possibly be removed by using a randomized
complete block design structure, but it may not be possible to control the number
of wheat plants per plot (unless the seeds are sewn thickly and then the emerging
plants are thinned to a given number of plants per plot). The researcher wishes to
compare the treatments as if each treatment were grown on plots with the same
average fertility level and as if every plot had the same number of wheat plants. The
use of a randomized complete block design structure in which the blocks are constructed such that the fertility levels of plots within a block are very similar will
enable the treatments to be compared by averaging over the fertility levels, but the
analysis of covariance is a procedure which can compare treatment means after first
adjusting for the differential number of wheat plants per plot. The adjustment
procedure involves constructing a model that describes the relationship between
yield and the number of wheat plants per plot for each treatment, which is in the
form of a regression model. The regression models, one for each level of the
treatment, are then compared at a predetermined common number of wheat plants
per plot.

1.2 THE COVARIATE ADJUSTMENT PROCESS
To demonstrate the type of adjustment process that is being carried out when the
analysis of covariance methodology is applied, the set of data in Table 1.1 is used
in which there are two treatments and five plots per treatment in a completely
randomized design structure. Treatment 1 is a chemical application to control the
growth of weeds and Treatment 2 is a control without any chemicals to control the
weeds. The data in Table 1.1 consist of the yield of wheat plants of a specific variety
from plots of identical size along with the number of wheat plants that emerged

© 2002 by CRC Press LLC

C0317c01 frame Page 2 Sunday, June 24, 2001 1:46 PM

2

Analysis of Messy Data, Volume III: Analysis of Covariance

TABLE 1.1
Yield and Plants per Plot Data for the Example
in Section 1.2
Treatment 1
Yield per plot
951
957
776
1033
840

Treatment 2

Plants per plot
126
128
107
142
120

Yield per plot
930
790
764

989
740

Plants per plot
135
119
110
140
102

Yield per plot

1100

1000

900

X
Means
X

800

700
1

2
Treatment Number

FIGURE 1.1 Plot of the data for the two treatments, with the “X” denoting the respective
means.

after planting per plot. The researcher wants to compare the yields of the two
treatments for the condition when there are 125 plants per plot.
Figure 1.1 is a graphical display of the plot yields for each of the treatments
where the circles represent the data points for Treatment 1 and the boxes represent
the data points for Treatment 2. An “X” is used to mark the means of each of the
treatments.
If the researcher uses the two-sample t-test or one-way analysis of variance to
compare the two treatments without taking information into account about the
number of plants per plot, a t statistic of 1.02 or a F statistic of 1.05 is obtained,
indicating the two treatment means are not significantly different ( p = 0.3361). The
results of the analysis are in Table 1.2 in which the estimated standard error of the
difference of the two treatment means is 67.23.
© 2002 by CRC Press LLC


C0317c01 frame Page 3 Sunday, June 24, 2001 1:46 PM

Introduction to the Analysis of Covariance

3

TABLE 1.2
Analysis of Variance Table and Means for Comparing
the Yields of the Two Treatments Where No Information
about the Number of Plants per Plot is Used
Source
Model

Error
Corrected total

df
1
8
9

SS
11833.60
90408.40
102242.00

MS
11833.60
11301.05

FValue
1.05

ProbF
0.3361

Source
TRT

df
1

SS (type III)

11833.60

MS
11833.60

FValue
1.05

ProbF
0.3361

Parameter
Trt 1 – Trt 2

Estimate
68.8

StdErr
67.23

t Value
1.02

Probt
0.3361

TRT
1
2

LSMean
911.40
842.60

ProbtDiff
0.3361

1100

el

od

m
t1

en

atm

Tre

Yield per plot

1000

l

de

2
nt

mo

e

atm

Tre

900

800

Treatment 1 data
Treatment 2 data

700
100

110

120

130

140

150

Number of plants per plot

FIGURE 1.2 Plot of the data and the estimated regression models for the two treatments.

The next step is to investigate the relationship between the yield per plot and
the number of plants per plot. Figure 1.2 is a display of the data where the number
of plants is on the horizontal axis and the yield is on the vertical axis. The circles
denote the data for Treatment 1 and the boxes denote the data for Treatment 2. The
two lines on the graph, denoted by Treatment 1 model and Treatment 2 model, were
computed from the data by fitting the model yij = αi + βxij + εij, i = 1, 2 and j = 1,
© 2002 by CRC Press LLC


C0317c01 frame Page 4 Sunday, June 24, 2001 1:46 PM

4

Analysis of Messy Data, Volume III: Analysis of Covariance

TABLE 1.3
Analysis of Covariance to Provide the Estimates of the Slope
and Intercepts to be Used in Adjusting the Data
Source
Model
Error
Uncorr Total

df
3

7
10

SS
7787794.74
5737.26
7793532.00

MS
2595931.58
819.61

FValue
3167.28

ProbF
0.0000

Source
TRT
Plants

df
2
1

SS(Type III)
4964.18
84671.14

MS
2482.09
84671.14

FValue
3.03
103.31

ProbF
0.1128
0.0000

Parameter
Trt 1 – Trt 2

Estimate
44.73

StdErr
18.26

tValue
2.45

Probt
0.0441

Parameter
TRT 1
TRT 2

Plants

Estimate
29.453
–15.281
7.078

StdErr
87.711
85.369
0.696

tValue
0.34
–0.18
10.16

Probt
0.7469
0.8630
0.0000

2, …, 5, a model with different intercepts and common or equal slopes. The results
are included in Table 1.3.
Now analysis of covariance is used to compare the two treatments when there
are 125 plants per plot. The process of the analysis of covariance is to slide or move
the observations from a given treatment along the estimated regression model (parallel to the model) to intersect the vertical line at 125 plants per plot. This sliding
is demonstrated in Figure 1.3 where the solid circles represent the adjusted data for
Treatment 1 and the solid boxes represent the adjusted data for Treatment 2.
The lines join the open circles to the solid circles and join the open boxes to

the solid boxes. The lines indicate that the respective data points slid to the vertical
line at which there are 125 plants per plot.
The adjusted data are computed by

(

) (

)

(

yAij = yij − αˆ i + βˆ xij + αˆ i + βˆ125 = yij + βˆ 125 − xij

)

ˆ i + βˆ xij) i = 1,2 and j = 1,2,…,5 are the residuals or deviations of
The terms yij – (α
the observations from the estimated regression models. The preliminary computations of the adjusted yields are in Table 1.4. These adjusted yields are the predicted
yields of the plots as if each plot had 125 plants.
The next step is to compare the two treatments through the adjusted yield values
by computing a two-sample t statistic or the F statistic from a one-way analysis of
variance. The results of these analyses are in Table 1.5.
A problem with this analysis is that it assumes the adjusted data are not adjusted
data and so there is no reduction in the degrees of freedom for error due to estimating
the slope of the regression lines. Hence the final step is to recalculate the statistics

© 2002 by CRC Press LLC

C0317c01 frame Page 5 Sunday, June 24, 2001 1:46 PM

Introduction to the Analysis of Covariance

5

125 plants per plot

1100

el

od

m
t1

en

Yield per plot

tm

a
Tre

1000

el

od

m
t2

Slide observations
parallel to regression line
to meet the line of 125
plants per plot

en

tm

a
Tre

900
Adjusted data symbols

800

Treatment 1 data
Treatment 2 data

700
100

110

120

130

140

150

Number of plants per plot

FIGURE 1.3 Plot of the data and estimated regression models showing how to compute
adjusted yield values at 125 plants per plot.

TABLE 1.4
Preliminary Computations Used in Computing Adjusted Data
for Each Treatment as If All Plots Had 125 Plants per Plot
Treatment
1
1
1
1
1

Yield Per Plot
951
957
776
1033
840

Plants Per Plot
126
128
107
142
120

Residual
29.6905
21.534
–10.8232
–1.5611
–38.8402

Adjusted Yield
943.922
935.765
903.408
912.67
875.391

2
2
2
2
2

930
790
764

989
740

135
119
110
140
102

–10.2795
–37.0279
0.6761
13.3294
33.3019

859.218
832.469
870.173
882.827
902.799

by changing the degrees of freedom for error in Table 1.5 from 8 to 7 (the cost of
estimating the slope). The sum of squares error is identical for both Tables 1.3 and
1.5, but the error sum of squares from Table 1.5 is based on 8 degrees of freedom
instead of 7. To account for this change in degrees of freedom in Table 1.5, the
estimated standard error for comparing the two treatments needs to be multiplied
by 8 ⁄ 7 , the t statistic needs to be multiplied by 7 ⁄ 8 , and the F statistic needs to
be multiplied by 7/8. The recalculated statistics are presented in Table 1.6. Here the
estimated standard error of the difference between the two means is 18.11, a 3.7-fold
reduction over the analysis that ignores the information from the covariate. Thus,

© 2002 by CRC Press LLC


C0317c01 frame Page 6 Sunday, June 24, 2001 1:46 PM

6

Analysis of Messy Data, Volume III: Analysis of Covariance

TABLE 1.5
Analysis of the Adjusted Yields (Too Many Degrees
of Freedom for Error)
Source
Model
Error
Corrected Total

df
1
8
9

SS
5002.83
5737.26
10740.09

MS
5002.83

717.16

FValue
6.98

ProbF
0.0297

Source
TRT

df
1

SS (Type III)
5002.83

MS
5002.83

FValue
6.98

ProbF
0.0297

Parameter
Trt 1 – Trt 2

Estimate

44.734

StdErr
16.937

t Value
2.641197

Probt
0.0297

TRT
1
2

LSMean
914.231
869.497

ProbtDiff
0.0297

TABLE 1.6
Recalculated Statistics to Reflect the Loss of
Error Degrees of Freedom Due to Estimating
the Slope before Computing the Adjusted Yields
Recalculated
Recalculated
Recalculated
Recalculated

estimated standard error
t-statistic
F-statistic
significance level

18.11
2.47
6.10
0.0428

by taking into account the linear relationship between the yield of the plot and the
number of plants in that plot, there is a tremendous reduction in the variability of
the data. In fact, the analysis of the adjusted data shows there is a significant
difference between the yields of the two treatments when adjusting for the unequal
number of plants per plot (p = 0.0428), when the analysis of variance in Table 1.2
did not indicate there is a significant difference between the treatments ( p = 0.3361).
The final issue is that since this analysis of the adjusted data overlooks the fact the
slope has been estimated, the estimated standard error of the difference of two means
is a little small as compared to the estimated standard error one gets from the analysis
of covariance. The estimated standard error of the difference of the two means as
computed from the analysis of covariance in Table 1.3 is 18.26 as compared to 18.11
for the analysis of the adjusted data. Thus the two analyses are not quite identical.
This example shows the power of being able to use information about covariates
or independent variables to make decisions about the treatments being included in

© 2002 by CRC Press LLC


C0317c01 frame Page 7 Sunday, June 24, 2001 1:46 PM

Introduction to the Analysis of Covariance

7

the study. The analysis of covariance uses a model to adjust the data as if all the
observations are from experimental units with identical values of the covariates.
A typical discussion of analysis of covariance indicates that the analyst should
include the number of plants as a term in the model so that term accounts for
variability in the observed yields, i.e., the variance of the model is reduced. If
including the number of plants in the model reduces the variability enough, then it
is used to adjust the data before the variety means are compared. It is important to
remember that there is a model being assumed when the covariate or covariates are
included in a model.

1.3 A GENERAL AOC MODEL AND
THE BASIC PHILOSOPHY
In this text, the analysis of covariance is described in more generality than that of
adjusting for variation due to uncontrollable variables. The analysis of covariance
is defined as a method for comparing several regression surfaces or lines, one for
each treatment or treatment combination, where a different regression surface is
possibly used to describe the data for each treatment or treatment combination.
A one-way treatment structure with t treatments in a completely randomized
design structure (Milliken and Johnson, 1992) is used as a basis for setting up the
definitions for the analysis of covariance model. The experimental situation involves
selecting N experimental units from a population of experimental units and measuring k characteristics x1ij, x2ij, …, xkij on each experimental unit. The variables x1ij,
x2ij, …, xkij are called covariates or independent variables or concomitant variables.
It is important to measure the values of the covariates before the treatments are
applied to the experimental units so that the levels of the treatments do not effect
the values of the covariates. At a minimum, the values of the covariate should not

be effected by the applied levels of the treatments. In the chemical weed treatment
experiment, the number of plants per plot occur after applying a particular treatment
on a plot, so the value of the covariate (number of plants per plot) could not be
determined before the treatments were applied to the plots. If the germination rate is
affected by the applied treatments, then the number of plants per plot cannot be used
as a covariate in the conventional manner (see Chapter 2 for further discussion). After
the set of experimental units is selected and the values of the covariates are determined
(whent possible), then randomly assign ni experimental units to treatment i, where
N = Σ ni. One generally assigns equal numbers of experimental units to the levels
i1
of the treatment, but equal numbers of experimental units per level of the treatment
are not necessary. After an experimental unit is subjected its specific level of the
treatment, then measure the response or dependent variable which is denoted by yij.
Thus the variables used in the discussions are summarized as:
yij
x1ij
x2ij
xkij

is
is
is
is

the
the
the
the

dependent measure

first independent variable or covariate
second independent variable or covariate
kth independent variable or covariate

© 2002 by CRC Press LLC


C0317c01 frame Page 8 Sunday, June 24, 2001 1:46 PM

8

Analysis of Messy Data, Volume III: Analysis of Covariance

At this point, the experimental design is a one-way treatment structure with
t treatments in a completely randomized design structure with k covariates. If there
is a linear relationship between the mean of y for the ith treatment and the k covariates
or independent variables, an analysis of covariance model can be expressed as:
y ij = βoi + βli x lij + β2 i x 2 ij + … + β ki x kij + ε ij

(1.1)

for i = 1, 2, …, t, and j = 1, 2, …, ni, and the εij ~ iid N(0, σ2), i.e., the εij are
independently identically distributed normal random variables with mean 0 and
variance σ2. The important thing to note about this model is that the mean of the
y values from a given treatment depends on the values of the x’s as well as on the
treatment applied to the experimental units.
The analysis of covariance is a strategy for making decisions about the form of
the covariance model through testing a series of hypotheses and then making treatment comparisons by comparing the estimated responses from the final regression
models. Two important hypotheses that help simplify the regression models are
H01: βh1 = βh2 = … = βht = 0 vs. Ha1: (not H01:), that is, all the treatments’

slopes for the hth covariate are zero, h = 1, 2, …, k, or
H02: βh1 = βh2 = … = βht vs. Ha2: (not Ho2:), that is, the slopes for the hth
covariate are equal across the treatments, meaning the surfaces are parallel
in the direction of the hth covariate, h = 1, 2, …, k.
The analysis of covariance model in Equation 1.1 is a combination of an analysis
of variance model and a regression model. The analysis of covariance model is part
of an analysis of variance model since the intercepts and slopes are functions of the
levels of the treatments. The analysis of covariance model is also part of a regression
model since the model for each treatment is a regression model.
An experiment is designed to purchase a certain number of degrees of freedom
for error (generally without the covariates) and the experimenter is willing to sell
some of those degrees of freedom for good or effective covariates which will help
reduce the magnitude of the error variance. The philosophy in this book is to select
the simplest possible expression for the covariate part of the model before making
treatment comparisons.
This process of model building to determine the simplest adequate form of the
regression models follows the principle of parsimony and helps guard against foolishly selling degrees of freedom for error to retain unnecessary covariate terms in
the model. Thus the strategy for analysis of covariance begins with testing hypotheses
such as H01 and H02 to make decisions about the form of the covariate or regression
part of the model. Once the form of the covariate part of the model is finalized, the
treatments are compared by comparing the regression surfaces at predetermined
values of the covariates.
The structure of the following chapters leads one through the forest of analysis
of covariance by starting with the simple model with one covariate and building
through the complex process involving analysis of covariance in split-plot and

© 2002 by CRC Press LLC


C0317c01 frame Page 9 Sunday, June 24, 2001 1:46 PM

Introduction to the Analysis of Covariance

9

repeated measures designs. Other topics discussed are multiple covariates, experiments involving blocks, and graphical methods for comparing the models for the
various treatments.
Chapter 2 discusses the simple analysis of covariance model involving a oneway treatment structure in a completely randomized design structure with one
covariate and Chapter 3 contains several examples demonstrating the strategies for
situations involving one covariate. Chapter 4 presents a discussion of the analysis
of covariance models involving more than one covariate which includes polynomial
regression models. Models involving two-way treatment structures, both balanced
and unbalanced, are discussed in Chapter 5. A method of comparing parameters via
beta-hat models is described in Chapter 6. Chapter 7 describes a method for variable
selection in the analysis of covariance where many possible covariates were measured. Chapter 8 discusses methods for testing the equality of several regression
models.
The next set of chapters (9 through 11) discuss analysis of covariance in the
randomized complete block and incomplete block design structures. The analysis
of data where the values of a characteristic are used to construct blocks is described,
i.e., where the value of the covariate is the same for all experimental units in a block.
In the analysis of covariance context, inter- or between block information about the
intercepts and slopes is required to extract all available information about the regression lines or surfaces from the data. Usual analysis methods extract only the intrablock information from the data. A mixed models analysis involving methods of
moments and maximum likelihood estimation of the variance components provides
combined estimates of the parameters and should be used for blocked experiments.
Chapter 12 describes models where the levels of the treatments are random effects
(Littell et al., 1996). The models in Chapter 12 include random coefficient models.
Chapter 13 provides a discussion of mixed models with covariates and Chapter 14
presents a discussion of unequal variance models.
Chapters 15 and 16 discuss problems with applying the analysis of covariance
to experiments involving repeated measures and split-plot design structures. One

has to consider the size of experimental unit on which the covariate is measured.
Cases are discussed where the covariate is measured on the large size of an experimental unit and when the covariate is measured on the small size of an experimental
unit. Several examples of split-plot and repeated measures designs are presented. A
process of selecting the simplest covariance structure for the repeated measures part
of the model and the simplest covariate (regression model) part of the model is
described. The analysis of covariance in the nonreplicated experiment is discussed
in Chapter 17. The half-normal plot methodology (Milliken and Johnson, 1989) is
used to determine the form of the covariate part of the model and to determine which
effects are to be included in the intercept part of the model.
Finally, several special applications of analysis of covariance are presented in
Chapter 18, including using the covariate to construct blocks, crossover designs,
nonlinear models, nonparameteric analysis of covariance, and a process for examining mixed models for possible outliers in the data set.
The procedures of the SAS® system (1989, 1996, and 1997) and JMP® (2000)
are used to demonstrate how to use software to carry out the analysis of covariance

© 2002 by CRC Press LLC


C0317c01 frame Page 10 Sunday, June 24, 2001 1:46 PM

10

Analysis of Messy Data, Volume III: Analysis of Covariance

computations. The topic of analysis of covariance has been the topic of two volumes
of Biometrics, Volume 13, Number 3, in 1957 and Volume 38, Number 3, in 1982.
The collection of papers in these two volumes present discussions of widely diverse
applications of analysis of covariance.

REFERENCES

Littell, R. C., Milliken, G. A., Stroup, W. W., and Wolfinger, R. D. (1996) SAS® System for
Mixed Models, SAS Institute Inc., Cary, NC.
Milliken, G. A. and Johnson, D. E. (1989) Analysis of Messy Data, Volume II: Nonreplicated
Experiments, Chapman & Hall, London.
Milliken, G. A. and Johnson, D. E. (1992) Analysis of Messy Data, Volume I: Design
Experiments, Chapman & Hall, London.
SAS Institute Inc. (1989) SAS/STAT® User’s Guide, Version 6, Fourth Edition, Volume 2,
Cary, NC.
SAS Institute Inc. (1996) SAS/STAT® Software: Changes and Enhancements Through Release
6.11, Cary, NC.
SAS Institute Inc. (1997) SAS/STAT® Software: Changes and Enhancements Through Release
6.12, Cary, NC.
SAS Institute Inc. (2000) JMP® Statistics and Graphics Guide, Version 4, Cary, NC.

© 2002 by CRC Press LLC


C0317c02 frame Page 11 Monday, June 25, 2001 9:13 PM

2

One-Way Analysis
of Covariance —
One Covariate in a
Completely Randomized
Design Structure

2.1 THE MODEL
Suppose you have N homogeneous experimental
units and you randomly divide

t
them into t groups of ni units each where Σ ni = N. Each of the t treatments of a
i =1
one-way treatment structure is randomly assigned to one group of experimental
units, providing a one-way treatment structure in a completely randomized design
structure. It is assumed that the experimental units are subjected to their assigned
treatments independently of each other. Let yij (dependent variable) denote the jth
observation from the ith treatment and xij denote the covariate (independent variable)
corresponding to the (i,j)th experimental unit. As in Chapter 1, the values of the
covariate are not to be influenced by the levels of the treatment. The best case is
where the values of the covariate are determined before the treatments are assigned.
In any case, it is a good strategy to use the analysis of variance to check to see if
there are differences among the treatment covariate means (see Chapter 18).
Assume that the mean of yij can be expressed as a linear function of the covariate,
xij, with possibly a different linear function being required for each treatment. It is
important to note that the mean of an observation from the ith treatment group depends
on the value of the covariate as well as the treatment. In analysis of variance, the
mean of an observation from the ith treatment group depends only on the treatment.
The analysis of covariance model for a one-way treatment structure with one
covariate in a completely randomized design structure is
y ij = α i + βi X ij + ε ij ,
i = 1, 2, …, t. j = 1, 2, …, n i

(2.1)

where the mean of yi for a given value of X is µYi ΈX = αi + βiX. For making inferences,
it is assumed that εij ~ iid N(0, σ2). Model 2.1 has t intercepts (α1, …, αt ), t slopes
(β1, …, βt ), and one variance σ2, i.e., the model represents a collection of simple
linear regression models with a different model for each level of the treatment.

© 2002 by CRC Press LLC


C0317c02 frame Page 12 Monday, June 25, 2001 9:13 PM

12

Analysis of Messy Data, Volume III: Analysis of Covariance

Before analyzing this model, make sure that the data from each treatment can
in fact be described by a simple linear regression model. Various regression diagnostics should be run on the data before continuing. The equal variance assumption
should also be checked (see Chapter 14). If the simple linear regression model is
not adequate to describe the data for each treatment, then another model must be
selected before continuing with the analysis of covariance.
The analysis of covariance is a process of comparing the regression models and
then making decisions about the various parameters of the models. The process
involves comparing the t slopes, comparing the distances between the regression lines
(surfaces) at preselected values of X, and possibly comparing the t intercepts. The
analysis of covariance computations are typically presented in summation notation
with little emphasis on interpretations. In this and the following chapters, the various
covariance models are expressed in terms of matrices (see Chapter 6 of Milliken and
Johnson, 1992) and their interpretations are discussed. Software is used as the mode
of doing the analysis of covariance computations. The matrix form of Model 2.1 is
 y11  1
 M  M
 

 y1n1  1
 


 y21  0
 M  M
=

 y2 n2  0
 M  M
 

 ytl  0
 M  M
 

 ytnt  0
 


x11
M
x1n1
0
M
0
M
0
M
0

0
M
0

1
M
1
M
0
M
0

0
M
0
x21
M
x2 n2
M
0
M
0

L
L
L
L
L
L

0
M
0
0

M
0
M
1
M
1

0 
M 

0 

0 
M 

0 
M 

xtl 
M 

xtnt 


 α1 
 
 
 β1 
 
α 2 

 
  + ε.
 β2 
M
 
αt 
 
 
 βt 

(2.2)

which is expressed in the form of a linear model as y = Xβ + ε. The vector y denotes
the observations ordered by observation within each treatment, the 2t × 1 vector β
denotes the collection of slopes and intercepts, the matrix X is the design matrix,
and the vector ε represents the random errors.

2.2 ESTIMATION
The least squares estimator of the parameter vector β is βˆ = (X′X)–1X′y, but the least
squares estimator of β can also be obtained by fitting the simple linear regression
model to the data from each treatment and computing the least squares estimator of
each pair of parameters (αi, βi). For data from the ith treatment, fit the model
 y i1  1
  
 M  = M
  
y  1
 in i  
© 2002 by CRC Press LLC

x i1 

M 

x in i 


α i 
 
  + εi ,
 
 βi 
 

(2.3)


C0317c02 frame Page 13 Monday, June 25, 2001 9:13 PM

One-Way Analysis of Covariance

13

which is expressed as yi = Xiβi +εi. The least squares estimator of βi is βˆ i =
(X′i Xi)–1X′i y, the same as the estimator obtained for a simple linear regression model.
The estimates of βi and αi in summation notation are
ni

∑x y

ij ij

βˆ i =

− n i xi. yi.

j =1

ni

∑x

2
ij

− n i xi2.

j =1

and
αˆ i = yi. − βˆ i xi. .
The residual sum of squares for the ith model is
ni

SS Re si =

∑ (y

ij

)

2

− αˆ i − βˆ i x ij .

j =1

There are ni – 2 degrees of freedom associated with SSResi since the ith model
involves two parameters. After testing the equality of the treatment variances (see
Chapter 14) and deciding there is not enough evidence to conclude the variances
are unequal, the residual sum of squares for Model 2.1 can be obtained by pooling
residual sums of the squares for each of the t models, i.e., sum the SSResi together
to obtain
t

SS Re s =

∑ SS Re s .

(2.4)

i

i =1

The pooled residual sum of squares, SSRes, is based on the pooled degrees of
freedom, computed and denoted by
d.f .SS Re s =

t

t

i =1

i =1

∑ ( n i − 2 ) = ∑ n i − 2 t = N − 2 t.

The best estimate of the variance of the experimental units is σˆ 2 = SSRes/(N – 2t).
The sampling distribution of (N – 2t) σˆ 2/σ2 is central chi-square with (N – 2t) degrees
ˆ = (α
ˆ 1, βˆ 1,
of freedom. The sampling distribution of the least squares estimator, β′
ˆ
ˆ t, βt ) is normal with mean β′ = (α1, β1, …, αt , βt ) and variance-covariance
…, α
matrix σ2 (X′ X)–1, which can be written as
(X′X )−1
 1 1
−1
σ 2 ( X ′X ) = σ 2 
M
 0

© 2002 by CRC Press LLC

O
L

0 

M

−1
(X′t Xt ) 

(2.5)