2012 (methodology in the social sciences) noel a card phd applied meta analysis for social science research the guilford press (2011)

Applied Meta-Analysis for Social Science Research



Methodology in the Social Sciences
David A. Kenny, Founding Editor
Todd D. Little, Series Editor

This series provides applied researchers and students with analysis and research design
books that emphasize the use of methods to answer research questions. Rather than
emphasizing statistical theory, each volume in the series illustrates when a technique
should (and should not) be used and how the output from available software programs
should (and should not) be interpreted. Common pitfalls as well as areas of further
development are clearly articulated.

SPECTRAL ANALYSIS OF TIME-SERIES DATA
Rebecca M. Warner
A PRIMER ON REGRESSION ARTIFACTS
Donald T. Campbell and David A. Kenny
REGRESSION ANALYSIS FOR CATEGORICAL MODERATORS
Herman Aguinis
HOW TO CONDUCT BEHAVIORAL RESEARCH OVER THE INTERNET:
A Beginner’s Guide to HTML and CGI/Perl
R. Chris Fraley
CONFIRMATORY FACTOR ANALYSIS FOR APPLIED RESEARCH
Timothy A. Brown
DYADIC DATA ANALYSIS
David A. Kenny, Deborah A. Kashy, and William L. Cook
MISSING DATA: A Gentle Introduction

Patrick E. McKnight, Katherine M. McKnight, Souraya Sidani,
and Aurelio José Figueredo
MULTILEVEL ANALYSIS FOR APPLIED RESEARCH: IT’S JUST REGRESSION!
Robert Bickel
THE THEORY AND PRACTICE OF ITEM RESPONSE THEORY
R. J. de Ayala


THEORY CONSTRUCTION AND MODEL-BUILDING SKILLS:
A Practical Guide for Social Scientists
James Jaccard and Jacob Jacoby
DIAGNOSTIC MEASUREMENT: Theory, Methods, and Applications
André A. Rupp, Jonathan Templin, and Robert A. Henson
APPLIED MISSING DATA ANALYSIS
Craig K. Enders
ADVANCES IN CONFIGURAL FREQUENCY ANALYSIS
Alexander A. von Eye, Patrick Mair, and Eun-Young Mun
PRINCIPLES AND PRACTICE OF STRUCTURAL EQUATION MODELING,
Third Edition
Rex B. Kline
APPLIED META-ANALYSIS FOR SOCIAL SCIENCE RESEARCH
Noel A. Card


Applied
Meta-Analysis
for Social Science
Research
Noel A. Card
Series Editor’s Note by Todd D. Little

THE GUILFORD PRESS
New York   London


© 2012 The Guilford Press
A Division of Guilford Publications, Inc.
72 Spring Street, New York, NY 10012
www.guilford.com
All rights reserved
No part of this book may be reproduced, translated, stored in a
retrieval system, or transmitted, in any form or by any means,
electronic, mechanical, photocopying, microfilming, recording, or
otherwise, without written permission from the publisher.
Printed in the United States of America
This book is printed on acid-free paper.
Last digit is print number:  9  8  7  6  5  4  3  2  1
Library of Congress Cataloging-in-Publication Data
Card, Noel A.
  Applied meta-analysis for social science research / Noel A. Card;
Series editor’s note by Todd D. Little.
   p. cm. — (Methodology in the social sciences)
  Includes bibliographical references and index.
  ISBN 978-1-60918-499-5 (hbk.: alk. paper)
  1.  Social sciences—Statistical methods.  2.  Social sciences—
Methodology.  3.  Meta-analysis.  I.  Title.
  HA29.C286 2011
  300.72—dc23

2011019332

For my family—Jeanet, Gabby, and Angie



Series Editor’s Note

I am particularly proud and happy to introduce you to Noel Card and his
wonderful resource Applied Meta-Analysis for Social Science Research. The reason that I am so enthusiastic is that I have a close professional and personal
relationship with Noel spanning some 10-plus years now. Noel overlapped
with me for a number of years at the University of Kansas before he moved
to his current position at the University of Arizona. During this time, I have
seen Noel develop into one of the very finest pedagogically gifted quantitative scholars of our day. His honors include an early career award from the
Society for Research in Child Development. He is also a member of the Society of Multivariate Experimental Psychology (SMEP), a limited-membership
society devoted to the advancement of multivariate methods as applied in the
social and behavioral sciences. Noel’s election to SMEP at such a young age
speaks volumes about his ability and accomplishments.
When I became the Series Editor of the Methodology in the Social Sciences series, one of the first books I sought was a comprehensive book on
meta-analysis written in the accessible style of this series. I approached Noel
about writing this book knowing that he was the absolute best person to
write it. To begin, Noel has had a long-standing passion for meta-analysis,
and he has honed his pedagogy for quantitative synthesis techniques by
regularly teaching the University of Kansas Summer Institutes in Statistics
(“Stats Camps”; www.Quant.ku.edu) course on meta-analysis. Couple his
dogged determination to understand all that is meta-analysis with his gifted
ability to communicate quantitative methods to a diverse audience and you
have a recipe for a perfect book: a desk reference for those who are familiar
with meta-analysis and an all-in-one learning tool to use in classes at both

ix


x

Series Editor’s Note

the undergraduate or graduate level. Noel’s coverage is both broad and deep
and not tainted by preferences or restricted by a particular software focus.
As a resource work, Noel’s book covers many topics that others avoid, but
Noel integrates these topics so seamlessly that you’ll wonder which topics
are lacking in the other books (e.g., calculating appropriate and unique effect
sizes, thoroughly handling artifact corrections, evaluating publication bias
with bias alternative analytic representations, evaluating advanced statistical models of the assembled data, and so on). In each chapter, Noel provides
thoughtful and helpful advice on the key issues as well as offers alternative
tactics. He carefully presents the pros and cons of each alternative and issue.
As you read, you will immediately hear him reaching out to you and guiding
your understanding of the material. His “voice” as he writes is assuring and
clear, light-hearted yet authoritative. He deftly introduces a topic or idea at
the ground level and moves you step by step up the rungs of understanding
to the very top of where the field currently resides.
Throughout Noel’s book you will find a number of appealing pedagogical features. For those who are not comfortable with equations, for example,
Noel is very careful to explain the concepts in clear, simple language and in
equation form. He also annotates the equations so that beginners can learn
how to “read” a statistical equation. Equations are useful only when you the
reader can follow them. Noel makes sure you can follow them—primarily
as a supplement to the accompanying narrative. To illustrate all of the key
points throughout the book, Noel uses ample real data examples that he’s
discovered in his own meta-analytic work. Noel also has very thoughtfully

selected and annotated some recommended readings that you will find at the
end of each chapter.
You can see from the table of contents that the coverage is complete. You
will discover when you read each topic that Noel does not assume that you
have prior knowledge nor does he offend the seasoned expert. His motivation is genuine: He provides a comprehensive yet accessible work that will
advance scientific practice of quantitative synthesis in the social, behavioral,
and educational sciences. His many examples provide solid grounding for
and concrete clarifications of the concepts. Noel’s book is about as close to a
“page turner” as you can get.
Todd D. Little
At 30,000 feet between
Houston and Kansas City;
University of Kansas
Lawrence, Kansas


Preface and Acknowledgments

In some sense, I began this book over 10 years ago, when I was a graduate
student at St. John’s University. During this period of reading and trying to
draw conclusions from the research literature on childhood aggression and
victimization (my first area of substantive interest), I became discouraged by
the lack of accumulation of science, as evidenced by the discrepant conclusions drawn in (narrative) reviews of the literature and the numerous studies
conducted and published in what seemed an absence of knowledge of the
existing literature. During this time, I became motivated to find a better way
to summarize and synthesize the research literature in my particular field.
I soon was introduced to meta-analysis as a potential solution, and it did
not take long before I was convinced of its value. I began reading all I could
about this methodology and had the good fortune to attend a workshop by
Robert Rosenthal in the summer of 2004. Since that time, I have become

increasingly interested and immersed in meta-analysis through publishing
meta-analyses within my substantive area, teaching graduate classes and
workshops on the topic, and collaborating with researchers on their own
meta-analyses. So, when Todd D. Little, Editor of The Guilford Press’s Methodology in the Social Sciences series, approached me in 2007 about the possibility of writing a book, I was eager for the opportunity and believed I was
ready to do so.
The key phrase of the last sentence is “ . . . believed I was ready to do so.”
During the last 3 years of writing this book, I have learned that the methods of
meta-analysis are both broad and deep, and that this is a continuously evolving methodology. In this book, I have tried to capture each of these qualities
in presenting this methodology to you. First, I have tried to cover the full


xi


xii

Preface and Acknowledgments

breadth of meta-analysis, covering every aspect of the process of planning,
conducting, and writing a meta-analytic review. Second, I have delved into
the depth of meta-analysis when I believe that more advanced methods are
valuable, aiming to present this material in the least technical manner I can.
Finally, I have tried to present the state of the art of meta-analysis by covering recent advances that are likely to be valuable to you. In balancing these
sometimes competing demands of coverage, I have consistently thought back
to when I was preparing my first meta-analyses to consider what is the most
important material for beginning users of meta-analysis techniques to know.
The result, I hope, is that this book will help you learn about and then use
what I believe is an invaluable tool in the advancement of science.
Fortunately, I had many people supporting me during the writing of this
book. First and foremost, I want to thank C. Deborah Laughton (Publisher,

Methodology and Statistics, at Guilford) and Todd D. Little (Series Editor).
Both provided the perfect balance of patience and prompting, listening and
advice giving, and—above all—sincere friendship throughout the writing
of this book. I also thank C. Deborah for securing input from a number
of expert reviewers and am grateful to these reviewers for their thoughtful and constructive feedback: Adam Hafdahl, ARCH Statistical Consulting,
LLC; Mike Cheung, Department of Psychology, National University of Singapore; Blair Johnson, Department of Psychology, University of Connecticut;
Soyeon Ahn, Department of Educational and Psychological Studies, University of Miami; Jody Worley, Department of Human Relations, University of
Oklahoma; Robert Tett, Department of Psychology, University of Tulsa; John
Norris, Department of Second Language Studies, University of Hawaii; Brad
Bushman, Institute for Social Research, University of Michigan; Meng-Jia Wu,
School of Education, Loyola University; Tania B. Huedo-Medina, Department
of Psychology, University of Connecticut; and Jeffrey Valentine, Department
of Educational and Counseling Psychology, University of Louisville. I am
also thankful to the many individuals who provided prepublication copies
of their writings on meta-analysis, which was necessary to ensure that the
material presented in this book is up to date. This book also benefited from
feedback from students in a class at the University of Arizona (spring 2010)
and two workshops (2009 and 2010) in an ongoing course I teach at the
University of Kansas Summer Institutes in Statistics (affectionately known
as “Stats Camps”; see quant.ku.edu/StatsCamps/overview.html). The students
in these classes provided invaluable feedback and reactions to the material
that greatly improved the pedagogical value of this book. Finally, I am most
grateful to the support of my family throughout this process.


Contents

Part I. The Blueprint: Planning and Preparing
a Meta‑Analytic Review
1  •  An Introduction to Meta‑Analysis

3

1.1
1.2
1.3
1.4
1.5
1.6

The Need for Research Synthesis in the Social Sciences   3
Basic Terminology   4
A Brief History of Meta‑Analysis   8
The Scientific Process of Research Synthesis   9
An Overview of the Book   12
Practical Matters: A Note on Software and Information
Management   13
1.7 Summary   14
1.8 Recommended Readings   14

2  •  Questions That Can and Questions
That Cannot Be Answered through Meta‑Analysis

16

2.1 Identifying Goals and Research Questions
for Meta‑Analysis   17
2.2 The Limits of Primary Research and the Limits
of Meta‑Analytic Synthesis   19
2.3 Critiques of Meta‑Analysis: When Are They Valid

and When Are They Not?   23
2.4 Practical Matters: The Reciprocal Relation between Planning
and Conducting a Meta‑Analysis   29
2.5 Summary   31
2.6 Recommended Readings   32

3  •  Searching the Literature
3.1
3.2
3.3
3.4
3.5
3.6
3.7



34

Developing and Articulating a Sampling Frame   34
Inclusion and Exclusion Criteria   38
Finding Relevant Literature   42
Reality Checking: Is My Search Adequate?   52
Practical Matters: Beginning a Meta‑Analytic Database   55
Summary   58
Recommended Readings   58

xiii

xiv

Contents

Part II.  The Building Blocks:
Coding Individual Studies
4  •  Coding Study Characteristics

63

4.1
4.2
4.3
4.4

Identifying Interesting Moderators   64
Coding Study “Quality”   68
Evaluating Coding Decisions   73
Practical Matters: Creating an Organized Protocol
for Coding   77
4.5 Summary   82
4.6 Recommended Readings   82

5  •  Basic Effect Size Computation

85

5.1 The Common Metrics: Correlation, Standardized Mean Difference,
and Odds Ratio   85
5.2 Computing r from Commonly Reported Results   96

5.3 Computing g from Commonly Reported Results   107
5.4 Computing o from Commonly Reported Results   114
5.5 Comparisons among r, g, and o   118
5.6 Practical Matters: Using Effect Size Calculators
and Meta‑Analysis Programs   121
5.7 Summary   122
5.8 Recommended Readings   122

6  •  Corrections to Effect Sizes

126

6.1 The Controversy of Correction   127
6.2 Artifact Corrections to Consider   129
6.3 Practical Matters: When (and How) to Correct:
Conceptual, Methodological, and Disciplinary
Considerations   142
6.4 Summary   144
6.5 Recommended Readings   144

7  •  Advanced and Unique Effect Size Computation
7.1 Describing Single Variables   147
7.2 When the Metric Is Meaningful: Raw Difference Scores   154
7.3 Regression Coefficients and Similar Multivariate
Effect Sizes   156
7.4 Miscellaneous Effect Sizes   161
7.5 Practical Matters: The Opportunities and Challenges
of Meta‑Analyzing Unique Effect Sizes   166
7.6 Summary   169
7.7 Recommended Readings   169

147




Contents

xv

Part III. Putting the Pieces Together:
Combining and Comparing Effect Sizes
8  •  Basic Computations: Computing Mean Effect Size
and Heterogeneity around This Mean

175

8.1 The Logic of Weighting   176
8.2 Measures of Central Tendency in Effect Sizes   180
8.3 Inferential Testing and Confidence Intervals of Average
Effect Sizes   182
8.4 Evaluating Heterogeneity among Effect Sizes   184
8.5 Practical Matters: Nonindependence among Effect Sizes   191
8.6 Summary   195
8.7 Recommended Readings   195

9  •  Explaining Heterogeneity among Effect Sizes:
Moderator Analyses

198

9.1 Categorical Moderators   199
9.2 Continuous Moderators   207
9.3 A General Multiple Regression Framework
for Moderation   210
9.4 An Alternative SEM Approach   218
9.5 Practical Matters: The Limits of Interpreting Moderators
in Meta‑Analysis   222
9.6 Summary   226
9.7 Recommended Readings   226

10  •  Fixed‑, Random‑, and Mixed‑Effects Models

229

10.1 Differences among Models   230
10.2 Analyses of Random‑Effects Models   234
10.3 Mixed‑Effects Models   239
10.4 A Structural Equation Modeling Approach
to Random‑ and Mixed‑Effects Models   245
10.5 Practical Matters: Which Model Should I Use?   250
10.6 Summary   255
10.7 Recommended Readings   255

11  •  Publication Bias
11.1 The Problem of Publication Bias   257
11.2 Managing Publication Bias   260
11.3 Practical Matters: What Impact Do Sampling Biases Have
on Meta‑Analytic Conclusions?   275
11.4 Summary   276

11.5 Recommended Readings   276

257


xvi

Contents

12  •  Multivariate Meta‑Analytic Models

279

12.1 Meta‑Analysis to Obtain Sufficient Statistics   280
12.2 Two Approaches to Multivariate Meta‑Analysis   286
12.3 Practical Matters: The Interplay between Meta‑Analytic Models
and Theory   300
12.4 Summary   305
12.5 Recommended Readings   306

Part IV. The Final Product:
Reporting Meta‑Analytic Results
13  •  Writing Meta‑Analytic Results

313

13.1 Dimensions of Literature Reviews, Revisited   313
13.2 What to Report and Where to Report It   317
13.3 Using Tables and Figures in Reporting Meta‑Analyses   329
13.4 Practical Matters: Avoiding Common Problems in Reporting

Results of Meta‑Analyses   337
13.5 Summary   342
13.6 Recommended Readings   342

References

345

Author Index

357

Subject Index

363

About the Author

377


Applied Meta-Analysis for Social Science Research



Part I

The Blueprint
Planning and Preparing
a Meta‑Analytic Review

1

An Introduction to Meta‑Analysis

Meta-­analysis, also called quantitative research synthesis, is a powerful
approach to summarizing and comparing results from empirical literature. The
goal of this book is to provide you, the reader, with information to conduct and
evaluate meta-­analytic reviews.

1.1The Need for Research Synthesis
in the Social Sciences
Isaac Newton is known to have humbly explained his success: “If I have seen
further it is by standing upon the shoulders of giants” (1675; from Columbia
World of Quotations, 1996). Although the history of science suggests that
Newton may have been as likely to kick his fellow scientists down as he was
to collaboratively stand on their shoulders (e.g., Boorstin, 1983, Chs. 52–53;
Gribbin, 2002, Ch. 5), this statement does eloquently portray a central principle in science: That the advancement of scientific knowledge is based on
systematic building of one study on top of a foundation of prior studies, the
accumulation of which takes our understanding to ever increasing heights. A
closely related tenet is replication—that findings of studies are confirmed (or
not) through repetition by other scientists.
Together, the principles of orderly accumulation and replication of
empirical research suggest that scientific knowledge should steadily progress. However, it is reasonable to ask if this is really the case. One obstacle to
this progression is that scientists are humans with finite abilities to retain,
organize, and synthesize empirical findings. In most areas of research, stud

3

4

PLANNING AND PREPARING A META-ANALYTIC REVIEW

ies are being conducted at an increasing rate, making it difficult for scholars
to stay informed of research in all but the narrowest areas of specialization.
I argue that many areas of social science research are in less need of further
research than they are in need of organization of the existing research. A
second obstacle is that studies are rarely exact replications of one another,
but instead commonly use slightly different methods, measures, and/or samples.1 This imperfect replication makes it difficult (1) to separate meaningful
differences in results from expectable sampling fluctuations, and (2) if there
are meaningful differences in results across studies, to determine which of
the several differences in studies account for the differences in results.
An apparent solution to these obstacles is that scientists systematically
review results from the numerous studies, synthesizing results to draw conclusions regarding typical findings and sources of variability across studies.
One method of conducting such systematic syntheses of the empirical literature is through meta-­analysis, which is a methodological and statistical
approach to drawing conclusions from empirical literature. As I hope to demonstrate in this book, meta-­analysis is a particularly powerful tool in drawing these sorts of conclusions from the existing empirical literature. Before
describing this tool in the remainder of the book, in this chapter I introduce
some terminology of this approach, provide a brief history of meta-­analysis,
further describe the process of research synthesis as a scientific endeavor,
and then provide a more detailed preview of the remainder of this book.

1.2 Basic Terminology
Before further discussing meta-­analysis, it is useful to clarify some relevant
terminology. One clarification involves the distinction of meta-­analysis from
primary or secondary analysis. The second clarification involves terminology
of meta-­analysis within the superordinate category of a literature review.
1.2.1 Meta‑Analysis versus Primary

or Secondary Analysis
The first piece of terminology to clarify are the differences among the terms
“meta-­analysis,” “primary analysis,” and “secondary analysis” (Glass, 1976).
The term “primary analysis” refers to what we typically think of as data analysis—when a researcher collects data from individual persons, companies,
and so on,2 and then analyzes these data to provide answers to the research
questions that motivated the study. The term “secondary analysis” refers to




An Introduction to Meta-­Analysis

5

re-­analysis of these data, often to answer different research questions or to
answer research questions in a different way (e.g., using alternative analytic
approaches that were not available when the data were originally analyzed).
This secondary data analysis can be performed either by the original researchers or by others if they are able to obtain the raw data from the researchers.
Both primary and secondary data analysis require access to the full, raw data
as collected in the study.
In contrast, meta-­analysis involves the statistical analysis of the results
from more than one study. Two points of this definition merit consideration
in differentiating meta-­analysis from either primary or secondary analysis.
First, meta-­analysis involves the results of studies as the unit of analysis, specifically results in the form of effect sizes. Obtaining these effect sizes does not
require having access to the raw data (which are all-too-often unavailable), as
it is usually possible to compute these effect sizes from the data reported in
papers resulting from the original, primary or secondary, analysis. Second,
meta-­analysis is the analysis of results from multiple studies, in which individual studies are the unit of analysis. The number of studies can range from
as few as two to as many as several hundred (or more, limited only by the
availability of relevant studies). Therefore, a meta-­analysis involves drawing

inferences from a sample of studies, in contrast to primary and secondary
analyses that involve drawing inferences from a sample of individuals. Given
this goal, meta-­analysis can be considered a form of literature review, as I
elaborate next.
1.2.2 Meta‑Analysis as a Form of Literature Review
A second aspect of terminological consideration involves the place of meta­analysis within the larger family of literature reviews. A literature review can
be defined as a synthesis of prior literature on a particular topic. Literature
reviews differ along several dimensions, including their focus, goals, perspective, coverage, organization, intended audience, and method of synthesis (see Cooper, 1988, 2009a). Two dimensions are especially important in
situating meta-­analysis within the superordinate family of literature reviews:
focus and method of synthesis. Figure 1.1 shows a schematic representation
of how meta-­analysis differs from other literature reviews in terms of focus
and method of synthesis.
Meta-­analyses, like other research syntheses, focus on research outcomes (not the conclusion reached by study authors, which Rosenthal noted
are “only vaguely related to the actual results” (1991, p. 13). Reviews focusing
on research outcomes answer questions such as “The existing research shows