Advancing quaititative methods in second language research luke plonsky, routledge, 2015 scan

ADVANCING QUANTITATIVE
METHODS IN SECOND
LANGUAGE RESEARCH

By picking up where introductory texts have left off, Advancing Quantitative
­Methods in Second Language Research provides a “second course” on quantitative
methods and enables second language researchers to both address questions currently posed in the field in new and dynamic ways and to address novel or more
complex questions as well. In line with the practical focus of the book, each
chapter provides the conceptual motivation for and step-by-step guidance needed
to carry out a relatively advanced, novel, and/or underused statistical technique.
Using readily available statistical software packages such as SPSS, the chapters walk
the reader from conceptualization through to output and interpretation of a range
of advanced statistical procedures such as bootstrapping, mixed effects modeling,
cluster analysis, discriminant function analysis, and meta-analysis. This practical
hands-on volume equips researchers in applied linguistics and second language
acquisition (SLA) with the necessary tools and knowledge to engage more fully
with key issues and problems in SLA and to work toward expanding the statistical
repertoire of the field.
Luke Plonsky (PhD, Michigan State University) is a faculty member in the
Applied Linguistics program at Northern Arizona University. His interests include
SLA and research methods, and his publications in these and other areas have
appeared in Annual Review of Applied Linguistics, Applied Linguistics, Language Learning, Modern Language Journal, and Studies in Second Language Acquisition, among
other major journals and outlets. He is also Associated Editor of Studies in Second
Language Acquisition and Managing Editor of Foreign Language Annals.


SECOND LANGUAGE ACQUISITION RESEARCH SERIES
Susan M. Gass and Alison Mackey, Series Editors

Monographs on Theoretical Issues:

Schachter/Gass
Second Language Classroom Research: Issues and Opportunities (1996)
Birdsong
Second Language Acquisition and the Critical Period Hypotheses (1999)
Ohta
Second Language Acquisition Processes in the Classroom: Learning Japanese (2001)
Major
Foreign Accent: Ontogeny and Phylogeny of Second Language Phonology (2001)
VanPatten
Processing Instruction: Theory, Research, and Commentary (2003)
VanPatten/Williams/Rott/Overstreet
Form-Meaning Connections in Second Language Acquisition (2004)
Bardovi-Harlig/Hartford
Interlanguage Pragmatics: Exploring Institutional Talk (2005)
Dörnyei
The Psychology of the Language Learner: Individual Differences in Second
Language Acquisition (2005)
Long
Problems in SLA (2007)
VanPatten/Williams
Theories in Second Language Acquisition (2007)


Ortega/Byrnes
The Longitudinal Study of Advanced L2 Capacities (2008)
Liceras/Zobl/Goodluck
The Role of Formal Features in Second Language Acquisition (2008)
Philp/Adams/Iwashita
Peer Interaction and Second Language Learning (2013)
VanPatten/Williams

Theories in Second Language Acquisition, Second Edition (2014)
Leow
Explicit Learning in the L2 Classroom (2015)
Dörnyei/Ryan
The Psychology of the Language Learner—Revisited (2015)

Monographs on Research Methodology:
Tarone/Gass/Cohen
Research Methodology in Second Language Acquisition (1994)
Yule
Referential Communication Tasks (1997)
Gass/Mackey
Stimulated Recall Methodology in Second Language Research (2000)
Markee
Conversation Analysis (2000)
Gass/Mackey
Data Elicitation for Second and Foreign Language Research (2007)
Duff
Case Study Research in Applied Linguistics (2007)
McDonough/Trofimovich
Using Priming Methods in Second Language Research (2008)
Dörnyei/Taguchi
Questionnaires in Second Language Research: Construction, Administration, and
Processing, Second Edition (2009)
Bowles
The Think-Aloud Controversy in Second Language Research (2010)


Jiang
Conducting Reaction Time Research for Second Language Studies (2011)

Barkhuizen/Benson/Chik
Narrative Inquiry in Language Teaching and Learning Research (2013)
Jegerski/VanPatten
Research Methods in Second Language Psycholinguistics (2013)
Larson-Hall
A Guide to Doing Statistics in Second Language Research Using SPSS and R,
Second Edition (2015)
Plonsky
Advancing Quantitative Methods in Second Language Research (2015)

Of Related Interest:
Gass
Input, Interaction, and the Second Language Learner (1997)
Gass/Sorace/Selinker
Second Language Learning Data Analysis, Second Edition (1998)
Mackey/Gass
Second Language Research: Methodology and Design (2005)
Gass with Behney & Plonsky
Second Language Acquisition: An Introductory Course, Fourth Edition (2013)


ADVANCING
QUANTITATIVE
METHODS IN SECOND
LANGUAGE RESEARCH

Edited by
Luke Plonsky
NORTHERN ARIZONA UNIVERSITY

First published 2015
by Routledge
711 Third Avenue, New York, NY 10017
and by Routledge
2 Park Square, Milton Park, Abingdon, Oxon, OX14 4RN
Routledge is an imprint of the Taylor & Francis Group, an informa business
© 2015 Taylor & Francis
The right of Luke Plonsky to be identified as the author of the editorial
material, and of the authors for their individual chapters, has been asserted in
accordance with sections 77 and 78 of the Copyright, Designs and Patents
Act 1988.
All rights reserved. No part of this book may be reprinted or reproduced or
utilised in any form or by any electronic, mechanical, or other means, now
known or hereafter invented, including photocopying and recording, or in
any information storage or retrieval system, without permission in writing
from the publishers.
Trademark notice: Product or corporate names may be trademarks or registered
trademarks, and are used only for identification and explanation without
intent to infringe.
Library of Congress Cataloging-in-Publication Data
Plonsky, Luke.
  Advancing quantitative methods in second language research / Luke
Plonsky, Northern Arizona University.
   pages cm. — (Second Language Acquisition Research Series)
  Includes bibliographical references and index.
  1.  Second language acquisition—Resesarch.  2.  Second language
acquisition—Data processing.  3.  Language and languages—Study and
teaching—Research.  4. Language acquisition—Research.  5. Language
acquisition—Data processing.  6.  Quantitative research.  7.  Multilingual

computing. 8. Computational linguistics. I. 
Title.
  P118.2.P65 2015
 401'.93—dc23
 2014048744
ISBN: 978-0-415-71833-2 (hbk)
ISBN: 978-0-415-71834-9 (pbk)
ISBN: 978-1-315-87090-8 (ebk)
Typeset in Bembo
by Apex CoVantage, LLC


For Pamela


This page intentionally left blank


CONTENTS

List of Illustrations
xi
Acknowledgmentsxvii
xix
List of Contributors
PART I

Introduction1
 1 Introduction
Luke Plonsky

3

  2 Why Bother Learning Advanced Quantitative Methods in L2
Research?9
James Dean Brown
PART II

Enhancing Existing Quantitative Methods
  3 Statistical Power, p Values, Descriptive Statistics, and Effect
Sizes: A “Back-to-Basics” Approach to Advancing Quantitative
Methods in L2 Research
Luke Plonsky
  4 A Practical Guide to Bootstrapping Descriptive Statistics,
Correlations, t Tests, and ANOVAs
Geoffrey T. LaFlair, Jesse Egbert, and Luke Plonsky

21

23

46


x Contents

  5 Presenting Quantitative Data Visually
Thom Hudson
  6 Meta-analyzing Second Language Research
Luke Plonsky and Frederick L. Oswald

78
106

PART III

Advanced and Multivariate Methods

129

  7 Multiple Regression
Eun Hee Jeon

131

  8 Mixed Effects Modeling and Longitudinal Data Analysis
Ian Cunnings and Ian Finlayson

159

  9 Exploratory Factor Analysis and Principal
Components Analysis
Shawn Loewen and Talip Gonulal

182

10 Structural Equation Modeling in L2 Research
Rob Schoonen

213

11 Cluster Analysis
Shelley Staples and Douglas Biber

243

12 Rasch Analysis
Ute Knoch and Tim McNamara

275

13 Discriminant Analysis
John M. Norris

305

14 Bayesian Informative Hypothesis Testing
Beth Mackey and Steven J. Ross

329

Index347


ILLUSTRATIONS

FIGURES
3.1
3.2
3.3

3.4
3.5
3.6
3.7
3.8
3.9
4.1
4.2
4.3
4.4
4.5
4.6
4.7

A descriptive model of quantitative L2 research
24
Screenshot of effect size calculator for Cohen’s d32
Screenshot of effect size calculator for Cohen’s d with CIs
32
Linear regression dialogue box used to calculate CIs
for correlation coefficients
34
Statistics dialogue box within linear regression
34
Output for linear regression with CIs for correlation
35
Output for descriptive statistics produced through
Explore in SPSS
40
Descriptive statistics and CIs for abstracts

with vs. without errors
41
A revised model of quantitative L2 research
42
Explore main dialogue box
52
Bootstrap dialogue box
53
Bootstrap specifications
54
Descriptive statistics table with bootstrapped 95% CIs
for various descriptive statistics
55
Correlations output table with bootstrapped 95% CIs
for Pearson correlation coefficient
60
Bootstrapped correlation coefficients and Q-Q plot
62
Independent-Samples Test output table with
bootstrapped 95% CIs
63


xii Illustrations

4.8 Bootstrap mean differences, Q-Q plot, and jackknife-afterboot plot of the mean difference between English and
Vietnamese66
4.9 Plot of the bootstrap T-statistics, their Q-Q plot, and the
jackknife-after-boot plot
68

4.10 One-way ANOVA output table with bootstrapped
95% CIs
69
5.1 Cleveland’s 1993 graphic display of barley harvest data from
Immer, Hayes, & Powers (1934)
81
5.2 Types of graphics used over last four regular issues of five
applied linguistics journals
86
5.3 Bar chart showing means of listening scores for each
category of self-rated confidence ratings with
95% CI (N = 45)
88
5.4 Histogram of speaking scores (N = 45)
89
5.5 Grouped bar chart for speaking scores by gender
with 95% CI course level by gender with 95% CI
89
90
5.6 Percentage of students in each proficiency level by gender
5.7 Number of students in each proficiency level by gender
91
5.8 Box-and-whisker plot for the speaking test by gender
92
5.9 Box-and-whisker plots for the five proficiency levels
across the speaking test scores
93
5.10 Student scores (means and CIs) on five tests administered
three weeks apart over a semester (N = 45)
93

5.11 Mean scores and 95% CIs on reading, listening,
and grammar for three proficiency levels
94
5.12 Graphic representation of score data across levels with
box chart display of distributions
95
5.13 Scatter plot for the relationship between reading scores
and grammar scores (N = 45)
96
5.14 Mean state scores for NAEP data in Table 5.4
96
5.15 Mean state scores for NAEP data in Table 5.4 ordered
by state score
97
5.16 Scatter plot matrix of correlations between four subtests
98
5.17 Number of weekly online posts with sparklines showing
the online posting activity for each student
98
5.18 Example pie charts for student distribution
99
5.19 Initial SPSS bar chart for speaking mean scores by level
100
5.20 Edited SPSS bar chart for speaking mean scores by level
100
5.21 Listening score interaction of gender by proficiency level
101
6.1 Example of a forest plot
116

Illustrations  xiii

  6.2 Example of a funnel plot without the presence
of publication bias
  6.3 Example of a funnel plot with the presence
of publication bias
 7.1 Partial L value table
  7.2 Mahalanobis distance dialogue boxes in SPSS
  7.3 Mahalanobis distance column in SPSS data view
  7.4 Tolerance statistic dialogue boxes in SPSS
  7.5 Multiple regression analysis decision tree
  7.6 SPSS standard multiple regression dialogue boxes:
the first dialogue box and selections in the Statistics tab
  7.7 SPSS standard multiple regression dialogue boxes:
selections in the Linear Regression Plots dialogue box
  7.8 A scatter plot indicating normality
  7.9 A scatter plot indicating nonnormality
  7.10 SPSS hierarchical regression analysis dialogue boxes:
selections of PVs for the first model
  7.11 SPSS hierarchical regression analysis dialogue boxes:
selections of PV for the second and final model
and selection of statistics
  8.1 Q-Q plots for untransformed (left) and transformed
(right) proficiency scores
  9.1 Types of factor analysis
  9.2 Overview of the steps in a factor analysis
  9.3 Example of KMO measure of sampling adequacy
and Bartlett’s Test of Sphericity (SPSS Output)
  9.4 Adapted R-matrix

 9.5 Communalities
  9.6 Choosing EFA
  9.7 Main dialogue box for factor analysis
  9.8 Descriptives in factor analysis
  9.9 Dialogue box for factor extraction
  9.10 Total variance explained
  9.11 Scree plot
  9.12 Dialogue box for factor rotation
  9.13 Options dialogue box
  9.14 Unrotated component matrix
  9.15 Rotated factor loadings (pattern matrix)
  9.16 Factor scores dialogue box
  9.17 Labeling the factors
10.1 Two competing structural models

116
117
136
138
139
140
142
144
145
145
146
150
151
167
184

186
188
189
190
191
191
192
193
195
195
198
199
200
201
202
204–5
215


xiv Illustrations

10.2 Two competing structural models with measurement
part added
216
10.3 Two competing models: a one-factor model and a
three-factor model
218
10.4 PRELIS data definition options
226
10.5 Starting to build command lines

226
10.6 Adding latent variable command lines
227
230
10.7 Setup for the three-factor model
10.8 Importing data for one-factor model in AMOS
234
10.9 Output file three-factor model with correlated error
in AMOS
235
11.1 Step 1247
11.2 Step 2
248
11.3 Step 3, part 1249
11.4 Step 3, part 2249
11.5 Step 4, part 1250
11.6 Step 4, part 2251
251
11.7 Step 5, part 1
11.8 Step 5, part 2
253
254
11.9 Step 6
11.10Step 7
254
11.11Dendrogram of cluster analysis for 947 cases
255
11.12Truncated agglomeration schedule for 947
cases in the data set
256

11.13Distance between fusion coefficients
by number of clusters
258
11.14Step 9, part 1
259
11.15Step 9, part 2
259
11.16Data view with 2, 3, and 4 cluster solutions
260
11.17Step 9, part 3
261
262
11.18Step 10, part 1
11.19Step 10, part 2
262
11.20Step 11
264
11.21Cluster membership by task type for the
two-cluster solution
265
11.22Cluster membership by task type for the
three-cluster solution
265
11.23Cluster membership by task type for the
four-cluster solution
266
11.24Cluster membership by score level for the
two-cluster solution
267

Illustrations  xv

11.25Cluster membership by score level for the
three-cluster solution
11.26Cluster membership by score level for the
four-cluster solution
12.1 Sample person/item (Wright) map
12.2 Sample category characteristic curve
12.3 Sample facets map
13.1 Selecting the right analysis in SPSS
13.2 Selecting and defining grouping variables
13.3 Selecting predictor variables
13.4 Selecting statistics for the analysis
13.5 Selecting analysis and display option for classification
13.6 Two-dimensional output for three group average values
on two discriminant functions
14.1 Schematic person-item map with cut scores
14.2 Grouping identifiers and item difficulty estimates
14.3 Comparison of Means data input
14.4 Group observations in Comparison of Means
14.5 Confirmatory model specification
14.6 Entry of hypothesized mean hierarchies
14.7 Summary of hierarchy of hypotheses
14.8 Execution of Comparison of Means
14.9 Comparison of Means Bayesian analysis output

267
267
285–86

293
297
311
311
312
313
314
319
332
336
337
338
339
340
341
342
343

TABLES
  1.1
  3.1
  3.2
  3.3
  3.4
  3.5

Software used and available for procedures in this book
Data and results from Sample Study 1
Data and results from Sample Study 2
Data and results from Sample Study 3

Example results showing the inconsistency of p values
General benchmarks for interpreting d and r effect
sizes in L2 research
  5.1 Types of graphical charts and frequency of use found in
last four regular issues of five L2 journals
  5.2 2009 average reading scale score sorted by gender, grade
12 public schools
  5.3 2009 average NAEP reading scale scores by gender for
grade 12 public schools in 11 states (first revision)

7
25
25
26
28
38
79
84
85


xvi Illustrations

  5.4 2009 average NAEP reading scale scores by gender for
grade 12 public schools in 11 states sorted on state mean
scores (second revision)
85
  6.1 Suggested categories for coding within meta-analyses of L2
research110
  7.1 SPSS output for tolerance statistics

140
  7.2 SPSS output for variables entered/removed
147
  7.3 SPSS output for regression model summary
147
  7.4 SPSS output for ANOVA resulting from regression
148
  7.5 SPSS output for regression coefficients
149
  7.6 SPSS output for variables entered/removed in hierarchical
regression model
152
  7.7 SPSS output for hierarchical regression model summary
152
  7.8 SPSS output for ANOVA resulting from hierarchical
regression153
  7.9 SPSS output for hierarchical regression coefficients
153
  9.1 Parallel analysis
196
11.1 Reformatted fusion coefficients for final six clusters formed 257
11.2 Means and standard deviations for the two–cluster solution 263
11.3 Means and standard deviations for the three–cluster solution 263
11.4 Means and standard deviations for the four–cluster solution 264
12.1 Data type, response formats, Rasch models, and programs
278
12.2 Data input format for analyses not involving multiple raters 279
12.3 Data input format for analyses involving multiple raters
280
12.4 Sample person measurement report (shortened)

288
12.5 Sample item measurement report (shortened)
289
12.6 Sample item measurement report for partial credit data
292
12.7 Sample rating scale category structure report
293
12.8 Sample rater measurement report
298
13.1 ANOVA output for nine predictor variables
315
13.2 Box’s M output for testing homogeneity of covariance
across three groups
316
13.3 Canonical discriminant functions output
317
13.4 Relationship output for individual predictor variables
318
and functions
13.5 Classification output for each predictor variable
320
13.6 Accuracy of classification output for membership
in three groups
320
14.1 Grouping labels for analysis
335
14.2 Hypotheses tested in confirmatory technique
337
14.3 Comparison of Means software (exploratory and
confirmatory tests)

339


ACKNOWLEDGMENTS

I want to begin by expressing my sincere gratitude to the diverse set of individuals
who have contributed to this volume in equally diverse ways. I am very grateful,
first of all, to all 18 chapter authors. It is clear from their work that they are not
only experts in the statistical procedures they have written about but in their ability to communicate and train others on these procedures as well. I also thank the
authors for their perseverance and persistence in the face of my many requests.
In addition to my own comments, each chapter was also reviewed by at least one
reviewer from both the target audience (graduate students or junior researchers
with at least one previous course in statistics) and from the modest pool of applied
linguists with expertise in the focal procedure of each chapter. I am very thankful
for the comments and suggestions of these reviewers which led to many substantial improvements throughout the volume: Dan Brown, Meishan Chen, Euijung
Cheong, Joseph Collentine, Jersus Colmenares, Scott Crossley, Deirdre Derrick,
Jesse Egbert, Maria Nelly Gutierrez Arvizu, Eun Hee Jeon, Tingting Kang, Geoffrey LaFlair, Jenifer Larson-Hall, Jared Linck, Junkyu Lee, Qiandi Liu, Meghan
Moran, John Norris, Gary Ockey, Fred Oswald, Steven Ross, Erin Schnur, and
Soo Jung Youn. Along these lines, my thanks go to the students in my ENG 599
and 705 courses, who read and commented on prepublication versions of many of
the chapters in the book. Special thanks to Deirdre Derrick for all her help on the
index. I also thank Shawn Loewen and Fred Oswald, both of whom have had a
(statistically) significant effect on my development as quantitative researcher. A big
thanks go to Sue Gass and Alison Mackey, series editors, for their encouragement
and support in carrying this book from an idea to its current form. Last, thanks
to you, the reader, for your interest in advancing the field’s quantitative methods.
In the words of Geoff Cumming, happy reading and “may all your confidence
intervals be short!”

This page intentionally left blank


CONTRIBUTORS

Douglas Biber (Northern Arizona University)
James Dean Brown (University of Hawaii at Manoa)
Ian Cunnings (University of Reading)
Jesse Egbert (Brigham Young University)
Ian Finlayson (University of Edinburgh)
Talip Gonulal (Michigan State University)
Thom Hudson (University of Hawaii at Manoa)
Eun Hee Jeon (University of North Carolina, Pembroke)
Ute Knoch (University of Melbourne)
Geoffrey T. LaFlair (Northern Arizona University)
Shawn Loewen (Michigan State University)
Beth Mackey (University of Maryland)
Tim McNamara (University of Melbourne)
John M. Norris (Georgetown University)
Frederick L. Oswald (Rice University)
Luke Plonsky (Northern Arizona University)
Steven J. Ross (University of Maryland)
Rob Schoonen (University of Amsterdam)
Shelley Staples (Purdue University)


This page intentionally left blank


PART I

Introduction


This page intentionally left blank


1
INTRODUCTION
Luke Plonsky

Rationale for This Book
Several reviews of quantitative second language (L2) research have demonstrated
that empirical efforts in the field rely heavily on a very narrow range of statistical procedures (e.g., Gass, 2009; Plonsky, 2013). Namely, nearly all quantitative
studies employ t tests, ANOVAs, and/or correlations. In many cases, these tests
are viable means to address the research questions at hand; however, problems
associated with these techniques arise frequently (e.g., failing to meet statistical
assumptions). More concerning, though, is the capacity of these tests to provide
meaningful and informative answers to our questions about L2 learning, teaching, testing, use, and so forth. Also concerning is that the near-default status of
these statistics restricts researchers’ ability to understand relationships between
constructs of interest as well as their use of analyses to examine such relationships.
In other words, our research questions are being constrained by our knowledge
of statistical tools.
This problem manifests itself in at least two ways. First, it is not uncommon
to find researchers that convert intervally measured (independent) variables into
categorical ones in order for the data to fit into an ANOVA model. Doing so
trades precious variance for what appears to be a more straightforward analytical approach (see Plonsky, Chapter 3 in this volume, for further comments and
suggestions related to this practice). Second, and perhaps more concerning, the
relatively simple statistics found in most L2 research are generally unable to model
the complex relationships we are interested in. L2 learning and use are multivariate in nature (see, e.g., Brown, Chapter 2 in this volume). Many studies account

for the complexity in these processes by measuring multiple variables. Few, however, attempt to analyze them using multivariate techniques. Consequently, it is


4  Luke Plonsky

common to find 20 or 30 univariate tests in a single study leading to a greater
chance of Type I error and, more importantly, a fractured view of the relationships of interest (Plonsky, 2013).
Before going on I need to clarify two points related to the intentions behind
this volume. First, neither I nor the authors who have contributed to this volume
are advocating for blindly applied technical or statistical sophistication. I agree
wholeheartedly with the recommendation of the American Psychological Association to employ statistical procedures that are “minimally sufficient” to address
the research questions being posed (Wilkinson & Task Force on Statistical Inference, 1999, p. 598). Second, the procedures described in this book are just tools.
Yes, they carry great potential to help us address substantive questions that cannot
otherwise be answered. We have to remember, though, that our analyses must be
guided by the substantive interests and relationships in question and not the other
way around. I mention this because of the tendency, particularly among novice
researchers, to become fascinated with a particular method or statistic and to
allow one’s research questions to be driven by the method.
Having laid out these rationales and caveats . . . at the heart of this volume is
an interest in informing and expanding the statistical repertoire of L2 researchers.
Toward this end, each chapter provides the conceptual motivation for and the
practical, step-by-step guidance needed to carry out a relatively advanced, novel,
and/or underused statistical technique using readily available statistical software
packages (e.g., SPSS). In related disciplines such as education and psychology,
these techniques are introduced in statistics texts and employed regularly. Despite
their potential in our field, however, they are rarely used and almost entirely
absent from methodological texts written for applied linguistics.
This volume picks up where introductory texts (e.g., Larson-Hall, 2015) leave
off and assumes a basic understanding of research design as well as basic statistical
concepts and techniques used in L2 research (e.g., t test, ANOVA, correlation).

The book goes beyond these procedures to provide a “second course,” that is, a
conceptual primer and practical tutorial on a number of analyses not currently
available in other methods volumes in applied linguistics. The hope is that, by
doing so, researchers in the field will be better equipped to address questions currently posed and to take on novel or more complex questions.
The book also seeks to improve methodological training in graduate programs,
the need for which has been suggested as the result of recent studies surveying
both published research as well as researcher self-efficacy (e.g., Loewen et al.,
2014; Plonsky, 2014). This text will assist graduate programs in applied linguistics
and second language acquisition/studies in providing “in-house” instruction on
statistical techniques using sample data and examples tailored to the variables,
interests, measures, and designs particular to L2 research.
Beyond filling gaps in the statistical knowledge of the field and in available texts
and reference books, this volume also seeks to contribute to the budding methodological and statistical reform movement taking place in applied linguistics. The