ĐÁNH GIÁ VIỆC THIẾT kế BẢNG câu hỏi THU THẬP dữ LIỆUTRONG các NGHIÊN cứu KHẢO sát ở KHOA sư PHẠM TIẾNG ANH

ABSTRACT
The study, as its name reveals, has been conducted with the hope to review and
evaluate the validity and reliability of the employed questionnaires in ELT survey
research at FELTE, ULIS, VNU. With the design of evaluation research, the study
aims to present critical comments on what student practitioners have already
applied successfully from their research methodology background, and what they
need to improve in order for the research to be more successful.
Through the reviews of eight questionnaires employed in six survey graduation
papers from 2007 to 2012 at FELTE, ULIS, VNU, the limited validity and
reliability of those data collection instruments were revealed. The quality criteria of
the source, pilot procedure, main parts and the layouts of the questionnaires, as well
as the consideration of the questionnaire items should have received more
deliberation for the insurance of the quality of questionnaires.
From these findings, some practical implications were proposed with the hope to
suggest some solutions to the problem of limited validity and reliability of the
questionnaires, thus contributing to the value enhancement of the research projects
on ELT Methodology at FELTE, ULIS.
i
TABLE OF CONTENTS
Page
Acknowledgements i
Abstract ii
Table of Contents iii
List of Abbreviations vi
List of Tables and Figures vii
CHAPTER 1: INTRODUCTION
1.1. Statement of the Problem and Rationale for the Study 1
1.2. Aims of the Study and Research Question 2
1.3. Scope of the Study 2
1.4. Significance of the Study 3
1.5. Organization of the Study 3

CHAPTER 2: LITERATURE REVIEW
2.1. English Language Teaching Research 4
2.1.1. Introduction to English Language Teaching (ELT) 4
2.1.2. Research in ELT 4
2.2. Survey Research in ELT 6
2.2.1. Definitions of Survey Research 6
2.2.2. Values of Survey Research 7
2.2.3. Survey Research Procedure 10
2.2.4. Data Collection Instruments in Survey Research 13
2.3. Questionnaires in Survey Research 13
2.3.1. Definitions of a Questionnaire 13
2.3.2. Types of Data Collected from Questionnaires 14
2.3.3. Advantages and Disadvantages of Using Questionnaires 15
2.4. Questionnaire Construction 17
2.4.1. Questionnaire Construction Procedure 17
2.4.2. Quality Criteria for Constructing a Questionnaire 19
2.4.2.1. Quality Criteria for the Main Parts of the Questionnaire 19
2.4.2.2. Quality Criteria for the Content of the Questionnaire 20
2.4.3. Criteria for a Well-Constructed Questionnaire 21
2.5. Research Gaps in Questionnaire Construction Evaluation at FELTE, ULIS 23
ii
2.6. Summary 23
CHAPTER 3: METHODOLOGY
3.1. Research Context 24
3.2. Research Question 25
3.3. Research Approach: Evaluation Research 25
3.3.1. Evaluation Research 25
3.3.2. Evaluation Research Procedure 26
3.4. Sampling 28
3.4.1. Evaluands 28

3.4.2. Sampling Strategy: Criterion Sampling 28
3.4.3. Sampling Procedure and Results 30
3.4.4. Description of the Subjects of the Study 31
3.5. Data Collection Instrument 34
3.6. Data Collection Procedure 35
3.7. Data Analysis Procedure 35
3.8. Summary 36
CHAPTER 4: RESULTS AND DISCUSSIONS
4.1. Evaluation of the Reliability 37
4.1.1. Analysis of the Sources and the Pilot Procedure 37
4.1.2. Analysis of the Main Parts 39
4.1.3. Analysis of the Layouts 41
4.2. Evaluation of the Validity 43
4.2.1. Analysis of Questionnaire A 43
4.2.2. Analysis of Questionnaire B 47
4.2.3. Analysis of Questionnaires C1 and C2 49
4.2.4. Analysis of Questionnaires D1 and D2 50
4.2.5. Analysis of Questionnaire E 53
4.2.6. Analysis of Questionnaire F 55
4.3. The Extent to which the Student-Researchers Paid Attention to the Quality Criteria
of Questionnaire Construction 61
4.4. Summary 63
CHAPTER 5: CONCLUSION
5.1. Summary of the Major Discussions 64
iii
5.2. Pedagogical Implications 65
5.3. Limitations 66
5.4. Suggestions for Further Studies 67
REFERENCES 68
APPENDICES 70

iv
LIST OF ABBREVIATIONS
ELT: English Language Teaching
FELTE: Faculty of English Language Teacher Education
ULIS: University of Languages and International Studies
VNU: Vietnam National University
v
LIST OF TABLES AND FIGURES
LIST OF FIGURES
Figure 1: Broad Categories of Research 5
Figure 2: Stages in the Planning of a Survey 11
LIST OF TABLES
Table 2.1: Overview of Research Styles 7
Table 2.2: Steps of Conducting a Survey Research Study 10
Table 2.3: Equivalence between Survey Research Procedures provided by
Cohen et al., (2007) and Mertens (2005) 12
Table 2.4: Advantages and Disadvantages of Using Questionnaires 15
Table 2.5: The Process of Constructing a Good Questionnaire 18
Table 2.6: The Main Parts of a Questionnaire 19
Table 2.7: Types of Validity 21
Table 2.8: Criteria to Evaluate the Reliability of a Questionnaire 22
Table 2.9: Criteria to Evaluate the Validity of a Questionnaire 23
Table 3.1: Evaluation Research Procedure 27
Table 3.2: Criteria for Sampling 29
Table 3.3: The Sampling Procedure and Results 30
Table 3.4: Subjects of the Study 31
Table 3.5: Data Collection Instrument:
An Evaluation Framework for Questionnaire Construction 34
Table 4.1: Sample Questionnaire Sources and Pilot Procedures 38
Table 4.2: Main Parts of the Sample Questionnaires 40

Table 4.3: Layouts of the Sample Questionnaires 42
Table 4.4: Analysis of Questionnaire A 44
Table 4.5: Analysis of Questionnaire B 48
Table 4.6: Analysis of Questionnaire D1 51
Table 4.7: Analysis of Questionnaire D2 52
Table 4.8: Analysis of Questionnaire E 54
Table 4.9: Analysis of Questionnaire F 56
vi
CHAPTER 1: INTRODUCTION
1.1. Statement of the Problem and Rationale for the Study
Research is one major activity in the teaching and learning process at Faculty of
English Teacher Education (FELTE), University of Languages and International
Studies (ULIS), Vietnam National University, Hanoi (VNU). The student research
activity is conducted annually, calling for a large number of projects in various
fields, especially in English Language Teaching Methodology (ELT). More
importantly, research is used as an activity for assessment of senior students, in the
form of graduation paper. Annually, dozens of projects are conducted by students,
many of whom choose the survey approach and employ questionnaires as their
research instrument. In fact, it is evident that the questionnaire is generally
considered one of the most popular research methods utilized by the student
researchers.
Nonetheless, to which extent do the student practitioners pay attention to
questionnaire construction is questionable, especially in terms of validity and
reliability. The help of computers and the Internet have led to the ease of
questionnaire design, as Dornyei (2003, p.3) says: “it is possible to draw up
something that looks respectable in a few hours”, but the quality of data is another
issue. As Gillham (2000, p.1) states, “good research cannot be built on poorly
collected data”, and well constructed questionnaires will eliminate this risk. In other
words, “developing a questionnaire that will yield worthwhile data” (Gillham, 2000,
p.1) is the prerequisite for valuable findings in research in general and ELT research

in particular.
Meanwhile, the scarcity of research projects on the issues of questionnaire
construction at FELTE, ULIS is recognized. In the University Library, there has
been little or even no evidence of the reviewing and evaluating the current situation
of questionnaire construction. Those studies, if conducted may serve as a source of
reference for the enhancement in the quality of the questionnaires employed in their
researches as well as the quality of the students’ projects as a whole.
Therefore, the research “Questionnaire Construction in ELT Survey Research at
FELTE, ULIS: An evaluation study” was conducted as regards the graduation
1
theses at FELTE, ULIS, VNU with the close investigation of the questionnaire
construction in those survey research projects.
1.2. Aims of the Study and Research Question
The study was conducted with the hope to develop an Evaluation Framework for
questionnaire construction, to analyze the previously employed questionnaires and
to evaluate their validity and reliability. Through the investigation into the utilized
questionnaires, which are the products of the questionnaire construction process,
this project aimed to present critical evaluation on the questionnaire construction of
the previous theses, especially in regards to its relevance to the criteria for a well-
constructed questionnaire.
To be specific, the research project aimed at answering the following question:
To what extent do the questionnaires utilized in the ELT survey research
projects at FELTE, ULIS satisfy the quality criteria of questionnaire
construction?
1.3. Scope of the Study
From the aims of the study and the research questions, the scope of the study is
determined. The study concerns with only the graduation papers in the field of ELT
Methodology at FELTE, ULIS. As the research topic is questionnaire construction,
the graduation papers investigated involves attitudinal issues for the most typical
features of the questionnaires to be utilized. Of those research projects on attitudinal

issues, the projects which employ questionnaires as their main or only data
collection instrument are selected.
The researcher limits the investigated period of time to only six years from 2007 to
2012. The reasons for this decision is that 2006 was the year when the Course of
Research Methodology was first introduced to the third year students at FELTE,
ULIS. These students conducted their graduation papers in 2007; consequently, the
starting year for investigation is decided to be 2007 to ensure that the student
researchers had been some theoretical backgrounds of research methodology.
To sum up, the research Questionnaire Construction in ELT Survey Research at
FELTE, ULIS: An evaluation study concerns with only the survey graduation
2
papers which employ questionnaires as their main or only data collection instrument
from 2007 to 2012.
1.4. Significance of the Study
This research project, focusing on questionnaire construction, aims to provide a
clearer view of the questionnaire construction quality in ELT research at FELTE,
ULIS. The investigation into the situation of questionnaire construction, especially
regarding the validity and reliability of the employed questionnaires, is hoped to
present critical evaluation on questionnaire construction – what practitioners have
already applied successfully from their research methodology background, and what
they need to improve in order for the research to be more successful.
Furthermore, through the reviews of related studies on questionnaire, as well as the
evaluation tool presented in this study, the author hopes to raise the research
practitioners’ awareness of the “standardized questionnaire” as well as provide them
a suggestion of questionnaire self-evaluation. The study is expected to serve as a
useful source of reference for junior research practitioners who want to employ
questionnaire as a tool for their survey research, especially in ELT field.
1.5. Organization of the Study
Chapter 1 – Introduction - presents an overview of this evaluation research.
Chapter 2 is the review of relevant literature, setting the theoretical foundation for

data collection and analysis.
Chapter 3 describes the design of this study, including the approach, sampling
strategy, data collection instrument and procedure, as well as framework for data
analysis.
Chapter 4 presents results of the research and discussions from the analysis.
Finally, chapter 5 – Conclusion – is the summary of the findings as well as the
pedagogical implications and limitations of this evaluation study.
3
CHAPTER 2: LITERATURE REVIEW
In Chapter 2, the relevant literature of Survey Research on English Language
Teaching (ELT) Methodology is reviewed with the emphasis on the construction
procedure of questionnaires as the data collection instrument of the survey
research. Particularly, the quality criteria of a well-constructed questionnaire is
thoroughly investigated and presented in this chapter.
2.1. English Language Teaching Research
2.1.1. Introduction to English Language Teaching
English Language Teaching (ELT) is an applied linguistics branch of study of great
attention. This branch started to receive much interest from researchers in the late
sixteenth century with the arrival of the first textbooks “designed solely to teach
English as a foreign language” (Howatt, 1984, p. 6). Through the appearance and
development of various theories in English Language Teaching, this field has
become a profession worldwide since 1900. Today, ELT has become the topic of
interest of thousands of linguists around the world and also a major in numerous
university programs, usually under the name of TESOL – Teaching English to
Speakers of Other Languages, or Language Teacher Education.
2.1.2. Research in ELT
Research in ELT is generally understood as the systematic investigation in ELT. In
Brown and Rogers (2002, p. 12), this term is defined as “any systematic and
principled inquiry in language learning and teaching”. It means that English
language learning and teaching process is the overall objective of ELT research.

The importance of research in ELT is emphasized through the article “Language
Teacher Education” by Jourdenais (in The Handbook of Language Teaching, edited
by Long and Doughty, 2009, pp. 647-58). In this article, Jourdenais highlights the
role of research in ELT:
4
It is not enough that teachers be successful users of the target language; nor is it
sufficient that they understand how the language itself works. They must have a
level of awareness of language that enable them to assess, analyze, and present
it to learners in ways that will enhance acquisition. (p. 652)
Approaches and methods in ELT research quite varies and many authors present
these approaches and methods clearly in their books, some prominent names among
whom are Brown (2001), Brown and Rodgers (2002), Mertens (2005) and
Denscombe (2010). Among them, the best summary of approaches and methods is
probably the figure “Broad Categories of Research” by Brown (2001, p. 1) as
follows:
Figure 1: Broad Categories of Research (Brown, 2001, p.1)
Figure 1 demonstrates Brown’s idea to classify different types of research in ELT.
However, Brown (2001) uses this figure to illustrate the position of questionnaires
among various approaches and methods in ELT research, for it is the topic of the
book Using Surveys in Language Program. Brown and Rogers (2002), in their book
Doing Second Language Research, also use this chart, but this time a more detailed
discussion of research categories is presented. As can be seen clearly in the chart,
“research traditions available to second language scholars” (in their words, p. 10), is
classified into primary and secondary research, the former involves the findings
from original data, while the latter is based on secondary sources such as “other
researchers’ books and articles”. Each sub-type is divided into smaller ones, with
different instruments and designing techniques. Brown and Rogers also emphasize
5
the conclusion that the initial steps of any research project is “to decide what type,
or types, of research you are doing” (p. 12).

2.2. Survey Research in ELT
2.2.1. Definitions of Survey Research
Survey Research is a genre of social research. The term “Survey Research” is often
referred to as an approach or a strategy, different from the word “survey” only,
which means the tool or the method to collect data. The distinction between
“strategy” and “method” is quite clear, especially through books by Mertens (2005,
p.167) and Denscombe (2010, pp. 3-8). For example, Denscombe (2010) clarifies
that “research methods are the tools for data collection – things like questionnaires,
interviews, observation and documents” (p. 4), while he lists surveys (in plural form
– which refers to the design, rather than the tool) as one among the eight main
research strategies (pp. 5-6). He also presents three main characteristics of the
survey approach, namely Wide and inclusive coverage, at a specific point of time
and empirical research (pp. 11-12).
Brown (2001, p. 1) has similar ideas through his clarification of the term “survey
research”. He highlights the attributes of survey research, the most distinctive of
which are data-based and using interviews and questionnaires to collect
information. In the latter part of Chapter 1, Brown (2001) explores survey research
in linguistics field of study, which he calls “language surveys”. In his words,
“Language surveys are any survey research studies that gather data on the
characteristics and views of informants about the nature of language or language
learning through the use of oral interviews or written questionnaires” (p. 2).
Based on the definition of “language surveys” by Brown (2001), it can be
understood that survey research’s most prominent characteristic is the data
collection process to find out “characteristics and views” of the participants about
the interested topic of study, using interviews or questionnaires. It can also be
inferred from his definition that survey research is favourable thanks to its ability of
6
generalization, for survey research involves numerous participants to discover and
generalize those collected “characteristics and views”.
2.2.2. Values of Survey Research

Different types of research serve different purposes and it is the aim of the research
study that researchers choose the appropriate design or research styles. It should be
noted that each type has its own advantage and it is suggested that researchers have
a good overview of those types to make appropriate decision of their research
design.
There have been numerous books and journals about research methods, but it seems
that the summary of Cohen et al. (2007) organizes the information and key terms in
the most efficient way. This summary provides researchers an overview of research
approaches and designs with prominent characteristics of each type. The summary
is cited in Table 2.1:
Table 2.1: Overview of Research Styles
(Adapted from Box 3.2: Elements of Research Styles by Cohen et al., 2007, pp.84-6)
Model Purposes Foci Characteristics
Survey Gathering large-scale
data in order to make
generalizations
Generating statistically
manipulable data
Gathering context-free
data
Opinions
Scores
Outcomes
Conditions
Ratings
Describes and explains
Represents wide population
Gathers numerical data
Much use of questionnaires and
assessment / test data

Experiment Comparing under
controlled conditions
Making generalizations
about efficacy
Objective measurement
of treatment
Establishing causality
Initial states,
intervention and
outcomes
Randomized
controlled trials
Control and experimental
groups
Treats situations like a
laboratory
Causes due to experimental
intervention
Does not judge worth
Simplistic
continued
7
Table 2.1: Overview of Research Styles (continued)
Model Purposes Foci Characteristics
Ethnography Portrayal of events in
subject’s terms
Subjective and reporting of
multiple perspectives
Description, understanding
and explanation of specific

situation
Perceptions and
views of
participants
Issues as they
emerge over time
Context specific
Formative and emergent
Responsive to emerging
features
Allows room for judgements
and multiple perspectives
Wide database gathered over a
long period of time
Time-consuming to process
data
Action
research
To plan, implement, review
and evaluate an intervention
designed to improve
practice / solve local problem
To empower participants
through research
involvement and ideology
critique
To develop reflective
practice
To promote equality
democracy

To link practice and research
To promote collaborative
research
Everyday
practices
Outcomes of
interventions
Participant
empowerment
Reflective
practice
Social democracy
and equality
Decision-making
Context-specific
Participants as researchers
Reflection on practice
Interventionist – leading to
solution of ‘real’ problems and
meeting ‘real’ needs
Empowering for participants
Collaborative
Promoting praxis and equality
Stakeholder research
Case study To portray, analyse and
interpret the uniqueness of
real individuals and
situations through accessible
accounts
To catch the complexity and

situatedness of behaviour
To contribute to action and
intervention
To present and represent
reality – to give a sense of
‘being there’
Individuals and
local situations
Unique instances
A single case
Bounded
phenomena and
systems:
• individual
• group
• roles
• organizations
• community
In-depth, detailed data from
wide data source
Participant and non-participant
observation
Non-interventionist
Empathic
Holistic treatment of
phenomena
What can be learned from the
particular case
continued
8

Table 2.1: Overview of Research Styles (continued)
Model Purposes Foci Characteristics
Testing and
assessment
To measure achievement
and potential
To diagnose strengths and
weaknesses
To assess performance
and abilities
Academic and non-
academic, cognitive,
affective and
psychomotor
domains – low-order
to high-order
Performance,
achievement,
potential, abilities
Personality
characteristics
Materials designed to provide
scores that can be aggregated
Enables individuals and
groups to be compared
In-depth diagnosis
Measures performance
It can be concluded from Table 1 that survey research tends to be the most
advantageous in circumstances of factual information gathering with the close
relationship with large-scale population.

This conclusion agrees with Denscombe (2010, p. 12) that the data most frequently
and most effectively collected from survey research are often mass-data related,
concerned with “specific issues” within “groups or categories of people” and
“relatively uncomplicated”.
In particular, language surveys serve best for “curriculum development” and for
“research” (Brown, 2001, p. 12). Survey research benefits needs analysis and
program evaluation to improve an academic course, or explores opinions and
attitudes of targeted objectives thanks to the ability of “describing, exploring or
explaining physical characteristics, phenomena, behaviors, attitudes, and so forth”
(Brown, 2001, pp. 12-16).
However, the values of a particular survey study depend greatly on the quality of
instruments employed, because “good research cannot be built on poorly collected
data” (Gillham, 2000, p.1). As presented in 2.2.3, interviews and questionnaires are
9
two instruments used in survey research. This study will focus on analyzing
questionnaire construction in ELT survey research as an importance factor
contributing to the values of the studies.
2.2.3. Survey Research Procedure
Survey research procedure is discussed thoroughly in many books and projects on
research methodology, such as Brown (2000), Mertens (2005) and Cohen et al.,
(2007). Among them, Mertens (2005) and Cohen et al., (2007) seem to organize
and express ideas in the easiest way for junior practitioner in education research to
understand.
In this part, a comparison between two explanations of those two books is presented
to draw out the prominent steps of conducting a survey study.
Mertens (2005, pp. 166-212) explains thoroughly survey research procedure.
Chapter 6 in the book Research and Evaluation in Education and Psychology.
Integrating diversity with Quantitative, Qualitative and Mixed Methods presents
four steps of conducting a survey research study, which are briefly illustrated in
Table 2.2 as follows:

Table 2.2: Steps of Conducting a Survey Research Study
(based on Mertens, 2005, pp. 166-212)
Steps Description
1. Design phase Determination of purposes, objectives and types of information
needed for the survey
Consideration of design options for the survey
2. Sampling plan Identification of the respondents, population definition and
sampling decisions
3. Designing the
questionnaire
Design of the questionnaire used to collect data for the survey
research
4. Conducting
the survey
Process of inviting participants, collecting data and analyzing
data to answer research questions
10
Also in the discussion of survey research procedure, Cohen et al., (2007, p. 210)
present a detailed diagram illustrating the stages in the planning of a survey:
Figure 2: Stages in the Planning of a Survey (cited in Cohen et al., 2007, p. 210)
In comparison between the steps summarized by Mertens (2005) and Cohen et al.,
(2007), it can also be noted that the summary by Mertens (2005) is much simpler
with fewer steps than that of Cohen et al., (2007). However, in fact, the procedure
provided by Mertens (2005) is not simple. One explanation is that Mertens (2005)
groups numerous smaller steps into a bigger one with detailed explanation in latter
11
parts of the chapter and the book, which is similar to the explanation by Cohen et
al., (2007). Therefore, it can be concluded that two books share similar ideas about
the procedure of conducting a survey study.
The equivalence between two ways of description is summarized as follows:

Table 2.3: Equivalence between Survey Research Procedures
provided by Cohen et al., (2007) and Mertens (2005)
Cohen et al., (2007) Mertens (2005)
2.2.4. Data Collection Instruments in Survey Research
12
Design phase
Conducting
the survey
Sampling
plan
Designing the
questionnaire
As illustrated by Brown and Rodgers (2002, p.11), data collection instruments in
survey research are interviews and questionnaires. There are also other instruments
used in survey research, which are stated clearly in the 5
th
edition of Research
Methods in Education by Cohen et al., (2000, p. 171). Other instruments are
standardized tests and attitude scales. However, interviews and questionnaires
usually receive more attention in survey research, especially in the context of
Faculty of English Language Teacher Education, University of Languages and
International Studies.
Brown (2001, pp. 5-7) explains in more details the characteristics and the roles of
each type. According to this author, interviews are useful in collecting oral data
about the interested topic, but this instrument can be time-consuming and demands
the tactics of questioning to handle effectively. However, questionnaires prove their
edge over interviews in the issues requiring a large number of informants.
2.3. Questionnaires in Survey Research
2.3.1. Definitions of a Questionnaire
“Questionnaire” is not an uncommon term in the modern world today. It exists in

numerous situations and in most social fields of study. Questionnaires have been
considered an effective information gathering tool for researchers to serve their
work.
Researchers have pointed out some definitions of this term. In Questionnaires in
Second Language Research – Construction, Administration, and Processing,
Dornyei (2003, p.5) presents again the clarification of Aiken (1997) on different
names of “Questionnaire”. This term is sometimes called “inventories, forms,
opinionnaires, tests, batteries, checklists, scales, surveys, schedules, studies,
profiles, indexes/indicators” or “sheets”. Despite various names questionnaires
might be called, Dornyei focuses on the sense of “employed as research instruments
for measurement purposes to collect reliable and valid data”. (p.6)
13
A “questionnaire”, as defined in Longman Dictionary of Language Teaching and
Applied Linguistics (Richards et al., 1992, p. 303), is “a set of questions on a topic
or group of topics designed to be answered by a respondent”. The authors of this
book also emphasize that validity, reliability and unambiguity should be paid much
intention in constructing questionnaires. Other linguists such as Seliger and
Shohamy (1989, p. 172), Brown (2001, p. 6) and Denscombe (2010, pp. 155-6)
have similar ideas. For example, in Using Surveys in Language Programs, Brown
(2001) explains this term as “any written instruments that present respondents with
a series of questions or statements to which they are to react either by writing out
their answers or selecting from among existing answers” (p. 6).
Although there might be some other ways of defining the term, all authors agree on
two characteristics of a questionnaire. Firstly, a questionnaire consists of various
questions on a particular interested field. Secondly, questions, whichever type they
belong to, are systematically designed to serve the research purpose.
2.3.2. Types of Data Collected from Questionnaires
Although questionnaires are effective and advantageous, it is not possible for
questionnaires to be utilized in every circumstance. According to Dornyei (2003, p.
8) questionnaires can collect three types of information, namely factual, behavioral

and attitudinal information.
The first type of data, factual information, reveals background information of
respondents “that may be relevant to interpreting the findings of the survey”.
Typical examples of factual information in second language research are “learners’
language learning history, amount of time spent in an L2 environment, level of
parents L2 proficiency, or the L2 coursebook used.” The second type is behavioral
information, aiming to reveal “actions, life-styles, habits and personal history”.
Lastly, attitudinal information is about respondents’ thoughts, including “attitudes,
opinions, beliefs, interests and values”.
2.3.3. Advantages and Disadvantages of Using Questionnaires
14
The popularity of the questionnaire in research leads to a multitude books and
journals discussing the instrument’s characteristics, especially with regards to the
positive and negative attributes. Prominent authors discussing this issue are Dornyei
(2003), Gillham (2000) and Denscombe (2010).
Dornyei (2003, pp. 9-14) summarizes the advantages and disadvantages of using
questionnaires, which is presented briefly in Table 2.4 as follows:
Table 2.4: Advantages and Disadvantages of Using Questionnaires
(Dornyei, 2003, pp. 9-14)
Advantages Disadvantages
Ability of generalization while still:
- Saving researchers’ time
- Saving researchers’ effort
- Reducing the cost of research
- Simplicity and superficiality of
answers
- Unreliable and unmotivated
respondents
- Little or no opportunity to correct
the respondents’ mistakes

- Social desirability
- Self-deception
- Acquiescence bias
- Halo effect
According to Dornyei (2003, pp. 9-14), questionnaires prove their efficiency in
three main aspects, namely researcher time, researcher effort and financial
resources. Perhaps these three aspects are the main reasons for the wide use of
questionnaires in student research activity and in graduation papers at FELTE,
ULIS.
However, Dornyei also points out nine serious limitations of questionnaires. Those
nine limitations, which are presented in Table 2.4, refer to three main groups of
drawbacks. The first one concerns the depth of investigation, which is also the first
among those nine limitations. Questionnaires may limit “the depth of the
investigation” (p. 10) due to the short amount of time respondents are willing to
15
spend on questionnaires. Three other drawbacks concern the quality of answers
affected by questionnaire design. Respondents may misunderstand the real meaning
of the questionnaire items; therefore, they make mistakes. However, there is little or
no opportunity to correct these mistakes and the quality of the research is affected.
The rest four problems concern with respondents’ psychology factors, namely
unreliable and unmotivated respondents, social desirability bias, self deception,
acquiescence bias and halo effect. In short, respondents, due to psychology reasons,
may not provide true information during the completion of questionnaires.
The question follows Table 2.4 is why questionnaires still remain to be one of the
most popular data collection instruments when the number of disadvantages exceed
its advantages.
In fact, it is easy to misunderstand the point of Dornyei (2003) and careless readers
may think of the questionnaire popularity as a contradiction if they only look into
the number of advantages and disadvantages listed by Dornyei (2003, pp. 9-14).
However, it should be noted that the three advantages refer to the powerful ability

of generalization of this data collection instrument. Hardly can any other instrument
deal with large-scale research population as effectively as questionnaires.
Furthermore, those disadvantages stated above can be restricted and eliminated
from the high-quality construction of a particular questionnaire.
The advantages of questionnaires are confirmed by numerous other research studies
and books. Questionnaires are hightlighted as “a suitable method when it is
intended to collect some specific information from a large number of people”
(Manual for Educational Media Researchers, 2006). The powerful ability of
generalization is also emphasized by Gillham (2000) and Denscombe (2010).
Noticeably, Gillham (2000, p. 14) suggests that the questionnaire disadvantages can
be seen as “a list of pointers to producing a good questionnaire” and “as with many
other things in life, a successful outcome is a lot to do with trouble taken in
preparation”.
16
Therefore, it is essential to dig into questionnaire construction carefully to make the
most use of this data collection instrument.
2.4. Questionnaire Construction
2.4.1. Questionnaire Construction Procedure
Questionnaire construction, used in this study, is the term to describe the design of
questionnaire form for it to serve as the main (or only) method in ELT projects at
ULIS. As in Survey Questionnaire Construction, Martin (2006, p. 3) defines that
“[C]onstructing a questionnaire involves many decisions about the wording and
ordering of questions, selection and wording of response categories, formatting and
mode of administration of the questionnaire, and introducing and explaining the
survey.”
Questionnaire construction can be complicated as regards the elimination of the
disadvantages and enhancement of the advantages of the data collection instrument.
The quality criteria of a well constructed questionnaire will be discussed in the later
part of this chapter. Initially, the steps of constructing a good questionnaire should
be understood carefully as they are the prerequisite for a questionnaire to be an

effective data collection instrument.
A questionnaire can be conducted by adopting and adapting from the questionnaires
by other researchers, or by self-development of the author (Mertens, 2005, p. 179).
As for questionnaires of self-development, Dornyei (2003, pp. 16-7) discusses
thoroughly the steps the process of constructing a good questionnaire, which can be
summarized in Table 2.5 as follows:
Table 2.5: The Process of Constructing a Good Questionnaire
(Dornyei, 2003, pp. 16-7)
Step Work to be done Explanation
17
1 General features’ decision
making
Deciding on the general features of the
questionnaire, such as the length, the format,
and the main parts
2 Items writing Writing effective items/questions and drawing
up an item pool
3 Items selecting and
organizing
Selecting and sequencing the items
4 Instructions writing Writing appropriate instructions and examples
5 Piloting Piloting the questionnaire and conducting item
analysis
While the steps 1 to 4 are quite easy to understand, step 5 might be misunderstood
by the junior research practitioners. The pilot procedure involves the response of
some participants but it should not be confused with the questionnaire
administration procedure. Piloting in its basic meaning is the process of “trial runs”
to allow the researcher to “collect feedback about how the instrument works and
whether it performs the job it has been designed for” (Dornyei, 2003, p. 63).
The pilot procedure can be conducted at the point of item pool completion and at

the point which the almost-final-version of the questionnaire has been prepared
(Dornyei, 2003, pp. 63-8). The participants of the questionnaire pilot procedure can
be divided into three groups. The first group should not be “specialists” in the field,
who can help the questionnaire constructor with the jargon misunderstanding
elimination, the second one involves the professionals in survey research while the
third one knows “the target population well” (Dornyei, 2003, p. 66).
2.4.2. Quality Criteria for Constructing a Questionnaire
The term “Quality criteria” is referred to as one among “basic research issues” that
can influence researcher’s “methodological decisions” (Dornyei, 2007, p. 48).
Dornyei even ranks this issue the first in the order of importance, because “we can
18
only claim that our investigation is indeed a disciplined inquiry if we can set
explicit quality standards to achieve” (p. 48).
In the investigation of questionnaire construction, quality criteria often involve the
quality of the layout and quality of the content of the examined questionnaire.
2.4.2.1. Quality Criteria for the Main Parts of the Questionnaire
Mechanically speaking, a questionnaire should include five main parts which are
presented in the following table according to Dornyei (2003, pp. 25-30).
Table 2.6: The Main Parts of a Questionnaire (Dornyei, 2003, pp. 25-30)
Main Parts Characteristics
1. Title Be informative in terms of the domain of the investigation, the initial
orientation
Be able to activate various content schemata
2. Instructions
2.1. General
instruction
Be able to cover those following points:
1. What the study is about and why it is important
2. The organization responsible for conducting the study.
3. Emphasizing that there are no right or wrong answer;

requesting honest answers and trying to elicit integrity in
general
4. Promising confidentiality
5. Saying “thank you”
2.2. Specific
instructions
Be graphically highlighted
Be informative and clear for respondents
3. Questionnaire
items
Be clearly separated from the instructions
4. Additional
information
(optional)
Be at the end of the questionnaire
5. Final
“thank you”
Be at the end of the questionnaire
2.4.2.2. Quality Criteria for the Content of the Questionnaire
Quality criteria for the content of the questionnaire refers to its reliability and
validity. In fact, these two criteria are often listed out and discussed thoroughly in
numerous books for research methodogy, without under the classification of
19