The Science and Art of Implementing Quantitative Evaluation Surveys

Bạn đang xem bản rút gọn của tài liệu. Xem và tải ngay bản đầy đủ của tài liệu tại đây (174.77 KB, 67 trang )

(1)<div class='page_container' data-page=1>

The Science and Art of Implementing

Quantitative Evaluation Surveys

John Hoddinott

Deputy Director

Food Consumption and Nutrition Division

</div>
(2)<div class='page_container' data-page=2>

Introduction

 Conducting a quantitative evaluation survey is both a Conducting a quantitative evaluation survey is both a

science and an art.

 The science pertains to the construction of a sample that The science pertains to the construction of a sample that

is representative of the population of interest

 The art pertains to the implementation of a survey The art pertains to the implementation of a survey

instrument (such as a questionnaire) that generates the

information you need for your evaluation

</div>
(3)<div class='page_container' data-page=3></div>
(4)<div class='page_container' data-page=4>

The science …

 Deciding on your unit of observationDeciding on your unit of observation

 Describing the universe: The sample frameDescribing the universe: The sample frame
 Drawing the sampleDrawing the sample

</div>
(5)<div class='page_container' data-page=5>

The Unit of Observation

 The unit of observation is simply the unit that is of The unit of observation is simply the unit that is of

interest given the objectives of your study:

• StudentsStudents

• Young womenYoung women

</div>
(6)<div class='page_container' data-page=6>

The Sampling Frame

The frame for a sample is a list of the units in the

population (or universe) from which the units that will be

enumerated in the sample area are selected. It may be an

actual list, a set of index cards, a map, or data stored in a

computer. The frame is a set of physical materials

(census statistics, maps, lists, directories, records) that

enables us to take hold of the universe piece by piece.

(Casley and Lury, 1987, p. 52)

</div>
(7)<div class='page_container' data-page=7>

The Sampling Frame, cont’d

 It is important that you examine carefully any sampling It is important that you examine carefully any sampling

frame that is made available to you for:

• DuplicationsDuplications

• Errors (such as inclusion errors)Errors (such as inclusion errors)

• OmissionsOmissions

 The concern is that errors of inclusion or omission are The concern is that errors of inclusion or omission are

non-random

• Lists of studentsLists of students

</div>
(8)<div class='page_container' data-page=8>

Drawing the sample

 Probabilistic sampling methods use some mechanism Probabilistic sampling methods use some mechanism

involving chance to determine which observations appear

in the sample. These mechanisms include:

• Systematic samplingSystematic sampling

• Systematic random samplingSystematic random sampling

• Stratified random samplingStratified random sampling

</div>
(9)<div class='page_container' data-page=9>

Sampling

 Systematic methods involve the selection of every nth Systematic methods involve the selection of every nth

observation.

 For example, suppose we want a sample of 250 observations For example, suppose we want a sample of 250 observations

from our population of 1000 students.

 We could take the first student on our list, the fifth, the ninth We could take the first student on our list, the fifth, the ninth

and so on. This method is relatively straightforward.

 The drawback is that the ordering of firms from 1 to 1000 The drawback is that the ordering of firms from 1 to 1000

must be random. If there is some subtle, difficult-to-observe

ordering of the sample (for example, older children tend to

be counted as even numbers) the observations drawn will

not be a random sampling of the population.

</div>
(10)<div class='page_container' data-page=10>

Sampling, cont’d

 Simple random sampling is a better alternative.Simple random sampling is a better alternative.

 The simplest way to do this is to use a statistics package The simplest way to do this is to use a statistics package

like STATA.

 Suppose we have a listing of 1000 students and we want Suppose we have a listing of 1000 students and we want

to randomly select 50 of them for interview. We use the

command:

</div>
(11)<div class='page_container' data-page=11>

Sampling, cont’d

 There is a potential weakness with this approach. Suppose we There is a potential weakness with this approach. Suppose we

are drawing a sample of 100 students from a population of 1000. 
are drawing a sample of 100 students from a population of 1000. 
We know that 30% of these have completed grades 5-8 so our 
We know that 30% of these have completed grades 5-8 so our 
sample should contain 30 such students. However, this is only 
sample should contain 30 such students. However, this is only 
true on average! Though the likelihood is high that our sample 
true on average! Though the likelihood is high that our sample 
will contain 30 such students, it is also possible that it contains 
will contain 30 such students, it is also possible that it contains 
20, 25 or 40.

20, 25 or 40.

 The solution to this problem is random stratified sampling. The The solution to this problem is random stratified sampling. The

first step is to divide the population into groups or strata. Here, 
first step is to divide the population into groups or strata. Here, 
the division would be between the 300 students in grades 5-8 
the division would be between the 300 students in grades 5-8 
and 700 other students. Using the random number method, 
and 700 other students. Using the random number method,

select 10% of students in each category, so the resultant sample 
select 10% of students in each category, so the resultant sample 
contains 30 students in grades 5-8 and 70 others. The

contains 30 students in grades 5-8 and 70 others. The 
proportions in the sample are identical to those in the 
proportions in the sample are identical to those in the 
underlying population.

</div>
(12)<div class='page_container' data-page=12>

Sampling, cont’d

 The final approach is to use cluster based sampling.The final approach is to use cluster based sampling.
 Here, you select a unit of observation (eg a school) and Here, you select a unit of observation (eg a school) and

sample within it.

 Cluster based sampling is especially appropriate in the Cluster based sampling is especially appropriate in the

context of randomized designs where randomization

occurs at the cluster level.

</div>
(13)<div class='page_container' data-page=13>

Sample Size Calculations

Recall some basic statistical concepts:

• Significance levelSignificance level. The probability of rejecting a null . The probability of rejecting a null

hypothesis that is true. – also called Type I errors. This is 
hypothesis that is true. – also called Type I errors. This is

often expressed as a percentage so that a test of 
often expressed as a percentage so that a test of

significance level,

significance level, αα, is referred to as a 100, is referred to as a 100αα% level test% level test

• PowerPower. The probability, 1- . The probability, 1- ββ of correctly rejecting a null of correctly rejecting a null 
hypothesis that is false.

hypothesis that is false.

• For a given sample size, there is a trade off increasing power For a given sample size, there is a trade off increasing power 
and reducing Type I errors

and reducing Type I errors

• We can only increase power and reduce Type I errors We can only increase power and reduce Type I errors 
simultaneously by increasing sample size

</div>
(14)<div class='page_container' data-page=14>

Sample Size Calculations, cont’d

 In addition, in order to calculate sample sizes, we need to In addition, in order to calculate sample sizes, we need to

know:

• The size of the impact that we would like to detectThe size of the impact that we would like to detect

• Estimates of standard deviationsEstimates of standard deviations

• (If using a cluster design), estimates of intra-cluster (If using a cluster design), estimates of intra-cluster 
correlation, also called the design effect

correlation, also called the design effect

</div>
(15)<div class='page_container' data-page=15>

Sample Size Calculations, cont’d

 (If using a clustered design), first calculate the intra-(If using a clustered design), first calculate the

intra-cluster correlation coefficient:

cluster correlation coefficient:

 Suppose we have the variable score and our clusters are Suppose we have the variable score and our clusters are

defined by the variable, school_id

 The command:The command:

• loneway score school_idloneway score school_id

</div>
(16)<div class='page_container' data-page=16>

Sample Size Calculations, cont’d

 Suppose I want to detect a 20% increase as a result of my Suppose I want to detect a 20% increase as a result of my

intervention (from 100 for the control group to 120 for the

treatment group).

 I want to have statistical power of 0.80.I want to have statistical power of 0.80.
 Standard deviation is 60.Standard deviation is 60.

 I run the Stata command:I run the Stata command:

</div>
(17)<div class='page_container' data-page=17>

Sample Size Calculations, cont’d

Test Ho: m1 = m2, where m1 is the mean in population 1

and m2 is the mean in population 2and m2 is the mean in population 2
Assumptions:

Assumptions:

alpha = 0.0500 (two-sided)alpha = 0.0500 (two-sided)

power = 0.8000power = 0.8000

m1 = 100m1 = 100

m2 = 120m2 = 120

sd1 = 60sd1 = 60

sd2 = 60sd2 = 60

n2/n1 = 1.00n2/n1 = 1.00

Estimated required sample sizes:

n1 = 142n1 = 142

</div>
(18)<div class='page_container' data-page=18>

Sample Size Calculations, cont’d

 Finally, to account for clusteringFinally, to account for clustering

 Suppose I intend to survey 100 schools (50 treatments Suppose I intend to survey 100 schools (50 treatments

and 50 controls) and the intra-cluster correlation is 0.35

 I runI run

• sampclus, obsclus (25) rho (0.30)sampclus, obsclus (25) rho (0.30)

</div>
(19)<div class='page_container' data-page=19>

Sample Size Calculations, cont’d

 Sample Size Adjusted for Cluster DesignSample Size Adjusted for Cluster Design

 n1 (uncorrected) = 142n1 (uncorrected) = 142
 n2 (uncorrected) = 142n2 (uncorrected) = 142

 Intraclass correlation = .3Intraclass correlation = .3

 Average obs. per cluster = 25Average obs. per cluster = 25
 Minimum number of clusters = 94Minimum number of clusters = 94

 Estimated sample size per group:Estimated sample size per group:

</div>
(20)<div class='page_container' data-page=20>

Sample Size Calculations: Concluded

 The final point to note here is that having insufficient The final point to note here is that having insufficient

sample sizes dooms the evaluation survey even before

you leave the office

 You need to make these calculations on the basis of You need to make these calculations on the basis of

conservative assumptions regarding:

• Effect sizeEffect size

</div>
(21)<div class='page_container' data-page=21>

The art

 The “Hard Arts”The “Hard Arts”

• Questionnaire design and length; pilot testingQuestionnaire design and length; pilot testing

• Use of PDAsUse of PDAs

• Issues in data entry, checking and cleaningIssues in data entry, checking and cleaning

• BudgetsBudgets

 The “Soft Arts”The “Soft Arts”

• Selecting enumeratorsSelecting enumerators

• Selecting SupervisorsSelecting Supervisors

• Interacting with respondentsInteracting with respondents

– Informed consentInformed consent

– PaymentsPayments

</div>
(22)<div class='page_container' data-page=22></div>
(23)<div class='page_container' data-page=23></div>
(24)<div class='page_container' data-page=24>

The Hard Arts

 TopicsTopics

• Questionnaire design and length; pilot testingQuestionnaire design and length; pilot testing

• Use of PDAsUse of PDAs

• Issues in data entry, checking and cleaningIssues in data entry, checking and cleaning

</div>
(25)<div class='page_container' data-page=25>

Questionnaire Design

 Getting this right is critical to the success of your project.Getting this right is critical to the success of your project.
 The best way to design the questionnaire is to ‘work The best way to design the questionnaire is to ‘work

backwards’ . That is, start by thinking about what your

report will look like:

• What are the outcomes that you want to measure?What are the outcomes that you want to measure?

• (In the case of non-randomized designs), what variables (In the case of non-randomized designs), what variables 
determine participation. What covariates would you put in 
determine participation. What covariates would you put in

</div>
(26)<div class='page_container' data-page=26>

Questionnaire Design, cont’d

• Do treatment observations actually receive the treatment? Do treatment observations actually receive the treatment? 
Do they receive only partial treatment? Are there problems 
Do they receive only partial treatment? Are there problems

with quality? What constraints/problems did they face in 
with quality? What constraints/problems did they face in

accessing intervention
accessing intervention

– Helps explain why you might not find significant impactHelps explain why you might not find significant impact

– Allows you to set up a “treatment on the treated” model as an Allows you to set up a “treatment on the treated” model as an 
alternative to your “intent to treat”

alternative to your “intent to treat”

– Operational details are of considerable interest to program Operational details are of considerable interest to program 
managers

managers

• What are the characteristics of your sample? Are there What are the characteristics of your sample? Are there 
particular sub-groups that you want to identify?

</div>
(27)<div class='page_container' data-page=27>

Questionnaire Design, cont’d

 Practical Considerations:Practical Considerations:

• Develop a logical sequence of questions – think of this as a Develop a logical sequence of questions – think of this as a 
conversation rather than an interview

conversation rather than an interview

• Start with easy/gently questions of the “tell me about Start with easy/gently questions of the “tell me about 
yourself” type

yourself” type

• Consider which questions should be pre-coded and which Consider which questions should be pre-coded and which 
should be open-ended. For example, a question on marital 
should be open-ended. For example, a question on marital

status could be
status could be

– Precoded: Precoded: 1 if single; 2 if married; 3 if widowed; 4 if 1 if single; 2 if married; 3 if widowed; 4 if 
divorced; 5 if other.

divorced; 5 if other.

– Open ended: How would you describe your marital Open ended: How would you describe your marital

status?

• Use skip patterns:Use skip patterns:

– “If yes, go to Q10”“If yes, go to Q10”

</div>
(28)<div class='page_container' data-page=28>

Questionnaire Design, cont’d

• Think about recall periods:Think about recall periods:
– TelescopingTelescoping

– Important but infrequent eventsImportant but infrequent events

• Think carefully about how to approach sensitive topicsThink carefully about how to approach sensitive topics

• Break down complex questions into partsBreak down complex questions into parts
– Not. How much did you earn from wage work.Not. How much did you earn from wage work.

– Better. How many hours per day did you work in the last seven Better. How many hours per day did you work in the last seven 
days? How many days per week did you work in the last seven 
days? How many days per week did you work in the last seven 
days? How much are you paid for this wage work, allowing 
days? How much are you paid for this wage work, allowing 
respondent to answer in terms of hourly, daily or weekly wage
respondent to answer in terms of hourly, daily or weekly wage

</div>
(29)<div class='page_container' data-page=29>

Questionnaire Design: Pilot Testing

 Before starting your survey, you need to make sure that Before starting your survey, you need to make sure that

your questionnaire works – this is called pilot testing.

 You should try the questionnaire on 10-20 respondents, You should try the questionnaire on 10-20 respondents,

who represent a variety of respondent ‘types’

 Pilot testing should reveal the following:Pilot testing should reveal the following:

a)

a) Are definitions used in the questionnaire appropriate. This Are definitions used in the questionnaire appropriate. This 
applies both to definitions of units of observation (does the 
applies both to definitions of units of observation (does the 
definition of a household correspond with the definition 
definition of a household correspond with the definition 
used by the people being studied) and to particular

used by the people being studied) and to particular 
questions (eg "holdings"; "assets"; "income")

questions (eg "holdings"; "assets"; "income")
b)

b) Do respondents understand the questionsDo respondents understand the questions
c)

</div>
(30)<div class='page_container' data-page=30>

Pilot Testing, cont’d

d)

d) Are questions being asked that cause respondents unease Are questions being asked that cause respondents unease 
or do they refuse to answer

or do they refuse to answer
e)

e) Are there problems associated with translating particular Are there problems associated with translating particular 
concepts

concepts
f)

f) Is the layout and sequencing of questions sensibleIs the layout and sequencing of questions sensible

g)

g) Can greater use be made of pre-coding Can greater use be made of pre-coding

h)

</div>
(31)<div class='page_container' data-page=31>

Pilot Testing, cont’d

 After pilot testing, review results with enumeratorsAfter pilot testing, review results with enumerators
 Depending on results, re-do with a smaller number of Depending on results, re-do with a smaller number of

respondents (5-10)

</div>
(32)<div class='page_container' data-page=32>

Questionnaire design: A final word

 When doing an evaluation survey with a longitudinal When doing an evaluation survey with a longitudinal

design, it is extremely important that at the time of the

baseline, you obtain information on how to contact

respondents in the future:

• GPS is idealGPS is ideal

• Cellphone numbersCellphone numbers

• Information on people who could aid in a follow up contactInformation on people who could aid in a follow up contact

 Not only is this information helpful, it can also be used as Not only is this information helpful, it can also be used as

additional regressors in attrition probits

</div>
(33)<div class='page_container' data-page=33>

Using PDAs or handheld computers in surveys

 There is increasing interest in using Personal Digital There is increasing interest in using Personal Digital

Assistants (PDAs) such as PalmPilots or handheld

computers for data collection.

 Using these requires:Using these requires:

• Purchasing the hardware ($200-$500 each)Purchasing the hardware ($200-$500 each)

• Purchasing the software, such as Pendragon Forms or PC Purchasing the software, such as Pendragon Forms or PC 
Pocket Creations

Pocket Creations

• Some one to write the data entry program using this Some one to write the data entry program using this 
software

</div>
(34)<div class='page_container' data-page=34>

Handhelds: Advantages

 Speed: Data are available for use immediatelySpeed: Data are available for use immediately

 Filters and skips can ensure that unnecessary questions Filters and skips can ensure that unnecessary questions

are not asked

</div>
(35)<div class='page_container' data-page=35>

Handhelds: Non-Issues

 Its harder to train enumeratorsIts harder to train enumerators

• Not in our experience.Not in our experience.

 Battery life is a problemBattery life is a problem

• Buy extra batteriesBuy extra batteries

 Transferring data to computers is hardTransferring data to computers is hard

</div>
(36)<div class='page_container' data-page=36>

Handhelds: Disadvantages

 Need access to reliable source of electricityNeed access to reliable source of electricity
 Sometimes hard to see in bright daylightSometimes hard to see in bright daylight

 In our experience, data entry errors are more frequent In our experience, data entry errors are more frequent

when using PDAs compared to paper questionnaires and

back office data entry, although these may converge with

further experience

 In our experience, surveys take slightly longer when using In our experience, surveys take slightly longer when using

handhelds for numeric information and much longer for

text. Consequently, enumerators are much less likely to

</div>
(37)<div class='page_container' data-page=37>

Handhelds: Disadvantages

 If a response does not seem quite right, it is relatively If a response does not seem quite right, it is relatively

straightforward to take the questionnaire back into the

field. Note too that checking one response often leads to

revisions to other responses.

 If a PDA is lost, then all data are lost. In some cases (such If a PDA is lost, then all data are lost. In some cases (such

as Pocket PCs, this is also true if the battery dies)

 Safety/security of enumerators – carrying an expensive Safety/security of enumerators – carrying an expensive

electronic device can make them a target for criminals

</div>
(38)<div class='page_container' data-page=38>

Data Entry, Cleaning, and Management

 Researchers typically spend a lot of time thinking about Researchers typically spend a lot of time thinking about

critical issues at the outset of their project (sample size

calculations, questionnaires) and at the end of the project

(data analysis, report writing).

 They typically spend less time worrying about the They typically spend less time worrying about the

intermediate phase: data entry, cleaning and

management.

 This is a mistake. Many evaluation studies fall apart, or This is a mistake. Many evaluation studies fall apart, or

fall badly behind schedule, because insufficient attention

is paid to data management

</div>
(39)<div class='page_container' data-page=39>

Data Management, cont’d

 Avoiding these problems requires:Avoiding these problems requires:

• Paying attention to the design and implementation of the Paying attention to the design and implementation of the 
data entry software early

data entry software early

</div>
(40)<div class='page_container' data-page=40>

Data Management, cont’d

 Software. Good choices include:Software. Good choices include:

• CS PRO (CS PRO (www.cspro.orgwww.cspro.org))

• Microsoft Access (often bundled with Microsoft Office)Microsoft Access (often bundled with Microsoft Office)

• SPSS/Data Entry Module (but this can be expensive)SPSS/Data Entry Module (but this can be expensive)

 Start work on data entry programs as soon as Start work on data entry programs as soon as

questionnaire design (or designs of certain modules) is

finalized

</div>
(41)<div class='page_container' data-page=41>

Data Management, cont’d

 Example of data management protocol:Example of data management protocol:

1.

1. Enumerator checks questionnaire for completeness before Enumerator checks questionnaire for completeness before 
giving it to supervisor

giving it to supervisor
2.

2. Supervisor does quick check to make sure form is complete Supervisor does quick check to make sure form is complete 
and that critical topics are correct. She passes form to:

and that critical topics are correct. She passes form to:
3.

3. Data checkers/verifiers who go through form in detail. Data checkers/verifiers who go through form in detail. 
Forms are sent back to supervisors/enumerators for 
Forms are sent back to supervisors/enumerators for

correction. Otherwise, they are given to data entry team.
correction. Otherwise, they are given to data entry team.
4.

4. Data are entered:Data are entered:

– Question as to whether to use single or double data entryQuestion as to whether to use single or double data entry

– Compromise; enter 10-20% of data twice to do quality Compromise; enter 10-20% of data twice to do quality 
check on entry

check on entry
5.

5. Violations of range/value are sent back to field for checkingViolations of range/value are sent back to field for checking
6.

6. Additional checks are made when variable aggregates are Additional checks are made when variable aggregates are 
constructed

</div>
(42)<div class='page_container' data-page=42>

Data Management, cont’d



Finally, two major considerations:



You need system to keep track of

questionnaires – these cannot be lost!



Ensure data are backed up regularly.

• Minimum once a weekMinimum once a week

• This means that you have more than one copy held in This means that you have more than one copy held in

different locations

</div>
(43)<div class='page_container' data-page=43>

Budgets

 Budgeting for quantitative surveys has three Budgeting for quantitative surveys has three

components:

• What do you need?What do you need?

• How many ___ do you need?How many ___ do you need?

</div>
(44)<div class='page_container' data-page=44>

Budgets: What do you need?

 Typical line items in a survey budget:Typical line items in a survey budget:

• Pilot testing of questionnairesPilot testing of questionnaires

• Training of survey staffTraining of survey staff

• Enumerators (salary, payroll taxes and possibly per diem)Enumerators (salary, payroll taxes and possibly per diem)

• Supervisors (salary payroll taxes and possibly per diem)Supervisors (salary payroll taxes and possibly per diem)

• Drivers and TransportDrivers and Transport

• Printing of questionnairesPrinting of questionnaires

• Supplies (clipboards, pens, first aid kits etc)Supplies (clipboards, pens, first aid kits etc)

• Payments to respondents(?)Payments to respondents(?)

• VerifiersVerifiers

• Training of data entry staffTraining of data entry staff

</div>
(45)<div class='page_container' data-page=45>

Budget: How many

 ““How many” depends on:How many” depends on:

• The size of the sample you are collectingThe size of the sample you are collecting

• Whether it is concentrated in a few places or widely Whether it is concentrated in a few places or widely 
disbursed

disbursed

• How quickly you want to complete the surveyHow quickly you want to complete the survey

 Example.Example.

• My survey is working in 40 spatially disbursed locations.My survey is working in 40 spatially disbursed locations.

• I am interviewing 30 households in each location. I am interviewing 30 households in each location.

• Each interview lasts approximately two hours and so I Each interview lasts approximately two hours and so I

assume that an interviewer can complete two interviews per 
assume that an interviewer can complete two interviews per

day
day

• I assume that enumerators work five days, then have one I assume that enumerators work five days, then have one 
day off

</div>
(46)<div class='page_container' data-page=46>

Budget: How many

 If I hire three enumerators, they will complete the survey If I hire three enumerators, they will complete the survey

in one locality in five days

• Each day, 6 interviews are completed (3 enumerators x 2 interviews Each day, 6 interviews are completed (3 enumerators x 2 interviews

per day)

• In five days, 30 interviews are completed (6 per day x 5 days)In five days, 30 interviews are completed (6 per day x 5 days)
• One day to move to new site, one day offOne day to move to new site, one day off

• Repeat three timesRepeat three times

 This implies that one team of three enumerators will cover This implies that one team of three enumerators will cover

four localities in one month

 So, ten teams of enumerators (30 enumerators in total) So, ten teams of enumerators (30 enumerators in total)

will complete survey over a four week period

 Suppose you allocate one supervisor to two teams.Suppose you allocate one supervisor to two teams.

</div>
(47)<div class='page_container' data-page=47></div>
(48)<div class='page_container' data-page=48>

Topics

 Selecting enumeratorsSelecting enumerators

 Selecting SupervisorsSelecting Supervisors

 Interacting with respondentsInteracting with respondents

• Preparing the ground Preparing the ground

• Interview settingInterview setting

• Interacting with respondents, including:Interacting with respondents, including:

– Informed consentInformed consent
– PaymentsPayments

</div>
(49)<div class='page_container' data-page=49>

Enumerators

 The job description for the ideal enumerator would The job description for the ideal enumerator would

include:

• Communications skillsCommunications skills

• Good knowledge of English (or French, or Portuguese or Good knowledge of English (or French, or Portuguese or 
Spanish) as well as the local language(s)

Spanish) as well as the local language(s)

• A perceptive intelligence,A perceptive intelligence,

• inexhaustible patienceinexhaustible patience

• unfailing dependabilityunfailing dependability

• Wonderful people skillsWonderful people skills

• Willingness to work long hoursWillingness to work long hours

• An ability to get along with all elements of the local An ability to get along with all elements of the local 
population.

population.

</div>
(50)<div class='page_container' data-page=50>

Enumerators, cont’d

 Some considerations:Some considerations:

• Level of schoolingLevel of schooling

• GenderGender

</div>
(51)<div class='page_container' data-page=51>

Supervisors

 Supervisors should be able to do the followingSupervisors should be able to do the following

a)

a) Maintaining time schedules and co-ordinating workMaintaining time schedules and co-ordinating work
b)

b) Checking the enumerator's work by sitting in on interviewsChecking the enumerator's work by sitting in on interviews
c)

c) Checking completed questionnaires for misunderstandings Checking completed questionnaires for misunderstandings 
and recording errors

and recording errors
d)

d) Liaising and trouble-shooting with respondents and local Liaising and trouble-shooting with respondents and local 
community leaders

community leaders
e)

e) Acting as courier - transferring equipment and documents Acting as courier - transferring equipment and documents 
from the office to the field and vice versa

from the office to the field and vice versa
f)

f) Covering in emergencies (eg arranging interviews if an Covering in emergencies (eg arranging interviews if an 
enumerator falls sick)

enumerator falls sick)
g)

</div>
(52)<div class='page_container' data-page=52>

Interacting … : Preparing the ground

 You can’t just show up and start asking questions.You can’t just show up and start asking questions.

 In both rural and urban areas, even if you have research In both rural and urban areas, even if you have research

clearance, it is a good idea to contact local government

officials to inform them about the survey work you intend

to carry out.

 It is also helpful to have a standard ‘story’ or explanation It is also helpful to have a standard ‘story’ or explanation

that you and your research team can provide to any one

curious about your work.

</div>
(53)<div class='page_container' data-page=53>

Interacting with respondents: Informed consent

 Before an interview can take place, you must obtain Before an interview can take place, you must obtain

“informed consent” from your respondents.

 Specifically, you need to explain to Specifically, you need to explain to those being those being

interviewed:

• The purpose of their studyThe purpose of their study

• That participation is voluntaryThat participation is voluntary

• Who has funded and supported itWho has funded and supported it
• How the data will be collectedHow the data will be collected

• Who they can contact if they have questionsWho they can contact if they have questions

• How the results may be used.How the results may be used.

• How researchers will maintain anonymity and/or confidentiality.How researchers will maintain anonymity and/or confidentiality.

 Consent must be derived from the actual people involved, Consent must be derived from the actual people involved,

not just officials or leaders.

</div>
(54)<div class='page_container' data-page=54>

Informed consent, continued

 The best approach to informed consent is to prepare two copies The best approach to informed consent is to prepare two copies

of a document (or letter) which contains the information 
of a document (or letter) which contains the information 
described on the previous slide.

described on the previous slide.

 At the time of first contact, enumerators review the letter with At the time of first contact, enumerators review the letter with

respondents. The respondent keeps one copy and signs the 
respondents. The respondent keeps one copy and signs the 
other which the survey team retains.

other which the survey team retains.

 There should be a log kept indicating who has given voluntary There should be a log kept indicating who has given voluntary

consent.
consent.

 In the case of school children (or minors), there is an issue as to In the case of school children (or minors), there is an issue as to

</div>
(55)<div class='page_container' data-page=55>

Interacting … : The Interview Setting

 The decision regarding their setting has two components:The decision regarding their setting has two components:

• Where to conduct an interview, and Where to conduct an interview, and

• When. When.

 The principal decision with respect to interview location is The principal decision with respect to interview location is

the relative merits of a private versus public encounter.

 A public encounter may encourage a more general A public encounter may encourage a more general

discussion of the issues; and where information is in the

public domain, onlookers may assist or correct respondents

with particular questions (although there is no guarantee

that they will).

 When the research deals with matters that are personally When the research deals with matters that are personally

sensitive, the presence of outsiders or even other family

members may inhibit respondents, embarrassing them into

evasion or silence.

</div>
(56)<div class='page_container' data-page=56>

The Interview Setting, cont’d

 Onlookers may even encourage respondents to answer Onlookers may even encourage respondents to answer

untruthfully. For example, working during food crises is

problematic (apart from posing moral dilemmas) because

respondents have an incentive to understate their stocks

of grain, and their general wealth, in the expectation that

food aid will be brought into the community. This

tendency will be exacerbated in a public interview, since

the respondent who admits to being wealthy may face

demands for help from poorer neighbours.

 In general, therefore, the more sensitive the topic, the In general, therefore, the more sensitive the topic, the

stronger the case for conducting the interview in private.

</div>
(57)<div class='page_container' data-page=57>

The Interview Setting, cont’d

 However, often there is no choice. It may be difficult, for However, often there is no choice. It may be difficult, for

example, to interview workers in firms without their

supervisors, or fellow employees being present. In some

rural areas, it may be impossible to keep neighbours and

visitors away.

 But ways can be found around this problem.But ways can be found around this problem.

• Changed the subject of the interview to avoid sensitive Changed the subject of the interview to avoid sensitive 
topics while other people were around.

topics while other people were around.

• Another solution is to catch people alone - on coffee breaks, Another solution is to catch people alone - on coffee breaks, 
in the fields working, or on their way to market.

</div>
(58)<div class='page_container' data-page=58>

The Interview Setting: When to conduct?

 Interviews can be a real imposition on respondents' time. Interviews can be a real imposition on respondents' time. 
 It is important to meet on days and at times convenient to It is important to meet on days and at times convenient to

respondents.

 One way of ensuring that interviews are not an imposition One way of ensuring that interviews are not an imposition

is to make appointments to see people.

• For example, by making initial contact (and obtaining For example, by making initial contact (and obtaining 
informed consent) and leaving 'calling cards' and/or 
informed consent) and leaving 'calling cards' and/or

calenders.
calenders.

</div>
(59)<div class='page_container' data-page=59>

When to conduct, cont’d

 A related issue is deciding when in the year to conduct the A related issue is deciding when in the year to conduct the

interview.

 This is not usually an issue for traders or manufacturing This is not usually an issue for traders or manufacturing

firms (though food traders are likely to be very busy

around the harvest period) but can be a matter of concern

in rural localities.

 In semi-arid areas, for instance, it may be easier to work In semi-arid areas, for instance, it may be easier to work

mainly during the dry season when people who are

actively engaged in farming have time to spare.

 At a practical level, travelling around a community may be At a practical level, travelling around a community may be

debilitating during the heat of the dry season, but can

become virtually impossible during the rains.

</div>
(60)<div class='page_container' data-page=60>

When to conduct, cont’d

 On the other hand, farmers in semi-arid areas do not work On the other hand, farmers in semi-arid areas do not work

on their farms for half the year and sit around under trees

waiting to be interviewed by researchers for the other

half! Seasonal out-migration is common in such

communities, notably among young men, and it would be

myopic to defer interviewing until after the harvest if a

large number of respondents then disappears from the

village.

 Interviewing schedules should be built around the Interviewing schedules should be built around the

seasonal (and daily and weekly) pattern of domestic and

economic activities.

</div>
(61)<div class='page_container' data-page=61>

Paying respondents

 In general, researchers do not like to pay respondents in cash for In general, researchers do not like to pay respondents in cash for

interviews. But interviewing does take time, and can cost

respondents money in terms of lost income.

 When interviews imply a financial sacrifice, some form of When interviews imply a financial sacrifice, some form of

compensation may be appropriate.

 In the context of longitudinal or evaluation surveys, payments are In the context of longitudinal or evaluation surveys, payments are

a means of reducing attrition.

 One approach is to pay respondents the equivalent of a daily One approach is to pay respondents the equivalent of a daily

wage, in recognition of the fact that they were giving up a day's

labour.

 An alternative to cash payment is to offer small gifts as a tangible An alternative to cash payment is to offer small gifts as a tangible

expression of thanks (for education related interventions, pens,

notebooks or inexpensive calculators are an option).

</div>
(62)<div class='page_container' data-page=62>

Paying respondents, cont’d

 But compensation might create friction between But compensation might create friction between

respondents as well.

• If significant inequalities exist within the community, equal If significant inequalities exist within the community, equal

payments are inappropriate - a sizeable sum to a landless labourer

will be an insulting pittance to a wealthy merchant.

• Paying wealthy respondents more than the poor only exacerbates Paying wealthy respondents more than the poor only exacerbates 
existing inequalities, whereas offering payment or gifts only to the

existing inequalities, whereas offering payment or gifts only to the

poor appears to favour some people at the expense of others.

• Further, cash compensation can interfere with the data collection Further, cash compensation can interfere with the data collection

exercise, particularly if it is more than nominal, and if it relates

(directly or indirectly) to the research issues.

</div>
(63)<div class='page_container' data-page=63>

Paying respondents, cont’d

 Further, as soon as fieldworkers give gifts, pay informants in Further, as soon as fieldworkers give gifts, pay informants in

cash, or lend money, a precedent is set which inevitably leads to 
cash, or lend money, a precedent is set which inevitably leads to 
escalating expectations.

escalating expectations.

 Failing to meet these may compromise both the current Failing to meet these may compromise both the current

fieldwork and subsequent research projects. 
fieldwork and subsequent research projects.

 Conversely, if no payment is offered, non respondents may Conversely, if no payment is offered, non respondents may ‑‑

ridicule participants for being 'used' for nothing. 
ridicule participants for being 'used' for nothing.

 Others who are excluded from the sampling frame may want to Others who are excluded from the sampling frame may want to

be interviewed (and may resent being excluded, believing that 
be interviewed (and may resent being excluded, believing that 
those selected for interviews will benefit somehow), while some 
those selected for interviews will benefit somehow), while some 
who are selected may complain that they have been picked on to 
who are selected may complain that they have been picked on to 
answer interminable boring questions.

</div>
(64)<div class='page_container' data-page=64>

Paying respondents, cont’d

 In some cases, it may be more appropriate to offer a In some cases, it may be more appropriate to offer a

single, sizeable contribution to the wider community as a

whole.

 This, too, can take a number of forms (depending on what This, too, can take a number of forms (depending on what

seems appropriate), such as a social event, contributions

to community fund raising, a health fair.

 It is important to donate something to which all members It is important to donate something to which all members

of the community will have equal access and to ensure

that the gift does not distort research-related outcomes,

unless its presentation is deferred until departure.

</div>
(65)<div class='page_container' data-page=65>

Paying respondents, cont’d

 There is no simple answer to the question of payment to There is no simple answer to the question of payment to

respondents.

 There are strong reasons for providing some form of There are strong reasons for providing some form of

recompense for the time and trouble taken by

respondents

 But there are many dangers in doing so. But there are many dangers in doing so.

 Perhaps the best advice is to provide some tangible Perhaps the best advice is to provide some tangible

means of appreciation, but do so in a manner sensitive to

the circumstances of the research.

</div>
(66)<div class='page_container' data-page=66>

Interacting … : Post-Research Obligations

 Interactions with respondents does not end with the Interactions with respondents does not end with the

completion of the last questionnaire. Researchers have

responsibilities to their respondents.

 A primary obligation is to tell respondents' story A primary obligation is to tell respondents' story

truthfully.

 An equally important obligation is to maintain the An equally important obligation is to maintain the

anonymity of respondents. This extends beyond not

using names in the context of writing research reports;

there is a strong case for not entering individual names

onto computer files (and relying on identification numbers

</div>
(67)<div class='page_container' data-page=67>

Post-Research Obligations, cont’d

 Finally, respondents are often interested in the findings of Finally, respondents are often interested in the findings of

your research. Some means of disseminating results

should be built into the project design.

</div>


Tài liệu MIT Joint Program on the Science and Policy of Global Change: Effects of Air Pollution Control on Climate pdf

andy warhol the life and art of the prince of pop 1775

Manhood Of Humanity The Science and Art of Human Engineering potx

Integrative Pain Medicine The Science and Practice of Complementary and Alternative Medicine in Pain Management docx

creating emotion in games the craft and art of emotioneering

bridges the science and art of the worlds most inspiring structures mar 2010

The Science and Technology of Carbon Nanotubes III ppt

The Science and Technology of Carbon Nanotubes pptx

The Science and Technology of Materials in Automotive Engines Part 1 doc

The Science and Technology of Materials in Automotive Engines Part 3 doc

The Science and Art of Implementing Quantitative Evaluation Surveys

<b>The Science and Art of Implementing </b>

<b>The Science and Art of Implementing </b>

<b>Quantitative Evaluation Surveys</b>

<b>Quantitative Evaluation Surveys</b>

<i><b>John Hoddinott</b></i>

<i><b>John Hoddinott</b></i>

<b>Introduction</b>

<b>Introduction</b>

<b>The science …</b>

<b>The science …</b>

<b>Drawing the sample</b>

<b>Drawing the sample</b>

<b>Sampling</b>

<b>Sampling</b>

<b>Sampling, cont’d</b>

<b>Sampling, cont’d</b>

<b>Sampling, cont’d</b>

<b>Sampling, cont’d</b>

<b>Sampling, cont’d</b>

<b>Sampling, cont’d</b>

<b>Sample Size Calculations</b>

<b>Sample Size Calculations</b>

<b>Sample Size Calculations, cont’d</b>

<b>Sample Size Calculations, cont’d</b>

<b>Sample Size Calculations, cont’d</b>

<b>Sample Size Calculations, cont’d</b>

<b>Sample Size Calculations, cont’d</b>

<b>Sample Size Calculations, cont’d</b>

<b>Sample Size Calculations, cont’d</b>

<b>Sample Size Calculations, cont’d</b>

<b>Sample Size Calculations, cont’d</b>

<b>Sample Size Calculations, cont’d</b>

<b>Sample Size Calculations, cont’d</b>

<b>Sample Size Calculations, cont’d</b>

<b>Sample Size Calculations: Concluded</b>

<b>Sample Size Calculations: Concluded</b>

<b>The art</b>

<b>The art</b>

<b>The Hard Arts</b>

<b>The Hard Arts</b>

<b>Questionnaire Design</b>

<b>Questionnaire Design</b>

<b>Questionnaire Design, cont’d</b>

<b>Questionnaire Design, cont’d</b>

<b>Questionnaire Design, cont’d</b>

<b>Questionnaire Design, cont’d</b>

<b>Questionnaire Design, cont’d</b>

<b>Questionnaire Design, cont’d</b>

<b>Questionnaire Design: Pilot Testing</b>

<b>Questionnaire Design: Pilot Testing</b>

<b>Pilot Testing, cont’d</b>

<b>Pilot Testing, cont’d</b>

<b>Pilot Testing, cont’d</b>

<b>Pilot Testing, cont’d</b>

<b>Questionnaire design: A final word</b>

<b>Questionnaire design: A final word</b>

<b>Using PDAs or handheld computers in surveys</b>

<b>Using PDAs or handheld computers in surveys</b>

<b>Handhelds: Advantages</b>

<b>Handhelds: Advantages</b>

<b>Handhelds: Non-Issues</b>

<b>Handhelds: Non-Issues</b>

<b>Handhelds: Disadvantages</b>

<b>Handhelds: Disadvantages</b>

<b>Handhelds: Disadvantages</b>

<b>Handhelds: Disadvantages</b>

<b>Data Entry, Cleaning, and Management</b>

<b>Data Entry, Cleaning, and Management</b>

<b>Data Management, cont’d</b>

<b>Data Management, cont’d</b>

<b>Data Management, cont’d</b>

<b>Data Management, cont’d</b>

<b>Data Management, cont’d</b>

<b>Data Management, cont’d</b>

<b>Data Management, cont’d</b>

<b>Data Management, cont’d</b>



<b><sub>Finally, two major considerations:</sub></b>

<b><sub>Finally, two major considerations:</sub></b>