Assessment in counseling chapter 4

Validity and Item
Analysis
Chapter 4


Validity
∗ Concerns what instrument measures and how well
it does so
∗ Not something instrument “has” or “does not
have”
∗ Informs counselor when it is appropriate to use
instrument and what can be inferred from results
∗ Reliability is a prerequisite for validity


Traditional Categories of Validity
∗ Content-Related
∗ Criterion-Related
∗ Construct


Validation Evidence: Standards for
Educational and Psychological Testing
∗

Evidence Based on Test Content

∗

Evidence Based on Response Processes

∗

Evidence Based on Internal Structure

∗

Evidence Based on Relations to Other Variables

∗

Evidence Based on Consequences of Testing


Evidence Based on Test Content
∗ Degree to which the evidence indicates that items,
questions, or tasks adequately represent intended
behavior domain
∗ Central focus is typically on how the instrument’s
content was determined
∗ Content-related validation evidence should not be
confused with face validity


Evidence Based on Response Processes
∗ Concerns whether individuals respond in a manner
consistent with construct measured
∗ May also examine information processing differences
by subgroup
∗ Computer simulated performances

Evidence Based on Internal Structure
∗ Examining internal structure using factor analysis
∗ Can also examine internal structure of instrument for
different subgroups (differential item functioning)


Evidence Based on Relations to Other
Variables
∗

Correlational method

∗ Convergent and discriminant evidence
∗ Multitrait-multimethod matrix
∗

Prediction or instrument-criterion relationship

∗ Concurrent and predictive validity
∗ Regression
∗ Decision Theory
∗

Validity Generalization


Correlational Method
∗ Correlational method 
examines relationship

between instrument and
criterion or other
pertinent variables
∗ Validity coefficient

∗ Multitrait-multimethod
matrix  examines
relationships between
instrument and traits that
are theoretically related
and unrelated to it


Prediction/Instrument-Criterion
Relationship
∗ Concurrent validity
∗ Predictive validity


Regression

a = y-intercept
b = slope
X = score


Standard Error of Estimate
∗ The margin of expected error in the individual’s
predicted criterion score as a result of imperfect
validity

2

Sest = Sy 1− r xy


Decision Theory: Expectancy Table


Expectancy Table (cont.)


Expectancy Table (cont.)


Validity Generalization
∗

Method of combining validation studies to determine if validity evidence can be generalized

∗ Must have a substantial number of studies
∗ Must use meta-analytic procedures


Evidence-Based Consequences
of Testing
∗

Examples of social consequences:

∗ Group differences on tests used for employment selection
∗ Group differences in placement in special education
∗

Counselors should consider both validation evidence and social implications


Conclusion on Validation Evidence
∗ Gradual accumulation of evidence
∗ Counselor must evaluate the information to
determine if it appropriate for which client under
what circumstance
∗ Validation evidence should also be considered in
informal assessments


Item Analysis
∗ Examining and evaluating each item in
instrument
∗ Item difficulty:
p = # who answered item correctly
total # of individuals

∗ Item discrimination:
∗ Extreme group method: d = upper % – lower %
∗ Correlational method


Item Response Theory
∗ Focus is on each item and establishing items that

measure ability or level of a latent trait
∗ Involves examining item characteristic curve and
calibration of each individual item
∗ Parameters:
∗ Difficulty
∗ Slope (discrimination)
∗ Probability of guessing


Item Characteristic Curve