Ecological validity of neuropsychological tests focus on the california verbal learning test and the wisconsin card sorting test

Archives of Clinical Neuropsychology, Vol. 13, No. 6, pp. 523–534, 1998
Copyright  1998 National Academy of Neuropsychology
Printed in the USA. All rights reserved
0887-6177/98 $19.00 ϩ .00

PII S0887-6177(97)00038-3

Ecological Validity of Neuropsychological
Tests: Focus on the California Verbal Learning
Test and the Wisconsin Card Sorting Test
Michelle Y. Kibby
The University of Memphis

Maureen Schmitter-Edgecombe
Washington State University

Charles J. Long
The University of Memphis

To date, few studies have assessed the ecological validity of current measures of memory and executive functioning, particularly the California Verbal Learning Test (CVLT) and the Wisconsin Card
Sorting Test (WCST). A battery of tests, including the CVLT and the WCST, was administered to 28
severely head-injured adults who were 1 year or more postinjury. Hierarchical linear regression
was performed to assess the relationship between these cognitive tests and work functioning while
statistically controlling for physical disability, emotional distress, and memory compensation strategies utilized by the participants. Results suggest that the CVLT was best at predicting performance
on the job; however, the CVLT and the WCST were about equally predictive of the type of position
held by a participant. Hence, this study demonstrates the ecological validity of measures of memory
and executive functioning for predicting work-related skills.  1998 National Academy of Neuropsychology. Published by Elsevier Science Ltd

INTRODUCTION
There has been a changing focus in clinical neuropsychology. With the availability of increasingly sophisticated neuroradiological procedures, referral questions are now more focused
on predicting what a patient can do in the everyday world and less on diagnosing and localizing brain impairment. Clinical neuropsychologists are being asked to make statements regarding a patient’s functional skills, ability to return to work, and competence in tasks of community living. These concerns are what are referred to by the term ecological (Heinrichs, 1990;

Sbordone & Long, 1995). The changing referral questions bring up a critical issue: most
neuropsychological tests were developed to detect, and attempt to localize, the presence of
brain damage rather than to be ecologically sound. To date, research establishing the ecologiAddress correspondence to: Michelle Kibby, The University of Memphis, Department of Psychology, Memphis,
TN 38152. E-mail:

523


524

M. Y. Kibby et al.

cal validity of neuropsychological tests has been limited (for a detailed review see Heinrichs,
1990; Long & Kibby, 1995; Puente, 1992; Sbordone & Long, 1995). Neuropsychological
tests that were once validated for their ability to detect brain damage now must be evaluated
for their utility as ecological instruments. The present study investigated the ability of memory and executive functioning measures to predict employment status.
Memory and executive functioning have been among the most common cognitive constructs studied by researchers investigating the ecological validity of traditional neuropsychological tests. This may be because, unlike other cognitive constructs, such as attention, there
are widely accepted measures of memory and executive functioning that are normed with
demonstrated reliability and validity.
Researchers have begun to demonstrate the ecological validity of memory tests such as
Reitan’s Story Memory Test (Newnan, Heaton, & Lehman, 1978) and the Wechsler Memory
Scale (Heaton, Chelune, & Lehman, 1978). For example, tests of short- and long-term memory have been found to be important predictors of return to work (Heaton et al., 1978; Weddell, Oddy, & Jenkins, 1980), of average wage earned on the job (Newnan et al., 1978), and
of successful completion of a vocational evaluation (Ryan, Sautter, Capps, Meneese, & Barth,
1992). In addition, memory functioning has also been shown to predict individual’s ability
to perform self-care skills or to re-enter the community (Goldstein, McCue, Rogers, & Nussbaum, 1992; Richardson, Nadler, & Malloy, 1995).
Traditional measures of executive functioning (e.g., Category Test) have also been shown
to be predictive of return to work and of average wage earned. The Category Test, in association with other measures, was able to discriminate between employed and chronically unemployed brain-injured persons and to predict average wage earned on the job (Newnan et al.,
1978). Nonetheless, in this study, memory functioning better predicted occupational status
(type of position held by an individual) than the Category test. In a related study, Dikmen
and Morgan (1980) found that measures of memory functioning and flexibility in thinking

best discriminated between employed and unemployed individuals with epilepsy, although
the Category Test gained greater importance with increasing levels of occupational status.
This is not surprising as all jobs require some level of memory functioning to perform,
whereas manual labor positions may require much less executive functioning to perform than
higher-level positions.
While the literature on memory and executive functioning has made strides toward filling
the void of research on the ecological validity of neuropsychological tests, there are still two
important issues that need to be addressed. First, studies have only investigated whether
neuropsychological instruments can predict return to work. In contrast, little attention has
been given to the utility of these measures for predicting how competently an individual can
perform on the job. Many individuals are able to return to some type of employment after
brain injury. However, they often return to positions of lower status than they were working
at before the injury, or they need additional assistance to continue working at the same type
of job. Even those that are able to return to the same level position without additional assistance often experience emotional distress related to an inability to perform at the same level
as prior to the injury. Studies that only investigate whether an individual returns to work are
not able to assess how well the person is functioning on the job as compared to before the
injury. One of the purposes of this study was to investigate how well tests of memory and
executive functioning predict competency on the job postinjury (labeled ‘‘job performance’’).
A second major issue is that the majority of research conducted in his area has failed to
control for relevant factors, such as the presence of physical disability and/or emotional
disturbance. Both of these factors have been shown to be strong predictors of return to work.
For example, the presence of motor deficits and other system injuries have been found to
be inversely related to return to work (Fraser, Dikmen, McLean, Miller, & Temkin, 1988;


Ecological Validity of the CVLT and the WCST

525

Stambrook, Moore, Peter, Deviaene, & Hawryluk, 1990). Physical disability may effect one’s

ability to obtain and maintain employment, particularly in the case of laborer positions or
positions that entail a great deal of stamina. Stambrook et al. (1990) also found that patient
self-report of depression and psychosocial difficulties were inversely related to postinjury
vocational status. Failure to take into account such factors may confound the relationship
between neuropsychological test data and work functioning. In fact, researchers utilizing the
clinical scales of the Minnesota Multiphasic Personality Inventory (MMPI) in conjunction
with neuropsychological measures have found the combination of neuropsychological and
personality measures to be better at predicting return to work than using either measure alone
(Heaton et al., 1978; Newnan et al., 1978).
The present study investigated the ecological validity of two currently used measures of
executive and memory functioning that are appearing more frequently in the literature: the
Wisconsin Card Sorting Test (Heaton, 1981) and the California Verbal Learning Test (Delis,
Kramer, Kaplan, Ober, & Fridlund, 1987). Both of these instruments combine experimental
methodology with a standardized, clinical measure. Currently, however, little is still known
about the ecological validity of these instruments in terms of predicting occupational status
or job performance.
WISCONSIN CARD SORTING TEST
The Wisconsin Card Sorting Test (WCST) is a test of concept formation that evaluates
a person’s ability to ‘‘shift set’’ and avoid perseverative responding. It is possible for a person
to do well on the conceptually more difficult Category Test and quite poorly on the WCST
because of the latter task’s requirement of shifting response set (Bennett, 1988). Previous
research has demonstrated the use of the WCST for diagnosing brain impairment. For example, the measure of perseverative responding is especially sensitive to frontal lobe dysfunction
(Heaton, 1981). Recent research has also begun to demonstrate its ecological validity. The
WCST Perserverative Responses significantly correlated with head-injured males’ ability to
perform activities of daily living (Little, Templer, Persel, & Ashley, 1996). Lysaker, Bell,
and Beam-Goulet (1995) found that the WCST Trials to First Category and Total Number
Correct successfully predicted task orientation at a vocational work placement in a sample
of persons with schizophrenia. Research to date has yet to investigate the utility of the WCST
for predicting job performance and occupational status, however.
CALIFORNIA VERBAL LEARNING TEST

The California Verbal Learning Test is a measure of memory for auditorily presented
material. Research has shown that persons with head injury performed worse on immediate
recall, delayed recall, and recognition measures than controls (Crosson, 1988). Persons who
had sustained head injuries were also less effective at utilizing internal encoding strategies
that may have led to their deficient recall. As with most neuropsychological instruments,
research has attempted to demonstrate its utility in diagnosing brain impairment, yet much
is still unknown as to how this instrument predicts competency in meeting the demands of
everyday life. This issue is critical, given some of the psychometric limitations that have
been raised about the instrument (e.g., inflated norms) (Randolph, Gold, Kozora, Cullum,
Hermann, & Wyler, 1994; see Elwood, 1995 for a review).
Three factors make this study unique. First, the study focuses on a chronic population.
Neuropsychologists in outpatient settings often see clients with severe head injuries for the
first time several months postinjury, and few studies have investigated the ecological validity


526

M. Y. Kibby et al.

of neuropsychological tests under such conditions. Second, physical disability and emotional
distress were assessed so their relationship to return to work could be investigated and statistically controlled. The use of memory compensation techniques were also measured, as use
of these techniques may act to improve performance on the job despite poor memory functioning and, thereby, influence the relationship between the two. Third, this investigation takes
evaluating return to work a step further by assessing the qualitative aspects of how an individual is performing on the job (‘‘job performance’’), as well as assessing whether the individual
returns to work.
Based upon the literature review, it is predicted that both the WCST and the CVLT will
be predictive of job performance and occupational status when physical disability, emotional
distress, and use of memory compensation techniques are taken into account. However, it is
hypothesized that the CVLT may be a better predictor of job performance than the WCST.
This hypothesis is based on research by Dikman and Morgan (1980) who found that memory
was a strong predictor of employment status in general; executive functioning was predictive

only as occupational status increased.
METHOD
Participants
This study utilized 28 participants with a diagnosis of closed head injury who were from
Memphis, Tennessee and the surrounding area. Participants were classified as having a severe
head injury if they suffered from posttraumatic amnesia (PTA) of 7 days or more; 89% of
the participants reportedly had PTA of over 2 weeks, and 82% had PTA over 1 month. PTA
was assessed retrospectively by careful clinical questioning of the participant and a significant
other. Additionally, because we were interested in evaluating the chronic affects of severe
closed head injury, only those who were at least 1 year postinjury were studied; 89% of
those studied were more than 2 years postinjury, and 57% were more than 5 years postinjury.
The mean age at testing was 33.20 (SD ϭ 10.00, range ϭ 19.24–54.80 years), and mean
education was 13.89 years (SD ϭ 2.061, range ϭ 11–19 years). Fluid intelligence was assessed with the Culture Fair Intelligence Test (M ϭ 88.96, SD ϭ 13.15, range ϭ 50–105).
Procedure
Participants were tested as part of a larger study (Schmitter-Edgecombe, 1996). The WCST
and the CVLT were administered along with other cognitive neuropsychological measures.
Measures of physical disability, emotional distress, and memory compensation strategies
were also obtained. Physical disability was measured on a 3-point scale (1 ϭ no disability,
2 ϭ mild to moderate disability, 3 ϭ severe disability). Emotional disturbance was evaluated
with the Hopkins Symptom Checklist-58 (Derogatis, Lipman, Rickels, Uhlenhuth, & Covi,
1974). Memory compensation strategies were assessed by use of a checklist where participants rated the frequency with which they used each compensation strategy on a 4-point
scale. A low score on this measure is indicative of frequent use of memory compensation
strategies. The possible compensation strategies were based upon those found in Cognitive
Rehabilitation Therapy: Late Stages of Recovery (Ylvisaker, Szekeres, Henry, Sullivan, &
Wheeler, 1987).
WCST and the CVLT
From the WCST, the measure of perseverative responses was utilized as it has been found
to be the most useful for detecting the presence of brain damage (Heaton, 1981) and to be

Ecological Validity of the CVLT and the WCST

527

TABLE 1
Descriptive Data on Independent Variables Analyzed
Variable
Physical Disability
HSCL-58
Memory Strategies
WCST PR
CVLT Total Recall 1–5
CVLT Short-Delay-Trial 5
CVLT Discriminability

M

SD

Minimum

Maximum

1.64
1.46
19.48
23.71
41.96
Ϫ33.11
84.75

.83
.42
10.08
21.09
13.04
33.67
16.39

1.00
1.00
4.00
4.00
20.00
Ϫ100.00
41.00

3.00
2.48
40.00
75.00
65.00
20.00
100.00

CVLT ϭ California Verbal Learning Test; HSCL ϭ The Hopkins Symptom Check List; M ϭ mean; SD ϭ standard
deviation; WCST PR ϭ Wisconsin Card Sorting Test perseverative responses.

ecologically valid for predicting ADL performance (Little et al., 1996). From the CVLT,
immediate, short-delay and long-delay recall measures were selected in order to determine

if there are differences in ecological validity between these three types of memory. Crosson
(1988) found all three to successfully discriminate between individuals with head injury and
controls. For immediate memory, total number of words recalled on List A Trials 1–5 was
used. Delayed recall measures are heavily influenced by immediate recall performance. In
order to assess the decrement in performance from immediate recall to short-delay recall, a
formula from the CVLT computerized scoring package was utilized: Short-Delay Free Recall
minus Trial 5. Discriminability (the ability to discriminate previously learned words from
distracter words after a 20-minute interval) was chosen to represent long-term memory, as it
is more independent of immediate recall performance than long-term delayed recall (Crosson,
1988). See Table 1 for descriptive information on the independent variables.
Job Performance and Occupational Status
In terms of job performance, Lam, Priddy, and Johnson (1991) reported a need to expand
traditional measures of return to work (i.e., those that state whether or not the individual is
employed) to include any difficulties a patient may be experiencing when employed. For this
purpose, a questionnaire assessing job performance was developed, as such a scale was not
found in the literature. The 9-point job performance scale is illustrated in the Appendix. As
this is a newly developed scale, return to work was also ranked on the type of scale used
more traditionally by the literature (2 ϭ employed full-time, 1 ϭ employed part-time, and
0 ϭ unemployed). Occupational status was ranked according to the Hollingshead’s (1975)
Four Factor Index of Social Status. Those who are unemployed were given a score of ‘‘0.’’
Work-related measures were given only to those participants who were working prior to
injury (N ϭ 19) in an attempt to reduce the contribution of premorbid factors that may have
led to employment problems before the injury. Hence, all analyses were conducted on the
19 individuals who were working at the time of the accident. See Table 2 for descriptive
information on the dependent measures.
In order to lessen bias introduced by self-report, spouse’s or parent’s ratings of the participant’s job performance were also obtained and compared to those of the participant. Research
has shown that patients may overstate their functional abilities when compared to objective
assessment by nursing personnel (Rubenstein, Schairer, Weiland, & Kane, 1984).
Data Analysis
Pearson correlations between significant others’ and participants’ ratings on job performance were conducted. Hierarchical Linear Regression was performed to investigate the

528

M. Y. Kibby et al.

TABLE 2
Descriptive Data on Dependent Measures
Variable
Job performance a
Hours currently working b
Hollingshead’s rating c

M

SD

Minimum

Maximum

4.16
.74
2.00

3.22
.81
2.40

1.00

.00
.00

9.00
2.00
7.00

M ϭ mean; SD ϭ standard deviation.
a
Authors’ 9-point scale developed to assess quality of performance on the job.
b
Traditional scale of return to work: 0 ϭ unemployed, 1 ϭ employed part-time, 2 ϭ employed full-time.
c
Type of position held as assessed by the Hollingshead Occupational Scale.

relationship between the WCST and CVLT and measures of job performance and occupational status. In order to statistically control for physical disability, emotional distress and
memory strategies utilized, these variables were entered into the equation in Step 1 (Pedhazur,
1982). The WCST was entered in Step 2 and the CVLT measures in Step 3. The CVLT was
entered into the regression equations last, as this position is the most difficult in which to
explain additional variance, and, hence, it provides the CVLT with the most conservative test.
This method was chosen as it was hypothesized that the CVLT would be a better predictor of
job performance than the WCST.
RESULTS
Significant Other Ratings
Significant others’ ratings correlated significantly with participants’ ratings on job performance (r ϭ .9807, p Ͻ .000). Therefore, analyses were performed only on participants’
ratings.
Job Performance Versus Return To Work
The correlation between the authors’ measure of performance on the job and the traditional
measure of return to work was high (r ϭ .8405, p Ͻ .000). However, with the correlation
being less than 1.0, there are enough differences between the two measures to justify the

need for a measure of quality of performance on the job. The scale measuring job performance
incorporates the traditional measure of return to work (full- versus part-time versus unemployed) and goes beyond it to measure quality of performance on the job (refer to Appendix).
The high correlation between the authors’ scale and the traditional scale demonstrates the
new scale’s concurrent validity. External validity was demonstrated by utilizing the scale on
a sample of normal adults (N ϭ 35). Of the adults, 68.4% achieved a maximum score of 9.
Of the 31.6% who did not achieve a 9, some were using additional memory compensation
techniques or were having emotional distress on the job. Average score for the normal participants on this measure was 7.95 (SD ϭ 1.84), while the mean for the participants with CHI
was 4.16 (SD ϭ 3.22).
Return to Work
When the traditional scale of return to work was used as the dependent measure, none of
the analyses using the WCST and the CVLT were significant. The only bivariate correlation


Ecological Validity of the CVLT and the WCST

529

TABLE 3
Results of Hierarchical Regression Analysis in Predicting Job Performance When CVLT Total Recall
Trials 1–5 is Entered Last Into the Equation

Variable
Step 1
Physical disability
Emotional distress
Memory strategies
Step 2
WCST PR
Step 3
CVLT total recall 1–5

Explained Variance
(%)

B

β

Ϫ.63
Ϫ2.95
.06

Ϫ.16
Ϫ.42
.19

1.95
11.69
2.03

.04

.29

4.24

.20

.78

34.21*

CVLT ϭ California Verbal Learning Test; WCST PR ϭ Wisconsin Card Sorting Test perseverative responses.
Note. Explained variance determined by semipartial correlation.
F ϭ 4.68 (5, 12); R2 ϭ .66; p ϭ .01.
*p Ͻ .01.

between the independent variables and the traditional measure to reach significance was the
measure of physical disability (r ϭ Ϫ.5686, p ϭ .01).
Job Performance
When the job performance scale was used as the dependent measure, several important
findings emerged. Two measures of the CVLT, total number of words recalled on learning
trials 1–5 (immediate recall) and Discriminability (long-term recognition memory), were
significantly correlated with the authors’ job performance scale. The measure of short-delay
free-recall failed to reach significance. Immediate recall accounted for 34% of the variance
in job performance, while recognition memory accounted for 32% of the variance. WCST
perseverative responses did not significantly correlate with level of job performance in either
of the analyses. Of the variables entered in Step 1, emotional distress approached significance
when in the equation with CVLT immediate recall ( p ϭ .07), and physical disability was
significant when in the equation with CVLT Discriminability. See Tables 3 and 4 for results
of analyses investigating the relationship between the WCST and CVLT and job performance.
TABLE 4
Results of Hierarchical Regression Analysis in Predicting Job Performance When CVLT
Discriminability is Entered Last Into the Equation

Variable
Step 1
Physical disability
Emotional distress
Memory strategies

Step 2
WCST PR
Step 3
CVLT discriminability

Explained Variance
(%)

B

β

Ϫ1.73
Ϫ2.31
.08

Ϫ.45
Ϫ.33
.26

.06

.41

7.11

.16

.75

32.44**

16.36*
7.25
3.67

CVLT ϭ California Verbal Learning Test; WCST PR ϭ Wisconsin Card Sorting Test perseverative responses.
Note. Explained variance determined by semipartial correlation.
F ϭ 4.33 (5, 12); R 2 ϭ .64; p ϭ .02.
*p Ͻ .05; **p Ͻ .01.


530

M. Y. Kibby et al.

TABLE 5
Results of Hierarchical Regression Analysis in Predicting Occupational Status When CVLT Total Recall
Trials 1–5 is Entered Last Into the Equation

Variable
Step 1
Physical disability
Emotional distress
Memory strategies
Step 2
WCST PR
Step 3
CVLT total recall 1–5

Explained Variance
(%)

B

β

Ϫ1.42
Ϫ1.34
.03

Ϫ.46
Ϫ.25
.13

15.30
4.43
1.09

.10

.52

16.04

.11

.54

17.06*

CVLT ϭ California Verbal Learning Test; WCST PR ϭ Wisconsin Card Sorting Test perseverative responses.
Note. Explained variance determined by semipartial correlation.
F ϭ 3.40 (5, 12); R 2 ϭ .59; p ϭ .04.
*p Ͻ .05.

Occupational Status
As forecasted, the WCST perseverative responses predicted occupational status, as measured by the occupational scale from the Hollingshead Four Factor Index of Social Status
(Hollingshead, 1975). Correlations between occupational status and WCST perseverative responses approached significance ( p ϭ .052) when in an equation with the CVLT immediate
recall measure and reached significance when in an equation with CVLT Discriminability.
Immediate recall was significant for predicting occupational status, while recognition memory
approached significance ( p ϭ .06). Of the variables entered in Step 1, physical disability
approached significance when in an equation with CVLT immediate recall ( p ϭ .06) and
was significant in an equation with CVLT Discriminabilty. See Tables 5 and 6 for results
of analyses looking at the relationship between the WCST and CVLT and occupational status.
DISCUSSION
Neuropsychologists often need to make decisions about a patient’s capacities to function
in the everyday world. They are faced with questions such as, ‘‘is the patient ready to return
TABLE 6
Results of Hierarchical Regression Analysis in Predicting Occupational Status When CVLT
Discriminability is Entered Last Into the Equation

Variable
Step 1
Physical disability
Emotional distress
Memory strategies
Step 2
WCST PR
Step 3

CVLT Discriminability

B

β

Explained Variance
(%)

Ϫ1.73
Ϫ1.13
.06

Ϫ.56
Ϫ.22
.26

25.09*
3.19
3.78

.08

.67

18.77*

.10

.57

15.06

CVLT ϭ California Verbal Learning Test; WCST PR ϭ Wisconsin Card Sorting Test perseverative responses.
Note. Explained variance determined by semipartial correlation.
F ϭ 3.14 (5, 12); R2 ϭ .57; p ϭ .05.
*p Ͻ .05.


Ecological Validity of the CVLT and the WCST

531

to work or live independently?’’ Many of these neuropsychologists work in outpatient settings where they typically see clients with chronic disorders for the first time months after
onset of the disorder. Investigating the ecological validity of cognitive neuropsychological
tests in a postacute, severely head-injured population has yielded some interesting and surprising results.
Measures of immediate recall and long-term recognition memory from the CVLT were
significant predictors of current level of job performance as measured by the authors’ scale,
which combines employment status (full- versus part-time versus unemployed) with quality
of performance on the job. Immediate recall also correlated significantly with occupational
status (i.e., unskilled laborer, professional), and long-term recognition memory approached
significance. The measure of short-term delayed recall failed to reach significance in any of
the equations. The ability of immediate recall and recognition memory to predict both job
performance and occupational status lends further support to the ecological validity of memory tests in predicting current work functioning.
In terms of executive functioning, perseverative responses from the WCST were not predictive of job performance. In contrast, perseverative thinking was predictive of occupational
status. This finding concurs with that of Dikmen and Morgan (1980) who found measures
of executive functioning to be predictive of employment status only as occupational status
increased. The hypothesis stating that measures of memory functioning would predict job
performance better than the WCST was supported.
Equations predicting occupational status yielded some interesting findings. In these equations, the WCST was not significant when in an equation with immediate recall, but was

significant when in an equation with long-term recognition memory (Discriminability). Discriminability was not significant in predicting occupational status but tended toward significance. It appears that both memory and perseverative responses are predictive of occupational
status and are competing for shared variance. This conclusion is supported by the fact that
WCST perseverative responses are correlated with CVLT Discriminability (r ϭ Ϫ0.58; p Ͻ
.001) and with CVLT immediate recall (r ϭ -0.55; p Ͻ .001). Research has shown that those
who do not use internal strategies to categorize material at encoding perform worse at retrieval
(Crosson, 1988; Kibby, Schmitter-Edgecombe, & Long, 1994). This might reflect the fact
that a certain level of executive functioning is necessary in order to most effectively organize
and encode new material. Indeed, a relationship between measures of retrieval and executive
functioning has been demonstrated by Kibby, Schmitter-Edgecombe and Long (1995). Perhaps the overlap between memory and executive functioning helps to explain why the WCST
and recognition memory are competing for the same variance when predicting occupational
status.
In terms of variables entered into the first step of the hierarchical linear regression analyses,
only physical disability was significant. Neither emotional distress nor memory compensation
techniques utilized were significant predictors of job performance or of occupational status.
Emotional distress approached significance for predicting job performance when entered into
an equation with immediate recall. Stambrook et al. (1990) and Newnan et al. (1978) found
psychosocial variables to be predictive of number of hours working. With a larger number
of participants, emotional distress may have been significant for predicting performance on
the job, which includes the number of hours currently working. Physical disability was significant in predicting both job performance and occupational status. This is also consistent
with findings of Stambrook et al. (1990). Having injuries to the body in addition to the brain
can make it extremely difficult to return to work, particularly if one holds a position that
entails manual labor or high levels of stamina.
When discussing predicting return to work, Lam et al. (1991) stated a need for more
comprehensive measures of work functioning. Measures of work functioning should go be-


532

M. Y. Kibby et al.

yond stating whether a person can return to work to include how the individual is performing
on the job. The job performance scale included in this study was an attempt at meeting this
need. The authors’ measure of job performance and the traditional measure of return to work
(i.e., full-time employment, part-time employment or unemployed) are highly correlated, yet
in this study the cognitive measures were not predictive of return to work alone. Perhaps the
cognitive measures utilized in this study are better predictors of how well a person can perform on the job than solely of if, and how long, a person can work each day. Physical
disability, on the other hand, was a significant predictor of return to work. This is intuitively
logical, as having a physical disability can reduce one’s stamina, making it difficult to work
full-time.
The nonsignificant findings for the WCST and for emotional distress in predicting performance on the job need to be interpreted with caution. A small sample size was utilized.
Occupational status was skewed toward the lower end of the distribution, as there were no
individuals from the first category of Hollingshead scale (e.g., physicians, psychologists).
The sample studied tended to be of low average IQ; however, this measure probably reflects
the affects of the injury and not premorbid status as the average educational level was 13.89
years. Finally, the sample was composed of only severely head-injured participants 1 year
or more postinjury, which may reduce the findings’ generalizability to acutely injured populations or other disorders.
Further research is needed on this topic utilizing a larger sample size or different populations of participants. In addition, the study should be replicated by testing participants at
various times since injury so that the ability of cognitive measures to predict later work
performance can be investigated. Furthermore, more work needs to be done utilizing scales
or tests that measure qualitative aspects of job performance. Often persons with head injury
are able to return to work part- or full-time, but they have difficulty performing at the same
level of competency as they had before the injury.
Knowledge of how well neuropsychological tests predict a patient’s competency on the
job will aid neuropsychologists in determining whether a patient is ready to return to work
and if any accommodations are going to be necessary to help the patient succeed. This knowledge could be gained through collaborating with vocational rehabilitation. Those in vocational rehabilitation have the skills necessary to develop measures that evaluate how well a
patient is performing on the job. Neuropsychologists could compare how well our test data
predict patients’ performance in vocational rehabilitation and patients’ performance on the
job once placed. By doing this, neuropsychologists will have a heightened awareness of
which of our tests are correlated with performance on the job. This type of collaboration is
essential if ecological research is to expand into predicting job performance.

Acknowledgments: Portions of this data were presented at the Thirteenth Annual Meeting of the National Academy of Neuropsychology, 1994, Phoenix. We gratefully acknowledge the assistance of Amanda Neblett and Betsy
Shaver for their help in collecting and scoring the data.

REFERENCES
Bennett, T. (1988). Use of Halstead-Reitan Neuropsychological Test Battery in the assessment of head injury. Cognitive Rehabilitation, May/June, 18–24.
Cattell, R. B., & Cattell, A. K. S. (1973). Handbook for the individual or group Culture Fair Intelligence Test:
Scale 2. Champaign, IL: IPAT.
Crosson, B. (1988). California Verbal Learning Test (CVLT) performance in severely head-injured and neuropsychologically normal adult males. Journal of Clinical and Experimental Neuropsychology, 10, 754–768.


Ecological Validity of the CVLT and the WCST

533

Delis, D. C., Kramer, J., Kaplan, E., Ober, B. A., & Fridlund, A. (1987). The California Verbal Learning Test.
New York: Psychological Corporation.
Derogatis, L. R., Lipman, R. S., Rickels, K., Uhlenhuth, E. H. & Covi, L. (1974). The Hopkins Symptoms Checklist
(HSCL): A self-report symptom inventory. Behavioral Science, 19, 1–15.
Dikmen, S., & Morgan, S. (1980). Neuropsychological factors related to employability and occupational status in
persons with epilepsy. Journal of Nervous and Mental Disease, 168, 236–240.
Elwood, R. W. (1995). The California Verbal Learning Test: Psychometric characteristics and clinical application.
Neuropsychology Review, 5, 173–201.
Fraser, R., Dikmen, S., McLean, A., Miller, B., & Temkin, N. (1988). Employability of head injury survivors: First
year post-injury. Rehabilitation Counseling Bulletin, 31, 276–288.
Goldstein, G., McCue, M., Rogers, J., & Nussbaum, P. D. (1992). Diagnostic differences in memory test based
predictions of functional capacity in the elderly. Neuropsychological Rehabilitation, 2, 307–317.
Heaton, R. K. (1981). A manual for the Wisconsin Card Sorting Test. Odessa, FL: Psychological Assessment Resources.
Heaton, R. K., Chelune, G. J., & Lehman, R. A. W. (1978). Using neuropsychological tests to assess the likelihood
of patient employment. Journal of Nervous and Mental Disease, 166, 408–416.
Heinrichs, W. (1990). Current and emergent applications of neuropsychological assessment: Problems of validity

and utility. Professional Psychology: Research and Practice, 21, 171–176.
Hollingshead, A. B. (1975). Four Factor Index of Social Status. Unpublished manuscript.
Kibby, M. Y., Schmitter-Edgecombe, M. E., & Long, C. J. (1994). The California Verbal Learning Test (CVLT)
as a measure of memory functioning in severely head-injured adults and neurologically normal controls. Poster
session presented at the fourteenth annual meeting of the National Academy of Neuropsychology, Orlando, FL.
Kibby, M. Y., Schmitter-Edgecombe, M. E., & Long, C. J. (1995). Relationships between measures of Auditory
Verbal Learning and Executive Functioning in closed head injury. Poster session presented at the fifteenth annual
meeting of the National Academy of Neuropsychology, San Francisco, CA.
Lam, C. S., Priddy, D. A., & Johnson, P. (1991). Neuropsychological indicators of employability following traumatic
brain injury. Rehabilitation Counseling Bulletin, 35, 68–75.
Little, A. J., Templer, D. I., Persel, C. S., & Ashley, M. J. (1996). Feasibility of the neuropsychological spectrum
in prediction of outcome following head injury. Journal of Clinical Psychology, 52, 455–460.
Long, C. J., & Kibby, M. Y. (1995). Ecological validity of neuropsychological tests: A look at neuropsychology’s
past and the impact that ecological issues may have on its future. Advances in Medical Psychotherapy, 8, 59–
78.
Lysaker, P., Bell, M., & Beam-Goulet, J. (1995). Wisconsin Card Sorting Test and work performance in schizophrenia. Psychiatry Research, 56, 45–51.
Newnan, O. S., Heaton, R. K., & Lehman, R. A. W. (1978). Neuropsychological and MMPI correlates of patients’
future employment characteristics. Perceptual and Motor Skills, 46, 625–642.
Pedhazur, E. J. (1982). Multiple regression in behavioral research: Explanation and prediction. Fort Worth, TX:
Harcourt Brace Jovanovich College Publishers.
Puente, A. E. (1992). The status of clinical neuropsychology. Archives of Clinical Neuropsychology, 7, 297–312.
Randolph, C., Gold, J. M., Kozora, E., Cullum, C. M., Hermann, B. P., & Wyler, A. R. (1994). Estimating memory
function: Disparity of Wechsler Memory Scale-revised and California Verbal Learning Test indices in clinical
and normal samples. Clinical Neuropsychologist, 8, 99–108.
Richardson, E. D., Nadler, J. D., & Malloy, P. F. (1995). Neuropsychologic prediction of performance measures
of daily living skills in geriatric patients. Neuropsychology, 9, 565–572.
Rubenstein, L. Z., Schairer, C., Weiland, G. D., & Kane, R. (1984). Systematic biases in functional status assessment
of elderly adults: Effects of different data sources. Journal of Gerontology, 39, 686–691.
Ryan, T. V., Sautter, S. W., Capps, C. F., Meneese, & Barth, J. T. (1992). Utilizing neuropsychological measures
to predict vocational outcome in a head trauma population. Brain Injury, 6, 175–182.

Sbordone, R. J., & Long, C. J. (Eds.). (1995). The ecological validity of neuropsychological testing. Delray Beach,
FL: St. Lucie Press.
Schmitter-Edgecombe, M. (1996). Effects of divided attention on implicit and explicit memory performance following severe closed head injury. Neuropsychology, 10, 155–167.
Stambrook, M. Moore, A. D., Peter, L. C., Deviaene, C., & Hawryluk, G. A. (1990). Effects of mild, moderate
and severe closed head injury on long-term vocational status. Brain Injury, 4, 183–190.
Weddell, R., Oddy, M., & Jenkins, D. (1980). Social adjustment after rehabilitation: A two year follow-up of patients
with severe head injury. Psychological Medicine, 10, 257–263.
Ylvisaker, M., Szekeres, S. F., Henry, K., Sullivan, D. M., & Wheeler, P. (1987). Topics in cognitive rehabilitation
therapy. In M. Ylvisaker & E. M. R. Gobble (eds.), Community re-entry for head injured adults (pp. 137–220).
Boston: College-Hill Press.


534

M. Y. Kibby et al.

APPENDIX

Job Performance Scale
Say to the participant ‘‘tell me the one number that best describes what you are doing right
now.’’ If the individual is also going to school so he or she is not working full time, please
note this on the bottom of the page.
1) not working because
2) working in a highly structured environment (sheltered workshop).
3) working under close supervision or in a supported work environment. Tasks are given
with detailed instructions, and the participant is checked regularly to see how he/she
is performing on the task.
4) working without much supervision beyond what non-head injury survivors need, but
with decreased quality of work. The participant is having difficulty initiating activities,
completing tasks or relating to others.

5) working without much supervision beyond what non-head injury survivors need with
fairly good quality with the aid of compensation strategies (detailed lists of things to
do or how to do them, memory aids). OR working independently with a reduced load
(some work is given to other people to do) or with modified job tasks (tasks are made
simpler for the individual).
6) working at a comparable level to non-head injury survivors, but at a lower level job
than before the injury.
7) working at a comparable level to non-head injury survivors at the same job level as
before the injury, but with reduced hours compared to before the injury (not due to
school).
8) working at the same type of job as before the injury at the same number of hours and
at a comparable level to non-head injury survivors; however, the participant has fatigue,
headaches, depression, memory or concentration problems. He/She is not using extra
compensation strategies or in need of special assistance, but is having difficulty with
the job.
9) working at the same type of job as before at the same number of hours and with the
same level of ability as before the injury, but without the difficulty and the complaints
mentioned above in #8.