Effects of check incheckout on behavioral indices and mathematics generalization

Effects of Check-in/checkout on Behavioral Indices and
Mathematics Generalization
Michael D. Mong
University of Southern Mississippi
Kristin N. Johnson
Eastern Illinois University
Kristi W. Mong
Mong Psychological Associates
ABSTRACT: Check-in/checkout (CICO) is a behavioral intervention that is used to provide
systematic feedback about a student's behavior at the beginning and end of each school day. The
purpose of the present study was to evaluate the effectiveness of CICO on decreasing problem
behaviors and the collateral effects on mathematics performance for 4 at-risk elementary school
students. A multiple-baseline across-participants design using dyads was used to analyze problem
behavior collected through direct classroom observations. As ancillary measures, office discipline
referrals per week as well as mathematic performance (i.e., digits correct per minute [DCPMj) were
collected for each student. Treatment integrity and acceptability and social validity were also
measured. Results indicate a decrease in problem behaviors as well as an increase in DCPM for each
participant.
• When children receive office discipline
referrals (ODRs), they can often simultaneously exhibit a multitude of issues including academic and behavioral problems.
These problems rarely exist in isolation, and
in combination they put students in more
dramatic risk of school failure (Mclntosh,
Flannery, Sugai, Braun, & Cochrane, 2008).
Thus, the relationship between academic
performance and problem behaviors provides concern because of their documented
interaction (Maguin & Loeber, 1995; Roeser
& Eccles, 2000). Students with early behavior
difficulties are at greater risk for developing
academic problems (Fleming, Harachi,
Cortes, Abbott, & Catalano, 2004), and

students with early academic difficulties are
at greater risk for developing problems in
social behavior (Mclntosh, Horner, Chard,
Boland, & Good, 2006; Morrison, Anthony,
Storino, & Dillon, 2001).
Given the documented relationship between academic and behavior problems,
positive behavior support (PBS) experts recommend a three-tiered model of behavior
supports to prevent and intervene with problem behavior (Sugai & Horner, 2001). As
students progress through the tiers, the intensity of the intervention increases, as does the
Behavioral Disorders, 36 (4), 225-240

cost of resources. The purpose of the first tier is
to provide primary prevention for all students.
Those students whose behaviors continue to
be discrepant from their peers are identified for
additional support at Tier 2.
Approximately 15% to 20% of the population will benefit from this level of support
(Walker & Shinn, 2002). Tier 2 interventions
offer at-risk students additional opportunities
to learn expected behaviors that lead to
educational success (Lee, Sugai, & Horner,
1999). These services are provided in addition
to the core instruction (National Association of
State Directors of Special Education, 2005).
Tier 3 interventions are for those students who
are highly discrepant from their peers either in
behavioral excesses or deficits. In addition,
these students referred to Tier 3 may be
nonresponsive to Tier 2 interventions.
For behavior. Filter and colleagues (2007)

recommended interventions be implemented
with students who have two or more ODRs but
whose problem behaviors do not pose an
immediate danger to self or others. These
interventions should be de!signed to be quickly
accessed, flexible, and to bring about improvement (Hawken & Horner, 2003). One intervention that may meet these requirements is
the check-in/checkout (CICO) program.
August 2011 / 225


The CICO program was developed as an
efficient intervention for reducing problem
behavior. The CICO program was designed
to increase feedback and positive adult attention (Crone, Horner, & Hawken, 2004).
Previous research has shown that CICO is
an easy-to-implement behavioral strategy and
is effective at decreasing ODRs (Hawken &
Horner, 2003; March & Horner, 2002), decreasing observed problem behaviors (Fairbanks, Sugai, Guardino, & Lathrop, 2007), and
increasing appropriate behavior (Todd, Campbell, Meyer, & Horner, 2008). The CICO
program has been shown to be a relatively
simple and inexpensive intervention (Hawken,
MacLeod, & Rawlings, 2007). The CICO
program has also been shown to have relatively high levels of teacher and staff acceptability. Additionally, previous research (Todd
et al., 2008) has shown that typical school
personnel were able to effectively implement
the CICO intervention.
If implementing CICO demonstrates students engaging in fewer problem behaviors
and spending less time in the school office,
improved academic achievement may likely
follow (Hawken et al., 2007). Indeed, Hawken

and Horner (2003) documented increases in
academic engagement following CICO implementation by assessing if the student was (a)
looking at the teacher while the teacher was
giving instructions/directions, (b) working with
a peer when instructed to do so, (c) reading
silently or completing a writing assignment, (d)
participating in a teacher-approved activity if
work was completed, or (e) talking about
academic material with the teacher or aide
for at least 7 s. Although the primary focus of
the study was on classroom problem behaviors, a secondary analysis indicated that the
CICO intervention was associated with increases in mean level of academic performance in mathematics for all 4 students.
While the previous research is indeed
promising, few studies have examined CICO's
effect on outcome variables such as academic
performance and achievement (Hawken et al.,
2007). The main idea is that although CICO
may or may not have direct effects on academic
achievement, students who spend less time
engaging in inappropriate behaviors may replace those behaviors with appropriate behavior. Time on task or engagement leads to better
academic performance. Furthermore, few studies have provided measures of CICO effect
size. Reporting effect sizes is considered best
226 / August 2011

practice when presenting empirical research
findings in many fields (Wilkinson, 1999).
Another limitation of previous CICO research
involves the manner in which social validity
was assessed. To date, few studies have
measured the social validity of CICO using a

measure with validated psychometric properties.
Thus, the purpose of the present study was to
(a) extend the literature by examining CICO's
effect on classroom problem behaviors as
measured by direct behavioral observations
and ODRs, (b) determine whether the effects of
a behavioral intervention (CICO) affected student mathematics performance, and (c) assess
the social validity of CICO using a measure with
validated psychometric properties.

Method
Participants and Setting
The study was conducted in a suburban
elementary school located in the southeast
United States with approximately 415 students
(Grades 3-5), 65% of whom qualified for free
or reduced lunch and 36% of whom were from
ethnic minority backgrounds. The school had
been implementing schoolwide PBS for more
than 2 years. The Schoolwide Evaluation Tool
(Horner, Sugai, Todd, & Lewis-Palmer, 2005)
results indicated that the school was implementing its schoolwide behavior support plan
with 82% treatment integrity.
Students were selected for participation in
the study if (a) they received five ODRs in a
single month, (b) their problem behavior
occurred frequently across multiple settings
throughout the school day as noted from ODR
analysis and teacher interviews, and (c) the
results of a functional behavioral assessment

(FBA) indicated that the function of the
students' problem behavior was hypothesized
to be attention seeking. The problem behaviors
exhibited by these students are particularly
alarming given that the long-term outcomes for
students who exhibit early patterns of maladaptive behavior are continued poor academic performance, referral for special education identification and placement, social
rejection, low self-esteem, potential for developing a more chronic psychological disorder,
and increased school dropout rates (Walker,
Ramsey, & Gresham, 2004).
Informed consent to participate was obtained for each student from his or her parent/
Behavioral Disorders, 36 (4), 225-240


legal guardian. Of the 4 students selected for
intervention, 2 were boys and 2 were girls.
Students ranged in age from 8.4 to 9.1 years,
with a mean age of 8.7 years. All students were
enrolled in the third grade. In terms of race,
there were 2 Caucasian students and 2 African
American students.
Lauren. Lauren was an 8-year-old Caucasian girl who has never received special
education services, has no previous diagnoses,
and has no history of retention. The majority
(87%) of Lauren's ODRs resulted from inappropriate behavior in the classroom. She was
referred by her classroom teacher for talking
out during classroom instruction, which accounted for 18% of her total ODRs; noncompliance with teacher demands (40%); disrupting peers during independent seat work (29%);
and off-task behavior across settings (13%)
including the classroom and cafeteria. Based
on the results of the FBA, Lauren's problem
behaviors were hypothesized to be maintained

primarily by adult attention with peer attention
as a secondary function. During the immediate
3 months before the study, Lauren averaged
2.1 (range, 1-3) ODRs per week.
Andrew. Andrew was an 8-year-old African
American boy who has never received special
education services, has no previous diagnoses,
and has no history of retentions. Andrew was
receiving Title I services for reading and math.
The majority of Andrew's ODRs (93%) resulted
from inappropriate behavior in the regular
classroom or music classroom during instruction. He was referred by his classroom teacher or
music teacher for noncompliance with teacher
demands (37%), oft'-task behavior (22%), and
talking out (41%) in the general education class
as well as music and physical education classes.
Based on the results of the FBA, Andrew's
problem behaviors were hypothesized to be
maintained by adult attention. During the
immediate 3 months before the study, Andrew
averaged 4.1 (range, 2-5) ODRs per week.
Pam. Pam was a 9-year-old African
American girl who has never received special
education services, has no previous diagnoses,
and has no history of retentions. The majority
of Pam's ODRs (92%) occurred in the classroom during instruction and independent seat
work. She was referred by her classroom
teacher for refusing to complete assignments
(14%), noncompliance with teacher demands
(54%), talking out in class (24%), and noncompliance with bus driver demands (8%).

Based on the results of the FBA, Pam's problem
Behavioral Disorders, 36 (4), 225-240

behaviors were hypothesized to be maintained
primarily by adult attention with peer attention
as a secondary function. During the immediate
3 months before the study, Pam averaged 1.9
(range, 1-3) ODRs per week.
Stanley. Stanley was an 8-year-old Caucasian boy who has never received special
education services, has no previous diagnoses,
and has no history of retentions. The majority
of Stanley's ODRs (92%) occurred in the
classroom during independent seat work. He
was referred by his classroom teacher for offtask behavior (34%), noncompliance with
teacher demands (17%), and talking out in
class (41%) as well as off-task behavior in
physical education (4%) and in schoolwide
specials (4%). Based on the results of his FBA,
Stanley's problem behaviors were hypothesized to be maintained primarily by adult
attention with peer attention as a secondary
function. During the immediate 3 months
before the study, Stanley averaged 2.0 (range,
1-3) ODRs per week.
In addition, two guidance counselors and
the students' classroom teachers agreed to
participate in the study, with the counselors
identified as the CICO specialists. Prior to the
initiation of each phase, the primary investigator trained the teachers and CICO specialists on
the required procedures associated with each
phase of the study. The training consisted of the

primary investigator describing the procedures,
modeling the procedures, and having the staff
member practice the procedures with the
primary investigator providing feedback. This
format was implemented until the individuals
were able to implement the procedures independently. The primary investigator was available for any questions or concerns from the
teachers and CICO specialists as they arose.
FBA Procedures
Prior to initiation of the study, an FBA was
conducted for each student. The assessment
process involved a 20- to 40-min interview
conducted by the primary author with each
participant's teachers using the Functional Analysis Informant Record for Teachers (FAIR-T;
Edwards, 2002). The purpose of the EAIR-T
was to determine and help define the problem
behaviors, to determine appropriate replacement behaviors, to determine potential reinforcers for appropriate behavior, and also to help
formulate the hypothesized function of the
presenting problem behavior. To date, studies
August 2011 / 227


have supported the use of the FAIR-T in such a
manner (Doggett, Edwards, Moore, Tingstrom, &
Wilczynski 2001; Dufrene, Doggett, Henington,
& Watson, 2007; Edwards, 2002). Following the
FAIR-T, three 20-min conditional probability
observations were conducted in the student's
natural classroom setting prior to intervention. A
FAIR-T hypothesis statement was judged to be
supported if the direct observation data provided

similar information to the antecedent and
consequent events defined in the FAIR-T hypothesis statement. As CICO was designed to
increase feedback and positive adult attention
(Crone et al., 2004), it was important to match
the behavioral intervention to the function of the
participant's problem behaviors.

Measurement
Direct observation of problem behavior.
Problem behavior was observed 3 days per
week using a 20-min, 10-s partial interval
recording system. For each participant, observations took place during the same academic
class period each day. The specific class period
for each student was determined by teachers'
reports of the most problematic time of day
based on the FAIR-T interview. Problem behaviors included (a) noncompliance, (b) talking
out, and (c) off-task behavior. Noncompliance
was defined as failure to complete assigned
instructions or failure to initiate commands
within 5 s. Talking out was defined as the
student engaging in vocalizations that were not
preceded by a raised hand and/or were not
initiated by an adult. Off-task behavior was
defined as the student oriented away (e.g., face
or body) from the teacher or materials during
instruction or oriented toward irrelevant persons or objects, manipulating materials or
objects inappropriately that are relevant to the
assigned task or activity, or doing other
behaviors that are not related to his or her
assignment or task for a period of 10 s or more.

For all students, the observations occurred in
mathematics as it was considered by the
teachers as one of the most problematic times
of the day. Direct observation of problem
behavior was calculated by dividing the number of intervals with observed problem behavior
by the total number of intervals and multiplying
this ratio by 100 to obtain a percentage.
Office discipline referrals. The ODRs were
collected to compare the rates of problem
behaviors before and during the CICO
program and reported as a weekly measure.
228 / August 2011

As a measure of behavior, ODRs possess
sufficient construct validity and adequate
concurrent validity with a number of standardized measures of individual behavior (as cited
in Irvin, Tobin, Sprague, Sugai, & Vincent,
2004) as well as predictive validity for negative
school outcomes, including physical assaults
and dropout (Tobin & Sugai, 1999). Office
discipline referrals have also been identified as
an effective and efficient measure for decision
making in schools (Irvin et al., 2006).
Percentage of daily progress reports points
earned. Each student's percentage of daily
progress report (DPR) points earned was
examined as a measure of appropriate behavior. Evaluations of the DPR suggest that it
possesses internal consistency, temporal stability, and concurrent validity and that its
sensitivity allows for detection of treatment
effects (Pelham, Fabiano, & Massetti, 2005).

Treatment Integrity
The CICO treatment integrity checklist.
Treatment integrity was measured in a similar
manner used by Hawken et al. (2007). For
each integrity assessment, the first author
examined the CICO daily student intervention
protocol to determine the degree with which
the CICO intervention was implemented as
prescribed. Treatment integrity was evaluated
during 33% of the sessions evenly distributed
across all phases of the study based on
completion of a checklist during the session.
Treatment integrity was calculated by the
number of items on the checklist completed
divided by the total number of items on the
checklist and multiplied by 100.
Academic Pretreatment Assessment
A Web-based computer program. Math
Worksheet Generator (interventioncentral.org),
was used to generate curriculum-based mathematics worksheets. The program allows the
user to design worksheets requiring the use of
specific skills. State benchmarks were used to
determine which skills were representative of
each grade level. The program was then used to
create a worksheet specific to a particular grade
level and state benchmark. The computer
program randomized the order of the (a)
problems within a worksheet and (b) factors
within each problem. Each worksheet listed

problems in six rowsof four problems in portrait
orientation on a regular 8 'A- by 11 -inch sheet of

Behavioral Disorders, 36 (4), 225-240


white paper. Each worksheet contained the 24
problems. Prior to beginning the study, the third
author used curriculum-based assessment
(CBA) procedures to identify each student's
instructional level. Multiple skill CBA probes
were administered to determine the current
grade-level performance of each of the students. The student was given one worksheet at
his or her current grade placement in school. For
example, a third-grade student was given an
opportunity to complete a third-grade-level
probe. Each student was given 60 s to complete
each worksheet. The digits correct per minute
(DCPM) on each worksheet was determined by
the number of digits written correctly during a 1 min Interval, divided by the number of seconds
worked and multiplied by 60. Errors per minute
(EPM) served as a secondary dependent variable.
Responses were scored as errors if incorrect
digits were written below the line or if digits
were written in the wrong place or omitted.
If performance was determined to be in the
instructional-level range, a worksheet at the
same grade level was administered. If the student
performed at the frustrational level (less than 14
DCPM), a worksheet at a lower grade level was

administered. These procedures were repeated
until a median instructional level performance
was obtained across three worksheets within the
same grade level to establish baseline.
According to Burns, VanDerHeyden, and
Jiban (2006), a student's independent instructional level was the point in the curriculum at
which he or she could complete math problems with 14 to 31 digits correct if enrolled in
first through third grade. The independent
instructional level was the point in the
curriculum at which a student could complete
math problems obtaining 24 to 49 digits if
enrolled in Grades 4 and higher.
During the 8 weeks of CICO intervention,
all students received math instruction once per
day and did not receive any math interventions. Furthermore, analysis of the classroom
teachers' weekly lesson plans indicated that
mathematics instruction focused on skills (i.e.,
estimating and rounding, analyzing graphs)
other than mathematics computation. Thus,
the instruction in the classroom did not
necessarily directly address the skills on the
mathematic computation probes.
Academic Progress Monitoring
To determine student progress, each student
completed three multiple skill grade-level
Behavioral Disorders, 36 (4), 225-240

probes at the predetermined instructional level
to determine the student's median DCPM.
Problems included addition with two-digit numbers with regrouping (i.e., 23 -i-18), addition with

two three-digit numbers with regrouping (i.e.,
156 -H 379), subtraction with two two-digit
numbers with regrouping (i.e., 48 — 19), subtraction with two three-digit numbers without
regrouping (i.e., 275 - 130), multiplication facts
to 10 (i.e., 3 X 5), and division facts to 10 (i.e., 8
-H 2). The multiple skill instructional-level probes
were administered once per week. The use of the
multiple skill probes allowed the evaluation of
level and rate of progress changes on instructional-level material.
Social Validity
The Behavior Intervention Rating Scale
(BIRS; Treuting & Elliott, 1991) is a 24-item
scale that employs a 6-point Likert-type format
(1 = strongly disagree to 6 = strongly agree) to
measure teachers' perceptions of treatment
acceptability and the perceived efficacy of
classroom interventions. The BIRS is composed of three factors (Acceptability, Effectiveness, and Time of Effect) and a Cronbach's
alpha of .97 for the total scale (Elliott &
Treuting, 1991). The Acceptability factor is
composed of 15 items with a Cronbach's alpha
of .97. The Effectiveness factor consists of
seven items with a Cronbach's alpha of .92.
The Time of Effect factor is composed of two
items with a Cronbach's alpha of .87 (Elliot,
1998). The BIRS was completed by both
guidance counselors and the participant's
classroom teacher at the end of each week
for a total of eight measurements per respondent. The mean scores for each factor across
the CICO intervention phase will be reported.
The five-item BEP Acceptability Questionnaire (Hawken & Horner, 2003) was used to

assess the social validity of the intervention
with the participants. Questions on the BEP
Acceptability Questionnaire assessed the extent to which the CICO was perceived to (a)
improve problem behavior at school, (b)
improve academic performance, (c) be worth
the time and effort, (d) be worth recommending to others, and (e) be easy to implement.
Scores on the BEP Acceptability Questionnaire
were recorded on a Likert-type scale (1-6),
with higher scores indicating a more favorable
impression of the CICO intervention. The
participants responded to the questionnaire
once per week for a total of eight measure-

August 2011 / 229


ments per respondent. The mean scores for
each question across the CICO intervention
phase will be reported.

Reliability of Measures
Interobserver agreement for problem behavior observation data. Interobserver agreement (lOA) for problem behavior data was
collected using a second independent observer
for 33% of the sessions evenly distributed across
all phases of the study. The lOA was calculated
by adding the number of intervals of agreement
of problem behavior for each session, then
dividing by the total number of observed
intervals for each session, and then multiplying
this ratio by 100 to obtain a percentage.

Interobserver agreement of CICO sessions.
As an additional measure of treatment integrity, the third author completed the CICO daily
intervention student protocol during direct
observation of 33% of all the CICO intervention sessions. Treatment integrity was calculated by the number of items on the checklist
completed divided by the total number of
items on the checklist and multiplied by 100.
Interscorer agreement. The primary author
designed a list of scoring instructions for the
mathematics probes used in the investigation.
One scorer and the primary investigator scored
a sample of 15 probes independently. The
rules were clarified and revised until there was
at least 90% agreement on a set of 45 sample
probes. Scorers were then cleared for scoring
of the probes. Approximately 33% of the total
probes were independently scored by the one
scorer and the primary investigator across all
phases of the study.
Design and Procedure
The experimental design for this study was
a combined series multiple baseline across
students design. The CICO intervention was
first applied to the student with the most stable
baseline in terms of problem behaviors. After
an intervention effect was demonstrated and
the subsequent students' baselines remained
stable, the intervention was applied to the
student with the next most stable baseline
(Carr, 2005). To demonstrate empirical control
and to avoid delays to intervention, students

were paired in dyads to create two multiplebaseline pairs for the current project. Two
phases were implemented for this study:
baseline and CICO.
230 / August 2011

Visual analyses for level, trend, and
variability were used to determine effects as
well as two statistical procedures for effect
sizes. Effect sizes were calculated using the
percentage of nonoverlapping data points all
(PND; Olive & Smith, 2005). PND is calculated by dividing the number of nonoverlapping
data points with baseline by the total number
of intervention data points. The lowest baseline data point was used to establish the
overlap of baseline data points with intervention data points for observed behavior and
ODRs, whereas the highest baseline data point
was used for the percentage of DPR points
earned. Benchmarks for PND scores have also
been established by Scruggs and Mastropieri
(2002). Specifically, PND scores below 50%
suggest an ineffective intervention effect,
scores between 50% and 70% suggest a
questionable intervention effect, scores between 70% and 90% suggest an effective
intervention effect, and scores greater than
90% suggest a very effective intervention
effect. Previous researchers (Campbell, 2004;
Olive & Smith, 2005) concluded that PND is
valid for documenting the effects of interventions in single-subject research.
Baseline
During baseline, typical schoolwide behavior support procedures were in place for all
students, including those participating in this

study. During baseline, direct observation of
problem behavior and assessment of academic
skills for target students was conducted. The
ODRs were monitored through collection of the
written ODR reports. In addition, the students'
teachers were given three practice DPRs the
week prior to implementation of CICO to ensure
that only the target behaviors were rated. These
practice DPRs served as a baseline measure of
the percentage of DPR points earned.
Check-in/Checkout
The CICO process involved the following
five elements: (a) Students were required to
check in with the guidance counselor of their
choice before school. The counselor provided
the student with a DPR form that was carried to
class for feedback throughout the day. When
students checked in, they were asked if they
had their DPR from the day before signed by
their parents and if they had their materials
ready for the school day. They received praise
Behavioral Disorders, 36 (4), 225-240


and a lottery ticket for a weekly drawing for
checking in. Also during check in, students
were prompted to identify daily goals and
given feedback to encourage success, (b) At
three specified times of the day, students
approached the teacher with the CICO report

card, and the teacher provided the student
with feedback about the student's behavior
by rating either 0 (did not meet expectations),
1 {somewhat met expectations), or 2 (met
expectations). Teachers also provided immediate verbal praise for students who met
behavioral expectations for that time period
and corrective feedback if students did not
meet the expectations, (c) At the end of the
school day, students took the DPR to the
counselor to check out. The percentage of
points earned for the day was calculated, and
students received verbal praise and rewards if
they met their daily point goal. Students could
choose from among specific rewards determined by a forced-choice preference assessment including stickers, pencils, time with a
preferred adult, or extra recess time the
following day. Similar to the procedures used
by Hawken et al. (2007), 80% of the total
points earned was the daily goal for all
students in this study. If students did not meet
their daily goal, the counselor provided
information on what to work on for the
following school day. (d) Students then took
their DPR home to be signed by a parent/
guardian, (e) The DPR was signed by a parent/
guardian and returned the following morning.

Results
Direct Observation of Problem Behavior
Figure I summarizes the results across
participants. During baseline, all 4 participants

displayed variable levels of problem behaviors, with an overall mean of 32.8% (range,
21-42.7). Upon introduction of the CICO
intervention phase, Lauren and Andrew displayed an immediate reduction in level and
trend, whereas Pam and Stanley displayed
gradual decreases in level and trend. The
participants were observed to engage in an
overall mean of 21.4% (range, 16.9-29.3)
across the CICO phase.
Lauren. At baseline, Lauren displayed problem behaviors in 31.3% (range, 28%-34%),
with an increasing trend and moderate variability. With regard to the CICO intervention phase,
an immediate decrease in level and trend (mean
Behavioral Disorders, 36 (4), 225-240

of 17.9%; range, 12%-25%) was observed as
compared with the baseline phase. The PND for
Lauren was 100%, suggesting that the CICO
intervention was effective at decreasing her
observed problem behaviors.
Andrew. On average, Andrew displayed
problem behaviors in 42.7% (range, 39%48%) with relatively no trend with moderate
variability across the baseline phase. With
regard to the CICO intervention phase, an
immediate decrease in level and trend (mean
of 21.6%; range, 19%-26%) was observed as
compared with the baseline phase. The PND
for Andrew was 100%, suggesting that the
CICO intervention was effective at decreasing
his observed problem behaviors.
Pam. At baseline, Pam displayed problem
behaviors in 2 1 % (range, 17%-24%) with

increasing trend and moderate variability.
With regard to the CICO intervention phase,
a gradual decrease in level and trend was
observed as compared with the baseline
phase. Pam averaged 16.9% (range, 10%25%) of problem behaviors across the phase.
The PND for Pam was 63%, suggesting that
the CICO intervention was somewhat effective at decreasing her observed problem
behaviors.
Stanley. At baseline, Stanley displayed
problem behaviors in 36% (range, 32%-40%)
with a gradually increasing trend and moderate variability. In the CICO intervention phase,
a gradual decrease in level and trend was
observed as compared with the baseline
phase. Stanley averaged 29.3% (range, 25%6%) of problem behaviors across the phase.
The PND for Stanley was 75%, suggesting that
the CICO intervention was effective at decreasing his observed problem behaviors.

Office Discipline Referrals
Figure 2 summarizes the results across
participants. During baseline, all 4 participants
displayed slight variability with ODRs, with an
overall mean of 3.3 (range, 1.7-5.0) ODRs per
week. Upon introduction of the CICO intervention phase, all 4 participants displayed a reduction in level and trend. Overall, the participants
had a mean of 1.2 (range, 0.63-2.3) ODRs.
Lauren. At baseline, Lauren had 2.3 ODRs
per week (range, 2-3) with a relatively level
trend and slight variability. In the CICO
intervention phase, a gradual decrease in level
and trend was observed as compared with the
baseline phase. Lauren averaged 1 ODR per

August 2011 / 231


100

Baseline

CICO

80

Lauren

60
40
20
12 3

6

9

12

15

18

21

24

27

30

33

100
Andrew?

80
60
O

40

X
ta

20
I

123

6

9

12

15 IE

I

I—I—I

21

I

I

24

I—I—I

I—I

I

27 30

I

I

i_j

33 36 39 42

Pam

60
40
20
0
12 3

6

12

15

18

21

24

27

30

33

100
Stanley

80
60
40
20
0

-•—>—I—I—I—I—I I

123

6

9

I—I—i—I

12

I

15

I

iXi

t—1 I

I

•

•

• •

18 21 24 27
SESSIONS

30

33 36

39 42

Figure 1. Percentage oí intervals with problem behavior for Lauren, Andrew, Pam, and Stanley.

week (range, 0-3). The PND for Lauren was
75%, suggesting that the CICO intervention
was effective at decreasing the student's
weekly ODRs.
Andrew. At baseline, Andrew had 4.3
ODRs per week (range, 4-5) with relatively
no trend and slight variability. In the CICO
intervention phase, an immediate decrease in
level and trend was observed as compared
232 / August 2011

with the baseline phase. Andrew averaged 1
ODR per week (range, 0-3). The PND for

Andrew was 100%, suggesting that the CICO
intervention was effective at decreasing the
student's weekly ODRs.
Pam. At baseline, Pam had 1.7 ODRs per
week (range, 1-2) with an increasing trend and
slight variability. For the CICO intervention
phase, a gradual decrease in level and trend
Behavioral Disorders, 36 (4), 225-240


10
9
8
7
6
5
-1
3
2
1
O

Baseline

CICO
Lauren

10

4

5

8

9

10

11

12

13

11

14

10
9
8
7
6
5
4
3
2
1
O

Pam

10

10
9

11

Stanley

8
7
6
5
4
3
2
1
Û

1

2

3

4

5

6

7

8

9

10

11

12

13

14

•WEEKS
Figure 2. Weekly office discipline referrals (ODRs) for Lauren, Andrew, Pam, and Stanley.

was observed. Pam averaged 1 ODR per week
(range, 0-3). The PND for Pam was 63%,
suggesting that the CICO intervention was
somewhat effective at decreasing the student's
weekly ODRs.
Stanley. At baseline, Stanley had 5.0
ODRs per week (range, 4-6) with relatively

Behavioral Disorders, 36 (4), 225-240

no trend and slight variability. For the CICO
intervention phase, a gradual decrease in level
and trend was observed. Stanley averaged 2.3
ODRs per week (range, 0-5). The PND for
Stanley was 75%, suggesting that the CICO
intervention was effective at decreasing the
student's weekly ODRs.
August 2011 / 233


Percentage of DPR Points Earned
Lauren. During baseline, Lauren earned an
average of 48% (range, 44%-51%) of her
possible DPR points. Under CICO, Lauren
earned an average of 79% (range, 66%-96%)
of her possible weekly DPR points. The PND
for Lauren was 100%, suggesting that CICO
was effective at increasing the percentage of
daily DPR points earned.
Andrew. During baseline, Andrew earned
an average of 37% (range, 32%-41%) of his
possible DPR points. Under CICO, Andrew
earned an average of 82% (range, 69%-91%)
of his possible weekly DPR points. The PND
for Andrew was 100%, suggesting that CICO
was effective at increasing the percentage of
daily DPR points earned.
Pam. During baseline, Pam earned an

average of 56% (range, 53%-59%) of her possible DPR points. Under CICO, Pam earned an
average of 75% (range, 58%-88%) of her possible
weekly DPR points. The PND for Pam was 92%,
suggesting that CICO was effective at increasing
the percentage of daily DPR points earned.
Stanley. During baseline, Stanley earned
an average of 44% (range, 41%-47%) of his
possible DPR points. Under CICO, Stanley
earned an average of 78% (range, 47%-98%)
of his possible weekly DPR points. The PND
for Stanley was 92%, suggesting that CICO
was effective at increasing the percentage of
daily DPR points earned.

Academic Progress Monitoring
Figure 3 summarizes the DCPM and EPM
obtained for each student on multiple skill
grade-level probes. The data were visually
analyzed separately with regard to changes in
level, trend, and variability. Students' actual
rate of improvement was also visually analyzed on grade-level multiple skill probes.
Lauren. According to the pretreatment
assessment, Lauren was performing at the lower
end of the instructional range for her grade level,
with a median score of 14 DCPM. Visual
inspection revealed a level trend with a slight
degreeof variability with a mean of 14.7 DCPM
(range, 14-15) across the baseline phase. For the
CICO intervention phase, a gradual increase in
level, trend, and variability was observed with a

mean 16.3 DCPM (range, 14-20). Visual
analysis of errors suggests a gradually decreasing trend and slight variability with a mean of
1.3 EPM (range, 1-2) across baseline. Errors in
234 / August 2011

the CICO intervention phase were observed to
decrease in level, trend, and variability with a
mean of 0.38 EPM (range, 0-1 ).
Andrew. According to the pretreatment
assessment using Curriculum Based Measurement (CBM), Andrew was performing at the
appropriate instructional level for his grade, with
a median score of 15.5 DCPM. Visual inspection
revealed a gradually increasing trend with some
degree of variability with a mean of 15.3 DCPM
(range, 14-16) across the baseline phase. With
regard to the CICO intervention phase, a gradual
increase in level, trend, and variability was
observed with a mean 17.0 DCPM (range, 1618) across the phase. Visual analysis of errors
suggests a relatively level trend and slight
variability with a mean of 1.3 EPM (range, 0-2)
across baseline. Errors in the CICO intervention
phase were observed to decrease in level, trend,
and variability with a mean of 0.38 EPM (range,
0-2) across the phase.
Pam. According to the pretreatment assessment using CBM, Pam was performing at
the lower end of the instructional range for her
grade level, with a median score of 14 DCPM.
Visual inspection revealed a gradually increasing trend with some degree of variability with
a mean of 13.7 DCPM (range, 13-14) across
the baseline phase. With regard to the CICO

intervention phase, a gradual increase in level,
trend, and variability was observed with a
mean of 14.1 DCPM (range, 11-17) across the
phase. It may be interesting to note that Pam
achieved her lowest performance (11 DCPM)
during the second week of the CICO intervention. This low score coincided with her highest
percentage of intervals with observed problem
behavior (25%).
Visual analysis of errors suggests a decreasing trend with slight variability with a
mean of 1.7 EPM (range, 1-2) across baseline.
Errors in the CICO intervention phase were
observed to gradually decrease in level and
trend with moderate variability and a mean of
0.38 EPM (range, 0-1) across baseline.
Stanley. According to the pretreatment
assessment using CBM, Stanley was performing
at the appropriate instructional level for his grade,
with a median score of 21 DCPM. Visual
inspection revealed a gradually increasing trend
with some degree of variability with a mean of
20.8 DCPM (range, 20-22) across the baseline
phase. With regard to the CICO intervention
phase, a gradual increase in level, trend, and
variability was observed with a mean 22.3 DCPM
(range, 20-24) across the phase. Visual analysis of
Behavioral Disorders, 36 (4), 225-240


•I-]-

Baseline

25
20
15

10
5
0

EPM

1

2

3

4

5

6

25

7

8

9

10

11

Andrew

20
15
10
5
! O
I
I
I

7

R

9

10

11

1?

14

I

i 25
' 20
15
10
5
0

Pam

1

2

3

4

5

25

6

7

8

9

10

9

10

11

Stanley

20
15
10
5
0
1

2

3

4

3

6
7
8

"WEEKS

U

12

13

14

Figure 3. Digits correct per minute (DCPM) and errors per minute (EPM) for Lauren, Andrew,
Pam, and Stanley on multiple skill worksheets.

errors suggests a relatively level trend with slight
variability was observed with a mean of 1.8 EPM
(range, 0-3) across baseline. Errors in the CICO
intervention phase were observed to gradually
decrease in trend, level, and variability with a
mean of 0.75 EPM (range, 0-2) across baseline.

Behavioral Disorders, 36 (4), 225-240

Social Validity
Table 1 summarizes the guidance counselors' and the participant's classroom teachers'
perceptions of social validity using the BIRS.
Mean counselor ratings on the Acceptability

August 2011 / 235

TABLE 1
Guidance Counselor and Teacher Ratings
on the BIRS

BIRS Scale

Guidance Counselors,
M (Range)

Teachers,
M (Range)

Acceptability

76 (74-77)

Effectiveness
Time of Effect

TABLE 2
Student Ratings on the BEP Acceptability
Questionnaire
BEP Question

Student Ratings,
M (Range)

69 (67-71)

Improves problem behavior at school

4.5 (4-5)

33 (31-35)

36 (33-38)

Worth time and effort

4.5 (4-5)

8 (7-8)

6 (5-7)

Note. The maximum scores for the factors are 90 for
Acceptability, 42 for Effectiveness, and 12 for Time of Effect.
Higher scores indicate agreement with each factor.

Recommended to others

5 (5-5)

Easy to participate

4 (4-4)

Improves academic performance

3.5 (3-4)

Note. Higher scores indicated agreement with the question.
The maximum score for each question was 6.

factor were 76 (range, 74-77) out of the possible
90. Mean counselor ratings on the Effectiveness
factor were 33 (range, 31-35), out of the
possible 42. Mean counselor ratings on the
Time of Effect factor were 8 (range, 7-8), out of
the possible 12. This finding indicates that both
guidance counselors viewed the CICO intervention as acceptable and effective but were
slightly concerned with the amount of time
needed to implement CICO.
Mean teacher ratings on the Acceptability
factor were 69 (range, 67-71) out of the
possible 90. Mean teacher ratings on the
Effectiveness factor were 36 (range, 33-38).
Mean counselor ratings on the Time of Effect
factor were 6 (range, 5-7). This finding
indicates that the participant's classroom
teachers viewed the CICO intervention as
acceptable and effective but were somewhat
concerned with the amount of time needed to
implement CICO.
Table 2 summarizes the students' perceptions of social validity using the five-item BEP
Acceptability Questionnaire (Hawken & Horner, 2003). Mean participant ratings for
whether the CICO intervention was perceived
to improve problem behavior at school and
whether CICO was worth the time and effort
were 4.5 on a 6-point scale. The highest mean

participant rating (5) involved whether the
intervention was worth recommending to
others. Mean participant ratings for whether
CICO was easy to participate in and for
whether CICO improved academic performance were 4 and 3.5, respectively.
Interobserver Agreement for Problem
Bebavior Observation Data
The actual overall mean lOA was 94%
(range, 93%-98%); for Lauren, it was 95%
(range, 94%-95%); for Andrew, 95% (93%236 / August 2011

98%); for Pam, 94% (range, 93%-95%); and
for Stanley, 94% (range, 93%-95%).

Treatment Integrity
For Lauren, mean treatment integrity was
88%, (range, 85%-92%); for Andrew, 98%
(range, 95%-100%); for Pam, 92% (range,
90%-94%); and for Stanley, 90% (89%-91%).
Overall mean treatment integrity for this study
was 92% (range, 85%-100%). The lOA for
treatment integrity for all participants across all
observed CICO sessions was 100%.

Interscorer Agreement
For Lauren, the mean interscorer agreement was 99% (range, 98%-100%); for
Andrew, 99% (range, 97%-100%); for Pam,
99% (range, 97%-100%); and for Stanley,
99% (97%-100%). Overall mean interscorer
agreement was 99% (range, 97%-100%).

Discussion
Given that approximately 13% of elementary students demonstrate between two and six
ODRs per year despite the presence of
primary-level preventative measures (Horner
et al., 2005), there is a great need for effective
and cost-efficient secondary interventions.
Concurrent with the findings of previous
research (Filter et al., 2007; Hawken & Horner,
2003; Hawken et al., 2007; Todd et al., 2008),
the CICO intervention effectively reduced the
problem behaviors of all 4 participants.
In terms of direct observations of problem
behaviors, the 4 participants displayed variable levels of problem behaviors with an
overall mean of 32.8% of observed intervals
Behavioral Disorders, 36 (4), 225-240


with problem behaviors. Upon introduction of
the CICO intervention, all 4 participants
displayed a reduction in level and trend. The
participants engaged in an overall mean of
21.4% (range, 16.9%-29.3%) of observed
intervals with problem behaviors across the
CICO phase. With respect to weekly ODRs,
the 4 participants displayed an overall mean of
3.3 (range, 1.7-5.0) ODRs per week across the
baseline phase. Upon introduction of the
CICO intervention, all 4 participants displayed
a reduction in level and trend in terms of their

weekly ODRs. Overall, the participants had a
mean of 1.2 (range, 0.63-2.3) ODRs. Also
concurrent with previous research, this study
has shown that both school faculty and staff
were able to implement the necessary procedures with a high degree of treatment integrity.
Actual overall mean treatment integrity for this
study was 92% (range, 85%-100%).
Hawken and Horner (2003) demonstrated
that the CICO intervention was associated with
increases in the mean level of academic
engagement for all participants involved, yet
few studies have examined CICO's effect on
outcome variables such as academic performance and achievement. Perhaps a contribution of the present study is the examination of
CICO's effect on mathematics performance as
measured by DCPM and EPM. Mean DCPM
for all participants increased over baseline
when the CICO intervention was implemented. Furthermore, EPM for all participants
decreased from baseline when the CICO
intervention was implemented. These results
seem to be congruent with the Hawken et al.
(2007) hypothesis that if implementing CICO
results in students engaging in less problem
behavior and spending less time in the school
office, improved academic achievement may
likely follow. For all participants, FBA results
indicated that the primary function of problem
behaviors was adult attention. CICO was
designed to increase feedback and positive
adult attention (Crone et al., 2004). By implementing an intervention designed to address the
hypothesized function of the participant's problem behaviors, all participants engaged in less

problem behavior and may have experienced
ancillary academic benefits.
Another contribution of the present study
is the addition the PND effect size measure.
Historically, single-subject researchers have
not used statistics to support conclusions for
intervention effectiveness (Derenne & Baron,
1999). Rather, single-subject researchers have
Behavioral Disorders, 36 (4), 225-240

relied on strong internal validity of designs and
use of visual analysis to document intervention
effectiveness (Marascuilo & Busk, 1988; Parsonson & Baer, 1978). To date, no previously
published study investigating the CICO intervention has employed a measure of effect size.
Recently, however, single-subject researchers
have begun reporting effect sizes for singlesubject experiments (Olive & Smith, 2005).
Indeed, the publication manual of the American Psychological Association (2009) suggests
that all manuscripts submitted for publication
include effect size calculations to facilitate
interpretation of intervention outcomes. Given
this, PND all was calculated to help evaluate
both the behavioral and academic outcomes.
With respect to the present investigation, the
PND measure of effect size suggests that the
CICO intervention was effective in reducing
the participant's problems behaviors in terms
of direct observations and ODRs.
The manner in which social validity was
evaluated during the present investigation may
also serve as a contribution. Previous CICO

studies (Hawken & Horner, 2003; Hawken et
al., 2007; Todd et al, 2008) examined social
validity using the BEP Acceptability Questionnaire (Hawken & Horner, 2003), which has no
published psychometric properties. Given the
relative lack of documented psychometric
properties, the present investigation employed
the BIRS to assess social validity with both
guidance counselors and the participant's
classroom teachers. The results revealed that
both the guidance counselors and the teachers
rated the CICO intervention as acceptable and
effective. Both groups also expressed some
concern with the Time of Effect factor. More
specifically, the teachers and counselors rated
disagree or slightly disagree in response to the
statement, "The intervention would quickly
improve the child's behavior." These results
are congruent with those obtained by
McCurdy, Kunsch, and Reibstein (2007),
which used the Intervention Rating Profile
and the Children's Intervention Rating Profile
to measure perceptions of a CICO intervention. Results also indicated teachers and
students had strong satisfaction with the CICO
intervention.
An additional positive outcome of the
study was the direct behavioral observations
that were implemented with a high degree of
interobserver agreement. Some previous studies (i.e., Hawken et al., 2007) have relied on
ODRs as evidence of behavior change. ConAugust 2011 / 237

current with the methodology of previous
CICO researchers (Campbell & Anderson,
2008; Fairbanks et al., 2007; Hawken &
Horner, 2003; Todd et al., 2008), the addition
of independent observations lend support for
the ODR findings, which provide two measures that demonstrate the intervention effects.
The high degree of treatment integrity as
measured throughout all phases of the study
may also be a strength of the current study.
Treatment integrity represents a critical component in evaluating the effects of an intervention. Despite the importance of treatment
integrity, the majority of researchers within the
CICO literature have failed to examine it
appropriately. For example, two studies have
explored the effectiveness of CICO programs
that were already implemented in schools (Filter
et al., 2007; Hawken, 2006; Hawken et al.,
2007). Thus, treatment integrity could not be
evaluated across all phases of CICO implementation. Other studies (Todd et al., 2008) have
failed to present any fidelity data, although they
reported that the data had been collected.
Although this study has added to the
research base, there are a few limitations that
are worthy of discussion. First, the study was
conducted in one school district that has specific
demographic, geographic, and ecological factors that limit the scope of the generalization of
the results. These results can be generalized
only to a population of students similar to the
ones that participated in the study.
Second, CICO's effects on mathematics

may need to be interpreted with caution in that
the results could be due to maturation alone.
Students have demonstrated improvement with
exposure to the learning environment as demonstrated in several universal screening or
benchmarking data sets. However, during the
8 weeks of CICO intervention, the participants
exhibited fewer problem behaviors, making it
highly likely that they spent more time in class
and experienced fewer classroom disruptions.
This combination of more time in the learning
environment coupled with fewer distractions
may have led to increases in math engagement.
Third, the acceptability measures used in
this study either have been used with CICO
interventions but lack psychometric data or the
measure has psychometric data but has no
previous association with CICO. Thus, the results
pertaining to acceptability should be interpreted
with caution. However, the individual analysis
has garnered interesting information about
potential issues of acceptability with CICO.
238 / August 2011

Fourth, pairing the participants in dyads
with only two phases may raise threats to
internal validity. Future research may address
this concern by employing a withdrawal design
or a multiple baseline across three or more
participants where a withdrawal is impractical.
Although dyads were used, the participants

were in different classrooms, different classes of
behaviors (i.e., problem behaviors versus academic performance) systematically changed
with the implementation of CICO, and each
student had a hypothesis that his or her
respective problem behaviors were attention
maintained.
Finally, because some data were collapsed into a weekly aggregate score, nuances
in the individual data points cannot be
evaluated. However, this practice is not
uncommon in behavioral research. Collapsing
data either into a weekly score or using
medians can reduce any variability in data
that can mislead or draw attention to an outlier
that is not necessarily as important in the total
picture. In fact, by the authors using PND all
as well as visual inspection, they have
attempted to account for such variance and
not make decisions about the data inappropriately.
With these limitations in mind, future
research is warranted on CICO. This study is
promising but could be extended to examine
whether CICO would have an effect on other
academic measures including but not limited
to reading, comprehension, other mathematics
computation or application, or other areas. In
addition, the present study wanted to determine if there were any generalization to
academic variables without directly intervening on mathematics, but programming generalization may have better effects than hoping
for generalization.
Although there is a firm research base on
CICO and its effects on behavior, more

research is needed to examine the contributory
value of each component in the CICO
package. Future studies could explore the
components by conducting a component
analysis or adding and removing components
to determine the relational or causal effects on
both behavioral and academic variables.
In addition, future research should also
examine the effects of CICO with other target
behaviors or response classes. Finally, it may
be interesting to examine the effects of CICO
in school districts with various levels of PBS
implementation (e.g., no implementation, low
Behavioral Disorders, 36 (4), 225-240


levels of implementation, moderate levels of
implementation, and high levels of implementation).
In summary, all participants displayed
fewer problem behaviors, as measured through
direct observation and ODRs, when CICO was
in place. Furthermore, the decrease in problem
behaviors may have contributed to minimal
gains in basic math skills. Although more
research is needed, the CICO intervention
appears to be an effective Tier 2 intervention
as it is continuously available, requires minimal staff effort, and allows for ongoing data
collection and evaluation.
REEERENCES
American Psychological Association. (2009). Publication manual of the American Psychological

Association (6th ed.). Washington, DC: Author.
Burns, M., VanDerHeyden, A., & Jiban, C. (2006).
Assessing the instructional level for mathematics: A comparison of methods. School Psychology Review, 35, 401-418.
Campbell, J. M. (2004). Statistical comparison of four
effect sizes for single-subject designs. Behavior
Modification, 28, 234-246.
Campbell, A., & Anderson, C. (2008). Enhancing
effects of check-in/check-out with function
based support. Behavioral Disorders, 33(4),
233-245.
Carr, J. E. (2005). Recommendations for reporting
multiple baseline designs across participants.
Behavioral Interventions, 20, 219-224.
Crone, D. A., Horner, R. H., & Hawken, L. (2004).
Responding to problem behavior in the school:
The behavior education program. New York:
Guilford.
Derenne, A., & Baron, A. (1999). Huan sensitivity to
reinforcement: A comment on Kollins, Newland, and Critchfield's (1997) quantitative literature review. Behavior Analyst, 22, 35-41.
Doggett, R. A., Edwards, R. P., Moore, J. W.,
Tingstrom, D. H., & Wylczynski, S. M. (2001).
An approach to functional assessment in general
education classroom settings. School Psychology Review, 30, 313-328.
Dufrene, B. A., Doggett, R. A., Henington, C, &
Watson, T. S. (2007). Functional assessment and
intervention for disruptive classroom behaviors
in preschool and head start classrooms. Journal
of Behavioral Fducation, 16, 368-388.
Edwards, R. P. (2002). A tutorial for using the
functional assessment informant record for

teachers. Proven Practice, 4, 31-33.
Elliott, S. N. (1988). Acceptability of behavioral
treatments in educational settings. In J. C. Witt,
S. N. Elliott, & F. M. Gresham (Eds.), Handbook
of behavior therapy in education (pp. 121-150).
New York: Plenum.

Behavioral Disorders, 36 (4), 225-240

Elliot, S. N., & Treuting, M. V. (1991). The Behavior
Intervention Rating Scale: Development and
validation of a pretreatment acceptability and
effectiveness measure. Journal of School Psychology, 29, 43-51.
Fairbanks, S., Sugai, C , Guardino, D., & Lathrop, M.
(2007). Response to intervention: Examining
classroom behavior support in second grade.
Fxceptional Children, 73, 288-310.
Filter, K. J., McKenna, M. K., Benedict, E. A., Horner,
R. H., Todd, A. W., & Watson, J. (2007). Check
in/check out: a post-hoc evaluation of an
efficient, secondary-level targeted intervention
for reducing problems behaviors in schools.
Fducation and Treatment of Children, 30, 69-84.
Fleming, C. B., Harachi, T. W., Cortes, R. C, Abbott,
R. D., & Catalano, R. F. (2004). Level and change
in reading scores and attention problems during
elementary school as predictors of problem
behavior in middle school. Journal of Fmotional
and Behavioral Disorders, 12, 130-144.
Hawken, L. S. (2006). School psychologists as leaders

in the implementation of a targeted intervention.
School Psychology Quarterly, 27(1), 91-111.
Hawken, L. S., & Horner, R. H. (2003). Evaluation of
a targeted intervention within a schoolwide
system of behavior support. Journal of Behavioral Fducation, 12, 225-240.
Hawken, L. S., MacLeod, K. S., & Rawlings, L.
(2007). Effects of the behavior education program (BEP) of office discipline referrals of
elementary school students. Journal of Positive
Behavior Interventions, 9, 94-101.
Horner, R. H., Sugai, G., Todd, A. W., & LewisPalmer, T. (2005). Schoolwide positive behavior
support: An alternative approach to discipline
in schools. In L. Bambara, & L. Kern (Eds.),
Individualized supports for students with problem behaviors: Designing positive behavior
plans (pp. 359-390). New York: Guilford Press.
Irvin, L. K., Horner, R. H., Ingram, K., Todd, A. W.,
Sugai, G., Sampson, N. K., & Boland, J. B.
(2006). Using office discipline referral data for
decision making about student behavior in
elementary and middle schools: An empirical
evaluation of validity. Journal of Positive Behavior Interventions, 8, 10-23.
Irvin, L. K., Tobin, T. J., Sprague, J. R., Sugai, G., &
Vincent, C. G. (2004). Validity of office discipline referral measures as indices of school-wide
behavioral status and effects of school-wide
behavioral interventions. Journal of Positive
Behavior Interventions, 6, 131-147.
Lee, Y., Sugai, G., & Horner, R. (1999). Effect of
component skill instruction on math performance and on-task, problem, and off-task
behavior of students with emotional and behavioral disorders. Journal of Positive Behavior
Interventions, 1, 195-204.
Maguin, E., & Loeber, R. (1995). Academic performance and delinquency. In M. Tonry (Ed.),

Crime and justice: An annual review of research

August 2011 / 239


(Vol. 20, pp. 145-264). Chicago, IL: University
of Chicago Press.
Marascuilo, L. A., & Busk, P. L. (1988). Combining
statistics for multiple-baseline AB and replicated
ABAB designs across subjects. Behavioral Assessment, W, 1-28.
March, R. E., & Horner, R. H. (2002). Feasibility and
contributions of functional behavior assessments
in schools. Journal of Emotional and Behavioral
Disorders, 10, 158-170.
McCurdy, B. L., Kunsch, C, & Reibstein, S. (2007).
Tier II prevention in the urban school: Implementing the Behavior Education Program. Preventing School Eailure, 51, 12-19.
Mclntosh, K., Flannery, K., Sugai, C , Braun, D., &
Cochrane, K. (2008). Relationships between
academics and problem behavior in the transition from middle school to high school. Journal
of Positive Behavior Interventions, 10, 243255.
Mclntosh, K., Horner, R. H., Chard, D. J., Boland, J.
B., & Good, R. H. (2006). The use of reading and
behavior screening measures to predict nonresponse to school-wide positive behavior support: A longitudinal analysis. School Psychology
Review, 35, 275-291.
Morrison, G. M., Anthony, S., Storino, M., & Dillon,
C. (2001). An examination of the disciplinary
histories and the individual and educational
characteristics of students who participate in an
in-school suspension program. Education and
Treatment of Children, 24, 276-293.

National Association of State Directors of Special
Education (NASDE). (2005). Response to intervention: Policy considerations and implementation. Alexandria, VA: NASDE.
Olive, M. L., & Smith, B. W. (2005). Effect size
calculations and single subject designs. Educational Psychology, 25, 313-324.
Parsonson, B. S., & Baer, D. M. (1978). The analysis
and presentation of graphic data. In T. R.
Kratochwill (Ed.), Single subject research: Strategies for evaluating change (pp. 101-166). New
York: Academic.
Pelham, J., Fabiano, G., & Massetti, G. (2005).
Evidence-based assessment of attention deficit
hyperactivity disorder in children and adolescents. Journal of Clinical Child & Adolescent
Psychology, 34, 449-476.
Roeser, R. W., & Eccles, ). S. (2000). Schooling and
mental health. In A. J. Sameroff, M. Lewis &
S. M. Miller (Eds.), Handbook of developmental
psychopathology (2nd ed., pp. 135-156). New
York: Kluwer Academic/Plenum.
Scruggs, T. E., & Mastropieri, M. A. (2002). On
babies and bathwater: Addressing the problems

240 / August 2011

of identification of learning disabilities. Learning
Disability Quarterly 25, 155-168.
Sugai, G., & Horner, R. (2001). School climate and
discipline: Coing to scale. Paper presented at
the National Summit on the Shared Implementation of IDEA, Washington, DC.
Tobin, T. |., & Sugai, G. M. (1999). Using sixth-grade
school records to predict school violence,
chronic discipline problems, and high school

outcomes. Journal of Emotional and Behavioral
Disorders, 7, 40-53.
Todd, A. W., Campbell, A. L., Meyer, G. G., &
Horner, R. H. (2008). The effects of targeted
interventions to reduce problem behaviors:
Elementary school implementation of check incheck out. Journal of Positive Behavior Interventions, 10, 46-55.
Treuting, M., & Elliott, S. N. (1991). The Behavior
Intervention Rating Scale: Development and
validation of a pretreatment acceptability and
effectiveness measure. Journal of School Psychology, 29, 43-5^.
Walker, H. M., Ramsey, E., & Gresham, F. M.
(2004). Antisocial behavior in school with
infotrac: Evidence-based practices. Belmont,
CA: Wadsworth.
Walker, H. M., & Shinn, M. R. (2002). Structuring
school-based interventions to achieve integrated
primary, secondary, and tertiary prevention
goals for safe and effective schools. In M. R.
Shinn, H. M. Walker, & G. Stoner (Eds.),
Interventions for academic and behavior problems II: Preventive and remedial approaches
(pp. 1-25). Bethesda, MD: National Association
of School Psychologists.
Wilkinson, L. (1999). Statistical methods in psychology journals: Guidelines and explanations.
American Psychologist, 54, 594-604.

AUTHORS' NOTE
Address correspondence to Michael Mong,
The University of Southern Mississippi, 730 E.
Beach Blvd, Long Beach, MS 39560; E-mail:

MANUSCRIPT
Initial Acceptance: 2/27/11
Final Acceptance: 4/14/11

Behavioral Disorders, 36 (4), 225-240


Copyright of Behavioral Disorders is the property of Council for Children with Behavioral Disorders and its
content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's
express written permission. However, users may print, download, or email articles for individual use.