Title: Jessica TurpakCarey Masse, B.S.Brandon Schultz, Ed.S.
1Jessica Turpak Carey Masse, B.S. Brandon Schultz,
Ed.S.
Treatment Fidelity A Look At a School-Based
Academic Intervention for Youth with Attention
Deficit Hyperactivity Disorder (ADHD)
RESULTS
ABSTRACT
- CHP participant response frequencies were
examined and findings were as follows - There was 100 agreement between the students,
mentors, parents, and consultants when asked, Is
an assignment notebook required? - When asked Is there a requirement for teacher
initials? There was high agreement between
student, mentor, and consultant (ranging from 70
to 80). Agreement was slightly lower between
parents and the school consultant with an
agreement rate of 50, and even lower between the
parent, mentor, and student with frequencies at
40 (see table). - Regarding the question, How often is the
assignment notebook checked? there was a high
agreement between the mentors and students (70)
and a low agreement between the parents and
students (30), as well as among parents and
consultants (40). -
- Treatment fidelity demonstrates that an
experimental manipulation is conducted as planned
and that content and process of the intervention
are equal across participants (Dumas, Lynch,
Laughlin, Smith, Prinz, 2001) Various studies
indicate that treatment fidelity plays a vital
role in ensuring overall validity in a treatment
program (Calsyn, 2000). Fidelity was assessed in
the Challenging Horizons Program (CHP), a school
based intervention for adolescents with ADHD.
This poster presents fidelity assessment data on
one of the variety of interventions in the CHP.
Methods for assessing fidelity are reviewed and
the rationale for selecting survey methodology
for the CHP is explained. Initial data reflected
high and low agreements among respondents
regarding key components of the assignment
tracking method. Implications for the assessment
of fidelity in school-based mental health
services are discussed.
Response Rates Among Parents, Mentors, Students
and School Consultant
DISCUSSION
INTRODUCTION
Assessing treatment fidelity within the
Challenging Horizons Program serves the practical
and vital role of evaluating the delivery of
school-based treatment interventions. After
completing the fidelity questionnaires, initial
data revealed incongruity among participant
responses. The highest levels of inconsistency
were between parents and all other raters.
Miscommunication, misunderstanding and lack of
knowledge regarding students intervention plans
may play a large role in low agreement rates.
Communication between parents and the mentor may
be difficult to accomplish due to time
constraints and scheduling issues. Poor
communication about specific events at middle
school, between parents and their children, is a
common complaint of parents. Other possible
sources of disagreement include inadequate
knowledge of the intervention procedures. Other
issues that may effect fidelity may include
respondent exaggeration and questionnaire
defects. Possible solutions to communication
problems may include requiring monthly parent and
mentor meetings to review and update the students
interventions. Additional training sessions may
be necessary to increase adherence to specific
details of the intervention. The data presented
in this poster represents only a sub-sample of
the fidelity data being collected in this
project, but it does have implications for the
measurement of fidelity. Parents may be poor
sources of information about the implementation
of procedures at middle schools. In addition,
fidelity questions need to be very focused and
directly ask about specific aspects of the
intervention. Future analyses should attempt to
evaluate fidelity data in relation to treatment
outcomes.
The Challenging Horizons Program is a
school-based psychosocial treatment intervention
for adolescents with Attention Deficit
Hyperactivity Disorder (ADHD). Parents of the
adolescents, mentors and the school consultant
collaboratively determine the appropriate
intervention methods for each student, providing
a tailored treatment program based on established
academic and psychosocial treatment methods (e.g.
Assignment Notebook, Organizational Skills
Interpersonal Skills Group) (Evans, Serpell,
Schultz Williams, 2003). Treatment fidelity is
used to demonstrate if a manipulation was
conducted in an experiment as intended (Dumas et
al., 2001). Moncher and Prinz (1991) state that
monitoring treatment fidelity is critical to the
outcome of a treatment program, explaining if
fidelity goes unchecked, treatment outcomes may
be due to an unknown variable rather than the
treatment. It may appear that the treatment
program is ineffective when in fact it simply has
poor implementation. Treatment fidelity of the
CHP was assessed to determine if the program was
being implemented according to manualized
procedures. Treatment fidelity can be assessed
through multiple methods observation, video or
audiotape, and questionnaires (Moncher Prinz,
1991). Questionnaires were chosen as the primary
form of fidelity measurement for the CHP, in
light of observational measurement constraints.
Multiple respondents were used to obtain data
from various sources and improve the validity of
the assessment. Higher levels of fidelity
positively affect validity, giving the study the
ability to be more successfully generalized to
other settings (Evans, Langberg, Williams
2003).
Note D/K Dont Know response, (--) Missing Data
METHODS Continued
- Materials
- Assignment Notebook Questionnaire
- Distributed to parents, students, mentors and the
school consultant. - Consisted of multiple questions aimed at
determining the extent to which core aspects of
the intervention were being completed as planned.
- Some questions consisted of multiple choice
answers, while others gave the respondent the
option of choosing yes/no or dont know. - The number of items on each questionnaire
reflected each respondents role in the
assignment notebook intervention. - 3 questionnaire items that demonstrated the key
aspects of a successful assignment notebook
intervention were examined in the initial
analysis. - Procedure
- Questionnaires were completed during the first
half of the academic school year by parents,
mentors,students and the school consultant. The
gathered data was analyzed by running frequency
distributions and comparing those results among
respondents.
REFERENCES
Calsyn, R.J. (2000). A checklist for critiquing
treatment fidelity studies. Mental Health
Services Research, 2(2), 107-113. Dumas, J.E.,
Lynch, A.M., Laughlin, J.E., Smith E.P., Prinz,
R.J. (2001). Promoting intervention fidelity
Conceptual issues, methods, and preliminary
results from the Early Alliance prevention
trial. American Journal of Preventive Medicine,
20(1S), 38-47. Evans, S.W, Langberg, J.,
Williams, J. (2003). Achieving generalization in
school-based mental health. In M.D. Weist, S.W.
Evans N.A. Lever (Eds.), Handbook of School
Mental Health Advancing Practice and Research
(pp.335-348). New York, NY Kluwer
Academic/Plenum Publishers. Evans, S.W.,
Serpell, Z., Schultz, B., Williams, A.L.
(2003). Developing and transitioning
school-based treatment for ADHD The Challenging
Horizons Program (CHP). Emotional and Behavioral
Disorders in Youth, 3(4), 95-103. Moncher, F.J.,
Prinz, R.J. (1991). Treatment fidelity in
outcome studies. Clinical Psychology Review, 11,
247-266.
METHODS
Participants A portion of the CHP sample (11)
who enrolled earliest in the project were
included in this study. Parents of the students,
mentors (teachers serving as primary counselors),
and the school consultant (research staff),
completed the treatment fidelity assessments.
Ten (10) students received Assignment Notebook
as part of their personalized intervention
program this intervention used to address
difficulties the students have recording and
organizing their assignments.