Weaving the Essential Tapestry

1
Weaving the Essential Tapestry
OBM Research to Practice in Training Autism
Personnel On-Line
Richard K. Fleming Beth Sulzer-Azaroff
2
Todays Presentation

• Describe the development, evaluation and
  dissemination of a comprehensive distance
  learning curriculum for personnel and parents of
  children with autism.
• Discuss the tailored application of OBM theory
  and research in this endeavor.
• Present results from our field evaluation.
• Discuss future research and dissemination.

3
Our Charge

• To design evaluate and disseminate a
  behaviorally-based undergraduate level distance
  curriculum to train educators and parents to
  apply intensive behavioral interventions among
  children with autism.
• We use the acronym, BIA, which stands for
  Behavioral Interventions in Autism
• Educators, broadly defined to include
• Regular and special education teachers
• Early intervention personnel
• Speech and language therapists
• And others

4
Project Personnel

• UMMS/Shriver
• Charles Hamad, Principal Investigator
• Richard Fleming, Project Manager
• Megan Tupa, Research Associate
• Susan Dyer, Research Associate
• Anncy Graziano, Admin. Asst.
• Robert Bass, NE Index (technology)
• Jay Mavor, NE Index
• John Rochford, NE Index
• Lee Vorderer, Project Coordinator
• The Browns Group of Naples
• Beth Sulzer-Azaroff, Director of Curriculum
  Development

• Rockman et al
• Michelle Weissman, External Evaluator
• Other Key Personnel
• Jennifer Crockett, Instructor
• Alicia Perry, Instructor
• The Groden Center
• Marcie Desrochers, St. Lawrence College
• Paul Roselli, Corporate Video Productions

5
Our Personal Goals

• To develop a BIA curriculum to
• Effectively teach teachers and parents, to, in
  turn become effective teachers of their students
• Support their
• participation in and completion of BIA
• satisfaction with the experience
• continuation within the ABA field
• Attract numerous other students in the future

6
Learning the Hard Way

• Innovation the OBM/ABA way
• Consider the special contingencies of
  reinforcement and punishment affecting personnel
  and parents, and how they either support/maintain
  or impede progress
• internal and external stakeholders
• obvious and subtle reinforcing aversive
  stimuli
• Negative and Positive examples of stakeholders
  and contingencies affecting them
• School visits
• Failing to prepare students to supply only
  positive feedback
• Safety programs union reaction
• Failing to involve them at the start
• Setting up a demonstration program
• Surveying all stake-holders
• You probably have examples of your own

7
Identifying and Analyzing Multiple and
Interlocking Critical Contingencies within
Complex Projects

• applied behavior analysts should, when working
  in any setting, measure the variables maintaining
  behavior before changing the behavior (Austin,
  2000, p. 343).
• With BIA, that has meant observing our students
  performance before, during and after our training
• Pre-tests of verbal/conceptual skills
• Baseline performance measures
• Verbal statements about content, coverage,
  clarity, style, overall value and so on
• And the verbal statements (generally via
  questionnaire) of our colleagues and consumers
  early and later during our students involvement
• Field facilitators
• Administrators
• BIA Instructors
• Parents
• Other constituencies?

8
Constituencies Impacted by BIA Program

• Our Consumers
• Our supervisor
• Partners
• Colleagues
• Students
• Mastery of content and rate of progress through
  the course sequence
• Satisfaction
• Instructors
• Funding agency

• Our Students
• Consumers
• The progress of their students with autism
• Satisfaction of
• Cooperating teachers parents
• ABA in autism colleagues
• Administrators of field setting
• Others?

9
Sources of Contingency Control for Our BIA
Students and Their Associates

• Individual child with autism
• Progress rate
• Side effects
• Families
• Need for care-giving
• Educating
• Enabling self sufficiency
• Interacting with and gaining affection from child
• Avoiding melt-downs
• Tantrums and self-injury
• Principal
• Seeing student progress, learning inclusion
• Avoiding irate parents teachers

• Teachers and associates
• Seeing student progress
• Maintaining health and safety of student and
  others
• Satisfying team members, supervisors, families
• Relations friends
• Interacting with and gaining affection from child
• Avoiding melt-downs tantrums and self-injury
• Other instructional specialists (OT, PT, SLP)
• Seeing student progress, e.g., functional
  communication
• Gaining positive, supportive feedback
• Better able to assist educators and caregivers

10
Other Sources of Contingency Control

• Project Personnel Managers
• Maximizing performance of personnel
• Retaining effective personnel
• Boosting child performance within managers unit
• Minimizing human effort and material resources
  Administrator(s)
• Training investment
• Retention and advancement of personnel
• Project Production Staff
• Satisfaction with product
• Efficiency
• New skills

• Funding Agency (DOE, FIPSE)
• Publicity
• Product
• Completion
• Favorable evaluation by consumers
• Meet mission
• Dissemination Plan an mechanism for broad
  dissemination
• Project Administrators
• Recognition
• Future grants
• Minimizing resources

11
Still Other Sources of Contingency Control

• Curriculum developers
• BIA student progress
• Ours and theirs
• Approval
• PI
• Funding agency
• Colleagues within ABA/autism field
• Opportunity to learn and be creative
• Outside evaluators
• Data for
• Granting agency
• Partners
• Future business
• Parent, other advocacy groups

• ABA constituency
• BA Certification Board
• Quality assurance
• Recognition
• Attracting candidates
• Passing exam
• Colleagues
• Methods based on sound scientific data
• Training students to mastery of ABA conceptual
  and applied skills
• Interest and participation from other countries

12
Still More . Disseminators of Program

• Instructors other direct service providers
• Satisfaction with student progress
• Satisfaction with rewards of job
• Interaction with students colleagues
• Associations with academia
• Positive feedback from managers, peers, parents,
  data
• Others

• Institutions of higher education
• Satisfied students
• Students subsequently successful on job
• Managers of service delivery programs (schools,
  workshops, day programs etc.)
• Effective student learning
• Smoothly operating delivery systems
• Dedicated, enthusiastic, skilled teaching staff

13
A Sample Letter to the Administrator of the
Organization

• Your address
• Date
• Dear (Senior Administrator),
• I want to tell you of an exciting opportunity you
  may find beneficial to your program as well as to
  me. I am enrolled as a student at _____________
  in a course entitled Practicum in Behavioral
  Interventions in Autism (BIA). It is designed
  to enable students to gain proficiency in the
  practice of applied behavior analysis (ABA) among
  youngsters with autism spectrum disorder. As you
  probably know, scientific studies have
  demonstrated these methods to be especially
  effective with this population.
• Although we have solid knowledge of many
  important ABA concepts, now we need to begin to
  put that knowledge into practice. Consequently,
  we have been asked to locate a program serving
  children with autism spectrum disorder, and to
  volunteer to participate in specific ways. Under
  the supervision of our BIA instructor, a person
  with advanced training and considerable
  experience in this field, we are asked to spend a
  minimum of six hours a week at the site for
  purposes of
• 1. identifying a student
• 2. selecting a constructive instructional goal or
  objective suited to the student and approved by
  his or her primary educator (and/or parent).
• 3. using an all positive approach to design and
  teach the student to progress toward
• the objective.
• 4. collecting performance data to track the
  students progress and as a basis for
• making sound revisions in the process.
• 5. making data-based revisions in the plan
• 6. preparing a report summarizing the project
• 7. submitting the report to our instructor
• 8. assisting in other ways, as mutually
  determined
• The program asks little of you and your staff,
  other than for your assistance in helping us
  identify a child to work with and an adult (i.e.,
  a field facilitator) to handle local
  arrangements related to our weekly assignments.
  While, as Im sure you know, we cannot guarantee
  of success with a student, we do promise to make
  our best efforts.
• I shall follow up this letter with a telephone
  call during the next few days.
• Sincerely yours,

14
Field Facilitator Agreement

• I agree to allow _______________________ (name of
  BIA practicum student) to attend our program for
  a minimum of ____ hours a week for ___ weeks and
  in keeping with the programs schedule. During
  that time s/he will conduct an instructional
  program with a student1 and possibly
  participate in our activities in other ways, to
  be mutually determined. I understand this
  practicum students Behavioral Interventions in
  Autism (BIA) instruction will be supervised
  directly or from a distance by ___________________
  (name of BIA instructor), a qualified
  Instructor associated with _______________________
  ___________
• (name of college or university).
• Further, I understand this BIA student will
  adhere to professional confidentiality policies
  in reference to his or her student. Although
  s/he may discuss aspects of the instruction with
  her/his instructor privately or classmates during
  formal group discussions, s/he will withhold any
  identifying information, such as names, addresses
  and so on.
• I also agree to help this student identify a
  child in our program to teach a specific learning
  objective acceptable to the childs regular
  teacher and parent(s), record and report on
  progress in writing to his or her instructor and
  to obtain appropriate permissions for
  video-taping and sharing these videos in
  confidence with his or her instructor.
• Further, I agree to complete an interim and final
  rating form that includes a set of the practicum
  students professional skills, such as
  friendliness, attendance, timeliness, helpfulness
  and cooperation.
• _____________________ __________________
  ____________
• Print Name
  Signature Date
• Please add any questions or comments below
• 1 More advanced practicum students may elect
  to develop a program for staff, parents or other
  service providers.

15
Teaching Contract

• As a student of Behavioral Interventions in
  Autism I, (your name) ________________, have been
  assigned to work with ____________________
  (student pseudonym). His or her teacher (/or
  parent) and I have conferred about an appropriate
  instructional objective for me to try to teach
  this child. We have agreed that under my BIA
  Instructors supervision, the objective of this
  teaching is as follows
• __________________________________________________
  _____________________________________________
• __________________________________________________
  _____________________________________________
• Our reasons for selecting this objective are
• __________________________________________________
  _____________________________________________
• __________________________________________________
  _____________________________________________
• __________________________________________________
  _____________________________________________
• __________________________________________________
  _____________________________________________
• I plan to measure __________________s progress
  toward reaching this objective and report the
  results weekly to my course instructor. From
  time to time, I also plan to send video
  recordings of our teaching interactions for my
  instructors review. These and any other
  identifying information will be kept
  confidential, except in the case of the parent(s)
  and teachers future written agreement to share
  them with specifically designated others.
• Parent _______________________
  __________________________ __________
• Print name
  Signature
  Date
• Teacher _______________________
  __________________________ __________
• Print name
  Signature
  Date
• BIA student ___________________
  __________________________ __________
• Print name
  Signature
  Date

16
Resolution

• Cannot satisfy all
• e.g., current suitability for professional level
  staff. Adapt curriculum
• up (BCBA)
• down (how-to for specific behavioral procedures
  to satisfy others
• Set priorities Pareto 8020 rule
• Features of reinforcing contingencies
• Value power
• Time schedule (fixed or variable)
• Frequency (ratio)
• Dependability (fixed or variable)
• Immediacy - delay
• Discriminated- signaled by clear S-Ds

Biggest Bang for the Buck
17
Which Ones Did Program Choose to Emphasize? Why?

• Funding agency (DOE, FIPSE)
• Future Grants
• Powerful but delayed
• Personnel managers administrators
• Powerful more immediate
• Especially with regular reports and inquiries
• Production staff
• Bigger bang for the Buck
• Immediate
• Efficient affects
• progress of project
• quality of material
• ABA constituency
• Delphi
• Requested their input
• BACB input
• Our own priorities

18
Curriculum Developers Priorities

• ABA- Peer supported content
• Evidence-based content
• ABA-based shaping of both knowledge and skills
• Mastery learning - PSI
• Advantages
• Fosters student progress
• Data demonstrate superior student learning and
  retention
• Though the response cost is high
• Early dropouts
• Labor and technology costs
• Our own repertoires
• Basic and applied research
• Conceptual systems analysis
• Research to practice

• Behavioral instruction, PSI Research
• Pear Crone-Todd (1999), PSI in Cyberspace,
  JABA.
• Burton, Moore Magilaro (2002). Behaviorism and
  Instructional Technology.
• Johnson Ruskin (1977). Behavior instruction An
  instructive review. APA.
• JOBM JABA publications on staff and parent
  training, reinforcement and feedback, systems
  analysis, etc.

19
Methodological Sequence

• Analyze the contingencies operating
• Adjust to those contingencies and/or develop new
  one
• possible antecedents and consequences and do an
  armchair analysis (as in the Step 4 of Daniels
  A-B-C Analysis) of the most powerful variables at
  play
• Obvious
• Subtle
• Similar to Brethowers Total Performance System
• Input
• Process
• Product
• Followed by feedback on any discrepancies

20
Methods

• Choose content
• Delphi survey
• Select curriculum
• Choose methods
• Based on behavior analytic research and
  development
• Mastery Instruction
• Active participation
• Discussions
• Simulations
• Actual field experiences
• Rapid feedback and reinforcement

• Develop curriculum for four courses
• 1. Introduction to Behavioral Intervention in
  Autism 
• 2. Positive Teaching and Behavior
• 3. Behavioral Intervention Program Models in
  Autism
• 4. Behavioral Intervention Practicum in Autism

21
Methods (continued)

• Prepare study questions, quizzes, annotated
  answer keys
• Design laboratory and field experiences
• Discussion topics
• Prepare instructor guidelines

• Recruit students
• Distance
• Face-to-face
• Implement curriculum
• Evaluate learning and satisfaction
• Revise content and methods
• Disseminate

22
ResultsDelphi Practitioner Skills
23
Results of Field TrialsData on Student
Performance

• Student demographics
• Pre- and post-test measures, three courses
• Student retention
• Profiles of retakes on mastery assessments
• Participant satisfaction (external evaluation)

24
Demographics Education Level

• Beginning of BIA sequence
• Introduction to BIA
• Two sections
• 1a n16
• 1b n17
• Total reporting N 32 (of the 33)

P E R C E N T
HS
V/T
BS/ BA
MA/ MS
MD/ PhD
25
Reasons for EnrollingCourse 1, two sections,
percent rating each reason as very important
26
ABA Activities Prior to EnrollmentCourse 1, two
sections, percent rating each activity as very
important
27
Pre- and Post-Test MeansCourses 1-3
C O R R E C T
N 26
N 14
N 11
Pre/Post
Pre/Post
Pre/Post
28
Participant RetentionEntire Sequence Courses
1-4(Typical retention in distance courses lt 50)
P A R T I P A N T S
(74)
(75)
(73)
(81)
29
Courses 1 2 Satisfaction with Content
Assignments (Mean rating, 5 most satisfied)
30
Helpfulness of Study Questions(n19, Course 1)
How did the study questions help you?
31
Sample Mastery Assessment Item with Feedback

• 7. Which of the following describes the greatest
  benefit to intervening in the childs natural
  environment?
• More time-efficient, given you can target
  multiple students behaviors at one time.
• Less transfer of training required
• Better generalization to other natural
  environments - Correct!
• Better ability to spot other children with
  intervention needs 7Solution or
  ExplanationAlthough each of the options could
  occur through natural context interventions, the
  greatest benefit is providing the child with
  skills that will generalize. (SQ 12)

32
Mastery Assessments Course 1Mean of takes for
successful students (N12)
Course 1 -----
---- Course 2
Successive Mastery Assessments
33
Satisfaction with Mastery Assessments (Mean
rating, 5 most satisfied)
34
Perceived Competence

• Perceived competence to teach children with
  autism as a function of new knowledge and skills
  acquired in the online courses
• End of Course 1 mean, n15 4.53
• (4 somewhat more competent, 5 considerably
  more competent)
• End of Course 2 mean, n10 4.40
• End of Course 3 mean, n9 4.67
• Perception versus behavior The need for direct
  measurement of performance

35
Course 4 Practicum Mid-term Ratings Scale of 1
to 5, with 5 being the most satisfied

36
Data from Student Project Learning to Mand
37
The Picture Postcard

• I would collect all of the people who have
  benefited or developed a better understanding of
  the "nuts and bolts" of behavioral intervention
  and autism, because of the information shared
  during the 4 BIA courses.  That group would
  include parents, my fellow SLP's, sped teachers,
  instructional aides, volunteers, regular ed
  teachers, student interns, group home staff,
  their other clients, day program staff, ...the
  list is endless...AND...we can't know the
  people that they've shared information with, who
  have also shared.  

38
Overall Results to Date

• All four courses disseminated relatively smoothly
• Considerable support and Satisfaction from the
  many constituencies mentioned
• Many future offshoots

39
Lessons Learned

• Building a distance, mastery training personnel
  training program in ABA has been a real challenge
  requiring
• Analysis of critical metacontingencies
• Coordinated teamwork
• Persistence to incorporate behavioral instruction
  methodology
• The investment of major financial and personal
  resources
• The need is enormous, if not differentiated
• We seem to have demonstrated some successes and a
  good perspective on future directions

40
More Lessons Learned

• Distance learning courses that utilize behavioral
  instruction are an effective means of educating
  students, personnel and parents interested in
  behavioral intervention in developmental
  disabilities.
• But verbal learning is not enough!
• Running field, practicum and other direct
  experiences with the assistance of
  telecommunications technology holds great promise
  of becoming an effective method of distance
  training. Research is needed!

41
What Did We Overlook?

• Higher Education Disseminators
• Failed to explore with disseminators the mastery
  learning method in advance
• Quizzing as instructional method and as
  assessment method
• Concept of mastery learning foreign
• Current approaches and solutions
• Identifying other programs to handle portions of
  the curriculum
• Graduate programs
• Undergraduate programs
• Continuing education
• Collaboration with programs and universities to
  deliver field practicum

42
Dissemination A Lesson in Generality across
Participants and Settings

• Adjusting the curriculum content level
• Graduate-level, BACB-approved version
• Suite of practical modules for paraprofessionals
• Parent education in ABA
• Educating Parents Behavioral Intervention in
  Autism (NIH) underway.
• International market?
• Participation interest from Philippines,
  England, Japan, Nigeria
• Panel discussion ABA Beijing, 2005
• Implications?

43
Current and Future Changes

• Continuing Education
• Completed
• Undergraduate course credit
• In the works
• Revision suitable for graduate level
• In process
• Others

44
Conclusion

• Building a distance, mastery training personnel
  training program in ABA has been a real challenge
  requiring the investment of major financial and
  personal resources
• But the need is enormous
• And we seem to have demonstrated some successes
  and a good perspective on future directions

45
Additional Data Follow
46
The tapestry is nearly complete!
47
Data on Student Performance

• Pre mid-term post-assessment measures
• Number of retakes (?)
• Student retention
• New enrollees
• External evaluations
• Behavioral change
• First versus last teaching tape
• Also loose data in the form of anecdotal
  evidence and comments from parents and teachers.

48
Pre vs. Post-Test ScoresIndividual Students,
Matched Items, Course 1b
49
Knowledge and Skills
50
ABA Technology
51
Personal and/or Work Interest?

• Course 1
• Introduction to BIA
• Two sections
• 1a n16
• 1b n17
• Total reporting 33

Parents/ Relatives
Employees
52
Satisfaction with Threaded Discussion (mean
rating, 5 most satisfied)
53
Mastery Assessments Course 2Cumulative takes
for 3 students who dropped and 3 who finished
Drop
Drop
Drop
54
Other Possibly Important Sources of Contingency
Control in this Program

• How many sources of contingency control can you
  think of in a case like this?
• Hint Think of all the stakeholders and their
  reinforcers and punishers