Advancing

1
Research Brought to Life
The Quality Assurance Process

• Advancing
• Excellence

2009 Juvenile Justice Education Institute and
Southern Conference on Corrections
2
Presentation Outline

• JJEEPs Mission Vision
• QA Results from 2008-09
• Advisory Board/Standards Revision
• Overview of 2009-10 QA Standards
• Quality Assurance (QA) Review Protocol
• Needs Assessment
• QA Indicator Explanation
• Technical Assistance/Corrective Action Plans

3
Mission Vision of JJEEP

• JJEEPs mission is to ensure that each student
  assigned to a Department of Juvenile Justice
  (DJJ) program receives high-quality educational
  services that increase that students potential
  for future success.

4
JJEEPs Four Main Functions

• Conduct research that identifies educational best
  practices
• Conduct annual QA reviews of DJJ educational
  programs
• Provide technical assistance to improve
  educational programs
• Provide annual policy recommendations to the DOE

5
Results from 2008-2009

• Increased the number of exemplary programs from
  27 to 68
• Exemplary programs in every region
• Overall residential average increased from 5.08
  to 5.77
• Used over 30 peer reviewers this review cycle
• Only three programs scoring below 4.00 in the
  state

6
JJEEPs efforts for improvement

• QA reports provided to programs and school
  districts within 45 days of the review
• Communication via listserv, materials posted on
  our Web site, reviewer contacts
• Focus on student outcomes
• Use of research/data for continuous improvement
  of QA standards and process
• Partners for improvement

7
Networking/Mentoring

• Opportunities provided during the conference
• Mentor programs/school districts assigned to
  lowest performing programs

8
Advisory Board/Standards Revision

• Standards are reviewed revised annually with
  solicited input from stakeholders
• Developed Advisory Board with school district,
  private providers, and educational program
  representatives to help guide SR this year

9
Changes to the 2009-2010 QA Standards Process

• No numerical scores assigned
• Combined educational personnel qualifications and
  professional development requirements under
  curriculum and instruction
• Integrated community involvement, collaboration,
  and learning environment expectations into
  curriculum and instruction

10
Changes to the 2009-2010 QA Standards Process

• Incorporated specially designed instruction and
  related services indicator into curriculum and
  instruction
• New standard for exit transition

11
2009-10 QA Review Protocol

• Schedules are developed annually by program
  administrators from the 5 DJJ regions and the
  JJEEP Director
• On-site reviews will include a needs assessment
  to prepare for the shift in 2010-2011 to a more
  student outcome-oriented review
• Average three-day review

12
QA Review Protocol

• Self-reports due July 17 (updated in Jan.)
• JJEEP contacts school districts and programs to
  review self-report information the Wednesday
  prior to the review
• Reviewer reviews last years QA and TA reports
  and CAPs received

13
On-Site Protocol

• Education entrance meeting
• Explanation of process
• Identification of contact person
• Scheduling interviews, exit meeting
• Schedule agreed upon
• Tour of facility

14
QA Methodology

• Document Review
• Interviews
• Observations

15
Document review

• Review
• self-report documents
• student files (open and closed)
• curricular documents
• contract/cooperative agreement
• personnel documentation
• school improvement plan
• current SD comprehensive reading plan
• policies and procedures
• lesson plans/ grade books

16
Document review is a guide for interviews and
observations

• Interview Observe
• Students Classrooms
• Teachers Transition meetings
• Support staff Treatment Team
• School district staff
• Guidance/Transition Staff

17
Communication

• Daily debriefing with DJJ
• Daily debriefing with lead educator and other
  interested parties to discuss concerns, clarify
  questions, provide list of other information
  needed
• As needed, with contract manager and/or program
  administration
• Exit meeting on last day

18
QA Review Protocol and Rating Guidelines

• New methods/requirements will be piloted
• All benchmarks rated pass or fail
• Multiple data sources to evaluate quality
• Policy, document review, interviews, and
  observations
• Preponderance of evidence to determine whether
  the intent of the indicator is being met
• POLICY PRACTICE OUTCOME

19
Exemplary Programs

• Exemplary status will not be assigned during
  2009-10 review cycle
• Exemplary programs still asked to help provide
  assistance to low-performing programs

20
Needs Assessment - Purpose

• Pilot for the 2010-11 QA review cycle
• Focuses on program and student outcomes and the
  processes that impact student outcomes
• Collaborative process among JJEEP, the program,
  and the school district
• Needs Assessment findings will not impact QA
  results
• Results will be used to inform the 2010-2011 QA
  standards and methods

21
Needs Assessment - Purpose

• Some changes have already occurred
• Focus on transition, curriculum and instruction,
  and teacher qualifications
• Less emphasis on administrative requirements
• New self-report process and information

22
Needs Assessment - Content

• Program processes and decision making in
• Transition
• Curriculum and Instruction
• Staff Qualifications and Retention
• School District Oversight and Evaluation
• Possible outcomes for the 2010-2011 cycle
• Student gains based on entry/exit BASI results
• Reading gains based on PMRN or MIS
• Student progression rates
• Attendance rates (day treatment)

23
Needs Assessment - Conducting

• Full Reviews The Needs Assessment will be
  conducted on site as part of the QA review.
• To identify programs strengths and weaknesses
  and assist them in preparing for the 2010-2011 QA
  review cycle

24
Needs Assessment - Conducting

• Exemplary Reviews The Needs Assessment will be
  conducted through a Web based and/or telephone
  reporting system.
• To identify promising practices in Exemplary
  programs and inform the 2010-2011 QA standards
  and process

25
Needs Assessment - Findings

• Transition process and how it prepares individual
  students for return to school and/or employment
• Curriculum/instruction and how it provides
  learning opportunities for individual students

26
Needs Assessment - Findings

• Staff retention rate and retention strategies
• School district oversight and evaluation methods
  and the use of student performance measures to
  improve services

27
Needs Assessment - Findings

• Commendations Since programs will not be
  receiving numerical QA ratings in 2009-2010,
  Commendations noted for exceptional practices
  will be noted in the Needs Assessment
• Considerations Needs Assessment Considerations
  will be developed with input from program and/or
  school district administrators

28
Needs Assessment - Examples

• Process Questions
• What student progression opportunities does the
  program provide (i.e., credit recovery,
  competency-based programs, or online learning)?
• How is monitoring of student progress used to
  enhance instruction and services?
• How does the program prepare students for
  returning to school and/or the community?

29
Needs Assessment - Examples

• Outcomes
• What student outcomes are used to enhance
  instruction (i.e., entry/exit testing, reading
  progress monitoring, credits earned, student
  progression, diplomas earned, return to school
  rates)?
• What is the programs in-field teaching rate?
  Retention rate?
• What are the programs student gains or student
  progression outcomes?

30
Needs Assessment

• During the 2009-2010 review cycle, JJEEP
  reviewers will meet regularly to discuss Needs
  Assessment results
• Reviewers will solicit input regarding program
  and student outcomes for the 2010-2011 cycle and
  the processes that are important to achieving
  these outcomes

31
EDUCATIONAL STANDARD ONE ENTRY
TRANSITIONIndicator 1 Entry Transition Services

• Enrolling students in appropriate courses in the
  MIS upon entry based on re-entry educational
  goals, past records, entry assessment scores, and
  FCAT results (Courses must be grade appropriate
  and include reading, English/language arts, math,
  social studies, science, physical education
  P.E., and an approved career and education
  planning course, as needed, for student
  progression and high school graduation.)

32
EDUCATIONAL STANDARD ONE ENTRY
TRANSITIONIndicator 1 Entry Transition Services

• P.E. required for
• Grades K-5 150 minutes per week
• Grades 6-8 one period/day/semester/year
• Requirement waived if
• 1) The student is enrolled in a remedial
  course
• 2) The students parent indicates in writing
  that
• - The student is enrolled in another
  course
• offered as an option by the school
  district.
• - The student is participating in
  physical
• activities outside the school day
  which are
• equal or in excess of the
  requirement.
• Website nichole.wilder_at_fldoe.org

33
EDUCATIONAL STANDARD ONE ENTRY
TRANSITIONIndicator 1 Entry Transition Services

• For Detention Programs
• Intensive reading is required for
• long-term students who need
• reading remediation.
• New PE rule does not apply.

34
EDUCATIONAL STANDARD ONE ENTRY
TRANSITIONIndicator 1 Entry Transition Services

• Advising all students with regard
• to their individual
• Abilities and aptitudes
• Educational/occupational opportunities
• Diploma options
• Major areas of interest
• Post-secondary opportunities
• Educational status and progress

35
EDUCATIONAL STANDARD ONE ENTRY
TRANSITIONIndicator 2 Testing and Assessment

• Residential/Day Treatment
• Administer
• The BASI for reading, mathematics, and
  writing/language arts to students within 10
  school days of entry into the program to plan
  instruction

36
EDUCATIONAL STANDARD ONE ENTRY
TRANSITIONIndicator 2 Testing and Assessment

• Detention
• Administer
• Any assessment but the BASI
• Most commonly administered
• OWLS (Oral Written Language Scales)
• MBA (Mini-Battery of Achievement)
• WJ-R, WJ-III (Woodcock-Johnson Test of
• Achievement Revised or 3rd
  Edition)
• WIAT (Weschler Individual Achievement
  Test

37
EDUCATIONAL STANDARD ONE ENTRY
TRANSITIONIndicator 2 Testing and Assessment

• Detention continued
• BASIS (Basic Academic Skills Individual
  Screener
• STAR (Standardized Test for Assessment of
  Reading
• New Century
• WRAT (Wide Range Achievement Test)
• HAMAT (Hammill Multiability Achievement Test)
• Writing Prompt-FCAT writing sample

38
EDUCATIONAL STANDARD ONE ENTRY
TRANSITIONIndicator 2 Testing and Assessment

• Administer
• Career/technical aptitude assessments and/or
  career interest inventories to students within 10
  school days of entry and using the results to
  enhance employability and career/technical
  instruction

39
EDUCATIONAL STANDARD ONE ENTRY
TRANSITIONIndicator 2 Testing and Assessment

• Career/technical aptitude assessments
• Career Interest Survey
• Self-Directed Search (SDS)
• Career Cruiser interest assessment
• Choices Interest Profiler

40
EDUCATIONAL STANDARD ONE ENTRY
TRANSITIONIndicator 3 Student Planning

• Using entry assessment results, past records, and
  re-entry educational goals to develop age- and
  grade-appropriate IAPs for all non-ESE students
  that
• Are used to guide instruction
• Are developed w/in 15 school days
• Language change
• New Focus

41
EDUCATIONAL STANDARD ONE ENTRY
TRANSITIONIndicator 3 Student Planning

• (IAPs)
• Include specific, individualized, and measurable
  long-term goals for reading, writing/language
  arts, math, and career/technical areas
• Include at least two short-term instructional
  objectives per goal
• Identify remedial strategies
• Include a schedule for determining progress

42
EDUCATIONAL STANDARD ONE ENTRY
TRANSITIONIndicator 3 Student Planning

• Reviewing students progress toward achieving
  their IAP goals and objectives by an educational
  representative in treatment team or other formal
  meetings and revising IAPs when goals or
  short-term objectives are met

43
EDUCATIONAL STANDARD ONE ENTRY
TRANSITIONIndicator 3 Student Planning

• Convening IEP meetings and/or amending the plans,
  as needed, to include measurable annual IEP goals
  and short-term objectives or benchmarks that
  directly relate to students identified academic,
  behavioral, and/or functional deficiencies and
  needs
• Includes Detention long-term students

44
EDUCATIONAL STANDARD ONE ENTRY
TRANSITIONIndicator 3 Student Planning

• Reviewing students progress toward meeting their
  IEP goals and providing IEP progress reports to
  the parents as often as progress reports are sent
  home for all students
• Day Treatment
• Requesting conditional release students exit
  transition plans, and educational portfolio from
  previous residential commitment program and
  modifying transition goals as needed.
• Attendance Plan

45
EDUCATIONAL STANDARD TWO SERVICE
DELIVERYIndicator 4 Academic Curriculum
and Instruction

• Individualizing instruction based on the course
  descriptions for the courses in which students
  are enrolled and the current FSSS using a
  variety of instructional strategies based on
  students individual assessment results and
  progression needs to engage students in classroom
  learning activities
• (Merged 3 benchmarks
  from last year)
• New focus is on individual student needs
• Detention 22 days or more
• Less than 22 days literacy, social skills,
  remediation

46
Indicator 4 Academic Curriculum and
Instruction

• Implementing students individual academic plans
  (IAPs) and individual educational plans
  (IEPs) as written
• ESE services

47
Indicator 4 Academic Curriculum and
Instruction

• Hiring core academic teachers who have Florida
  professional or temporary teaching certification,
  a valid statement of eligibility, or proof of
  accepted application for teaching certification
• Moved
• Now specific to academics, reading, and career
  education

48
EDUCATIONAL STANDARD TWO SERVICE
DELIVERYIndicator 4 Academic Curriculum
and Instruction

• Ensuring that teachers participate in a beginning
  teacher program, as appropriate and that all
  teachers receive professional development
  training throughout the year or continuing
  education based on educational program needs,
  actual instructional assignments, the school
  improvement plan (SIP), and professional
  development plans
• Moved
• Now specific to academics, reading, career ed.

49
EDUCATIONAL STANDARD TWO SERVICE
DELIVERYIndicator 4 Academic Curriculum
and Instruction

• Providing adequate educational resources that
  include educational support staff,
  technology, and instructional materials
• Moved
• Now specific to academics, reading, and career
  education

50
EDUCATIONAL STANDARD TWO SERVICE
DELIVERYIndicator 4 Academic Curriculum
and Instruction

• Ensuring that students receive a minimum of 300
  minutes of daily instruction or the weekly
  equivalent
• Moved

51
EDUCATIONAL STANDARD TWO SERVICE
DELIVERYIndicator 5 Reading Curriculum
and Instruction

• Explicit reading instruction that
• Addresses the reading goals/objectives
• on students academic plans
• Provides more than one class period of
• reading intervention for disfluent
• secondary level students
• Includes curricula identified in the
• current school district comprehensive
• reading plan
• Is guided by progress monitoring and
• diagnostic reading assessment results
• Reading instruction is required in Detention
  under Indicator 3 Curriculum and Instruction

52
EDUCATIONAL STANDARD TWO SERVICE
DELIVERYIndicator 5 Reading Curriculum
and Instruction

• Reading teachers who have Florida professional or
  temporary teaching certification, a valid
  statement of eligibility, or proof of accepted
  application for teaching certification
• These qualifications for reading teachers applies
  to Detention Centers as well.

53
EDUCATIONAL STANDARD TWO SERVICE
DELIVERYIndicator 5 Reading Curriculum
and Instruction

• Reading teachers who participate in a beginning
  teacher program, as appropriate, and professional
  development training throughout the year or
  continuing education based on educational program
  needs, actual instructional assignments, the SIP,
  and professional development plans

54
EDUCATIONAL STANDARD TWO SERVICE
DELIVERYIndicator 5 Reading Curriculum
and Instruction

• Adequate educational resources that include
  educational support staff, technology, and
  instructional materials

55
EDUCATIONAL STANDARD TWO SERVICE
DELIVERYIndicator 6 Employability Career
Curriculum and Instruction

• (Not required in Detention programs)
• Post-secondary, employability, social, and life
  skills instruction,
• career exploration, and/or hands-on technical
  training for students who have high school
  diplomas or the equivalent

56
EDUCATIONAL STANDARD TWO SERVICE
DELIVERYIndicator 6 Employability Career
Curriculum and Instruction

• Career exploration opportunities and resources
  that address the required career
  competencies in middle school career instruction

57
EDUCATIONAL STANDARD TWO SERVICE
DELIVERYIndicator 6 Employability Career
Curriculum and Instruction

• Teachers who have teaching certification or
  documented approval to teach, according to the
  school board policy for use of noncertified
  instructional personnel based on documented
  expert knowledge/skill
• Moved
• Now specific to employability/career ed.

58
EDUCATIONAL STANDARD TWO SERVICE
DELIVERYIndicator 6 Employability Career
Curriculum and Instruction

• Teachers who attend professional development
  training throughout the year or continuing
  education based on educational program needs,
  actual instructional assignments, the SIP,
  professional development plans, and/or annual
  teacher evaluations teachers who participate in
  a beginning teacher program, as appropriate
• Moved
• Now specific to employability/career ed.

59
EDUCATIONAL STANDARD TWO SERVICE
DELIVERYIndicator 6 Employability Career
Curriculum and Instruction

• Adequate educational resources and support staff,
  technology, and instructional materials
• Moved
• Now specific to employability/career ed.

60
EDUCATIONAL STANDARD TWO SERVICE
DELIVERYIndicator 6 Employability Career
Curriculum and Instruction

• Employability, social, and life skills courses
  offered for credit or
• curricula that are integrated into other
  courses that are based on state and school board
  standards instruction that follows the course
• descriptions and individualized course work
  based on students career interests

61
EDUCATIONAL STANDARD TWO SERVICE
DELIVERYIndicator 6 Employability Career
Curriculum and Instruction

• Type 2 Programs
• A broad scope of career exploration and
  prerequisite skill training based
• on students' interests and/or aptitudes in
  career courses offered for credit or via
  activities that are integrated into other courses
  for credit
• Merged two benchmarks

62
EDUCATIONAL STANDARD TWO SERVICE
DELIVERYIndicator 6 Employability Career
Curriculum and Instruction

• Type 3 Programs
• Appropriate access to hands-on career/technical
  competencies and prerequisite training
  required for entry into a specific occupation in
• career courses offered for credit or
  activities that are integrated into other
  courses for credit
• Merged two benchmarks

63
EDUCATIONAL STANDARD THREE EXIT
TRANSITIONIndicator 7 Exit Transition Services

• Administering the BASI for reading,
  writing/language arts, and mathematics
• to all exiting students who have been in the
  program for 45 or more school days and
  documenting transmittal of entry/exit BASI
  standard scores and growth scale values to the
  school district
• for MIS reporting or the direct input of the
  scores into the MIS

64
EDUCATIONAL STANDARD THREE EXIT
TRANSITIONIndicator 7 Exit Transition Services

• Developing (for 8th graders) and/or reviewing
  (for 9th graders)
• electronic Personalized Education Plans
  (ePEPs) based on students
• re-entry educational and career goals via the
  online Florida Academic Counseling and Tracking
  for Students (FACTS) system at FACTS.org

65
EDUCATIONAL STANDARD THREE EXIT
TRANSITIONIndicator 7 Exit Transition Services

• Developing exit transition plans that include
  students desired diploma options, anticipated
  next educational placements, post-release
  educational plans, aftercare providers, job or
• career/technical training plans, and parties
  responsible for implementing the plans

66
EDUCATIONAL STANDARD THREE EXIT
TRANSITIONIndicator 7 Exit Transition Services

• Notifying the transition contacts in the
  receiving school districts at least one week
  prior to students' scheduled release from the
  program and documenting transmittal of students
  educational exit packets to the transition
  contacts in the receiving school districts prior
  to their exit

67
EDUCATIONAL STANDARD THREE EXIT
TRANSITIONIndicator 7 Exit Transition Services

• Exit packets shall include, at a minimum,
  cumulative transcripts reporting credits earned
  prior to and during commitment, school district
  withdrawal forms with grades in progress, current
  IEPs/IAPs, exit plans, and career education
  certificates.

68
EDUCATIONAL STANDARD THREE EXIT
TRANSITIONIndicator 7 Exit Transition Services

• Unique to Day Treatment
• Exit plans do not include aftercare providers
• Notify the next educational placements or
  receiving school districts transition contacts
  at least one week prior to the students
  scheduled release
• Some exit information accessible on MIS

69
EDUCATIONAL STANDARD THREE EXIT
TRANSITIONIndicator 7 Exit Transition Services

• Unique to Detention
• Participation of educational representative in
  detention hearings
• Transmittal of in-county students school
  district withdrawal forms with numerical grades
  in progress to their next educational placements
  at the time of exit

70
EDUCATIONAL STANDARD THREE EXIT
TRANSITIONIndicator 7 Exit Transition Services

• Detention contd.
• Transmittal of out-of-county educational
  records to the next educational placements,
  transportation staff, or JPOs at the time of exit

71
EDUCATIONAL STANDARD FOUR CONTRACT
MANAGEMENTIndicator 8 School District
Monitoring, Accountability, and Evaluation

• The school district ensures that the program
• Submits its electronic bi-annual self-reports and
  required documents in a timely manner

72
EDUCATIONAL STANDARD FOUR CONTRACT
MANAGEMENTIndicator 8 School District
Monitoring, Accountability, and Evaluation

• Accurately reports all student data under the
  programs individual school number, including
  grades, total credits earned student progression,
  certificates, entry and withdrawal dates,
  withdrawal codes, entry/exit BASI growth scale
  values and standard scores, attendance, reading
  progress monitoring scores, and diplomas earned
  in the school district MIS (Reading progress
  monitoring scores may be reported in the Progress
  Monitoring and Reporting Network PMRN or the
  MIS.)

73
EDUCATIONAL STANDARD FOUR CONTRACT
MANAGEMENTIndicator 8 School District
Monitoring, Accountability, and Evaluation

• Participates in the Adequate Yearly Progress
  (AYP) process and
• accurately reports its statewide assessment
  participation rate data
• Be sure to provide the program the opportunity to
  correct the enrollment file
• Not a requirement in detention standards

74
EDUCATIONAL STANDARD FOUR CONTRACT
MANAGEMENTIndicator 8 School District
Monitoring, Accountability, and Evaluation

• Receives appropriate oversight and assistance by
  the contract manager that includes conducting and
  documenting annual evaluations of the educational
  program, monitoring implementation of
• the school district comprehensive reading
  plan, and ensuring that the terms of the
  cooperative agreement with the DJJ and the
  contract with the private educational
• provider (if applicable) are followed

75
After the On-Site Visit

• Reviewer discusses findings with JJEEP staff QA
  Director
• Reviewer writes QA review report
• Buddy Reviews are conducted
• QA Director reviews/approves report
• Report is edited and revised
• Report is submitted to the DOE for final approval
  and mailing

76
Corrective Action Process Residential Programs

• Programs that fail one or more of Standards 1,
  2, or 3 will receive a corrective action plan
  (CAP)
• Failing 3 or more benchmarks in Standard 1 Entry
  Transition will result in failing the overall
  standard

77
CAPs Residential Programs

• Failing 7 or more benchmarks in Standard
  2Service Delivery will result in failing the
  overall standard
• Failing 2 or more benchmarks in Standard 3Exit
  Transition will result in failing the overall
  standard

78
Program CAPs
79
School District CAPs

• School districts that fail Standard 4 for two or
  more consecutive years will receive a CAP
• Failing 2 or more benchmarks in Standard
  4Contract Management will result in failing the
  overall standard

80
School District CAPs
81
CAP Completion

• Establish a corrective action team
• Develop the action plan
• Complete and return CAP to QA Director (within 90
  days)
• Ensure superintendent signs the implementation
  page AFTER the CAP has been implemented

82
Technical Assistance (TA)

• On-site TA during QA reviews
• Through recommendations on QA reports
• Networking
• E-mail, telephone, fax
• Web site resources
• By request
• CAP follow-up
• New program- mock QA review

83
TA and CAPs (2008-2009)

• 11 total CAPS
• 4 of 11 are school district only CAPs
• 3 of 11 are program and school district CAPs
• Remaining four are program only CAPs
• 1 program with a CAP closed
• All 10 programs/school districts have or will
  receive appropriate TA based on need
• 5 programs are on DOE Intervention list

84
Research Brought to Life
Contact Us for Information

• JJEEP
• 325 John Knox Road
• Bldg. L, Suite 102
• Tallahassee, FL 32303
• [phone]

Visit our Web site for information on research,
standards, technical assistance documents, and
links related to juvenile justice
education www.criminologycenter.fsu.edu/jjeep