Certificate of Mastery

1
Certificate of Mastery

• House Education Committee
• January 16, 2003
• Dr. Terry Bergeson
• State Superintendent of Public Instruction

2
Purpose of Education Reform
to provide students with the opportunity to
become responsible citizens, to contribute to
their own economic well-being, and to that of
their families and communities, and to enjoy
productive and satisfying lives. - Basic
Education Act (Goal)
3
State Learning Goals

• Read with comprehension, write with skill, and
  communicate effectively and responsibly in a
  variety of ways and settings
• Know and apply the core concepts and principles
  of mathematics social, physical, and life
  sciences civics and history geography arts
  and health and fitness
• Think analytically, logically, and creatively,
  and to integrate experience and knowledge to form
  reasoned judgments and solve problems and
• Understand the importance of work and how
  performance, effort, and decisions directly
  affect future career and educational
  opportunities.

4
New Graduation Requirements to Accomplish Our
Goals(effective for graduating class of 2008)

• 19 credits including English, mathematics,
  social studies, science, health and fitness,
  visual or performing arts, occupational education
• High School Education Plan
• Culminating Project
• Certificate of Mastery
• Other District Requirements

5
What HB 1209 says aboutthe Certificate of
Mastery

• Once State Board of Education finds the 10th
  grade WASL sufficiently valid and reliable as a
  graduation requirement, successful completion
  shall lead to a Certificate of Mastery (CoM).
• Most students shall obtain a CoM at about the age
  of 16.
• CoM is evidence that student has mastered the
  EALRS
• COM required for graduation but not the only
  requirement

6
HB 1209 EALRs and assessments serve multiple
purposes

• Goal
• Establish student expectations at
  internationally competitive levels.
• System
• A tool to evaluate instructional practices.
• Initiate appropriate educational support for
  students who have not mastered the EALRs.
• Students
• To assess individual student skills/knowledge for
  graduation purposes.

7
Student Performance Where We are Now
Some Special Education
Some English Language Learners (ELL)
Have demonstrated the skills and knowledge
Other Students
Have NOT demonstrated the skills and knowledge
8
Student Performance Where We Need to Go
Have NOT demonstrated the skills and knowledge
Some Special Education
Some English Language Learners (ELL)
Other
Demonstrated by alternative means
Have demonstrated the skills and knowledge
9
Ensuring Fair and Successful Implementation of
CoM as a Graduation Requirement

• Reliability and Validity of WASL Scores
• Reviewing Content and Performance Standards
• Designing a Composite (Blended) CoM Model
• Retake Opportunities
• Accelerated Learning Plans with Instruction
• Other Opportunity to Learn Issues
• Guidelines for Special Education Students and
  English Language Learners
• Alternate Method for Demonstrating Proficiency
• Implementation of Meaningful Incentives

10
1) Three questions regarding evidence for
reliability of scores

• Are judges consistent in applying rubrics to
  short-answer and extended-response items?
• Would students earn same total score no matter
  which judge assigns scores to their work?
• Do students perform consistently across the items
  in the test?

11
Evidence for Reliability of WASL Scores

• High level of agreement between raters
• Likelihood of students getting the same total
  score regardless of the rater is extremely high
  (r .93 to .97)
• Student internal consistency is high for math and
  reading (r .90)
• Student internal consistency in writing is
  acceptable if students can take tests more than
  once (r .75 to .80)
• Student internal consistency in listening will
  be acceptable beginning in 2003

12
Three questions regarding evidence for validity
of scores

• Do the items on tests measure the Essential
  Academic Learning Requirements?
• Are scores on WASL tests well correlated with
  scores on other tests that measure the same
  content?
• Do patterns of scores suggest that mathematics,
  reading, and writing tests require different
  underlying knowledge and skills?

13
Evidence for the Validity of Grade 10 WASL Scores

• Ongoing test development process requires regular
  checking of items match to EALRs.
• External review by Stanford Research Institute
  found that mathematics items match EALRs.
• Statistical data (correlations with ITED scores
  and patterns of students responses on ITED
  subtests and WASL strands) provide solid evidence
  for validity of scores.

14
2) Reviewing Content and Performance Standards

• Grade 7 and 10 mathematics test items aligned to
  EALRS (SRI)
• 10th grade mathematics test is appropriate level
  of difficulty for 10th graders (SRI)
• Standard setting procedures were state of the
  art in 1999 (SRI)
• Standard setters must consider high school
  graduation
• Should incorporate latest research on standard
  setting prior to graduation requirement

15
3) Designing a Composite (Blended) Model

• Current model Conjunctive
• Student must achieve a score of 400 in four areas
  (reading, mathematics, writing, listening) to
  earn CoM
• Possible alternative Composite (Blended) Model
• Higher scores within the standard error of
  measurement (SEM) may compensate for lower scores
  within the standard error of measurement
• Must achieve a composite score of 1600 across all
  four areas (reading, mathematics, writing,
  listening)
• Allows for slight variations in scores while
  still holding students to meeting standards on
  the whole

16
Standard Error of Measurement (SEM)

• All assessment has measurement error.
• Positive error happens when students guess or
  copy and get higher scores than they should.
• Negative error happens when students make
  careless errors and get lower scores than they
  should.
• SEM allows us to decide how confident we are in
  students scores.
• If a test has an SEM of 8, we can expect that 70
  of the students with a true score of 400 will
  have an observed score between 392 and 408.
• Composite model allows us to give students
  benefit of the doubt.

17
The Current Model (Conjunctive)
460
440
420
Standard
400
380
360
340
320
300
Reading
Writing
Listening
Mathematics
Student did not meet writing standardCertificate
of Mastery NOT awarded
18
Composite (Blended) Model A Possible Alternative
430
420
410
403
408
Standard error of measurement
400
396
396
395
390
380
370
360
350
Reading
Writing
Listening
Mathematics
Students scores are all within the standard
error of measurement of 400 with an average
across all four areas greater than or equal to
400 CoM awarded.
19
Comparison of Conjunctive and Composite (Blended)
Models
On the WASL scales, a score of 400 meets
standard. For the blended model, a composite
score of 1600 is required to meet
standard. Students lowest scores must be no
lower than the bottom of the standard error band
(to be determined) students points above the
standard error band may not contribute to the
composite score of 1600.
20
Composite (Blended) Model A Possible Alternative
430
426
420
410
Standard error of measurement
400
390
396
393
396
385
380
370
360
350
Reading
Writing
Listening
Mathematics
Students composite score equals 1600. Very high
Listening Score does not compensate for very low
scores in reading, writing and math writing
score falls below SEM CoM not awarded
21
Composite (Blended) Model A Possible Alternative
430
422
420
410
Standard error of measurement
400
390
396
393
389
396
380
370
360
350
Reading
Writing
Listening
Mathematics
Students composite score equals 1600. Very high
Listening Score does not compensate for very low
scores in writing and math CoM not awarded
22
Benefits of a Blended Model

• Recognizes that individuals have different
  strengths
• Recognizes the possibility of minor errors in
  test scores
• Overall standards for knowledge and skills are
  maintained
• Graduation will be dependent on a composite of
  WASL knowledge and skills
• Similar approach used in other high-stakes tests
  (e.g., SAT and GRE)

23
4) Retake Opportunities

• Students can bank points until they their
  average scale score meets or exceeds a 400
• State will provide at least 4 opportunities to
  retake the WASL or any part of the WASL
• Students would not be required to retake area(s)
  where they have met mastery
• Opportunities for retakes will be offered twice
  each year after a students sophomore year
• Students get the benefit of the doubt can keep
  the highest score

24
5) Accelerated Learning Plans with Instruction

• Districts should provide accelerated instruction
  (i.e., remediation) for students not meeting
  standards following 7th and 10th grade WASL,
  respectively.
• Plans should focus timely interventions in areas
  assessed in which students did not meet
  standards.
• OSPI will make best practices on specific
  interventions available statewide.

25
6) Other Opportunity to Learn Issues

• Ample Prior Notice to Students and Their Families
• EALRs
• CoM as Graduation Requirement
• Aligned Curriculum
• EALRs as Commonly Understood Standard
• WASL assesses EALRs

26
7) Guidelines for Special Education and ELL
Students

• Complex Issues
• Current SBE rules ALL students must earn CoM to
  earn regular diploma
• Rite of Passage vs. Skill Diploma
• Task force will continue to meet to develop
  guidelines and identify best practices

27
8) Alternate Methods for Demonstrating
Proficiency

• Alternate methods are for students judged to be
  at or above standard by the district but not on
  the WASL students must attempt WASL twice.
• Portfolio of students work that demonstrates
  same standards measured on WASL
• Semi-standardized classroom-based performance
  assessments used to collect evidence
• Multiple-sources of evidence required for each
  tested area
• Portfolio submitted with assurance by district
  that portfolio includes only students own work
• Collection of work evaluated as met or did not
  meet standard for each tested area at state level

28
9) Possible Incentives before the COM is a
Graduation Requirement
Graduation Requirement
Apprenticeship consideration
4 year admission requirement
Running Start requirement
4 year admission consideration
CC/TC Placement
Scholarships
Business Incentives
Transcripts
Time
For recent public high school graduates
29
Current K-12/Higher Ed Collaboration on
Assessments and Incentives

• UW Study - Relationship between WASL scores and
  Performance in the First Year of College
  (WASL/SAT/HSGPA/First Year GPA)
• OSPI/SBCTC Study Relationship Between WASL and
  Placement Tests Used at Community and Technical
  Colleges (WASL/ASSET/COMPASS/Accuplacer)
• HECB Promise Scholarship Study
• Discussions with SBCTC, Council of Presidents,
  and HECB

30
Once students take the WASL seriously, scores
will improve

• In Spring 2002, approximately 30 of 10th graders
  taking the WASL met all four standards.
• Indiana
• First administration 54 met standards
• After multiple retakes 86 met standards
• Final graduation rate 89.5 (3.5 graduated
  through alternative routes)
• Massachusetts (Initial administration of the
  assessments)
• For class before it was a grad requirement 51
  met standards
• For class after it became a grad requirement 73
  met standards

31
Immediate Critical Issues

• Funding Retake Opportunities
• Timely Decisions for Ninth Grade Class of 2004
  (Graduating Class of 2008)