Los Angeles Unified School District Local District E Instructional Support Services Standards-Based Education Performance Assignment Training Session for Analytic Scoring 2003-04

1
Los Angeles Unified School DistrictLocal
District EInstructional Support
ServicesStandards-Based Education Performance
Assignment Training Session for Analytic
Scoring2003-04
2
PERFORMANCE ASSIGNMENTELEMENTARY AND MIDDLE
SCHOOLLOGISTICS

• Data Results
• Changes
• Testing Schedule
• Rater Reliability

3
PERFORMANCE ASSIGNMENTDATA RESULTS

• In grades 2-9, scores increased 2 (total
  students tested 276,319)
• 40 scored proficient or above in English
  (107,497)
• 42 scored proficient or above in Spanish
  (3,181)
• 29 improved their performance level (53,608)

4
PERFORMANCE ASSIGNMENTDATA RESULTS

• Responses from Student and Teacher Questionnaire
  validate the correlation between students
  opportunity to learn and their Performance
  Assignment scores.
• Positive relationship between higher frequency of
  quality reading and writing experiences and
  higher scores.
• Students who have less practice with reading and
  writing score lower.

5
PERFORMANCE ASSIGNMENT CHANGES FOR 2003-04

• Elementary Grades
• Performance Assessment aligns with OC
• Third periodic assessment
• Middle School
• Third periodic assessment
• SBACs Literacy Coaches work together
• Analytic scoring replaces holistic scoring

6
PERFORMANCE ASSIGNMENTTESTING SCHEDULE

• Elementary
• Third Periodic Assessment
• Middle School
• Third Periodic Assessment
• Single Track February 23-27
• Three Track A March 29 - April 3 B
  C January 26-30
• Four Track A B March 1-5 C D January
  26-30

7
PERFORMANCE ASSIGNMENTRATER RELIABILITY

• Target Percentage
• 2001-02 50-60
• 2002-03 55-65
• 2003-04 60-70

8
PERFORMANCE ASSIGNMENTRATER RELIABILITY
SCHOOLS 70-79 RATER AGREEMENT

• Allesandro Coop. Learning Magnet (grade 4)
• Alta Loma (grade 2)
• Arlington Heights (grade 4)
• Atwater (grade 3)
• Buchanan (grade 3)
• Bushnell Way (grade 4)
• Clifford (grade 4)

9
PERFORMANCE ASSIGNMENTRATER RELIABILITY
SCHOOLS 70-79 RATER AGREEMENT

• Dayton Heights (grade 2)
• Fletcher Drive (grades 2 and 3)
• Garvanza (grade 3)
• Grant (grade 4)
• Hobart (grade 3)
• Ivanhoe (grade 4)
• Micheltorena (grade 4)

10
PERFORMANCE ASSIGNMENTRATER RELIABILITY
SCHOOLS 70-79 RATER AGREEMENT

• Monte Vista (grade 5)
• Pio Pico (grade 3)
• Ramona (grades 2 and 3)
• San Pascual (grade 3)
• Selma (grade 4)
• Wilshire Crest (grade 3)
• Wilton Place (grade 3)

11
PERFORMANCE ASSIGNMENTRATER RELIABILITY
SCHOOLS 80-89 RATER AGREEMENT

• Atwater (grade 2)
• Franklin (grade 3)
• Gardner (grade 5)
• Grant (grade 3)
• Selma (grade 3)
• Wilton Place (grade 5)

12
PERFORMANCE ASSIGNMENTRATER RELIABILITY
SCHOOLS 90-100 RATER AGREEMENT

• Aragon (grade 2)
• Arroyo Seco Museum Sci. Magnet (grade 5)
• Cheremoya (grade 4)
• Dorris Place (grade 4)
• Eagle Rock G/HA Elem. Magnet (grade 3)

13
PERFORMANCE ASSIGNMENTRATER RELIABILITY
SCHOOLS 90-100 RATER AGREEMENT

• Micheltorena (grade 3)
• Pio Pico (grade 2)
• Queen Anne (grades 4 and 5)
• Selma (grade 2)
• Rater Reliability Data from Use of Periodic
  Assessments to Promote Literacy Achievement

14
PERFORMANCE ASSIGNMENTRATER RELIABILITY
SECONDARY SCHOOLS

• Eagle Rock (gd. 8) 74
• Pio Pico (gd. 7) 58
• Irving (gd. 8) 57
• Burbank (gd. 6) 54
• Burroughs (gd. 8) 51
• LeConte (gd. 6) 51
• King (gd. 6) 50
• Mount Vernon (gd. 7/8) 47

15
PERFORMANCE ASSIGNMENT Analytic Scoring for
Elementary
16
SCORING Holistic Scoring

• Procedure which yields a single score based on
  an overall impression of students performance

17
SCORINGAnalytic Scoring

• Procedure which yields multiple scores based on
  two or more separate scales for evaluating
  performance on selected dimensions or traits
  each trait receives a separate score.

18
SCORING
ANALYTICAL
HOLISTIC
analyzes separate aspects of student writing
looks at the entire writing piece
a form of assessment that looks at writing
writing is read through for each score
captures an overall picture of students
writing
scores are given for each analytic category
both use rubrics
both can be indicators for teachers
next steps
gives one score
uses multiple scales
Provides more detailed info for improving
instruc- tion and communicating with
students
a single scale
is often more efficient
19
SCORINGAnalytic Scoring for Unit 3

• Applications (genre) 45
• Writing Strategies (traits) 45
• WOEL Writing and Oral English Language
  (Conventions) 10

20
PREPARING FOR A SCORING TRAINING SESSIONScoring
Handbook (Part I)Major Responsibilities

• Determine who will attend
• Schedule training session
• Find appropriate location
• Prepare training packets

21
PREPARING FOR A SCORING TRAINING SESSIONScoring
Handbook (Part I)Materials

• Training packet (grade level)
• Prompt and rubric
• Set of anchor and training papers
• Scoring Handbook
• Assessment form for recording student scores
  (Literacy Coach)

22
PREPARING FOR A SCORING TRAINING SESSIONScoring
Handbook (Part I) Materials

• Training Paper Score Sheet
• Rater Agreement Summary Sheet
• Calculator
• Training Papers Key

23
PREPARING FOR A SCORING TRAINING SESSIONScoring
Handbook (Part I) Time

• 1.5 to 2.5 hours

24
CONDUCTING A SCORING TRAINING SESSIONScoring
Handbook (Part II)

• Open Court Writing
• Unit 1
• Scoring Materials
• Grade 3

25
CONDUCTING A SCORING TRAINING SESSIONScoring
Handbook (Part II)Step 1

• Read the task (prompt) carefully
• Think about the stories you have read in the
  Friendship Unit and what it is like to be a
  friend. Write a story about your best friend and
  tell about a special time or memorable event that
  you have shared as friends. Be sure to use
  concrete sensory details to describe where you
  were. Describe what was special to you and your
  friend and tell why this special time was
  something you will always remember.

26
CONDUCTING A SCORING TRAINING SESSIONScoring
Handbook (Part II)Step 2

• Review the rubric for Applications
• Working with a partner, carefully read aloud the
  description of each score point in the rubric.
  Start with level 3/Proficient.
• 5 minutes

27
CONDUCTING A SCORING TRAINING SESSIONScoring
Handbook (Part II)Step 3

• Read and discuss anchor paper focusing only on
  Applications
• Anchor papers represent the lowest performance
  level
• In packet, there are anchor papers for 2, 3, and 4

28
CONDUCTING A SCORING TRAINING SESSIONScoring
Handbook (Part II)Step 3

• Focusing only on Applications
• Read Anchor 3 paper aloud. Briefly discuss
  the qualities of the paper, referring to each
  component of the rubric. Describe why it
  received the score it did. Repeat for Achors 2
  and 4. Discuss distinctions in levels referring
  to the rubric as needed.
• 10 minutes

29
CONDUCTING A SCORING TRAINING SESSIONScoring
Handbook (Part II)Step 4

• Repeat the same process from Steps 2 and 3
  focusing only on the Writing Strategies category.
• Step 2 review the rubric
• Step 3 read and discuss anchor papers
• 10 minutes

30
CONDUCTING A SCORING TRAINING SESSIONScoring
Handbook (Part II)

• Step 5
• Repeat the same process from Steps 2 and 3
  focusing only on the WOEL (Conventions)
  category.
• Step 2 review the rubric
• Step 3 read and discuss anchor papers
• 5 minutes

31
CONDUCTING A SCORING TRAINING SESSIONScoring
Handbook (Part II)

• Step 6
• Score training papers
• Training papers already have assigned scores by
  expert raters
• Goal is to obtain at least 70 agreement

32
CONDUCTING A SCORING TRAINING SESSIONScoring
Handbook (Part II)

• Step 6 Score Training Papers
• A. Score Training Paper A
• Read Training Paper A. Compare each response to
  the set of anchor papers and rubric criteria for
  each category. Use the Training Paper Score
  Sheet (at the bottom of your packet) to record
  scores.
• 5 minutes

33
CONDUCTING A SCORING TRAINING SESSIONScoring
Handbook (Part II)

• Step 6 Score Training Papers
• A. Score Training Paper A
• You should have 3 scores for paper A. Using the
  Rater Agreement Summary Sheet, record the
  frequency of each score (Applications, Writing
  Strategies, Conventions). In each category,
  circle the highest number.
• 5 minutes

34
CONDUCTING A SCORING TRAINING SESSIONScoring
Handbook (Part II)

• Step 6 Score Training Papers
• A. Score Training Paper A
• Overall Score 2

35
CONDUCTING A SCORING TRAINING SESSIONScoring
Handbook (Part II)

• Step 6 Score Training Papers
• B. Score Training Paper B
• (use same procedures as before)
• Read Training Paper B. Compare each response to
  the set of anchor papers and rubric criteria for
  each category. Use the Training Paper Score
  Sheet (at the bottom of your packet) to record
  scores.
• 5 minutes

36
CONDUCTING A SCORING TRAINING SESSIONScoring
Handbook (Part II)

• Step 6 Score Training Papers
• B. Score Training Paper B
• (use same procedures as before)
• Read Training Paper B. Compare each response to
  the set of anchor papers and rubric criteria for
  each category. Use the Training Paper Score
  Sheet (at the bottom of your packet) to record
  scores.
• 5 minutes

37
CONDUCTING A SCORING TRAINING SESSIONScoring
Handbook (Part II)

• Step 6 Score Training Papers
• B. Score Training Paper B
• Overall Score 3

38
CONDUCTING A SCORING TRAINING SESSIONScoring
Handbook (Part II)

• Step 6 Score Training Papers
• C. Score Training Paper C D
• Score training papers C and D following the same
  procedures used to score training papers A and B.
• 5 minutes

39
CONDUCTING A SCORING TRAINING SESSIONScoring
Handbook (Part II)

• Step 6 Score Training Papers
• C. Score Training Paper C D
• Training Paper C Score3
• Training Paper D Score2

40
CONDUCTING A SCORING TRAINING SESSIONScoring
Handbook (Part II)

• Step 6 Score Training Papers
• D. Check rater agreement
• Using the Rater Agreement Summary Sheet, and the
  sample page WS-4 in your Scoring Handbook, follow
  the directions to calculate rater agreement.
• 10 minutes

41
CONDUCTING A SCORING TRAINING SESSIONScoring
Handbook (Part II)

• Step 7 Student Writing Samples
• Score student writing samples and record on the
  Classroom Assessment Summary Form provided by the
  Literacy Coach.