Progress Monitoring in Written Expression Within and Across Grades: Using CurriculumBased Measures t

1
Progress Monitoring in Written Expression Within
and Across GradesUsing Curriculum-Based
Measures to Make Data-Based Decisions

• Presented by Kristen McMaster,
• University of Minnesota
• to the Pennsylvania Department of Education
  Conference
• Wednesday, April 15, 2009

2
Acknowledgements

• Heather Campbell
• Stan Deno
• Xiaoqing Du
• Troy Ellis
• Christine Espin
• David Parker
• Anna-Lind Petursdottir
• Sue Rose
• Teri Wallace
• Seungsoo Yeo
• Teachers and Students in Minneapolis Public
  Schools
• Research Institute on Progress Monitoring (RIPM)
  (Grant No. H324H030003) awarded by the Office of
  Special Education Programs (OSEP) to the
  Institute on Community Integration in
  collaboration with the Department of Educational
  Psychology, College of Education and Human
  Development, University of Minnesota

3
Objectives

• Identify Curriculum-Based Measures in Written
  Expression (CBM-W) that are appropriate within
  and across elementary and secondary grades.
• Learn to administer, score, and graph CBM-W.
• Discuss appropriate uses of CBM-W for data-based
  instructional decision-making.

4
Presentation Overview

• What is CBM and how is it used?
• What does the research say about CBM-W?
• How do I administer and score CBM-W?
• How do I use CBM to make instructional decisions?

5
What is CBM and how is it used?
6
What is CBM?

• A simple, efficient set of procedures for
  repeated measurement of student growth over time.
• Designed to help teachers make instructional
  decisions.

7
Criteria for Curriculum-Based Measures

• Technically adequate
• Reliable
• Valid (e.g., related to important constructs,
  such as overall writing proficiency, and to
  important educational outcomes, such as high
  stakes tests)
• Sensitive to growth
• Practical
• Easy to use
• Brief
• Easy to understand
• Easy to teach
• Repeatable
• Durable
• Appropriate for range of age/skill levels

8
How is CBM used to monitor progress?
1. Determine current level of performance
2. Set individual annual goal -- tied
to general education standards
4. Evaluate and modify instruction on basis of
rate of progress towards goal
3. Monitor individual progress towards goal.
Valid and reliable indicators of performance
9
Why monitor progress?

• To detect relatively small but important gains in
  short periods of time
• To monitor rate of growth as well as level of
  performance
• To deliver effective efficient instruction
• Intense
• Relentless
• Part of identification process for special
  education

10
Progress Monitoring Case Study
11
Jaime

• Jaime is a 16-year old tenth grader who has yet
  to pass the statewide graduation test in writing.
  
• He has been placed in a standards English class
  in an effort to help him pass the high-stakes
  tests.
• His English teacher decides to monitor his
  writing progress using CBM.
• She will use this progress monitoring data to
  decide whether her instruction is effective.

12
Things to consider

• What type of CBM measure should Jaimes teacher
  select?
• How should she administer and score Jaimes CBM
  performance?
• How should she document his progress?
• How should she use the data to make instructional
  decisions?

13
What does research say about Cbm-W?
14
CBM Research in Written Expression

• 1970s-80s
• University of Minnesota Institute for Research on
  Learning Disabilities (IRLD)
• Elementary
• 1990s to present
• Extensions of IRLD work
• Research Institute on Progress Monitoring (RIPM
  2004-2009)
• Elementary Secondary
• Beginning Writers

15
What does research tell us to date?

• What we know

• What we still need to learn

• CBM-W generally produces reliable and
  criterion-valid scores
• CBM-W appears to be sensitive to growth over time

• We are still examining characteristics of slopes
  produced from progress-monitoring data
• We are still examining effects of teachers use
  of the data to make instructional decisions

16
General Findings Across Grades
Full Table with References is in Appendix
17
Three Cautions

• Selection of measures will depend on the
  characteristics of the students in the class.
• To measure performance and progress in writing,
  teachers must think carefully about how they or
  their district define good writing.
• Because little work has been done on the use of
  the measures as progress measures, some
  information, for example typical growth rates,
  may need to be determined locally.
• TIP
• Continued research is being conducted on CBM-W.
  It is best to keep abreast of the continued
  development of CBM-W measures.

18
Sample measures
19
Sample Picture Prompt
(from RIPM CBM Writing Studies, Grades 3, 5, 7)
20
Sample Narrative Prompt
(from RIPM CBM Writing Studies, Grades 3, 5, 7)
21
Sample Expository Prompt
(from RIPM CBM Writing Studies, Grades 3, 5, 7)
22
Sample Story Prompt(from RIPM CBM Writing
Studies, Grade 1)

• Translation
• It was my birthday party. My friends came over.
  It was very fun. We went to the hotel too. We
  went swimming too. We had so much fun that

23
Sample Sentence Copy Prompt(from RIPM CBM
Writing Studies, Grade 1)
24
Sample Picture-Word Prompt(from RIPM CBM Writing
Studies, Grade 1)
25
How do I administer, score, and monitor progress
with cbm-w?
26
Select Materials

• Select type of measure based on
• Grade level
• Purpose of progress monitoring
• What will provide YOU with the most meaningful
  information.
• Determine the number of prompts needed.
• Create prompts
• That tap background knowledge that your students
  are likely to have.
• That end mid-sentence (if narrative/story
  starters).
• That include a tell why component (if
  expository).
• Print prompts at top of pages that students will
  write on.
• TIP
• Keep materials and procedures consistent
  throughout the monitoring period.

27
Sample Prompts

• Narrative (Story Starters)

• Expository

• One night I had a strange dream about
• It was the last day of school so I decided to
• One day, when I got home from school
• One day my friend told me the strangest story
• I was walking home when I found a 100 bill on
  the sidewalk and
• One day I went to school but nobody was there
  except me, so I
• I was on my way home from school and
• It was the first day of school and
• I was talking to my friends when, all of a
  sudden
• One day I woke up and was invisible and
• One day I found the most interesting thing and
• One summer I went on a trip and
• I was walking down the street when I saw...

• Describe a game you like to play and tell why you
  like it.
• Describe your favorite day of the week and tell
  why you like it.
• Describe your favorite time of the year and tell
  why you like it.
• Describe your favorite thing to do and tell why
  you like it.
• Describe a place you like to go and tell why you
  like to go there.
• Describe the clothes that kids wear in your
  school and tell why kids wear them.
• Describe the friends you have and tell why they
  are your friends.

28
Sample Prompts

• Sources for prompts
• Research Institute on Progress Monitoring
• www.progressmonitoring.org
• Write Source
• http//www.thewritesource.com
• NWREL
• http//www.nwrel.org/assessment/pdfGeneral/Prompts
  _BlowingAway.pdf

29
Administration Procedures

• Administration procedures
• Read prompt to students
• Give students 30 seconds to think
• Reread prompt
• Give students X minutes to write (choose time
  based on grade level, and stay consistent all
  year)

30
Scoring Procedures

• Words written (WW) The number of words written
  in the sample. A word is defined as a sequence
  of letters separated by a space from another
  sequence of letters.
• Words spelled correctly (WSC) The number of
  correctly spelled words written in the sample. In
  some studies, words correct are counted
  regardless of whether they are used correctly
  within the context of the sentence (e.g., I saw
  a bare when I was camping.), and in others, only
  when the words are used correctly within the
  context.

31
Scoring Procedures

• Correct word sequences (CWS Videen et al.,
  1982) The number of adjacent, correctly spelled
  words that are syntactically and semantically
  correct within the context of the sample.
• Capitalization and punctuation are taken into
  account.
• Example Themansawa bare inthetreeand
  begn torun. CWS 9.

32
Scoring Procedures

• Correct minus incorrect word sequences (CIWS
  Espin et al., 1999). The number of incorrect word
  sequences subtracted from the number correct
  sequences. The number of CIWS from the examples
  above would be 3 (7-4) and 5 (9-4) respectively.
• Example Theman saw avbarevinthetree
  andvbegnvtorun. CWS 9, IWS 4, CIWS 5.

33
Graphing Data

• Baseline Collect 2-3 samples. Graph the scores.
  
• Draw a vertical line to separate the baseline
  data from the first phase of instruction.
• Draw a goal line corresponding to the desired
  weekly increase for the total number of weeks of
  instruction.

34
Setting the Goal

• Determine the following
• b Students baseline performance
• Example b 24 CWS
• m Desired rate of progress
• Example m 1.5 CWS per week
• x Amount of time the student will be monitored
• Example x 35 weeks
• Solve for y (the students goal)
• y mx b
• Example y (1.5 35) 24 52.5 24 76.5

35
Setting the Goal Practice

• Baseline scores 27 CWS, 32 CWS, 22 CWS
• Use median or mean
• Median (middle score) 27
• Mean (27 32 22)/3 27
• Desired rate of progress 2 CWS per week
• Monitoring period 20 weeks
• Goal y mx b

36
Setting the Goal

• There are no set guidelines for determining
  desired rate of progress.
• You might wish to use
• Published norms (e.g., AIMSweb www.aimsweb.com)
• Local school or district norms
• Classwide data
• TIP
• Set reasonable/attainable yet ambitious goals.

37
Monitoring Progress

• Administer a different writing prompt once per
  week, score the writing sample, and graph the
  scores.
• Monitor writing performance to see whether the
  student is improving and on track to reach the
  goal.
• When you see the student is not reaching the
  goal, modify instruction.

38
Back to Case Study
39
Jaime

• Which type of writing prompt would you select?
• For how many minutes would you administer each
  probe?
• What scoring procedure would you use?

40
Jaimes Writing Sample

• The cave was very dark. I try to close my eyes,
  so I couldnt see anything, but that didnt help.
  Than I hear some one breathing. I try to
  stream, but nother came out. The breathing
  became close and close to me, and the worst Part
  was that I couldnt see athing. At first I
  thought meself that I am Just emaging stuff.

41
Case Study Jaimes Graph
42
Questions for Discussion

• What possible benefits do you see in using CBM-W
  to monitor student writing progress?
• What are the possible drawbacks or barriers to
  using CBM-W?

43
Questions for Discussion

• Writing is a complex, multi-dimensional process.
  How might you account for changes in the quality
  of student writing?
• How might you use CBM-W to monitor your own
  students progress and to make instructional
  decisions?

44
For More Information

• Visit the RIPM Website
• http//progressmonitoring.org
• Contact Kristen McMaster
• mcmas004_at_umn.edu