Montana University System Writing Assessment

1
Montana University System Writing Assessment

• Improving Teacher Quality

2
Writing Proficiency A Brief History

• 1995 Board of Regents (BOR) approves
  Proficiency Admissions Requirements and
  Developmental Education in the MUS
• 1998 OPI convenes Composition Transition Team
• 2000 BOR approves Joint K-16 Composition
  Standards Committee recommendation to field test
  a writing assessment
• 2001 Montana/ACT Field Test begins
• 2003 ACT and SAT announce new Essay Sections
• 2004 BOR passes Writing Proficiency Policy
• 2007 Writing Proficiency Policy applies to
  incoming freshmen
• 2007 BOR passes Composition Placement Policy,
  which applies to two-year and four-year campuses
  of the MUS
• 2009 Writing Proficiency Policy reaches
  proficient levels (3.5 or 7) for FULL admission
  to the four-year programs of the MUS

3
Policy defines Writing Proficiency

• Any student granted full admission to a four-year
  degree program at MSU, MSU-B, MSU-N, UM, UM-Tech,
  or UM-W must satisfy a writing proficiency
  standard.
• For Fall 2009 and the following years, students
  must earn a minimum score of
• 7 on the Writing Subscore or 18 Combined
  English/Writing section of the Optional Writing
  Test of the ACT or
• 7 on the Essay or 440 on the Writing Section of
  the SAT or
• 3.5 on the Montana University System Writing
  Assessment or
• 3 on the AP English Language or English
  Literature Examination.
• With these minimum scores, students can be placed
  directly into college-level composition.

4
Placement into Developmental Coursework
Provisional Admission

• A student who has not yet demonstrated the
  ability to meet these standards may be admitted
  (without condition) to a two-year degree program
  or admitted provisionally to a four-year degree
  program on any campus of the Montana University
  System.

5
Composition Placement Policy

• Applies Policy 301.16 to exempted students and
  two-year students
• Aligns writing placement exams on campuses with
  the MUSWA
• Sets thresholds based on writing assessment
  taken in high school (MUSWA, ACT, SAT, AP)
• students can skip on-campus placement exam with
  entry scores
• students can challenge placement
• students can be placed directly into remedial or
  advanced courses
• Aligns placement practices in two-year and
  four-year programs
• Eases transferability through consistent
  placement practices

6
Implementing Placement Policy
Summer 2008 1,332 students
7
Over 60 of Montanas Class of 2008 Took Entrance
Exams
Of 11,202 total students (WICHE, March, 2008)
8
ACT, SAT, MUSWA Reveal Similar Score
Distributions
Whole numbers reflect perfect scorer agreement.
A score of 7 represents the proficiency
threshold.
9
Costs of MUSWA Shared
The graph includes all operating costs and .5 FTE
for Academic Initiatives Title II Improving
Teacher Quality grants to campuses for onsite
facilitators, training costs and college credits
and an estimate for the cost to K-12 schools for
substitute teachers and teacher travel
10
What Does the MUSWA Save?

• If the 7,100 students whose scores were reported
  in 2008 had paid for the Optional ACT Writing
  Test 106,500
• If the estimated 9,000 students who will test in
  2009 paid for the ACT 135,000
• If the value of a day of professional development
  is 75 45,000
• If we reduce the number of placement tests
  administered and scored on campus by 63 (UMs
  figures) and each essay cost a campus only 2,
  the savings to campuses for freshmen alone would
  be 8,866.
• If you add the professional development value,
  placement test savings, and consider that only
  half of this years students use the MUSWA
  instead of ACT, the savings would be 121,366
  per year.

11
Test Validity

• Measures students ability to engage in the
  academic writing expected in the MUS
• Allows students to demonstrate what they can do,
  rather than find instances of what they do not
  know
• Correlates with successful performance in the
  content area of interest writing
• Offers modes of composition used by students

12
The Montana University System Writing Assessment
(MUSWA)

• Voluntary Participation
• Networking of K-12 teachers and college faculty
• Professional development
• To clarify and share standards across levels
• To improve curriculum and instruction
• To bring consistency to grading and placement
  practices
• Development of Learning Communities

13
Voluntary Participation Increasing
MUSWA is voluntarily used in 76 of Montanas 170
High Schools.
14
Schools Select Testing Mode

• Handwritten in Test Booklets
• Word-Processed, Inserted into Test Booklets
• Bubble-in Student Information Page
• Composed, Word-Processed, Printed
• Online at www.MUSWA.com
• Teachers Register Classes Online
• Students Provide Information Online
• Students Compose in WP Program, Copy and Paste
  into Text Box, Submit
• OCHE Prints for Scoring
• Students with Special Needs use Appropriate Mode

15
Use By Mode (2009 Based on Registration Data)
16
6-Point Holistic Rubric

• Focus on topic
• Taking a position defined in the prompt
• Organization
• Development of Ideas
• Supporting the position with evidence
• Language Fluency
• Sentence control, clarity of expression, word
  choice
• Language Conventions
• Grammar, usage, mechanics

17
On a Six-Point Rubric, How is Proficiency
Described?

• These papers state and support a position on the
  issue defined in the prompt with some elaboration
  or relevant explanation.
• Organization is generally clear.
• Sentences are usually well controlled, expression
  of ideas is usually clear, and word choice is
  appropriate for the topic.
• A competency with language is apparent, even
  though there may be some errors in grammar,
  usage, and mechanics.

18
Preparing Students

• Write persuasive essays using prompt format
• Deconstruct arguments found in advertising,
  opinion pages, speeches.
• Learn organizational and language skills
• Practice scoring other papers with rubric
• Use website to submit practice essay

19
Using Writing Process Improves Scores
20
Students Must Demonstrate Thinking Skills

• Show the ability to make and articulate judgments
  and grasp the complexity of issues, e.g., by
  considering implications.
• Present specific and relevant support or
  evidence.
• Use logical reasoning that shows the ability to
  distinguish between assertions and evidence and
  to make inferences based on the support or
  evidence.

21
Prompt Features

• Specific, hypothetical situation involving a
  choice between two alternatives
• Specific criterion that will be used to weigh
  alternatives
• Specific audience

22
A Retired Prompt

• Your local school board is planning a building
  project to improve the quality of students
  experience in your local high school. Students,
  teachers, and community members have offered two
  proposals 1) to build a new school auditorium
  with a stage to provide a place for school plays,
  talent shows, and other performances 2) to build
  a new gymnasium for the schools team sports, PE
  programs, and intramural activities.
• Write a letter to the School Board in which you
  make a case either for building 1) a new
  auditorium, 2) a new gymnasium, or 3) some other
  facility that you think is more important,
  explaining why you think the proposal you
  selected is more likely to improve the quality of
  students experience in school.
• Begin your letter Dear School Board

23
Http//Webwriters.msugf.edu

• Sample essays with explanations about how essays
  are scored.
• Released prompts for students to write as
  practice.
• The capacity to accept student essays submitted
  electronically and scored by trained volunteer
  teacher/scorers.
• Activities for improving writing
• Tackling the Prompt
• Resources and links
• Instructional ideas for teachers.
• Funded by Student Assistance Foundation of
  Montana.

24
What do Students get from MUSWA?

• Students with 5.5 and 6 scores receive Letters of
  Recognition
• Students bound for the MUS may save 15.00 by
  avoiding the ACT Optional Writing Test
• Potential statewide savings 49,300
• Students may use the MUSWA as practice for the
  ACT or SAT essays
• Students who score below the thresholds can
  improve their skills and retake as seniors

25
Achieving Transparency

• Young people should not have to wait until they
  have a high school diploma in hand to learn that
  they are unqualified for college-level courses or
  for work.
• The Lost Opportunity of Senior Year Finding A
  Better Way (2001)

26
81 of 2008 Test-takers Plan to Attend Colleges
or Universities
27
Remediation in English Composition of Montanas
HS Graduates Fall Semester in the MUS
After the Writing Proficiency Policy was adopted
in 2004, composition remediation spiked to 14.6,
but with implementation in 2007, it dropped to
11.4, then edged up to 12.4 in 2008.
28
What Do Teachers get from the MUSWA?

• Student, class, school data
• Analysis of writing strengths and weaknesses
• Regional collaboration
• OPI renewal units or college credits
• Sharing of standards, curriculum, and
  instructional practices across K-16
• Alignment of placement and admission practices

29
K-12 and College Staff Volunteer to Score
In 2001-02, scoring was held in three sites in
2003, five sites in 2004-06, seven sites and
2007-08, eight sites.
30
Scoring Reliability Check
Reliability Coefficient Cronbachs Alpha 0.873
31
Scoring Site Data from 2008
32
What Do Schools get from MUSWA?

• Breakdown of Scores by State, School, Teacher,
  and Classroom
• Data on Student Aspirations and Skills
• Professional Development Collaboration
• Potential Awards of Merit
• Press Release Templates

33
Strength and Weakness Data
34
Comparing Two Schools Data
35
How do schools participate?

• Register for number of tests needed, select mode,
  commit scorers (1 per 40)
• Schedule one day, February 2-27 to test with one
  day for make-up
• Help students prepare
• Watch mail for materials (two envelopes for
  booklets) then keep them secure
• For online, explore site register classes at
  least a week prior to testing
• For booklets, spend 10 minutes before test day
  for students to bubble-in information
• Read Directions for Test Administration before
  test day
• On test day, distribute prompts, monitor testing,
  destroy prompts
• Return test booklets

36
What happens after testing?

• Trainers meet in Helena to add new samples to
  anchor and practice sets, write new annotations,
  and practice training
• Tests are mixed, sorted, and boxed at OCHE for
  scoring
• Scoring sessions are held March 9 April 2
• Score sheets are scanned, data examined, and
  reports generated after scoring is complete
• Schools receive scores about April 29
• Schools receive additional data, transcript
  stickers, and electronic files by May 22
• Colleges and universities receive scores

37
Trainers Responsibilities

• Select anchors and practice papers that will
• 1) reinforce the rubric and
• 2) settle a few issues that will be raised.
• Recruit scorers.
• Learn to persuade scorers that the anchors are
  reliable samples.
• Help make training/scoring a positive experience
  for all.

38
Score Sheets and Scoring

• If student didnt do it, bubble in prompt number
• If printed prompt is wrong, bubble in right
  number
• Bubble in Solution 1, 2, or 3 (other)
• All score sheets must have Reader 1 2
• Reader 3 (resolver) bubbles in 2 scores
• All readers share Comments Strength and Weakness
• Online score sheets MUST show two matching test
  s
• After scoring, scorers may check for agreement
• Table leader should check for accuracy and
  discrepancies, then hold discussions
  outsideparticularly early in the scoring process

39
Data Glitches

• Incomplete student name score without student
• Blank test booklet student without score
• No test number score without student
• Blank online test student without score
• Mixed header sheets, booklets mismatch
• CHECK NUMBERS GIVE BLANKS 0s

40
Manage the Materials

• Keep score sheets with tests until information
  and scores are final
• Ensure score sheets are scanner-ready
• Completed neatly and correctly
• Stacked with same orientation
• Stack tests by score for easy research
• Collect complete, ordered training materials
  ready for next scoring site

41
Professional Conversations

• What strengths does this writer demonstrate?
• How does this essay reflect the rubric at score
  X? At score Y?
• How would you help this writer improve?
• Does this writer demonstrate the capacity to
  succeed in a college-level composition course?

42
You Are Making a Difference!