Lessons from the Field Computer Based State Testing: the Good, the Bad, and the Ugly

1
Lessons from the Field -- Computer Based State
Testing the Good, the Bad, and the Ugly
2
West Virginia Department of
EducationOnline Writing Assessment2004 - 2006
3
West Virginia Department of EducationOffice
of Student Assessment ServicesCTB/McGraw-Hill
4
BACKGROUND Writing Assessment Redesign
5
RATIONALE

• Promote 21st Century Skills
• Meet National Writing Standards
• Model ACT/SAT Writing
• Prepare for NAEP Writing

6
Fall 2003

• Redesign Committee
• Redesign of Rubric
• 4 point rubric
• 6 point rubric
• Analytic Traits
• Organization
• Development
• Sentence Structure
• Word Choice
• Mechanics

7
Online Writing Assessment7th and 10th GradesMay
September 2004

• Collected 13,000 essays
• Written by WV students
• Computer generated saved to disks
• Emailed to WVDE -- CTB
• Hand scored by CTB
• Selected 100 essays per score point/prompt/trait

8
2005 Hand Scoring Procedure

• Prompts reviewed by table leaders
• Anchor papers chosen
• Readers trained using anchor papers
• Scored by analytic traits total of 30 points
• IntelliMetric scoring engine

9
IntelliMetric Scoring and Validation

• Validity of scoring
• Based on how well the system replicates human
  raters
• Steps in training engine
• 1st papers with scores given by human raters
• 2nd blind papers no numeric score
• 3rd compare agreement
• High rate of agreement
• Algorithm has adequately captured human judgment

10
Spring 2005 Administration
11
March 2005

• 44,000 students
• Public and non-public
• 7th and 10th grades
• Administered electronically
• Four-week window
• Scored electronically
• Same year reports

12
Student Experience

• Access through a secure desktop (VANGuard)
• Easy to use interface
• Access to writers checklist and rubric

13
State/County/School Concerns

• Connectivity
• Internet/computer experience of teachers/students
• Scheduling
• Weather

14
State/County/School Concerns

• Number of labs
• Number of computers
• Security updates and patches
• Windows 2000/XP
• Deep Freeze and Fortress
• Bandwidth/connectivity at the building level

15
County/School Concerns

• Afternoon vs. morning
• Slow/fast students
• Script in manual
• Scheduling of students in labs
• Work space arrangement

16
2005 Bandwidth/Connectivity
17
2005 Bandwidth/Connectivity
18
Student Response

• POSITIVE
• Word processing
• Untimed
• Reports
• Turn around time

19
Student Response

• NEGATIVE
• Connectivity
• Losing essays

20
Improvements for 2006

• Increasing of states bandwidth
• Writing Roadmap practice sessions
• Non-scorable essay review
• Additional essays/prompt to train engine

21
Improvements for 2006

• Moving of SAVE button
• Adjustment of AutoSave to minimize internet
  traffic
• County-by-county test schedule
• Inter-rater study by CTB trained hand-scoring
  raters

22
Improvements for 2006

• Aggregated confidential summary report
• Electronic state reports
• Technical report

23
2006 Results

• 7th Grade -- 75 At or Above Mastery
• 10th Grade 79 At or Above Mastery

24
Recommendations
(Charleston Gazette-Mail -- June 3, 2006)

• WVDE Advisory Committee recommends the following
• Increase hardware, software, infrastructure,
  distance learning, training and technology staff
• Increase funding from 22.44 million to 71.86
  million in the next fiscal year

25

Now is the time for the West Virginia
Department of Education to prioritize the next
steps and nail down what is feasible. Our
obligation is to provide an infrastructure that
we believe will give our teachers and students
the ability to be globally competitive. Dr.
Steve Paine State Superintendent of
Schools (Charleston Gazette-Mail -- June 3, 2006)
26
Contact Information

• WVDE -- Office of Student Assessment Services
• 304-558-2546
• Brenda West, Assistant Director
  bwest_at_access.k12.wv.us
• Sandra Foster, Coordinator
• slfoster_at_access.k12.wv.us
• Jason Perdue, Coordinator jperdue_at_access.k12.w
  v.us

27
Support/Contact Information

• CTB/McGraw-Hill Online Support
• Amy Fitts, Online Program Manager
• 831-393-6238
• JoAnne Edde, Online Support Manager
• 866-282-2250
• CTB/McGraw-Hill Product Information
• Lynn Hunley, Sr. Evaluation Consultant
• 704-896-1778