Assessing Oral Communication and Critical Thinking Skills Across the Curriculum

1
Assessing Oral Communication and Critical
Thinking Skills Across the Curriculum

• P. Anand Rao and Roy Weinstock
• University of Mary Washington

2
Do we know what to look at?
3
Assessment!!

• SCHEV Mandated Assessment
• Part of the Reports of Institutional
  Effectiveness (ROIE) and New Core Competencies
• UMW started work on this assessment project in
  Fall 2003
• Assessment Proposal was based on our use of the
  SI Program

4
Three Assessment Projects

• SCHEV Mandated Assessments
• Modeled SCHEV Assessment project on WI Programs
  assessment of writing proficiency
• Goal is to evaluate 200 eligible speeches for
  each assessment project
• Internal Assessment
• Opportunity to evaluate and reflect upon the SI
  Program
• Will also look at first speeches and first SI
  course

5
Timeline for the Project

• Developed Proposal Fall 2003-Fall 2004
• Proposal Approved October 2004
• Speeches Taped 2005 Calendar Year
• Evaluation Sessions in May 2005 and
• January 2006
• Report Due to SCHEV June 2006

6
What are we evaluating?

• SI classes were selected from departments across
  the University (10 different disciplines were
  represented)
• Student workers taped at least two rounds of
  speeches in each class (as many as 3-4 rounds)
• Class assignments were used
• CT Project all speeches eligible (if viewed as
  appropriate by evaluators)
• OC Project 1st round and 1st SI course removed

7
Spring 2005

• 13 courses were initially selected speeches
  from 11 of those were used
• Over 200 speeches taped 183 were viewed and
  evaluated (including reevaluations)

8
Fall 2005

• 20 courses were initially scheduled we planned
  to tape speeches from 15, and eventually used
  speeches from 13
• More than 230 speeches were taped, and 220 were
  viewed and evaluated (including reevaluations)

9
How They Were Evaluated

• Worked in teams of two viewed speeches on DVD in
  a classroom
• Independently evaluated each speech
• No speeches from their own class may evaluate
  speech from a past student
• May decline to evaluate a particular speech if
  you feel you cannot conduct the evaluation in an
  impartial manner

10
How They Were Evaluated

• Each presentation was to be viewed only once
• Included both individual and group presentations
• Basic information about the course was available
  (type of course, course level, number of
  students), but no course, faculty, or speaker
  identifications

11
How They Were Evaluated

• Evaluation sheets were collected by me, and if
  evaluations differed on either section, then the
  presentation was viewed by another evaluator
  until two matching scores were reached
• Each session (morning and afternoon) began with a
  collective discussion about a sample speech
• Evaluators rotated partners

12
How They Were Evaluated

• Speeches are only to be used for assessment
  project

13
Not Like Grading a Speech in Class

• Class assignments are not always driven by the
  same rationale as the assessment mastery of
  material vs. skills proficiency
• Viewed in a vacuum
• Proficiency ratings not the same as assigning a
  grade

14
Importance of the Assessment

• SCHEV report
• Internal Assessment for the SI Program
• Not constructed with a particular outcome in
  mind- we want to do well, but more importantly,
  we want to do right

15
Whats Next?

• We have about 80 speeches to be evaluated or
  reevaluated this spring
• Our report to SCHEV is due in June
• Plan to prepare a report to the University
  community on our findings over the summer
• Sample Student Speeches Volume
• I plan to use this information as part of a
  review of the SI Program