ARE They Thinking WCET Tides, Shoals,

1
ARE They Thinking?WCET Tides, Shoals,
HarborsCharting the Voyage for E-Learning in
Higher EducationSan Diego, California, November
4, 2003
CTLT
Gary Brown, Director The Center for Teaching,
Learning, Technology browng_at_wsu.edu
2
Were They Thinking?
3
Enrollments in WSU Online Learning Spaces
4
But Are They Thinking?
5
Assessment
6
(No Transcript)
7
The Rubric Paradigm

• Guide faculty gradingand can reveal how students
  are thinking
• Guide student learningdemystify expectations
  (goals)
• Provide measures of growth
• Beginning of course to end of course
• Beginning of program to end of program
• Beginning of enrollment to graduation

8

• Critical thinking is something you will need
  for the rest of your life. You may not be able
  to answer every question that is thrown at you
  right off the bat, but critical thinking can help
  you find a way to answer the question, or throw
  it back in a different way
• a student

9
Dimensions of Critical Thinking

• Identifies and summarizes the problem/question at
  issue (and/or the source's position).
• Identifies and presents the STUDENTS OWN
  perspective and position as it is important to
  the analysis of the issue.
• Identifies and considers OTHER salient
  perspectives and positions that are important to
  the analysis of the issue.
• Identifies and assesses the key assumptions.
• Identifies and assesses the quality of supporting
  data/evidence and provides additional
  data/evidence related to the issue.
• Identifies and considers the influence of the
  context on the issue.
• Identifies and assesses conclusions, implications
  and consequences.

10
Critical Thinking and Measures of Growth

• Identifies and summarizes the problem/question at
  issue (and/or the source's position).

Emerging____________________________ Mastering
11
(No Transcript)
12
Findings from WSU
13
Critical ThinkingOne coursetwo semesters
14
Critical Thinking8 Courses 4 with CT, 4 w/o CT
15
Faculty Development

• Faculty who used the rubric were enthusiastic and
  expressed plans to integrate the rubric more
  intensively in future courses.
• The critical thinking rubric was valuable for
  helping faculty communicate expectations to
  students.

16
What Students Say

• The grading rubric was very demanding.
• Having to think and not just write took some
  getting used to.
• I have carried this skill on to my other classes
  as well.

17

• This is something Ive been hoping would
  actually happen somedayIts very refreshing to
  see that there are teachers that like to actually
  encourage thought, something that goes a lot
  farther in the real world than knowing a plethora
  of historical facts that will never serve you any
  farther in life than maybe to be good at watching
  Jeopardy.

18
What About Content?

• By taking the emphasis off of learning
  material to answer test questions and placing it
  more on critical thinking and formulating new,
  original ideas, students are forced to really
  think about the material they are researching and
  the discussions held in class and to make their
  own decisions about the meaning and assumptions
  behind the facts.

19
Critical Thinking StudyResults

• Significant gains in courses when rubric is used.
• Significant gains from first to junior years.
• Butthe critical thinking of upper division
  students had a mean of only 3.1 on scale of 6.
• Developing but not mastering...

20
Additional Findings Implications

• The greatest gains by upper division students
  reflect improved abilities to analyze issues from
  multiple perspectives.
• Students gained the least in their abilities to
  articulate their own viewpoints.
• Comparisons to WSUs writing assessment
• As critical thinking scores rise, writing
  placement scores and portfolio exam scores
  sink...
• The faculty questionnaire revealed a focus on
  gradingrather than on critical thinking for
  broader life-long learning.

21
PlagiarismReporting Frequency (of respondents)

• 1999 2001
• Written cut paste 40
  43
• Written plagiarism 16
  8
• Internet cut paste 10
  41
• Internet plagiarism 5
  5
• (e.g., paper mills)

22
Cartoon Fetching is for Losers
23
The Messy NSSEGeorge Kuh

• Student Engagement
• Most students come to college expecting to be
  more engaged than they are.
• Students typically dont exceed their own
  expectations, particularly with regard to
  academic work.
• But students will go beyond what they think they
  can do under certain conditions, one of which is
  that their teachers expect, challenge, and
  support them to do so.
• Are we willing to make the effort that such
  practices demand of us?
• From Change,
  March/April 2003 p 25-32)

24
The Next LevelAssessing Assignments

• To what extent do assignments predict students
  critical thinking?
• Faculty meet to use the rubric to rate each
  others assignments.
• Establish inter-rater reliability, or consensus.

25
Findings from WSU
26
Assessing Assignments

• Total CT assignments 23
• Total Instructors 23
• Total assignment assessments 272
• average Inter Rater Reliability .8129
• Total CT student papers 240
• average Inter Rater Reliability .7682

27
Assignment Assessment the Echo Effect
28
Planning for the Next Step
29
The Age of InteractionBetween People
People are most definitely not doing the things
which the Internet was originally designed to do,
moving large volumes of data around, getting
remote access to supercomputer facilities, or
whatever. . . . They're not connecting to other
computers, but to other people. --Paul
Dourish, Senior Researcher _at_ Apple
30
The Online WastelandIn the South (1/20/2001
60902 PM), by Rhondaviolence as I see it
(1/22/2001 95825 AM), by Rob Personal Insight
(1/23/2001 74324 PM), by Sue Informal
Statistics (1/24/2001 120912 PM), by Toni Rape
(1/25/2001 100233 AM), by Decker Shame and
Fear (1/25/2001 62707 PM), by Ramsey Informal
statistics - commonality of violence (1/26/2001
95808 AM), by Harvey cast iron skillets and
whiskey (1/26/2001 74109 PM), by Mary Dear
Mary (1/27/2001 113321 AM), by Rhonda Thanks
for responding Rhonda (2/1/2001 75319 PM), by
Mary Violence in a small community (1/30/2001
95719 PM), by Mallory What I think! (2/1/2001
50856 PM), by Gerri Informal Stats (2/2/2001
100418 PM), by Dan In my own circle (2/4/2001
32931 PM), by Dara
I hope we are able to exchange with each
other more within the class.
31
Toward Critical Engagement
32

• Another Model, Another Exchange
• Eric's Estimable Explorations (2/12/1998
  115831 AM), by Eric
• Dennis's Angst-Ridden Qualifications (2/19/1998
  71708 AM), by Dennis
• Confession (2/22/1998 55037 PM), by Gary
• none (2/13/1998 93348 AM), by Mike
• Response to Mike (2/16/1998 104139 PM), by
  Christina
• Hey, wait for me (2/20/1998 73542 AM), by John
• Defining Power (2/23/1998 43718 PM), by
  Christina
• Hyperflitter A Digger, Too (2/17/1998 44659
  PM), by Gary
• Hi from Muncie (5/19/2001 125620 PM), by Dennis
• Hackers anti-social? (2/13/1998 93443 AM), by
  Mike
• Are You a "Hacker"? (2/16/1998 12749 PM), by
  Eric
• sometimes (2/17/1998 34540 PM), by Tina
• History (2/16/1998 101518 PM), by Christina
• What Is Hypertext? (2/17/1998 92318 AM), by
  Eric
• How would you like your tripe? (2/18/1998
  102847 AM), by BRIAN
• Anonymity/Credibility/Accountability (2/18/1998
  115529 AM), by Eric
• Anonymity (2/19/1998 121141 PM), by Dennis
• Speech is Speech (2/24/1998 11705 PM), by BRIAN
  

33
(No Transcript)
34
Assessing Engagement Online
Assignment
Facilitation
Evaluation Criteria
Critical Thinking Outcomes
35
The Critical Engagement Criteria

• Scale
• Capping or Engaging
• Dimensions
• Assignment
• Evaluation Criteria
• Facilitation
• Student Interaction
• Situated Alignment

36
Qualitative Quantitative Findings

• Traditional Faculty facilitation tends to cap
  critical engagement.
• Faculty assignments tend to over or parallel
  prompt.
• Outlining formula answers
• Experienced, mature learners will self-facilitate
• Minimal, intermittent faculty participation
  motivates
• Mature is not necessarily older.
• Mature learners do tend to appreciate older,
  experienced learners
• As students perceptions of the efficacy of
  learning activities matures.
• Tests and homework assignments wane and peer
  critiques and interactions wax

37
Engagement Models What Learners Say

• Our class interaction was THE MOST valuable
  tool. This set up reminds me of seminar style
  classes, where you present your paper or views
  and others discuss and share their views.
• My ideas were constantly challenged and changed
  by the students and opinions in this course. I
  think that this is what made the class such a
  valuable experience for all of us.

38

• I appreciated all the comments to my posts,
  especially those comments that pushed me to
  further explain my position or clarify my
  thoughts on a particular subject. I think this
  interaction with other classmates helped me to
  fully develop my thoughts and understanding of
  the subject matter and to get another point of
  view that I would not have been able to get in
  just a video course.

39

• What really amazes me the most through this
  course is the way MY perspectives are changing
  through each lesson
• Since the professor actually encouraged the
  use of personal experience as a resource or
  reference in this section, the posts have been
  particularly informative.
• I certainly will be able to use all that I
  learned in this course and apply it to my own
  life and understanding of myself and others.

40

• I am a firm believer that the way in which
  these online courses are set up really puts the
  emphasis on learning, as opposed to the
  traditional binge and purge, i.e. memorize,
  take a test, then forget the information.

41

• I have to agree with you on that. I think
  everything that we had to discuss has really
  stuck in the long term memory rather than
  memorizing it and spitting it back out. I hope
  they do more courses in this format. It takes the
  pressure off and makes it more enjoyable to learn
  and actually retain it.

42
Conclusions?
43

• Students value interaction
• They learn to value challenging their own and
  each others ideas
• They develop their ideas more fully
• Value validation of personal experience
• And the subsequent application to their lives
• Learn material more deeply
• They enjoy the interaction and therefore the
  learning

44

• Students will demonstrate very sophisticated
  thinking when we design courses that encourage
  them to do so
• They think when we let them!

45
Evaluating Online Discussions Four Different
Frames of Analysis

• Katrina A. Meyer
• Assistant Professor of Educational Leadership
• University of North Dakota
• 2003 WCET Conference

46
Four Frames

• King and Kitchener Reflective Judgment Model
• Perry Intellectual and Ethical Development Model
• Garrison et al. Critical Thinking Model
• Bloom Taxonomy of Educational Objectives

47
Findings

• Of 278 total postings, 74 were coded by all four
  frames (26 dropped).
• Of 206 postings, 63 or 130 postings had unique
  combinations of the four frames.
• Of the over 1500 possible permutations of the
  four frames, only 130 combinations were found,
  and 31 of these only appeared twice.

48
Findings

• 33.4 of postings captured the request for and
  response from students of personal experiences
  and beliefs (KK3 and P7).
• 32.5 of postings were related to requesting and
  responding with analyses that were either
  exploring or integrating ideas (Bloom
  Garrison).

49
Findings

• The type of triggering question influences the
  level of response from students.
• Questions created to trigger personal stories did
  so and questions targeted to elicit information
  or higher-level analysis did so.

50
Findings

• 45.3 were at levels 5-7 (King and Kitchener)
• 100 were at levels 5-9 (Perry)
• 52.2 were at the two highest levels (Garrison)
• 54.3 were at levels 4-6 (Bloom)
• Appropriate for doctoral-level students in a
  doctoral-level class.

51
Findings

• Each frame has value.
• Each frame focuses attention on a particular
  aspect or quality of the student and his/her
  thinking
• There is no one best frame, or perhaps one frame
  might be better suited for a particular
  discussion or a particularly set of learning
  objectives.
• Need to develop multiple frames for analyzing
  online discussions intended to address different
  learning situations.

52
Findings

• Perry was the most difficult framework.
• Distinctions between levels were difficult to
  discern.
• This may preclude it being used on a more
  frequent basis.

53
Findings

• King and Kitchener was less difficult to apply.
• Both King and Kitchener and Perry are
  classification schemas for an individual.
• Both might work better if the discussion were set
  up to generate a particular reflection or if the
  post was sufficiently long to give a sense of the
  students reasoning.
• The level of a specific post (or even several
  postings) may or may not indicate the students
  predominant developmental stage.

54
Next Level of Analysis

• A stage may indicate the students primary level
  of response or
• A stage the group must pass through (i.e.,
  information) in an effort to develop higher-level
  analyses.
• Use frames to analyze the ebb and flow of online
  discussions as a group effort, rather than
  focusing on the individual postings as a
  reflection of the students level of thought.

55
Cautions

• Danger that a posting might become colored by the
  frame.
• Each frame focused the analysis at the same time
  it eliminated other perspectives.
• It is a lens that filters out AND focuses.

56
More Frames Needed

• Assessment of social presence the ability of
  students to contribute in such a way as to make
  their personalities come to life in their
  postings.
• Assessment of group functioning how online
  groups work together to develop an understanding
  of and solutions to a problem.