Critical Thinking at Texas Tech

1
Critical Thinking at Texas Tech March 6, 2009
2
Teaching and Assessing Critical Thinking How to
Make Critical Thinking a REAL Learning Outcome

• Diane F. Halpern, PhD
• Claremont McKenna College, Claremont, CA

With special thanks to a lifetime of coauthors,
especially Milt Hakel, Art Graesser, Keith
Millis, my NCUEP Colleagues, and the real
thinkersour wonderful students
3
Our Studentsthe Millennials

• They are the children of late Baby Boomers
  (1945-64) or Gen Xers (1965-81)
• They spend 6.5 hours a day on various media
• They are 36 of US population
• They are more diverse than any prior group

4
Lets Start by Thinking About the Lives of Our
Students Present and Future

• What do they need to know in a world where
  knowledge is accumulating an unprecedented rate?
• Where our students will work at jobs that do not
  exist today.
• Where a college degree is a requirement for 90
  of the fastest growing jobs.
• Where pollution is a major problem, along with
  racism, poverty, and terrorist attacks

5
Information, information, information

• "A weekday edition of The New York Times contains
  more information than the average person was
  likely to come across in a lifetime in 17th.
  century England."
•  Information Anxiety, R.S.Wurman

6

• Multitaskers--Attention is limited, and when it
  is divided among too many tasks or the tasks are
  difficult, performance suffers. Of course, people
  have always been able to multitask (that is,
  attend to more than one thing) Parents monitor
  their children while cooking dinner, we carry on
  conversations while walking, students do homework
  while listening to music, and so on. But those
  who overload their systems by, for example,
  studying while checking e-mail, instant
  messaging, and watching television will do worse
  at all these tasks than they would if they
  focused on one at a time.
• The more they use the web, the more difficult it
  becomes to stay focused on long pieces of
  writing.

7
Rethinking the Purpose of Education

• The best education for life in the 21st century
  must be built on the twin pillars of "learning
  how to learn" and how to think critically about
  the vast array of information that confronts us.

8
Using What We Know About Teaching and Learning

• Systematically apply what we know about how
  people learn to curricular design and class
  planning.
• Better use of technologyfor example learning
  games
• Recognize what students really need to knowhow
  to think critically and learn in a world where we
  are drowning in information

9
So, if you are thinking critically, you are
wondering if it is possible to help students
improve how they think? Why not? We teach
writing, oral communication, math with the belief
that these skills will transfer to appropriate
situations.
10
Evidence That Better Thinking Can Be Learned with
Appropriate Instruction

• "Blind" evaluations of programs designed to
  enhance thinking skills (e.g., the Venezuela
  project)
• Student self reports (weak evidence, but students
  believe that they have improved)
• Gains in cognitive growth and development (e.g.,
  Piagetian tasks that measure cognitive stages)
• More expert-like mental representations following
  instruction (relative to control groups)
• Decision makers trained to reorganize existing
  knowledge in naturalistic settings showed more
  expert-like performance
• Tests of cognitive skills (e.g., standardized
  tests for critical thinking)
• Spontaneous and uncued transfer (e.g., call
  students at home months after the class is
  completed under the guise of a survey)
• Inductive reasoning tasks were taught to college
  students using realistic scenarios form many
  different domains. The authors conclude that
  critical thinking is a skill and that it is
  transferable

11
Most important robust finding

• The majority of studies report positive impact
  on pupils attainment across a range of
  noncurriculum measures (such as reasoning or
  problem-solving). No studies reports reported
  negative impact on such measures (Thinking Skills
  Review Group, October 2003).

12

• Critical thinking is the use of those cognitive
  skills and abilities that increase the
  probability of a desirable outcome.
• It is purposeful, reasoned, and goal directed.
  It is the kind of thinking involved in solving
  problems, formulating inferences, calculating
  likelihoods, and making decisions. Critical
  thinkers use these skills appropriately, without
  prompting, and usually with conscious intent, in
  a variety of settings. That is, they are
  predisposed to think critically. When we think
  critically, we are evaluating the outcomes of our
  thought processes--how good a decision is or how
  well a problem is solved. Critical thinking also
  involves evaluating the thinking process--the
  reasoning that went into the conclusion we've
  arrived at or the kinds of factors considered in
  making a decision.

13
Important Robust Findings from Research

• Critical thinking will transfer when it is taught
  for transfer.
• The effect may not be even across all groups.
• Some of the benefit derives from making the
  thinking skills explicit.
• The disposition to think critically, like the
  learning-performance distinction is important

14
Thinking About Critical Thinking Instruction
15
Effective critical thinking instruction is
predicated on two assumptions

• There are clearly identifiable and definable
  thinking skills that students can be taught to
  recognize and apply, and
• (2) If recognized and applied, the students will
  be more effective thinkers.

16
Teaching Learning to Think Critically A
Four-Part Model

• 1.Explicitly teach/learn the skills of critical
  thinking
• 2.Encourage/develop the disposition of effortful
  thinking and learning
• 3.Direct learning activities in ways that
  increase the probability of transfer
• 4.Make metacognitive monitoring explicit and
  overt (Halpern, 1998, 2004)

17
Here is a generic list of thinking skills that
would be applicable in many situations

• recognizing that a problem exists
• developing an orderly, planful approach so that
  tasks are prioritized and problems are recognized
  as differing with regard to how serious and
  urgent they are
• understanding how cause is determined,
• recognizing and criticizing assumptions,
• analyzing means-goals relationships,
• giving reasons to support a conclusion,
• assessing degrees of likelihood and uncertainty,
• incorporating isolated data into a wider
  framework
• using analogies to solve problems.

18
Generic Thinking Skills, continued

• Relating new knowledge to information that was
  previously learned
• Using numerical information, including the
  ability to think probabilistically and express
  thoughts numerically
• Using matrices and other diagrams to communicate
• Synthesizing information from a variety of
  sources
• Determining credibility and using this
  information in formulating and communicating
  decisions
• Selecting among alternatives with the use of a
  reasoned method

19
Build on critical thinking skills throughout the
curriculum

• Identify the skills you want students to learn in
  your General Education critical thinking courses,
  then be sure they are used in other classes
• Deliberately add new skills throughout the
  curriculummay be some differences by discipline,
  but ensure overlap among GE courses
• Some critical thinking skills that are firmly
  rooted in psychology
• Understanding that small samples yield more
  extreme results
• Recognizing and avoiding hindsight bias
• Knowing why we need control groups
• Avoiding either-or thinking (e.g., is it nature
  or nurture)
• Being aware of the fallibility of memory
• Self-serving attributes for success and failure
• Halo effects

20
Making Arguments WorksheetExample 1 Does
violence on television really have a negative
influence on children's behavior?1. State your
conclusion. (although you may begin your formal
writing here, but sure that the conclusion
follows from your reasons). As you work, this is
the last part this is filled in, not the
first. 2. Give three reasons (or some other
number) that support your conclusion.  3. Rate
each reason as weak, moderate, strong, or very
strong.
21
4. Give three counterarguments (or some other
number) that weaken your conclusion. Rate how
much each counterargument weakens the conclusion
little, moderate, much, or very much. 5. List
any qualifiers (limitations on the reasons for or
againstfor example some evidence may be
restricted to early childhood) 6. List any
assumptions. 7. Are your reasons and
counterarguments directly related to your
conclusion? 8. What is the overall strength of
your argument weak, moderate, strong, or very
strong? Now that you have completed this
worksheet, rate the overall strength of your
argument.
22
Dispositions for effortful thinking and learning
Model and provide explicit instruction in the

• willingness to engage in and persist at a complex
  task
• conscious use of plans and suppression of
  impulsivity
• flexibility and open-mindedness
• willingness to abandon nonproductive strategies
  and self-correct

23
We need to direct learning activities that make
transfer more likely

• Draw a diagram/graphic that organizes
  information.
• What additional information would you want before
  answering the question?
• Explain why you selected (a particular) multiple
  choice question.
• State the problem in at least two ways.
• Which information is most important? Why?
• Which information is least important? Why?
• Categorize the information in a meaningful way.

24
What Makes Great Teachers Great?

• Create a natural critical learning environment.
  "Natural" because what matters most is for
  students to tackle questions and tasks that they
  naturally find of interest, make decisions,
  defend their choices, sometimes come up short,
  receive feedback on their efforts, and try again.
  "Critical" because by thinking critically,
  students learn to reason from evidence and to
  examine the quality of their reasoning, to make
  improvements while thinking, and to ask probing
  and insightful questions. This is, by far, the
  most important principle -- the one on which all
  others are based and which commands the greatest
  explanation.
• Ken Bain
• Chronicle of Higher Education, April9, 2004
  Volume 50, Issue 31, Page B7

25
Measuring Critical Thinking
26
Halperns Taxonomy of Critical Thinking SkillsA
Guide for Instruction Assessment
27
Halperns Critical Thinking Assessment

• Verbal reasoning ability to comprehend defend
  against persuasive techniques embedded in
  everyday language
• Argument Analysis ability to judge how well
  evidence supports a conclusion
• Hypothesis testing accumulating observations,
  forming beliefs/hypotheses/seeking convergent
  evidence

28
HCTA, continued

• Likelihood Uncertainty recognizing regression
  to the mean, understanding the limits of
  extrapolation probability chance.
• Decision making/problem solving Seeking
  information to reduce uncertainty listing
  alternatives recognizing bias.

29
Assessing Critical Thinking the Halpern
Critical Thinking Assessment (HCTA)

• Available on the Internet/Paper (2 versions)
• Scenario-based
• Each competency is evaluated with forced- choice
  constructed responses
• Forced-choice recognition memory
• Constructed response free recall

30
Halpern Critical Thinking Assessment Sample
Question

• A recent report in a magazine for parents showed
  that adolescents who smoke cigarettes also tend
  to get low grades in school. As the number of
  cigarettes smoked each day increase, GPA
  decreased. One suggestion made in this report
  was that we could improve school achievement by
  preventing adolescents from smoking.
• Based on this information, would you support this
  idea as a way of improving the school achievement
  of adolescents who smoke?
• Type yes or no and explain why or why not.

31

• Based on this information, which is the best
  answer?
• 1. School grades probably will improve if we
  prevent adolescents from smoking.
• 2. School grades might increase, but we cannot be
  certain because we only know that grades go down
  when smoking increases.
• 3. There is no way to know because we only know
  that smoking and grades are related, not whether
  smoking causes grades to change.
• 4. There will probably be no effect on grades
  if we prevent adolescents from smoking because
  the magazine is written for parents, so it is
  probably biased against teen smoking.

32

• Based on this information, which is the best
  answer?
• 1. School grades probably will improve if we
  prevent adolescents from smoking.
• 2. School grades might increase, but we cannot be
  certain because we only know that grades go down
  when smoking increases.
• 3. There is no way to know because we only know
  that smoking and grades are related, not whether
  smoking causes grades to change.
• 4. There will probably be no effect on grades
  if we prevent adolescents from smoking because
  the magazine is written for parents, so it is
  probably biased against teen smoking.

33
IES Grant that is incorporating what we know
about the science of Learning

• A program to teach critical thinking/ scientific
  reasoning skills using what we know about current
  students (play games on line, use teaching
  agents, and more).
• Grant is with Keith Millis at Northern Illinois
  University and Art Graesser at University of
  Memphis(they are the brains behind this project)
• ARIES Acquiring Research Investigative and
  Evaluative Skills (a brief look early in the
  project)
• Special thanks to other ARIES folksPatricia
  Wallace, Zhiqiang Cai, Cristian Moldovan, Carol
  Forsyth, Anne Britt, Joseph Magliano, Katja Wiemer

34
ARIESAcquiring Research Investigative and
Evaluative Skills

• Animated Agentsa teacher and a student guide the
  student through the tutor lessons and will be an
  expert on scientific inquirythe heart of
  critical thinking
• For example My roommate and I got into an
  argument yesterday on who was more influential on
  hip hop James Brown or Stevie Wonder to which
  the Teacher-Agent might respond, You know, you
  could have resolved the argument by using what
  scientists call an operational definition.

35
Some Key Concepts in Scientific Inquiry

• Developing Research Ideas
• Theories, hypotheses, pseudoscience,
  falsifiability
• The Independent and Dependent Variables
• Operational definitions, reliability, accuracy,
  precision, validity, objectivity of scoring
• Experimental Control
• Comparison groups, random assignment, subject
  bias, attrition/mortality
• The Sample Experimenter
• Representative, sample size, experimenter bias,
  conflict of interest
• Drawing conclusion
• Alternative interpretations, limits of
  correlation research, quasi-experimental designs,
  replication of results

36
ARIESAcquiring Research Investigative and
Evaluative Skills

• We use science of learning in the program
• Self-explanation
• Generate reasons why a study is faulty or not
  faulty
• Reciprocal teaching
• Students teach the fellow student
• Spacing, testing effects
• Students must recognize concepts across many
  examples
• Variable encoding
• Psychology, biology and chemistry problems
• Authentic learning
• Case studies are magazine, news articles,
  advertisements
• Motivation, engagement
• Consequences for their performance
• Auto-tutor platform that allows students to hold
  a dynamic conversation with the learner (dialog
  interactivity).
• Principles of serious gaming.

37
Operation ARIES! Your mission to become an
agent of the Federal Bureau of Science (FBS) to
hunt down Fuath spies who are on Earth stealing
our resources. Be careful because they look and
act human! The Fuaths have been publishing
faulty research in a variety of fields in order
to confuse future human generations about the
scientific method. They are also milking our
economy dry by selling products on the Internet,
based on suspect research.
38
Aliens taking over the earth with bad sciencenow
that is scary!!
39
How can you become an FBS agent?
Step 1 Take a Science Training Course
You will learn key concepts in the scientific
method like control groups, validity,
independent and dependent variables (20 in
all). You will read (and be tested on) a science
book used by the Fuath spies. You will be guided
by an FBS handler and be joined by a fellow
(animated) student
40
How can you become an FBS agent?
Step 2 Analyze Case Studies
You will analyze examples of research written by
the Fuaths. They are from magazines, the
Internet, and newspapers, covering topics in
Psychology, Biology and Chemistry. With help
from your FBS handler and a Fuath defector, you
will learn to identify flaws in the research.
Pay attention because the Fuaths use the flaws to
communicate with one another. You will likely
uncover important clues to their plans.
41
How can you become an FBS agent?
Step 3 Interrogate Suspected Alien Scientist
Spies
You are close to finding and stopping the
mastermind behind the invasion. FBS has
captured a number of scientists, some of whom are
Fuaths and some whom are Human. It is your job
to tell the difference between them by asking
questions about their research. If a study is
faulty, then you have found a Fuath and you are
one step closer to saving the Earth. Be alert!
42
A Sample Screen
43
Example of Research to EvaluateStudents Have to
Ask Questions
44
Scientific Thinking/ Critical Thinking

• For each chapter
• Multiple choice questions
• Types Definitional, Function/importance, Example
  
• 2 of each, 6 total
• Challenge Test
• Options take before or after reading chapter
• If before, and any error on first three, then
  given chapter
• Order of questions
• D F E D F E
• Trialog after the last 3 questions

45
Types of Trialogs

• Dr. Quinn (teacher) to Glass (student agent)
• Vicarious learning for students with low scores
• Low knowledge good for vicarious learning (Craig
  et al., 2004)
• Dr. Quinn (teacher) to human
• Standard AutoTutor for students with intermediate
  scores
• Partial knowledge, zone of proximal development
• Human to Glass Tealman (student agent)
• Teachable agent for students getting all or most
  questions correct
• Reinforce and use existing knowledge

46
Here is an example of a session in which the
student evaluates information to decide if the
person doing the research is a human (good
research) or an alien (flawed research)
47
Summary

• ARIES is an initial attempt to incorporate
  AutoTutor in a game-like setting
• Dialogs in AT
• Make learning active, increasing motivation
• Can serve different tasks (testing, evaluating
  studies, promoting questioning skills,
  advancing story line)
• Making engaging games that lead to deep learning
  is still a new frontier in education

48
Many Experiments are Underway

• 1. Does Having Different Content Domains in ARIES
  Increase Transfer?
• 2. Does Role Playing and the Type of Activity
  Affect Learning?
• 3. Do We Need Full Dialog Exchanges for all
  Active Applications Problems for Deep Learning to
  Occur?
• 4. Does the Reciprocal Teaching in ARIES Affect
  Learning?
• 5. Does the Effectiveness of Teachable Agents
  Depend on the Perceived Knowledge of the Agent?

49
Contact Information

• Dr. Diane F. Halpern
• Claremont McKenna College
• 850 Columbia Ave.
• Claremont, CA 91711
• [phone]
• To try out the critical thinking assessment, send
  an e-mail to diane.halpern_at_cmc.edu