Title: Training in Experimental Design: Developing scalable and adaptive computer-based science instruction
1Training in Experimental DesignDeveloping
scalable and adaptive computer-based science
instruction
TED
- Mari Strand Cary, David Klahr
- Stephanie Siler, Cressida Magaro, Junlei Li
- Carnegie Mellon University University of
Pittsburgh
2Overview of the TED project
- Curriculum Experimental design, evaluation, and
interpretation - Age 5th-8th grade students
- Schools 6 inner city
- 4 low SES challenging classroom environments
- 2 mid-high SES
- End goal Computer-based adaptive tutor
- 1 student 1 computer in classroom environment
- Provides individualized, adaptive instruction
- Supplements (does not replace!) teacher
3What do we mean by Experimental design?
- CVS Control of Variables Strategy
- Simple procedure for designing unconfounded
experiments - (Vary one thing at a time)
- Conceptual basis for making valid inferences from
data - (Isolating the causal path)
4CVS and RampsTest whether the ramp surface
affects the distance that a ball travels.
Variable Ramp 1 Ramp 2 Ramp 2
Confounded Unconfounded
Surface Smooth Rough Rough
Track length Short Long Short
Height High Low High
Ball Golf Rubber Golf
5Why do we need to teach CVS?
- Core topic in science instruction
- State standards
- High stakes assessments
- Science component of NCLB
- Has real-world applications
- Essential to evaluating product claims, and news
reports - Students do not always learn CVS on their own
(low SES students, in particular)
6What do students do wrong?
- Common errors
- Vary everything
- Hold target variable constant and vary other
variables - Partially confounded
- Nothing varied (identical)
- Their justifications
- I dont know
- You told me to test x!
- Describe their set-up
- Want to see if x happens
- Want to see if this setup is better than that
setup
7Why do they take these approaches?
- By accident
- misread question
- working carelessly
- Are led astray
- by saliency of physical apparatus (e.g., ramps)
- dont understand written representations (e.g.,
tables) - On purpose
- different goals (e.g., engineering)
- misconception of experimental logic
- think other variable(s) dont matter
- Just guessing
8Whats the best way to teach CVS?
- As a society (educators, researchers, and
legislators), we dont know - Our research team knows of one effective way
9Our basic CVS instruction
- Students design experiments
- Students answer questions
- Instructor provides explicit instruction about
CVS - One domain
- Short instructional period
10Effective in the lab and in classrooms of high
SES and achievement levels
- One-on-one Chen Klahr (1999) Klahr Nigam
(2004), Strand Cary Klahr (in preparation) - Full class Toth, Klahr Chen (2000)
- Physical and virtual materials Triona Klahr
(2003)
11Would it work for lower-achieving students in
low-SES schools?
12Effective in low-achievement classrooms (Li,
Klahr Jabbour, 2006)
- Raises item-scores above national norms
- Enables students to catch up with untrained
peers from high-SES schools - BUT, repeated and varied forms of instruction are
required for generalized CVS understanding - Many days
- Multiple domains
13Thus, our starting point
- Brief, focused CVS instruction is differentially
efficient and effective for different student
populations, settings, and transfer tasks. - We want to reach ALL students!
- To improve our instruction for the entire student
population, we must engage in modification
individualization
14A computer tutor could facilitate differentiated
instruction
- Computer-based instruction
- Individualized self-paced
- Provides instruction, practice, and feedback
- Teacher freed to provide coaching as needed
15How are we building our tutor?
- 4 development phases
-
- Iterative design process
164 development phases
- Information gathering
- What are the novice models students hold and how
can we address those? - Refining the basic instruction and going
virtual - Building a computer tutor with a few paths
- Building an adaptive computer tutor with a web
of paths
17- An evolving CVS computer tutor
18Our iterative design process
19- What are we learning from each version that will
help us design the final, adaptive tutor?
- VERSION 1 (Completed)
- Database of student biases, misconceptions,
errors areas of difficulty - Inventory of successful tutoring approaches
- familiar domains
- instruction in prerequisite skills
- step-by-step approach
- Student-friendly terminology, definitions, and
phrasing - Requiring explicit articulation by student
20- What are we learning from each version that will
help us design the final, adaptive tutor?
- VERSION 2 (Ongoing)
- Information regarding
- classwide implementation of successful tutoring
approaches - feasibility of multiple domains
- effect of emphasizing domain-generality
- interface usability
- worksheet usability
21- What are we learning from each version that will
help us design the final, adaptive tutor?
- VERSION 3 (being developed)
- Information regarding
- individual tutor usability and pitfalls
- comparative efficacy of set learning paths
- efficacy of immediate computer feedback
22The adaptive tutor will include
- Pre-testing and ongoing monitoring of student
knowledge - Self-paced instruction
- Diverse topics matching students interests
- An interactive and engaging interface
- Teacher-controlled and/or computer-controlled
levels of difficulty - Level of scaffolding, feedback, and help aligned
with students needs - Computerized assessments
- Logging capability
23Beyond our classroom instruction
- Where on the contextual / abstract continuum
should this type of instruction be focused? When? - Single vs. multiple domains?
- Static pictures vs. simulations vs. tabular
representations - Best mix of explicit instruction, exploration,
help, feedback, etc.
24Questions? Comments?
Many thanks to the Institute of Education
Sciences for supporting our work
- MariStrandCary_at_cmu.edu
- Klahr_at_cmu.edu
25(No Transcript)
26- VERSION 1
- Database of student biases, misconceptions
areas of difficulty - Inventory of successful tutoring approaches
- familiar domains
- instruction in prerequisite skills
- step-by-step approach
- Student-friendly terminology, definitions, and
phrasing - Requiring explicit articulation of understanding
and reasoning
Create best outcome or Most dramatic difference
Ignore the data or Biased by expectations
Pets, Sports drinks, Cars, Study habits, Running
races
Learn about all variables at once
Variable vs. Value Experiment Result vs.
Conclusion
Table format Remembering the target
variable Drawing conclusions based on the
experiment
Read carefully, Identify question, Identify
variables
Good vs. Fair vs. Informative vs. True Variable
something that can change
27What IS an intelligent tutor?
- Computer-based instructional system
- Contains an artificial intelligence component
- Encodes cognitive objectives of the instruction
- Tracks students state of knowledge
- Compares student performance to expert
performance - Tailors multiple features of instruction to the
student (Anderson, Boyle, Corbett, Lewis, 1990
Anderson, Conrad, Corbett, 1989 Corbett
Anderson, 1995 Greeno, 1976 Klahr Carver,
1988).
28Ramp apparatus
29CVS and RampsA completely confounded test for
determining the effect of ramp surface on the
distance that a ball travels.
Variable Ramp 1 Ramp 2
Surface Smooth Rough
Track length Short Long
Height High Low
Ball Golf Rubber
30Classroom CVS with urban 5th 6th graders
(Klahr, Li Jabbour, 2006)
31Low training vs. high comparison group
Training group (5/6th grade, low achieving school)
Comparison group (5-8th grade, high achieving
school)
32Every version
Stand-alone, detailed lesson plan with visual aids
Feedback
Assessments (formative and summative)
Asks students to explain, justify, and infer
Examples of exp. designs (good and bad)
Students designing experiments
33Increasing complexity and adaptiveness
- Physical apparatus ? Virtual simulations
-
- Full class ? Full class individual computer
use - Inflexible ? Individually-adaptive self-paced
- One domain ? Multiple domains
34Why SES differences?
- Found them in our previous studies
- Classroom environment
- Reading comprehension
- Experience with this type of thinking
(expectations, appropriate challenge and/or
scaffolding, amount of practice)
35What if later versions are less effective than
earlier versions?
- Stop the presses!
- Look for obvious reasons
- Examine lesson components individually
- Consider what is missing
36Prerequisites
- Science mindset
- Problem decomposition
- Vocabulary!
- Identify and understand question
- Identify key variables
- Notice and complete component steps
- Analogical reasoning
- Reading listening carefully
37Procedures
- Test one variable at a time
- Make the values for the variable youre testing
be DIFFERENT across groups. - Make the values for the variables youre not
testing be the SAME across groups.
38Concepts
- You need to use different values for the variable
youre testing in order to know what effect those
different values have. - You need to use the same value for all the other
variables (hold all the other variables constant
control the other variables) so that they cant
cause difference in the outcome. - If you use CVS, you can know that only the
variable youre testing is causing the
outcome/result/effect.