The Influence of the Use of an Open-Ended Classroom Response System on Student Outcomes

1
The Influence of the Use of an Open-Ended
Classroom Response System on Student Outcomes
Hoky Min, Gregory K. W. K. Chung, Rebecca
Buschang, Lianna Johnson, William Kaiser
California Educational Research
AssociationAnnual Meeting Rancho Mirage, CA
December 5, 2008
2
Overview of Talk

• Background of research (Greg)
• Survey constructs (Greg)
• Analyses (Hoky)
• Results (Hoky)
• Implications and next step (Greg)

3
Background

• Develop a classroom response system
• UCLA developed (Elec. engineering by Bill Kaiser)
  3I Individualized, interactive instruction
• Different from clickersfocus on the process of
  problem solving, not just the final answer
• Test the machinery
• To what extent can teachers make use of real-time
  student responses?
• How do students perceive the experience?
• How does the mode affect student learning?

4
Overview of Research

• Develop and validate a survey measure of
  students perceptions of processes experience
  with a classroom response system
• Examine technical quality of measure
• Examine relation between perception and outcomes

5
3IIndividualized, InteractiveInstruction

• Use computers to help with immediate feedback and
  formative assessment
• Typical lesson
• Present problem / question / prompt
• Students type their response
• Teacher interprets student responses and adjusts
  instruction immediatelymoves on, reviews,
  elaborates, discusses,

6
(No Transcript)
7
(No Transcript)
8
Students View
9
Instructors View
10
Instructor Perceptions

• All students in session participated, drastically
  improved interaction
• Clear and immediate feedback
• Rate of receiving questions and observing
  responses to problems is much higher than
  conventional sessions
• Method exceeds interactivity of one-on-one from
  instructor perspective

11
Student Perceptions

• Interviewed students and gathered written
  responses during pilot tests
• Learning, interaction, interest
• Comfort participating
• Engagement
• Developed survey items based on qualitative data
• Examine technical quality of measure and relation
  to student outcomes (this study)

12
Scales

• Learning
• Interaction
• Interest
• Comfort participating
• Engagement

13
Method

• Undergraduategenetics
• 59 students
• 3I used for weekly discussion sessions (9 weeks)
• Middle schoolsummer school remedial math
• 104 students (6th, 7th, 8th grade)
• 3I used for guided practice sessions (twice over
  4 weeks)
• Minimal instructor training

14
Analyses and Results
15
What We Did
16
Research Questions

• To what extent does the survey measure students
  perceptions on the use of the technology in
  class?
• To what extent do students perceptions influence
  their class achievement?

17
Structural Equation Modeling (SEM)

• A statistical technique that tests hypotheses,
  theories, and models as to relationships among
  variables
• Latent variables Theoretical constructs
  underlying performance or scores on measures
• Observed variables Scores or performance on
  measures

18
Structural Equation Modeling (SEM)

• Measurement model (confirmatory factor analysis)

19
Structural Equation Modeling (SEM)

• Structural regression model

20
Measurement Model (College)
21
Structural Regression Model(College)

• Model 1

22
Structural Regression Model (College)

• Model 2

23
Measurement Model (Middle School)
24
Structural Regression Model (Middle School)

• Model 1

25
Structural Regression Model (Middle School)

• Model 2

26
Summary of Findings

• For college and middle-school levels, the survey
  measures are valid indicators of students
  perceptions of the learning processes evoked from
  the use of 3I
• Students perception does not predict class
  achievement
• Students perception and class achievement are
  both affected by their existing knowledge on the
  subjects

27
Implications

• Why was there no relation between students
  perception of classroom processes and outcomes?
• Classroom interaction doesnt matter
• Poor measure
• Duration of use too short
• Ceiling effect with university students
• Relative coverage of content (with respect to
  outcome) in 3I sessions was much less than
  lectures
• Instructor training

28
Next Steps

• Improve instructor support
• Develop structured problem sets
• a priori -- Common errors , possible knowledge
  gaps behind errors, instructional strategies
• Experimental design
• With 3I vs. without 3I (business as usual),
  control for content
• Challenging

29
greg_at_ucla.edu
30
Perceived Learning

The sessions helped to reinforce what I had learned from lectures and the book. It was a good way to solidify any potential questions I may have had regarding specific circuits. Using the computer based tools was a nice alternative to pencil and paper or white-boarding. I think that the answer to this question is based on the type of individual. From my perspective, it is easier for me to take notes on problems and go over it at a later time, individually. I felt some pressure when solving the problems in a group setting.
31
Perceived Comfort

I think that maintaining anonymity is very crucial in the interaction aspect of the discussion. Many, including myself, may feel a little embarrassed asking a "dumb" question but w/ this method, I don't feel that people will hesitate to ask those questions. The whole "instant messaging" system was cool, but seemed impersonal. Also, it felt intimidating to message the professor. It seemed to make more sense if we just asked the questions in person rather than messaging.
32
Perceived Engagement

I was definitely more prone to sit and give my full attention in this section than I am normally in any discussion. I did not fall asleep, where normally I will doze off during normal discussion. I think it's a lot easier to pay attention because I feel like I actually have to do the problem myself, rather than sit back and let some brainy kid figure it out for me, like I will tend to do when I feel lazy normally. Whenever we were assigned a problem to do, I always ended up taking out a piece of paper and pencil to write out the problem. Having the problem on the computer made it harder to see the whole problem because the screen was too small to fit the problem into the screen.
33
Typical Approach

• Whole-group instruction
• Difficult to get immediate feedback from students
• Feedback is usually only from a few students
• Not all students may be engaged