Evaluating the Effectiveness of Problem Based Learning

1
Evaluating the Effectiveness of Problem Based
Learning
DEE Conference Cambridge 6-7 September 2007

• John Sedgwick and Guglielmo Volpe
• Department of Economics, Finance and
  International Business
• London Metropolitan University
• j.sedgwick_at_londonmet.ac.uk, g.volpe_at_londonmet.ac.u
  k

Research funded and supported by the Economics
Network of the Higher Education Academy
www.economicsnetwork.ac.uk
2
Structure of Talk

• Aims of Project
• Evaluating Problem Based Learning the literature
• Structure of PBL Project
• Evaluation of Experience
• Evaluation of Performance
• Students Perception
• Staff Evaluation
• Projects Next Steps

3
Evaluating PBL the literature

• Evaluation of Outcomes
• Knowledge mixed results or insignificant
  differences between PBL and other methods
  (Albanese and Mitchell, 1993) greater knowledge
  with PBL (Smits, 2002), (Farrell, 2003)
• Lifelong Learning Skills PBL superior to
  conventional methods (Bransford et al., 1989)
• Learning Process deeper approaches to learning (
  Sobral, 1995)
• Team Skills PBL facilitates development of
  collaborative skills (Cockrell et. al., 2002)
• Control Group Analysis in some contexts PBL may
  lead to worse outcomes for some students (Newman,
  2004)

4
Control Group Analysis

• Structure of Project
• Industrial Economics Module
• Spring Semester Module
• Taught at North and City Campus to two separate
  cohorts of students
• PBL at North Campus while Traditional Approach at
  City Campus
• Different lecturers in the two campuses
• Same assessment but slightly broader syllabus at
  City campus
• Control for students background and
  characteristics
• Evaluate and compare students experience and
  performance

5
Control Group Analysis

• Methodology evaluation framework (Newman, 2004)

Objective Measure
Students participation Class registers Class participation Logs of meetings
Students satisfaction Questionnaires at end of semester Focus group Module evaluation form
Students outcomes and students performance Assessment performance End of semester questionnaires Statistical analysis of performance Follow up to test knowledge retention
6
Students Performance
Average Median F D C B A
Overall 57.9 59.5 15.0 5.0 30.0 40.0 10.0 City
55.7 58.0 7.9 13.2 39.5 23.7 15.8 North

Exam 57.3 63.5 12.5 12.5 16.7 33.3 25.0 City
54.6 56.5 10.5 18.4 26.3 28.9 15.8 North

Coursework 53.9 51.0 15.0 25.0 15.0 25.0 20.0 City
56.0 56.5 10.0 5.0 42.5 20.0 22.5 North
7
Students Performance
Does experience in PBL lead to a better
performance?
  Coefficients t Stat P-value
Intercept 55.90439 17.11719 2.38E-17
Difference in Year 23 average 0.900969 5.500858 5.11E-06
PBL in Semester A 7.079753 2.183598 0.036683
Mature student -5.45856 -1.68865 0.101323
International student -4.16997 -1.4207 0.165385
Disability -8.72376 -1.52794 0.136667
Semester B starter 4.960657 1.216063 0.23314
Regressor Overall performance in module by North
Campus studentsOLS method R20.650, Adjusted R2
0.582 Observations 38
8
Students Performance
Is PBL correlated with a better performance in
the module?
  Coefficients t Stat P-value
Intercept 60.76463 27.27263 1.78E-32
Difference in Year 23 average 0.947212 7.512146 7.56E-10
PBL at North Campus 0.598059 0.223027 0.824388
Mature students -8.10651 -3.36355 0.001451
International students -4.69812 -1.75774 0.084677
Semester B starters 1.736257 0.457838 0.648976
Regressor Overall performance in module by all
studentsOLS method R20.576, Adjusted R2
0.535 Observations 58
9
Students Evaluation
Students evaluation at end of semester
North campus (PBL) North campus (PBL) City Campus City Campus
Overall Score 75 8 68 10
1. Lecturer well organised 97 0 83 0
2. Timely assessment information 97 0 92 8
3. Clear presentation 83 0 69 15
4. Appropriate level of difficulty 68 6 77 8
5. Lecturer encourages questions 90 0 85 0
6. Clear syllabus and assessment 77 0 54 8
7. Right number of topics 84 0 62 8
8. Module recommended 70 7 54 15
9. Tutorial complements lecture 77 0 77 15
10
Students Evaluation end of semester
questionnaire
City Campus Students (15) City Campus Students (15) City Campus Students (15) North Campus Students (33) North Campus Students (33) North Campus Students (33)
SA A SD D S A A S D D
I would have liked to have experienced PBL (traditional method) in this module 53 33 33 30
I prefer the traditional lecture/seminar (PBL) approach to any other method 53 20 42 18
A PBL (traditional approach) approach would be appropriate for this module 53 33 52 24
Final year students should be required to learn more independently (less lectures/seminars) 43 43 64 24
I expect to learn more than the North (City) campus students 33 13 33 6
I will be able to retain more knowledge of IE than the NC (CC) students in the future 20 13 47 6
All students should follow the same teaching method 67 13 70 9
A PBL approach is more appropriate for final year modules 53 20 70 12
How much/well you learn does not depend on the teaching method 47 47 23 47
11
Students evaluation

• Students views from questionnaires and focus
  group (PBL students)
• You are understanding while learning, as usually
  in lectures you do not retain the information
• I hear and I forget, I see and I remember, I do
  and I understand
• I believe with PBL students are forced to
  independently attain a wider knowledge because
  they need to research more and by so doing we
  come across more things than we would get in
  class teaching
• PBL is good as it relates to real world issues
  and that tends to stay in my head. Group learning
  is also important as it is easier to understand a
  discussion than a lecture
• PBL is not necessarily better. The tutor who is
  delivering the module and his ability to teach is
  important!
• PBL is not good at all especially for final year
  students who need much more support at that time

12
Staff Evaluation

• For me, these results are much stronger than I
  was getting in the module a few years ago, so I
  feel very encouraged
• The experience was such a good one in terms of
  classroom buzz
• I enjoyed the experience and will repeat it next
  year
• The PBL experience induced me to introduce the
  approach in other modules

13
Knowledge Retention

• Both City and North Campus students were
  contacted after completion of moduleThey were
  asked to answer three questions without
  consulting any notesAttempts were marked blindly
  by both City and North campus tutors

Evaluation of post exam knowledge retention test (mark out of 5) Evaluation of post exam knowledge retention test (mark out of 5) Evaluation of post exam knowledge retention test (mark out of 5)
Modules mark City Tutor North Tutor Average
City Student 69 2 2 2
North Student 1 67 5 4.6 4.8
North Student 2 67 2.5 3.3 2.9
North Student 3 58 2.5 3 2.8
14
Overall Evaluation
Green Yellow Red
Positive correlation between PBL and performance Students appreciation of approach Students perception of deeper / independent / dynamic learning Students desire to be involved with approach Students perception that approach pushes them more than other approaches Staff satisfaction and appreciation of its value Students development of transferable skills Working with others Self-directed learning Knowledge retention and deeper understanding Differences in CW and exam performances but not statistically significant PBL students seems to retain more knowledge Statistical correlation between performance and PBL not significant Lack of prior experience in PBL can hinder performance Difficulty in handling a large class
15
Steps to complete research

• Still two steps to complete research
• Full analysis of focus group with identification
  of key themes
• Further evaluation of questionnaires by
  distinguishing responses by gender and age

16
Overall Evaluation

• The PBL students perform better in the coursework
  but less well in the exam compared to the non-PBL
  students
• Overall the non-PBL students have performed
  better but the difference in performance is not
  statistically significant
• Students who experience PBL in the first semester
  perform better than those without such an
  experience
• PBL is a positive but not significant determinant
  of performance across the two campuses
• The majority of students appreciate the value of
  the PBL approach in particular for final year
  students
• About half of the City students would have liked
  to experience PBL in the module
• Students express some concerns about their
  ability to engage with PBL without having any
  prior experience
• Staff enjoy the approach and the ability to be
  much closer to the students learning process