Development and Evaluation of Audience Response Systems and ComputerInstruction Tutorials

1
Development and Evaluation of Audience Response
Systems and Computer-Instruction Tutorials

• Paul K. Schick, MD
• pksone_at_comcast.net
• Lankenau Hospital and Drexel University,
  Pennsylvania

Mary Burke, MD and James Burke, MD Lankenau
Hospital, Pennsylvania
Support Sharpe-Strumia Foundation and the
Kitchen Foundation
2
Audience Response Systems
A quarter is worth
7

• 5 cents
• 25 cents
• 10 cents
• 50 cents
• 75 cents

3
Merits of Audience Response Systems

• During presentations and lectures
• Participants - Greater interest and
  participation
• Lecturers Aware of audience comprehension
• Can address gaps of knowledge.

• Post-hoc analysis of responses
• Quality of questions
• Detection of strengths and weakness in baseline
  knowledge
• Evaluate comprehension of lecture
• Demographics and assessment of competency

Software for post-hoc analysis was written by
Paul K. Schick
4
Lecture on Hematologic Emergences (Audience
Response System)
Twenty topics were covered, such as indications
for transfusions. Since residents should have had
experience with hematologic emergencies,
questions were asked prior to the discussion to
determine baseline knowledge. 24 residents and 8
medical students.
Feedback (Scale of 1 to 5 1 Strongly agree)
Will help me manage hem/onc emergencies 1.47
Audience Response Systems improve lectures 1.22
5
Post-hoc Analysis

• Difficulty
• Ideal difficulty 50 to 85
• Difficult
• Easy 85

• 2) Discriminative value
• Grades were sorted.
• Ratio of correct answers - Upper/Lower Quintile
• A discriminative question Value 1.5

6
Post-Hoc Evaluation
Evaluation of questions Average Difficulty
52.6 (Ideal range) Average Discriminative
value 2.73 (Very high)
7
Post-hoc Evaluation of Individual Questions
Discriminative Value (DV)
8
Use of Data to Improve Presentations and to
Strengthen Our Educational Program

• Improving the quality of questions
• Ambiguous questions was improved or deleted.
• Improving the medical school curriculum. For
  example, addressing the following
• Medical students were not adequate taught to
  consider cardiac symptoms when deciding to
  transfuse patients.
• Improving our resident training program For
  example, addressing the following
• Medical students were better instructed to manage
  a common complication of sickle cell anemia.

9
Baseline Knowledge Demographics
ANOVA and Scheffe Analysis n 32
a,
b
a,b
10
Testing for Competency
Outstanding (Mean 1 SD Poor (Mean 1 SD)
11
Blood Cell Morphology (Audience Response System)
18 Residents and 6 Medical Students Subject
matter not familiar to residents Asked questions
prior to discussing topics to test for baseline
knowledge Asked questions after discussing topics
to test for comprehension
12
Questions Prior or After Discussion of a Topic
Prior to showing slides and discussing
topics Appropriate for residents Can
establish baseline knowledge After showing
slides and discussing topic Appropriate for 2nd
year medical students Can assess comprehension
topics
13
Audience Response Systems

• A) Audience interest and interactivity
• B) Post-hoc analysis
• Quality of questions (critical for interactive
  programs)
• Quality of lectures and conferences
• Baseline knowledge
• Comprehension of lecture and conferences
• Accreditation

14
Computer-Assisted Instruction

• We demonstrated two computer-based tutorials
  on Bleeding Disorders and on ECG interpretation
  during the presentation.
• Our study on the tutorial on ECG
  Interpretation is In Press in Medical Teacher.
  The article is entitled Critical analysis of a
  Computer-Assisted Tutorial on ECG interpretation
  and its ability to determine competency.

15
Comparison of Approaches
16
Contact Information

• Please contact me if you have any question or
  any interest in collaborating on a computer-based
  educational project.
• Paul K. Schick, MD
• pksone_at_comcast.net