Designing Electronic Performance Support Systems to Facilitate Learning

1
Designing Electronic Performance Support Systems
to Facilitate Learning

• PSYC 512
• October 20, 2005
• Christina Abbott

2
Citation

• Van Schaik, P., Pearson, R Barker, P.,
  Designing electronic performance support systems
  to facilitate learning. Innovations in Education
  Teaching International, 39(4), 289-306.

3
Why This Article

• Short
• Has some glaring problems
• Relates to terms we know
• Provides a decision-making tree for descriptive
  and inferential statistics.

4
What an EPSS Should Do?

• Shiak, Pearson, and Barker state that an EPSS1
  facility designed for educational need should
• Assess skill fill in the gaps in performance
• Act as a transfer agent by using on-demand
  mechanisms.
• Act as an augmentation aid improve performance
  over and above natural ability
• 1EPSS Electronic Performance Support System

5
Design of This EPSS

• Goal To design a learning tool for psychology
  students that aids in their understanding of
  various statistical methods.
• Step 1 Needs Analysis
• Step 2 Design and Development
• Database
• Help System
• Advisor System
• Personal Area and Notes

6
Advisor System

• The Advisor System is a series of decision-making
  trees. The one below is for descriptive
  statistics.

7
Advisor System

• The decision-making tree below is for inferential
  statistics.

8
Purpose

• There were three main purposes of the study
• Analyze students knowledge of concepts in
  quantitative research methods.
• Establish students performance with system.
• Establish students overall acceptance of the
  system as a learning aid.

9
Hypothesis

• The researchers could have increased the
  criticality of the study by including a
  hypothesis for each of the above purposes.
• For example
• Researchers could have developed a hypothesis for
  the performance of a group who did not use the
  EPSS to answer knowledge questions (control
  group) with a group that has full access to the
  system.

10
Research Design

• Participants
• Sample of 89 sophomore psychology students
• Students had taken two research classes in their
  first year of study
• Three parts to the research
• Knowledge of Concepts in Quantitative Research
  Methods
• Use of the EPSS
• System Acceptance

11
Phase 1 - Knowledge of Concepts

• Independent Variables the test questions
  (nominal)
• Dependent Variables the answers (ordinal)
• Confounds
• confidence level with computers and prior
  knowledge of statistical methods
• There was no control group specified, i.e. one
  that answered the questions without using the
  EPSS.
• Could have made the study a within subjects
  design and measured their knowledge before and
  after using the EPSS.

12
Phase 2 Use of the EPSS

• Independent Variables the test questions
  (nominal)
• Dependent Variables the answers (ordinal)
  (either correct or found)
• Confounds
• confidence level with computers and prior
  knowledge of statistical methods were confounds.
  
• there was no control group specified, i.e. one
  that answered the questions without using the
  Advisor, Help, and Tools.
• Could have made the study a within subjects
  design and measured their knowledge before and
  after using the tool.

13
Phase 3 System Acceptance

• Survey was a 7 point Lickert Scale with values
  ranging from strongly agree to strongly disagree.
• Independent Variables the survey questions
  (nominal)
• Dependent Variables the answers (ordinal)
• Confounds
• The students success in Phase 1 and 2 may
  contribute to their overall system acceptance.
• In previous studies I have done, when the user is
  asked to rate the performance of a system, they
  are really rating his or her success with the
  system. These users had low expectations of the
  system to begin with and expected to struggle.
  Satisfaction level self-reports can be
  misleading.
• There are more probing questions the researchers
  could have asked, such as where did you get
  stuck and when were you lead astray by the
  system.

14
Hypothesis Implications

• The null hypothesis was not stated at the
  beginning of the study. 
• Discussion Section Statements implied that the
  researchers, though misguided, may have had a
  hypothesis in mind.
• The significant correlations between acceptance
  measures confirmed the association between
  intention of use and perceived usefulness found
  in previous research.
• The researchers also cited a correlation, but did
  not show the data to back up their claim that
• Overall students performance using the EPSS was
  higher than unaided performance control group
  on knowledge questions.
• The researchers could have easily correlated the
  relationship between performance and confidence,
  as they had already gathered the data.

15
Results

• There was scope for improving students
  performance.
• Students performance using the EPSS was high.
• Students performance using the EPSS was higher
  than unaided performance on knowledge questions.
• Non-significant correlation between students
  performance in terms of knowledge of concepts and
  success in using the EPSS.
• Students expected that the EPSS would be useful
  to them for learning, revision, and completing
  assignments.

16
Parting Thoughts

• Ways to improve this study
• State the hypothesis clearly.
• Have multiple hypothesis to aid in the
  criticality of study.
• Correlate performance with acceptance this is
  more powerful than each alone.
• Ways to improve the article
• Do not use the term - confirm.
• State all aspects of the research design -
  include any control group information.
• Median measure used for Lickert Scale Data ?