Experimental Research methods

1
Experimental Research methods

• Dr. Geof Staniford

2
Introduction

• In this talk we are going to consider three types
  of experimental design
• Pre-Experimental Design
• True Experimental Design
• Quasi-Experimental Design
• and relate them to IS Research through the medium
  of a simple IS example.

3
Outline of Our Approach

• Briefly consider the underlying philosophy behind
  experimental design
• Describe the IS example
• Explore the experimental design methods in some
  detail
• Compare and contrast the methods
• Point to other ideas worth investigation

4
Philosophy Underlying Experimental Research

• Experimentation is a process of observation , to
  be carried out in a situation especially brought
  about for that purpose.
• The embryology of the chick - (Aristotle BCE 340)
• Scientific observation is controlled
  investigation (Nagel 1961)
• Experiments must strike a balance between
• richness of worldly realism
• tightness of control
• Experiments are undertaken to produce new
  knowledge
• The practical use of knowledge establishes its
  value
• knowledge is what works or satisfies us (James
  1907)
• New knowledge is derived by reasoning from
  experiments

5
Reasoning From Experiment
Scope

Inference
Reliability
Real World
Experiment
Results
Theory
Knowledge
6
Information System Example

• The main feature of experimental research is the
  deliberate control and manipulation of conditions
  (one or more independent variables), the
• model of the natural sciences.
• We will be interested in the situation in which
  we wish to investigate whether changes in font
  size of the on screen text window influence the
  effects of fatigue that occur when users carry
  out routine tasks using a graphical user
  interface.

Cause
Effect
7
Pre-Experimental Design

• The One Group Pre-Post Test
• O1 Observes/measures the baseline level of the
  dependent variable before exposure to the
  treatment condition (e.g. task efficiency
  level).
• X Exposure to the treatment condition/independent
  variable (e.g. reduce window font size).
• O2 Observes/measures the level of the dependent
  variable (task efficiency level) after exposure
  to treatment condition.

Observation Measurement
Experimental Treatment
Observation Measurement
O1
X
O2
8
Pre-Experimental Design

• Problems With the One Group Pre-Post Test
• IF Treatment is given to Subject
• THEN Response occurs.
• T IMPLIES R (modus ponens)
• vulnerable to the logical fallacy of affirming
  the consequent.
• Logic tells us that a false antecedent can imply
  a true consequent.

9
True Experimental Design

• Pre-Post Test Control Group Design
• Experimental Group O1 X O2
• Control Group O1 O2
• There are two major features of this design
• Randomisation ensures that variation between the
  subjects is equalised between the groups
• Control Group ensures that the conclusions drawn
  about the experimental group and the effect of
  the independent variable are valid

10
Pre-Post Test Control Group Design

• Problem
• the control group O2 measure may be influenced
  by the prior exposure to the O1 measure.
• Pre-Post Test Control Group Design (Var 1)
• Experimental Group O1 X O2
• Control Group 1 O1 O2
• Control Group 2 O2

11
Pre-Post Test Control Group Design

• Pre-Post Test Control Group Design (Var 2)
• Experimental Group 1 X O2
• Experimental Group 2 O1 X O2
• Control Group 1 O1 O2
• Control Group 2 O2
• This adds a further level of sophistication and
  internal checking of the possible effect of the
  O1 observation on the experimental group.

12
Quasi-Experimental Design

• Non-Equivalent Control Group Design
• This design uses experimental and control groups
• Often we have to use existing (intact) groups
• These groups are technically not equivalent
• because they have not been equated using
  randomisation procedures
• Disadvantage
• the performance levels of the groups may be
  different irrespective of, or in addition to, any
  effect of the independant variable

13
Further Topics

• Validity of experiments
• internal validity
• external validity
• Laboratory experiments in IS
• Experiments on individuals in IS
• Small group research in IS
• Field experimentation in IS

14
Discussion Topic

• Validity in Information Systems Experiments