Experimental Design

1
(No Transcript)
2
Lecture 2

• Principles of Economic Experiments and
  Experimental Design

3
But 1stresults from last weeks 2nd experiment

• Goal was to find out the role our subconscious
  plays in decision making
• General result
• the 2nd guess in not necessarily better than the
  1st, but the average of the two is better than
  the 1st
• What this means is that even though you dont
  have any more info on the correct answer,
  subconsciously you know whether you over- or
  underestimated your 1st guess

4
But 1stresults from last weeks 2nd experiment

• Our result
• Remember, the actual distance from Vancouver to
  HCMC is roughly 11,500km
• Circumference of Earth 40,000km
• Distance from Earth to Moon 400,000km
• Average of 1st guess 228,051km
• Average of 2nd guess 180,498km
• Average of both guess 204,274km

5
But 1stresults from last weeks 2nd experiment

• Conclusions
• The subconscious is powerful, and we need to
  understand it better
• Why was our 2nd guess (and average of the two)
  better than our 1st when our 1st guess was
  supposed to be our best guess?
• Many students need to travel to get a better
  sense of distance
• Yet another reason to look into Study Abroad

6
Tutorials next week!

• Tutorials begin next week
• We will have our 1st experiment
• It is very important to show up on time
• If you have issues with the tutorial section you
  are in, come see me at the break
• Issues must be serious (i.e. scheduling conflict
  with other course/work)

7
Course Schedule
Date Lecture Tutorial
May 9 Introduction NO TUTORIALS
May 16 Experimental Design NO TUTORIALS
May 23 NO CLASSES Experiment 1 Markets
May 30 Data Analysis Qualitative NO TUTORIALS
June 6 Data Analysis Quantitative Experiment 2 Public Good
June 13 Markets How to use excel, Sample questions
June 20 Public Goods ASS1 due
June 27 MIDTERM NO TUTORIALS
July 4 Game Theory Experiment 3 Ultimatum
July 11 Social Preferences Experiment 4 Crime
July 18 Social ID Sample questions
July 25 Crime ASS 2 due
August 1 NO CLASSES NO TUTORIALS
August 8 Development NO TUTORIALS
8
Tutorials
Section Day Time Room TA
D101 Tues. 1030-1120 WMC 1651 Bakhit
D102 1030-1120 EDB 9651 Graeme
D103 1230-120 WMC 2220 Ji Hoon
D104 1230-120 BLU 11901 Bakhit
D105 1130-1220 K8664 Graeme
D106 1130-1220 RCB 6136 Bakhit
D107 330-430 WMC 2501 Ji Hoon
D108 330-430 RCB 6125 Bakhit
D109 Wed. 1030-1120 WMC 3510 Yang
D110 1130-1220 WMC 2268 Yang
D111 1230-120 WMC 1651 Ji Hoon
D112 130-220 WMC 2501 Ji Hoon
9
TA Information
Name Office Office hours email
Yang 3605 1230-130 (Wed.) ywa81_at_sfu.ca
Bakhit 3621 230-330 (Wed.) bemberge_at_sfu.ca
Graeme 2696 300-400 (Mon.) gmw1_at_sfu.ca
Ji Hoon 1655 330-430 (Thurs.) jhl14_at_sfu.ca
10
Principles of Economic Experiments
11
Big Questions to be Answered

• How do you choose and present the rules governing
  an experimental economy?
• How do you choose and motivate subjects?

12
I. Realism and Models
13
What is the goal of designing an experiment?

• To make the lab resemble the real-world as much
  as possible?
• Too complex
• The more complex the design of an experiment, the
  more expensive it is to conduct
• Reality has an infinite amount of detail
• Need to choose only the most important details
  relevant to the research question
• e.g. rules and rewards not fashion style and
  scent of air

14
What is the goal of designing an experiment?

• 2. To replicate the assumptions of the formal,
  theoretical model?
• Even if the observed behaviour of subject is
  consistent with the implications of the formal
  model, this only serves as weak support for the
  model
• It would be stronger if you had observed the same
  behaviour by relaxing some of the more stringent
  assumptions of the model
• e.g. of sellers in a competitive market

15
What is the goal of designing an experiment?

• 3. Your goal should be to find a design that
  offers the best opportunity to learn something
  useful and answer the questions that motivate
  your research

16
Analogy to Art

• An artist wishes to express a human event, say
  slavery
• He is unable to re-enact the event since it took
  place so long ago
• He finds it undesirable to replicate it closely
  for moral reasons
• He chooses a medium, say canvas or stone
• The quality of his painting will be judged by how
  well it simplifies reality to capture and
  communicate the essence of being a slave
• Likewise, an experiment should be judged by its
  impact on our understanding, not how close it
  replicates reality

17
Analogy to Art
The Captive Slave (1827) by British portraitist
John Philip Simpson
18
II. Induced-Value Theory
19
How does the experimenter gain control of the
subjects?

• Induced-value Theory
• Proper use of a reward allows an experimenter to
  induce pre-specified characteristics in
  experimental subjects
• With the proper reward, the subjects innate
  characteristics become largely irrelevant
• this is extremely important when we want to start
  analyzing and interpreting the results of an
  experiment

20
What are the necessary conditions to induce
subjects characteristics?

• Monotonicity
• Subjects always prefer more reward
• Dont choose a reward that people are bounded by
• e.g. pieces of cake, glasses of juice, or
  anything people can get full of
• NOTE the best and most commonly used reward is
  cash
• Easy to satisfy

21
What are the necessary conditions to induce
subjects characteristics?

• 2. Salience
• Relation between actions and rewards implements
  the desired institution
• Fixed payment (e.g. 5 to show-up)
• NOT SALIENT because payment does not depend on
  subjects actions
• Performance-based payment (e.g. 1 per point of
  profit earned)
• This IS SALIENT
• Salience is what differentiates surveys from
  controlled economic experiments

22
What are the necessary conditions to induce
subjects characteristics?

• 3. Dominance
• Subjects are only motivated by their reward (i.e.
  not motivated by what others are getting)
• Need for privacy
• This is why many experiments are conducted in a
  lab using computer terminals as the interface

23
What have experimenters learned from
induced-value theory?

1. To create a controlled economic environment, need
   to motivate subjects by paying them in cash
2. Average payment should exceed the average
   opportunity cost of the subjects
3. Find subjects whose opportunity costs are low and
   whose learning curves are steep (e.g.
   undergrads!)
4. Create the simplest possible economic environment
   in order to promote salience and reduce
   ambiguities in interpreting the results

24
What have experimenters learned from
induced-value theory?

• 5. Check instructions and verify subjects
  understand in dry runs or quizzes
• Avoid loaded words in instructions
• e.g. Prisoners Dilemma ? actions A B vs. Loyal
  Betray
• 7. Do not deceive or lie to subjects
• Salience and dominance are lost if subjects doubt
  the announced relation between actions and
  rewards

25
ANY QUESTIONS?
26
TIME FOR A BREAK

• COME BACK AT 130

27
Experimental Design
28
Introduction

• How we design our experiment dictates the
  questions we can ask and answer
• Last weeks Battle of the Sexes experiment was
  very simple in design, and thus could only make
  very limited comments on coordinating behavior

29
Introduction

• How could we have changed last weeks experiment
  to make comments on
• The effects of using hockey and ballet as the
  labels for the actions
• The effects of being punished for not coordinating

30
Introduction

• Consider the following experimental design

2 1 a b
A 3,3 1,1
B 0,0 3,3
2 1 hockey ballet
hockey 3,3 1,1
ballet 0,0 3,3
31
Introduction

• Now, consider the following experimental design

2 1 a b
A 3,3 1,0
B 0,-1 3,3
2 1 a b
A 3,3 1,1
B 0,0 3,3
NO PUNISHMENT
PLAYER 2 IS PUNISHED
32
What is the Goal of an Experimental Design?

• SHARPEN the focus variables and minimizing the
  BLURRING of nuisance variables

33
What is the Goal of an Experimental Design?

• Focus variable
• The few variables whose effects you are
  interested in
• This is, in fact, the point of the experiment!
• AKA Treatment variable
• e.g. the labeling of actions and the severity of
  punishment

34
What is the Goal of an Experimental Design?

• Nuisance variable
• Other variables that are of no direct interest,
  but may affect your results
• Types
• Controllable (e.g. sex, age, education, income,
  etc.)
• Uncontrollable (e.g. subjects interest,
  alertness, amount of fatigue)

35
I. Direct Experimental Control

• Constants and Treatments

36
How do we Sharpen Focus Variables?

• Focus variables are controlled for at 2 or more
  different levels
• Need to vary all treatment variables
  independently to obtain the clearest possible
  effects
• Need to ensure all possible explanations for our
  outcome of interest (i.e. ability to coordinate)
  are covered

37
How do we Sharpen Focus Variables?
Confounded Treatments
Independent Treatments
No punish w/ punish
A B observations observations
Hockey Ballet observations observations
No punish w/ punish
A B observations NONE
Hockey Ballet NONE observations
If we notice a difference in behaviour for the
treatments we have observations for, it is
impossible to know whether it was the labelling
or the punishment that caused it.
38
II. Indirect Experimental Control

• Randomization

39
How do we Blur Nuisance Variables?

• Uncontrollable nuisance variables can cause
  inferential errors if they are confounded with
  focus variables
• A variable is confounded if it is correlated with
  both the outcome variable and the treatment
  variables
• Independence among controlled variables prevents
  some confounding problems
• Need to ensure that any subject does not have a
  biased opportunity to be in a particular
  experimental session based on some controlled
  variable (e.g. sex)

40
How do we Blur Nuisance Variables?

• Randomization provides indirect control of
  uncontrolled nuisance variables by ensuring their
  independence of treatment variables
• EXAMPLE Role Assignment by Order of Attendance
• Subjects personal idiosyncrasies and habits are
  uncontrollable
• Dont assign early birds to one role and late
  comers to the other role
• Randomizing roles based on order of attendance
  ensures differences between players is due to
  their roles, not due to differences in subjects
  personal characteristics

41
How do we Randomize?

• Types of randomization techniques
• Completely Randomized
• Each treatment is equally likely to be assigned
  in each period of an experimental trial
• Quite effective when you can afford to run many
  periods
• Independence is established after many periods
• This can be improved upon (i.e. fewer periods)
  with the appropriate combination of control and
  randomization

42
How do we Randomize?

• Completely Randomized
• EXAMPLE

Period Treatment
1 AB, punish
2 Hockey Ballet, no punish
3 AB, punish
4 AB, no punish
5 Hockey Ballet, punish
43
How do we Randomize?

• 2. Random Blocks
• Difference from completely randomized design is
  that 1 or more nuisances are controlled as
  treatments rather than randomized
• Between Subjects
• Treatments are only varied across subjects
• Subjects only receive 1 treatment
• Our original Battle of the Sexes class experiment
  used this design
• Within Subjects
• Treatments are varied for each subject
• In other words, every single subject is exposed
  to every single treatment
• Subjects receive each treatment in a random order

44
How do we Randomize?

• Between Subjects
• ADVANTAGES
• Avoids carryover effects common in Within Subject
  Design
• Lowers the chances of subjects suffering boredom
  after a long series of tests
• Lowers the chances of subjects becoming more
  accomplished through practice and experience, and
  thus skewing the results

45
How do we Randomize?

• Between Subjects
• DISADVANTAGES
• Practicality
• Requires a large number of subjects to generate
  useful data since subjects are only exposed to 1
  treatment
• Individual variability Assignment bias
• Since subjects are only part of 1 group, it is
  difficult to control for all possible individual
  differences
• Environmental factors
• Usually arise from poor experimental design
• Suppose, for time reasons, you test one group in
  the morning and one in the afternoon
• Many studies show that most people are at their
  mental peak in the morning, so this will
  certainly have created an environmental bias

46
How do we Randomize?

• ii. Within Subjects
• ADVANTAGES
• This gives as many data sets as there are
  conditions for each subject
• Requires far fewer subjects than Between Subjects
  Design
• Provides a way of reducing the amount of error
  arising from natural variance between individuals

47
How do we Randomize?

• ii. Within Subjects
• DISADVANTAGES
• Carryover effects where the first treatment
  adversely influences the others
• e.g. Fatigue and Practice
• In a long experiment, with multiple conditions,
  the participants may be tired and thoroughly fed
  up of researchers prying and asking questions and
  pressuring them into taking tests.
• This could decrease their performance on the last
  study.

48
How do we Randomize?

• 3. Crossover Design
• Variation of Within Subject Design
• Used when you suspect your treatment variables
  have carryover effects (i.e. effects that last
  for some time)

49
How do we Randomize?

• 3. Crossover Design
• EXAMPLE
• Back to the Battle of Sexes to punish or not
• Suppose we are concerned that being in the
  punishment treatment (P) first will affect
  behaviour in the subsequent no punishment
  treatment (N)
• Simple Design NP and PN
• This design confounds time and learning with the
  treatment variables
• Crossover Design NPN and PNP
• Using this design, the difference in the average
  outcome for P and N indicates the effect of your
  focus variables

50
How do we Randomize?

• 4. Factorial Design
• Most important general method when you have 2 or
  more treatment variables
• More efficient than completely randomized design

51
How do we Randomize?

• 4. Factorial Design
• EXAMPLE
• 2 treatment variables A,B
• Levels AL,M,H, BL,H
• 3x2 factorial each of the 6 treatments
  LL,LH,ML,MH,HL,HH occurs k times
• If k4 then 3x2x424 trials are required

52
How do we Randomize?

• 4. Factorial Design
• Required number of trials increases quickly as
  the number of treatments (and levels of
  treatments) increases

53
How do we Randomize?

• 4. Factorial Design
• EXAMPLE
• Suppose we have 2 treatment variables, each with
  2 levels
• 22x2 24 16 trials
• Suppose we have 3 treatment variables, each with
  2 levels
• 22x2x2 28 256 trials
• NOTE both of these examples uses k1

54
How do we Randomize?

• 5. Fractional Factorial
• Alleviates the problem with the increasing number
  of trials
• Basic idea run a balanced subset of the
  Factorial Design
• Less robust than Factorial Design

55
How do we Randomize?

• 5. Fractional Factorial
• EXAMPLE
• 3 treatments, each with 2 levels ,-
• 8 possible treatments ,-,-,--,-,--,--
  ,---
• If you choose the first 4 or every other one then
  you will have unbalanced treatments because some
  variables are held constant or some pairs of
  variables are correlated

56
How do we Randomize?

• 5. Fractional Factorial
• EXAMPLE
• To get a balanced subset, use the following rule
• The 3rd element equals the product of the first
  two
• Balanced subset ,--,--,--
• NOTE the balanced subset of a 2x2x2 factorial
  design requires ½ the number of treatments (i.e.
  4 vs 8) and greatly reduces the number of trials
  (i.e. 16 vs. 256)

57
What are some of the chronic nuisances in
experiments?

• Experience and Learning
• Subjects behaviour changes over time as they
  come to better understand the lab environment
• learning is a nuisance when we want to test a
  static theory, but a focus when we want to
  characterize behavioural change over time
• When it is a nuisance, control it by
• Using only experienced subjects (CONTROL AS A
  CONSTANT)
• Using a balanced Crossover Design (CONTROL AS A
  TREATMENT)

58
What are some of the chronic nuisances in
experiments?

• Fatigue and Boredom
• Not all experiments are exciting to participate
  in
• Keeping experimental session under 2 hours is a
  good rule of thumb
• Using the occasional payoff switchover is another
  way to help keep attention
• Noninstitutional Interactions
• Subjects behaviour may be affected by
  interactions outside the lab
• e.g. Battle of Sexes players getting together at
  a break and coordinating on a strategy
• Careful monitoring during the break or a change
  in parameters after the break is advisable

59
What are some of the chronic nuisances in
experiments?

• 4. Selection Biases
• When subjects or behaviour is unrepresentative
  because their selection was biased
• Biased implies not random
• Problems with self-selection
• e.g. recruiting subjects from a advanced finance
  course for a market trading experiment
• Subject/Group Idiosyncrasies
• A group of subjects may somehow reinforce each
  other in unusual behaviour
• Need replications with different subjects

60
Dont forget, tutorials start next week!