In search of

1
In search of

• Slow evolution (going on two years) includes a
  very large scale pilot study (i.e., scrapped
  data) that led to control improvements and the
  current data.
• The long process of communicating a vague goal
  (within two people) at what point is it worth
  consuming others time??? We debated this
  question as recently as last night.
• More rigorous write-ups (with context) exist we
  were shooting for a write-up you could read in
  under a ½ hour.

2
The Experiments Timeline
Both AB form a final three percentages (after
viewing the partners suggestion)
Subject A forms a sample estimate
ABs sample estimates are mechanically combined
Both AB can suggest revisions the three
percentages (goal match Bayes)
Final is compared to Bayes, profit determined,
feedback given to both partners then a new round
begins (with a new sample)
Subject B forms a sample estimate
3
Intuition vs. Problem Solving
Ian
Gerd
Leonard
4
Do the sample estimates get better over
time? (all probabilities are stated in high
side termscolumns are absolute error relative
to Bayes normatively correct response)
Above, each subject sees each problem in four
rounds (but the exact problem is disguidedBlack
vs White Marble, High vs Low)
5
(No Transcript)
6
Reversion / non-reversion
Interesting Small samples are very
volatilebut big vs. small results are quite
different!
Yellow is Big Sam 75/67 Green is Big Sam 50/60
7
Control choices tradeoffs

• The small sample volatility is interesting
• we have some combinations that allow for rigorous
  (or somewhat rigorous) comparisons
• 2 of 3 ? 3 of 4
• 2 of 3 ? 2 of 4
• 3 of 4 ? 3 of 5
• we also match big small simple probabilities
  (75, 67, 60, 50) for somewhat valid
  comparisons
• The cost of the preceding benefits is lost
  tractability meaningthe potential for
  undesigned / unwanted confounds when attempting
  to systematically look at collaboration phase
  using combined evidence (see next slide)

8
Turning four complete rounds into psuedo-eight
rounds.
ANY systematic difference in these four problem
matches means we need to think about unwanted
confounds.
Dynamic issues the possibility of workers
signaling to the boss (we currently see zero
evidence of thisso we treat worker boss roles
identically). We decided to keep individual
problems matched (which still seems like a good
idea).
9
These are starting points during each
psuedo-round formed by mechanically combining
the assessors probabilities.
Obviously, the second combination has relatively
big percentage error ( error is the combined
odds vs. Bayesian correct odds). One reason is
that problems 2 and 8 are relatively pure with
respect to always reverting to 50 (assessor odds
are outside Bayes only 5 0, respectively).
This virtually eliminates the possibility of
offsetting error reduction.in contrast, Ps35
have the largest chance of offsetting error (41
of all assessments are outside Bayes)not
definitivebut some evidence. The pattern of no
learning does not differ among problem
combinations.
10
Round Sequence
Both AB form a final three percentages (after
viewing the partners suggestion)
Subject A forms a sample estimate
ABs sample estimates are mechanically combined
Both AB can suggest revisions the three
percentages (goal match Bayes)
Final is compared to Bayes, profit determined,
feedback given to both partners then a new round
begins (with a new sample)
Subject B forms a sample estimate
11
A Difficult Mechanical Problem?

• Our subjects get no better over-time (i.e., over
  rounds) at the mechanical starting point for
  matching the Bayesian optimal (recall, matching
  the Bayesian three percentages is the
  compensation function).
• This is not terribly surprising given that
  subjects individual sample estimates got no
  better over time however, improvement could have
  happened through some subjects offsetting their
  partners errorsthus keeping individual sample
  estimate error high, but combined three
  percentage error (and thus total compensation)
  might decrease.
• In our view, the preceding are the mechanical
  solution possibilitiesin other words, if I want
  to do better (earn more money) in this task, and
  I view the task as a mechanical problem, I must
  either (1) improve my initial sample estimate or
  (2) compensate for error in my partners sample
  estimate through my sample estimate.

12
Vocabulary Word Final Nodes
Both AB form a final three percentages (after
viewing the partners suggestion)
Subject A forms a sample estimate
ABs sample estimates are mechanically combined
Both AB can suggest revisions the three
percentages (goal match Bayes)
Final is compared to Bayes, profit determined,
feedback given to both partners then a new round
begins (with a new sample)
Subject B forms a sample estimate
13
Why should the final nodes matter?
14
Intuition
15
By, Gerd!...he might be right
negative you should have done nothingbad
movement
positive good move! ? ? ?
16
Pseudo-rounds (8) collapsed into real rounds (4)
because the graph looks scary-jagged when you
dont do so
17
You can see both of the highlighted results in
the graph (these are all 8 psuedo rounds in
spite of the scary jaggednessthe result is quite
significant)
18
Huh? Why did that happen? How did that
happen?What happened?
19
The Debate Over Whether or Not We Were (Are?)
Far Enough Along to Present This Thing

• Stanovich West (2000) find correlations between
  intelligence and base-rate problemswe have some
  support for this sort of thing via analyzing
  subject GMAT scores.
• We have some analysis on how good groups and
  bad groups did different things with the three
  probabilities. Specifically, good subjects
  seem to figure out they must increase the odds of
  one black one white marble bag. Its a good
  heuristichow do they know to do it? (Blink?)

20
The GMAT Story

• Each team experiences the four team problems
  eight times (recall the psuedo rounds).
• For each team problem, both partners provide a
  final three probabilities, the final from the
  partner playing the boss role is used for team
  compensation (i.e., compared to Bayes three
  probabilities)but partners rotate who is in the
  boss roll throughout the experiment (nuisance
  control manipulation).
• Although there are game-theoretic reasons why the
  final would change depending on the role (e.g.,
  the worker might try to signal the boss), we find
  no differences based on rolein other words, each
  round, subjects seem to simply provide their best
  final probabilities regardless of the role they
  are playing.

21
The GMAT Story (continued)

• Questions?
• Anyway, it seems groups who disagree about the
  final assessment (no teamwork or learn from
  each other or some similar concept) might be
  different from groups who do, generally agree on
  the final.

22
(No Transcript)
23
(No Transcript)
24
The within-group final difference is affected by
both the teams AVG GMAT (high vs. low ADIF
poor label) and by the difference between team
member GMATs (big vs. small BDIF. Sadly, the
between subjects stuff isnt significant
25
Using average GMAT as a continuous control
(rather than a high vs. low fixed factor), the
between subjects GMAT difference result becomes
marginally significant you can sort-of see how
via in the graphif you imagine the dichotomous
AVG GMAT is representative of the continuous
variable.
26
Does any of that consistency difference translate
into performance difference? The overall error
drops across timebut it doesnt differ by high
vs. low GMAT
27
Power Outage?
Sadlynothingyou can pick up a significant w/in
subjects effectbut not a between subjects one
(even if you eliminate the 1st or the 1st two
pseudo rounds).
28
error by round movement (initial to final) by
round has a really cool correlation chart (hard
to fit those in Powerpoint).but at 150the
analysis must stop transition to other
analysis