Checklists%20for%20Reporting%20Social%20Science%20Experiments

1
Checklists for Reporting Social Science
Experiments

• Alan Gerber
• Dave Doherty
• Conor Dowling

2
Outline

• Definition checklist
• A list of items in order that should be
  reported/performed.
• 1. The benefits of checklists generally, slow
  adoption and resistance
• 2. The benefits of checklists in research
• 3. Checklists for reporting medical research
  (CONSORT).
• 4. Compare and Contrast Social Science and
  Medical Research how do the differences affect
  reporting priorities?
• 5. Modifications of CONSORT for social science
  research
• 6. Next steps

3
Benefits to Checklists in General/Story of
Checklists

• Checklists used in situations where a
  complicated, step by step task is attempted.
• Pilot safety check.
• Medicine.
• Checklists work but often not used even when
  beneficial.
• Somewhat odd situation.

4
Benefits to Checklists in Research

• Suppose the checklist said that in section 1
  there should be a clear description of the sample
  (from initial sample through all
  exclusions/attrition, etc.), sample means, and a
  clear description of the treatment and treatment
  assignment.
• Readers will be provided the information they
  need to evaluate the work.
• Researchers will know what they will need to show
  and so will be careful to take necessary steps.

5
Checklists in Medicine

• The problem of inadequate reporting was
  widespread in medical research and a couple of
  groups got together to address the problem,
  merged.
• Result CONSORT statement - checklist.

6
The Consort List
Appendix A

7
Comment

• Basically this is great.
• Lots of things here. Some seem a bit superfluous,
  but it cant hurt.
• Can we make it better for social sciences?
• What should be added/given greater emphasis?

8
Medicine versus Social Science

• Sources of Bias, Problems of Generalization. A
  few ideas.
• 1. Biochemistry versus Choice.
• For Choices objectives, actions/resources,
  beliefs (connect actions to outcomes).
• If the outcome variable is a choice, the
  treatment effect may depend on all of these
  things.
• heterogeneous treatment effects
• Implication may need more detailed accounting of
  these conditions to interpret results. Lots of
  context, boring as it is. Perhaps a standardized
  list of things to be placed into the appendix?
  Theory and prediction?

9

• 2. For field contexts, social processes may be
  important.
• Not clear that there is spillover type problems
  in randomized medical studies.
• Implication clear reporting of opportunities for
  treatment group to contaminate control group,
  discussion of magnitude of bias, evidence related
  to this.

10

• 3. Subtle strategic behavior by subjects makes
  outcome measures tricky.
• If the outcome measures are not comprehensive, it
  may be hard to accurately interpret the treatment
  effect.
• Relatively clear in medicine, but what about
  social science?
• Medicine overall mortality, days in hospital
• Social Science example experiment on effect of X
  on fraud at the polling place. The opportunities
  for fraud include unregistered voters voting,
  ballot stuffing, poll worker manipulation, etc.
• Implication Report the range of important
  behavioral effects predicted, whether measured or
  not. Try to measure as many as possible.

11

• 4. Non-standard treatments.
• Drug tests 50 mg of X.
• Social Science Experiment Exposure to Mailing
  that I designed myself.
• In medicine, there are many fewer studies of
  devices and surgery than of drugs, perhaps
  because of this problem at least in part.
• Implication Provide treatment materials. Perhaps
  minimize treatment innovations across studies
  (avoid gratuitous innovations).

12

• 5. Ex-ante versus ex-post hypotheses.
• Its a matter of degree, but the endpoints and
  subject populations are generally more obvious in
  medical studies than social science research.
• Implication Clearly report ex ante hypotheses
  versus ex post hypotheses.
• 6. List of Other differences

13
Conclusion

1. Checklists appear to work but rarely spring up
   and are slow to catch on.
2. There are excellent starting points for social
   science reporting already.
3. The challenges to the validity and
   generalizability of social science research are
   different in some ways from medical research.
4. Modification/Additions to the medical research
   checklist can produce a useful checklist for
   social science research.
5. Talk focused on experiments. The paper version
   discusses observational work as well.