Beyond Publication Bias: PrecisionEffect Testing using MetaRegression Analysis T'D' Stanley Tom Hend

1
Beyond Publication BiasPrecision-Effect Testing
using Meta-Regression AnalysisT.D. Stanley
(Tom)Hendrix College, USA Stanley_at_Hendrix.edu
(P)ublication bias is leading to a new
formulation of Greshams lawlike bad money, bad
research drives out good. Bland (1988, p.
450) The controversy over the effects of
antidepressants (front page, Aug. 7) illustrates
a problem resulting from the intense
commercialization of medical research -
publication bias. When huge financial gains
hinge on showing success for a drug in a clinical
trial, negative findings may be withheld from
publication. D. Cohen, NY Times, 2003.
2
Many other commentators have addressed the issue
of publication bias. . . . All agree that it is
a serious problem. Begg and Berlin (1988,
p.421)

• A. On the Importance of
• 1. Doing Meta-Regression Analysis
• 2. Correcting Publication Selection Bias

3

• 1. Doing Meta-Regression Analyses (MRA)
• MRA can filter out ubiquitous publication and
  misspecification biases and detect genuine
  empirical effects that might lie buried
  underneath.
• A Meta-meta-analysis of 50 economics
  meta-analyses reveals the vast majority have
  substantial or severe pubbias.
• Meta-Regression Analysis serves as scientific
  literature review. (Stanley, 2001)

4

• 2. Correcting Publication Selection Bias
• a. File drawer problem.
• b. Register or Perish! The Paxil and Vioxx
  scandals have forced leading medical journals to
  require prior registration of trials. Drug
  Companies demand results
• c. The Journal of Economic Surveys has devoted a
  Special Issue to Meta-Regression Analysis
  Issues of Publication Bias in Economics (July
  2005).
• d. Publication biases can make empirical effects
  seem much larger than they are this has policy
  implications. In economics, for example, there
  is a 3 or 4-fold exaggeration of water
  elasticities and a 10-fold exaggeration of common
  currency effect on trade (Stanley, 2005a Rose
  and Stanley, 2005)
• . . . So the UK may be wise not to
  join the European Monetary Union

5

• B. Picturing Publication Bias
• 1. Funnel Graphs
• The simplest and most commonly used method to
  detect publication bias is an informal
  examination of a funnel plot. Sutton et al.
  (2000b, p.1574)
• A funnel graph is a scatter diagram of precision
  (1/Se) vs. estimated effect.
• In the absence of publication selection,
  estimates will vary randomly and symmetrically
  around true effect (b).
• The expected inverted funnel shape is dictated by
  predictable heteroskedasticity.

6
Figure 1 Funnel Plot of Union-Productivity
Correlations source Doucouliagos, C. and
Laroche, P. (2003)
7
Figure 2 Funnel Graph of Price Elasticity for
Water Demand source Dalhuisen, J. et al.
(2003)
8
Figure 3 Funnel Graph of Common Currency Trade
Effect Source Rose, Andrew K. Stanley, T.
D. (2005)
9
C. Meta-Regression Analysis (MRA)

• 1. Basic Framework/MRA Model
• effectj b b0Sej SakZjk ej (j1, 2,
  L) OLS (1)
• tj b0 b (1/Sej) SakZjk/Sej vi
  WLS (2)
• The conventional t-test of the intercept of
  equation (2) is a test for publication
  biasfunnel asymmetry test or FAT ? H0 b0
  0.
• Testing b0 may be considered a test of
  authentic effect, corrected for publication
  selectionprecision-effect test or PET ? H0 b
  0.
• Zjk are moderator variables, which measures
  relevant characteristics of a specific empirical
  study and are used to explain its systematic
  variation from others in the literature.
• Equation (1) can be derived from a Heckman
  regression (Doucouliagos and Stanley, 2006).

10
2. Connection to Funnel Graph

• Invert Compress ?

Gives MRA Equation (1)
Reverse axes ?
11
ti b0 b (1/Sei) ei

• 2. FAT-PET-MRA-- Examples

FAT
PET
12
Simulations1. Power and level of
precision-effect testing (PET)
13
2. Two-Stage precision-effect (PETS)

• effectj b b0 ej (j1, 2, L)
  OLS (3)
• From this first-stage, equation (3),
  is used to estimate the magnitude of study is
  publication bias . Next, the
  estimated publication bias of each study is
  subtracted from that studys reported estimated
  effect, and a second MRA model is run on equation
  (2), this time using corrected t-values and
  forced through the origin. As before, the
  coefficient on precision (1/Sei) in this second
  stage version of equation (2) defines the PETS
  estimate of effect corrected for publication
  bias. And, it seems to work quite well (in many
  applications and in simulations) in removing
  publication bias.

14
2. Two-Stage Precision- Effect (PETS)
15
Details of these methods and simulations can
be found in Stanley, T.D. (2008).
Meta-regression methods for detecting and
estimating empirical effects in the presence of
publication selection, forthcoming in Oxford
Bulletin of Economics and Statistics. Stanley,
T.D. (2006). Two-stage precision-effect
estimation and Heckman meta-regression for
publication selection bias. School Working Paper,
Economics Series 2006/25, Deakin University.
Both online at http//www.deakin.edu.au/buslaw/
aef/publications/economics_wp.php