How Many Scientists Fabricate and Falsify Research A Systematic Review and MetaAnalysis of Survey Da

1
How Many Scientists Fabricate and Falsify
Research?A Systematic Review and Meta-Analysis
of Survey Data
Daniele Fanelli
2
The million dollar question.
-Only misconduct that has been discovered, and
(presumably) proven to be intentional
Ultimately, only scientists know about their own
intentions!
3
Over the years, many surveys have asked
scientists directly
different things, in different ways
Form of misconduct
Question
Outcome
Since entering medical school have you?
Fabricated data
Yes No
Modified research or experimental results to
improve the outcome
Never SometimesFrequently
Have you participated in research involving
during the last 10 years?
Failing to present data that contradict one's
own previous research
0 1-5 gt5
Indicate the number of members you have
observed/experienced exhibiting within the
last 5 years
Seriously misleading interpretation of results
Results appeared inconclusive and dificult to
compare
4
Tricks in the analysis

• How many committed or observed X at least once
• Only questions on fabrication, falsification,
  alteration and QRP that distort scientific
  knowledge. No plagiarism, professional misconduct
  etc
• Mixed questions were excluded

Question by question effect size and weight

• No other measure of study quality (its
  controversial)
• Included all eligible studies that specified
  their methods
• Entered methodological factors in inverse
  variance weighted regression

5
The search
"research misconduct" OR "research integrity" OR
"research malpractice" OR "scientific fraud" OR
"fabrication, falsification" OR "falsification,
fabrication"
42 literature databases, 14 journals, 8 grey
literature db, 2 internet scientific search
engines, and references lists
Potentially relevant studies obtained from
electronic search (n3276)
Studies excluded because were not surveys on
research misconduct (n3207)
Studies excluded for one of the following reasons
(n48) -Did not have any relevant or original
data -Sample not exclusively composed of
researchers -Misconduct not related to research
(e.g. cheating on school projects) -Does not
distinguish fabrication and falsification from
other forms of misconduct not relevant to this
review -Presents data only in format not usable
in this review
Studies retrieved for examination of full text
(n69)
Studies included in review (n21)
Studies excluded from meta-analysis because did
not meet quality criteria (n3)
Studies included in meta-analysis (n18)
6
Characteristics of studies

• Conducted between 1986-2005
• USA (15), UK (3), multinational (2), and
  Australia (1)
• Medical/clinical (8), biomedical (6),
  multidisciplinary (6), economists (1)
• In total 85 questions
• about fabrication, falsification, alteration,
  modification (meta-analysis)
• Questionable research practices (systematic
  review only)

(Full data set available soon in PLoS ONE)
7
Scientists who admit fabrication, falsification,
or alteration of results
Scientists who know a colleague who fabricated,
falsified, or altered results
b -0.140.05 P0.006
1.97 (N7, 95CI 0.86-4.45)
If only asked fabrication, falsification 1.06
(N4, 95CI 0.31-3.51)
14.12 (N12, 95 CI 9.91-19.72)
If only asked fabrication, falsification 12.34
(N11, 95CI 8.43-17.71)
8
Questionable Research Practices
(e.g. failing to publish data that contradicts
ones previous research dropping data points
based on a gut feeling)
9
What influences admission rates?
Inverse variance-weighted regression
bSE
P
-
Asking about self vs colleagues
-4.530.81
lt0.0001
82 of variance explained (N15)
-
Using fabrication or falsification vs
alteration or modification
-1.020.39
0.0086

Handed-out surveys vs mailed
1.170.4
0.0032
Controlling for these factors, tested for
differences between
Year
USA / other
Researcher / other
n.s.
Medical / other
b0.850.28
Biomedical / other
P 0.0022
Social Sc. / other
10
Leave-one-out sensitivity analysis
Scientists who admit fabrication, falsification,
or alteration of results
Scientists who know a colleague who fabricated,
falsified, or altered results
Martinson 2005 is outstanding, as conservative!
11
Repairing misconduct
Around half of recalled cases had no action
whatsoever taken against them
12
Summary of key findings

• Data fabrication, falsification and alteration
  was
• admitted on average by around 2 (1 - 4)
• directly observed by 14 (10 - 20)
• Questionable Research Practices were
• admitted on average by up to 34
• directly observed by up to 72
• Overall admission rates (self-/non-self) were
  higher in
• Non-self reports, questions not using
  fabrication or falsification, handed out
  questionnaires
• Medical/clinical and related research

13
How Reliable Are These Numbers?
Self-reports
Conservative
Unlike surveys for other criminal
behaviour, Scientists always lose by admitting
misconduct
Non-self-reports

• Unclear
• Risk of multiple reporting
• Muhammad Ali effect
• Unaware of all cases
• Unwilling to damage their field

• Regression analysis
• Medical research not robust to all sensitivity
• Differences in methodology masked most effects
• Non-significant effects not necessarily
  non-significant

14
Conclusions

• On average, 2 of scientists admitted misconduct,
  and 34 QRP
• Actual frequencies probably higher
• Probably vary depending on field and many other
  factors, which meta-analysis currently cannot
  detect
• Future surveys might benefit by
• Focusing on correlates of misconduct
• Common methodologies