Evaluation and evaluative research in healthcare: analysis of an example

1
Evaluation and evaluative research in healthcare
analysis of an example
Enrique Bernal Delgado, PhD Marisol Galeote
Mayor, PhD Félix Pradas Arnal, MD Salvador Peiró
Moreno, PhD Soledad Márquez Calderón, PhD
2
Types of studies of interventions and
organizations
Descriptive studies Describe and/or quantify
what organizations or interventions, or some
aspect of them, are like and/or how they
function. Development studies Design or
plan interventions (practices, processes,
programs, policies) or new organizations.
Explanatory studies Understand how organizations
or interventions work, and the factors that
influence them. Evaluation studies Formulate
judgements about interventions, or some aspect of
them. Evaluative research studies
Find out if interventions are effective and/or
efficient
3
Evaluative research

• The objective should be to find out if
  interventions are effective and/or efficient.
• Therefore, the design should define
• the causal model behind the hypothesis
  established.
• the variables of the study dependent,
  independent y other prognostic variables.
• the units of observation.
• the instruments used for measurement.
• the time frame in which measurement or
  observation take place
• the factors that can affect the studys internal
  and external validity

4
Designs for evaluative studies
EG
EG neqCG
EG eqCG
X 0 -- 0
X 0
Post-test
0 X 0 0 -- 0
0 X 0
Pre-test
000 X 000 000 -- 000
000 X 000
Time series
000 X 000 000 -- 000
R
5
Factors that threaten internal validity

• Uncontrolled concomitant prognostic factors
• Background changes in the environment that are
  simultaneous with the intervention.
• Maturation changes in the individuals included
  in the program in response to natural
  developments.
• Administration of the TEST the effect of having
  administered a PRE-TEST on subsequent POST-TEST.
• Instruments changes in the measurement
  instruments or in the observers.
• Regression to the mean.
• Selection biases the assignation of individuals
  to the groups may be differential.
• Mortality or losses differential losses of
  participants.

6
Factors that threaten external validity

• The effect of a reaction to, or an interaction
  between the tests the pre-test changes the
  sensitivity of the participant to the
  intervention.
• Effect of the interaction between a selection
  bias and the intervention The response to the
  intervention is differential in the experimental
  group.
• Effect of reaction to the experiment (Hawthorne)
  distortion of the effect because the people know
  they are being studied.
• Mortality or losses differential loss of
  participants.
• Variability response variability depends on
  multiple factors.
• Generalization of groups or individuals
• Generalization from a homogeneous group to a
  population.
• Generalization from the average response of the
  group to an individual.
• Generalization to other places, time frames or
  programs.

7
Designs in evaluative research control of threats
R random assignation --- without random
assignation 0 measurement or observation X
Intervention evaluated EGExperimental
group CG Control group
8
Pretest-postest with equivalent group
Multiple time series with equivalent group

• Bernal-Delgado E, Galeote-Mayor M, Pradas-Arnal
  F, Peiró S. Evidence-based educational outreach
  visits
• Effects on prescription of non-steroidal
  anti-inflammatory drugs.
• JECH 2002 56 653-8.

9
Summary of the study

• Objective To evaluate the effectiveness of group
  educational outreach visits based on the
  systematic review of evidence, to change
  prescription patterns for drugs in Primary Care.
  
• Population studied 24 teams with 158 general
  practitioners in Primary Care in the Healthcare
  Area of Teruel.
• Design experimental, single blind, with 1 EG
  and 2 equivalent CGs.

10
Causality Model (1) detection of variations in
prescriptions of different NSAIDs in small areas
Piroxicam 7
Tenoxicam 18
Diclofenac 22
11
Causality Model ( 2) Hypothesis the Variations
in Medical Practices (VMP) are due to ignorance
or uncertainty about the advantages and
disadvantages of the different NSAIDs prescribed,
that can be mitigated with educational
strategies.
Causality Model for Variations in Medical
Practices (VMP)
12
Are the groups equivalent?
13
Clinical Problem
Search Strategy
Analysis of evidence
Summary of evidence
Evidence-based outreach visit
Dissemination of recommendations
14
Have changes occurred after the interventions?
15
Some common determinist models
(1) ya exp(bt)
(3) yexp(ab/t)
(2) yK 1 a exp (bt)-1
16
Have changes occurred after the interventions?
17
Control of threats to internal validity

• The findings may be attributable to
• A bias in selection?
• Interference from other, simultaneous
  interventions?
• Maturation?
• Regression to the mean?
• Contamination?
• Misclassification of effect and differential
  losses?
• Effect of persons conducting the outreach
  visits?

18
Control of threats to external validity

• Can these results be generalized?
• Inclusion criteria were limited.
• Was there a reaction to the pre-test?
• Was there a reaction to the experiment
  (Hawthorne) or biases due to the novel effect or
  social desirability?

19
CONCLUSIONS The intervention was more
effective than doing nothing and, although there
is no proof that this type of intervention is
better than others, this type does seem to
produce an incremental effect that surpasses the
effect of the placebo. In systems offering few
incentives, such evidence-based educational
sessions should be developed.