Becoming an Informed Consumer of Research

1
Becoming an Informed Consumer of Research

• Evaluating Research

2
An informed consumer of research is able to

• Critically evaluate research studies (primary
  sources)
• Critically evaluate review articles (secondary
  sources)
• Conduct a review of the literature

3
There is no study too fragmented, no hypothesis
too trivial, no literature citation too biased or
too egotistical, no design too warped, no
methodology too bungled, no presentation of
results too inaccurate and too contradictory, no
analysis too self-serving, no argument too
circular, no conclusion too trifling or too
unjustified, and no grammar and syntax too
offensive for a paper to end up in print. -
Rennie
question all the answers Crosby, Stills and
Nash Wasted on the Way
Rennie D. Guarding the guardians A conference on
editorial peer review. JAMA. 19862391-2392.
4
Initial Screening

• Scholarly? Substantive? Popular?
• If scholarly Is it theory, fact, opinion,
  methods?
• If scholarly Is it written by a reputable
  scholar or researcher?
• If scholarly Is it a primary or secondary
  source?
• How recent is the source?

5
Evaluation of Research Studies

• Evaluate the study in terms of
• Utility
• Trustworthiness
• Elements

6
Utility

• The degree to which a study is useful to a
  particular practitioner

7
Trustworthiness

• Type of publication authority of authors
  refereed articles commentaries
• Validity (well-grounded, sound, categories of
  evidence)
• Reliability (accuracy, stability, consistency)

8
Categories of Evidence

• Level A Well-conducted RCT
• Level B Well-conducted case-control study,
  poorly controlled or uncontrolled study,
  observation studies for high potential for bias,
  case series or case reports
• Level C Expert opinion

Hadorn DC, Baker D, Hodges JS, Hicks N. Rating
the quality of evidence for clinical practice
guidelines. J Clin Epidemiol. 199649749-754.
9
Elements

• Title
• Abstract
• Body of the paper
• Introduction (problem, purpose, literature
  review)
• Methods (subjects, instruments, procedures, and
  data analysis)
• Results (presented with no comments tables)
• Discussion (interpretation of results in light of
  previous research related to the findings,
  limitations, clinical relevance, future research)
• Conclusions (restates important findings ties
  into the purpose)
• References (cited in the article)

10
(No Transcript)
11
(No Transcript)
12
(No Transcript)
13
(No Transcript)
14
(No Transcript)
15
(No Transcript)
16
Six Steps to Evaluating Research Studies

• Classify the research variables
• Compare purposes and conclusions
• Describe design and control elements
• Identify threats to validity
• Place the study in the context or other research
  studies
• Evaluate the personal utility of the study

17
Step 1 Classify the Research and Variables

• Experimental
• At least one independent variable is subjected to
  controlled manipulation by the researcher
• Quasiexperimental nonrandom assignment of
  subjects to groups or repeated treatments of same
  group
• Non-experimental
• No manipulation of an independent variable (e.g.
  retrospective chart reviews)

18
Step 2 Compare Purposes and Conclusions

• Provides an indication of whether the study is
  internally consistent (or did the researcher
  stray from the purpose?)
• Guides critique of methods, results and
  discussion
• If significant findings be critical of how well
  the author controlled for alternative explanation
  of the results
• If not significant be critical of data analysis
  methods methods might not be able to detect
  differences that do exist

19
Step 3 Describe Design and Control Elements

• What is the sequence of measurement and
  manipulation? How did the researchers try to
  control for extraneous variables?
• Is the sequence of measurement and manipulation
  (if applicable) clear to the reader?
• Were there any other factors not taken into
  account by the researchers that could explain the
  differences (or lack of differences) found by the
  researchers?

20
Step 4 Identify Threats to Validity

• Internal validity (Are there few alternative
  explanations?)
• Construct validity (Is the measurement tool truly
  measuring what its suppose to? Examples Does IQ
  measure intelligence? Does ROM measure function?)
• External validity (generalizability)
• Statistical conclusion validity (Are the
  appropriate statistical tools used?)

21
Step 5 Place the Study in the Context of Other
Research

• How much new information does the study add to
  what is already known?
• Does it support previous findings?
• Does it contradict previous findings?

22
Step 6 Evaluate the Personal Utility of the Study

• Does the study have meaning for your own
  purposes? How useful is the information to you,
  the student or practitioner?

23
Practice

• Review the sample article provided by the
  instructor and complete the six steps for
  evaluating a research study to the best of your
  ability.
• Homework Find a research study in an area of
  interest to you and complete the six steps for
  evaluating a research study.

24
Evaluation of Review Articles

• Assess the clarity of the review question.
• Evaluate the article identification and selection
  strategies.
• Determine how the authors assess validity of the
  studies.
• Evaluate the results against the strength of the
  evidence.
• Evaluate the personal utility of the review.

25
Step 1 Assess the Clarity of the Review Question

• Is it clear
• What (risk factors, interventions, a test) is
  being reviewed?
• What outcome (functional status, success rates,
  validity and reliability) is expected?
• What population is being studied?
• What comparisons are being made?

26
Step 2 Evaluate the Article Identification and
Selection Strategies

• Is the review comprehensive?
• Time span of the review?
• Search terms used?
• Databases accessed?
• How were studies excluded?

27
Step 3 Determine How the Authors Assess Validity
of the Studies

• Did they?
• How?

28
Step 4 Evaluate the Results Against the Strength
of the Evidence

• Are sufficient valid studies cited to draw a
  conclusion?

29
Step 5 Evaluate the Personal Utility of the
Review

• How useful is the information to you?

30
(No Transcript)
31
Practice

• Review the sample article provided by the
  instructor and complete the five steps for
  evaluating a review article to the best of your
  ability.

32
Conducting a Review of the Literature

• Determine the purpose of the review.
• Identify and select studies for inclusion.
• Identify the designs and constructs of the
  studies.
• Determine the validity of the individual studies.
• Make comparisons across studies.
• Specify problems that need further study.

33
Step 1 Determine the Purpose of the Review

• Guide treatment decisions
• Compare own treatment outcomes with those of
  others
• Determine how others measure success for
  particular types of patients
• Develop a research agenda in an area of interest

34
Step 2 Identify and Select Studies for Inclusion

• Conduct a focused literature search
• Decide which studies are relevant and which are
  not

35
Step 3 Identify the Designs and Constructs of
the Studies Selected

• The nature of the studies
• The nature of the variables under study

36
Step 4 Determine the Validity of the Individual
Studies

• Is the study well-grounded and sound, as
  previously described?

37
Step 5 Make Comparisons Across Studies

• Do the studies agree (consensus) or disagree
  (controversy)?
• If there is controversy, why? What was different
  about the studies that may be responsible for the
  differences?

38
Step 6 Specify Problems That Need Further Study

• Areas of controversy
• Areas with little or no research