The Nature of Evidence

1
The Nature of Evidence Evidence-Based
Assessment Tools

• Paul Stolee, Ph.D.
• University of Waterloo, Waterloo
• Lawson Health Research Institute, London
• RGPs of Ontario, April 8, 2005

2
Outline

• The Nature of Evidence
• Evidence and evidence-based practice
• Evidence-Based Assessment Tools
• The assessment challenge in geriatrics
• A checklist for evaluating assessment tools
• Future directions

3
What is evidence?

• A note about data
• If only we had hard data

4
The Trouble with Hard Data Influences on Health
Policy

• Knowledge
• Values and Attitudes
• Institutional Structures
• - Lomas, 1989

5
The Trouble with Hard Data

• The Lake Wobegon Effect

6
Reduction in Lake Wobegon Effect following
P.I.E.C.E.S. training Rating of Level of Care
compared with similar facilities
7
Hard and Soft Data

• distinctly human clinical information is often
  regarded as soft, and either omitted or
  deliberately excluded from the hard data of
  disease that are usually used in scientific
  statistical assessments. - Feinstein, 1992

8
What is evidence?

• A note about Critical Appraisal

9
Guides for reading articles to distinguish useful
from useless or harmful therapy I

• Was the assignment of patients to treatments
  really randomized?
• Were all clinically relevant outcomes reported?
• Were the study patients recognizably similar to
  your own?
• Sackett et al., Clinical Epidemiology, 1985

10
Guides for reading articles to distinguish useful
from useless or harmful therapy II

• Were both clinical and statistical significance
  considered?
• Is the therapeutic maneuver feasible in your
  practice?
• Were all patients who entered the study accounted
  for at its conclusion
• Sackett et al., Clinical Epidemiology, 1985

11
Studies of Geriatric Mental Health Outreach
Programs meeting these 6 criteria
12
The Case of Geriatric Day Hospitals

• systematic review of 12 randomized trials
  comparing a variety of day hospitals with a range
  of alternative services found no overall
  advantage for day hospital care
• Forster, et al., BMJ, 1999

13
Evidence (from CHSRF workshop, 2004)

• Is a lot more than research, and it includes a
  lot of contextual information
• Types of evidence (from Rudolf Klein)
• Research evidence produced by accepted research
  methods
• Organizational evidence an organizations
  capacity to complete the tasks
• Political evidence how key public, politicians
  and other players react to policies, affecting
  chances of success

14
Evidence-Based Medicine (or Practice)

• the conscientious, explicit and judicious use of
  current best evidence in making decisions about
  the care of individual patients.
• Sackett, et al., Evidence-Based Medicine, 1997

15
The Practice of EBM the next sentence

• The practice of evidence-based medicine means
  integrating individual clinical expertise with
  the best available external clinical evidence
  from systematic research
• Sackett, et al., Evidence-Based Medicine, 1997

16
Individual clinical expertise

• The proficiency and judgment that individual
  clinicians acquire through clinical experience
  and clinical practice.
• Sackett, et al., Evidence-Based Medicine, 1997

17
Best available external clinical evidence

• Clinically relevant research, often from the
  basic sciences of medicine, but especially from
  patient-centred clinical research
• Sackett, et al., Evidence-Based Medicine, 1997

18
Evidence-Based Practice

• is based on a self-directed learning model,
  whereby practitioners must not only continue
  learning but also continue evaluating their
  techniques and practice in light of this learning
  to see what can be improved.
• Law, Evidence-Based Rehabilitation, 2002

19
Practical Wisdom and The Practical Syllogism
(with apologies to Aristotle)

• Syllogism
• All horses have four legs
• Seabiscuit is a horse
• Seabiscuit has four legs

• Practical Syllogism
• CGA is effective for frail older persons (the
  desired end)
• Mrs. Paulgaard is a frail older person (the
  particular situation)
• I will undertake a CGA with Mrs. Paulgaard (an
  action to be taken)

20
Practical Wisdom and The Practical Syllogism II
(more apologies to Aristotle)

• Knowledge is incomplete outside its practical
  application
• Therefore suggest Knowledge IN Practice
• A combination of clinical judgment and expertise,
  an appraisal of the situation and contextual
  factors, as well as the application of research
  evidence

21
Evidence-Based Assessment requires

• An appropriate context organizational and
  political support, AND
• Evidence-based assessment tools, AND
• Clinical expertise, judgement and reflective
  practice AND
• An integration of hard, scientific data and
  soft, clinical information and judgement.

22
Evidence-Based Assessment Tools

• The assessment challenge in geriatrics
• A checklist for evaluating assessment tools
• Focus on reliability and validity
• Future directions

23
The Challenge of Geriatrics The Black Box of CGA
-Burns, 1994
24
The Challenge of Geriatrics Who is being assessed
Rockwood, et al., CMAJ, 1994
25
Frail Older Persons

• Multiple, complex problems
• Same outcomes not always relevant or feasible for
  all patients or situations
• Same outcomes can be positive or negative,
  depending on the patient or situation
• Very small gains can have major clinical or
  program implications
• Quality of life often more relevant than survival
  or length of life

26
The Challenge of Geriatrics Inconsistent and
Ambiguous Outcomes
LOW
HIGH
Rubenstein Rubenstein, 1992 Stuck, et al., 1993
27
Better Demonstrated CGA Outcomes When

• Appropriate targeting
• More intensive interventions
• Control over longer-term management follow-up
  and implementation of recommendations
• Usual care control group

28
Which Outcome?

• The demonstrated impacts of geriatric programs
  are greater when appropriate, responsive outcome
  measures are chosen

29
Geriatric Day Hospitals Revisited

• Commonly used measures of disability may be
  insensitive to change in the day hospital
  Forster, et al., 1999

30
Evidence-Based Assessment Tools in Geriatrics

• A clinical priority
• Assessment, treatment and care planning
• Outcome evaluation
• To guide practice
• A policy and planning priority
• Benchmarking
• Resource allocation
• A research priority

31
Research Priorities for Geriatric Rehabilitation

• Canadian Consensus Workshop on Geriatric
  Rehabilitation
• May, 2003, Ottawa, 75 participants
• Major theme of research priorities assessment
  tools, outcome measures, information systems
• Stolee, Borrie, Cook, Hollomby, et al.
  Geriatrics Today J Can Geriatr Soc 2004 738-42

32
Research Priorities for Alzheimer Disease and
Related Dementias

• Ontario Consensus Workshop
• March 31-April 1, 2005, Toronto, 50 participants
• Identified priorities included Clinically
  relevant outcome measures, and research to
  identify these measures

33
A Checklist for Evaluating Assessment Tools
(Streiner, Can J Psychiatry, 1993)

• Items
• Where did they come from?
• Previous scales
• Clinical observation
• Expert opinion
• Patients reports
• Research FindingsTheory

34
A Checklist for Evaluating Assessment Tools
(Streiner, Can J Psychiatry, 1993)

• Items
• Were they assessed for
• Endorsement frequency
• Restrictions in range
• Comprehension?
• Lack of ambiguity?
• Lack of value-laden or offensive content?

35
A Checklist for Evaluating Assessment Tools
(Streiner, Can J Psychiatry, 1993)

• Reliability
• Internal Consistency
• Test-retest reliability
• Inter-rater reliability
• How was reliability calculated?
• On what groups were the reliabilities estimated?

36
A Checklist for Evaluating Assessment Tools
(Streiner, Can J Psychiatry, 1993)

• Validity
• Face validity
• Content validity
• Criterion validity
• Construct validity
• With what groups has the scale been validated?

37
A Checklist for Evaluating Assessment Tools
(Streiner, Can J Psychiatry, 1993)

• Utility
• Completed in a reasonable amount of time?
• How much training is needed?
• Easy to score?

38
Reliability and Validity Why Bother?

• Accuracy
• Are we consistently measuring what we think were
  measuring?
• Associations
• Evaluation
• Resources
• Clinical/Individual Implications
• Policy Implications

39
Reliability and Validity Why Bother?

• Any measurement True Value Error
• Error can be Random or Systematic

40
Reliability and Validity Why Bother?

• Random Error
• Imprecise answers and attenuated associations
  dont find connections that are really there
• The realm of reliability
• Systematic Error
• Bias get the wrong answer
• The realm of validity

41
Reliability Definition

• ability to measure something in a reproducible
  and consistent fashion
• stability or equivalence in repeated or separate
  administrations (assuming no real changes)
• repeatability, reproducibility, stability,
  consistency
• the proportion of variance in measurement that
  reflects actual variance in subjects

42
Reliability Calculation using ANOVA
R
(Intraclass Correlation Coefficient)
43
Other Forms of the Reliability Coefficient

• Pearson Correlation
• Agreement
• Kappa
• K (po - pe)
• 1 - pe
• Weighted Kappa

44
Reliability Approaches/Types

• Test-Retest/Intra-Observer
• Inter-Observer
• Alternate Forms
• Internal Consistency

45
How to Improve Reliability

• Reduce Error Variance
• Clear, standardized instructions
• Training
• Improve survey/scale design
• Increase True Variance
• Modify survey/scale items
• Increase survey/scale items
• Cheat
• Test in more heterogeneous group than your
  patient population

46
Standards for the Reliability Coefficient

• No sound basis for reliability standard, but
• Litwin (95) .70
• McDowell Newell (96) 0.85
• Streiner (93) .60s marginal,
• .70s acceptable, .80 very high
• Hays et al. (93) .70 for clinical trials
• Nunnally (78) .90 at individual level
• Higher for individual decisions than for
  group/population research

47
Validity Definition(I)

• Oh. I know! exclaimed Alice, Its a vegetable.
  It doesnt look like one, but it is.
• I quite agree with you, said the Duchess and
  the moral of that is - Be what you would seem to
  be - Never imagine yourself not to be otherwise
  than what it might appear to others that what you
  were or might have been was not otherwise than
  what you had been would have appeared to them to
  be otherwise.
• I think I should understand that better, Alice
  said very politely, if I had it written down.
  (Carroll, 1865)

48
Validity Definition(II)

• The extent to which a measurement method measures
  what it is intended to
• The range of interpretations that can be
  appropriately placed on a measure

49
Validity Types

• Face Validity
• Content Validity
• Domains
• Range
• Sensibility

It helps if the vegetable looks like a vegetable
50
Validity Types

• Criterion Validity
• Concurrent
• Predictive

Gold Standard
Yes
No
a
b
Yes
Test
c
d
No
51
Validity Types

• Construct Validity
• Extreme Groups
• Convergent
• Divergent/Discriminant

52
Responsiveness

• The ability of a measure to detect clinically
  important change
• Measures
• Effect Size
• Relative Efficiency
• ANOVA
• ROC Curves

53
Future Directions

• Pilot multi-site studies on outcome measures
• GAS in the GDHs of the RGPs
• Assessment databases and information systems
  (e.g., NRS, interRAI)
• What is the role for specialized geriatric
  services?
• Clinical decision support systems
• WHO ICF as assessment framework
• Standardized coding for assessment items
  including individualized measures?
• Biomarkers and linkages with clinical indicators
• E.g., MRI and individualized clinical indicators
• Hard data meet soft data

54
References

• Kramer MS, Feinstein AR. Clinical Biostatistics
  LIV. The Biostatistics of Concordance. Clin
  Pharmacol Ther 29111-123, 1981.
• McDowell I, Newell C. Measuring Health A Guide
  to Rating Scales and Questionnaires. 2nd Ed. New
  York Oxford University Press, 1996
• Nunnally JC, Bernstein I. Psychometric Theory,
  3nd Ed. New York McGraw-Hill, 1994.
• Streiner DL, Norman GR. Health Measurement
  Scales A Practical Guide to their Development
  and Use. Third Edition. Oxford Oxford University
  Press, 2003.
• Streiner DL. A checklist for evaluating the
  usefulness of rating scales. Can J Psychiatry
  38140- 148, 1993.