Methods to Detect and Control Bias in Comparative Analyses of Disability Statistics

1
Methods to Detect and Control Bias in Comparative
Analyses of Disability Statistics
5th Annual Meeting of the Washington Group on
Disability Statistics Rio de Janeiro, Sept 21-23
2005
Bedirhan Üstün Nenad Kostanjsek Classifications
Assessment Terminology Measurement and Health
Information Systems www.who.int/classifications/ic
f
2
(No Transcript)
3
Temperature Hot, Mild or Cold ???

• Measured Real temperature 20 C
• Iceland - Hot
• Germany- Mild
• Brazil - Cold

4
Big Mac Index

• University Professor in Turkey gets 3000
  USD.
• University Professor in the Switzerland gets
  8000 USD.
• Big Mac in TR 1.5 USD
• Big Mac in CH 4 USD
• Salary 2000 Big Macs
• in both

5
Health Disability Survey Data

• prove useful to complement National Health
  Disability Information Systems
• more representative of entire populations
• direct information from individuals on their own
  experiences rather than professionals or
  surveillance information
• can be tailored to specific management and
  evaluation purposes in mind
• provide empirical data for system performance
  assessment
• More like to be standard and accurate
• gathered more carefully

6
DOWNSIDE of Surveys

• Numerous
• National and International Health Disability
  Surveys exist and they differ in
• Content
• Questions
• Response scales
• Samples
• Timing

7
Problem of Comparability in International Health
Data

• Mortality statistics 3 x
• Health Survey data 6 8 x
• Disability Survey data 30 60 x
• Health Interventions 40 300 x

8
Methods for Comparability1. Same Concepts

• Examples
• Disability Impairment, Activity limitation
  Participation restriction (ICF definition)
• Disability Impairments simple activities
• Hearing body function
• Communicating activity and participation domain
• The self-care category includes the
  sub-categories washing, dressing, toileting.

9
Methods for Comparability2. Same Domains
Questions
10
Methods for Comparability2. Same Domains
Questions (cont.)
11
Methods for Comparability3. Same response scale
(cont.)

• Different response categories for severity scale
• none, mild, moderate, severe
• Numerical scales (0,1,2,3,4) (1,2,3,4,)
• Likert visual
• Percentage visual scale
• Different response categories for frequency
  scales
• always, usually, sometimes, never or all the
  time,
• once, a few times, sometimes, many times

12
Methods for Comparability3. Same response scale
(cont.)
13
Methods for Comparability4. Translation protocols

• Translation/interpretation of questions
• Translation/interpretation of response scales
• The meaning of the underlying construct may vary
• The relation between the questions and the
  underlying construct may vary

14
Methods for Comparability4. Translation
protocols (cont.)

• The item is difficult or impossible to translate
  because there is no equivalent term in the local
  language.
• The item, when translated, has a meaning that is
  more narrow than the original term. Only part of
  the connotations of the original term carries
  over into the local language.
• The item, when translated, has a meaning that is
  expanded beyond the meaning found in the
  original. The translated items has additional
  meanings that would change the interpretation of
  the item.
• The item is translated into exactly the same word
  that has been used for another item in the
  questionnaire. Words that are different concepts
  in the questionnaire become merged into the same
  concept in the local language.
• The item can be translated, but there is a
  cultural applicability problem with the item, or
  with the definition, or with the examples used
  for the item.

15
Methods for Comparability5. Same application
mode same stimulus
Input
Respondent
Output
16
Methods for Comparability6. Measuring and
adjusting for shifts in cut points Different
Rulers Do response categories mean the same in
different populations?
17
Systematically inconsistent reporting in
different surveys

• men and women
• age groups
• education groups
• economic level
• education levels
• cultural and ethnic groups
• occupations
• levels of health insurance.

18
Methods for adjusting for shifts in cut points
Calibration Tests Vision

• used extensively in international studies
• Distant Vision
• standard Snellens chart
• Near vision
• standard card for use in illiterate populations

19
Methods for adjusting for shifts in cut points
Calibration Tests Cognition

• Verbal fluency
• names of animals in one minute
• record correct number errors
• Simple Memory
• Immediate recall
• delayed recall (20 min.) of word list
• Colour cancellation test
• time taken
• number of correct and wrong cancellations

selected from WHO international studies on
Culture-fair Cognitive Battery
20
Methods for adjusting for shifts in cut points
Calibration Tests Mobility

• Posturo-Locomotor-Manual test
• Time to stand up
• Time taken to complete task
• 3 trials
• stand up from sitting position
• walk one step
• lift up weight of 2 kgs
• walk 6 metres
• place weight at shoulder weight
• lift up weight
• walk back towards chair
• place weight back
• return to chair and sit down

21
Methods for adjusting for shifts in cut points
Vignettes

• Mobility
• Mary has no problems with walking, running or
  using her hands, arms and legs. She jogs 4
  kilometres twice a week.
• Anton does not exercise. He cannot climb
  stairs or do other physical activities because he
  is obese. He is able to carry the groceries and
  do some light household work.
• David is paralyzed from the neck down. He is
  unable to move his arms and legs or to shift body
  position. He is confined to bed.

22
biasDifferential Item Functioning

• a distortion (possibly systematic and
  non-random) of the results
• due to a measurement error affects all
  measurements in the same way
• violates the fair comparison of results across
  different groups
• substantive implications?
• statistically significant?
• random or systematic?

23
Methods for Cross-Population Comparability

• Same concepts
• Same domains questions
• Same response scales
• Translation protocols
• Same application mode same stimulus
• Correction for confounders (education, income)
  using known calibrators
• Use modern Item Response Theory (IRT) approaches