Quality of vaccination histories reported from selected Immunization Information Systems (IIS): Results of the 2002 and 2004 National Immunization Survey (NIS) Experiments Meena Khare1, James A. Singleton2, Diana Bartlett2 1National Center for

1
Quality of vaccination histories reported from
selected Immunization InformationSystems (IIS)
Results of the 2002 and 2004 National
Immunization Survey (NIS)ExperimentsMeena
Khare1, James A. Singleton2, Diana
Bartlett21National Center for Health
Statistics, CDC2 National Immunization Program,
CDCDisclaimer The findings and conclusions of
this presentation are those of the authors and do
not necessarily represent the views of the CDC
2
Issues

• Approximately 30 of the children (19-35 months)
  in the NIS have missing or inadequate
  immunization histories from the providers.
• 13 did not have consent to contact
  provider
• 17 had consent but have missing or
  incomplete provider data
• Can vaccination data from IIS help to
• replace missing and incomplete NIS provider data?
• measure completeness of provider-reported
  vaccination data?
• compare and resolve vaccination data
  discrepancies?
• improve validity of the NIS vaccination coverage
  rates?

2
3
National Immunization Survey (NIS)

• Large ongoing random-digit-dialing telephone
  survey, conducted by CDC since 1994
• Measures vaccination coverage among children aged
  19-35 months at national, state, and urban-area
  levels
• Monitors Healthy People 2010 goals for
  vaccination coverage
• gt90 for 4 DTaP, 3 Polio, 1 MMR, 3 Hib, 3
  Hep B, and 4313, 43133 series
• Data collection
• Household Screener and Interviews (CATI)
• Provider reports (mailed IHQs)
• Estimates
• Vaccination coverage rates based on
  provider-reported data at national, state, and
  urban-area levels

3
4
Immunization Information Systems (IIS)

• Confidential, population-based, computerized
  information systems that collect vaccination data
  within a defined geographic area (e.g., state,
  community)
• Consolidates vaccination records from multiple
  health-care providers monitors low immunization
  coverage areas
• U.S. Healthy People 2010, objective 14.26
  Increase to 95 the proportion of children (aged
  lt6 years) in a fully operational population-based
  IIS by 2010
• In 2004, only 10 of the 56 grantees had fully
  operational IIS with gt95 of the children
  enrolled in the database
• (Source MMWR, 54(45), 2005)

4
5
The 2002 and 2004 NIS Experiments

• States with mature IIS (covering gt67 children in
  the area) were selected in the study
• Four geographically diverse states were included
  in 2002
• Four new states were selected in 2004
• To assess feasibility of using IIS data to
  replace missing or incomplete provider data
• To assess match rate and completeness of the IIS
  and the provider data
• To compare vaccination coverage rates based on
  data from the IIS and the providers

5
6
Sources of data from the NIS Experiment
1 2 3 4 . . . . . . . . . . . 34,000
7
Percent of children with consent and vaccination
history data, by source and pretest site
Pretest site (children with household interview) Children with IIS consent Children with adequate provider data (P) Children with IIS data (I) Children with provider and/or IIS data (T) Increase in Children with Vaccination History Data (T-P)
A (n217) 87.1 59.4 73.3 79.3 19.9
B (n107) 86.0 62.6 57.9 74.8 12.2
C (n217) 87.1 65.0 71.9 83.4 18.4
D (n98) 87.8 66.3 51.0 74.5 8.2
E (n217) 86.2 72.4 78.3 82.5 10.1
F (n191) 83.2 73.3 73.8 80.1 6.8
G (n165) 85.5 74.5 82.3 81.2 6.7
H (n192) 83.3 68.2 71.3 80.2 12.0
Children matched in the IIS based on Name,
Gender and DOB Sites A-D included in 2002 and
Sites E-H included in 2004
7
8
Comparison of 4313 UTD status based on
provider and IIS data by source and pretest site
Sites A-D included in 2002
Sites E-H included in 2004
8
9
Comparison of estimates of Hep B coverage by
source and pretest site
Sites A-D included in 2002
Sites E-H included in 2004
9
10
Comparison of estimates of DTaP 4 coverage by
source and pretest site
Sites A-D included in 2002
Sites E-H included in 2004
10
11
Vaccine-specific agreement in UTD status among
children with data from both sources by pretest
site, 2004
Vaccine type Agreement in vaccination status in children with data from both sources Agreement in vaccination status in children with data from both sources Agreement in vaccination status in children with data from both sources Agreement in vaccination status in children with data from both sources
Site E Site F Site G Site H
DTaP 4 87.2 85.9 94.3 78.1
Hep B 3 88.5 89.2 95.2 82.5
Varicella 1 91.9 92.2 96.2 88.6
4313 UTD 87.2 84.4 92.4 75.4
n from both sources (n148) (n128) (n105) (n114)
Sites A-D had agreements 55-83
11
12
Number of IIS-reported DTaP doses for children
UTD with 4 doses of DTaP from providers by site,
2002
IIS- reported doses of DTaP Provider UTD with 4 doses of DTaP Provider UTD with 4 doses of DTaP Provider UTD with 4 doses of DTaP Provider UTD with 4 doses of DTaP
Site A Site B Site C Site D
Missing records 9.9 31.0 18.5 38.5
0 3.3 0.0 2.5 3.8
1-3 39.6 12.1 16.8 7.7
4 (UTD) 47.2 56.9 62.2 50.0
All 100.0 (n91) 100.0 (n58) 100.0 (n119) 100.0 (n52)
Missing 1-3 doses in the IIS
12
13
Number of IIS-reported DTaP doses for children
UTD with 4 doses of DTaP from providers by site,
2004
IIS- reported doses of DTaP Provider UTD with 4 Doses of DTaP Provider UTD with 4 Doses of DTaP Provider UTD with 4 Doses of DTaP Provider UTD with 4 Doses of DTaP
Site E Site F Site G Site H
Missing records 6.5 8.9 13.4 11.6
0 3.6 4.9 0.0 0.0
1-3 3.6 3.3 5.0 19.6
4 (UTD) 86.3 82.9 80.6 68.8
All 100.0 (n138) 100.0 (n123) 100.0 (n118) 100.0 (n112)
Missing 1-3 doses in the IIS
13
14
Number of provider-reported DTaP doses for
children UTD with 4 doses of DTaP from IIS by
site, 2004
Provider-reported doses of DTaP IIS UTD with 4 doses of DTaP IIS UTD with 4 doses of DTaP IIS UTD with 4 doses of DTaP IIS UTD with 4 doses of DTaP
Site E Site F Site G Site H
Missing records 10.8 9.0 9.6 15.2
0 0.7 2.5 0 1.0
1-3 6.1 3.3 0 4.0
4 (UTD) 82.4 85.2 90.4 79.8
All 148 122 104 99
Missing 1-3 doses from providers
14
15
Distribution of provider reported facility type
by pretest site
Pretest site (Children with reported facility type) All public providers All private providers All hospital providers Mixed provider type and others
A (n133) 15.0 69.2 3.0 12.8
B (n68) 25.0 48.5 14.7 11.8
C (n146) 15.8 50.7 15.1 18.4
D (n67) 23.9 47.8 6.0 22.3
E (n158) 38.0 35.4 5.1 21.5
F (n141) 22.7 31.2 24.1 22.0
G (n123) 11.4 61.8 8.1 18.7
H (n134) 11.2 61.9 12.7 14.2
Excluding children with missing facility type
from all providers on the IHQ during the last 6
months of 2004, 76 of public vaccination
provider sites and 39 of private sites
submitted immunization data to an IIS (MMWR,
54(45), 2005
15
16
Summary

• High consent rates (83-88) in all pretest
  states
• High match rates (gt70) except for sites B and D
  match rate varied by site
• Substantial underreporting in vaccination data
  from the IIS. Vaccination coverage rates were
  generally lower from the IIS differences in
  estimates were smaller in 2004 in comparison to
  2002
• High agreement (gt75) in the UTD status when
  vaccination data were available from both sources
• In 7 out of the 8 sites, children received more
  of their immunizations in the private sector
  (range of 31-69)

16
17
Summary (continued)

• Completeness and quality of vaccination histories
• Adding the IIS query to the NIS provider record
  check increased the percentage of children with
  vaccination data by 6-20
• Entire vaccination records for DTaP were missing
  for 9-15 children from the providers in 2004
• Entire vaccination records for DTaP were missing
  for 6-13 children in the IIS in 2004
• Approximately 3-20 children in 2004 had missing
  or incomplete information on 1-3 DTaP doses from
  the IIS
• Mixed patterns were observed in vaccine-specific
  missing doses (e.g., first vs last missing dose
  for a multi-dose vaccine) by vaccine and site

17
18
Limitations

• Only 8 geographically diverse states were
  included in the pretest
• There was no overlap in states between 2002 and
  2004 to assess improvement over time
• Only mature IIS sites were included
• Quality and completeness varied by site
• Comparability across states need to be evaluated
• Some sites had small sample sizes

18
19
Conclusion

• Public as well as private providers need to
  report complete vaccination histories to the
  registries private providers were the majority
  delivering vaccines to surveyed children in most
  sites
• Although matching rates for the NIS children were
  very high in the IIS, vaccine-specific dose
  information were missing or incomplete.
  Therefore, vaccination coverage rates were
  under-estimated in seven of the eight pretest
  sites using the IIS data alone
• Until completeness of vaccination data improves
  to equivalent levels across all IIS in the U.S.,
  NIS will continue using provider-reported
  histories to compare state level vaccination
  coverage rates in the U.S.

19