Monitoring and Evaluation: Calculating and Interpreting Coverage Indicators

1
Monitoring and Evaluation Calculating and
Interpreting Coverage Indicators
2
Learning Objectives

• By the end of the session, participants will be
  able to
• Identify sources of data for calculating coverage
  indicators
• Estimate denominators for routine coverage
  estimates
• Calculate and interpret coverage indicators from
  routine data
• Use online resources for estimating coverage
  indicators
• Assess the quality of relevant data sources
• Reconcile coverage estimates from different data
  sources

3
Maternal Health Coverage Indicators

• Proportion of pregnant women who received at
  least two antenatal care visits
• Proportion of deliveries occurring in a health
  facility
• Proportion of deliveries with skilled attendant
  at birth
• Proportion of women attended at least once during
  postpartum period (42 days after delivery) by
  skilled health personnel for reasons related to
  childbirth

4
Why Coverage Indicators Are Important

• Understand how effective program is
• See if one target group is reached more
  effectively than another
• Identify underserved area/regions

5
Child Health Coverage Indicators

• Immunization Programs
• DTP3 vaccine coverage
• Measles vaccine coverage
• BCG vaccine coverage
• OPV3 coverage
• HepB3 coverage
• Fully immunized child
• Nutrition programs?
• Control of diarrheal disease programs?

6
Coverage Indicators for HIV/AIDS Care Treatment
Programs

• Number of clients receiving public/NGO VCT
  services
• Number of clients provided with ARVs
• Percent of children in need receiving
  cotrimoxazole prophylaxis
• Percent of HIV patients receiving DOTS
• Coverage of PMTCT programs?

7
Where Do We Get the Data?

• Censuses
• Surveys
• Registrations
• Health management information systems
• Program statistics
• Patient registers

8
Estimating Coverage From Routine Data
9
Indicators From Program Statistics Numerators

• HMIS and routine reports give information on
  numerators
• Numerators number of deliveries in health
  facilities, measles vaccinations, pills
  distributed, voluntary counseling and testing
  clients etc.
• Denominators ?

10
ExampleImportance of denominator

• Town A vaccinated 200 infants
• Town B vaccinated 400 infants
• Town C vaccinated 600 infants

• Population size
• Town A 10,000
• Town B 30,000
• Town C 60,000

11
Indicators From Program Statistics What
Denominators Are Needed?

• Denominators population composition
• Population composition
• How many women are of childbearing ages?
• How many children are under five?
• How many adolescents? 15-19? 20-24?
• How many men are 15-59 years?
• How many children are of school going age?
• How many infants are there?
• How many babies are born each year?

12
How Do We Get Denominators?

• Population registers
• Censuses
• Population projections
• Population growth rate (r)
• Rate of natural increase crude birth rate (CBR)
  minus the crude death rate (CDR)
• Net migration rate inmigration - outmigrants per
  1000 population
• CBR no. of births per 1000 population in 1 year
• CDR no. of deaths per 1000 population in 1 yr
• Population growth rate of natural increase
  net migration rate

13
Spectrum Model

• DemProj projects population of country/region by
  age and sex based on assumptions about fertility,
  mortality, and migration
• Urban and rural population projections can also
  be prepared
• EasyProj supplies data needed to make a
  population projection from estimates provided by
  the Population Division of the UN
• www.tfgi.com

14
Spectrum
15
Calculating Denominators

• Population at time t P(t) P(0) exp(rt),
  where
• P(t) is the population size after t years
• P(0) is the population size at the last census
• Example
• 300,000 people at census
• Growth rate 3 (0.03),
• What is the population after 10 years?
• 404,958 people

16
Estimating Number of Live Births

• Where data on the number of live births are
  unavailable
• Total expected births Total population x crude
  birth rate
• Where the crude birth rate (CBR) is unknown
• Total expected births Total population x 0.035

Source WHO 1999a WHO 1999b
17
Estimating Number of Surviving Infants

• Target population for childhood immunization
• Surviving infants lt12 months of age in a year
• Where data on the number of surviving infants are
  unavailable
• Total expected number of surviving infants
• Total population x CBR x (1 infant mortality
  rate)

18
Estimating Number of Surviving Infants CBR Known

• Total population 5,500,000
• CBR 30/1000
• Infant mortality rate (IMR) 80/1000
• Number of surviving infants
• Total population x CBR x (1 IMR)
• 5,500,000 x 30/1000 x (1 - 0.080)
• 5,500,000 x 0.030 x 0.920
• 151,800

Source Immunization Essentials A Practical
Field Guide (USAID, 2003)
19
Estimating Number of Surviving Infants CBR
Unknown

• Where data on the number of surviving infants,
  CBR or IMR are unavailable, multiply total
  population by 4
• Expected no. of surviving children lt 12 months
• Total population x .04
• If the total population is 30,000, then the
  number of children under one year 30,000 x
  4/100 1200

Source WHO, 2002b
20
Estimating the Monthly Target Population

• Monitoring immunization and vitamin A coverage
  should
• be done monthly at the facility and district
  levels,
• requiring estimations of the monthly target
  population
• Monthly target population Estimated number of
  children under 1 year of age divided by 12
• Example
• Annual target population of children lt 12 months
  1200
• Monthly target 1200/12 100

21
Example Immunization Coverage From Routine Data

• Total population of district in 1990 99,000
• CBR 40 per thousand
• IMR 80 per thousand
• Population growth (r) 3 per year
• 3,000 measles vaccinations were given to infants
  in district in 1998
• What is the measles coverage rate for 1998?
• Numerator No. immunized by 12 months in a given
  year
• Denominator Total no. of surviving infants lt 12
  months in same year

22
Immunization Coverage From Routine Data Answer

• Estimate district total population in 1998
• Pop1998 99,000 exp(.038) 125,410
• Estimate number of surviving infants in 1998
• 125,410 x (40/1000) x (1 - .080) 4615
• Estimate measles coverage rate
• Measles coverage 3000/4615 x 100 65

23
Case Study 1 Immunization Coverage from Facility
Data

• Estimate total population in 2003
• Calculate coverage for DTP1, DPT3, and measles
  vaccine in 2003
• Evaluate trends in coverage
• Estimate drop-out rates
• Analyze the problems in 2003
• Is coverage low or falling?
• What are possible causes?
• What are the differences in coverage in different
  areas?
• What action can managers take if coverage
• data indicate problems?

24
Challenges in Estimating Coverage from Routine
Data

• Limited knowledge of target pop/denominators
• Low timeliness completeness of reporting
• Poor data quality
• Lack of written standard reporting procedures
• No systematic supervision on data management
• Dual reporting systems (EPI, HMIS)
• Inclusion of data from private sector

25
Assessing Reliability of Routine Coverage
Indicators

• Understand how denominators are derived
• Understand the process of collecting the
  information
• Look for inconsistencies and surprises

26
Assessing Reliability of Routine Coverage
Indicators

• Look for reliable data from other sources to use
  as a basis for comparison
• Cross-check

27
Estimating Coverage from Survey Data
28
Survey Tools for Coverage Estimation

• WHO-EPI surveys
• Lot quality coverage surveys
• Large-scale population-based surveys
• USAID Demographic and Health Surveys
• UNICEF Multiple Indicator Cluster Survey
• Arab League PAPCHILD surveys
• CDC Reproductive Health Surveys
• Seventy-five household survey
• Knowledge-Practice-Coverage Surveys
• Other local surveys

29
How Do Administrative Data Compare With Survey
Data?
30
Reconciling Coverage Estimates From Different
Data Sources

• Age group geographic scope
• Health cards versus recall
• Different sources for different purposes
• Not all coverage data can be compared in
  constructive way
• Differences in inclusion of private sector
• Selectivity

31
On-line Resource STATcompiler

• Innovative online database tool
• Allows users to select numerous countries and
  hundreds of indicators to create customized
  tables that serve specific needs
• Accesses nearly all population and health
  indicators published in DHS final reports
• http//www.measuredhs.com/statcompiler

32
STATcompiler
33
On-line Resource DOLPHN

• DOLPHN Data Online for Population, Health and
  Nutrition
• Online statistical data resource
• Quick access to frequently used indicators from
  multiple sources, including
• DHS, BUCEN, CDC, UNAIDS, UNESCO, UNICEF, World
  Bank, WHO
• www.phnip.com/dolphn

34
Advantages and Disadvantages of Routine-based
Coverage

• Advantages
• Provides information on more timely basis
• Makes use of data routinely collected
• Can be used to detect and correct problems in
  service delivery
• Disadvantages
• Denominator errors
• Poor quality reporting

35
Advantages and Disadvantages of Survey-based
Coverage

• Advantages
• Avoids problems with denominators
• Includes information from non-reporting
  facilities
• Disadvantages
• Coverage survey has low precision
• Larger standard errors at sub-national levels
• Irregular and expensive
• Survey timing may affect coverage rates

36
Case Study 2 Estimating Vitamin A Coverage

• Calculate coverage from routine data
• Use tally sheets to determine number of children
  who received vitamin A compared to target
  population
• Compare coverage estimates from routine data
  with estimates from survey data
• Estimate missed opportunities

37
References

• WHO. 1999a. Indicators to Monitor Maternal
  Health Goals Report of a Technical Working
  Group, Geneva, 8-12 November 1993. Division of
  Family Health Geneva WHO.
• WHO. 1999b. Reduction of Maternal Mortality A
  Joint WHO, UNFPA, UNICEF, World Bank Statement.
  Geneva WHO.
• WHO (2002) Increasing Immunization at the Health
  Facility Level. Geneva, Switzerland World
  Health Organization