Monitoring and Evaluation: Information Sources and Systems - PowerPoint PPT Presentation

Loading...

PPT – Monitoring and Evaluation: Information Sources and Systems PowerPoint presentation | free to download - id: 404c30-M2Q2N



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Monitoring and Evaluation: Information Sources and Systems

Description:

... (elderly, youth) Risk ... (e.g. retrospective attitudes recall bias) Household survey programs ... To provide data for evaluating program impact ... – PowerPoint PPT presentation

Number of Views:80
Avg rating:3.0/5.0
Slides: 82
Provided by: cpc2
Learn more at: http://www.cpc.unc.edu
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Monitoring and Evaluation: Information Sources and Systems


1
Monitoring and Evaluation Information Sources
and Systems
2
Session Objectives
  • At the end of this session, participants will be
    able to
  • Name the main information sources for PHN ME
  • Describe the main strengths and weaknesses of
    different data sources
  • Discuss the main data-quality issues that need to
    be considered
  • Explain why complementary data sources are often
    required to monitor and evaluate health systems
  • Identify potential data sources that might apply
    in a specific program context.

3
Overview
  • Types of information
  • Strengths and weaknesses of selected data sources
  • Data quality
  • Linking data sources
  • Exercise

4
The Finagles Laws of Information
  • The information you have is not the information
    you want
  • The information you want is not the information
    you need
  • And the information you need is usually not
    available

5
Definitions (1)
  • Data the raw facts that are collected and form
    the basis for what we know.
  • Information the product of transforming the data
    by adding order, context, and purpose
  • Knowledge the product of adding meaning to
    information by making connections and comparisons
    and by exploring causes and consequences

6
Definitions (2)
  • Health system
  • all resources, organizations and actors that
    are involved in the regulation, financing, and
    provision of actions whose primary intent is to
    protect, promote or improve health. (WHO, 2000)
  • Program
  • A set of procedures to conduct activities. The
    objective is normally the solution to a problem
  • Neither a health system or program is a static
    phenomena. They experience a continuous process
    of changes due to pressure from both outside the
    system and from within the system.

7
Definitions (3)
  • Health Information System (HIS)
  • A health-information system (HIS), similar to a
    health management information system (HMIS)
  • a system that provides specific information
    support to the decision-making process at each
    level of an organization (Hurtubise, 1984)
  • Data Systems
  • a way of talking about the whole set of ME
    indicators in a performance monitoring-and-evaluat
    ion plan, and all of the data and other
    information that need to be gathered and
    understood in an orderly fashion that makes sense
    and help in program management and implementation

8
Types of Information
  • Surveillance
  • Epidemiological
  • Behavioral
  • Routine service reporting
  • Special program reporting systems
  • Administrative systems
  • Vital registration systems
  • Facility surveys
  • Household surveys
  • Censuses
  • Research and special studies

9
Frequency of Data Collection
  • ROUTINE or continuous data collection
  • NON-ROUTINE or periodic data collection

10
Classify the previous information types by
frequency of data collection
  • Routine
  • Non-Routine

11
Frequency of Data Collection
  • ROUTINE or continuous data collection
  • Health facility-based (patient information and
    service statistics)
  • Community-based (service-statistics)
  • Program-based (administrative)
  • Vital registration
  • Sentinel reporting/demographic surveillance
  • NON-ROUTINE or periodic data collection
  • Household or facility-based surveys
  • Population census
  • Rapid-assessment procedures (RAP)
  • Special studies/research

12
Geographic System Levels
  • National
  • Sub-national (e.g. district)
  • Program area

13
The Health Information System Data for Planning,
Monitoring and Evaluation in the PHN Sector
Routine
Non-Routine
TYPE
USE
Aggregated Service Statistics Aggregated Mgmt
Data Aggregated Surveillance Data Financial
Data Vital Registration Systems
Policy-Making Strategic Planning Program
Tracking Disease Surveillance Technical
Logistical Support
National Level
Population-based surveys e.g. DHS
Facility-based surveys e.g. Situation Analysis,
SPA
Rapid Assessment Methods
Aggregated Service Statistics Aggregated Mgmt
Data Sentinel Sites Observation
Checklist Self-Evaluation (e.g. COPE)
Planning (Access) Management (Quality/Efficiency)
Supervision (Performance) Disease Surveillance
District Level
Special Studies e.g. EPI cluster surveys, KAP
studies, etc.
Facility/ Client
Client Records Financial Records Supply
Records Facility logbooks/data records Aggregated
Community Data
Client Mgmt and Follow-Up Health Unit
Management Work Planning/Priority Setting
Census
Birth and Death Records School Records CBD
logbooks Drug Revolving Fund records
Client Mgmt and Follow-up Supplies
Management Community Awareness
Community
14
Data-Collection Levels
  • Policy or program
  • Service environment
  • Client
  • Population
  • Spatial/geographic

15
Data Sources at the Policy/Program Level
  • Official documents (legislative, administrative)
  • National budgets or other accounts data
  • Policy inquiries
  • Reputational rankings
  • Program effort scores

16
Trends in Family-Planning Effort Score 1972-1999
17
Data Sources Service Environment Level
  • Administrative records
  • Service statistics
  • Management information
  • Financial data
  • Service-delivery point
  • Routine service statistics
  • Audits/inventories
  • Facility surveys
  • Agent, staff or provider
  • Performance, competency
  • Training records

18
EXAMPLE SERVICE ENVIRONMENT LEVEL DATA
19
HEALTH MANAGEMENT INFORMATION SYSTEMS (HMIS)
  • Note An important way of monitoring routine data
    over time is through a Health Management
    Information System. An HMIS is a system for
    ongoing (routine) collection and reporting of
    data about client-service delivery. In many
    countries, this system operates at the national
    level. Ideally, these routine data are collected
    from a comprehensive set of service delivery
    points, and should cover topics such as
  • Costs
  • Stockouts
  • Births
  • Mortality
  • Morbidity
  • Numbers of clients seen, referred (inpatient
    outpatient)
  • Numbers of clients by types of service

20
Data Sources Client
  • Client-exit interviews
  • Case surveillance
  • Epidemiology of disease
  • Provider-client observation
  • Management of the sick child
  • Vendor-client interaction
  • Contact or visit registers
  • Customer record

21
Data Sources Population
  • Census
  • Vital registration system
  • Sample household surveys
  • Special population surveys
  • Demographic (elderly, youth)
  • Risk groups (CSWs, MSMs, IDUs)
  • Occupational (farmer, skilled labor)
  • Area-based (catastrophe-affected)
  • Biomarkers

22
Spatial/Geographic Data Sources
  • Satellite imagery
  • Aerial photography
  • Digital line graphs
  • Digital elevation models
  • Cadastral maps
  • Global Positioning System data
  • PLACE (site-based surveys)

23
Aerial Photography
Digital Elevation Models
Integrated GIS Database
Satellite Imagery
Cadastral Data
Digital Line Graphs
GPS Data
Chris Betz 1757 Millbrook Ln 28226 Y
2 Christian Carl 1761 Millbrook Ln 28226 Y
1 Chris McAfee 1765 Millbrook Ln 28226 Y
2 Dale Legere 1776 Millbrook Ln 28226 N
6 Donna Black 1780 Millbrook Ln 28226 Y 2
Demographic Data
Differential GPS
24
High Transmission of HIV Guguletu, Cape
Town South Africa Carolina Population
Center University of North Carolina at Chapel Hill
Neighborhood Statistics Showing High Transmission
Sites within 500 Meters
1 - 13
14 - 25
Air Photo Showing Potential High
Transmission Sites in Guguletu
26 - 37
Individual Structures Can be Identified
38 - 49
50 - 61
No Data
25
Strengths and Weaknesses of Selected Data Sources
26
Focus ME Data Sources
  • Facility-based routine information systems
  • Facility surveys
  • Population-based surveys
  • Program records/administrative data

27
Facility-Based RHIS Types of Information
Generated
  • Service statistics
  • Outcomes of health interventions if individual
    patient records kept
  • Not coverage (but can be estimated in some cases
    with other data)
  • Not incidence (except nosocomial infections)
  • Not prevalence

28
What is Wrong with Existing RHIS?
  • Irrelevance of information gathered
  • Poor data quality
  • Duplication and waste among parallel health
    information systems
  • Lack of timely reporting and feedback
  • Poor use of information
  • Centralization of information management without
    feedback to lower levels

29
Strengths of Routine Health Information Systems
  • Continuously collected suitable for frequent
    reporting
  • Existing system no new data collection builds
    local capacity sustainability
  • Typically available at lowest administrative
    levels (e.g. district, facility)
  • Integral part of health system direct link to
    health system actions

30
Common Problems With Facility-Based RHIS
  • Variation in quality and completeness of
    reporting
  • Timeliness of reporting
  • Difficulty of providing coverage estimates
  • Indicators may not be exactly what you want in a
    particular context
  • May only cover government facilities
  • Double-counting

31
Facility Surveys Types of Information
  • Readiness to provide services (inventory)
  • Infrastructure, staffing, hours of operation,
    fees
  • Health worker knowledge
  • Provider interviews
  • Quality of Care
  • Client-provider observation
  • Client satisfaction
  • Exit interviews

32
Strengths of Facility Surveys
  • Can cover both public and private health
    facilities
  • More detailed information than is typically
    available in routine systems
  • Can be tailored to specific program needs
  • Timing can coincide with program implementation
  • Can combine with population survey for outcome
    monitoring and impact evaluation
  • Quality control may be easier than in routine
    systems

33
Limitations of Facility Surveys
  • Survey sampling design and analysis may be
    complex
  • Expensive, time-consuming
  • Stand-alone sustainability concerns less
    connected to ongoing program decision-making
  • Information rapidly outdated, unless repeated
    not available regularly
  • Coverage/sample size constraints
  • National vs. sub-national
  • By type of facility
  • Small client sample sizes for some services (e.g.
    FP, STIs)

34
Facility Survey Initiatives and Tools
  • Service Provision Assessment (SPA/HSPA) DHS
  • Service Availability Mapping (SAM) WHO
  • Quick Investigation of Quality (QIQ) M/Eval (FP
    only)
  • Situation Analysis (SA) Population Council (FP
    only)
  • JICA facility surveys and mapping

35
Population-based surveys Types of information
collected
  • Knowledge and attitudes
  • Practices
  • Coverage

36
Strengths of population-based surveys
  • Representative of the general population no
    selection bias
  • Wide range of outcome-level indicators can be
    collected
  • Program coverage
  • Well-tested instruments quality control

37
Limitations of population-based surveys
  • Coverage national versus sub-national not
    suitable for district-level estimates
  • Frequency typically only conducted every 3-5
    years.
  • Cannot detect small changes or changes over short
    periods of time without large sample sizes
    (expensive)
  • Not suitable for some types of information (e.g.
    retrospective attitudes recall bias)

38
Household survey programs (national)
  • Demographic and Health Surveys (DHS)
  • CDC Reproductive Health Surveys (RHS)
  • UNICEF Multiple Indicator Cluster Surveys (MICS)
  • PVO Knowledge Practices and Coverage Survey (KPC)
    (not national)
  • CDC Young Adult Reproductive Health Surveys
    (YARHS)

39
Class Activity (1)
  • How to improve facility-based routine
    information systems in developing countries?

40
Class Activity (2)
  • What are the determinants of health information
    systems performance?

41
Data Quality
42
Overview
  • Types of information
  • Strengths and weaknesses of selected data sources
  • Data quality
  • Linking data sources
  • Exercise

43
Data Quality Issues
  • Coverage
  • Completeness (census, sample)
  • Accuracy measurement error biases
    double-counting
  • Frequency of collection
  • Reporting flow
  • Reporting schedule
  • Public accessibility
  • Supervision

44
  • Hierholzer (Am. J. Med 1991 91 21-26) has
    called data the Researchers (ME expert) sand. A
    lens maker takes sand, refines it, melts it, and
    through a long process of grinding and smoothing,
    fashions a lens with which to see the world more
    clearly. Similarly, a ME expert takes data,
    refines it and smoothes it until a clearer
    picture of nature is revealed. If the sand is
    dirty or impure, the lens will be cloudy and
    distorted. If data is unreliable or invalid, the
    ME experts understanding of nature will be
    clouded and distorted.

45
  • By paying close attention to the data collection
    process from the conception of the data
    collection document through the editing of the
    data after it is collected, the ME expert help
    keep his sand pure so that, in the end, nature
    may be viewed with much clarity and possible
  • No amount of sophisticated analysis can salvage
    either a poorly designed or a badly carried out
    study

46
Linking data
47
Linking data
  • Data can be linked from different sources, across
    different levels, or over time
  • Linking data appropriately requires planning,
    preferably prior to data collection
  • Understanding linked data can provide depth and
    continuity to enrich otherwise discrete points of
    information

48
Linking Data
  • Why link?
  • Survey data sets (e.g., household and facility
    information) can be linked to compare services
    available and health outcomes across geographical
    units
  • Geographical and survey data can be linked to
    examine the effects of physical attributes on
    service utilization
  • Time series and panel data can help build causal
    explanations of program or project effects
  • Why not link?
  • May not be necessary for a given program in a
    given context
  • Improper methodology can confuse issues more than
    explain them
  • Analyzing linked data more appropriate for
  • evaluation than monitoring

49
Linking Data
  • Examples
  • Population and facility data can be linked to
    ascertain health outcomes correlated with service
    availability, training, or quality of care (e.g.
    of live births in catchment area attended by a
    trained personnel or of women exclusively
    breastfeeding until 6 months among women going to
    facilities where provider training took place.)
  • Facility and client data can be linked to learn
    about program expenditures per new family
    planning acceptor
  • Facility and staff data can be combined to
    provide information about the proportion of
    clients per provider or the proportion of doctors
    per facility

50
Population-based data highlight limitations in HIS
51
Facility-based data provide additional
explanations
52
Comparing HIS and population-based data
confirming trends...
53
..and raising questions
54
What do you need for evaluating program impact?
  • A specific question
  • Are program inputs X influencing behavior
    (outcome) Y?,
  • If yes, by how much?
  • A conceptual framework
  • Appropriate Data
  • An empirical model and estimation procedure

55
Linking Inputs to Outcomes for Evaluating
Program Impact .....
Individual/ Household Characteristics
Healthy Practice
Healthy Outcome
Service Utilization
Health Service Supply and Community Infrastructure
Other donors, NGOs, and the Government
USAID Program
56
Linking Inputs to Outcomes for Evaluating
Program Impact (sources of data)
Individual/ Household Characteristics
Healthy Practice
Healthy Outcome
Service Utilization
Health Service Supply and Community Infrastructure
Household Survey
Facility Surveys Community survey
Other donors, NGOs, and the Government
USAID Program
Compilation of data on relevant interventions
57
Levels of factors influencing health behavior and
outcomes
58
Data Sources
59
Data System needed for Program Impact Evaluation
  • Population - based measures of individual
    demographic and health
  • behaviors and outcomes, and household
    characteristics
  • ? Household survey
  • Objective measures of health service supply
    and program inputs
  • ? Facility Survey
  • Objective measures of other community
    characteristics
  • ? Community Survey
  • Key Areal Linkage of Surveys

60
Unlinked Program Inputs to Outcomes
Individual/ Household Characteristics
Healthy Practice
Healthy Outcome
Service Utilization
Health Service Supply and Community Infrastructure
Other donors, NGOs, and the Government
USAID Program
61
Facility Survey linked to Household Survey
  • Health facilities selected on basis of household
    survey clusters
  • Objectives
  • ? Describe womens accessibility to health
    services
  • ? Describe communities health service supply
    environment
  • Example accessibility Indicators
  • women who have a SDP with FP within 10
    kms.
  • women who have a physician within 5
    kms.
  • Sampling strategies Service Availability Module
    (SAM)
  • Family Life Survey
  • Living Standard Measurement Survey
  • Concentric clusters

62
Linked SurveysService Availability Module (SAM)
  • Identify facilities by interviewing key community
    respondents with Community Questionnaire, then
    visit the closest facility of each type if
    distance lt 30 km.
  • Key issue Access to services
  • Pros Facility indicators valid for Average
    woman or household
  • (access indicators)
  • Easy and cheap to implement
  • Linked to household survey, so impact
    evaluation possible
  • Cons It does not necessarily give estimates
    for universe of facilities
  • (but, see Hermalin,A. et.al.,1996 paper)
  • Partial description of communities health
    service supply
  • environment, likely

63
Service Availability Module (SAM) contd.
X
?
30 km.
X
DHS cluster
X
Health facility
64
Linked Facility SurveysFamily Life Surveys (FLS)
  • Identify facilities by asking household survey
    respondents on sources of services, compile list,
    and then visit facilities most frequently
    mentioned (up o a quota and lt 45 minutes driving)
  • Key Issue Community characteristics, not only
    Health supply
  • Pros Facility indicators valid for Average
    woman or household
  • (access indicators)
  • Linked to household survey, so impact
    evaluation possible
  • More complete picture of supply of services /
    objective measures?
  • Cons It does not necessarily give estimates
    for universe of facilities
  • (but, see Rand IFLS documentation)
  • Less easy to implement (close coordination
    with HS)

65
Linked Facility SurveysLiving Standard
Measurement Survey (LSMS)
  • Identify facilities by interviewing key community
    respondents with Community Questionnaire, then
    visit the closest facility of each type
  • Key issue Basic Community characteristics,
    access to services
  • Pros Facility indicators valid for Average
    woman or household
  • (access indicators)
  • Easy to implement
  • Linked to household survey, so impact
    evaluation possible
  • Cons - It does not necessarily give estimates
    for universe of facilities
  • - Partial description of communities health
    service supply
  • environment, likely

66
Linked Facility Survey Contiguous Clusters
Approach
  • Steps
  • 1 Define one or two rings of clusters around
    the DHS cluster
  • 2 Canvass the DHS cluster and the surrounding
    ring of clusters to
  • compile list of facilities
  • 3 Conduct interview in all facilities in that
    area
  • 4 Collect measure of size of clusters
    (population size) to calculate sample
  • weights

67
Linked Facility Survey Contiguous Clusters
Approach

X
X
X
Cluster boundary
Household Survey cluster
X
Health facility
X
68
Linked Facility Survey Contiguous Clusters
Approach
  • Reliability of Estimates Provides unbiased
    estimates of facility

  • characteristics, but less efficient
  • Analytical Utility Provides proxy of health
    service supply environment,
  • It makes evaluation of program impact
    possible,
  • without limiting monitoring
  • Cost It is a variation of Area Frame Sample
    (1st. stage areas are DHS clusters)
  • so, cost should be similar to
    stand-alone FS with area sampling and,
  • lower than FS with list sampling (of
    same sample size).
  • Cost also reduced if coordination of
    DHS and FS operations
  • Practical Feasibility Needs careful planning
    since survey preparation
  • location of clusters and their population size
    information required for
  • sample selection and weights

69
Tanzania Reproductive and Child HealthFacility
Survey, 1999
  • Objective 1. To collect information on
    availability and characteristics of
    reproductive and child health services
  • 2. To provide data for evaluating program
    impact
  • Sampling Design linked to TRCH Household
    Survey, contiguous
  • clusters
    approach
  • Clusters 146 (mainland)
  • Government NGO/Private Total
  • Hospitals 77 11 88
  • Health Centers 40 22 62
  • Dispensaries 138 117 255
  • UMATI / MS /oth 5 35
    40
  • Total 260 185
    445
  • Pharmacies 306

70
Tanzania Reproductive and Child Facility Survey,
1999
71
(No Transcript)
72
  • WRAP UP

73
Supplemental SlideRHIS Issues
  • Do we trust HIS data?
  • Reliability and validity of routine data
  • Timeliness and relevance of data
  • Costs of data collection and sustainability
  • Rational process and parsimonious selection of
    indicators
  • Is there an information culture with clear
    purposes of data collected for managerial and
    strategic decisions?
  • Environment that facilitates the direct use of
    routine data
  • Does HIS meet data need of decentralized health
    services?
  • Does HIS monitor referral systems effectively?
  • Does HIS monitor private sector activities that
    are of public health interest (what are the
    incentives, if any?)

74
Group Work Case study 1
  • Instructions
  • You have been asked to advise the following two
    programs on their ME plan. Identify the
    potential data sources that you might explore to
    provide information for the ME plan. List the
    factors that you consider as you assess the
    suitability of different data sources for each
    program.

75
  • Program A is an NGO-run RH/MCH program operating
    in 3 districts in a country. The program aims to
    improve use of MCH services such as immunization,
    ANC, and family planning use in the districts in
    which it works. It provides training to staff in
    MOH clinics in the districts to improve the
    quality of services provided. Private sector
    health services are limited in the program areas
    so most people use government sector services.
    The program also undertakes community
    mobilization through community health workers and
    local radio spots to promote use of services. The
    program wishes to use some of the ME plan data
    for ongoing program management and will be
    required to report to its donor annually on its
    performance as well as at the end of the project
    on its overall results.

76
  • Program B is a national AIDS prevention
    program. The program includes a mass media
    campaign on the ABCs aimed at reducing risk
    behaviors in the general population, the
    initiation of a PMTCT program and the expansion
    of its VCT program. The PMTCT and VCT program
    activities include training of health workers to
    provide quality VCT and PMTCT services,
    strengthening logistics systems to provide
    reliable supplies of HIV test kits to PMTCT and
    VCT sites as well as ARVs to PMTCT sites, opening
    new sites to increase the physical accessibility
    of these services to the population, and
    community mobilization to use VCT and PMTCT
    services through local media and community-based
    activities in areas where sites are located. In
    addition, new data collection forms will be added
    to the RHIS for PMTCT and VCT sites to collect
    service statistics on the new services, and sites
    will receive regular supervisory visits during
    their first few years of operation. The program
    wishes to use the ME plan data for ongoing
    program management and annual reporting, as well
    as to fulfil relevant UNGASS and donor reporting
    requirements.

77
Group Work Case study 2
  • Instructions
  •  
  • Identify for each indicator the appropriate
    source of data needed in its calculation. Present
    and discuss your work in a plenary session.

78
  • Proportion of households with at least one ITN
  • Proportion of children under 5 years old who
    slept under an ITN the previous night
  • Proportion of children under 5 years old with
    fever in last 2 weeks who received antimalarial
    treatment according to national policy within 24
    hours from onset of fever
  • Proportion of pregnant women who slept under an
    ITN the previous night
  • Proportion of women who received intermittent
    preventive treatment for malaria during their
    last pregnancy
  • HIV seroprevalence among all TB patients

79
  • Number of health facilities involved in DOTS with
    sufficient drug and laboratory supplies
  • Number of health facilities and laboratories
    involved in DOTS with sufficient capacity for
    DOTS
  • Number of health facilities where TB and HIV
    services are both available
  • Number of project staff trained
  • of project budget spent on health
    infrastructure
  • of project beneficiaries (patients) who are
    accurately referred

80
  • Number of networks/partnerships involved in
    project
  • Number of project service deliverers trained in
    ME
  • of overall project budget spent on ME
  • Number of project services deliverers trained in
    procurement and supply management
  • of project service delivery points with
    sufficient drug supplies
  • 18. Unit cost(s) of project drug(s) and
    commodities

81
  • Number of people reached by the services
  • Number of service points supported by the funding
  • Number of providers trained in the service
  • HIV-infected pregnant women receiving a complete
    course of antiretroviral prophylaxis to reduce
    the risk of MTCT (number and percentage)
  • Districts with access to donor recruitment and
    blood transfusion
  • Transfused blood units screened for HIV
  • People with advanced HIV infection receiving
    antiretroviral combination therapy (number and
    percentage)
About PowerShow.com