Parallel Session on Monitoring and Evaluation to Inform Action and Set Direction fro Research

1
Parallel Session on Monitoring and Evaluation to
Inform Action and Set Direction fro Research

• Five -Year (5YE) Evaluation of the Global Fund
• Tuesday, 18 November 2008
• 1100 - Meeting Room Wa Kamissoko
• 2008 Global Ministerial Forum on Research for
  Health Bamako, Mali

2

• Burkina Faso Team

3
Scope and Scale of 5YE in Burkina Faso

• National Health Accounts (2003-2006)
• National record reviews (HIV, Tb, Malaria,
  General)
  
• Secondary analysis of national surveys (HIV, Tb,
  Malaria, General)
  
• Primary data collection DCA
• Facility-based surveys
• Community-based surveys
• CBO interventions inventory

The National Reference Group for NHA and 2
research institutions selected (IRSS, GREFSaD)
13 Districts selected, 1 per health region, 3
urban, 10 rural, with high (6), medium (3) ,
low (4) exposition to support/interventions
Meda N, Soudré R, Simaga F, Tiendrebeogo AJ,
on behalf of MACRO/WHO/ Burkina Faso 5YE Team
4
Values of 5YE Linkages with Research

• Evidence of poor quality of the routine health
  information system for the monitoring of VCT, ARV
  procurement treatment, PMTCT, etc.
  
• Need of technical support (tools, training,
  archiving)
  
• Establish a research-based observatory of
  universal access
  
• Low access to PMTCT services (only 14)
• Use coverage of target population to follow the
  programme instead of number of facilities with
  PMTCT
  
• Introduce in round 9 grant operational research
  on barriers and facilitators to access PMTCT
  services by comparing good and bad performers
  health facilities
• Rising incidence of TB and malaria 1998-2007
• Better detection rate of TB?
• Better surveillance of malaria?

Meda N, Soudré R, Simaga F, Tiendrebeogo AJ,
on behalf of MACRO/WHO/ Burkina Faso 5YE Team
5
Gaps in Linking Evaluation Outcomes with Research
How to Address Them?

• Gaps
• In-depth analysis of the huge amount of data sets
  gathered not yet occurred (only imposed
  indicators have been computed)
  
• In-depth description and analysis of the impact
  on health system indicators not yet done
  
• Distribution by disease, coverage, and quality of
  CBO devoted to the fight against HIV, Tb
  Malaria unclear?
  
• How to Address Them?
• Support for further analysis of datasets and
  publications
  
• Study the added value of CBO in the fight against
  HIV, Tb Malaria?
  

Meda N, Soudré R, Simaga F, Tiendrebeogo AJ,
on behalf of MACRO/WHO/ Burkina Faso 5YE Team
6
Specific Recommendations Enhance the Link
between ME and Research

• Data sets gathered need further analysis to
  properly answer the basic evaluation questions.
  
• The report of GF5YE in Burkina will provide a
  solid platform for evidence-based policy-making
  and advocacy in fund-raising (future GF rounds or
  other partners) and will raise key research
  questions to explore
• Operational research has to be funded to address
  research questions raised by GF5YE findings (low
  access to PMTCT, rising incidence of malaria and
  Tb, effectiveness of ARV treatment, lack of CBO
  in malaria and TB programmes).

Meda N, Soudré R, Simaga F, Tiendrebeogo AJ,
on behalf of MACRO/WHO/ Burkina Faso 5YE Team
7
Acknowledgements

• Global Fund to Fight HIV/AIDS, Tb Malaria
• WHO country office and headquarter
• MACRO international consultants
• Burkina Ministry of Health/Cabinet
• CCM/Burkina
• Task Force members
• SP/CNLS (NAC secretariat)
• Research groups and their field workers
• Community and health facilities surveyed

Meda N, Soudré R, Simaga F, Tiendrebeogo AJ,
on behalf of MACRO/WHO/ Burkina Faso 5YE Team
8

• Ethiopia Team

9
Scope and Scale of 5YE in Ethiopia

• Trends in HIV, TB, malaria (3Ds) over past 5
  years (since GFs inception)
  
• Not restricted to GF inputs
• Triangulation across data sources
• wider health systems
• Building analytical capacity
• Using data from multiple sources
• Modelling to fill data gaps
• Strengthen country systems for future evaluation
• Strengthening of specific data sources,
  collecting baseline information for future ME

Amha Kebede and Dereje Alemayehu
on behalf of Ethiopia 5YE Team
10
Values of 5YE Linkages with Research

• Capacity building in data collection, assessment
  and analysis
  
• A data depository to facilitate future evaluation
  
  
• A model impact platform to help predict impact
  based on existing data in country
  
• Evidence to support a sustainability plan to
  strengthen HIS, ME and quality of services (both
  at national and district level
  
• Trends in disease burden over recent years (or at
  least established baseline levels) to assess
  progress toward the MDGs
  
• Linkages between health financing, availability
  of services/intervention and utilization of these
  services/intervention at district level
  
• Assessment of system effects and quality of
  services in districts according to the level of
  investment
  

Amha Kebede and Dereje Alemayehu
on behalf of Ethiopia 5YE Team
11
Gaps in Linking Evaluation Outcomes with Research
How to Address Them?

• Lack of mortality data (functioning vital
  registration system)
  
• Limited impact of PMTCT investments
• Assessing survival rates and quality of life for
  ART patients
  
• Sero-prevalence survey of high-risk populations
  for HIV
  
• Impact of PMTCT program
• Trends in welfare status of AIDS orphans
• Population-based data on the prevalence and
  incidence of TB
  
• Trends in STI prevalence STI surveillance
• Study impact on human resource capacity as
  possible bottleneck to service provision
  

Amha Kebede and Dereje Alemayehu
on behalf of Ethiopia 5YE Team
12
Specific Recommendations Enhance the Link
between ME and Research

• Prioritization needs
• Avoid redundant efforts (eg World Bank, Resource
  Mapping, PHRPlus study, CGD study)
  
• Avoid multiplicity of implementing organizations
• Under-utilization of potential of CBOs, including
  FBOs
  
• Assessment of HMIS data quality
• VCT, PMTCT, ART
• Follow up of lost-to-treatment cases
• ART and DOTS
• Use both qualitative and quantitative methods
• Conduct service utilization survey

Amha Kebede and Dereje Alemayehu
on behalf of Ethiopia 5YE Team
13

• Malawi Team

14
Scope and Scale of 5YE in Malawi

• The main sources for national-level data for both
  the Primary and Secondary Data Analysis Countries
  are from existing
  
• Funding records of major national health accounts
  
  
• National service statistics for the three
  diseases
  
• National facility surveys
• National household surveys
• ANC surveillance data
• Other data sources that would inform the
  evaluation
  

Damson Kathyola and Biziwick Mwale
on behalf of Malawi 5YE Team
15
Scope and Scale of 5YE in Malawi

• Primary data or district assessments in nine
  districts
  
• Health Facility Survey rapid assessment of
  public health facilities
  
• Hospital record review All private and public
  health facilities within the nine districts were
  included
  
• Patient follow-up study
• ART A sample of patients from a retrospective
  adults enrolled in ART
  
• Information on demographic and clinical
  characteristics of patients
  
• Collected information of characteristics of
  patients in terms of facilities available for
  follow-up and diagnosis
  
• Challenges Reluctance of ART clinics to release
  information especially in districts whose clinics
  are predominantly under the administration of
  NGOs
• Planned to extract data for 1800 but because of
  the above challenge only managed fro 1200
  patients
  
• TB Purposive sampling of all defaulters and
  patients who did not submit sputum after
  finalising treatment
  
• Interviewed defaulters
• Collected specimen for other patients
• Quality control training of the researchers was
  undertaken for three days
  

Damson Kathyola and Biziwick Mwale
on behalf of Malawi 5YE Team
16
Values of 5YE Linkages with Research

• Although there was a national taskforce,
  development of indicators and tools was
  centralised at Consortium level
  
• This was a lost opportunity to build and
  strengthen in-country capacity on ME
  
• Challenges with the Economic/financial component
  evaluation
  
• Research capacity strengthening within the
  HIV/AIDS and health sectors on health economics
  and financial assessment skills
  
• There were gaps in information/data from the
  health centre or national records and lack of a
  culture at facility level of keeping
  comprehensive health records e.g. patient
  registers and other records at facility level-
  missing/incomplete data
• Capacity building of the Human resources
  responsible for ME within MoH and HIV/AIDS
  players in ME to ensure quality
  
• Coordination of ME within the HIV/AIDS sector is
  a big challenge- this affects the quality,
  consistency and comparability of data
  

Damson Kathyola and Biziwick Mwale
on behalf of Malawi 5YE Team
17
Values of 5YE Linkages with Research

• Although the finance and resource tracking
  provides an overview of the expenditures on TB,
  HIV/AIDS and malaria, there is need to undertake,
  cost-benefit and cost-effectiveness analysis of
  different interventions
• Understand which interventions are more
  effective
  
• Economic benefits of the interventions
  implemented
  
• Different designs for different surveys created
  challenges in comparisons
  
• Harmonisation of the conduct of surveys to
  maximise resources
  
• Proper legal frameworks on the conduct of
  surveys
  
• Qualitative assessment of the underlying factors
  leading to observations in outcomes and
  functionality of the systems
  

Damson Kathyola and Biziwick Mwale
on behalf of Malawi 5YE Team
18
Global perspective ME and Research

• Capacity building initiatives at Global and
  regional levels should be based on understanding
  of the gaps in the ME systems and availability
  of data for policy decision within the developing
  countries
• There is also need to not only build the capacity
  of the technical staff but also raising awareness
  among policy makers at global, regional and
  nation levels on the role of research and ME in
  policy formulation to attract political will and
  support
• Global plans on diseases and targets should also
  encompass strategies for strengthening M E
  systems to enhance the evaluation of the impact
  of the interventions

Damson Kathyola and Biziwick Mwale
on behalf of Malawi 5YE Team
19

• Zambia Team

20
Scope and Scale of 5YE in Zambia

• Measure the impact of recent interventions or
  establish a baseline of recent interventions
  against Malaria, HIV/AIDS and TB
  
• Assess the disruption or benefits for other
  diseases interventions and health care programmes
  of the scaled-up interventions against the three
  diseases
• Develop institutional capacity to utilise
  information from the Monitoring and Evaluation
  system for strategic decision making
  
• Work still on-going with phase of health
  facilities records review in three districts with
  fairly good records and art and PMTCT from
  computerised records of all districts compiled by
  national implementing partners

Christopher Simoonga, Kumbutso Dzekedzeke and
Osward Mulenga
on behalf of Zambia 5YE Team
21
Values of 5YE Linkages with Research

• Some findings in the 5YE partially answer some
  research questions but largely bring them out
  
• Just what led to the sharp reductions in child
  mortality levels ? It should be many factors but
  its not so clear which was dominant among nets,
  exclusive breastfeeding, better parenting from
  the improving education of mothers and health
  education or PMTCT. Answers could be used to
  focus in cost-effective areas in order to achieve
  the MDGs in this area.
• Coverage of interventions did not translate into
  the same impacts. Are there some unseen factors
  or agents of interventions? Shouldnt there be
  some focus on some research for best health
  intervention management models?

Christopher Simoonga, Kumbutso Dzekedzeke and
Osward Mulenga
on behalf of Zambia 5YE Team
22
Gaps in Linking Evaluation Outcomes with Research
How to Address Them?

• There are two levels at which the success of
  interventions can be assessed
  
• Impact (done in 5YE)
• Diseases burden and its changes (Could not be
  done in the 5YE)
  
• In order to assess the changes in the diseases
  burden and causes of death, at least a sample of
  facilities should capture individual data over a
  long period about initial assessment-investigation
  -treatment-outcome. Currently, pre-coded,
  unlinked and mostly uni-variate data in the HMIS
  makes it difficult to do this.

Christopher Simoonga, Kumbutso Dzekedzeke and
Osward Mulenga
on behalf of Zambia 5YE Team
23
Specific Recommendations Enhance the Link
between ME and Research

• Expand and improve the data and statistical
  skills in Health sector. Small establishment
  overwhelmed with routine compilation of data
  leaving little time to use the data to answer
  evaluation and research questions.
• Health sector should strengthen relationships
  with local research institutions. Deliberate
  policy to task such agencies to analyse
  information would make-up for the shortage of
  skills and time in health sector agencies
• Benchmarks linked to incentives for health sector
  service delivery managers
  
• Clear annual targets couched with research and
  evaluations all the way to the health post
  
• Data collection, compilation and transmission to
  the centre should be a major benchmark for
  managers at different levels
  

Christopher Simoonga, Kumbutso Dzekedzeke and
Osward Mulenga
on behalf of Zambia 5YE Team