Evaluation: A Systematic Approach

1
EvaluationA Systematic Approach

• Joan Mangan, PhD, MST
• Gorgas Tuberculosis Initiative
• University of Alabama at Birmingham

2
Why Evaluate?

• Essential to the design, implementation, and
  appraisal of any program
• Enable program administrators and staff to refine
  and increase effectiveness of a program
• Established
• New

3
Why Evaluate?

• Assist administrators to satisfy the
  accountability requirements of program sponsors
• Internal Sponsors
• Ministry of Health
• Ministry of Security
• External Sponsors
• USAID
• Global Fund
• Sponsors require evidence that what was paid for
  and deemed desirable was actually accomplished

4
(No Transcript)
5
The Keys to Success

• In order for a TB program to achieve objectives,
  3 criteria must be met
• Measurable objectives have been specified
• Demonstrate links in the operation of the TB
  Program
• The individuals in charge have the motivation,
  ability, and authority to manage the program

6
Different Purposes. Different Strategies

• Three major classes of evaluation
• Situational Analysis
• a.k.a. - Planning Evaluation
• Process Evaluation
• Impact / Outcome Evaluations

7
Situational Analysisa.k.a Planning Evaluation

• Assess
• The extent and severity of the TB problem
• Current resources
• Begin to identify needs
• Conceptualization and design
• New programs
• Expansions to existing programs

8
Process Evaluation

• Determine a programs success in delivering
  resources, services, and benefits
• Monitoring program implementation
• What is being provided
• Alternative approaches to the diagnosis of TB?
• Alternative approaches to treatment and follow-up
  of patients
• How well services are provided
• All levels
• staff , patients , patient families , surrounding
  communities

9
Process Evaluation

• Monitoring program implementation
• When Where
• Day to day operations are efficient and meet the
  needs of the patient population
• Staff are equipped with needed resources to
  accomplish objectives
• Equipment, documentation forms, etc.

10
Impact / Outcome Evaluations

• Examine the extent to which a program is the
  cause of a positive change
• Traditional indicators
• Numbers of new TB cases
• Numbers of patients who complete treatment
• etc

11
Impact / Outcome Evaluations

• Aid in decision-making to continue, expand, or
  cut back a program
• Consider cost in relation to the benefits of a
  program
• Compare program costs with alternative strategies
  when working with few resources

12
(No Transcript)
13
The Process Evaluation

• Some may look at the process evaluation as an
  audit there are similarities but they are not
  identical activities
• Audits compare what is happening with what
  should be happening
• Process Evaluation identify and measure what
  has happened and why it has happened
• Example monitoring may reveal a delay in
  physicians obtaining the results of sputum smears
  to confirm a diagnosis of tuberculosis due to (a)
  not enough laboratory staff to read smears and
  (b) collection of sputum from numerous
  un-symptomatic individuals

14
The Process EvaluationMonitoring for Success

• Coverage - Is the program providing care to those
  at greatest need / highest risk for developing
  TB?
• Measuring Coverage
• Review program records
• Surveys of TB patients
• Community surveys
• Examine difference between persons who take
  advantage of the TB programs services versus
  those who are eligible but dont use program
  services or default on treatment
• Question can the population access TB program
  services?

15
Monitoring for Success

• Delivery of TB Program Services Systems
• Three main questions to ask
• Is diagnosis /treatment being provided? . or if
  the services for diagnosis/ treatment are
  available in an area are there sufficient
  services to cover the population?
• Is the correct diagnosis / treatment being
  provided?
• Is the diagnosis / treatment standardized?
• Also Important Presentation - Is the program
  presented in such a way that patients accept the
  diagnosis procedures and treatment that is being
  offered

16
Monitoring for Success

• Measuring Delivery of TB Program Services Systems
  
• Observation of activities
• Checklist
• Medical records
• Accuracy in diagnosis?
• Standardized treatment?
• Healthcare providers and staff
• Individual or group interviews with patients

17
Analysis of Monitoring Data

• Address 3 Issues
• Describe the TB program
• Compare TB program implementation between sites
• Assess conformity to the TB programs policies
  and procedures

18
Analysis Description of the Program

• Annual Program Reports
• Estimates coverage of population
• Describes the services delivered
• Reactions of patients to the program
• Examples
• Patients and community knowledge of TB, program
  services, access to care
• Patients seeking services why? / why not?
• Defaulting why? / why not?

19
AnalysisComparison between Sites

• Helps identify reasons for the differences that
  may exist in outcomes (case detection, treatment
  completion, etc.)
• These differences may be good or bad
• An assessment between sites can identify best
  practices . Or why some sites are more
  effective than others
• Sharing this information can enhance the overall
  program
• Increase standardization

20
AnalysisComparison between Sites

• Bi-annual evaluation meetings in Honduras
• An opportunity for staff on the front lines to
  learn and share ideas creative problem solving
• Provide teaching moments / clarify a policy or
  procedure
• Increased owner-ship in the program
• Motivation to improve a nicer form of
  accountability
• Allow program administrators to identify and
  prioritize those sites that need more
  attention/assistance

21
NTP Evaluation Meeting FormEfficiency and
Effectiveness
22
AnalysisConformity to the TB Program Policies
and Procedures

• Discrepancies between how a program is operating
  and the policies and procedures may lead to
• Efforts to move closer to the originally
  planned policies and procedures
• Redesign policies and procedures to reflect the
  best practices that arise out of necessity or
  creativity of staff on the front lines

23
Using Evaluation Information

• Decision making
• A rationale for action / change
• Go / No-Go Decisions
• Accountability
• Advocacy
• Policy

24
Take Home Message

• An evaluation of the process will provide you
  with specific targets for action with specific
  targets there is a decreased risk of wasting time
  and money

25
Take Home Message

• To obtain good information there is a need to
  get out in the field and talk to those people
  responsible for the day to day operation of a
  program as well as patients and their family
  members
• Data obtained from traditional TB Program
  indicators are not always enough
• Those on the front lines are in the best position
  to tell you
• What works,
• What doesnt work,
• What is needed,
• .. and are a good source of the best ideas

26
The Pragmatic Side of Any Evaluation

• Interests of sponsoring organizations may change
• Are these interests incorporated / reflected in
  evaluation activities?
• Unanticipated problems/solutions may arise and
  impact the way a program works
• Early identification and action can
• Avoid further damage
• Reap the benefits (example IEC programs in
  Honduras)

27
The Pragmatic Side of Any Evaluation

• Partial evaluations may provide knowledge that a
  program is not producing the desired outcomes
  the need for program refinement may be identified
  sooner rather than later