Targeted Evaluation of Five Programs Supporting Orphans and Vulnerable Children: Background and Meth

1
Targeted Evaluation of Five Programs Supporting
Orphans and Vulnerable ChildrenBackground and
Methods

• Florence Nyangara, PhD
• MEASURE Evaluation/Futures Group
• Dissemination Meeting, September 3rd, 2009
• Washington, DC

2
The number of OVC and their corresponding
programs increasing
Background

• SSA - 12 million orphans (0-17 yrs), 2003
• 2010 - over 18 million orphans (O)
• Other millions are made vulnerable HIV/AIDS,
  dire poverty, war, etc (V)
• Response increased attention to the plight of
  OVC (funds, programs)

Sub-Saharan Africas population of children
orphaned by AIDS increasing
Children on the Brink, 2004
3
Response to the OVC Crisis
Background

• OVC programs emergency response to areas most
  HIV-affected
• Strategies used were based on existing cultural
  support systems, conventional wisdom, and lessons
  learned from other program areas
• Support community-based responses (capacity
  resources)
• Household/family support (capacity resources)
• Direct support to families OVC (access to
  essential services)
• Gap lack of evidence to guide OVC programs
• Call for evidence based programming
• 2006 - USAID funded MEASURE evaluation to
  conduct targeted evaluations to fill this
  evidence gap

4
Evaluation Goals

• Find out what works in terms of
• intervention models and program components
• cost effectiveness, and
• outcomes (benefits) for OVC and their caregivers
  in resource poor settings
• Provide evidence to guide program decisions such
  as
• Scaling-up of best practices (models,
  strategies), and
• Modify improve interventions - to make them
  effective

5
Research Preparation Activities

• Funds were made available USAID/PEPFAR/PHE (4)
  and USAID/Tanzania mission funded (1) program
  evaluation.
• Research team formed - MEASURE Evaluation
• Extensive literature reviews (early 2006)
• Consultation meetings with stakeholders
• Identified OVC programs to be evaluated
• Research protocol developed
• Ethical approvals obtained US, Kenya, and
  Tanzania
• Identified local research partners (PSRI KE
  AXIOS - TZ)

6
Programs Evaluated

• Evaluated Five programs 2 in Kenya 3 in
  Tanzania
• They have different intervention models with
  varied combinations of child, family/household,
  and/or community centered approaches
  (multi-faceted).
• Although, the approaches vary, the goal for
  all of these programs is to improve the
  well-being of OVC and their families.

7
Programs Evaluated

• Kenya (2)
• Kilifi OVC Project, Catholic Relief Services
  (CRS)
• Operating in Kilifi District for two years
• Community Based HIV/AIDS Prevention, Care and
  Support Program (COPHIA), Pathfinder Integrated
  AIDS Program (IAP)
• Operating in Thika District for 4 years
• Tanzania (3)
• CARE Tumaini Project, Allamano, CARE, FHI
  (Allamano)
• Operating in Iringa Region for five years
• Mama Mkubwa Kids Club, Salvation Army (TSA)
• Operating in Mbeya Region for 2-years
• Jali Watoto, Pact/SAWAKA (Jali Watoto) Field
  funded
• Operating in Karagwe, Kagera Region for four
  years

8
Overview of Programs Strategies Evaluated
9
Key Research Questions

• Impact of indirect support
• How do efforts targeted at the structural systems
  surrounding children household and community
  affect
• Children well-being
• Caregivers well-being
• Community attitudes and support of OVC
  families?
• Impact of direct support on child outcomes
• What is the impact of educational, health, legal
  support, and other direct services on child
  families?

10
Methods

• Case studies (2006 and 2007)
• Site visits, interviews, program document review
• Provide understanding of program strategies,
  components, goals, and expected outcomes
• Document lessons learned from implementation
• Case Studies available http//www.cpc.unc.edu/meas
  ure/ovc
• Program expenditures (2006)
• Expense data collected and social costs estimated
• Quantified the costs corresponding to specific
  intervention (e.g. food supplementation,
  psychosocial service, educational support)

11
Methods

• Outcome evaluation (2007 and 2008)
• Post-test study design with intervention
  comparison groups
• Exposed Vs. Non-exposed
• Surveyed children age 8-14 or 7-15 their
  caregivers
• Up to 2 children per household
• Four questionnaires were applied in each
  household
• Q1 Household Questionnaires
• Q2 Parent/Guardian/Caregiver Questionnaire
• Q3 Parent/Guardian/Caregiver Regarding Child
  Questionnaire
• Q4 Child (age 8-14 or 7-15) Questionnaire

12
Ideal Impact Assessment
R
13
Post-test Study Design Used
The Groups are Not Randomly Assigned
Exposed
X O1
O2
Non-exposed
Jali Watoto Study compared intact groups
of intervention versus comparison
14
Research Designs and Sampling
15
Principles Guiding Questionnaires Development

• Capture multiple measures for each domain
• Use existing standardized scales where possible
  (PSS, SES)
• Intervention Exposure questions to be specific to
  each program
• To facilitate comparison across countries and
  program models, same survey instruments were used
  except intervention modules
• Multi-faceted programs necessitated sufficient
  questions across multiple domains
• Multiple perspectives on child well-being (child
  and caregiver)
• Measures of caregiver, household community
  well-being

16
Outcomes Examined

• Psychosocial well-being multiple measures
  standard scales used where appropriate (child and
  caregivers)
• Education enrollment attendance (child)
• Health self-reported health status and access
  to health services (child and caregiver)
• HIV-prevention HIV-knowledge (child)
  HIV-testing (caregiver)
• Legal protection birth registration, alternate
  caregiver
• Community support stigma and in-kind support
  (child caregiver)

17
Study Strengths and Limitations

• Strengths
• Yielded immediate data on program effects
• Results can be used to improve current programs
• Ethical - not withholding services for experiment
  sake
• Limitations
• Post-test design no baseline data - impossible
  to make conclusions concerning change in outcomes
  resulting from program exposure
• Selection bias - self-selection to participate
  and those who did not -makes it difficult to
  conclude with certainty that the interventions
  are responsible for the observed differences

18
Analyses Plan

• Who are the OVC/MVC program beneficiaries?
• Effects of community level interventions i.e.
• Community care and support meetings/sensitization
  
• Effects of household or caregiver level
  interventions i.e.
• Community volunteer or Health Worker home visits
• Caregivers participation in OVC care seminars
• Effects of child level interventions i.e.
• Kids clubs
• Basic needs support (e.g., education, health,
  legal)

19
Statistical Analyses

• Descriptive analysis (Univariate)
• Bivariate analysis (ANOVA and Chi-square)
• Multivariate (logistics, and linear regression)
• Control variable non-program factors e.g.
  socio-demographic
• Child Level Age, sex, orphan status,
  relationship to caregiver, and number of
  different homes the child had lived in the past
  year.
• Caregiver level Age, sex, marital status,
  education, illness, SES, and children

20
Description of OVC Program Beneficiaries

• Who is enrolled in OVC programs (MVC Profile)-
• Although, these programs targeted geographic
  areas most affected by HIV/AIDS, MVC were
  identified and assisted regardless of specific
  causes of vulnerability
• Majority of children enrolled in OVC programs are
  vulnerable in several fronts not just
  orphanhood

21
OVC Profile

• Orphans (over 66 across programs)
• Living in food insecure HH (over 80 across
  programs)
• Poorest households (lt 2assets) over 40
• Living with chronically ill primary caregiver
  (over 20)
• Living with caregiver aged 50 (about 23)
• Lived in two or more households in past year (14)

22
Thank YOU!

• Key Findings are presented next.

23

• MEASURE Evaluation is funded by the U.S. Agency
  for
• International Development through Cooperative
  Agreement
• GHA-A-00-08-00003-00 and is implemented by the
  Carolina
• Population Center at the University of North
  Carolina at
• Chapel Hill, in partnership with Futures Group
  International,
• ICF Macro, John Snow, Inc., Management Sciences
  for
• Health, and Tulane University. The views
  expressed in this
• presentation do not necessarily reflect the views
  of USAID or the United States government.
• Visit us online at http//www.cpc.unc.edu/measure.