An analysis of influence of evaluation in the context of Socio-Economic Development Plan (SEDP) implementation in Vietnam - The case of Hai lang district, Quang tri province - PowerPoint PPT Presentation

Loading...

PPT – An analysis of influence of evaluation in the context of Socio-Economic Development Plan (SEDP) implementation in Vietnam - The case of Hai lang district, Quang tri province PowerPoint presentation | free to view - id: 8002f1-MmM0Y



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

An analysis of influence of evaluation in the context of Socio-Economic Development Plan (SEDP) implementation in Vietnam - The case of Hai lang district, Quang tri province

Description:

An analysis of influence of evaluation in the context of Socio-Economic Development Plan (SEDP) implementation in Vietnam - The case of Hai lang district, Quang tri ... – PowerPoint PPT presentation

Number of Views:73
Avg rating:3.0/5.0
Slides: 41
Provided by: eduv49
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: An analysis of influence of evaluation in the context of Socio-Economic Development Plan (SEDP) implementation in Vietnam - The case of Hai lang district, Quang tri province


1
An analysis of influence of evaluation in the
context of Socio-Economic Development Plan (SEDP)
implementation in Vietnam - The case of Hai lang
district, Quang tri province
  • Ha Minh Tri
  • HCMC, 2 April 2015

2
Outline (1/2)
  • Introduction
  • 1.1 The problem statement
  • 1.2 The research context
  • 1.3 The purpose of the study
  • 1.4 Overview of research steps
  • Literature review
  • 2.1 Literature review on evaluation use
    influence
  • 2.2 Conceptual framework
  • 2.3 Research questions
  • Case study setting (Hai Lang district)

3
Outline (2/2)
  • Research methodology
  • 4.1 Research design
  • 4.2 Research instruments data collection
  • 4.3 Hypotheses
  • 4.4 Limitations
  • Analysis and findings
  • 5.1 Data analysis
  • 5.2 Findings
  • 5.3 Construct validity and reliability
  • Research and policy implications
  • 6.1 Contributions to theory
  • 6.2 Implications for future research policy

4
1. Introduction
5
1.1 The problem statement
  • Importance of accountability and learning.
  • Accountability challenge for public sector
    management.
  • Monitoring and evaluation (ME) findings
    underused to serve SEDP targets and objectives.
  • Empirical studies on evaluation influence none
    in the public sector Vietnam.

6
1.2 Research context Monitoring Evaluation
in Vietnam
  • Inspection system still limited to cope with
    state management requirements.
  • ME system for Poverty Reduction Strategy Paper
    (PRSP) and SEDP at central level switching from
    traditional to evidence-based. This shift is at a
    slower speed at district and commune levels.

7
1.3 Purpose of the study
  • - To identify and explain factors which may
    affect evaluation influence in the public sector
    of Hai Lang.

8
1.4 Overview of research steps
Developing a conceptual framework based on Theory
of Evaluation Influence
Exploring research setting ME in Vietnam
Conducting literature review to contextualise
the research
Start with problem statement
Research design methodology Mixed Methods
Explanatory Design (1) QUAN methodology uses
EFA, CFA and Structural Path Model and (2) QUAL
study is used to explain QUAN results.
Developing specific research questions
hypotheses
Developing research instruments collecting data
Empirical research data analysis, presenting
findings, analysis discussion (with QUAL
analysis enhancing QUAN findings)
Conclusions implications
9
2. Literature review
10
2.1 Literature review Main findings on
evaluation use (1/6)
  • Evaluation that was underused in the US in late
    1960s stimulated research on evaluation use.
  • Literature on evaluation use emerged since 1970s.
  • Five most common types of evaluation use
    instrumental, conceptual, persuasive, process and
    imposed use.

11
2.1 Factors affecting evaluation use (2/6)
7 factors under evaluation impl. 6 factors under decision/policy setting 9 factors under stakeholder involvement
Evaluation quality Information needs Involvement and information needs
Credibility Decision characteristics Involvement and credibility
Relevance Political climate Involvement and relevance
Communication quality Competing information Involvement and communication quality
Findings Personal characteristics Involvement and findings
Timeliness Commitment /receptiveness to evaluation Involvement and commitment/receptiveness to evaluation
Evaluator competence Involvement and personal characteristics
Involvement decision characteristics
Direct stakeholder involvement
12
2.1 Main findings on evaluation influence (3/6)
  • Evaluation influence appeared since 2000s as
    scope of evaluation use was limited.
  • Evaluation influence evolved with 2 remarkable
    models/theories (1) Integrated model of
    influence (Kirkhart, 2000), and (2) Theory of
    Evaluation Influence (Mark and Henry, 2004)

13
2.1 Mark and Henrys Theory of Evaluation
Influence (2004)
14
2.1 Where is this dissertation positioned? (6/6)
Evaluation use (since 1970s)
Instrumental use
Conceptual use
Persuasive use
Process use
Imposed use
Evaluation influence (since 2000s)
Integrated model of influence (Kirkhart, 2000)
Theory of Evaluation Influence (Mark and Henry, 2004)
This work
Evaluation influence in Vietnam public sector
15
2.2 Conceptual framework (1/2)
Evaluation Inputs
Evaluation Activities
Evaluation Influence
  • Evaluation context
  • Lessons learned/ best practices in evaluation
  • Evaluation partners
  • Evaluation capacity
  • Involvement of programme holders in evaluation
    processes
  • Generation of evaluation reports
  • Dissemination of evaluation reports/findings
  • Evaluation influence at individual level
  • Decision/Policy setting
  • Evaluation plans methods
  • Contingencies in environment
  • Facilitating factors e.g. willingness to
    cooperate, etc.
  • Inhibiting factors e.g. wrong time, time
    pressure, etc.

Source Developed from Mark Henry (2004).
16
2.2 Conceptual framework (2/2)
  • Theory-based evaluation approach was used to
    elaborate the conceptual framework to establish a
    theory of change of evaluation influence
    framework.
  • This theory of change was used as a guide to
    develop a research instrument.

17
2.3 Research questions
Two central questions guided the study
  1. What are the factors which may affect evaluation
    influence at district and commune levels of Hai
    Lang district as perceived by the staff involved?
  2. How do the identified factors impact on
    evaluation influence at district and commune
    levels of Hai Lang district?

18
3. Case study setting
19
3. Case study setting (1/2)
20
3. Case study setting (2/2)
  • Consisting of 21 communes and township.
  • A typical rural, in-land district, not a border,
    remote or mountainous district.
  • Not a district with a high percentage of ethnic
    minorities.
  • Supported by a Finnish-funded Rural Development
    Programme for 13 years (1997-2009) in 3 phases.
    Phase I (1997-2000) for 14/21 communes and
    township. Phases II III for all communes and
    townships.
  • Programmes main objectives include (1) improving
    local livelihoods, and (2) strengthening local
    capacity (participatory planning, ME, public
    administration reform, etc.)

21
4. Research methodology
22
4.1 Research design
  • The study applied a mixed methods explanatory
    design
  • Methodology QUAN survey QUAL study

Quantitative Data collection and analysis
Qualitative Data collection and analysis
Follow up with
Interpretation
Source Creswell (2012).
23
4.2 Research instruments data collection (1/2)
  • For quantitative survey
  • Used a 39-item questionnaire measured by a
    7-point Likert scales for a sample of 275
    participants
  • Measured 8 dimensions
  • (1) Lessons learned/best practices in evaluation
  • (2) Programme partners
  • (3) Evaluation capacity
  • (4) Evaluation plans and methods
  • (5) Involvement of programme holders in
    evaluation processes
  • (6) Generation of evaluation reports
  • (7) Dissemination of evaluation
    reports/information to stakeholders
  • (8) Evaluation influence

24
4.2 Research instruments data collection (2/2)
  • For qualitative study
  • Used semi-structured interviews and document
    review.
  • Interviewed 55 purposefully selected participants
    with 12 different characteristics (see next
    slide).
  • Documents for review include inspection reports,
    project/programme proposals, progress and
    evaluation reports.

25
Group code Characteristics of individuals
Group 1 With longer exposure to Programme 5
Group 2 With shorter exposure to Programme 4
Group 3 District 4
Group 4 Commune 4
Group 5 Socio-economic sector 5
Group 6 Agriculture, Natural Resources and Environment sector 5
Group 7 Health and Education sector 4
Group 8 Administration and Justice sector 5
Group 9 Culture and Information sector 5
Group 10 College 5
Group 11 Manager 4
Group 12 Staff 5
Total 55
26
4.3 Hypotheses
  • H1 Involvement of evaluation partners (Factor 1)
    is positively associated with evaluation
    influence.
  • H2 Evaluation capacity (Factor 2) is positively
    associated with evaluation influence.
  • H3 Evaluation plans and methods (Factor 3) is
    positively associated with evaluation influence.
  • H4 Generation of evaluation report for
    management (Factor 4) is positively associated
    with evaluation influence.
  • H5 Generation of evaluation report for
    partnership (Factor 5) is positively associated
    with evaluation Influence.
  • H11 Path model (see figure next slide) holds for
    groups with longer and shorter exposure to
    Finnish-funded programmes (incl. 5 factors and 5
    controls).

27
4.4 Limitations
  • Participants challenges to recall effects of
    evaluation may affect explanations of
    quantitative results.
  • Lack of a comprehensive database of documents and
    limited accessibility to documents might
    influence the explanations and confirmations of
    quantitative results.
  • Bias in interpretation of qualitative results due
    to fact that author was previously working in the
    Programme (1 year).

28
5. Analysis findings
29
5.1 Quantitative data analysis Steps of analysis
Research question 1 What are the factors which
may affect evaluation influence at district and
commune levels of Hai Lang district as perceived
by the staff involved?
30
5.2 Quantitative findings (1/3)
  • 5 factors resulted from EFA process
  • (1) Evaluation partners
  • (2) Evaluation capacity
  • (3) Evaluation plans methods
  • (4) Generation of evaluation report for
    management, and
  • (5) Generation of evaluation report for
    partnership
  • CFA used to test multi-group measurement model,
    construct validity reliability, incl. groups
    with longer shorter exposure Finnish-funded
    programme.

31
5.2 Quantitative findings (2/3)
  • Obtained 3 statistically significant factors (1)
    evaluation capacity, (2) evaluation plans
    methods, and (3) generation of evaluation report
    for partnership.
  • No significant difference between groups with
    longer and shorter exposure to Finnish-funded
    programme.
  • None of the control factors (gender, age, years
    of education, level of responsibility and level
    of administration) confounds the relationships
    specified in the model.

32
5.2 The path model (3/3)
33
5.3 Reliability construct validity
  • Criteria
  • Construct reliability (CR) gt .70
  • Construct validity
  • Results
  • 24 out of 28 items satisfied construct validity
    and reliability criteria

Convergent Validity Factor loadings .50 Average variance extracted (AVE) .50
Discriminant Validity Maximum shared variance (MSV) lt AVE Average shared variance (ASV) lt AVE
34
5.1 Qualitative data analysis
  • Research question 2 How do the identified
    factors impact on
  • evaluation influence at district and commune
    levels of Hai Lang district?
  • Interview data was analysed using Nvivo 8
  • A visual overview of qualitative data analysis

Initially reading through text data Dividing text into segments of information Labelling segments with codes Collapsing codes into categories and themes/findings Comparing themes/findings and categories
                   
Pages of text         Segment of text     30-40 codes     5-7 themes/findings       Similar or different themes/findings
                             
35
5.2 Qualitative findings
  • QUAL findings confirm the QUAN findings that both
    evaluation capacity, and evaluation plans and
    methods positively affect evaluation influence
    regardless of the 12 characteristics of
    interviewees.
  • Generation of evaluation report for partnership
    positively affects evaluation influence
    regardless of their college level, level of
    administration, sectors, groups with longer
    shorter exposure to programme.
  • A group of managers perceived no effect of
    generation of evaluation report for partnership
    on evaluation influence.

36
6. Research and policy implications
37
6.1 Contributions to theory
  1. The studied conceptual framework found valid in
    explaining evaluation influence in Hai Lang
    public sector.
  2. The studys results contributed construct
    measures capturing evaluation influence in
    Vietnam district public sector.
  3. The studys results provided new affirmation that
    evaluation capacity, evaluation plans and
    methods, and generation of evaluation report for
    partnership positively affect evaluation
    influence in Hai Lang.
  4. Qualitative data contributed to an understanding
    of evaluation influence in Hai Lang.

38
6.2 Implications for future research (1/2)
  1. Replicating this study in other districts with
    other characteristics than Hai Lang might provide
    a more comprehensive research setting for better
    insights of evaluation influence.
  2. Research at interpersonal and collective levels
    might provide a more complete picture of
    evaluation influence in SEDP implementation in
    Hai Lang.
  3. Longitudinal design might be used to capture
    perceived changes at different stages of
    evaluation processes.

39
6.2 Implications for policy (2/2)
  1. Maximize significant factors by using programme
    staff as motivators.
  2. Balance accountability and learning purposes in
    design and reporting of inspection and evaluative
    exercises.
  3. Involve non-governmental actors in ME processes
    of SEDP implementation.
  4. Enhance evaluation capacity of Ministries of
    Natural Resources Environment, and Planning
    Investment climate change mitigation to improve
    use of ME information.

40
Thank you for your attention!
About PowerShow.com