World Bank Independent Evaluation Group - PowerPoint PPT Presentation


PPT – World Bank Independent Evaluation Group PowerPoint presentation | free to download - id: 8dcac-ZDc1Z


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation

World Bank Independent Evaluation Group


Understanding the communication style of the decisionmaker ... tried to educate him in the classical rational way, without reference to any ... – PowerPoint PPT presentation

Number of Views:32
Avg rating:3.0/5.0
Slides: 59
Provided by: michaelb159
Learn more at:


Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: World Bank Independent Evaluation Group

World Bank Independent Evaluation Group
  • How to Increase the Utilization of Evaluations
  • Michael Bamberger

Session outline
  1. Defining and measuring evaluation utilization
  2. Reasons why many evaluations are under-utilized
  3. Examples of evaluation utilization the World
    Bank Influential Evaluations study
  4. Ways to strengthen utilization
  5. If time permits .. Further discussion on
    presenting the message

  • 1. Defining and
  • measuring
  • evaluation utilization

Defining evaluation outcomes and impacts
  • Use
  • How evaluation findings are utilized by
    policymakers, managers and others
  • Influence
  • How the evaluation influenced decisions and
  • Consequences
  • How the process of conducting the evaluation, the
    findings and the recommendations affected the
    agencies involved and the target populations..
  • Consequences can be
  • Positive or negative
  • Expected or unanticipated

Measuring evaluation outcomes and impacts
  • Changes in individual
  • Knowledge
  • Attitudes
  • Behavior
  • Changes in organizational behavior
  • Changes in program design or implementation
  • Changes in policies and planning
  • Decisions on project continuation, expansion and

Measurement issues
  • Time horizon
  • Intensity
  • Reporting bias
  • Many agencies do not acknowledge they have been
  • Attribution
  • How do we know the observed changes were due to
    the evaluation and not to other unrelated

Attribution analysis
  • How do we know if observed changes were due to
    the evaluation and not to other influences?
  • Stakeholder interviews
  • Surveys pretest/posttest or posttest only
  • Analysis of planning and policy documents for
    evidence of evaluation influence
  • Analysis of mass media
  • Key informants

Examples of attribution methodology used in
Influential Evaluations
  • See
  • Influential evaluations Detailed case studies
    pp. 69-72

Attribution analysis framework
2. Assess whether there is a plausible case
for attributing part of the effects to the
1. Identify potential Impacts (effects)
3. What proportion of the effects can be
attributed to the evaluation?
A. Comparison of 2 user surveys Stakeholder
opinion survey
  • Comparison of sample surveys of service users (in
    1993 and 1999) found reported improvement of
    services potential impact
  • Sample of 35 public service agencies,
    policy-makers, mass media and civil society
    corroborated influence of survey in influencing

Bangalore Citizens Report Cards
B. Testimonials from stakeholders
Bulgaria Metallurgical Project and Indonesia
Village Water Supply
  • Clients (Metallurgical company, Development Bank
    AUSAID Project Dept, Indonesian water agency)
    asked to send testimonials in letter or e-mail
    confirming influence of the evaluation.
  • In Bulgaria clients asked to confirm validity of
    benefit projections and to confirm benefits were
    attributable to the evaluation.

India Employment Assurance Program
  • Considered too bureaucratically difficult to
    solicit Government views on effectiveness of
    government agency (Evaluation Organization).
  • Opinions of Bank Resident Mission and other
    experts solicited on the influence of the
    evaluation and the credibility of the estimates
    of cost-savings and employment generation.
  • Paper trail specific references in the Five
    Year Plan and follow-up sector planning documents
    to how the evaluation was used.

D. Logical deduction from secondary sources
Public expenditure tracking study (education)
  • Follow-up PETS study estimated increased funds
    utilization (potential evaluation impact)
  • Extensive coverage of the report in the media
  • Government documents show how the findings were
  • Reports show how community groups use the budget
    information posted in schools/ media.

2. Reasons why evaluations are under-utilized
1 Lack of ownership
  • Evaluation focus and design are determined by
    donor agencies or outside experts with little
    real input from client.
  • The goals definition game alienates clients
  • Limited consultation with, and feedback to,
  • Evaluation seen as a threat

2 Timing
  • The evaluation findings are
  • presented too late to be useful
  • Too soon before policymakers or managers have
    started to focus on the issues discussed in the

3 Poor communication between evaluator and
  • Clients are not kept in the loop
  • Clients may not like evaluators communication
  • Language problems
  • Conceptual problems
  • The objectivity paradigm limits contact and
    communication between evaluator and client
  • Client does not share information with other

4 Lack of flexibility and responsiveness to
client needs
  • Rigid design that cannot be adapted to client
    needs or changing circumstances
  • Quasi-experimental design that cannot adapt
    indicators and data collection methods to
    changing circumstances.
  • Objective stance of evaluator limits
    interaction with clients.
  • Timing too early or too late
  • continues next page

  • Finance ministries try to force evaluation
    indicators and focus to correspond to budget line
  • National evaluation systems sometimes introduce
    top-down, uniform evaluation/reporting systems
    not reflecting the reality of different agencies

5 Resource constraints
  • Budget constraints affect
  • Data collection
  • Data analysis
  • Bringing staff together to participate in the
    evaluation process
  • Translation into local languages
  • Report preparation and dissemination
  • Limited local expertise

6 Time constraints
  • Too many demands on client and stakeholders time
  • The evaluators do not have enough time to
  • Consult with clients during evaluation planning
  • Design and implement the evaluation properly
  • Discuss the draft report with clients
  • Organize effective dissemination meetings

7 Relevance
  • The evaluation does not address priority
    information needs of clients
  • Much of the information is not considered useful
  • The information is not analyzed and presented in
    the way that clients want
  • Too detailed
  • Too general

Factors external to the evaluation affecting
  • Problems with the evaluation system
  • Dissemination mechanisms
  • Political ethics (attitudes to transparency)
  • Clients lack of long-term vision
  • Government perception of evaluation

3. Examples of evaluation utilization The World
Bank Influential Evaluations study
How are evaluations used? When are they
  • Evaluation never the only factor. How does the
    evaluation complement other sources of
    information and advice.
  • Political cover for difficult decisions
  • Identifying winners and losers and showing
    how negative impacts can be mitigated.
  • Credibility and perceived independence of the
    evaluator may be critical

  • continued next page

  1. The big picture helping decision-makers
    understand the influence of the social, economic
    and political context.
  2. Help managers understand how political and other
    pressures limit project access to certain groups
  3. Providing new knowledge or understanding
  4. Catalytic function bringing people together or
    forcing action.

Types of influence that evaluations can have
  • India Employment Assurance
  • Broader interagency perspective helped identify
    duplications and potential cost savings.
  • Evaluation Office had high-level access to
    Planning Commission
  • India Citizen Report Cards
  • Alerting management to service problems and
  • providing quantitative data to civil society
    pressure groups
  • Indonesia Village Water Supply
  • Making policy-makers aware of importance of
    gender issues and participatory approaches
  • continued next slide

  • Large Dams
  • Created political space for introducing new
    social and environmental criteria for evaluating
    dams and
  • launching dialogue that facilitated creation of
    World Commission on Dams.
  • Pakistan Wheat Flour Ration Shops
  • Political cover for sensitive political decision
  • Showed how to mitigate negative consequences
  • continued next page

  • Uganda Education expenditures
  • Developed methodology to document what everyone
    suspected (expenditure wastage)
  • provided documentation to civil society to
    pressure for improvements
  • 7. Bulgaria Metallurgical Project
  • Alerting borrowers and Development Bank to new EU
  • showing how to avoid fines
  • how to advance launch of mineral production
  • continued next page

  • China Forestry Policy
  • Legitimized questioning the logging ban
  • promoting more in-depth policy research
  • facilitating creation of Forestry Task Force


What difference did the evaluation make?
  • Major cost savings (India, Bulgaria, Pakistan)
  • Increased financial benefits (Uganda, Bulgaria)
  • Forced action (Bangalore, Uganda)
  • Strengthened gender and participatory planning
    and management of water (Indonesia)
  • Introduced social assessment of dams but
    discouraged future investments (Dams)
  • Continued next slide

  1. Increased efficiency of service delivery (India,
    Bangalore, Indonesia)
  2. Facilitated creation of important policy agency
    (Dams, China)

4. Ways to strengthen evaluation
Ways to strengthen evaluation utilization
  • 1. Deciding what to evaluate
  • 2. Timing
  • When to start
  • When to present the findings
  • 3. Deciding how to evaluate
  • Choosing the right methodology
  • 4. Ensuring effective buy-in
  • Stakeholder analysis and building alliances
  • The importance of the scoping phase
  • Formative evaluation strategies
  • Constant communication with clients

  • 5. Evaluation capacity building
  • 6. Deciding what to say see next section
  • 7. Deciding how to say it see following
  • Effective communication strategies
  • 8. Developing a follow-up action plan

6. Deciding what to say
  • Technical level
  • Amount of detail
  • Focus on a few key messages
  • Target messages to key audiences

Sources of lessons about a program
  • Evaluation findings
  • Experience of practitioners
  • Feedback from program participants
  • Expert opinion
  • Cross-discipline connections and patterns
  • Strength of linkages to outcomes

Identifying evaluation lessons and generating
  • Tactics for generating meaning (Handout 1)
  • Identifying high quality lessons (Handout 2)

7. Presenting the message
  • Communication style and choice of media (Handout
  • Focus report on intended users
  • Quantitative and qualitative communication styles
    (Handout 4)
  • The clients preferred communication style
    (Handout 5)
  • Making claims
  • The importance of graphics
  • Who receives the evaluation report and who is
    invited to comment

If time permits ..
  • More detailed discussion on presenting the message

Presenting the message
  1. Communication style and choice of media
  2. Utilization-focused reporting principles
  3. Quantitative and qualitative communication styles
  4. The clients preferred communication style
  5. Rules for written reports
  6. Making claims
  7. The importance of graphics
  8. Who receives the report and who is invited to

1. Communication style and choice of media
  • Continuous communication throughout the
  • No surprises
  • Educating the client how to think about
  • Short versus long reports
  • Combining verbal and written presentations
  • Slide shows
  • Informal versus formal

Communication style .. continued
  • Alternative media
  • Internet
  • Video
  • Theater
  • Dance
  • Murals/paintings
  • Posters
  • Signs

Communication style .. continued
  • Using the right language
  • Making sure the written report is available in
    stakeholder languages
  • Economical ways to translate
  • Personal testimony
  • Project visits
  • Working with the mass media

2. Utilization-focused reporting principles
  • Be intentional and purposeful about reporting
  • Focus reports on primary intended users
  • Avoid surprising stakeholders
  • Think positive about negatives
  • Distinguish dissemination from use
  • Source Patton (1997) 330-337

3. Quantitative and Qualitative communication
  • Tables versus text
  • Types of evidence
  • Statistical analysis
  • Case studies
  • Stories
  • Photos
  • Boxes
  • Site visits
  • Testimonials

3. The clients preferred communication style
  • Written and/or verbal
  • Quantitative/qualitative
  • Multiple presentations to different audiences

Understanding the communication style of the
  • High intellectual level Robert MacNamara, Elliot
  • These two individuals were perfectly capable of
    understanding the most complex issues and
    absorbing details absorbing the complexity,
    fully considering it in their own minds.
  • Lawrence Lynn. Prof. Public Administration. The
    Kennedy School. Patton 1997. PP 58-9.

Decisonmakers communication styles .. continued
  • Short personal stories Ronald Reagan
  • Preferred Readers Digest type personal stories
    and anecdotes.

Decisonmakers communication styles .. continued
  • Political animals Joseph Califano U.S. Secy
    Health, Education and Welfare
  • Califano is a political animal and has a
    relatively short attention span highly
    intelligent, but an action-oriented person. The
    problem that his political advisers had is that
    they tried to educate him in the classical
    rational way, without reference to any political
    priorities .. or presenting alternatives that
    would appeal to a political, action-oriented
  • Source Patton (1997) p.59.

5. Rules for written reports
  • Show, dont tell
  • Be brief and clear
  • Build the story with paragraphs
  • Write clear sentences
  • Omit needless words
  • Avoid jargon
  • Vaughan and Buss

Reporting results
  • Arranging data for ease of interpretation
  • Focusing the analysis
  • Simplicity in data presentation
  • Interpretation and judgment
  • Making claims
  • Useful recommendations
  • A futures perspective on recommendations
  • Source Patton (1997) pp 321-4

See next slide
5. Making claims
Importance of claims Importance of claims
Major Minor
Rigor of claims Strong
Rigor of claims Weak
Characteristics of claims of major importance
  • Involves having an impact
  • Deals with important social problem
  • Affects large numbers of people
  • Saves money and/or time
  • Enhances quality
  • Show something can really be done about a problem
  • Involves model or approach that could be

Characteristics of strong claims
  • Valid, believable evidence
  • Data covers a long period of time
  • Claim is about clear intervention with solid
  • About clearly specified outcomes and impacts
  • Includes comparisons to program goals, over time,
    with other groups, with general trends or norms

Strong claims continued
  • Evidence for claims includes replication
  • More than one site
  • More than one staff member obtains outcomes
  • Same results from different cohort groups
  • Different programs obtained comparable results
    using the approach
  • Based on more than one kind of evidence
  • Clear, logical linkages between intervention and
    claimed outcomes
  • Evaluators independent of staff

6. The importance of graphics
  • Line graphs trends over time
  • Pie charts parts of a whole
  • Cluster bar chart comparing several items
  • Combination chart bar chart and trend line
  • Dont overload graphs

7. Who receives the evaluation report and who is
invited to comment?
  • Restricted to few key decision-makers and
  • Participatory presentation/ consultation with
  • Public comment
  • Public hearings
  • Via civil society
  • Copies available to the press