Donor Demands and Uses for Evidence of Research Impact the Case of the CGIAR

1
Donor Demands and Uses for Evidence of Research
Impact the Case of the CGIAR

• David A. Raitzer
• Consultant to CGIAR Science Council, Standing
  Panel on Impact Assessment

2
Background

• Three commonly referenced primary uses for ex
  post impact assessment (epIA) findings
• accountability
• allocation
• internal learning.
• Accountability and allocation -gt donors, while
  learning -gt researchers managers
• First systematic attempt to understand
  accountability-related needs and demands for
  information related to research impact.
• Internal learning is the focus of a forthcoming
  initiative.

3
Methodology

• 2 Components
• Email survey of CGIAR Member representatives
• Telephone interviews of CGIAR donor
  representatives (conducted by Klaus Winkel)

4
Email survey of CGIAR donors

• An email survey was sent to representatives of
  all 63 CGIAR Members.
• The survey focused on the following
• Expectations for the role of epIA
• Satisfaction with epIA studies to date
• Use of epIA in allocation decisions
• Factors that facilitate epIA use
• Demanded metrics and methods
• Readership of epIA as compared with other forms
  of evaluation

5
Email survey

• 24 responses from the following 22 donors (69 of
  CGIAR Member funding)
• ACIAR, ADB, Austria, Belgium, DANIDA, DFID,
  EIARD, EU, GTZ (2), IADB, IFAD, KARI, Mexico,
  Morocco, Netherlands, Philippines, Rockefeller
• Responses indicated general importance of IA for
  accountability.

6
Email survey results - epIAs rated as important
information source in decisions
7
Email survey results - distance down the impact
pathway important determinant of use
8
The most poverty-relevant metrics were most
highly rated
9
Email survey results

• Nearly 60 claimed external epIA conduct
  important
• Moderate ratings for epIA credibility (median
  7/10), relevance (7), coverage (6)
• Greater readership reported for epIA than other
  forms of evaluation
• Approximately 50 reported direct use in
  decisions

10
Donor Interviews

• Telephone interviews of officials from 26 CGIAR
  donor agencies (that provide 90 of CGIAR
  funding) queried
• Investment decisions and the role of impact
  findings
• Donor perceptions of impact and how they are
  informed
• Determinants of epIA readership
• Examples of use of epIA results
• Specific demands for impact-related information

11
Interview results

• 26 CGIAR Member representatives interviewed
• ACIAR, Belgium, BMZ, CIDA, Columbia, DANIDA,
  DFID, EU, FAO, France, IADB, IDRC, IFAD, India,
  Ireland, Japan, Netherlands, Norway, Philippines,
  Rockefeller, SDC, Syngenta, SIDA, UNDEP, USAID,
  World Bank
• Overall funding level typically determined at
  high bureaucratic/political level few details
  are clear.
• Funding within the CGIAR typically decided within
  the domain of the official representative to the
  Group.
• Three-quarters reported moderate to high
  flexibility to shift funding according to new
  information.
• Half mentioned use of epIA to defend budgets.

12
Past impact is reported as a relatively minor
consideration in funding decisions
13
Perceptions of impact loosely follow epIA findings
14
Impact assessment reported as a minor source for
views on impacts
15
Preference for detailed analysis of effects at
the household level
16
Specific preferences - high demand for briefs and
more poverty-relevant metrics
17
Interview results

• Over a quarter of respondents expressed
  skepticism about the accuracy of past epIAs
• About ½ of interviewees could identify specific
  epIAs that they had used
• Most respondents indicated a desire to pair epIA
  findings with other kinds of information.

18
Analysis

• The observations of the present study largely
  agree with prior studies of evaluation use in
  other contexts.
• Instrumental use of evaluation findings as a
  direct basis for decisions has been rarely
  documented (Leviton and Hughes, 1981 Cousins and
  Leithwood, 1986).
• Conceptual or indirect use has been much more
  commonly observed, as has symbolic or advocative
  use (Rich, 1977 Shulock, 1999).

19
Analysis

• Similarly, indirect conceptual and symbolic
  use of epIA seem prevalent in the CGIAR.
• Most information sources cited for impact
  perceptions embed epIA findings.
• EpIA is important for confidence in the Centres
  and affects perceptions of capacity and
  relevance.
• It may be appropriate to rely heavily on other
  information sources for allocation decisions, due
  to methodological problems for epIA of many
  research areas and long lag times to impact.

20
Implications

• EpIAs are important for confidence,
  justification, particularly at critical
  junctures
• Other forms of evaluation are important conduits
• Certain demands (emphasis on independentconduct,
  end of impact pathway metrics) may be
  incompatible with internal learning needs
• Briefs and other dissemination media important
  (backed up by rigourous analysis)
• EpIAs need to be more transparent about
  assessment and present more research details
• Demand for wider topical coverage

21
Conclusions

• High demand for epIA among donors, but influence
  is more difficult to observe.
• EpIA is an appropriate accountability tool, but
  it is not applied to allocation decisions in
  isolation.
• EpIA findings reach key audiences indirectly, and
  need to be embedded in other forms of evaluation.
• Some specific changes have the potential to
  increase epIA influence

22
Many thanks for your attention!