The Meaning of Success: Tracking Recommendations

1
The Meaning of Success Tracking Recommendations

• Yves Genest
• Office of the Auditor General of Canada

Canadian Evaluation Society Vancouver June 3rd,
2003
2
Why recommendations?

• From a policy perspective
• Provide policy alternatives
• Provide operational options
• From an audit/evaluation perspective
• Suggest corrective actions

3
What is a good recommendation?

• Fully supported
• Responding to the underlying cause of deficiency
• Clear and succinct
• Action-oriented
• Positive in tone and content
• Cost-effective
• Consistent with other recommendations
• Able to be followed-up

4
The OAG approach to follow-ups

• Until 2001
• Every Value-for-Money chapters followed-up after
  two years
• Since 2001
• Status Reports some key strategic chapters
  followed-up in depth (re-audited) after two or
  three years
• Yearly monitoring of recommendations

5
Issues in assessing status of recommendations

• They vary in complexity
• Acceptance by the organization audited/evaluated
• What is progress?
• Dichotomous (Done/Not Done)
• Incremental (Shades of grey)
• Time

6
Old Rating Scale of the OAG

• Implementation complete and to our satisfaction
• Action complete because the recommendation is no
  longer appropriate or has been passed by events
• Progress is being made at a satisfactory pace
• Progress is being made but not satisfactory
• Entity rejects the recommendations
• Unknown not available

7
Results of Old Rating Scale (2000)
8
Some problemsand their solutions

• Complex recommendations and government-wide
• Level of satisfaction vs. progress
• Level of complexity
• Unknown / not available?

9
Schematic of rating system
Level of Difficulty
Level of Progress
Level of Satisfaction
10
Difficulty Rating
11
Level of Progress
12
Level of Satisfaction
13
In summary

• Identification and sequencing of factors
  influencing the status of recommendations
• Subjective element of judgment is acknowledged
• Creation of a more detailed and flexible database
• Performance reporting