World Bank Policy on the Selection and Employment of Consultants: Study of its Effectiveness Authors

1
World Bank Policy on the Selection and Employment
of Consultants Study of its EffectivenessAutho
rs Gian Casartelli, OPCPR Prof.Dr. Elmar
Wolfstetter, Humboldt University
Operational Policy Country Services
Procurement The World Bank

• Key Findings and Recommendations
• Gian E. Casartelli - Consulting Services Advisor,
  OPCPR- WB

2
Study Objective

• Objective Assess effectiveness of the Guidelines
  in leading to efficient contract allocation by
  considering quality, efficiency economy, fair
  competition, transparency and the development of
  consultants from LDC
• Key Assumptions Effective selection depends to a
  large extent on
• (1) quality of regulation
• (2) capacity of users

3
Study Approach and Main Features

• Bank data on financed consulting assignments
  FY03/04
• 120 prior review client executed contracts
  randomly selected (6 incomplete)
• 88 QCBS, 2 QBS, 4 FBS, 2 CQS, 1 LCS, 17 SSS
• Each case examined from EOI to contract signature
• A large excel database was created (see example)
• 60 TTLs were interviewed
• 10 internal and external peer reviewers were
  involved
• Effort100 staff weeks, 14 months
• Funding CTFs and Bank budget

4
Finding 1 Efficiency and Economy

• The selection process takes too long and is too
  expensive
• Median duration 11 months (average 17) for
  preparing short list and RFP. Proposals include
  400 to 700 pages of technical text. Median
  contract value 432K.
• Duration and cost discourage consultants, clients
  and Bank. The validity of proposals and
  commitment diminish as work plans and CVs need to
  change before starting the assignment (moral
  hazards on supply and demand side).
• Capacity to implement well and timely plays a
  role!

5
Finding 2 Participation

• Averages 21 EoIs, 5.3 firms shortlisted, 4.1
  proposals received, 3 financial envelopes opened
• Participation of most known consulting firms has
  decreased and is low.
• Consideration for quality may be affected, but
• Participation of national consultants, a key
  consideration, has increased substantially, but
  what about access to quality and innovation?

6
Finding 3 Quality

• Clients dont consider risk level and complexity
  of ToR
• when assembling the selection process
• Para 2.3 on ToR does not link to the selection
  method
• Clients adopt QCBS in 92 of cases even when
  trading quality for price is not advisable or the
  short list is mixed (not homogeneous)
• The location wording of references to Selection
  Method, Price and Short List (Para 1.5, 2.1,
  2.23, 2.8) contribute to uncritical choices of
  selection method, to the Q/P weight
• Issues Bank staffs advise is too standard, and
  borrowers risk indifference plays a role

7
Type of Services versus Selection Method How
Things Are
Type of Services
Selection Methods
Strategy, Master Plans, Complex Study Design
Quality Based Selection
2 QBS
Qualification bs Quality/Cost Selection Fixed
Budget Selection
Standard TA and Capacity Building Services Bid
Docs, Detailed Design, Supervision Simple Projects
CQS 2 FBS 4 QCBS 91
Very Simple Services
Least Cost
1 LCS
8
Sample ToR suggest the following distribution
Type of Services
Selection Methods
Strategy, Master Plans, Feasibility Studies,
Services for Complex Projects
Quality Based Selection
35 QBS
Standard TA Services and Capacity Building Bid
Docs, Det Design Supervision Services Simple
projects
Qualif. Based Quality/Cost Selection,Fixd
BudgetSelect.
CQS 10 FBS 10 QCBS
40
Very Simple Services
Least Cost
LCS 5,
9
Finding 3 Quality

• In 35 of cases QCBS is adopted although the
  shortlist is unbalanced (Para 2.8 on mixed
  shortlists is not as clear as it should be and is
  disregarded)
• In 32 of cases or more QCBS is adopted while
  assignment is complex or risky (disregard of Para
  3.2)
• Minimum Technical Score, score distribution to
  evaluation criteria, Weights of Q and P, Type of
  Contract are not correlated with the level of ToR
  risk and complexity. Selection algorithm becomes
  an imprecise but very complex search mechanism

10
Finding 4 Opportunity to compete fairly

• Lot of room to improve short list
• Para 2.6 call for short list of six consultants
  with a wide geographic spread and not more than
  two firms from the same country but at least one
  from a developing country is met before
  considering consultants qualifications
• The provision on short list when rigidly applied
  can lead to unbalanced short lists (35 of cases)
  by including less qualified consultants (Para 2.8
  is not sufficiently clear and Para. 2.3 does link
  ToR with short list)
• Wording of Para 2.5. on Advertising does not hint
  to consultants qualifications

11
Finding 5 Evaluation Committee

• Correlation between quality of proposal and
  technical score assigned by EC members can be
  weak
• 38 of EC members are specialists, the rest EA
  directors and higher authority delegates
  (principal agent problems). In 37 of cases EC
  members position are not identified
• Decisive disagreement by individual EC members is
  19. Evaluation and TER approvals averages 131
  days
• Guidelines recommend (Para 2.15) the use of
  Evaluation Committee composed of specialists but
  text could be more effective without affecting
  borrowers ownership

12
Finding 6 Evaluation Results

• Also weak correlation between quality and price
• ECs assign similar technical scores to
  differently priced proposals, and our study finds
  no significant correlation between quality and
  price.
• The mean technical score dispersion is 1.12 while
  the mean price dispersion is 1.79.
• The highest technical score is awarded the
  contract in 64 of cases. The lowest financial
  proposal is awarded the contract in 67 of cases,
  and in 37 of cases the highest technical score
  is also the lowest priced proposal.

13
Finding 7 The Scoring Formula (Para 2.22)

• The scoring formula for QCBS
• S ? St (1 ?)Sf where Sf pmin/p 100
• The formula is flawed in the following ways
• It favors higher priced proposals (paradox!)
• It depends on the lowest price, pmin, in a way
  that makes it impossible for consultants to
  predict how the rule will eventually reward them
  for quality
• pmin makes it difficult for the Client to
  calibrate the scoring
• These characteristics are troublesome because the
  mean price dispersion is above 2 in 28 of cases.
• Even if client knows how to evaluate and is
  honest, the formula can distort the ranking.

14
Annex 2 Relationship between MRQ and price p
(old and new formula)
15
Finding 8 National Consultants

• GL provisions promoting national consultancy have
  contributed to increased participation by
  national consultants
• However Gresham law seems at work
• Recurrent use of QCBS and prevalently price-based
  domestic regulation lead to disregard for quality
  and low remuneration levels. This is hurting the
  sustainability of domestic consulting industries
  ( brain drain etc)
• 77 of key staff is national. However reasons not
  clear a) better quality of service, b)
  consideration for development of nationals, or
  because c) lower proposal prices. Bait and
  switch frequently observed.

16
Finding 9 Transparency

• Guidelines provisions on transparency aim at
  preventing discrimination of consultants, ensure
  fair access and verifiability. Study results show
  the Guidelines effective in this regard. But, is
  it enough?
• Transparency of the evaluation process itself is
  not mentioned in the GL but something needs to be
  said without affecting Borrowers ownership
• Transparency of the Guidelines and of the RFP is
  not considered
• Symmetry is missing!

17
Conclusions Warning Signs

• Guidelines are often imprecise and too
  complicated leading to rigid application and
  increasingly ineffective compliance
• Excessive emphasis on cost savings leads clients
  to disregard for quality and project risks
• Evaluation Committee often not up to its task
• Transparency is treated as an end not as a mean

18
Recommendations

• The study focuses on 8 Key Recommendations that
  should help Bank staff and clients improve
  application of the GL
• These could dealt without significant policy
  change by
• Correction Rephrasing for Quality
• Clarification for Transparency Integrity
• Simplification for Efficiency Economy
• They are currently under discussion within the
  Bank. Further discussions with the industry will
  also take place.

19
Key Recommendation 1

• Enhance Efficiency and Economy
• Combine compatible and staged assignments
  whenever justified by technical and
  administrative circumstances, or by incumbents
  good performance
• Unify Requests for EOI under a coherent one
  whenever possible
• Encourage use of available simple selection
  methods (CQS) and Simplified Technical Proposals
  (STP)
• Limit bureaucratic overload by inducing
  sincerity of consultants expressing interest or
  limit extent of EoI submissions, or limit
  scope/length of submissions
• More TTL and PS attention!

20
Key Recommendation 2

• Strengthen consideration for quality
• Rephrase the Guidelines to ensure of selection
  methods and assignment ToR need to be linked
• Describe more clearly principles and
  circumstances for use different selection methods
• CQS and SSS under the correct circumstances past
  is often easier to evaluate objectively than the
  future
• Above should increase quality of participation by
  improving the accurateness of the search mechanism

21
Key Recommendation 3

• Ensure balanced short list and link it to a
  suitable selection method
• The importance of the qualifications for the
  assignment should be strengthened
• State flexibly the short list requirement of six
  firms with a wide geographic spread (para 2.6)
  and
• Reformulate Para 2.8 to better emphasize the
  concept of mixed short lists.
• Above should improve competitive fairness and
  encourage qualified consultants to participate

22
Key Recommendation 4

• For Maximum Strength of Evaluation Committee,
  Guideline should briefly but firmly stress
• Need of a reliable EC capable of evaluating
  quality (Subjectivity is not a problem when
  evaluator is professional and impartial)
• EA Independence from principals in Evaluation
  Phase
• Well designed evaluation protocols
• Independent observers in risky cases (no voice or
  vote)
• Professional evaluation will improve quality and
  credibility of the selection but TTL assistance
  and training matter

23
Key Recommendation 5 Improve QCBS Scoring Formula

• The currently used QCBS formula can compromise
  an evaluation, particularly with high dispersion
  of prices
• New QCBS formula aSt 300 (1 a)ln(p)100
  eliminates the flaws of the currently used. It
  needs more testing.
• It is recommended that the current formula be
  used with prudence i.e. when short list is
  balanced and proposals comparable so that price
  constitutes an efficient, fair and transparent
  (revealing) evaluation criterion.

24
Key Recommendation 6 National Consultants

• Promote National Consultant Participation and
  Growth
• Bank can consider a) maintaining present
  preferential provisions, or b) promote more
  strongly sustainability of national consulting
  with support to SM Consulting Firms.
• Describe better circumstances for assignments
  limited to national competition, stressing
  freedom of including international consultants.
• All selection methods as a possibility CQS, QBS,
  FBS, QCBS and SSS justified by the
  characteristics of the TOR and of the short list

25
Key Recommendation 7 Transparency

• Objective Guidelines and RFP should be self
  revealing!
• Indicate that transparent evaluation procedures
  can improve credibility (quality/integrity) of
  outcome
• Recommend EC members based on professional
  criteria
• Recommend EC to follow protocols to manage the
  process, evaluate and score the proposals
  individually, discuss scoring and agree on award!
• Result Evaluations should reflect impartiality,
  intersubjectivity and agreement

26
Key Recommendation 8 Conclusions

• Asymmetries of information will be the
  characteristic of this industry. Risks poor
  evaluations and unethical decisions
• There is no golden rule but GL should focus on
  effectiveness under balanced consideration of
  prioritized principles. This cannot be solely
  achieved through strict compliance and sanctions
  but applying judgment
• Best practice approach A combination of
  compliance with results and capacity building
  of Borrowers and TTL may lead more mature
  relations between clients and consultants.

27
Conclusions Action Agenda

• Text improvements in the Guidelines sharpening
  key concepts, improving definitions, clarifying,
  shortening text to gain effectiveness. Deepen
  economic and institutional analysis
• Capacity building, best practice notes, on ToR
  preparation for Borrowers and Bank staff, on
  shortlisting evaluation protocols
• Food for Thought
• Texts Structure Improvements
• Part 1 on principles
• Part 2 on rules

28
Annex 1a Combinations that reach the same total
score (St and p) under current formula