Lessons from the evaluation of OPTA

1

• Lessons from the evaluation of OPTA
• Results, methodologies and challenges
• Martijn Poel (TNO) Marieke de Wal
  (Berenschot)
• IDATE seminar, Reviewing the Review, 14 November
  2006

2
Reviewing the Regulator

• Context of the OPTA evaluation
• Objectives
• Methodology
• Summary of results
• Research challenges in the evaluation of NRAs

3
Context of the OPTA evaluation

• Netherlands strong evaluation culture, focus on
  governance and learning, with a trend towards
  Cost Benefit Analysis and Impact Assessment
• OPTA is to be evaluated every four years
• 2001 OPTA scored well, partly due to responses
  and interests of entrants do not rely on ex
  post regulation and competition authority
• New Telecommunications Law in 2004 market
  analysis approach
• Second evaluation in 2004/2005 Berenschot,
  Ecorys and TNO
• Assignor Ministry of Economic Affairs, to which
  OPTA is accountable

4
Objectives

• Most relevant are the OPTA Law (1997) and Telecom
  Law (1997)
• The main objectives infrastructure and services
  competition are linked to their relevance for
  first class infrastructures and services in the
  Netherlands, quality, choice and societal
  interests
• Objective of the OPTA evaluation
• Analyse how OPTA has fulfilled its tasks and how
  it has used its instruments, in terms of
  efficieny and effectiveness
• Assess internal organisation and management
• Assess collaboration with stakehoders
• Address recommendations of 2001 evaluation, e.g.
  transparancy and allocation of costs,
  collaboration with Ministry, quality of operations

5
Approach and methodologies
Six case studies Desk research (facts) Stakeholder
interviews National experts International
experts International benchmark
Regulatory interventions
Local Loop Unbundling
Number portability
Interconnection tariffs
Number issueing
End user tariffs
Quality of postal serv
6
Approach and methodologies
Six case studies Desk research (facts) Stakeholder
interviews National experts International
experts International benchmark Four topics Desk
research (facts) Interviews National
benchmark International benchmark
Regulatory interventions
Local Loop Unbundling
Number portability
Interconnection tariffs
Number issueing
End user tariffs
Quality of postal serv
Organisation and management
Operations
Allocation of costs
Quality and service
Efficiency input/output
7
Approach and methodologies
Six case studies Desk research (facts) Stakeholder
interviews National experts International
experts International benchmark Four topics Desk
research (facts) Interviews National
benchmark International benchmark Overview and
examples Desk research (facts) Interviews
Regulatory interventions
Local Loop Unbundling
Number portability
Interconnection tariffs
Number issueing
End user tariffs
Quality of postal serv
Organisation and management
Operations
Allocation of costs
Quality and service
Efficiency input/output
Collaboration with...
Ministry
Sector
Frequency regulator
Competition authority
Privacy regulator
Media regulator
IRG and ERG
EC
8
Methodology regulatory interventions (1)

• Eight criteria, based on Better Regulation and
  Good Governance documents, e.g. UK, Netherlands,
  OECD, Telecom Directives
• Send and explained to all interviewees
• Scored insufficient - poor - sufficient - good -
  very good

Responsiveness
Reasonable
Transparancy
Proportionality
Consistency and predictability
Lawfulness
Prudence and timeliness
Impact
9
Methodology regulatory interventions (2)

• First order impact telecom and postal markets
  rather than ICT sector, economy and society
• Quick scan of impact, based on interviews,
  statistics and national experts
• Why?
• No serious concerns on costs vs. benefits, e.g.
  EC statistics and recent examples of CBA by Oxera
  (including compliance costs)
• No intention to merge OPTA and competition
  authority, e.g. use CBA to analyse added value of
  sector specific regulator
• Workload for evaluators and OPTA (in four months)

10
Methodology organisation and management

• Allocation of costs includes issue of direct and
  indirect costs, according to OPTAs and other
  methods
• Efficiency output indicators?
• Quality of operations includes financial systems,
  cost control, Human Resource Management,
  information systems
• Quality of services includes complaints
  procedures and systems for quality control and
  improvement
• National benchmark of regulators based on desk
  research and recent evaluations, e.g. costs per
  FTE, systems used
• International benchmark for some indications on
  FTEs and costs

11
Methodology collaboration

• With sector part of the six case studies on
  regulatory interventions
• With policy makers and regulators
• Desk research on the formal protocols and
  statements on division of tasks, expertise,
  communication, information sharing, consultation,
  collaboration, approval
• Desk research and interviews with examples from
  cases with and without tensions and conflicts

12
Summary of results regulatory interventions

• Overall between sufficient and good
• Lawfulness sufficient, due to several cases lost
  (interc. tariffs, bitstream, number portability)
  but taking into account the flaws of the old law
• Transparancy sufficient, with criticism on use
  of stakeholder input, arguments for specific
  instruments (e.g. penalties) and going
  bilateral
• Consistency and predictability good (and
  responsive?)
• Impact appears to be above EU average, with
  attribution issues such as role of cable
  networks, small and flat country, international
  economy
• The new law allows for improvement of
  transparancy and lawfulness (timeliness?)
• OPTA and the ministry continue to improve
  transparancy, e.g. policy guidelines on penalties

13
Summary of results organisation and management

• Overall sufficient, which is a light improvement
• Efficiency difficult to measure due to lack of
  indicators for output
• Indirect costs are above 50, e.g. communication,
  strategy, support
• No target costs for specific products/output,
  with soms exceptions
• No system for quality control and improvement
  (light svp)
• FTEs and costs are within bandwidth of
  benchmark(s)
• Following the evaluation OPTA has
  intensified work on output indicators, cost
  allocation and cost reduction

14
Summary of results collaboration

• Collaboration with ministry has improved
  sufficient
• With others good
• With ministry build on clear roles,
  communication (bilateral and in public) and
  add trust and collaboration
• Build on good relation with competition
  authority, e.g. exchange of staff, joint or
  similar HRM, shared services
• With sector increase transparancy but also
  improve process skills related to agenda
  and priority setting, consensus building,
  mutual respect and prevention of formal cases

15
Research challenges in the evaluation of NRAs

• Benefit from ICT monitoring and Impact Assessment
  tools
• Process and output indicators, e.g.
  responsiveness and decisions
• Address compliance costs, uncertainy and risk
• Tackle attribution and additionality issues, e.g.
  national context, mergers
• Acknowledge the policy mix regulation and
  regulators are part of a set of policy
  instruments and public actors
• Better benchmarking, e.g. tasks and context of
  NRAs (market, institutional)
• Evaluation is learning regulatory monitor
• Expert judgements (scores) and stakeholder
  participation (insight)
• A tool box for the evaluation of
  NRAs?