MAESTRO Evaluation Showcase

1
What a Realization An Ex Post Impact
Evaluation of a Performance-Based
Program(2002/2003 SPC Evaluation)

• MAESTRO Evaluation Showcase
• July 26-27, 2006
• Project Manager Pierre Landry, SCELead
  Consultants Mike Rufo, Itron Keith Rothenburg,
  Southern Exposure Engineering KEMA Inc.

2
Overview of Presentation

• Evaluation Context and Background
• Approach
• Results and Key Findings
• Recommendations

3
Context Custom, Bidding, and SPC Type Programs

• History and types
• custom rebates, bidding, SPC
• Goals
• resource acquisition and market transformation
• Common features
• focus on custom efficiency measures in large CI
• encouragement of comprehensive projects
• inclusion of technical engineering review
• requirements for proof of project installation

4
Common Issues for Large CI Programs

• Uncertainty in savings estimates
• Risk of gaming and fraud
• Costs of measurement and verification
• Keeping application requirements manageable yet
  effective
• Distributing funds equitably
• Minimizing free riders/maximizing net impacts
• Supporting the efficiency services market

5
Background on 2002-2003 CA SPC

• 3 IOUs (PGE, SCE, SDGE)
• Targeted at Large CI
• Incentives by end use
• Lighting (0.05), HVACR (0.14), Process/Other
  (0.08)
• Calculated or Measured Savings Path
• Determined by utility
• 90 calculated
• All projects require Site Installation Report

6
Why An Ex Post Evaluation of Performance
Contract Program?

• PY2002 PY2003 first years SPC had independent
  ex post impact evaluation
• Originally (98/99), all SPC projects required
  in-program MV
• Initially, 2 years of MV, then 1 year
• MT policy environment in CA (1998-2000)
• Virtually no attention to ex post impact
  evaluation
• MV dropped as requirement in 2000
• Utilities retained right to require
• Most projects done under calculated savings path
• Shift back to resource acquisition focus with
  energy crisis and 2006-2008 program admin decision

7
PY2002 SPC Evaluation Objectives and Constraints

• Objectives
• Re-establish process for site-specific ex post
  impact evaluation
• Implement process for representative sample
• Estimate program realization rate
• Estimate net-to-gross
• Recommendations for improvement

• Constraints
• Total budget vs. program size, project
  complexity, of utilities
• Per site budgets
• Limited sub-metering
• Took 2 years for most of sample to install
  projects
• Limited pre-metering available
• Self report NTGR method

8
Approach

• Develop and implement sample design (ratio
  estimation)
• Obtain files savings documentation
• Review applications and prepare ex post analysis
  plans
• Conduct on-site data collection (limited
  metering)
• Develop ex post impact estimates
• Prepare detailed, site-specific documentation
• Carry out quality control review
• Extrapolate final ex post results to the
  population
• Estimate free ridership
• self-report method
• CADMAC protocol-level battery

9
Population

• Combined 2002-2003 SPC Population (apps)
• Combined 2002-2003 SPC Population (kWh)

10
Sample Design

• PY2002-PY2003 SPC Sample (apps)PY2002-PY2003
  SPC Sample (kWh)

11
Gross Realization Rate Results
12
Gross Realization Rate Results
13
Freeridership and Net-to-Gross
DEER SPC NTGR 0.7
14
Key Findings

• PY2002-2003 SPC more like a custom rebate than
  performance contract program
• Overall realization rate reasonably high but
  below one
• Wide range in individual realization rates
• Importance of influence of largest projects
• Gross savings may not be as conservative as SPC
  program managers intended
• NTGR slightly lower than 1998-2001 period

15
Other Savings-Related Key Findings

• Wide range in the quality of applications and
  supporting documentation
• Need for increased verification and documentation
  of assumptions in project files
• Experience and expertise levels of the reviewers
  vary widely
• Difficulties in assessing complex industrial
  process projects
• Limited estimation of kW peak demand savings

16
Impact-Related Recommendations

• Consider targeted increases in the level of
  technical documentation
• Consider a stronger application affidavit
  statement regarding savings assumptions
• Further standardize the review approach and
  documentation requirements for recurring complex
  projects
• Consider providing or requiring more technical
  support for applicants for complex projects

17
Savings-Related Recommendations (Cont.)

• Improve reviewer documentation
• Consider increasing
• conservatism for calculated path savings
  estimates
• measurement for large complex projects
• incentive premium for measured projects
• Increase pre-installation measurement for very
  large projects with highly uncertain baseline
  conditions
• Consider independent review of the SPC calculator

18
Free Ridership-Related Considerations

• Increase efforts to reduce free ridership
• Consider
• Higher incentive levels for higher payback
  measures or emerging technologies
• Incorporating a payback floor
• Bonus payments for first-time participants
• Custom baselines for process improvements
• Excluding projects that are obvious free riders

19
Evaluation-Related Recommendations

• Consider shifting to ex post impact evaluations
  from program-year to paid-year basis
• or a combination
• Expand scope of ex post measurement
• more measurement per site
• more sites in samples
• incorporate uncertainty in ex post estimates into
  program realization rate confidence interval
• Integrate evaluator early into program process to
  enable pre-measurement where necessary