Jackie Berger

1
National WAP EvaluationField Process Study

• Jackie Berger
• APPRISE

2
Presentation Overview

• Process Field Study Design
• Challenges and Resolution
• Overview of Findings
• Audits
• Measure Installation
• Final Inspections
• Opportunities - Training, Equipment, and
  Management

3
Process field study design
4
Process Field Study Focus

• Audit approach and implementation
• Energy education delivery
• Measure installation/operation of crews
• Post-weatherization quality assurance inspection
  approach and implementation
• Client interaction
• Interactions among agency staff
• Implications for management, procedures, and
  training

5
Process Field Study Approach

• Go beyond anecdotes
• Quantify findings across all visits
• Develop check lists and rating scales
• Train weatherization experts to implement
  consistently
• Enrich data with descriptive information
• Examples used to illustrate and clarify
• Not used to draw conclusions
• Make recommendations for program improvement
  based on prevalent issues

6
Observation Team

• Social Scientists
• One week observation at agency
• Weatherization manager and staff interviews
• Observations client intake, audit, measure
  installation, final inspection
• Written agency summary report
• Weatherization Experts
• 3-Week Observations at Agency
• Observations audit, measure installation, final
  inspection
• Post Observation Data Entry
• Training
• Monthly update calls
• Final debriefing

7
Challenges and resolution
8
Study Challenges (1)

• Program is different in every state and even
  within states
• Approximately 1,000 local agencies
• Varying guidelines and standards
• Different level of access to training and
  equipment
• On what standards should the jobs be assessed?
• Decision Weatherization best practices as agreed
  on by a national team of weatherization experts
• Assessment of where there are opportunities
  standards, equipment, training

9
Study Challenges (2)

• How to pick 20 agencies to represent the program

10
Agency Selection

• WAP Agencies
• Almost 1,000 deliver services across the country
• 400 comprised agency billing data sample
• Field Process Study 20 agencies in 20 different
  states
• Selection based on
• Number of states per region based on WAP funding
• State selection based on part of region and
  program size
• Agencies selected to represent

Urban/Rural Education Based on agency survey
PY 2008 planned jobs Training Based on agency survey
Measure selection Quality Control Based on agency survey
11
Study Challenges (3)

• How to pick staff/jobs to observe at each agency
• Best foot forward approach
• Agency managers to select staff/jobs
• Where are there opportunities for improvement for
  even the best teams at each agency

12
Study Challenges (4)

• Obtaining consensus among weatherization experts
• Number of issues, tests, measures to consider
• Where to start
• .and where to stop? (still trying to figure out)

13
Field StudyFlow Chart
14
Audit Observations

• Check Lists
• Audit preparation
• Audit introduction
• Audit home walk through
• Heating assessment
• Air conditioning assessment
• Ventilation assessment
• Water heater assessment
• Air leakage and insulation diagnostics
• Diagnostic testing
• Combustion safety testing
• Refrigerator assessment
• Lighting assessment
• Exit interview
• Should have been performed, was performed,
  performed correctly
• Described on write-up, recorded thoroughly and
  accurately

• Ratings
• Audit introduction
• Auditor attempt to engage client
• Client engagement in audit
• Auditor communication skills
• Quantity of education
• Quality of education
• Exit interview
• Completeness of inspection
• Completeness of testing
• Quality of testing
• Auditor efficiency
• Auditor professionalism
• Auditor respect of clients home
• Auditor safety practices
• Audit comprehensiveness
• Planned work comprehensiveness
• Audit write up
• Excellent, good, poor

15
Measure Installation Observations

• Check Lists
• Measure installation preparation
• Client interaction
• Introduction
• Air sealing procedures
• Attic insulation
• Thermostat installation
• CFLs

• Ratings
• Windows sealed, interior storm, exterior storm,
  sash kit, repair
• Doors weather stripping, treated, replacement,
  installation
• Air sealing (8 different locations rated)
• Overall air sealing quality, cleanliness
• Attic insulation quality, cleanliness
• Attic hatch work quality
• Walk-up attic work quality, cleanliness
• Wall insulation quality, cleanliness, siding
  preservation/interior wall finishing
• Basement insulation quality, cleanliness
• Crawl space - vapor barrier insulation, perimeter
  wall insulation, vents sealing, ceiling
  insulation and air barrier installation
• Ventilation existing fans ducted to outside,
  whole house, attic, kitchen, bathroom

16
Measure Installation Observations

• Ratings (continued)
• Heating system tune-up, repair, replacement,
  installation, vent correction
• Thermostat installation
• Air conditioning sealing around, tune-up,
  replacement
• Evaporative cooler repair, replacement
• Duct sealing, duct insulation, duct improvement
  cleanliness
• Hot water temperature adjustment, heater wrapped,
  pipes insulated, heater repaired, replaced,
  heater venting work
• Subcontractor treatment of refrigerator and home
• Dryer venting
• Low cost AC cooling coils cleaned, HVAC filters
  cleaned or replaced, faucet aerators,
  showerheads, CO detector, smoke detector, fire
  extinguisher
• Crew efficiency, professionalism, cleanliness,
  safety practices
• Communication, attempt to engage client, client
  engagement, education quantity, education quality

17
Final Inspection Observations

• Check Lists
• Home walkthrough
• Diagnostic tests
• Combustion safety tests
• Draft tests
• Occupant interaction
• Should have been performed, was performed,
  performed correctly

• Ratings
• Inspection completeness
• Testing completeness
• Testing quality
• Inspector attempt to engage client
• Client engagement
• Inspector communication skills
• Education quantity
• Education quality
• Inspector efficiency
• Inspector professionalism
• Inspector cleanliness
• Inspector safety practices

18
Study Challenges (5)

• Consistent implementation of assessment tool
• Designed to maximize consistency
• Training of weatherization experts
• Extensive review of data entry

19
Check List ExampleAudit Air Leaking
andInsulation Diagnostics
Applicable Observations Action Taken Action Taken
Applicable Observations
Measured surfaces 100 94 94
Inspected windows 100 96 96
Inspected all accessible attics 78 69 88
Measured insulation in all accessible attics 79 70 89
Created access to inaccessible attics 33 3 10
Measured insulation in exterior walls 95 47 49
Measured insulation in basement/crawlspace 74 55 74
Inspected for all typical bypasses 100 62 62
Visual inspection for air sealing opportunities 100 83 83
Used blower door while inspecting for leaks 96 64 67
20
Rating Design
Rating 1 2 3 4 5
of Points needed 0-49 50-74 75-89 90-99 100
Total points Number of Points needed for each rating Number of Points needed for each rating Number of Points needed for each rating Number of Points needed for each rating Number of Points needed for each rating
4 0-1 2 3 -- 4
5 0-2 3 4 -- 5
6 0-2 3-4 -- 5 6
7 0-3 4 5 6 7
8 0-3 4-5 6 7 8
9 0-4 5-6 7 8 9
10 0-4 5-7 8 9 10
11 0-5 6-7 8-9 10 11
12 0-5 6-8 9-10 11 12
13 0-6 7-9 10-11 12 13
14 0-6 7-9 10-12 13 14
15 0-7 8-10 11-13 14 15
21
Rating ExampleAttic Insulation Quality
Quality of Attic Insulation Quality of Attic Insulation
1 All air sealing work completed first
2 Exhaust fans vented to exterior as needed
3 Heat producing devices or systems protected from insulation contact
4 Attic checked for knob and tube wiring
5 Workers wore respirators, safety glasses, gloves, and hard hats while insulating attic
6 Insulation installed in sufficient quantity (bags per ft2) to meet R-value requirement
7 Proper insulation material chosen for attic conditions
8 Open blow insulation is level and of consistent depth
9 Attic ventilation maintained
10 Confined areas blown to dense pack
11 Proper containment used to protect client and belongings
Needs Improvement Needs Improvement Excellent Excellent Mean Rating Rated
Rating 1 2 3 4 5 Mean Rating Rated
Total Points needed 0-5 6-7 8-9 10 11 Mean Rating Rated
Bold Points needed 0 0 0 6 6 Mean Rating Rated
Attic Insulation Quality 4 18 42 19 18 3.3 57
22
Study Challenges (6)

• Remaining observers, rather than trainers,
  coaches, mentors
• Weatherization experts are used to coaching
• Told only to intervene in case of serious health
  and safety issue
• Weatherization staff expect to receive coaching
• Asked weatherization experts for feedback and
  advice

23
Study Challenges (7)

• Connection between work quality and impacts
• Not enough data to assess
• Assumption is if
• Conduct comprehensive and accurate testing
• Correctly diagnose problems in the home
• Effectively engage clients about energy use
  practices and potential changes
• Accurately transfer audit data and information to
  installers
• Effectively implement weatherization measures
• Inspect for quality and comprehensiveness
• Then results will be meet or exceed expectations

24
Process field study audit observation findings
25
Process Field StudyObservations Completed
Number of Observations Number of Observations Number of Observations
Social Scientist Weatherization Expert Total
Audit 43 112 155
Measure Installation 45 114 159
Final Inspection 37 91 128
Total 125 317 442
26
Audit Strengths

• Inspection 99 inspected every accessible room
• Heating system 94 inspected
• Filters 72 inspected
• Ventilation
• 72 inspected the kitchen
• 78 inspected the bathroom ventilation.
• Water heater 93 inspected
• Air conditioning 82 inspected outside unit.
• Insulation 89 measured all accessible attics.
• Testing 97 conducted blower door test (84
  correctly)

27
Audit Opportunities for Improvement

• Bathroom ventilation flow 24 assessed
• Hot water 39 checked temperature at the faucet
• Shower flow none of the auditors measured flow
• Insulation 49 measured in exterior walls.
• Blower door 67 used while inspecting for leaks
• Zonal pressure diagnostics 42 done (87
  correctly)
• IR camera 49 used (70 correctly)

28
Communication of Audit Findings

• Audit write-ups do not convey educational
  opportunities
• Work order not always clear or specific enough
  for installer to understand the desired outcome
• Tools that improved information transfer
• Pre-work walk through with installer
• Attaching photos of critical areas to the work
  order
• In process check-ups with the installers
• Air sealing is a particular area for improved
  communication
• Need for flexibility to change work order when
  problems are found during installations

29
Audit Findings

• Opportunities
• Utilize energy bills
• Increase client engagement
• Identify client-specific opportunities for
  reducing energy use
• Auditor understanding of pressure boundaries
• Auditor understanding of testing purpose and
  procedures
• Conduct worst case draft testing

• Strengths
• Respect and concern for clients
• Explanation of WAP program and process
• Use of data collection forms
• Auditors meet with contractors to explain audit
  findings and work scope

30
Process field study Measure Installation
Observation findings
31
Crews and Contractors Observed
Number Percent
Crews 35 31
Contractors 74 65
Both 5 4
TOTAL 114 100
32
Measure Installation Strengths

• Air sealing prioritized the right areas
• 77 percent prioritized sealing at the top and
  bottom of the envelope
• Attic insulation followed best practices
• 77 percent completed attic floor sealing prior to
  insulation
• 88 percent had no gaps or voids
• 95 percent used appropriate materials.

33
Measure InstallationMid-High Ratings
Area Ratings
Window work 3.5 4.0
Door work 2.8 3.5
Attic insulation 4.0
Wall insulation 2.9
Basement insulation 3.6
Crawl space work 3.1 4.0
Ventilation work 4.1 4.8
Thermostat installation 4.1
Air conditioning work 3.0 4.0
Water heater work 3.1 4.0
34
Measure Installation Opportunities for Improvement

• Air sealing diagnostics
• 22 percent used the blower door to guide air
  sealing
• 11 percent used the zonal pressure test to affirm
  appropriate pressure boundaries.
• Air sealing
• 57 percent sealed all major opportunities.

Low Ratings
Area Ratings
Air sealing 2.4
Heating system work 2.0 2.5
Smoke detector installation 2.6
CO detector installation 2.0
35
Measure Installation Findings

• Opportunities
• Increased use of blower door when air sealing
• Respect for clients homes (booties, covering
  furniture)
• Crew member safety
• Increased assessment of HVAC contractors
• Explain CFLs when installing
• Client education

• Strengths
• Crew flexibility and dedication (above and beyond
  work scope)
• Crew problem solving
• Some very high quality work
• Experienced HVAC contractors

36
Process field study final inspection Observation
findings
37
Final Inspection Strengths

• The inspectors discussed the work that was
  performed
• 75 percent discussed the work that was performed
  with the client.
• Inspections were complete
• 90 percent examined all accessible rooms.
• Blower door testing was usually conducted
• 85 percent conducted this test (85 percent of
  those performed correctly)

38
Final Inspection Opportunities for Improvement

• Zonal pressure test
• 33 percent performed the test (87 percent done
  correctly)
• IR camera
• 44 percent used the camera (95 percent done
  correctly)
• Combustion safety
• 31 percent performed this test on the gas stove
  (100 percent done correctly)
• Worst case draft test
• 60 percent performed this test on the heating
  system (92 percent correctly)

39
Final Inspection Findings

• Opportunities
• Increased client education
• Explain measures installed
• Reinforce client action plan
• Improved testing quality
• Increased assessment of installation quality
• Reduced use of auditor to conduct final
  inspection additional perspective

• Strengths
• Good job when following work order closely
• Time saved when contractor attends
• Information to clients who to call if they have
  problems
• Referrals to additional assistance programs

40
Summary
41
Average RatingsBy Agency and Overall
42
Training Needs

• Energy bill utilization to assess needs and
  educate clients
• Building science fundamentals
• Critical thinking assessing unique situations
• Combustion safety testing
• Zonal pressure testing
• IR camera use
• Ventilation assessment
• Safe work practices worker safety and lead safe
  work
• Interviewing skills how to understand the
  clients needs
• Client education

43
Equipment Needs

• GPS
• IR camera
• Boroscope and fiber optic scope and video
• Personal safety equipment

44
Management Opportunities

• Standards and procedures
• Policy manuals
• Forms and checklists