Customer Expectations

1
Customer Expectations WTP For Improvements in
Service from Electricity DNOs

• Pilot Debrief
• February 2008

2
Agenda

• Fieldwork Methodology
• Interviewer Feedback
• Stated Preference Methodology
• Stated Preference Findings
• Summary of Recommended Changes

3
Fieldwork Methodology

• Objectives of pilot were to test
• the recruitment process
• the survey hit rate
• the clarity and flow of the questionnaire
• the appropriateness of the language used
• the accuracy of all routings
• ease of use of the show material
• design and understanding of the stated preference
  exercises
• the interview duration
• Approach
• Businesses 40 CATI phone-post/email-phone (41
  achieved)
• Residential 100 CAPI in-home interviews (99
  achieved)
• Quotas
• Business
• 10 EPN (11 achieved), 10 SPN (10 achieved), 20
  LPN (20 achieved)
• Mix of size (20 small 15 medium 6 large)
  sector
• Minimum 10 with outage experience (39 achieved)
• Residential
• 25 EPN (25 achieved), 25 SPN (25 achieved), 50
  LPN (49 achieved)

4
General Feedback Interview Duration
DOMESTIC Mean 32 minutes
BUSINESS Mean 43 minutes
5
General Feedback Comments on Domestic
Questionnaire (non SP)

• Problems with clarity flow
• Question 34 did you find each of the levels of
  service described easy to follow? if you said
  no, Question 35 then said which levels werent
  clear to you? seem like separate issues
• Need showcard for income question
• Problems with routing
• Q22b if they say dont know should skip
  subsequent question

• General comments on what would make it easier
• Concern that people will think we are trying to
  sell them services for an electricity company.
  Also, many people less aware of Ofgem than of
  Ofwat or the FSA, so they were very cautious.
  Maybe a more elaborate explanation of the survey
  might help.
• Interview duration
• Longer than budgeted for in most instances
  cannot be done within current fee (due both to
  time required for each and impact of length on
  hit rates) and current incentive (due to length)
• Other
• Some issues with defining rural versus urban we
  will use a postcode dbase to do so in the
  mainstage
• Add all screening questions (eg age, SEG) to main
  questionnaire
• Add question about number of children 5

6
General Feedback Comments on Business
Questionnaire (non SP)

• Problems with clarity flow
• Some people assumed it was a sales call despite
  the current intro. Need to further emphasize that
  Ofgem are a regulatory body and that the research
  is looking at improvements that could be made to
  the services provided to their company
• RQ13 need a dont know code
• Q23 some said both add this code?
• Q26 needs a dont know or cant remember
• Q40 needs re-phrasing so it doesnt suggest
  anything will be sent needs to separate leaflet
  and email needs to be multicode
• Problems with routing
• none

• Interview duration
• Longer than budgeted for in most instances
  cannot be done within current fee (due both to
  time required for each and impact of length on
  hit rates)
• Other
• Some issues with defining rural versus urban
  again, we will use a postcode dbase to do so in
  the mainstage

7
Stated Preference Methodology

• Objective to obtain estimates of customers
  willingness-to-pay for improved service levels
  (or willingness-to-accept for reduced service
  levels)
• No revealed preference evidence, so need to rely
  on stated preferences
• Customers are not offered different service
  levels and prices in real world
• Two possible stated preference approaches
• Contingent Valuation questions (CV) ask
  respondents directly questions about valuations
• Discrete choice experiments (SP DCE) elicit
  valuations through tradeoffs
• Based on hypothetical choices
• Essential that price is one of the attributes

8
Example Choice Experiment 1 (Domestic)
9
CV vs SP DCE
10
Attributes Tested for Domestic Customers
Cost also included in all experiments
11
Attributes Tested for Business Customers
Cost also included in all experiments
12
Stated Preference Methodology

• Potential problem with separation of attributes
  across experiments is that results may
  over-estimate total wtp
• Therefore we also include a higher-level
  experiment, including all attributes to test
  total wtp
• We also include a series of CV questions to
  elicit total wtp for best service
• The higher-level experiment is simplified so that
  group of attributes are varied simultaneously
• Two levels tested
• Lowest service level
• Highest service level
• Still complicated

13
Example higher-level choice 6 choices
14
Stated Preference Methodology Analysis

• Analysis based on principles of utility
  maximisation
• Utility assigned to each alternative, made up
  of
• Attributes of the alternative
• Characteristics of the respondent, e.g. income,
  etc.
• Assumption that error distribution is extreme
  value (iid) leads to logit model
• Maximum likelihood techniques used to obtain
  values of unknown parameters
• Find the ßs that optimise explanation of the
  model
• Model outputs
• Coefficient estimates (and t-statistics)
• Model fit statistics

15
Utility Specification
16
Logit Model Formulation

• Assumption that random terms (?) are independent
  and identically distributed across alternatives
  (IID) and that they vary with an Extreme-Value
  distribution

17
Stated Preference Methodology Analysis

• Coefficients used to
• Determine the relative importance of attributes
• Determine the monetary value for attributes
• Specify utility functions for prediction models
• Monetary values for attributes identified in
  lower-level games
• May be adjusted by values for groups of
  attributes identified by higher-level game or by
  CV question
• This is a judgement call

18
Stated Preference Did it Work?

• Trading behaviour
• Background Questions on respondent understanding
• Model Results

19
Most domestic respondents were willing to choose
new service options in at least one of the choices

• Domestic London
• Domestic Other DNOs

20
Most business respondents were willing to choose
new service options in at least one of the choices

• Business London
• Business Other DNOs

21
Examining choices at different cost levels
implies that many domestic customers were willing
to pay 15 increases in DUOS for service
improvements
Other DNOs
Experiment 1
Experiment 2
Experiment 3
22
A wider range of costs was investigated in the
higher-level experiment with large numbers of
domestic users choosing alternatives with large
cost increases
23
Examining choices at different cost levels
implies that many business customers were willing
to pay 15 increases in DUOS for service
improvements
Other DNOs
Experiment 1
Experiment 2
Experiment 3
24
A wider range of costs was investigated in the
higher-level experiment with large numbers of
business users choosing alternatives with large
cost increases
25
Most respondents understood the experiments

• Were you able to make the comparisons in the
  choices we presented to you
• Domestic
• 48 of 49 respondent in London reported they were
  able to make the comparisons
• 47 out of 50 respondents in other DNOs reported
  they were able to make the comparisons
• Business
• 19 of 20 respondents in London reported they were
  able to make the comparisons
• 20 of 21 respondents in other DNOs reported they
  were able to make the comparisons

26
A sizeable proportion of domestic customers felt
that the levels of service in the choices were
unrealistic, and the experiments were difficult

• Were the levels of service we have been asking
  about in the choices realistic?
• London, 11 of 49 domestic respondents felt the
  service levels were not realistic or too
  difficult
• A 1 in 10 year electricity cut is not realistic
  because we are having electricity cut 6 times in
  a year
• I dont think all the changes will be possible
  within the costs stated
• It is seriously insane to raise the bills again
  and again
• No electricity company would cut emissions, as
  they are not efficient enough to replace the
  current system
• There were too many options and everything was
  too similar
• I dont believe the whole concept, its not clear
  enough
• They are all the same
• Too much complicated
• Other, 8 of 50 domestic respondents felt the
  service levels were not realistic or too
  difficult
• Because they are not achievable, ie the cost of
  putting all these cables underground isnt going
  to be economically viable
• The last thing I want is for my bill to go up
• They are talking about increasing expenditure by
  certain percentages, but the original expenditure
  is not listed so 15 increase of what?
• The comparisons are not obvious, got to really
  read in order to give an answer
• The different choices or packages didnt always
  seem logical.
• Too much to absorb all at one for the average
  consumer who is having enough problems paying
  their bills

27
Most business respondents felt that the levels of
service in the choices were realistic

• Were the levels of service we have been asking
  about in the choices realistic?
• London, 3 of 20 respondents felt the service
  levels were not realistic
• 7 power cuts in 10 years is outrageous
• Because we had 3 power cuts in the last 2 years
  of long duration
• They do not employ enough people, so
  realistically it is not going to happen
• Other, 4 of 21 respondents felt the service
  levels were not realistic
• Because some reductions are not able to be
  guaranteed. There are elements outside your
  control, i.e. the power cut we experienced was
  due to accidental cutting through power cables.
• Some options offered more for less.doesnt make
  sense.
• Typical of todays climate, people want all these
  things, but are not prepared to pay for them.
  Realistically, these improvements would probably
  cost too much.
• Some of the improvements would be very difficult
  to achieve.

28
Model results are encouraging

• Domestic Customer Models
• Results are very encouraging
• Cost coefficient is significantly negative
• Most other coefficients have intuitive signs,
  many are significant
• Resulting wtp values look reasonable
• Business Customer Models
• Many statistically insignificant terms, probably
  a result of small sample sizes and heterogeneity
  of sample, e.g. in terms of size of business
• Expect model to be improved with larger sample
  size in main survey
• Most important variables
• Power cuts
• Restoration of supply
• Reduction of carbon emissions

29
Interviewer Feedback on SP

• Very complex
• Too much to consider
• Consider reducing and rotating over 2 interviews
  to make each survey shorter
• Maybe cut down on the sets of choices if
  possible, 6 instead of 8. Choice sets are quite
  long
• Most respondents found the exercise difficult or
  complex at first but then adapted to it. It is a
  bit difficult to keep them interested in the
  survey till the end
• The interview should be shortened if possible
• Its a lot of reading

30
Conclusions

• Subject area is abstract and the choices are
  complex, but despite some concerns on the part
  of interviewers respondents seem to have been
  able to undertake the SP experiments
• Some, mostly minor, revisions required for main
  survey
• Main area for review is cost levels
• Mainstage interviews
• 2,100 domestic face-to-face interviews (150
  interviews conducted in each of the 14 DNO
  territories)
• 1050 CATI business interviews (75 per DNO
  territory)

31
Recommended Changes

• Review cost levels
• Pilot cost levels
• Fewer reductions and more increases for
  Experiments 2 and 3, e.g. 2 reductions, 6
  increases and base
• Recommend that we incorporate larger cost
  increases
• Review wording of attributes and levels to
  simplify as much as is possible (may not be much
  scope for this)
• Include better than now and worse than now
  descriptors, where appropriate
• Review Network resilience to flooding
  definition
• Complaint that it was difficult to make
  comparisons when base level is not defined
• Reduce number of choices in lower-level
  experiments from 8 per experiment to 6 per
  experiment

32
Other Recommendations

• Minor Text Changes
• Compensation for Multiple Interruptions minor
  error in wording which will entitles you, plus
  (x) in text to be removed
• Network Resilience to Major Storms include a ,
  separator in the numbers, e.g. 180,000 rather
  than 180000
• Minor Changes to Questions/Routing
• Slightly reword Q34 and/or Q35 (domestic) / Q46
  and/or Q47 (business)
• Amend routing for Q22(domestic) / Q33 (business)
  dont knows
• Use postcode dbase to define rural urban
• Add age SEG data from screener to main
  questionnaire (domestic only)
• Add question about number of children 5
  (domestic only)
• RQ13 (business) need a dont know code
• Q18 (domestic)/Q23 (business) some said both
  add this code?
• Q20 (domestic)/Q26 (business) needs a dont know
  or cant remember
• Q40 (business only) needs re-phrasing so it
  doesnt suggest anything will be sent needs to
  separate leaflet and email needs to be multicode
• Expand explanation of survey of Ofgem
• Reduce the length of the questionnaire or
  increase fees incentives