Predictors of customer perceived software quality

1
Predictors of customer perceived software quality

• Paul Luo Li (ISRI CMU)
• Audris Mockus (Avaya Research)
• Ping Zhang (Avaya Research)

2
Need to View Quality from the Customers
Perspective
We translate these advanced technologies into
value for our customers -IBM (9 on the
Fortune 500)
Our strategy is to offer products, services and
solutions that are high tech, low cost and
deliver the best customer experience. -HP (11
on the Fortune 500)
We deliver unparalleled value to our customers.
Only by serving our customers well do we justify
our existence as a business -Avaya (401 on the
Fortune 500)
3
What Would be Ideal

• Predict customer perceived quality
• Using customer characteristics
• For each customer

Key idea Focus on the customer
4
Possible Applications of Predictions

• How do I plan deployment to meet the quality
  expectations of the customer?
• How do I target improvement efforts?
• How do I allocate the right resources to deal
  with customer problems

• Predict customer experience for each customer
• Identify possible causes of problems
• Predict customer interactions

5
Solutions for Software Producers

• How do I plan deployment to meet the quality
  expectations of the customer?
• How do I target improvement efforts?
• How do I allocate the right resources to deal
  with customer problems

• Predict customer experience for each customer
• Identify possible causes of problems
• Predict customer interactions

6
To Improve Customer Perceived Quality

• How do I plan deployment to meet the quality
  expectations of the customer?
• How do I target improvement efforts?
• How do I allocate the right resources to deal
  with customer problems

• Predict customer experience for each customer
• Identify possible causes of problems
• Predict customer interactions

7
Gaps in Current Research

• Prior work examined
• Software defect prediction for a single customer
  (Musa et al. 1987, Lyu et al. 1996)
• Software defect prediction for modules or
  features (Jones et al. 1999, Khoshgoftaar et al.
  1996)

Is not scalable
8
Not Focused on Customers

• Prior work examined
• Software defect prediction for a single customer
  (Musa et al. 1987, Lyu et al. 1996)
• Software defect prediction for modules or
  features (Jones et al. 1999, Khoshgoftaar et al.
  1996)

Tell us nothing about a specific customer
9
Does not Capture other Aspects of Customer
Perceived Quality

• Prior work examined
• Software defect prediction for a single customer
  (Musa et al. 1987, Lyu et al. 1996)
• Software defect prediction for modules or
  features (Jones et al. 1999, Khoshgoftaar et al.
  1996)

Does not predict other aspects of customer
perceived quality that are not code related.
10
Research Contributions

• Predict software defects for each customer in a
  cost effective manner
• Predict other aspects of customer perceived
  quality for each customer
• Empirically validate deployment, usage, software,
  and hardware predictors

11
Rest of This Talk

• The setting
• Customer interactions (outputs)
• Customer characteristics (inputs)
• Results
• Conclusion

12
Empirical Results from a Real World Software
System

• Avaya telephone call processing software system
• 7 million lines of C/C
• Fixed release schedule
• Process improvement efforts
• Tens of thousands of customers
• 90 of Fortune 500 companies use it
• Professional support organization

13
Data Used are Commonly Available

• Customer issue tracking system
• Trouble ticket database
• The equipment database
• Change management
• Sablime database

Data sources available at other organizations
e.g. IBM and HP
Data collected as a part of everyday operations
14
At Other Organizations

• Customer issue tracking system
• Trouble ticket database
• The equipment database
• Change management
• Sablime database

Data sources available at other organizations
e.g. IBM and HP
Data collected as a part of everyday operations
15
Customer Interactions (Outputs)

• We assume customer interaction customer
  perceived quality
• Five customer interaction (Chulani et al. 2001,
  Buckley and Chillarege 1995) within 3 month of
  deployment
• Software defects high impact problem
• System outages high impact problem
• Technician dispatches
• Calls
• Automated alarms

Important for Avaya and likely for other
organizations as well
16
Examine Customer Installations
Number of deployments
5
1
Months after general availability
17
Capture Characteristics of Each Installation
Customer 1 Deployed first month, a Large system,
Linux Customer 2 Deployed first month, a Small
system, Windows Customer 3 Deployed first
month, a Large system, Proprietary Os Customer
4 Deployed first month, a Small system,
Linux Customer 5 Deployed first month, a Large
system, Linux
Number of deployments
5
1
Months after general availability
18
Analyze Using Statistical Analysis
Customer 1 Deployed first month, a Large system,
Linux Customer 2 Deployed first month, a Small
system, Windows Customer 3 Deployed first
month, a Large system, Proprietary Os Customer
4 Deployed first month, a Small system,
Linux Customer 5 Deployed first month, a Large
system, Linux
Number of deployments
5
Similarities
Differences
1
Months after general availability
19
Category of Predictors (Kinds of Inputs)

• We examine
• Deployment issues
• Usage patterns
• Software platform
• Hardware configurations
• Prior work examines
• Software product
• Development process

Common sense issues, but lack empirical validation
20
Category of Predictors (Kinds of Inputs)

• We examine
• Deployment issues
• Usage patterns
• Software platform
• Hardware configurations
• Prior work examines
• Software product
• Development process

Key idea From the customers perspective, they
are not good predictors (i.e. do not vary for a
single release)
21
Specific Predictors (Inputs)

• Total deployment time
• deployment issues
• Operating system
• software platform, hardware configurations
• System size
• hardware configurations, software platform, usage
  patterns
• Ports
• usage pattern, hardware configurations
• Software upgrades
• deployment issue

22
Recap

• Predict for each customer (outputs)
• Software defects
• System outages
• Technician dispatches
• Calls
• Automated alarms
• Using Logistic regression and Linear regression

• Using predictors (inputs)
• Total deployment time
• Operating system
• System size
• Ports
• Software upgrades
• For a real world software system

23
Example Field Defect Predictions
24
Predictors
25
Nuisance Variables
26
All Predictors are Important
27
The Most Important Predictor

• Total deployment time (deployment issue)
• Systems deployed half way into our observational
  period are 13 to 25 times less likely to
  experience a software defect

28
May Enable Deployment Adjustments

• Total deployment time (deployment issue)
• Systems deployed half way into our observational
  period are 13 to 25 times less likely to
  experience a software defect
• May be due to software patching, better tools,
  more experienced technicians

29
Another Important Predictor

• Total deployment time (deployment issue)
• Operating system (software platform, hardware
  configurations)
• Systems running on the proprietary OS are 3 times
  less likely to experience a software defect
  compared with systems on running the open OS
  (Linux)
• Systems running on the commercial OS (Windows)
  are 3 times more likely to experience a software
  defect compared with systems running on the open
  OS (Linux)

30
May Allow for Targeted Improvement or Improved
Testing

• Total deployment time (deployment issue)
• Operating system (software platform, hardware
  configurations)
• May be due to familiarity with the operating
  system
• May be due to operating system complexity

31
More Results in Paper

• The complete results and analyses for field
  defects
• Predictions for other customer interactions

32
Validation of Results and Method

• We accounted for data reporting differences
• Included indicator variables in the models to
  identify populations (e.g. US or international
  customers)
• We independently validated the data collection
  process
• Independently extracted data and performed
  analyses
• We interviewed personnel to validate findings
• Programmers
• Field technicians

33
Summary Identified Predictors of Customer
Perceived Quality

• We identified and quantified characteristics,
  like time of deployment, that can affect customer
  perceived quality by more than an order of
  magnitude

34
Summary Modeled Customer Interactions

• We identified and quantified characteristics ,
  like time of deployment, that can affect customer
  perceived quality by more than an order of
  magnitude
• We created models that can predict various
  customer interactions and found that predictors
  have consistent effect across interactions

35
Summary Deployment is Important for High
Reliability

• We identified and quantified characteristics ,
  like time of deployment, that can affect customer
  perceived quality by more than an order of
  magnitude
• We created models that can predict various
  customer interactions and found that predictors
  have consistent effect across interactions
• We learned that controlled deployment may be the
  key for high reliability systems

36
Improve Customers Experiences

• You can target improvement efforts
• You can allocate the right resources to deal with
  customer reported problems
• You can adjust deployment to meet the quality
  expectations of your customers

37
Predictors of customer perceived software quality

• Paul Luo Li (paul.li_at_cs.cmu.edu)
• Audris Mockus (Avaya Research)
• Ping Zhang (Avaya Research)

38
Predicted Number of Calls Match Actual Number of
Calls
Calls for the next release
Calls
Predictions are made here
Time