Measuring the Effectiveness of ERP systems

1
Measuring the Effectiveness of ERP systems

• Peter Seddon, PhD
• Senior Lecturer
• Department of Information Systems
• The University of Melbourne
• p.seddon_at_dis.unimelb.edu.au
• http//www.dis.unimelb.edu.au/staff/peter

2
Measuring the Effectiveness of ERP systems

• Chinese version translated by
• Bin Hu
• Department of Information Systems
• The University of Melbourne
• bin_at_staff.dis.unimelb.edu.au

3
Measuring the Effectiveness of ERP systems

• 1. An example calculation of ROI
• 2.Why is measuring IT Effectiveness so hard?

• 3.What do we know about IT Effectiveness
  measurement in practice?
• 4. Individuals IT effectiveness
• 5. Senior Managers ERP effectiveness
• 6. Summary and Lessons

4
Measuring the Effectiveness of ERP systems

• 1. An example calculation of ROI
• 2.Why is measuring IT Effectiveness so hard?

• 3.What do we know about IT Effectiveness
  measurement in practice?
• 4. Individuals IT effectiveness
• 5. Senior Managers ERP effectiveness
• 6. Summary and Lessons

5
1. Example calculation of ROI

• Report sponsored by SAP
• Analysis of The Boston Beer Companys investment
  in an SAP R/3 system
• Implementation, January, 1996
• Standard IRR calculation ROI of 83

Data from The ROI Report, Scalea and Company,
346 Beacon Street, Boston MA, August, 1997
6
Boston Beer in 1997

• Leading craft brewer in the US
• contract brewer - using excess capacity of other
  brewers - plus one brewery in Cincinnati
• 380 employees 190 in sales, 120 in brewing, and
  70 in administration
• 1991-1996 net sales growth 46 compound, from
  30M to 191M

7
Boston Beers SAP R/3 Modules

• FI Financials
• SD Sales and Distribution
• AM Asset Management
• MM Material Management
• CO Controlling
• PP Production Planning
• PA Profit Analysis
• BS Basis (middleware)

8
The Investment, 1995-1996
9
Estimated Benefits, 1996-2000

• Reduction in Gen. Admin. Expense 5 of 12.5m
  300,000 p.a. (rising to 900,000 by year 2000)
  
• Interest Savings on A/R reduction 6 of
  reduction of 4m in Working Cap 240,000
  (rising to 500,000 by year 2000)
• Improved Budget Control of Sales Expenses
  300,000 (rising to 800,000 by year 2000)
• Improved Control of POS Expenses 240,000
• Increased A/P Discounts 100,000 p.a

10
(No Transcript)
11
Calculating ROI (IRR), 1996?
12
Boston Beer Share Price 1996-99

• Source http//www.etrade.com, 27 Nov 1999

13
Measuring ERP Effectiveness

• Boston Beers sales fell 3 during 1996-98.
• Most of the expected cost savings (which assumed
  sales growth) were not realized.
• The expected ROI was not achieved.
• Does this mean the investment in the ERP system
  was a failure?
• How should we measure ERP effectiveness?

14
Measuring the Effectiveness of ERP systems

• 1. An example calculation of ROI
• 2.Why is measuring IT Effectiveness so hard?

• 3.What do we know about IT Effectiveness
  measurement in practice?
• 4. Individuals IT effectiveness
• 5. Senior Managers ERP effectiveness
• 6. Summary and Lessons

15
2. Why is measuring IT Effectiveness so hard?
(not just for ERP systems)

• Often hard to link benefits to any specific IT
  investment (e.g., benefits of a LAN).
• Cost savings are the easier to quantify, but
  these are not the only benefits.
• There are many different goals for evaluation
  (e.g., feasibility, audit)
• Different stakeholders evaluate the same system
  differently
• Evaluations change over time

16
Willcocks and Lester list six types of goals for
IT investment

• Infrastructure building (cant cost justify)
• Cost efficiency (e.g., Boston Beer)
• Service to the business (e.g., better
  information, better decision-making)
• Enable business improvement (processes)
• Differentiating the business competitively
• Revenue generation

Willcocks and Lester, Beyond the IT Productivity
Paradox Wiley 1999
17
Five Key Questions to get clear about IT
Effectiveness evaluation

• Q1. From whose perspective is effectiveness being
  judged?
• Q2. What is the system being evaluated?
• Q3. What is the purpose of evaluation?
• Q.4 What time frame is employed? (short, long)
• Q.5 How is effectiveness to be judged?

Based on Seddon, Staples, Patnayakuni, and
Bowtell, Dimensions of IS Success, Communications
of the AIS October 1999
18
Q1. From whose perspective is effectiveness being
judged?

• independent stakeholder
• individual user
• a group of people
• project manager
• IT management
• owner/senior management
• society

19
Q2. What is the system being evaluated?

• part of an information system (e.g., GUI)
• one particular system (e.g., the HR module)
• a type of system (e.g., ERP)
• all IT applications in an organization
• an inter-organizational information system
• an IT implementation process (e.g., ASAP)
• the IT function in an organization

20
Q.3 What is the purpose of the evaluation?

• Performance appraisal
• to learn how to do better feasibility-study
  evaluations in future
• expect different levels of cooperation from staff
  depending on purpose of evaluation
• some evaluations may be much more comprehensive
  than others

21
Q.4 What is the time frame is employed?

• short term (e.g., on time and within budget for
  implementation projects)
• long term

22
Q.5 How is effectiveness of the IT system to be
judged?

• Compared to
• some other organization (benchmarking)
• some ideal level of performance
• stated goals of the organization (e.g., the
  feasibility study for an IT project)
• past performance of the organization
• other desirable characteristics

23
Measuring the Effectiveness of ERP systems

• 1. An example calculation of ROI
• 2.Why is measuring IT Effectiveness so hard?

• 3.What do we know about IT Effectiveness
  measurement in practice?
• 4. Individuals IT effectiveness
• 5. Senior managers ERP effectiveness
• 6. Summary and Lessons

24
Firms evaluate effectiveness at 3 points in the
systems lifecycle
J.W. Ross, The ERP revolution Surviving versus
Thriving, Working Paper, CISR, MIT, 1998
25
Firms evaluate effectiveness at 3 points in the
systems lifecycle
Feasibility Study
J.W. Ross, The ERP revolution Surviving versus
Thriving, Working Paper, CISR, MIT, 1998
26
Firms evaluate effectiveness at 3 points in the
systems lifecycle
Feasibility Study
Development Stage
J.W. Ross, The ERP revolution Surviving versus
Thriving, Working Paper, CISR, MIT, 1998
27
Firms evaluate effectiveness at 3 points in the
systems lifecycle
Feasibility Study
Development Stage
Post-implementation
J.W. Ross, The ERP revolution Surviving versus
Thriving, Working Paper, CISR, MIT, 1998
28
What percentage of firms evaluate their
investments in IT? (formal reviews not just
ERP)
Source Seddon, Graeser, and Willcocks (1999)
data from IT managers in 80 large European and US
firms in 1998. Average annual turnover US4.5B.
Average annual IT budget was US38 million.
29
1. Feasibility Stage Evaluation problems(survey
of 97 UK firms, Ballantine 1993)
30
1. Feasibility Stage Evaluation Methods (80
respondents in US, UK, A NZ, Bacon 1992)
31
2. Development-stage Evaluation
32
2. Development-stage Evaluation

• For project managers, the key success measure is
  often on time and within budget.
• This has little to do with the economic value of
  the project.
• Some projects are abandoned, which means the
  entire cost of the project is wasted.
• Studying these abandoned projects gives insight
  into development-stage problems and evaluation

33
2. Development-stage Evaluation(Willcocks and
Lester 1996, n50)

• 80 of the 50 organizations had abandoned
  projects because of negative development-stage
  evaluations.
• The most important reasons for abandonment were
• project over budget (17/50 respondents)
• organizational needs changed (23/50 respondents)
• user requirements changed (22/50 respondents).

34
2. Development Stage Evaluation Reasons for
project abandonment (Norris 1996)

• Unacknowledged divisions between users on
  mandatory and desirable requirements and the
  scale of benefits expected from them.
• Gung-ho attitudes to managing risks.
• An assumption that short training courses at the
  launch of the system will be sufficient to change
  well-established working practices and to
  encourage users to adopt the system.
• The above are consistent with findings on BPR.

35
3. Post-implementation Evaluation
Post-implementation
36
3. Criteria for Post-Implementation
Review(Willcocks and Lester 1996, n50)

• 83 comparison to the feasibility study
• 63 cost-effectiveness
• 53 quality of product
• 48 systems availability
• 44 productivity
• 22 user satisfaction
• 15 of organizations used the top six criteria,
  and a further 15 used the top five (excluding
  user satisfaction).

37
3. Post-implementation Reviews
Question
Source Seddon, Graeser, and Willcocks (1999),
n80
38
3. Post-implementation Reviews
Question
Source Seddon, Graeser, and Willcocks (1999),
n80
39
3. Post-implementation Reviews
The remainder of this presentation focuses on
post-implementation benefits only, i.e., on
benefits from use, once the system is operational.
40
Measuring the Effectiveness of ERP systems

• 1. An example calculation of ROI
• 2.Why is measuring IT Effectiveness so hard?

• 3.What do we know about IT Effectiveness
  measurement in practice?
• 4. Individuals IT effectiveness
• 5. Senior Managers ERP effectiveness
• 6. Summary and Lessons

41
4. A model of factors affecting individual
perceptions of IS effectiveness (n119266)
0.2
Importance of the task
Knowledge of the system
0.2
ns
0.5
0.1
0.4
Support Staff
Ease of Use
Perceived Usefulness
0.3
ns
0.3
0.2
Information Quality
Perceived Net Benefit
0.2
42
Comparing scores on Old and New Systems
(September, 1996 and January, 1998. All scales
17)
Source Seddon and Staples (1999)
43
Measuring the Effectiveness of ERP systems

• 1. An example calculation of ROI
• 2.Why is measuring IT Effectiveness so hard?

• 3.What do we know about IT Effectiveness
  measurement in practice?
• 4. Individuals IT effectiveness
• 5. Senior Managers ERP effectiveness
• 6. Summary and Lessons

44
Why are senior managers interested in ERP
effectiveness?

• To improve IT investment decision-making in
  future
• To learn from reported experience of others
• to understand capabilities of ERP systems and so
  tap possible sources of competitive advantage
• to identify areas to target for future
  development

45
Answers to the five key questions for evaluating
IT effectiveness

• Stakeholder senior management
• System the entire ERP system
• Purpose of evaluation previous slide
• Time frame costs and benefits to date, plus
  costs and benefits for the next 2-3 years
  (including software and hardware upgrade costs)
• Effectiveness criteria Balanced IT Scorecard

46
Balanced IT Scorecard criteria

• 1. From the corporate financial perspective,
  e.g., ROI
• 2. The customer/user perspective (e.g., on-time
  delivery rate, satisfaction)
• 3. Business process (e.g., purchase invoices per
  employee)
• 4. An innovation/learning perspective (e.g., rate
  of cost reduction for IT services)
• 5. From the systems project perspective (e.g.,
  on-time, quality, cost)
• 6. A technical perspective (e.g., implementation
  efficiency, capacity utilization, response times)

Source Graeser and Willcocks (1998)
47
Norris says for post-implementation evaluation of
IT effectiveness, create a task force

• Different stakeholder interests must be
  represented
• Task force must have
• Authority vested from the top
• Board-level access
• Company-wide emphasis
• Business management insight.

Source Norris (1996)
48
Benefits checklist Analysis of vendor success
stories on WWW (Shang, PhD in progress, 1999)
49
Content analysis of the 182 WWW cases suggests
that, from the perspective of senior management,
there are five categories of benefits from ERP
systems (Shang 1999)

• 1. Operational
• 2. Management
• 3. Strategic
• 4. IT infrastructure
• 5. Organizational

50
1.Operational benefits (Shang 1999)

• Cost reduction labour, training, inventory
• Cycle-time reduction
• faster delivery to customer,
• faster administrative processes
• Increased productivity
• Improved data quality
• Improved customer service

51
2.Managerial benefits (Shang 1999)

• Better resource management inventory,
  maintenance, production scheduling, workforce
  management
• Better decision making improved market
  responsiveness, fast response to work changes,
  fast response to customer needs
• Better control analysis by line of business,
  product, customer, geographic area production
  costs management

52
3.Strategic benefits (Shang 1999)

• Support for future business growth
• Support for business alliances
• Lock-in customers (realtime data sharing,
  interactive customer service)
• build business innovation
• build cost leadership
• enhance product differentiation
• sustain competitiveness

53
4.Infrastructure benefits (Shang1999)

• Increased business flexibility
• reduction in IT costs legacy systems
  maintenance, mainframe replacement, year 2000
  compliance
• increased infrastructure capability global
  platform, database integrity
• flexibility adaptable modern technology,
  extendable, compatible

54
5.Organizational benefits (Shang1999)

• Support organizational changes (restructuring)
• business and employee skills learning
• greater customer focus for staff
• empowerment and accountability
• teamwork
• better employee morale and satisfaction

55
Downsides of ERP systems

• Vendor success stories do not mention the
  problems of ERP system use such as
• Dependence on vendor upgrades
• Poor response times
• Inflexibility/Expensive customization
• Staff leaving for better salaries
• others

56
Plan (for next year)

• Persuade a number of user organizations to use a
  combination of the IT Scorecard and the ERP
  Benefits Checklist to assess the effectiveness of
  their current systems.
• Will also ask them to identify downsides
  (unwanted consequences) of ERP systems.
• Hope to identify some common factors that led to
  greater perceived success.
• Expect user knowledge to be a key determinant of
  success.

57
Measuring the Effectiveness of ERP systems

• 1. An example calculation of ROI
• 2.Why is measuring IT Effectiveness so hard?

• 3.What do we know about IT Effectiveness
  measurement in practice?
• 4. Individuals IT effectiveness
• 5. Senior Managers ERP effectiveness
• 6. Summary and Lessons

58
6. Summary and Lessons

• IS effectiveness measurement is difficult because
  there is no simple causal relationship between IT
  expenditure and benefits.
• Financial measures such as Boston Beers ROI, and
  changes in corporate profitability, are clearly
  invalid in some circumstances.
• The previous section contains our best recipe for
  ERP effectiveness measurement from the senior
  management perspective
• Taskforce scorecard checklist.

59
Finally

• Establishing value for money depends on the
  business judgement of the managers involved - it
  is no more amenable to numerical analysis than
  any other value judgement. (Norris 1996)

Source Norris, G.D. Post-investment appraisal,
in Willcocks L. Investing in Information Systems,
London Chapman and Hall, 1996 193-221.
60
Questions?

• Peter Seddon and Bin Hu
• Department of Information Systems
• The University of Melbourne
• p.seddon_at_dis.unimelb.edu.au
• http//www.dis.unimelb.edu.au/staff/peter