A MultiObjective, MultiCriteria Approach for Evaluating IT Investments: Results from Two Case Studie

1
A Multi-Objective, Multi-Criteria Approach for
Evaluating IT Investments Results from Two Case
Studies

• G. S. Kearns
• Information Resources Management Journal
• Vol. 17, No. 1, pp. 37-62
• Jan-Mar 2004

2
Outline

• Introduction
• The IT Investment Decision
• The Analytic Hierarchy Process
• The IT Investment Model
• An Information Systems Example
• Evidence From Two Case Studies
• Results of Investment Decisions
• Discussion
• Study Contributions
• Conclusions

3
Introduction (1/3)

• A majority of CEOs
• IT investments were economically infeasible
• Confidence about the future ability of IT to
  provide strategic advantages
• Economic analysis of IT returns relies on
  quantitative measures

4
Introduction (2/3)

• Traditional approach
• Have not proven useful in the economic evaluation
  of IT-based investments
• Single criteria techniques
• Discounted cash flow, cost/benefit analysis
• Bias towards the tangible benefits
• IRR or net present value may ignore the soft ,
  qualitative benefits of IT applications
• Strategic applications
• Require a method
• Reliably measure all benefits in a consistent
  manner that is understood and supported by
  management

5
Introduction (3/3)

• Maximizing returns from IT investments requires a
  total portfolio planning approach
• Can not be accomplished by valuing each
  investment individually
• Mutually exclusive, mutual dependencies
• Should not be combined due to the total risk
• Combined with integer programming and the
  Analytic Hierarchy Process
• Support a multi-objective, multi-criteria
  approach
• Address several issues hindering the success of
  IT investments
• The purpose of the paper
• Demonstrate the MOMC approach to IT investment
  analysis
• The applicability of the proposed model using an
  illustrative example of five information systems
  projects
• The results of two case studies in which the
  model was successfully applied

6
The IT Investment Decision (1/3)

• There is little persuasive evidence
• Investment in IT positively impacts the financial
  position of the firm or increases productivity
• Measurement problem
• Time period between investment and realized
  benefits
• The direction of causality is difficult to prove
• The study examines a more direct method of
  influencing business performance
• Improving the quality of the IT investment
  portfolio

7
The IT Investment Decision (2/3)

• Traditional financial accounting measures
• Past evaluation of IT investments suffer from
• Isolation
• Difficulty in valuation of benefits
• Low explanatory power
• Ignore basic investment tenets
• All financial measures are sensitive to the
  valuation of benefits
• The approach assumes that each investment stands
  on its own merits without regard to other
  investments
• Some investments generally have failing marks
  under ROI and passing marks under net presents
  value
• Such as ERP

8
The IT Investment Decision (3/3)

• IT-related investments represent in excess of
  half the annual capital expenditures for many
  firms
• An agreed-upon approach to measuring IT
  investments does not exit
• Returns on IT investments have been
  unsatisfactory
• The selection of IT-based investments
• Produce the highest value for the firm
• Value must reflect a combination of both
  quantitative and qualitative criteria
• A decision support process is needed that will
  incorporate all relevant decision criteria

9
The Analytic Hierarchy Process

• AHP applications are numerous
• Strategic planning, microcomputer selection, etc
• AHP combines with other techniques
• multi-dimensional scaling and integer and linear
  programming
• No prior illustrations of this use
• The MOMC is an effective measurement process
• Rank alternative investments according to
  criteria
• Corporate strategies
• The strict time constraints of the planning
  process
• Support consensus among a diverse group of
  individuals
• Reflect investment precedence or exclusivity
  constraints
• Incorporate both quantitative and qualitative
  criteria
• Be understood by management

10
The IT Investment Model (1/7)

• Corporate strategies used as project ranking
  criteria
• The importance of linking IT strategies to
  corporate strategies has been well known
• Traditional discounted cash flow techniques lack
  linkage to corporate strategy
• AHP facilitate specification of criteria based
  upon corporate strategies

11
The IT Investment Model (2/7)

• Level of difficulty
• Include
• The flexibility of the measurement process in
  reflecting changes
• Perform sensitivity analysis
• Produce viable alternative solutions
• Provide an explanatory trail
• AHP methodology
• Use a paired-comparisons approach
• The criteria indicators represent typical
  investment alternatives
• The sum of each criterions value becomes the
  investments global score for final ranking

12
The IT Investment Model (3/7)

• Explanatory power
• The most valuable feature of AHP
• A convenient framework for concise representation
• Offer a formal, systematic, consistent approach
• When combined with an integer optimizing model
• The weights can readily be compared
• Managers are able to see into the process

13
The IT Investment Model (4/7)

• Creating consensus
• AHP is highly effective in distilling information
  from groups and fostering consensus
• By paired comparisons
• An important foundation for acceptance
• AHP creates quantitative rankings
• Use a systematic approach to capture priorities
• Measures the consistency of the overall process
• Cost, precedence, and exclusivity constraints
• Resource constraints limit the number of
  investments
• Precluded investment may be due to overlap in
  functionality or competition for non-cash
  resources
• Convert the multi-criteria resource allocation
  problems into integer programming
  maximization-type problems

14
The IT Investment Model (5/7)

• Structuring the AHP hierarchy

15
The IT Investment Model (6/7)

• AHP theory
• An overall view of the complex relationships
• Help the decision-maker assess the importance of
  the issue
• Support meaningful comparisons between attributes
• Steps of using
• Establish the decision hierarchy
• Create input data and make paired-comparisons of
  the decision elements
• Estimate the relative weights of the decision
  elements
• Aggregate the relative weights of decision
  elements to arrive at a final set of ratings
• For the decision alternatives

16
The IT Investment Model (7/7)

• Incorporating quantitative and qualitative
  investments
• In practice and theory
• No consensus on the appropriate mechanism for
  ranking IT investments
• Objective evaluation method
• Net present value, cost-benefit analysis, project
  risk, value analysis, benchmarking, multiple
  criteria approach, DSS evaluation, aggregate
  scoring technique, and anecdotal evidence
• Subjective method
• Attitude surveys and the opinions of users and
  analysts

17
An Information Systems Example (1/4)

• A simple hierarchy illustration
• Includes both financial and non-financial
  criteria
• Compare on the basis of corporate strategies
• Investment risk
• Revenue enhancing
• Operating efficiency
• Customer satisfaction
• Market growth

18
An Information Systems Example (2/4)

• Steps
• Define the decision hierarchy
• The goal is to rank the decision alternatives
• Input the data
• Expert ChoiceTM
• The input data are manipulated using matrix
  algebra to produce the relative weights or
  priorities
• Aggregation of all weights to produce a vector of
  composite relative weights between the criteria
  and the alternatives

19
An Information Systems Example (3/4)
20
An Information Systems Example (4/4)

• Optimizing using integer programming
• Maximize the AHP priority weights with the
  resource constraint
• The optimal solution is (1,1,0,1,0)
• The objective function value is equal to 0.709
• Higher values signify higher overall returns for
  the IT investments

21
Evidence From Two Case Studies (1/7)

• Research methodology
• Two case studies
• Use the IT investment model
• Contextual conditions could impact the outcomes
• The goals
• Ascertain the efficacy of the proposed ranking
  mechanism
• Collect and report the attitudes, behaviors, and
  perceptions
• Results were reviewed by the CIOs with minor
  corrections and revisions
• Use multiple cases
• Allow the investigator to replicate the results
  and improves generalizability
• The study will show
• Management involvement is necessary
• Organizational structure affects the success of
  the ranking process
• Hot and Lukewarm

22
Evidence From Two Case Studies (2/7)

• Case study background of companies
• Two U.S. utility companies
• North-central region and southern region
• Similarities
• Generators of electricity, retail and wholesale
  markets, sold surplus power, and controlled their
  own transmission and distribution systems
• Both had CIOs committed to IT planning

23
Evidence From Two Case Studies (3/7)

• Hot
• Smaller company under greater competitive
  pressure
• Highly participative management structure with
  younger management
• Previous experience in non-regulated industries
• Highly committed to planning and the strategic
  use of IT
• Lukewarm
• Relatively secure markets
• Issues of deregulation
• Shortly put markets under competitive pressures
• With traces of political rivalry
• Top management was without experience outside
  their field
• CEO and CIO had previous experience in
  non-regulated industries
• Committed to planning and increasing returns on
  IT investments

24
Evidence From Two Case Studies (4/7)

• IT planning and evaluation - Hot
• Interest in IT planning and using IT
  strategically
• Want a system
• Satisfy all areas of management
• Ask IT management for assistance in identifying
  technologies
• That might allow revision of business processes
  to improve efficiencies and customer service
• A combination of project evaluation tools
• ROI, payback, and a corporate model
• Useful but probably unreliable

25
Evidence From Two Case Studies (5/7)

• IT planning and evaluation - Lukewarm
• Delegate all IT planning to the CIO
• Complain about the time and cost of implementing
  systems
• IT steering committee
• Composed of several senior managers
• Rely heavily on the opinion of the CIO
• The IT plan contained
• A wish list of applications that continually
  changes with the political climate
• Use a cost/benefit and payback approach
• Selection of projects depends on
• How well managers could creatively assign dollars
  to benefits

26
Evidence From Two Case Studies (6/7)

• The Hot results
• The decision criteria and sub-criteria
• Originally developed by a team of IT managers
• Later modified by other members of management
• The participators were familiarity with AHP prior
  to the session
• Use a modified Delphi technique to decide the
  weights
• IT management played an impartial advisory role
• The initial analysis was completed
• Working with managers from finance, engineering,
  and marketing
• Use both the AHP and integer programming models
• Disadvantage
• Total time involved in making the
  paired-comparisons and estimating other
  parameters
• Advantage
• Their understanding of the process would help to
  make future estimates easier and cut the time
  requirement
• Select five IT investments with a capital
  requirement in excess of 18.5 million

27
Evidence From Two Case Studies (7/7)

• The Lukewarm results
• Expected to benefit from the results of the Hot
  experience
• Partly implemented and with less success
• Less efficient session
• A cross-functional management team
• Review and refine the comparisons after
  individual discussions with managers
• The team would have final authority
• The CEO supported the process but didnt
  participate directly
• On the advice of the team
• The investments identified as strategic, high
  cost, and high risk were evaluated
• 26 investments were analyzed
• Many were overlapping and mutually exclusive
• 8 investments were selected with a capital cost
  in excess of 34 million
• Problems
• Many managers continually requested revisions of
  the management team
• Use a spreadsheet program
• Perform a modified ROI analysis on the selected
  projects
• IT managers felt
• The direction was an improvement

28
Results of Investment Decisions (1/5)

• Acceptance
• Managers form both companies
• Enthusiasm
• The documentation for the methodology improved
  their understanding and made it easier for new
  managers to grasp
• Hot
• The internal environment and organizational
  structure are more conducive to acceptance of new
  processes
• Lukewarm
• Acceptance of the methodology had removed a major
  burden from IT planning
• No longer incurred the wrath of managers who had
  not been funded
• This supports
• One of the benefits of the MOMC approach is the
  balancing of conflicting objectives of different
  users and stakeholders

29
Results of Investment Decisions (2/5)

• Status of the IT investments selected
• There was no immediate pressure to cut capital
  investments
• Hot
• Lower earnings-per-share
• Delay one project to conserve cash and deploy
  resources to the other projects in order to
  realize the benefits more quickly
• All of the projects were on or under schedule and
  under budget
• Lukewarm
• Benefit from reduced political tensions
• Most of the projects were on schedule and within
  budget
• The delayed project had suffered from a political
  tug-of-war about infrastructure issues
• IT projects were an outstanding success

30
Results of Investment Decisions (3/5)

• Status of selection process
• Hot
• Managers were continuing to modify and enhance
  the model
• They wanted to be able to analyze individual
  investments on a stand-alone basis
• The use of a program to quickly generate an
  initial set of paired-comparisons
• Two strategic categories emphasized on valuation
  of intangible benefits
• Tow over 1 million categories emphasized on risk
  analysis
• Lukewarm
• The CEO had to contend with several presidents of
  the operating companies
• Less time to focus on IT
• Time period was not sufficient
• Little had been accomplished towards improving
  the process, primarily documentation of the
  process and the training of new managers
• The CIO was confident
• The next round of investment proposals would be
  handled more expeditiously

31
Results of Investment Decisions (4/5)

• Generalizable findings
• In one firm
• The CEO had greater knowledge of IS
• The CEO worked closely with the CIO and other
  managers followed the lead
• In the other firm
• The CEO had superficial knowledge
• The CEO did not work closely with the CIO
• Hot had capitalized on the new process to insure
  success and reduce the time requirement on
  management
• By extending the model and adding administrative
  controls
• Lukewarm accomplished less
• Managers in both firms had an improved attitude
• The new process improved the quality of
  information available to measure investment
  proposals, increased the involvement of managers,
  and added credibility to the final results
• An investments potential return may be reduced
• Because of implementation problems
• The inability to control quality during software
  system development

32
Results of Investment Decisions (5/5)

• Summary

33
Discussion (1/2)

• Benefits
• The ability of the model to handle a large number
  of criteria
• The ability to represent both tangible and
  intangible items
• The ability to model exclusivity and dependency
  of investments
• The ability to quickly reflect revisions
• The explanatory power of the model
• The support for group decision-making
• Limitations
• The lack of a financial measure of profitability
• The overall time requirements for management
• The problem of valuing intangibles, although
  ameliorated, remained

34
Discussion (2/2)
35
Study Contributions (1/2)

• Provide a tested process for prioritization and
  selection of IT investments
• Identify benefits and limitations inherent within
  the process
• Identify facilitators and inhibitors and
  generalizable findings to the approach
• Assist the introduction of the process

36
Study Contributions (2/2)

• Suggestions for future research
• Further case studies
• Suggested from different industries
• Provide more insights into the completeness of
  the approach
• Examine the impact of contextual variables on the
  success of the IT investment model
• The balancing of investment risk was not tested
  in this study
• The relationship between process credibility and
  subsequent development and implementation remains
  unresolved

37
Conclusions

• The MOMC approach merits attention as a
  investment selection and ranking tool
• Utilize AHP and integer programming
• Improve the IT investment process
• Strictly quantitative approaches have not yielded
  satisfactory results
• Subjective approaches lack explanatory power and
  can not be easily adjusted to reflect new
  knowledge
• Basing selection criteria on business strategies
  ensures the alignment of IT investments with
  these strategies