Analyzing the Risks of Information Security Investments with Monte-Carlo Simulations

1
Analyzing the Risks of Information Security
Investments with Monte-Carlo Simulations

• WEIS05

Workshop on the Economics of Information Security
James R. Conrad, University of Idaho Department
of Computer Science conr2286_at_uidaho.edu
2
Contents
Introduction to the Problem The Monte-Carlo
Solution Overview of Monte-Carlo
Simulations Example Analysis and
Critique Conclusions
3
Introduction to the Problem

• An information security investment may need to
  compete for resources with other business
  opportunities
• But many information security models rely upon
  experts estimates
• And the experts estimates may include
  significant uncertainty
• How can the analyst communicate an opportunity
  when so much is uncertain?

4
Monte-Carlo Solution

• Apply the Monte-Carlo technique to simulate and
  express uncertainty in information security
  models
• This is not a new model --- this is an
  enhancement of existing models
• While less common in the Computer Science
  discipline, many financial decision makers are
  already familiar with the Monte-Carlo approach

5
Monte-Carlo Simulations

• Specify uncertainty in probability distributions
• Monte-Carlo engine samples distributions
• Engine executes the security model once for each
  of several thousand iterations
• Monte-Carlo engine captures and collects the
  result of each iteration

distributions
engine
model
results
6
Monte-Carlo Simulations

• Engine simulates uncertainty in the model
  parameters
• Model continues to operate with discrete values
• Extra complexity largely confined to the
  Monte-Carlo engine
• Results can be charted as probability
  distributions

distributions
engine
model
results
7
Monte-Carlo Example

• Based upon Longstaff et als example appearing
  in Are we Forgetting the Risks of Information
  Technology? of IEEE Computer, December 2000
• Simulates the benefit/cost ratio of a proposed
  infosec investment for a financial enterprise
• Modeling parameters are similar to Longstaffs
  example with an added complication
• The experts dont agree!

8
Original (pre-Monte-Carlo) Parameters Model
intrusion rates
Intrusion Rate Parameters r1 2 Simulated
annual intrusion count w/o investment e 5.00E-01
Effectiveness of investment r2 r1e Annual
intrusion count with investment Other
Parameters p1 r1/365 Daily probability of
intrusion w/o investment p2 r2/365 Daily
probability of intrusion with investment X 20,000
,000,000,000 Asset value y1 100,000,000 Cost
of software assurance w/o investment y2 200,000,0
00 Cost of software assurance with
investment z1 1.00 Losses w/o
investment z2 0.50 Losses with
investment Model Calculations d1 p1z1 Calc
damage w/o investment d2 p2z2 Calc damage
with investment D y2-y1 Calc cost to provide
software assurance with investment d d1-d2 Calc
percentage of losses prevented by
investment b dX-D Calc net benefit of
investment bcr b/D Calc benefit/cost ratio for
investment (bcr7.22)
other parameters
model
benefit/cost ratio, bcr
9
Uncertainty in the Revised Example

• Consider a case in which the experts dont agree
  upon an single value estimate for the annual
  intrusion rate (fixed at r12 events/year in the
  original problem)
• The hypothetical disagreement stems from
  uncertainty in anticipated business practices
• Experts do agree there exists a 20 chance that
  business practices will change in a way that will
  raise the intrusion rate to 20 events/year and an
  80 chance that those practices will remain
  unchanged

10
Uncertainty in the Revised Parameters

• Model variability of optimistic intrusion rate
  as a Poisson process (for purposes of this
  example), rorandpoisson(2)
• Model variability of pessimistic intrusion rate
  as a Poisson process, rprandpoisson(20)
• Model uncertainty of anticipated business
  conditions by choosing the optimistic rate 80 of
  the time and the pessimistic rate 20 of the time
  using randdiscrete(0.80,0.20,ro,rp)
• Variability refers to a truly random process
• Uncertainty refers to the experts inability to
  anticipate future business conditions

11
Revised Params Model
intrusion rates
Intrusion Rate Parameters ro randpoisson(2)
Optimistic annual intrusion count w/o
investment rp randpoisson(20) Pessimistic
annual intrusion count w/o investment r1 randdisc
rete(0.8,0.2,ro,rp) 80 Chance of ro. 20 Chance
of rp. e 5.00E-01 Effectiveness of
investment r2 r1e Annual intrusion count with
investment Other Parameters p1 r1/365 Dail
y probability of intrusion w/o investment p2 r2/3
65 Daily probability of intrusion with
investment X 20,000,000,000,000 Asset
value y1 100,000,000 Cost of software
assurance w/o investment y2 200,000,000 Cost
of software assurance with investment z1 1.00 Lo
sses w/o investment z2 0.50 Losses with
investment Model Calculations d1 p1z1 Calc
damage w/o investment d2 p2z2 Calc damage
with investment D y2-y1 Calc cost to provide
software assurance with investment d d1-d2 Calc
percentage of losses prevented by
investment b dX-D Calc net benefit of
investment bcr b/D Calc benefit/cost ratio for
investment
other parameters
model
benefit/cost ratio, bcr
12
Simulation of Revised Example

• randpoisson() and randdiscrete() sample the
  probability distributions in each iteration of
  the simulation
• The Monte-Carlo engine recalculates the model
  for each iteration and captures the results (bcr)
• The Monte-Carlo engine charts the captured
  simulation results (next slide)

13
Simulation Results
14
Why not use a weighted average of r1 and r2?

• Why doesnt the revised model simply compute a
  weighted average of the two possible intrusion
  rates?
• r1 randpoisson(2)0.8randpoisson(20)0.2
• The randdiscrete() simulation preserves the
  bimodal nature of the experts disagreement.
• Any attempt to average away that uncertainty
  conceals the truth The experts dont agree.

15
Analysis

• The results reflect the experts strong
  preference for the optimistic intrusion rate in
  which the benefit/cost ratio remains unchanged at
  7.22. Risk-tolerant decision makers might manage
  to this value.
• The mean value lies at 22 between the two modes.
• The results also reflect a second mode at about
  81 along with a 10 chance of the benefit/cost
  ratio exceeding 81. Risk-adverse decision makers
  might manage to this value to avoid a catastrophe
  on their watch.

16
Critique

• But are real experts willing to provide even
  more estimates?
• The authors industry experience with
  Monte-Carlo models is that many experts are
  relieved to disclose the uncertainty they know to
  be in their estimates
• What real experts truly dislike is being held
  accountable to an expected value they know is
  merely representative of the possibilities

17
Additional Critique

• Given a tool to express uncertainty as
  probability distributions, which distributions
  closely model the empirical evidence?
• How to extend the Monte-Carlo approach to
  graphical models?

18
Conclusions

• Monte-Carlo techniques offer an approach to
  simulate uncertainty in expert estimates
• Enables the use of probability distributions for
  model parameters and forecast results
• The Monte-Carlo engine simulates random
  variables, allowing a security model to continue
  to manipulate discrete values with only minimal
  changes
• May be particularly useful for visualizing the
  potential of an extreme event, the unlikely
  possibility of a catastrophic outcome

19
Questions and Optional Slides
20
Why Poisson Distribution?

• The example problem uses a Poisson process to
  approximate intrusion attempts
• If and/or when the Poisson process usefully
  reflects empirical intrusion attempts is an open
  question
• Review Models the number of events occurring
  during a specified time interval for a Poisson
  process
• Review Continuous opportunity for independent
  events to occur
• Review Long-term rate is constant
• Review Used to model lightening strikes in a
  storm

21
Correlated Parameters

• Every iteration of a model must be a scenario
  that could physically occur. -- Vose.
• The parameters must make sense to the security
  model!
• One correlated parameter can usually be
  expressed as a function (relation) of another.
• Consider r1 and r2 in the example. These are
  likely related which is why r2 is calculated as a
  function of r1.
• If the relationship (e) between r1 and r2 is
  also uncertain, this too can be simulated.

22
Variability and Uncertainty

• Yes, this example lumped (simulated) variability
  and uncertainty together for simplicity
• Vose (Risk Analysis, 2000) offers an excellent
  treatment of this subject for those who need to
  keep them separated

23
Partitioning

• Yes, partitioning is an alternative technique
• The Monte-Carlo technique might be viewed as an
  automated approach to partitioning
• and the Monte-Carlo technique avoids the
  subjective choice of partition boundaries
• and the Monte-Carlo technique has commercial
  tool support for systems-level models.

24
Commercial Tools

• Yes, commercial off-the-shelf tools are
  available
• They are most useful for systems-level security
  models.
• They are less useful for low-level combinatorics
  security models
• Search for monte carlo simulation and pay
  particular attention to the Sponsored Links

25
Performance

• The authors industry experience includes
  Monte-Carlo simulations using hundreds of
  random distribution parameters
• Yes, they required several hours to run
• In 1997!
• My computer is more than 10X faster today.
• Simulation multiplies model complexity by n, the
  number of iterations. A simulation of an O(m2)
  model becomes nO(m2).
• Opportunities for parallel approaches when n
  cannot be ignored.