Neil Mitchison Neil'Mitchisonjrc'it

1
Systemic risks and the deployment of new
technology

• Neil MitchisonNeil.Mitchison_at_jrc.it

2
Risk management 4 failures

• Failure to identify hazards
• Failure to control changes
• Failure to identify assumptions
• Carry-over from the manual world
• Failure of useability
• Not my problem

3
Hazard identification

• A trusted world ...
• Vulnerabilities attract attackers
• Changes
• too many, too fast
• in typical IT systems, unpredicatble ripple
  effect

4
Assumptions

• The OS is stable
• Users will read the manual(all of it?)
• If it worked in the real-world itll work in the
  IT world (unconscious checks)
• ... the trickiest What assumptions do others
  make about my system?

5
Useability

• Definition of a computer password
• a collection of letters, numbers and symbols,
  whose destiny is to live on a small piece of
  yellow paper on the side of a computer terminal
• gt We need useability solutions
• But also do we know which risks a security
  feature protects us from?

6
and perhaps the most pernicious of all ...

• Its not my problem

7
Systemic risks

• arise from large-scale deployment
• difficult to predict from pointwise analysis
• arise from points of systemic vulnerability gt
  very serious
• may even arise only from systems of systems
  (e.g. data mining and privacy concerns)

8
Example biometrics

• Biometric systems are sure to appear
• Two important distinctions
• authentication / identification
• central / local
• Different sorts of systemic risks
• single failure point
• forgeable?

9
Biometrics contd.

• identification gt central system
• (but not vice versa)
• can we make a central system which is not a
  single failure point?
• non-forgeability gt
• either central verification
• or local cryptographic verification, robust
  against technological attack for ?10 years?

10
Data mining and privacy

• Not my problem
• system of systems problem
• Im just developing a technical solution (but
  once Ive deployed it, itll be too expensive to
  modify)
• Economic arguments for individual operators
• gt case for legislative/governmental intervention

11
Black-outs

• Worst problems come at system boundaries (e.g.
  national)
• By international agreement, generators break off
  at 47 Hz.
• Many generators break off within a few seconds gt
  system collapse

12
Remedies?

• strive to identify hazards(not routine work
  imagination needed)
• make assumptions explicit
• test for useability
• integrated system test
• also
• bounce it off the informed community
• and sometimes
• legislative intervention - but dont hold your
  breath

13
Conclusion

• Were still making the same mistakes
• The technology deployers need risk analysts
  (convince them!)
• Legislation is too slow
• and yet ...
• We are deploying an enormous number of new
  systems with fair success