Title: A Gift of Fire: Social, Legal, and Ethical Issues for Computers and the Internet
1- A Gift of Fire Social, Legal, and Ethical Issues
for Computers and the Internet
2A Gift of Fire
Errors, Failures, and Risks
- What Can Go Wrong?
- Case Study The Therac-25
- Increasing Reliability and Safety
- Perspectives on Failures, Dependence, Risk, and
Progress - Computer Models
3What Can Go Wrong?
- Facts About Computer Errors
- Error-free software is not possible.
- Errors are often caused by more than one factor.
- Errors can be reduced by following good
procedures and professional practices.
Q How do we distinguish between tolerable or
unavoidable errors in software versus careless
software development?
4What Can Go Wrong?
- The Roles of People in Computer-related Problems
- Computer User
- At home or work, users should understand the
limitations of computers and the need for proper
training and responsible use. - Computer Professional
- Understanding the source and consequences of
computer failures is valuable when buying,
developing, or managing a complex system. - Educated Member of Society
- Personal decisions and political, social, and
ethical decisions depend on understanding
computer risks.
5Categories of Computer Errors and Failures
- Problems for Individuals
- usually in their role as consumers.
- Example OCR, Speech Recognition
- who are incorrectly identified by inaccurate law
enforcement databases. - Wrong identity
- System Failures
- affecting large numbers of people and/or costing
large amounts of money. - Subway Control
- Safety-Critical Applications
- where people may be injured or killed.
- Nuclear Station
6- Problems for Individuals
- Billing Errors
- Lack of tests for inconsistencies and
inappropriate amounts (HK MTR). - Database Accuracy Problems
- Incorrect information resulting in wrongful
treatment or acts. - Terrorists Database (Brazilian killed by London
Police) - Military Database (Bombing of Chinese Embassy in
Yugoslavia)
Q Describe a computer error or failure that has
affected you.
7Causes for Computer Failures
- Large population. (Mistakes unavoidable)
- Human common sense not part of automated
processing. (When in doubt, verify never guess) - Overconfidence in the accuracy of data from a
computer. (User Responsibility GPS is an
example) - Errors in data entry. (Careless? Double Check?)
- Information not updated or corrected. (Chinese
Embassy in former Yugoslavia) - Lack of accountability for errors. (Whose duty?)
8- System Failures
- Communications
- Telephone, online, and broadcast services.
- Business
- Inventory and management software.
- Financial
- Stock exchange, brokerages, banks, etc..
- Transportation
- Reservations, ticketing, and baggage handling.
9- System Failures (contd)
- Causes
- Insufficient testing and debugging time.
- Significant changes in specifications (during and
after project begun). - Overconfidence in system.
- Mismanagement of the project.
Q Describe a recent system failure that affected
many people or resulted in a great monetary loss.
10- Safety-Critical Applications
- Military
- Power Plants
- Aircraft
- Trains
- Automated Factories
- Medicine
- others.
11Causes for Computer Failures (System)
- Safety-Critical Applications
- Overconfidence.
- Lack of override features.
- Insufficient testing.
- Sheer complexity of system.
- Mismanagement.
- More than one factor can occur at the same time!
- HK Air Cargo Terminal Case!
Q What activities do you participate in that are
controlled by safety-critical applications?
12Case Study The Therac-25
- The Therac-25 was a software-controlled
radiation-therapy machine used to treat people
with cancer. - Overdoses of radiation
- Normal dosage is 100200 rads.
- It is estimated that 13,000 and 25,000 rads were
given to six people. - Three of the six people died.
Q What determines whether the risks associated
with a safety-critical application are acceptable?
13Case Study The Therac-25
- Therac-25 Radiation Overdose
- Multiple Causes
- Poor safety design.
- Insufficient testing and debugging.
- Software errors.
- Lack of safety interlocks.
- Overconfidence.
- Inadequate reporting and investigation of
accidents. - How to Remind us????
14Increasing Reliability and Safety
- What Goes Wrong?
- Computer Systems Fail Because
- The job they are doing is inherently difficult,
and - The job is done poorly.
- Compounding the Reliability Issue
- Developers and users exhibit overconfidence in
the system. - Reused system software may not work in different
environments.
Q Identify the elements needed as an incentive
to increase reliability and safety.
15Increasing Reliability and Safety
- Professional Techniques (Code of Ethics)
- Follow good software practices.
- Exhibit professional responsibility at all levels
of development and use. - Construct well-designed user interfaces.
- Take human factors into account.
- Include built-in redundancy.
- Incorporate self-checking where appropriate.
- Follow good testing principals and techniques.
- Managers Guide!!
Q What human interface features should be
considered for ordinary business applications?
16Increasing Reliability and Safety
- Law and Regulation
- Criminal and Civil Penalties
- to recover loss from faulty or unsafe systems.
- Liability and Civil Laws
- to provide incentives to produce reliable and
safe systems. - Warranties
- to guarantee a certain level of quality.
- Federal or State Regulations
- to protect the public.
- IEEE, ACM, HKIE Code
Warranties, Insurance, Damage
17Increasing Reliability and Safety
- Law and Regulation (contd)
- Database Accuracy Enforcement
- to protect the public from inaccurate information
maintained by private companies and government. - Authority to use/enter/verify/retrieve data
- Mandatory Licensing of Software Developers
- to ensure proper training, competency, and
continuing education. - Graded Contractors
Q How can consumers protect themselves from
faulty software?
18Perspectives on Failures, Dependence, Risk, and
Progress
- Failures (Unavoidable)
- What are acceptable rates of failures?
- How accurate should software be?
- Dependence (Unavoidable)
- How dependent on computer systems are our
ordinary activities? - How useful are computer systems to our ordinary
activities? - Risk and Progress (Minimize Research)
- How do new technologies become safer?
- Can progress in software safety keep up with the
pace of change in computer technology? (Software
engineering/tools follow hardware progress?!)
19Computer Models
- Points to Consider
- Models are simplifications of either physical or
intangible systems. - Those who design and develop models must be
honest and accurate with results. - Computer professionals and the general public
must be able to evaluate the claims of the
developers.
Q What problems in your community have been or
could be studied with computer models?
20Complete Methodology
- We define a simplified capacity planning
methodology that covers - 1. Business level
- 2. Customer behavior level
- 3. Resource level
21Queueing Theory -- to predict performance
22C. Workload Forecasting- Example
- Online auto-buying is watching traffic grow very
fast - Plot the number of weekly sessions
- Forecast the increase in traffic to the site in
the future - Use regression techniques in Microsoft Excel to
add a trend line - A good fit to the curve is the exponential fit
- Number of sessions on week 11
23Computer Models
- Evaluating Models
- Why Models Might Not Be Accurate
- Developers have incomplete knowledge of the
system being modeled. - Data might be incomplete or inaccurate.
- Power of the computer might be inadequate.
- Variables are difficult to numerically quantify.
- Political and economic motivation to distort
results. (Election!?)
Q For each item above, give an example of a
model or simulation that was inaccurate.