Title: Lets Get Real: Disaster Recovery and Business Continuity in Public Safety
1Lets Get Real Disaster Recovery and Business
Continuity in Public Safety
- Is Yours Just a Paper Plan or a Real Way to
Prepare and Respond to Incidents and Disasters?
2Presentation Overview
- Key DR/BC Concepts and Issues
- Report card and dashboard
- Scenarios
- Requirements What has to operational by when
for work to be done by how many at what locations
serving what customers who are where? - Facilities
- People
- Systems
- Integration
- Coordination
- Daily readiness and simulated escalations
- Testing and independent verification and
validation - Implementation and triage
- Recovery, discovery, and improvements
- Player Scorecard Who Is In the Game and Why?
- DR/ BC Framework
- Action Steps to a Real Plan
- First steps
- Critical functions
- Funding and leveraging scarce resources
3Key DR/BC Concepts and Issues
4The Report Card and Dashboard
- All aspects of the plan, test, and implementation
should be scored simply (Red, Yellow, and Green) - Key indicators of planning and readiness need a
dashboard to enable assessment and action - Score or status
- Trend
- Key issue
5Public Safety Scenarios
- Public safety entities have a more difficult
challenge - Your IT DR/BC plan is intertwined with risk
scenarios - You may be affected by the risks of a given
scenario and your IT plan must address those
risks appropriately to maintain operations - You also have a role in response to the scenario
so the events will affect your operational
requirements
6Scenarios Overview
- Threat driven geographic circles of impact
- Kinds of threats and events
- Responsibility
- What will you do, what is shared, what do others
have to do for themselves - Tolerance for risk and uncertainty
- Lesson learned if you have a well known and
documented local risk - Have a real plan or get ready for a career change
7Source IBM
8Scenarios
- Identify Possible and Likely Natural Disasters
and Environmental Conditions By Kind and Duration
of Effects - Tornado
- Hurricane
- Tsunami
- Flood
- Snowstorm
- Drought
- Earthquake
9Scenarios
- Identify Possible and Likely Natural Disasters
and Environmental Conditions By Kind and Duration
of Effects - Electrical storms
- Fire
- Subsidence and landslides
- Freezing Conditions
10Scenarios
- Identify Possible and Likely Natural Disasters
and Environmental Conditions By Kind and Duration
of Effects - Contamination, Toxic releases and environmental
hazards - Epidemic
- Pandemic
- Animal or crop disease outbreak
11Scenarios
- Organized and/or Deliberate Disruption
- Act of terrorism
- WMD
- Acute and short lived (bomb)
- Acute and long lived (dirty bomb)
- Chronic
- Long term (contaminants and biohazards)
- Permanent (radioactivity, etc.)
- WLD (suicide bombers, car bombs, utility
sabotage) - Bioterrorism or genetically modified or inorganic
organisms - Direct contact
- Infectious
- Contact
- Airborne
12Scenarios
- Organized and/or Deliberate Disruption
- Act of Sabotage
- Product or food tampering
- Act of war
- Theft
- Arson
- Labor Disputes / Industrial Action
13Scenarios
- Loss of Utilities and Services
- Electrical power failure
- Loss of gas supply
- Loss of water supply
- Petroleum and oil shortage
- Raw materials
- Refined materials
- Communications services breakdown
- Loss of drainage / waste removal and trash pickup
14Scenarios
- Equipment or System Failure
- Internal power failure
- HVAC failure
- Equipment failure (excluding IT hardware)
15Scenarios
- Serious Information Security Incidents
- Cyber crime
- Malware
- Zombie attacks
- Denial of service
- Loss or alteration of records or data
- Disclosure of sensitive information
16Scenarios
- IT system failure (local or hosted)
- Hardware
- Software
- Commercial application
- Locally developed application
- Data
- Communications
17Scenarios
- Other Emergency Situations
- Workplace violence
- Public transportation disruption
- Neighborhood hazard
- Health and safety issues
18Scenarios
- Multiple and compound hazards and events
- Purposeful
- Coincidental
- Causally connected
- Interrelated
19IT Requirements
- What systems need to function
- How fast
- Maximum and optimum time frame for each system or
function to be restored - How well
- Sometimes minimal functionality is sufficient
20IT Requirements
- Where will it be used and by whom and will the
communications infrastructure support it? - Employees
- Users or beneficiaries
- By what priority will systems be restored
- The priority will be modified by what
contingencies - E.g. a long term total evacuation changes the
operational needs for criminal justice systems
and personnel
21Facilities
- Hot, warm, cold
- Mirrored, recoverable, reload-able
- Properly located
- EOC
- Non-EOC
- Operational
- IT facilities
- For user interaction with IT systems
22Facilities
- New kinds of mutual aid and sister
city/county/state arrangements - Work with friends, colleagues, associations, and
vendors - To match you with a comparable entities that are
located outside the various geographic threat
circles - Who can mirror your IT operations (hardware,
software, operating systems, and culture)
23People
- The right numbers, skills, location, redundancy,
etc. - Skills and abilities inventory
- Employees
- Contractors
- Vendors
- Mutual aid and the cavalry
24People
- Force in depthwho is the backup to the backup to
the backup? - Consider the actual health and physical abilities
and disabilities of a person when assigning tasks
for a disaster scenario - The disaster is not the time to find out the
electrician in the hazmat suit has a heart
condition - What family and personal duties may interfere
with performing official duties (e.g. save your
own kids or save a stranger)?
25Systems
- Daily operational
- Interdependent systems
- Emergency only
- Identity security and access management for
physical and logical security - Follow FIPS 201 for federal/state/local
interoperability
26Integration
- With whom should you work closely?
- Identify integration issues between
- Internal systems and public safety entities
- Other governmental systems
- Related actors
- Non-governmental systems and processes
- Example 911 and 311or its equivalent
- Normally separate but related
- Emergencies blur the line
- Co-location, cross training, and system
integration
27Coordination
- Within organization
- Within unit of government
- Across units of government
- Across levels of government
- Across public and private boundaries
28Daily Readiness and Simulated Escalations
- A disaster a day (What, thats not normal?)
- Realistic scenarios
- Captured lessons
- Learning and actually responding to lessons
learned within risk framework - A quality and security framework for daily
operations has substantial overlap with DR/BC
29Security Capabilities Models
- Like similar capability models from the Carnegie
Mellon SEI, SCMM models brings benefits - Helps close security holes
- Serves as a foundation for growth
- Guides security leadership
- Is evolutionary, not chaotic
- Supports point solutions
KPMG SCMM Model
30Capability Maturity
- Like the SCI CMM models, the KPMG Security
Capability Model has five levels of maturity
31Testing and Independent Verification and
Validation
- Does the planned response or action step actually
work? - Who verifies that it does?
- What do you do if it fails the test?
32Implementation and Triage
- Someone better be in charge
- Dispute resolution processes
- Who will be your Sensibility and Sanity Checker
(off site, not affected by the disaster, and
actually getting enough sleep to make sound
decisions)? - Baton Rouge example with Mayor Holden
33Recovery, Discovery, and Improvements
- What will the new normal be and when will it
happen - Learn from history, both recent and long past
- Document while the event occurs if at all
possible (make it someones job) or soon after
before memories fade
34Player Scorecard
- Who Is In the Game and Why
35Overlapping and Inter-Related Responsibilities
Disaster Preparedness and Recovery and Business
Continuity
Physical Security
Quality Assurance Methodologies
Cyber Security
Public Safety
36The Usual Suspects in Public Safety
- Police
- Fire
- Other sworn officers (transit, game, building or
branch based, etc.) - National Guard
- Public Health
- Public Works
- Transportation
- Environmental Protection
37The Usual Suspects in Emergency Management
- Federal, state and local emergency management
entities - National Guard
- NOAA, NWS, NSSL, other National Laboratories,
- Corps of Engineers
38IT Entities
- CIO, CTO, and Enterprise IT Shops
- Distributed IT Departments and leadership
- Government IT contractors
- DR/BC specific entities
- Applications developers and software
- Hardware
- Service providers (ASP, MSP, call centers, etc.
- Communications providers
39Policy Makers
- Executive, legislative, and judicial
- Those who hold the seat and those who actually
make the decisions - Go below the top level to ensure clarity,
alignment, and redundancy - EOC designees
- Emergency authorizers
40Non-Governmental Organizations
- Media
- Broadcast and satellite
- Emergency Broadcast System Members
- Print
- New media
- The Web
- Government site mangers
- Commercial site managers
- Citizens and bloggers
- Self-organizing communities (e.g. Craigs List)
41Non-Governmental Organizations
- Charities
- Businesses and business associations
- Community organizations
- Vital private services (hospitals, nursing homes,
etc. )
42A DR/BC Framework
43Business Operations and Technology
- Create a matrix, not a linear or organizational
view - Strategy
- Organization
- Processes
- Applications and data
- Technology
- Facilities
44Source IBM
45Action Steps to a Real Plan
46First Steps
47First Steps
- Leadership clarity, alignment, and commitment
- Authority or consensus?
- Stakeholders roles and responsibilities
- Be clear about risk tolerance
- Applications and IT assets inventory
- If needed, dust off and update your Y2K work
- Good data on plan status, readiness, test
results, response, and compliance
48First Steps
- Make a friend in accountingactuarially accurate
threat scenarios are more likely to be funded as
risk and cost can be properly balanced - Review existing plan or make a plan
- Borrow or buy a template
- Review peer plans and conduct site visits
- Communicate until it hurts
49Critical Functions
50Nail Down Your Critical Functions
- Law and order essentials (people, mobility,
tools, survival basics, etc.) - Communications
- Personnel management (policies, scheduling,
notification trees and systems, counseling, etc.) - Data and the connections to data and people
- Transactional systems
51Nail Down Your Critical Functions
- Rescue and response
- Pipeline to the health care system
- Building/location/hazmat information for fire and
first responders - Justice processing and incarceration
- Dispatch
52Nail Down Your Critical Functions
- Records
- Mobility
- Devices and local storage if communications are
intermittent or fail (e.g. mobile maps and
databases) - Know what you can actually cover (and what you
are just waiving your hands at and hoping it
either works or is never needed)
53Funding and Leverage
54Funding and Leverage
- Work within your risk/threat/cost/benefit matrix
and follow your own rules - How serious are you about being prepared?
55Funding and Leverage
- Stop building single purpose infrastructures and
reuse what you have - Ask not, what an infrastructure can do for you,
but what it can do for your taxpayers - Use shared services
- Follow standards or help create them if lacking
56Funding and Leverage
- Determine what pre-existing, unmet needs can be
addressed by a new investment - Determine whether existing public safety or
enterprise systems will do the job and if you can
use them - Invest wisely
- Vendors over inventors
- COTS over customization
- Web services over hard coding
57Think Out of the Box
58Think Third World
- Hand crank your computers
- Bike generators
- Solar and wind power
- Portable water purifiers
- Emergency shelter
- Runners and mountain bikes
- Hand tools
59Think New World
- Internet Protocol (IP) everything
- Bridge between radio, wireless data/WI-FI and use
each as IP conduits as needed - Gigs of portable flash memory
- Satellite data and telephony
60Think New World
- Instant Message
- Text and mobile email
- Cell On Wheels/Boat/Balloon
- Negotiate/legislate priority and bumping rights
in telecommunications provisioning
61Integrate With the Big DR/BC Picture
62The Big Picture
- Consult EM before, during, and after
- Once essential public safety systems have a DR/BC
IT and overall plan it can be incorporated into
the overall EM plan for the jurisdiction - Tie it all together in formal and informal
agreements - Create a focal point such as your EOC
63EOC Basics
- Not located in a hazard area (floodway)
- 500 square feet minimum floor space
- Communications section adjacent to EOC
- Three methods of communications with state EMA
and local responders - UPS and generator systems located above flood
level - Sleeping space for identified staff
- Kitchen space/food or meal contract
- New construction to International Building Code
Source Alabama EMD
64Conclusion Essential Public Safety Systems and
Organizations Must Be Disaster Resistant,
Flexible, Diversified, and Redundant(Or We Are
All In Big Trouble)
- Contact Information
- Richard J. H. Varn
- Center for Digital Government
- rjmvarn_at_msn.com
65Model Plan Outline
- What follows is a private sector based, but
broadly applicable tool that sells for 199 - To buy a copy of the business continuity plan
generator see http//www.eon-commerce.com/rusecure
/bcp.asp
66Model Plan Outline
- Business Continuity - Preparing the Plan
- Initiating the BCP Project
- Project Initiation Activities
- BC 010101 Review of Existing BCP (if available)
67Model Plan Outline
- BC 010102 Benefits of Developing a BCP
- BC 010103 BCP Policy Statement
- BC 010104 Preliminary BCP Project Budget
- BC 010105 Procedure for Approving BCP Content
68Model Plan Outline
- BC 010106 Communication on BCP Project to All
Employees - Project Organization
- BC 010201 Terms of Reference for BCP Project
Manager - BC 010202 Appoint BCP Project Manager and Deputy
- BC 010203 Select and Notify BCP Project Team
69Model Plan Outline
- BC 010204 Initial BCP Project Meeting
- BC 010205 Project Objectives and Deliverables
- BC 010206 Project Milestones
- BC 010207 Project Reporting Requirements and
Frequency - BC 010208 Required Documents and Information
70Model Plan Outline
- Assessing Business Risk and Impact of Potential
Emergencies - Emergency Incident Assessment
- BC 020101 Environmental Disasters
- BC 020102 Organized and / or Deliberate Disruption
71Model Plan Outline
- BC 020103 Loss of Utilities and Services
- BC 020104 Equipment or System Failure
- BC 020105 Serious Information Security Incidents
- BC 020106 Other Emergency Situations
- Business Risk Assessment
72Model Plan Outline
- BC 020201 Key Business Processes
- BC 020202 Establish Time-Bands for Business
Service Interruption Measurement - BC 020203 Financial and Operational Impact
- IT and Communications
73Model Plan Outline
- BC 020301 Specifications of IT and Communication
Systems and Business Dependencies - BC 020302 Key IT, Communications and Information
Processing Systems - BC 020303 Key IT Personnel and Emergency Contact
Information - BC 020304 Key IT and Communications Suppliers and
Maintenance Engineers - BC 020305 Existing IT Recovery Procedures
74Model Plan Outline
- Existing Emergency Procedures
- BC 020401 Summary of Existing Procedures for
Handling Emergency Situations - BC 020402 Key Personnel Responsible for Handling
Existing Emergency Procedures - BC 020403 External Emergency Services and Contact
Numbers
75Model Plan Outline
- BC 020500 Premises Issues
- BC 020501 Responsibility and Authority for
Building Repairs - BC 020502 Back-up Power Arrangements
- Preparing for a Possible Emergency
76Model Plan Outline
- Back-up and Recovery Strategies
- BC 030101 Alternative Business Process Handling
Strategy - BC 030102 IT Systems Back-Up and Recovery
Strategy - BC 030103 Premises and Essential Equipment
Back-up and Recovery Strategy
77Model Plan Outline
- BC 030104 Customer Service Back-up and Recovery
Strategy - BC 030105 Administration and Operations Back-up
and Recovery Strategy - BC 030106 Information and Documentation Back-up
and Recovery Strategy - BC 030107 Insurance Coverage
- Key BCP Personnel and Supplies
78Model Plan Outline
- BC 030201 Functional Organization Chart
- BC 030202 BCP Project Co-coordinator and Deputy
for Each Functional Area - BC 030203 Key Personnel and Emergency Contact
Information - BC 030204 Key Suppliers and Vendors and Emergency
Contact Information - BC 030205 Manpower Recovery Strategy
79Model Plan Outline
- BC 030206 Establishing the Disaster Recovery Team
- BC 030207 Establishing the Business Recovery Team
- Key Documents and Procedures
- BC 030301 Documents and Records Vital to the
Business Process - BC 030302 Off-site Storage
80Model Plan Outline
- BC 030303 Emergency Stationery and Office
Supplies - BC 030304 Media Handling Procedures
- BC 030305 Emergency Authorization Procedures
- BC 030306 Prepare Budget for Back-up and Recovery
Phase
81Model Plan Outline
- Disaster Recovery Phase
- Planning for Handling the Emergency
- BC 040101 Identification of Potential Disaster
Status - BC 040102 Involvement of Emergency Services
- BC 040103 Assessing Potential Business Impact of
the Emergency
82Model Plan Outline
- BC 040104 Project Management Activities
- Notification and Reporting During Recovery Phase
- BC 040201 Mobilizing the Recovery Team
- BC 040202 Notification to Management and Key
Employees
83Model Plan Outline
- BC 040203 Handling Personnel Families
Notification - BC 040204 Handling Media during the Disaster
Recovery Phase - BC 040205 Maintaining Event Log during Disaster
Recovery Phase - BC 040206 Disaster Recovery Phase Report
- Business Recovery Phase
84Model Plan Outline
- Managing the Business Recovery Phase
- BC 050101 Mobilizing the Business Recovery Team
- BC 050102 Assessing Extent of Damage and Business
Impact - BC 050103 Preparing Specific Recovery Plan
85Model Plan Outline
- BC 050104 Monitoring Progress
- BC 050105 Keeping Everyone Informed
- BC 050106 Handing Business Operations Back to
Regular Management - BC 050107 Preparing Business Recovery Phase
Report - Business Recovery Activities
86Model Plan Outline
- BC 050201 Power and Other Utilities
- BC 050202 Premises, Fixtures and Furniture
(Facilities Recovery Management) - BC 050203 Communication Systems
- BC 050204 IT Systems (Hardware and Software)
87Model Plan Outline
- BC 050205 Production Equipment
- BC 050206 Other Equipment
- BC 050207 Warehouse and Stock
- BC 050208 Trading, Sales and Customer Service
88Model Plan Outline
- BC 050209 Human Resources
- BC 050210 Information and Documentation
- BC 050211 Office Supplies
- BC 050212 Operations and Administration (Support
Services)
89Model Plan Outline
- Testing the Business Recovery Process
- Planning the Tests
- Develop Objectives and Scope of Tests
- Setting the Test Environment
- Environmental Disasters
90Model Plan Outline
- Organized and / or deliberate disruption
- Loss of Utilities and Services
- Equipment or System Failure
- Serious Information Security Incidents
- Other Emergency Situations
- Prepare Test Data
- Identify Who is to Conduct the Tests
91Model Plan Outline
- Identify Who is to Control and Monitor the Tests
- Prepare Feedback Questionnaires
- Prepare Budget for Testing Phase
- Training Core Testing Team for each Business Unit
92Model Plan Outline
- Conducting the Tests
- Test each part of the Business Recovery Process
- Test Accuracy of Employee and Vendor Emergency
Contact Numbers - Assess Test Results
- Training Staff in the Business Recovery Process
93Model Plan Outline
- Managing the Training Process
- Develop Objectives and Scope of Training
- Training Needs Assessment
- Training Materials Development Schedule
- Prepare Training Schedule
- Communication to Staff
- Prepare Budget for Training Phase
- Assessing the Training
94Model Plan Outline
- Feedback Questionnaires
- Assess Feedback
- Keeping the Plan Up-to-date
- Maintaining the BCP
95Model Plan Outline
- Change Controls for Updating the Plan
- Responsibilities for Maintenance of Each Part of
the Plan - Test All Changes to Plan
- Advise Person Responsible for BCP Training