Seeking Your Contingency Plan: Are You HOT, COLD, or WARM?

1
Seeking Your Contingency Plan Are YouHOT, COLD,
or WARM?
NCHICA AMC Security and Privacy Conference
September 26-28, 2005 Panel Members Bill
Rider, Panel Leader, Mgr. Information Security
and Disaster Recovery, Johns Hopkins Health
System and University Anne Marie Turner,
Information Systems Risk Mgr., University of
Rochester Medical Center
2
Agenda

• Overview of two Institutions
• Demographics
• IT Divisions
• DR Theories Baseline Strategies for Determining
  HOT, COLD, or WARM
• Institution DR Practices Comparisons
• Mainframe, Open Systems
• RTO and RPO
• Underlying factors internal vs. external
• Testing
• Maintenance
• Are You HOT, COLD, or WARM
• RPO and RTO Considerations
• Results
• Questions to Consider
• Overview Healthcare BCP/DRP Benchmarks
• Questions Answers

3
Johns Hopkins Health System University
Acute Care Beds Licensed 1,467 Opened 1,327
Of Occupancy 77.0 Discharges 77,962 Days 38
1,601 Outpatient Encounters 1,929,660 Emergency
Visits 208,682 Operating Room Cases 68,070 Inpa
tient 27,968 Outpatient 40,102 (Johns Hopkins
Hospital, Bayview Medical Center, Howard County
General Hospital, Johns Hopkins Clinical
Physicians, JHU School of Medicine)
4
1 IBM Mainframe Processor 406 MIPS (DR is
206) 8 Meg Memory (DR is 4) 1.3 Terabytes
Storage (DR is 1.3 Tb) Multiple DEC and AIX
Midrange Platforms Over 450 Distributed
Servers Multiple I.T. locations within 15 mile
radius Organization segregates I.T.
Operations Network Services Enterprise
Services Applications Support
5
Number of Beds Strong 750 beds, Highland
2751025 Number of In-Patient Discharges per
year 55K (SMH and HH) Number of Out-Patient
Discharges per year Outpatient Visits (not
discharges) SMH OP 360K UR Medical Faculty
Group 660K SMH Emergency Dept 90K SMH Lab
Specimens 500K (these are not actual patient
visits but do result in registrations) Eastman
Dental Center                          50K HH
Emergency Dept -                        25K HH OP
-                        75K Total
1.7M
Strong Memorial Hospital, Highland Hospital,
School of Medicine Dentistry, School of
Nursing, Research, Mt. Hope Family Center, UR
Medical Faculty Group, University Health Service,
Primary Care Network, Eastman Dental Center,
Long Term Care, Visiting Nurse Service
6
University of Rochester Medical Center
Information Systems Division

• IBM Mainframe Processor
• Multiple AIX Midrange Platforms
• Over 280 Open System Medical Center Servers in
  Data Center
• Information Systems Division

• Enterprise Operations
• Network Services
• Enterprise LAN
• Security
• Help Desk Support / Desktop Support

• Systems Interface
• Research Support
• Shared Services Organization
• Medical Informatics
• Clinical Systems Applications
• Project Management

7
Are You HOT, COLD, or WARM?
RTO RPO
RECOVERY TIME OBJECTIVE (RTO)
The period of time in which systems,
applications, or I.S. functions must be recovered
after an outage. RTO's are often used as the
basis for the development of recovery strategies,
and as a determinant as to whether or not to
implement the recovery strategies during a
disaster situation.
RECOVERY POINT OBJECTIVE (RPO)
The point in time to which systems and data must
be restored after an outage. RPO's are often used
as the basis for the development of backup
strategies, and as a determinant of the amount of
data that may need to be recreated after the
systems or functions have been recovered.
I.T. RTO and RPO Need To Be Negotiated With The
Business Units.In Order To Provide A Level Set
For User Expectations Regarding The Current
Version Of The Data, The Timely Restoration Of
The Systems, And For The Business Units To
Accurately Develop Their Business Contingency
Plans
8
The Bottom Line for Recovery
Are You HOT, COLD, or WARM?
9
Dispersed Processing Open Central Systems
Are You HOT, COLD, or WARM?
Print Server
Application Server
EPR Database Mainframe
Pharmacy Database
CITRIX Server
Database Server
PID Mainframe
Clinweb
Clindata
Outpatient Reg
Sybase Server
Inpatient Reg Mainframe
Clinweb
Outpatient Reg
Clindata
Sybase Server
Outpatient Reg
10
Recovery Strategy Needs to Recognize Diversity
Are You HOT, COLD, or WARM?
Print Server
Application Server
Pharmacy Database
EPR Database Mainframe
CITRIX Server
Database Server
PID Mainframe
Clinweb
Clindata
Outpatient Reg
Sybase Server
Internal Solution
External Solution
Inpatient Reg Mainframe
Clinweb
Outpatient Reg
Clindata
Sybase Server
Outpatient Reg
Open Systems DR Model
Centralized DR Model
11
Different Environments, Tactics, Controls
Are You HOT, COLD, or WARM?

• Open Systems DR Model
• Data Replication To Local Storage
• Failover and/or Quick Recovery
• Local Connections For High Volume
• Local AND Remote Recovery

• Centralized DR Model
• Traditional Offsite Storage
• Hotsite Location Approx 100 Miles From Primary
  Site
• T1 Connection Between Hotsite and Local Internal
  Solution

12
Mainframe and Open Systems

• URMC
• Multiple Platforms
• Mainframe/Midrange/Servers
• Centralized Locations
• Primary Data Center/Backup Data Center/Hospitals
• Strong Health Geography
• SMH, HH, and Data Centers within 5 miles
• Outlying Locations
• Urban, Suburban, and Rural

• Hopkins
• Multiple Platforms
• Mainframe/Midrange/Servers
• Dispersed Locations
• Hospital/University/Clinical Practices/Satellite
  JHU Campuses
• Campus Geography
• JHH and JHU within 10 miles
• Outlying Locations
• Urban and Rural

13
RPO, RTO, Outsourced vs. In-House

• URMC
• Criticality / BIA Driven
• Patient Safety
• Patient Quality of Care
• Workforce Productivity
• Financial
• Legal/Regulatory
• Reputation
• Service Level Agreements
• Educational
• Research Programs
• Mission Critical
• Dedicated DR systems data in backup data center
• Offsite media
• Business Critical
• Contract for shipment
• Offsite Media

• Hopkins
• Platform Size Driven
• Mainframe and Midranges at Hotsite
• Servers In-House

14
Underlying Factors Internal vs. Outsourcing

• Hopkins
• RTO
• RPO
• Cost Of Recovery
• Impact Of Recovery

• URMC
• Mission Critical
• RTO
• 2 to 24 hours
• RPO
• 0 to minimal data loss
• BIA
• Downtime Impact vs. Time to Recover Costs
• Dedicated DR Hardware
• DR Contract for Shipment

15
Testing HOT, WARM, COLD, Internal vs.
Outsourcing

• Hopkins
• Combination
• Relocation to hotsite 2 per year
• Servers tested throughout the year validated
  during hotsite tests
• Customers test locally
• High volume applications (email, images, etc.)
  test locally

• URMC
• Mission Critical Systems
• 2 per year
• Business Critical Systems
• 1 per year
• Systems tested throughout the year
• Hardware and Software Upgrades, project
  implementations, etc.
• Department Downtime Procedures Annual Testing

16
Maintenance HOT, WARM, COLD, Internal vs.
Outsourcing

• URMC
• Disaster Recovery System and Program Maintenance
  validated through
• In-House DR system testing, IT and End-User
  testing
• Technical Documentation Audits
• Skill Set Evaluations
• Internal and External Auditors
• EOC and ISD Command Center Team Exercises
• Change Management
• Project Management
• Incident Debriefing Sessions

• Hopkins
• All maintenance issues validated during hotsite
  tests
• In-House External Contracts
• Documentation
• Contact Lists
• Skill Sets
• Notification, Escalation, Declaration Procedures

17
Are You HOT, COLD, or WARM?
RTO RPO Considerations
Negotiate The Service Level Agreement Between
I.T. And Business Operations

• Use Both The I.T. And Business RTO RPO As The
  Basis
• Disaster Recovery Plan Test Results Quantify
  Timelines
• Business Contingency Plan Exercises Qualify
  Impact
• I.T. Capabilities Improve Timelines But At A
  Cost
• Business Contingencies Reduce Impact - But
  Require I.T. Capabilities

• Criticality Rankings
• Systems Recovery Sequencing
• Business Process Prioritization
• I.T. and Business Process Timelines
• Negotiated RTO and RPO

18
Are You HOT, COLD, or WARM?
Results
I.T.

• Better Understands the Customers Issues and
  Requirements
• Obtains A Clearly Documented Set of Customer
  Expectations for DRPs
• Clarify and Justify Budget Forecasts
• Establishes Specific Test Objectives
• Ensure Active Customer Involvement in Testing
  Recovery Processes

Business Units

• Better Understands the Role Of I.T. in the
  Contingency Process
• Obtains a Set of Parameters from which to Develop
  Their BCPs
• Workaround Procedures During Downtime
• Procedures For Capturing Lost Transactions From
  Downtime and During Recovery
• Restoration of Normal Environments

Everyone works towards a common interest, that of
ensuring that the business processes of the
organization, its mission, goals, and
objectives..and possibly the community at
largeare protected
19
Questions To Consider
Are You HOT, COLD, or WARM?

• Was the original disaster recovery initiative
  driven by I.T., business units, or Sr Management
  ?
• What are Sr. Managements expectations with
  respect to continuity of service ?
• Has a business impact analysis been done on some
  or all of the business units ?
• Quantified Impact
• Quantified Cost of DRP vs. Impact of Risk
• Acceptable Downtime Criteria (services,
  workstations, etc.)
• What discussions have taken place between I.T.
  and critical business units ?
• State of DRP
• State of BCP
• Quantified RTOs and RPOs
• Systems Development Life Cycles

20
Questions To Consider
Are You HOT, COLD, or WARM?

• What are the business units expectation with
  respect to current I.T. RTOs and RTOs ?
• Are they driven by I.T. technologies or business
  requirements ?
• Are there current SLAs ?
• Service Center
• Problem/Change Control
• Network Outage Response Time
• Are regulatory compliance, industry
  certification, or audit issues creating more
  compelling reasons for addressing DRP and BCP ?

21
Benchmark Survey Results

• 40 of Members responded to Benchmark Survey
• 58 of Responders have Steering Committees

83 Feel they were prepared for the HIPAA
deadline
75 Have DR Plans for Critical Systems
67 Use Planning Software
58 Have performed a BIA in last 12 months
22
Benchmark Survey Results
Disaster Recovery Plan Testing
25 Test Every Six Months
25 Test Annually
50 Currently Developing A Testing Strategy
23
Benchmark Survey Results
Percentage of IT Budget Spent for Disaster
Recovery

lt1
1
3
5
24
Benchmark Survey Results
Organizations with Recovery Solutions
35
30
25
Own
20
Co-Lo
15
Contract
No Response
10
33
33
26
5
8
0
Percentage with Recovery Solutions
25
Benchmark Survey Results
Minimum Recovery Timeframes for Critical Systems
35
30
33
25
25
20
15
17
17
10
8
5
0
Under 12 Hrs.
12-24 Hrs.
24 Hrs.
48-72 Hrs.
Not specified
26

• Business Continuity Planning Workgroup for
  Healthcare Organizations

27
BCPWHO Information

• 25 Academic Medical Centers and Healthcare
  Organizations
• Charter and Bylaws in Progress
• Website Coming
• Dedicated BCPWHO Chat Room for Questions, Issues,
  Discussion Coming
• Opportunities National Meetings and Regional
  Workshops
• DRP/BCP Logistical and Vendor Sponsorships
  Establishing and Growing
• Vendor Resources Communication Awareness

28
Interest Level Survey Results

• 60 Surveys Sent Out
• 22 Surveys Received Back
• 21 Clinical
• 13 Academic
• 10 Research
• 3 Disaster Recovery Planning Only
• 16 Combined Disaster Recovery and Business
  Continuity Planning
• 3 Were Other (Insurance)

29
Interest Level Survey Results

• Majority Of Responders Agree
• Formal Membership
• Membership Dues
• Charter With Bylaws
• Formal Board With Rotating Members
• Some type of regional users groups
• Tie annual meeting to conference venue
• DRJ CPM CI Strohl
• Virtual Workshops (Bi-monthly or Quarterly)
• Internet Chat Rooms With Participation Limited To
  Members Only

30
Interest Level Survey Results
And Who Are We All Reporting To ?
I.S. Security Director Director of Technical
Services VP Information Services Director,
Facilities Safety Programs Chief Security
Officer Assoc. Vice Chancellor Health Affairs,
Director Of Informatics Center V.P Information
Services CIO Assoc. Vice Chancellor Health
Affairs, Director Of Informatics Center Sr. Vice
President, Quality Care and Chief Medical
Officer Director of Engineering Services,
Enterprise Technology Services Executive
Associate Dean for Faculty Affairs Chief
Technology Officer Senior Director SR VP Legal
Affairs and HR
31
(No Transcript)
32
Business Continuity Planning Workgroup for
Healthcare Organizations Spring 2005 Disaster
Recovery Benchmark Survey   To BCPWHO Members
Please complete the healthcare disaster recovery
benchmark survey below and return the survey back
to Kathy Lee Patterson, pattersonkl_at_email.chop.edu
, of the BCPWHO Planning Committee by April 8,
2005. The information provided will be held in
strictest confidence, with published results sent
only to BCPWHO members. Healthcare
establishments names will not be published, only
the statistical results. General results of the
survey will be presented at the May Continuity
Insights Conference in New Orleans during the
BCPWHO session (C9, Healthcare/AMC DRP-BCP
Consortium A Whole Different Challenge). Thank
you for your participation. (While this survey is
generally I.T. DRP focused, future surveys will
be developed to address other areas of Business
Continuity Planning)
Hyperlink to benchmark survey
33
Seeking Your Contingency Plan Are YouHOT,
COLD, or WARM?

• Questions?

34
Engagement Process

• Facilitators
• Stimulate audience discussion with
• requests for questions and comments ,
• Pre-designed questions and instant polls that
  are designed to assess how the audience of AMC
  peers sees the topic and to start further
  questions and comments from the audience.
• Collect the results for reporting in the track
  reporting part of each plenary session and a
  planned GASP (Guidelines for AMCs on Security and
  Privacy) update.
• Audience (and panelists) Respond to the
  questions, comments, provide your own.

35
Instant Poll Rules

• Facilitators role
• Require audience members and panelists to shut
  their eyes (to promote more honest voting)
• Ask for a show of hands for each item to be voted
  on.
• Audience role
• Vote as you see fit.
• Voting is anonymous.
• Follow-up questions may ask voters to describe
  why they voted as they did, if they are
  comfortable doing so.
• Anonymity
• For some issues, you may wish to keep your vote
  private the eyes-shut voting rule is the main
  rule that assures this.
• Also, the facilitators will take only the notes
  that you see on the screen and will not identify
  you by name or institution unless you explicitly
  say that you are willing to be so identified.

36
Instant Poll Rules

• Facilitators role
• Ask audience members and panelist to shut their
  eyes (to promote more honest voting)
• Ask for a show of hands for each item to be voted
  on.
• Audience role
• Vote as you see fit.
• Voting is anonymous.
• Follow-up questions may ask voters to describe
  why they voted as they did, if they are
  comfortable doing so.

37
Conference Benchmarks -Disaster Recovery
Planning

• My AMC tests its disaster recovery plan every six
  months _____
• My AMC tests its disaster recovery plan every
  year _____
• My AMC has not tested its disaster recovery plan
  ____
• My AMC is still developing its disaster recovery
  plan ____
• Active test plans ___

38
Conference Benchmarks - Minimum Recovery
Timeframes for Critical Systems Based on Some
level of BIA

• Under 12 hours ____
• 12-24 hours ____
• 24 hours ___
• 48 - 72 hours ____
• Not specified _____

39
Contingency Planning - Discussion

• Should there be uniform standards?
• Should the government help pay?
• Is contingency planning a public health issue?
• What can AMCs be doing better?

40
Contingency Planning

• What aspects of contingency planning have not
  been adequately addressed today?
• What have you heard today that you want to pursue
  further?
• Any surprises in what you heard today?

41
What follow-up activities would be helpful to
AMCs in dealing with this topic?
42
Engagement Quality Instant Poll

• This session did a good job of engaging the
  panelists and the audience on the topic.
• 1 - Strongly Disagree ___
• 2 - Disagree ___
• 3 - Neither agree not disagree ___
• 4 Agree ____
• 5 - Strongly agree ____