Business Recovery Planning

1
Business Recovery Planning

• Past, Present, and Future

2
Presentation Overview

• Review evolution of business recovery planning at
  PERA
• Review business case for building a recovery
  facility
• Discuss facility location and design
• Discuss current recovery capabilities
• Discuss future direction for recovery and high
  availability

3
Business Continuity Planning Components

• Disaster Recovery Plans for restoring computing
  and telecomm services
• Business Resumption Workaround procedures for
  business processes until processes are recovered
• Business Recovery Plans for complete recovery
  of business processes, people, workspace, non-IT
  equipment, and facilities
• Contingency Planning Planning responses to
  external events
• Crisis Management Overall coordination of
  response to minimize damage to profitability,
  reputation, or ability to operate

4
PERAs Recovery History

• Contracted with Comdisco for recovery services in
  1990
• Initially focused on Information Systems
• Initial contract covered select midrange
  platforms only
• Contract expanded coverage over time to include
• Imaging System
• Core Network Servers
• Work area recovery with 50 PCs
• PBX with ACD

5
Recovery History (cont.)

• Disaster recovery tests were completed in New
  York, Chicago, and Englewood CO
• Contracts were updated regularly to account for
  hardware upgrades at PERA
• Systems were tested semi-annually
• Used Comdisco consulting to assist in creating
  organization wide recovery plans

6
What We Learned From Comdisco

• Corrected system backup problems
• Shifted focus from disaster recovery to business
  recovery
• Highlighted additional recovery needs
• Improved our recovery capabilities
• Improved our knowledge of business continuation
  planning and testing

7
Evaluating Alternatives to Comdisco

• In 1998 PERA priced out additional recovery
  services
• Investment information services
• Internet services
• In 1998 PERAs Business Continuation Steering
  Committee began writing a business case to
  evaluate the potential of building our own
  business recovery facility

8
Business Case

• Overall Goal Evaluate the positive and
  negative aspects of PERA developing its own
  secondary business recovery site within its
  business continuation plan.
• Plan Components
• Reviewed contract constraints and limitations
• Reviewed associated costs
• Defined a plan for recovering our own systems
• Analyzed facility location options
• Defined a conceptual plan for the facility design
• Analyzed potential improvements over outsourced
  recovery

9
Contract Constraints and Limitations

• Recovery test locations
• Scheduling test times
• Keeping contract current
• Increased fees
• Contract term extensions
• Bandwidth requirements

• Six weeks and out dilemma
• Ability to recover new technologies
• Internet services
• Investment systems
• Integration needs
• Incomplete solution
• Confidence level

10
Associated Costs

• Contract expenses for recovery services were
  approx. 150,000 per year
• Contract additions to cover limited investment
  services and Internet connectivity could have
  doubled the annual amount
• In the event of a disaster lasting 6 weeks,
  recovery expenses for contracted equipment and
  facilities would exceed 600,000.

11
Conceptual Systems Plan

• Leverage used equipment and modify minimally as
  needed
• Purchase maintainable used systems rather than
  new
• Increase disk capacity of AS/400 development
  platform to handle a production restore
• Purchase PBX and voice mail system
• Move development imaging system and purchase new
  optical jukebox
• Use the facility to store archive media

12
Conceptual Recovery Facility Plan

• Build or lease approx. 12,000 sq. ft.
• Include data center, call center, general work
  area, counseling office, meeting room, break
  room, storage, etc.
• Locate in Denver metro area
• 10 to 30 miles from Denver office
• Locate in area to serve our membership
• Presented an option to use the facility for as a
  production customer service office

13
Pros of Secondary Site

• Ease of scheduling test time
• LAN Bandwidth
• Close proximity
• Ownership of equipment
• Ability to keep current
• Ability to perform integrated tests
• Ability to extend the use of purchased equipment
• Ability to reduce recovery times

14
Pros of Secondary Site (cont.)

• Reduces risk
• Eliminates logistical problems
• No sharing with other clients
• Eliminates disaster declaration expense
• Compatible PBX and phone mail
• Increases recovery space (2500 12500)

15
Cons of Secondary Site

• Increase in capital costs
• Contract buyout expenses
• Increase in resource allocation toward disaster
  recovery
• Susceptible for area-wide disaster
• Loss of experience of Comdisco staff
• Not a common practice, On our own

16
Business Case Results

• In January of 1999 the business case was approved
  by PERAs Board of Trustees
• In January of 2000 PERA broke ground on the
  business recovery facility

17
From Conceptual to Actual

• 60,000 sq. ft. Office Building
• 40,000 sq. ft. leased
• 20,000 sq. ft. for PERA
• Recovery Center
• Data center
• Work areas
• Trading room
• Conference room
• Customer Service Office
• Storage
• Tenant amenities (absorbed in event of disaster)
• Break room
• Conference Rooms
• Restrooms with shower facilities

18
PERA Skill Sets Leveraged

• Property Management
• Real Estate
• IT Staff
• System Manager
• Telecomm Manager
• Network Managers

19
Determining the Location

• Risk of common outages / disaster impact
• Proximity to customers / staff
• Proximity to headquarters
• Land cost
• Ability to lease office space
• Technology considerations
• Telco capabilities
• Distance
• Line of sight potential

20
Location Selection Map
21
Location Selected
15.6 miles from downtown office
22
(No Transcript)
23
(No Transcript)
24
Recovery Center Floor Plan
25
Recovery Center Features

• Integration with Denver office
• Extension of Local Area Network
• Security system
• Telephone system
• Separation of recovery center from customer
  service center
• Customer service office turns site into a daily
  operation
• Designed for running multiple shifts
• Generator backup power
• Building amenities are shared as recovery center
  features
• Tenant revenue offsets recovery center expense

26
Completed Recovery Center
27
Data Center Features

• Work areas
• Operators
• Systems staff
• Printer / Sealer
• Fire suppression
• VESDA
• FM200
• UPS with generator backup
• Dual AC Units

• Dual entry fiber
• Networked to production data center
• Secondary ISP
• Secured storage for forms, checks, statements

28
Completed Data Center
29
Moving Away from Comdisco

• Construction was completed in October 2000
• Completed systems installation in March 2001
• Exceeded Comdisco contract capabilities in April
  2001
• Terminated contract with Comdisco in May 2001

30
Current Recovery Capabilities

• Benefit Administration Systems
• Optical Imaging System
• General Ledger, Accounts Payable, Accounts
  Receivable
• Portfolio and Trade Order Management
• Telephone System, Voice Mail, Automatic Call
  Distribution
• Network Services (E-mail, File Sharing, Print
  Serving etc.)
• 85 Workstations
• Customer Service
• Building Security System
• Internet Connectivity
• Investment Services (Trade Gateways, Bloomberg
  Information Services)

31
Business Recovery Topology
32
Advantages of the Recovery Facility

• Ability to test as needed
• Ability to recover at a more granular level
• Ability to use recovery equipment as replacements
  for failed production hardware
• Ability to test software upgrades before applying
  to production systems

• Turns business continuation into a daily
  operation
• Improved confidence in the ability to continue
  business in the event of disaster
• Reduces recovery time
• Merges disaster recovery and high availability

33
Cost Containment Questions

• What is our acceptable recovery window?
• What really needs to be recovered? When?
• Do we attempt to tune and enhance hardware
  performance before upgrading?
• Can we consolidate servers and services?
• How can we leverage this investment for other
  uses?

34
Upcoming Enhancements

• Rewrite of business continuity plans
• Dynamic load balancing for inbound/outbound
  Internet services
• Security certification for production and
  recovery sites
• Virtual Private Networking
• Eliminates PERA as ISP
• Provides broadband access to PERA systems from
  home
• Instantly leverages 40 laptops into recovery
  workstations
• Potential mirroring of select systems

35
Notes of Interest

• On July 16, 2001, Comdisco filed for
  reorganization under Chapter 11 bankruptcy
  protection
• Over 91 disaster declarations were received by
  Comdisco within two days of September 11th

36
Questions ???