A Business Continuity Plan for Government

1
A Business Continuity Planfor Government

• George BomarDianne CaseyTexas Department of
  Licensing and Regulation

2

• A practiced logistical plan for how an
  organization will recover and restore partially
  or completely interrupted critical functions
  within a predetermined time after a disaster or
  extended disruption.

3
The Focus on People

• For the main event, CIO Steve Yates wanted to
  test more than the company's technology
  procedures he wanted to incorporate the most
  unpredictable element in any contingency planning
  exercise the people.
• USAA Insurance Company

4

• Legacy of Y2k - Computer failures in banking,
  power, health, telecommunications and financial
  institutions
• September 11, 2001 Worst case scenario
  concept shifted

5
Selected Stats

• 80 of companies worldwide are not prepared for a
  pandemic or a natural disaster
• U.S. DOL estimates over 40 of businesses never
  reopen following a disaster
• Of the remaining 60, 25 close within 2 years.

6
Selected Stats

• Fires permanently close 44 of businesses
  affected
• 90 of companies that lose data are forced to
  shut down within 2 years
• 1993 World Trade Center bombing 150 of 350
  affected businesses failed

7

• More Arkansas Poultry Flocks Checked For Bird
  Flu (UPDATED SATURDAY, JUNE 14, 2008 555 PM CDT
  IN NEWS)By The Associated Press
• Within a few days all commercial chicken houses
  in the area had been tested and the 15,000 birds
  affected were killed and buried. The next step
  was for the commission to go door-to-door,
  checking for other cases.

8

• The Food and Drug Administration is expanding its
  warning to consumers nationwide that a
  salmonellosis outbreak has been linked to
  consumption of certain raw red plum, red Roma,
  and red round tomatoes, and products containing
  these raw, red tomatoes.
• June 5, 2008
• The Emergency Email and Wireless Network

9

• What does BCP look like
• Formal printed manual
• Full access by employees
• Stored in multiple locations
• Secondary work center
• Copies of critical materials

10

• Relationship to Disaster Recovery Plan
• DR - focused on information technology
  applications domain
• Overlap with BCP
• Crisis mgmt structure
• Secondary work center

11

• Data requirements between primary and secondary
  work centers
• Telecommunications architecture
• Data replication methodology
• Application and software availability
• Any physical data requirements at secondary
  site.

12

• Recommended BCP approach
• Smaller ones always contain partial elements of
  larger disasters
• BCP should be broader than disaster recovery
  alone or in case of emergency (ICE) procedures

Plan for the BIG disasters
13
BCP Purpose

• To enable leaders to
• maintain essential business
• processes and practices
• and equip the organization
• with means of becoming
• less vulnerable to incidents

14
The TDLR Plan

• Identifies management team members
• Designates remote site(s)
• Enumerates four (4) major scenarios
• Itemizes recovery steps to be taken within
• five (5) primary business functions

15
EventsThat might trigger an interruption

• Loss of key personnel
• Weather-related
• Infrastructure-related
• Internal system breakdowns

16
EventsThat might trigger an interruption

• Failure of an external business partner
• Health crisis impacting the work force
• A cyber attack
• An act of terrorism

17
Rating the Triggers
Probabilities of occurrence

• 1- Least likely to happen
• 4 - Most likely to happen

18
Impacts

• DURATION
• Will the effects be short-term, or longer?
• EXTENT
• How much of work force is impacted?

19
Devising a Template

• A questionnaire was circulated to capture
• Recovery procedure
• Recovery time objective
• Recovery location
• Dependencies
• Other considerations
• Summary of recovery steps

20
The Process

• Solicit written input from key personnel
• via templates
• Interview managers
• Prepare draft for each business function
• Obtain review comments and incorporate
• into revised draft

21
How About Prevention?

• Mitigate the impact of a disaster
• Practice good housekeeping
• Adhere to security procedures
• Observe information security procedures
• Maintain up-to-date operating guidelines

22

• An Emergency Management Team Convenes to decide
  
• Implement the BCP?
• Activation prompted by Team Lead

23
Alternate Location(s)

• Primary Site
• Alternate Site
• BCP provides directions to the sites

24
Scenario I

• The population of possible causes was condensed
  into four (4) major scenarios
• Loss of key executive personnel for a protracted
  period due to accident or other unforeseen
  event

25
Scenario II

• Loss of building access because of weather (or
  other natural disaster)-related event

26
Scenario III

• Contractor default, or other supplier of a
  critical service to the agency, abruptly goes
  out of business without warning and,

27
Scenario IV

• Health crisis (or act of terrorism) leads to an
  exorbitant rate of employee absenteeism (and
  temporary replacements are unavailable).

28
Functions Impacted

• The plan identifies five (5) main business
  functions adversely affected by the crisis
• Licensing of individuals and businesses
• Education and examination activities
• Measures to ensure compliance
• Administrative support
• Technological support

29
Initial Approach

• For each of the five (5) business functions,
• Identify impact,
• Recovery procedures, and
• Dependencies
• Redundancy

30
Adopted Approach

• For each of the four (4) scenarios
• Identify how each business function
• would be adversely impacted

31
Example I

• If key personnel were lost (Scenario I)
• Notify the agencys directors
• Convene emergency meeting of the Commission
• Formulate short-term succession plan
• Notify Governors office and key legislators
• Designate primary agency contacts
• Implement plans to notify the public, equip
  customer service, respond to complaints

32
Example II

• If building was inaccessible (Scenario II)
• Licensing
• Education and Examinations
• Compliance
• Administrative Support
• Technological Support

33
Example III

• If major contractor failed (Scenario III)
• Identify affected functions
• Marketplace alternatives?
• Make temporary process changes
• Procure new/other contractor

34
Example IV

• If a health crisis decimated the work force
  (Scenario IV)
• Identify skills of available staff
• Can skills be realigned?
• Determine what functions (e.g. inspections) can
  be postponed or suspended
• Consider tapping into regulated industries for
  temporary expertise

35
A Summary of Recovery Steps

• Plan must specify
• Key actions to be taken,
• By whom,
• In what order,
• For each business function.

36
Important Addenda

• Identify in an Appendix
• BCP Team Lead and Members
• with current contact information
• Name and address
• Phone number(s)
• E-mail address(es)

37

• Include
• a Phone Tree listing - who will contact whom
• Identify how information will be disseminated to
  employees
• List first group(s) to report to alternate site.

38

• Periodically,
• re-assess your BCP
• and update as needed!

39
Testing

• Purpose
• Achieve organizational acceptance
• Determine that the BCP solution is appropriate
  for recovery requirements
• Identify and correct design flaws
• Identify and correct implementation errors

40

• After 9/11, those companies with
• tested BCP manuals had business
• resumption within days.

41
Selected Stats

• 45 of companies with a BCP do not test it
  annually
• 80 of companies have not developed an IT crisis
  management function
• 40 of companies that have a crisis management
  plan do not have a dedicated crisis management
  team

42
Mistakes and Pitfalls

• Failing to gain senior level management support
• Not identifying all critical systems (including
  laptop data)
• Failing to bring the entire business into
  planning and testing
• Not identifying and planning for all gaps in
  recovery objectives
• Insufficient funding for testing

43
USAA Story

• 20,000 employees - needed HazMat training, an
  evacuation plan and a recovery plan
• Live exercises were confined to technology
  assets - recovering data from backup data
• Otherwise, passive exercises tabletop and
  paper simulations, role-play, guessing how
  people would react

44
USAA Story

• Post 9/11, built alternative center 200 miles
  away from San Antonio, on different power grid
  and water supply
• Steve Yates designed large scale continuity
  exercises
• At the first one, USAA discovered
• The setup process for computers and phones took
  nearly two hours leaving employees standing in
  the hot Texas sun.

45
USAA Story

• USAA take-away from testing
• Those who walked through the simulation were in
  the best position to find flaws and offer
  suggestions.
• Those who practice emergency situations are less
  likely to panic and are more likely to remember
  the plan.

46
Plan Maintenance Cycle

• Revisit annually or biannually
• Confirm information roll out to all staff
• Perform staff training
• Test and verify technical solutions for recovery
• Test organization recovery procedures

47

• Questions
• ????

48

• Presenters
• George Bomar 512-936-4313
• GBomar_at_license.state.tx.us
• Dianne Casey 512-463-7182
• Dianne_at_license.state.tx.us
• Texas Department of Licensing and Regulation