Sourcing Innovations How to Ensure your IT Department is Effective A Review of Outsourcing and Insou

1
Sourcing Innovations How to Ensure your IT
Department is EffectiveA Review of Outsourcing
and Insourcing Decisions and there Implications
2
Background

• Background
• Childrens Hospital Los Angeles
• 107 Years old
• 286 Bed Non-Profit Acute Care Facility
• Ranked 9 in US News and World Report
• New Building to Open in July 2010
• Education
• PhD Candidate in Information Services Management
• MSc in Business Information Technology, Kingston
  University, London
• BA in Applied Mathematics, Scientific
  Programming, UC San Diego
• Outsourcing Experience
• 1 Year with Perot Systems Healthcare
  Outsourcing, as Chief Operating Officer
• 6.5 Years with Computer Science Corporation,
  most recently as Account Executive
• 1 Year with San Diego County, IT Outsourcing as
  Manager
• Other Relevant Experience
• 2 Years as IT Global Procurement Director in
  London with over 300M annual Spend
• 5 Years in the Financial Industry in IT

3
Sourcing can be the Source of

• Massive Frustration
• Extensive Additional Costs
• Poor Quality
• Lack of Accountability
• Minimal Expertise
• Forced Contracts and HW/SW Packages
• Poor Performance
• Stretched Resources
• Relationship Challenges
• People are the Source of your Success or Failure

4
The Evolution of Change
5
EMR Current State
6
How Safe Are We
HealthCare
Total lives lost per year
Numbers of encounter for each fatality (Modified
from Leape)
7
CHLA Case Study

• In September 2006, CHLA Faced These Major
  Challenges (and many more)
• On November 26th 2006, the Email System Suffered
  a Catastrophic Failure
• The Average Age of a PC was 6 Years Old Windows
  2000
• Data Center Suffered from Regular Power and
  Cooling Failures
• There was No SAN in Place 10Mb Storage Available
  for Email Multiple Errors
• The EMR System was Down for Over 10 Hours per
  Month Crash During Peak Hours
• The Network Core Switches had not been
  Updated/Patched in 7 Years
• The Only Replacement Network Parts Available were
  via eBay
• Only select Clinical areas were Wireless and at
  802.11a/b
• The CHLA Internet Site was nearly Frozen, with
  Minimal Changes Allowed
• There were 8 Active Projects, with Only 3 Closed
  in the Prior Year
• No Redundancy or DR for Email, Network, EMR,
  Servers, Multiple Apps
• Customer Satisfaction was below 40 and Dropping
• The of IT Expense as Compared to Total CHLA
  Expense was gt 10
• Over 80 of the IT Budget was Related to
  Outsourcing Services
• CHLA was in Negotiations with CSC for an
  Outsourcing Contract Extension

8
How Do You

• Replace an Entire IT Organization?
• Regain the Organization, Board, and Executives
  Trust?
• Consolidate your Application Portfolio?
• Bring Stability to a Wavering EMR?
• Deal with an Increasing Application Demand?
• Insource Infrastructure Services?
• Replace the Entire CHLA Email System?
• Replace a Severely Aging PC Environment?
• Replace a Dying Network?
• Implement a New SAN?
• Replace the CHLA Internet Site?
• Start a Massive New Hospital Building Project?
• Quadruple the Number of IT Projects?
• Reduce the Operating Expenses by over 30?
• Improve the Hospital IT Survey Results by 40?
• All within 2 years

9
Organizational Distress

• 2006 CHLA IT Organization
• 67 of Management Positions were Outsourced
• No Accountability Model
• No Internal Project Management Dept
• All Financial Systems Reported to CFO
• Unclear Lines of Responsibility
• Consultants were Blaming each other for any
  Issues
• One Consulting Company was Suing CHLA for 9M
  and they were still working for CHLA
• Projects were Always Over Budget, Over Schedule,
  with Scope Creep

• Organization Transformation
• Strategic Focus Position Creation
• CMIO Further Integration
• Full Regular Infrastructure Staffing Level Review
• Full Regular Applications Staffing Level Review
• Full Regular Cross Functional Staffing Level
  Review
• No Management Positions are Outsourced

10
Outsourcing Struggles

• Sourcing Struggles
• Over 80 of all Application Support was
  Outsourced
• 100 of All Critical Applications were Outsourced
• No Accountability or SLA Model for the Situation

• 2007/2008 Sourcing Transitions
• CHLA Transitioned the Following Applications to
  New Suppliers
• Cerner Hardware Suite
• McKesson Application/Hardware
• PeopleSoft Application/Hardware
• EGate Application/Hardware Suite
• Department Application Support Suite
• The Combined Estimated Annual Bottom Line Savings
  for the Application Transitions is 1.23M
• Increased Staff Experience, Ability, Customer
  Satisfaction and Location
• Implemented Remote Hosting Offering (RHO) with
  the Application Providers which provides Hardware
  Stability, Regular Updates, DR, and Cost Savings

11
Application Sourcing Transformation
Non-CORE Patient Management Systems
Non-CORE Patient Financial Systems
CORE Financial Systems
CORE Patient Related Systems
Non-CORE Clinical Systems
Budgeting (TSI)
Lab (Advantage)
Clinical Systems
Time Attendance (LAR)
Path Lab (Co Path)
Patient Management
Registration (McKesson HBOC)
Blood Bank (HemoCare)
Other Systems
Scheduling (Oracle)
Radiology (ProgRIS/FUJI PACS)
Tight Integration is Essential
Medical Records (McKesson HBOC)
Pharmacy (WORx)
Tight Integration is Essential
ED Triage/Doc (None Available)
Critical Care (CareVue)
Patient Accounting (McKesson HBOC)
ADT (McKesson HBOC)
Patient Accounting (McKesson HBOC)
Anesthesia (CompuRecord)
Fixed Assets (Global)
Surgery (Mediware)
Infrastructure
Interface Engine (Open Hub - Eclypsis)
Website (CHLA Org) (TBD)
Department Centric Databases
Application Suppliers 23
12
Application Sourcing 2008
Non-CORE Patient Management Systems
Non-CORE Patient Financial Systems
CORE Financial Systems
CORE Patient Related Systems
Non-CORE Clinical Systems
Budgeting (TSI)
Lab (KIDS)
Clinical Systems
Time Attendance (LAR)
Path Lab (Co Path)
Patient Management
Registration (McKesson HBOC)
Blood Bank (Wyndgate)
Other Systems
Scheduling (KIDS)
Radiology (KIDS/FUJI PACS)
Tight Integration is Essential
Medical Records (KIDS)
Pharmacy (KIDS)
Tight Integration is Essential
ED Triage/Doc (KIDS)
Critical Care (CareVue)
Patient Accounting (McKesson HBOC)
Staff Scheduling
ADT (McKesson HBOC)
Anesthesia (CompuRecord)
Fixed Assets (PeopleSoft)
Patient Classification
Surgery (Mediware)
Infrastructure
Website (CHLA.org-Re Design)
Interface Engine (Open Hub - Eclypsis)
Department Centric Databases
Application Suppliers 13 Reduced Application
Suppliers 10
13
Application Sourcing Future
Non-CORE Patient Management Systems
Non-CORE Patient Financial Systems
CORE Financial Systems
CORE Patient Related Systems
Non-CORE Clinical Systems
Budgeting (TSI)
Lab (KIDS)
Clinical Systems
Time Attendance (LAR)
Path Lab (Cerner AP)
Path Lab (KIDS AP)
Patient Management
Registration (McKesson HBOC)
Blood Bank (Wyndgate)
Other Systems
Scheduling (KIDS)
Radiology (RadNet/FUJI PACS)
Tight Integration is Essential
Medical Records (KIDS)
Pharmacy (KIDS)
Tight Integration is Essential
ED Triage/Doc (KIDS)
Critical Care (KIDS iNet)
Critical Care (KIDS iNet)
Patient Accounting (McKesson HBOC)
ADT (McKesson HBOC)
Anesthesia (KIDS SurgiNet)
Anesthesia (CompuRecord)
Fixed Assets (PeopleSoft)
Surgery (KIDS SurgiNet)
Surgery (Mediware)
Infrastructure
Website (CHLA.org-Re Design)
Interface Engine (Open Hub - Eclypsis)
Department Centric Databases
Application Suppliers 6 Reduced Application
Suppliers 17
14
EMR Sourcing Transformation

• CHLA EMR Transformation
• EMR Struggles Prior to 2006
• The EMR Servers were Outsourced and Resided in
  San Diego
• There was Over 10.5 hours of Downtime Monthly
• Both CSC and Cerner were Blaming each other for
  the All Excessive Issues
• No Clear Line of Accountability Existed
• No Projects for the EMR Upgrade were Moving
• Minimal to No Upgrades were Being Performed
• No User Analytics were Available
• SLA Confusion and Lack of Enforcement
• EMR Transformation
• Transitioned the Servers to Cerner and
  Implemented Cerners RHO
• Reduced the Amount of Downtime Monthly by 81.5
  (Near 95 in last 12 Months)
• Clear Lines of Accountability with Cerner
• All Major Projects with Cerner have Started
• All Upgrades and Analytics are Now Available

15
Application Implementations

• Application Backlog
• Critical Upgrades had not been Performed in Over
  5 Years
• New Application Implementations did not Occur
• Regulatory, Support, and Maintenance Issues were
  at a Critical Executive Level

• 2007/2008 Application Upgrades
• Performed the Following Upgrades
• McKesson Double/Double Upgrade
• EC2000 Quadruple Upgrade
• PCON Triple Upgrade
• McKesson Navigator Upgrade
• CompuRecord Upgrade
• KIDS Upgrade to 2007.03 Code
• 2007/2008 Application Implementations
• Implemented HPM Cost Accounting System
• Implementation of Sodexho Interface with KIDS
• Wyndgate Implementation
• ePay Implementation
• UB04 Configuration Implementation
• E-MAR Implementation
• USC Faculty Profile System

16
Infrastructure Insourcing

• Infrastructure Transition Overview
• Analysis Performed in lt3 Months Executive and
  Board Presentations
• gt50 Reduction in Infrastructure Costs
• Realized Savings of 6.5M Directly to Bottom Line
  (Year over Year Budget)
• 55 Onsite Infrastructure Support Staff Increase
  which has Improved Backlog and Service
• Creation of a Project Management Office Reducing
  Project Consulting Costs
• Creation of a Web Development Team
• Provides for Full Software/Hardware Control which
  has produced Cost Savings, Improved Service and
  Increased Stability/Security
• Moved Multiple Offsite Services Back to CHLA
  providing Incredible Support, Training and
  Response Time Improvements
• Installed Active Directory, New Virus Protection,
  New Helpdesk System, New Monitoring Systems, New
  VPN Remote Access, and New Management Reports

17
Rapidly Aging PCs

• Severely Aging PC Environment
• Rapidly Deteriorating PC Environment
• Current Aging Statistics
• 40 (1408) PCs gt 7 Years Old
• 65 (2287) PCs gt 6 Years Old
• 86 (3026) PCs gt 3.5 Years Old
• Processor Speeds (Estimated)
• gt 7 Years 500Mhz to 750Mhz
• gt 6 Years 750Mhz to 1Ghz
• gt 3.5 Years 1Ghz
• Software for Older PCs is a Major Challenge
• Windows 2000
• Out of Support
• No Further Patches
• Application Compatibility Issues
• Microsoft Brought In to Help With
• Architectural Design Sessions
• Driver Injection

• Desktop Refresh Project
• Engineering Brought In to Help With
• Architectural Design Sessions
• Driver Injection
• Lab Setup
• Image(s) Creation
• Deployment Plan

18
Network Replacement

• After IDF Closet

• Before IDF Closet

• Network Refresh Project
• Current Challenges
• Lack of Redundancy
• Minimal Available Scalability
• Speed
• Out of Support
• No Available Replacements
• Power/Cooling Challenges
• New Network Refresh Benefits
• Best of Breed Solution
• Full Redundancy
• Full Scalability
• Lower Power Cooling Requirements
• 10Gb/sec Backplane Speeds
• 1Gb/sec to Desktop
• PoE Enhanced Reliability
• Distributed Antennae System (DAS)

19
SAN Replacement

• Storage Area Network Overview
• Past Challenges
• Regular Error Messages for Space
• Lack of Reliability
• Back-Up Issues
• Restore Delays and Challenges
• Minimal Available Storage
• Slow Retrieval Speeds
• Lack of Redundancy
• Relatively Small Supplier
• Current SAN Successes
• 45TB Useable Storage
• New Back-Up System Installed
• Server Consolidation/VMWare
• Network Application Implementation
• Additional Scalability
• Additional Reliability / SAN will Automatically
  Contact if Failure

20
Internet Site Insourcing

• Internet Site Insourcing
• Past Internet Site Issues
• Locked Contract with Outrageous TL Costs/Charges
• Minimal Functionality
• No Content Management System (CMS)
• Lack of Timely Modifications and Updates
• No Integration into Intranet, SharePoint, or any
  Other Internal Source of Information
• Insourcing Benefits
• Flexible, modern platform/CMS to allow Quick
  Turnaround on Content and Requested Changes
• Created a New In-house Team of 3 Highly Qualified
  Individuals
• New Functionality and Integration Available with
  Immediate Implementation and Web 2.0 Integration
• The New Team Completed the New Site within 6
  Months of Project Kick-Off
• Reduced Costs Substantially for Modifications,
  Updates, and New Implementations

21
PMO Department Creation

• Project Management Office Overview
• Prior Project Management Issues
• Utilization of Contractors for PM Roles
• Minimal CHLA Oversight or Control on Budgets /
  Schedule / Quality
• Utilization of Contractors Software, Hardware,
  and Reporting Tools
• Projects would Never End!
• Current PMO Success
• A Record 43 Open Projects Underway
• Over 5 Multi-Million Dollar Projects are
  Currently In Flight
• 100 of all Projects Completed by the New PMO
  were on Schedule, Budget, and within Scope
• Trust has been Restored at CHLA
• Major Project Successes
• RadNet
• Web Redesign
• ASC
• CNN Move

• Galileo Interface
• Wyndgate Implementation
• Minimally Invasive Suite Implementation
• Research Time Effort
• Solimar UNIX Reporting

22
Financial Reversal

• Financial Transformation
• Past Financial Challenges
• No Trust or Confidence from the Executives or
  Board to Implement Any IT Projects
• Prior IT Expenses were Dramatically Over Budget
  Annually and Progressively getting Worse
• There were Minimal Accounting Controls for the
  Outsourced Services with Costs Out of Control
• Financial Transformation
• IS Department Completed FY07 4M Under Budget
• Completed FY08 3M Under Budget
• Implemented New ITEC Capital Budget Tracking and
  Monitoring
• IS Contracts Organization Renegotiations with
  Enormous Financial Benefits
• Reduced the IS Budget in the First Full Fiscal
  Year by 30
• Reduced the IS Expense vs. Total Hospital Expense
  by 5, to 5.09 Annually
• Created a New PMO Team, Web Team, Finance Team
  and CTO without an Increase

23
Survey Results

• Email Communications Refresh
• Previous Surveys
• Reflected a Very Disgruntled, Untrusting, and
  Demanding Client Base
• The Clients were Calling for our Infrastructure
  Outsourcers Elimination
• 2008 Survey Results
• The Survey had Over a 21 Return New Record for
  the Department
• The Overall Response and Comments were Very
  Positive with Extremely Complimentary Comments
• Over 78 of Respondents Agreed that IS has
  Improved
• Nursing 71.9
• Research 78.3
• Executives 93.6
• Polite and Courteous were the Consistently
  Highest Scores
• Primary Areas of Improvement were Across the
  Board