VMware Data Recovery

1
VMware Data Recovery

• Presented by Kroll Ontrack at
• WI Area VMware Users Group

2
Kroll Ontrack Team

• Robert Bloomquist
• Principle Data Recovery Engineer
• Jodi Bredemeier
• Business Development Team Lead
• Ben Goluska
• Partnership Account Manager
• David Logue
• Sr. Remote Data Recovery Engineer

3
Common Data Loss Scenarios

• RAID/Drive problems
• VMFS File system corruption
• Format/reinstall
• Deleted virtual disks
• Internal VMDK or file corruption
• SQL, Exchange and Oracle
• Deleted files inside a Virtual Machine

4
RAID/Drive problems

• Scenario 1
• RAID 5 3 (500GB) drives
• MS SQL data
• RAID Controller died
• Replaced controller and rebooted
• Drives all offline
• Forced drives online
• No volumes

5
Customers plan

• Force drives online and rebuild
• Restore from backup
• Recreate the data

6
Additional options

• Customer called Ontrack Data Recovery
• Ontrack suggested
• Image the drives before rebuild
• If the rebuild fails
• Start a remote recovery
• Ship drives to Ontrack

7
Customers challenges

• RAID config damged
• One drive out of date
• Rebuild would cause additional damage
• Backups did not include SQL data
• Time to recreate data
• 3 months to 2 years

8
Data Layout RAID 5HEALTHY
Contiguous Files
DRIVE1 Pointer 1 - VM 1 Pointer 3 PARITY   VM 1 - 1 VM1 - 3 PARITY VM 2 - 1 VM2 - 3
DRIVE2 Pointer 2 - VM2 PARITY Pointer 5   VM1 - 2 PARITY VM1 - 5 VM2 - 2 PARITY
DRIVE3 PARITY Pointer 4 Pointer 6   PARITY VM1 - 4 VM1 - 6 PARITY VM2 - 4
9
Data Layout RAID 5HEALTHY
Fragmented Files
DRIVE1 Pointer 1 - VM 1 Pointer 3 PARITY   VM 1 - 1 VM1 - 3 PARITY VM1 - 4 VM1 - 2
DRIVE2 Pointer 2 - VM2 PARITY Pointer 5   VM 2 - 1 PARITY VM1 - 5 VM2 - 3 PARITY
DRIVE3 PARITY Pointer 4 Pointer 6   PARITY VM2 - 4 VM1 - 6 PARITY VM2 - 2
10
Data Layout RAID 5 DISORDERED
DRIVE3 PARITY Pointer 4 Pointer 6   PARITY VM1 - 4 VM1 - 6 PARITY VM2 - 4
DRIVE1 Pointer 1 - VM 1 Pointer 3 PARITY   VM 1 - 1 VM1 - 3 PARITY VM 2 - 1 VM2 - 3
DRIVE2 Pointer 2 - VM2 PARITY Pointer 5   VM1 - 2 PARITY VM1 - 5 VM2 - 2 PARITY
FILE1   VM 1 - 1 VM1 - 2 VM1 - 4 VM1 - 3 VM1 - 6 VM1 - 5
FILE2   VM 2 - 1 VM2 - 2 VM2 - 4 VM2 - 3
11
Resolution

• Ontrack used the original drives and was able to
  rebuild array
• 100 of data recovered
• If customer had proceeded with plans
• Parity would have been overwritten
• Data loss would have occurred
• Up to 1/3 of data would be missing or out of date

12
File System Corruption

• Scenario 2
• 2 SAN LUNs - RAID 5
• LUNs was accidentally formatted NTFS

13
Customers plan

• Restore from backup to same volume
• Recreate the data

14
Additional options

• Customer called Ontrack Data Recovery
• Ontrack suggested
• Restore data to different volume
• If the restore failed
• Start a remote recovery
• Ship drives and/or tapes to Ontrack

15
Customers challenges

• Backups corrupt
• Good backups did not include SQL
• Time to recreate data
• 2 months

16
Resolution

• Ontrack was able to rebuild the VMFS volume
• 100 of critical data recovered
• If customer had proceeded with plans
• Existing structures would have been overwritten
• Data loss would have occurred
• No recovery from DR company

17
Deleted Virtual Machine

• Scenario 3
• Windows 2003 Server
• 2 virtual disks
• Virtual machine was deleted

18
Customers plan

• Restore from backup to same volume
• Recreate the data

19
Additional options

• Customer called Ontrack Data Recovery
• Ontrack Suggested
• Restore data to different volume
• If the restore failed
• Start a remote recovery
• Ship drives and/or tapes to Ontrack

20
Customers challenges

• Backups corrupt
• Time to recreate data - 6 months

21
Data Recovery - Deleted

• What happens when a VM is deleted

22
Data Layout RAID 5HEALTHY
Contiguous Files
DRIVE1 Pointer 1 - VM 1 Pointer 3 PARITY   VM 1 - 1 VM1 - 3 PARITY VM 2 - 1 VM2 - 3
DRIVE2 Pointer 2 - VM2 PARITY Pointer 5   VM1 - 2 PARITY VM1 - 5 VM2 - 2 PARITY
DRIVE3 PARITY Pointer 4 Pointer 6   PARITY VM1 - 4 VM1 - 6 PARITY VM2 - 4
23
Data Recovery - Deleted

• What happens when a VM is deleted
• Pointer is deleted or zeroed out
• Data area is marked as free space

24
Data Layout RAID 5 DELETED
Before
DRIVE1 Pointer 1 - VM 1 Pointer 3 PARITY   VM 1 - 1 VM1 - 3 PARITY VM 2 - 1 VM2 - 3
DRIVE2 Pointer 2 - VM2 PARITY Pointer 5   VM1 - 2 PARITY VM1 - 5 VM2 - 2 PARITY
DRIVE3 PARITY Pointer 4 Pointer 6   PARITY VM1 - 4 VM1 - 6 PARITY VM2 - 4
After
DRIVE1 DELETED Pointer 3 PARITY   FREE FREE PARITY VM 2 - 1 VM2 - 3
DRIVE2 Pointer 2 PARITY Pointer 5   FREE PARITY FREE VM2 - 2 PARITY
DRIVE3 PARITY Pointer 4 Pointer 6   PARITY FREE FREE PARITY VM2 - 4
25
Resolution

• Ontrack was able to rebuild deleted virtual disks
• Manually assembled multiple fragments
• 100 of critical data recovered
• If customer had proceeded with plans
• Existing data would have been overwritten
• Data loss would have occurred
• Possibly no recovery from DR company

26
Internal Corruption

• Scenario 4
• Large storm, city lost power
• UPS engaged and generator started
• Generator ran out of fuel
• UPS failed
• SQL server crashed during large update

27
Customers plan

• Work with SAN vendor
• Work with database vendor
• Work with Microsoft
• Restore from backup to same volume
• Recreate the data

28
Additional options

• Customer called Ontrack Data Recovery
• Make copy of SQL database to server 2
• Restore data to different volume
• Start a remote recovery
• Ship drives and/or tapes to Ontrack

29
Customers challenges

• RAID and VMFS volume OK
• SQL file too corrupt for vendor to repair
• SQL file to corrupt for Microsoft
• Backups corrupt
• Time to recreate data
• Could not recreate data

30
Resolution

• DR Company was able to reconstruct database
  structures/access row data
• 100 of critical row data recovered into a new
  database
• If customer had proceeded with plans
• Database would have been further corrupted (no
  good copy)
• Further data loss would have occurred
• No recovery from DR company

31
Disaster Prevention
32
Disaster Prevention

• Create a Disaster Recovery Plan
• Define critical systems
• Time frames for restores
• Define critical contacts
• Internal and external
• Define procedures
• Define alternative options
• Backup site
• www.marshriskconsulting.com

33
Disaster Prevention

• Create a Disaster Recovery Plan
• Backup the Data

34
Disaster Prevention

• Create a Disaster Recovery Plan
• Backup the Data
• Test the Backups

35
Disaster Prevention

• Create a Disaster Recovery Plan
• Backup the Data
• Test the Backups
• Include Data Recovery in every Plan

36
Questions?
37
Questions

• Toll free Kroll Ontrack
• 1-800-872-2599
• Free phone consultation

38
Thank you
39
Ontrack

• One of the only companies to offer ESX server
  recovery
• Special tools recently developed for these
  recoveries
• Special training provided to engineers
• Development staff available to assist if needed
• Ability to handle complex, high end recoveries

40
What is Virtualization

• Virtualization is
• Software that allows the user to run multiple
  operating systems at the same time on the same
  physical computer by virtualizing the hardware
  of a complete physical computer. 

41
Terms to know

• Virtual (Guest) Machine
• A simulated computer in that it runs on a host
  computer but behaves as if it were a separate
  computer
• VMDK
• A single file that contains a virtual disk
• VMFS
• The VMware file system
• ESX
• The VMware operating system