SPLIT PERSONALITY MALWARE DETECTION AND DEFEATING IN POPULAR VIRTUAL MACHINES

1
SPLIT PERSONALITY MALWARE DETECTION AND DEFEATING
IN POPULAR VIRTUAL MACHINES

• Alwyn Roshan Pais
• Alwyn.pais_at_gmail.com

Department of Computer Science Engineering
National Institute of Technology, Karnataka
2
(No Transcript)
3
Objective

• To study the VM detection techniques in popular
  Virtual machines.
• Develop strategy to counter the detection.
• Prevent analysis aware malwares from detecting
  VM.

4
Plan of Action

• Introduction
• VM detection techniques
• Detection techniques in VMware, VirtualBox and
  VirtualPC.
• Related Work
• Prevent Analysis aware malwares from detecting
  VM.
• VMDetectGuard Tool to mask VM detection
  Windows
• Optimization of VMDetectGuard
• Results

5
Introduction
6
Malware

• Malware It is a collective term for any
  malicious software which enters the system
  without the authorization of the user of the
  system.
• Anti-virus/anti-malware products do not guarantee
  complete protection.

7
Present Scenario

• Security researchers use malware analysis tools
  to build defenses against the unknown malware
  forms.
• They then build patches for the newly discovered
  vulnerabilities and exploits.
• Virtualization has emerged as a very promising
  technology.
• Malware analyst use Virtual Machine Environment
  (VME), debuggers and sandboxes in their analysis
  work.

8
Virtualization

• A software based representation of a computer
  that executes programs in the same way as a real
  computer.
• Examples, VMware, Virtual PC, VirtualBox.
• Advantages
• Reduced capital and operational costs through
  more efficient use of hardware resources.
• Simplifies maintenance .
• Improves scalability and deployment agility.
• Improves reliability.

9
Benefits of Virtualization to Security Researchers

• Researchers can intrepidly execute potential
  malware samples without having their systems
  affected.
• If a malware destabilizes the OS, analyst just
  needs to load in a fresh image on a VM.
• Reduces time and cost.
• Increases productivity.

10
Analysis Awareness Functionality

• Malware developers have added a new functionality
  to malware.
• Detect the presence of analysis tools such as
  VMs, debuggers and sandboxes.
• Hide their malicious behavior on detection.
• Analysis Aware / Split Personality malware.

11
Related Work

• Carpenter (Carpenter et al., 2007) proposes two
  mitigation techniques.
• They aim at tricking the malware by
• Changing the configuration settings of the .vmx
  file present on the host system and,
• Altering the magic value to break the guest-host
  communication channel.

12
Drawbacks of the First Approach

• The configuration options break the communication
  channel between guest and host not just for the
  program trying to detect the VM, but for all the
  programs.
• Moreover the authors claim that these are
  undocumented features and that they are not aware
  of any side effects.

13
Related Work

• The work by Guizani (Guizani et al., 2009)
  provides an effective solution for Server-Side
  Dynamic Code Analysis.
• Small part of the solution deals with tricking
  the Split Personality malware that employ Memory
  Detection and VM Communication Channel Detection
  techniques.

14
Related Work

• Kalpa Vishnani et. al. 2011 Masks all the
  detection techniques used in Vmware.

15
Related Work

• Other works concentrate
• Detecting this category of malwares
• Running in host machine
• Save the current state
• quickly restore to previous state
• Virtual machines in the order of market share
• VMware, Virtual PC, and Virtual Box.

16
VM Detection Techniques

• Hardware fingerprinting
• Registry Check
• Process and File Check
• Memory Check
• Timing Analysis
• Communication Channel Check
• Invalid Instruction Check

17
Hardware Fingerprinting

• Involves looking for specific virtualized
  hardware.
• VMs give an abstracted view of many hardware
  components.
• Querying for such components reveals VM presence.
• For Example BIOS, Motherboard, SCSI Controllers,
  USB Controllers, etc.

18
Hardware Fingerprinting Results
19
Registry Check

• The registry entries contain hundreds of
  references to the string containing the name of
  the VM, Ex. Vmware, VirtualPC and VirtualBox.
• Checking the registry values for certain keys
  clearly reveals the VM presence.

20
Registry Check

• For Example
• HKEY_LOCAL_MACHINE\HARDWARE\DEVICEMAP\Scsi\Scsi
  Port1\Scsi Bus 0\Target Id 0\Logical Unit Id
  0\Identifier
• ?VMware, VMware Virtual S1.0
• HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\C
  lass\4D36E968-E325-11CE-BFC1-08002BE10318\0000\D
  riverDesc
• ? VMware SCSI Controller
• HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\C
  lass\4D36E968-E325-11CE-BFC1-08002BE10318\0000\P
  roviderName
• ? VMware, Inc.

21
Process and File Check

• Check - VM specific processes and files presence
• Eg.
• VBoxService.exe In VirtualBox for
  synchronization with host
• drivers like vboxhook.dll and vpcbus driver
  present in SYSDIR/drivers

22
Memory Check

• This involves looking for values of critical
  operating system data structures.
• These data structures are relocated on a virtual
  machine so that they do not conflict with the
  host system's copies.
• Store Interrupt Descriptor Table (SIDT), Store
  Local Descriptor Table( SLDT), Store Global
  Descriptor Table (SGDT), Store Task Register
  (STR), Store Machine Status Word (SMSW)
• Redpill.exe, ScoopyNG.exe use this method.

23
Timing Analysis

• Obvious yet rare attack.
• Involves looking at a local Time Stamp Counter
  (TSC) value.
• By noting down the time difference VM presence is
  detected.

24
VM Communication Channel Check

• This check involves detecting the presence of a
  host-guest communication channel.
• IN instruction and a magic number VMXh
• VmDetect.exe uses this check.
• Not applicable to VirtualPC and VirtualBox.
• Runs in VMware without exception.

25
Invalid Opcode Check

• Specific to VirtualPC
• Uses certain opcodes for guest host communication
• In host system raise exception and no exception
  in VirtualPC.

26
Vmware Detection
Hardware fingerprinting

• hardware details
• motherboard serial number, graphics card and
  network adapter captions
• Windows Management Instrumentation (WMI) contains
  classes
• hardware, display, registry etc.
• Check for VM specific strings

27
Registry Check

• Windows Registry stores
• configuration settings
• low-level operating system components
• Applications running
• Check for
• Strings like VirtualPC, VBOX, VirtualBox
• value that is specific to the corresponding
  virtual machine being testing on.

28
Process and File Check

• Check - VM specific processes and files presence
• Eg.
• VBoxService.exe In VirtualBox for
  synchronization with host
• drivers like vboxhook.dll and vpcbus driver
  present in SYSDIR/drivers

29
Memory Check

• involves looking at the values of specific memory
  locations
• STR (Store Task Register)
• stores the selector segment of the TR register
  (Task Register) in the speci?ed operand (memory
  or other general purpose register).
• Value specific in Virtual Machine

30
Invalid Opcode Check

• Specific to VirtualPC
• Uses certain opcodes for guest host communication
• In host system raise exception.

31
Detection of VM running Linux

• Techniques (tested on Vmware)
• Hardware Fingerprinting
• Dmesg check - prints the message buffer of
  the kernel
• /proc file system check - interface to internal
  data structures in the kernel.
• Communication channel check

32
Dmesg and /proc file system check

• Dmesg - prints the message buffer of the kernel
• Shows diagnostic message showing presence of
  hardware during boot
• contain strings like VMware,
• /proc file system - an interface to internal data
  structures in the kernel
• Contains system dependent information

33
Communication Channel Check

• IN instruction
• Raises exception EXCEPTION PRIV INSTRUCTION in
  host
• Runs in VMware without exception
• initiates guest to host communication by calling
  the IN instruction.

34
VMwareDetect

• Is the proof of concept tool.
• It employs the various VM detection techniques to
  detect the presence of VMware virtual machine.
• Memory Check
• VM Communication Channel Check
• Hardware Fingerprinting
• Registry Check
• Timing Analysis

35
VMwareDetect
36
VirtualMachineDetect - VirtualPC

• Check using all the methods

In VirtualPC In Native Machine
Hardware Fingerprinting Hardware Fingerprinting Hardware Fingerprinting
BIOS American Megatrenda L900781
Graphics Card Virtual PC Integration Components S3 Trio32/64 NVDIA GeForce 310
Baseboard Manufacturer Microsoft co-orporation LENOVO
System Name VIRTUALXP User-think
USB Controller USB Virtualisation Bus Driver Intel 5 Series /3400
Registry Check Registry Check Registry Check
SCSI HARDWARE\\DEVICEMAP\\Scsi\\Scsi Port 0\\Scsi Bus 0\\Target Id 0\\Logical Unit Id 0 Virtual HD Hitachi HDS721050CLA362
Control class for usb SYSTEM\\ControlSet001\\Control\\Class\\36FC9E60-C465-11CF-8056-444553540000\\0000 USB Virtualisation Bus Driver Intel 5 Series /3400
Control class for graphics SYSTEM\\ControlSet001\\Control\\Class\\4D36E968-E325-11CE-BFC1-08002BE10318\\0000 Virtual PC Integration Components S3 Trio32/64 NVDIA GeForce 310
Controlset for cd/dvd drive SYSTEM\\CurrentControlSet\\Enum\\IDE Disk Virtual_HD____1._1__ Registry not found
Invalid Opcode Did not raise exception Raised exception
File Check File Check File Check
Vpcubus Driver (Virtual USB Bus Driver) Present Not Present
Vpcgbus Driver (Virtual PC Guest Bus Driver) Present Not Present
Vpcuhub Driver (Virtual USB Hub Driver) Present Not Present
37
VirtualMachineDetect - VirtualBox
Virtual Box running windows Host Windows Machine
Hardware Fingerprinting Hardware Fingerprinting Hardware Fingerprinting
BIOS 0 L900781
Graphics Card Virtual Box Graphics Adapter NVDIA GeForce 310
N/W adapter AMD PCNET Family PCI Ethernet Adapter WAN Miniport(SSTP)
Processor Null CPU1
USB Controller Std Open HCD USB Host Controller Intel 5 Series /3400
Registry Check Registry Check Registry Check
Dsdt HARDWARE\\ACPI\\DSDT VBOX__ Registry not present
Scsi P0 HARDWARE\\DEVICEMAP\\Scsi\\Scsi Port 0\\Scsi Bus 0\\Target Id 0\\Logical Unit Id 0 VBOX HARDDISK Hitachi HDS721050CLA362
Scsi P1 HARDWARE\\DEVICEMAP\\Scsi\\Scsi Port 1\\Scsi Bus 0\\Target Id 0\\Logical Unit Id 0 VBOX CD-ROM Null
Vedio Bios Version HARDWARE\\DESCRIPTION\\System\VideoBiosVersion Oracle VM VirtualBox Version 4.1.2 VGA Bios Version 70.18.3E.00.05
System Bios Version HARDWARE\\DESCRIPTION\\System\SystemBiosVersion VBOX-1 LENOVO-133
Instruction Check Instruction Check Instruction Check
STR (store task register) 28 0 40 00
File Check File Check File Check
VBOXHook.exe Present Not Present
VBOXTray Present Not Present
VBOXService.exe Present Not Present
38
Virtual Machine Detect

• In VB

39
Remote Detection

• Scenario
• There is access to the terminal of a system
• need not be administrator access
• WMIC ( Windows management instrumentation command
  line) is used

40
Masking Detection of VM

• Using PIN API provided by Pin tool.
• Can get all the instructions, the arguments and
  return value
• Steps followed for masking
• Get each call made by binary.
• Check if matches a predefined list of calls. E.g.
• RegEnumValueA
• Str
• LoadLibraryA
• __emit

41
Masking Detection of VM

• Provide false values if
• VM specific values are read (matched from
  predefined list)
• Eg.
• Registry read returns the value VBOX
• Pin Tool gets the return value and modifies it in
  runtime.
• Registry read function returns modified value

42
Masking Detection of VM

• Binary does not detect manipulated value
  received.
• This currently supports
• 64 and 32 bit OS
• 64 and 32 bit applications

43
Masking Detection of VM
44
Our Approach
45
Our Approach

• STEP 1
• Maintain a list of all the hardware as well as
  registry querying API calls. Also maintain a list
  of all the VM specific instructions such as SIDT,
  SLDT, SGDT, STR, IN.

46
Our Approach

• Following is a partial list of API calls to be
  monitored.
• Hardware Querying APIs
• SetupDiEnumDeviceInfo
• SetupDiGetDeviceInstanceId
• SetupDiGetDeviceRegistryProperty
• Registry Querying APIs
• RegEnumKey
• RegEnumValue
• RegOpenKey
• RegQueryInfoKeyValue
• RegQueryMultipleValues
• RegQueryValue

47
Our Approach

• Step 2
• Perform dynamic binary instrumentation of the
  sample under test in order to obtain its low
  level information as well as to intercept all the
  API calls made by it.
• We hook into the sample under test by means of
  .dll injection.
• This is achieved using the pin framework.

48
Our Approach

• Step3
• Check to see if the sample under test makes a
  call or executes any of the monitored API calls
  or instructions respectively. If a match is
  found, set the OUTPUT to Split Personality
  Malware Detected. Also, log the activity and
  provide fake values to the sample so as to make
  it feel that it is running on a host system.

49
Implementation

• Designed, implemented and tested VMDetectGuard.
• Implemented in the framework provided by the Pin
  tool released by Intel Corporation.
• Pin is a tool for the instrumentation of
  programs.
• We made use of its framework to intercept the
  various API calls and low level instructions
  executed by the sample under test.

50
COUNTERING HARDWARE FINGERPRINTING

• Hardware emulation.
• APIs that query for BIOS, Motherboard, Processor,
  Network Adapter.
• Ex. VM returns a value none for motherboard
  serial number. VMDetectGuard returns a more
  appropriate string such as .16LV3BS.CN70166983G1X
  F instead.

51
Countering Registry Check

• VMDetectGuard monitors registry querying APIs
  such as the following
• RegEnumKey
• RegEnumValue
• RegOpenKey
• RegQueryInfoKeyValue
• RegQueryMultipleValues
• RegQueryValue
• If the output contains the string "VMware", our
  tool replaces this string with a more appropriate
  value that would have been returned on a non
  virtual system.

52
COUNTERING MEMORY CHECK

• SIDT, SLDT, and SGDT and STR instructions are
  monitored.
• The values of the target registers are then
  changed appropriately with the values that would
  have been obtained on a host OS.

53
COUNTERING MEMORY CHECK
54
COUNTERING VM COMMUNICATION CHANNEL CHECK

• Monitor execution of the IN instruction.
• We change the value of the magic number .
• This leads to generation of EXCEPTION PRIV
  INSTRUCTION exception.

55
COUNTERING TIMING ANALYSIS

• Instructions such as CPUID and RDTSC (Read Time
  Stamp Counter) are monitored.
• The tool maintains a log of each type of
  instruction executed.
• If the threshold value for a particular type of
  instruction is exceeded, it logs this activity
  too.
• Sample is tricked by deleting the CPUID
  instruction and modifying the values of ebx, ecx,
  and edx.

56
VMDetectGuard

• VMDetectGuard is our solution tool to counter
  Split Personality Malware.
• VMDetectGuard runs in two different modes.
• VM Guard Mode
• Non VM Guard Mode

57
VMDetectGuard

• Output Generated by VMDetectGuard
• Result Split Personality malware detected/not
  detected.
• VM Specific Log
• Instruction Trace
• System Call Trace
• Registry Trace
• Opcode Mix
• Instruction Count
• Diff Tool Feature

58
VMDetectGuard
59
Results Analysis
60
Redpill

• Red Pill is a very well known VM detection tool
  by Rutkowska J.
• Runs a single machine language instruction SIDT
  and analyses its result.

61
(No Transcript)
62
ScoopyNG

• ScoopyNG is a very well known tool for VM
  detection developed by Klein T.
• More reliable tool for VM detection in comparison
  to Red Pill.
• It performs the following checks
• SIDT check
• SLDT check
• SGDT check
• STR check
• IN check (VMware communication channel)

63
(No Transcript)
64
VmDetect

• This is another well known proof of concept VM
  detecting sample that makes use of the VMware
  communication channel to detect VMware Presence.

65
(No Transcript)
66
Backdoor.Win32.SdBot.fmn

• Captured this malware from the internet.
• Employs Memory check and Timing Analysis
  mechanisms .
• In the absence of VMDetectGuard This
  application cannot run under a Virtual Machine.
• In the presence of VMDetectGuard, it behaved
  malicious.

67
(No Transcript)
68
(No Transcript)
69
VMDetectGuard

• Running VMDetect in VirtualPC

• Running VMDetect under masking tool

70
VMDetectGuard

• Running DetectionChecks in VirtualBox

• Running DetectionChecks under masking tool

71
Optimization
Before (sec) After (sec) decrease in time taken
VirtualBox 167.310 112.411 32.08
VirtualPC 294.786 205.953 30.13
VMware 418.642 299.158 28.54
Running Firefox binary under masking tool, in all
the three virtual machines.
72
Results

• Tested VMDetectGuard
• Malwares captured from internet
• Proof of concept tools
• The results obtained after testing is given in
  table.

73
Results
Binary Detection Technique Used Run without tool Run under tool
Virtual Box Virtual Box Virtual Box Virtual Box
VBDetect calls others binaries for individual checks within. Registry Check File and Process Check Instruction Check Detected VirtualBox Did not detect VirtualBox
Rebhip File and Process Check Runs benignly Runs maliciously
VirtualPC VirtualPC VirtualPC VirtualPC
VPCDetect calls others binaries for individual checks within. Registry Check File and Process Check Invalid Opcode Check Detected VirtualPC Did not detect VirtualPC
Backdoor.Win32.SdBot.fmn File and Process Check Invalid Opcode Check Displays a message, This application cannot run under a Virtual Machine Ran maliciously
VMDetect Invalid Opcode Check Detects VirtualPC Does not detect VirtualPC
Trojen.Karsh-252 Invalid Opcode Check Displays a message, This application cannot run under a Virtual Machine Ran Maliciously
74
Conclusion

• Split Personality malware is on a gradual rise.
• Lack of academic research in this field.
• There does not exist any full-fledged tool to
  counter Split Personality Malware.
• We have designed, implemented and tested
  VMwareDetect, a proof of concept tool that
  detects the presence of Vmware.

75
Conclusion

• We also successfully designed and implemented
  VMDetectGuard, a tool to counter Split
  Personality malware.
• It detects as well as tricks the split
  personality binaries.
• Leads to the effective analysis of malware in the
  virtualized environment.
• Increases productivity.

76
Scope for Future Work

• Further testing of more number of malware.
• Tool is currently built for Vmware, VPC and VB.
• Providing solutions for other analysis tools such
  as debugger, sandbox etc.
• The work currently aims at Native binaries
• Can be extended to Managed binaries
• Extended to other operating systems.

77
References

• Rutkowska J. (2004). Red Pill.
  http//invisiblethings.org/papers/redpill.html
  (Nov 20, 2010)
• Quist D, Smith V. (2005). Detecting the
  Presence of Virtual Machines Using the Local Data
  Table. http//www.offensivecomputing.net/files/ac
  tive/0/vm.pdf, (Nov 14, 2010)
•  
• Klein, T. (2005) Scoopy Doo .
  http//www.trapkit.de/research/vmm/scoopydoo/index
  .html (Nov 4, 2010)
• P. Ferrie. Attacks on Virtual Machines. In
  Proceedings of the Association of Anti-Virus
  Asia Researcher Conference, 2007.
• Zhu D. and Chin E. (2007). Detection of VM-Aware
  Malware. http//radlab.cs.berkeley.edu/w/uploads/
  3/3d/Detecting_VM_Aware_Malware.pdf (Dec 1, 2010)
• Carpenter M., Liston T., Skoudis E. (2007).
  "Hiding Virtualization from Attackers and
  Malware". IEEE Security and Privacy, June 2007
• Lau B, Svajcer V. (2008). Measuring virtual
  machine detection in malware using DSD tracer.
  In the Proceedings of Virus Bulletin, 2008
• Balzarotti D., Cova M., Karlberger C., Kruegel C,
  Kirda E, Vigna G. (2010).Effcient Detection of
  Split Personalities in Malware. In the
  Proceedings of 17th Annual Network and
  Distributed System Security Symposium (NDSS
  2010),San Diego, February 2010

78
References

• VMware Inc. (2011), VMware KB Changing a MAC
  address in a Windows virtual machine.
  http//kb.vmware.com/selfservice/microsites/search
  .do?languageen_UScmddisplayKCexternalId100847
  3 (Jan 15, 2010)
• Pin (2004). Pin - A Dynamic Binary
  Instrumentation Tool. http//www.pintool.org/
  (Jan 10, 2010)
• Liston T. and Skoudis E. (2006). On the Cutting
  Edge Thwarting Virtual Machine Detection.
  http//handlers.sans.org/tliston/ThwartingVMDetect
  ion_Liston_Skoudis.pdf (Nov 1, 2010)
• Tiga, 2007. Sourpill, http//www.woodmann.com/co
  llaborative/tools/index.php/SourPill_VM_Detector
  (Nov 4, 2010)
• VMDetect (2005). VmDetect, Detect if your
  program is running inside a Virtual Machine.
  http//www.codeproject.com/KB/system/VmDetect.aspx
  (Jan 4, 2010)
• Guizani, W., Marion, J.-Y., Reynaud-Plantey, D.,
  Bp, C. S. (2009). Server-Side Dynamic Code
  Analysis. Analysis, 2009
• Omella A. (2006). Methods for Virtual Machine
  Detection. http//www.s21sec.com (Nov 24, 2010)
• OECD, Malicious Software (Malware) A Security
  Threat to Internet economy, (2007)
  http//www.oecd.org/dataoecd/53/34/40724457.pdf
  (Oct 20, 2010)

79

• Thank You!