Live Forensics Tutorial Part 3: Live Forensics

1
Live Forensics TutorialPart 3 Live Forensics

• Frank Adelstein, Ph.D.
• Technical Director, Computer Security, ATC-NY
• GIAC-certified Digital Forensics Investigator
• Golden G. Richard III, Ph.D.
• Associate Professor, Dept. of Computer Science,
  University of New Orleans
• GIAC-certified Digital Forensics Investigator
• Co-Founder, Digital Forensics Solutions, LLC

2
Live Forensics Tutorial
Undead

• Part 3 Live Forensics

3
Section Overview

• Goals
• Constraints
• Typical scenario
• Information available
• Analysis
• Wrap up

4
Goals

• Whats happening now
• Who is doing what
• Obtain another piece of the puzzle
• Help focus static analysis (what parts of the
  disk)
• Get passwords, unencrypted data, etc.
• Information can be used to
• Reconstruct sessions (e.g., web, ftp, telnet, IM)
• Find files (downloaded or accessed through
  network drives)
• Find passwords
• Identify remote machines

5
Why Live Forensics?

• Big disks
• Disk capacity keeps increasing (Oct06 500Gb for
  158) faster than processors
• Terabyte systems are big and common
• Searching (or indexing) takes time
• Mirroring takes time
• Minimal downtime (mission critical sys)
• Harder to seize systems (even with court order)
• Provide context for static analysis
• Low-profile examination
• Long data lifetimes
• Some data is only in RAM

6
Why The Lifetime of Volatile Data

• Chow (Shredding your Garbage paper, see
  references)
• Lifetime of volatile data
• usernames
• passwords/encryption keys
• credit card numbers
• private conversations
• Forensics analysis of physical memory reveals
  private data, even weeks after use
• Both a forensics and a privacy issue

7
Lifetime (2)

• Most current systems take no steps to overwrite
  sensitive data in memory
• Most applications that handle sensitive data
  werent specifically designed to deal with
  sensitive data
• e.g., Word processors
• Financial data, medical records, lists of
  passwords
• Password entered into web browser may be
  duplicated dozens of times
• kernel buffers, window manager, application
  buffers, dynamic memory allocator
• If software crashes, core dump may leak this
  information
• Live forensic analysis can reveal sensitive data
  months after last use

8
Chow Data Lifetime

• Ideal Lifetime period of time data is actually
  in use
• first write after allocation ? last read before
  deallocation
• Natural Lifetime period of time that data
  remains readable
• first write after allocation ? first write after
  reallocation (first overwrite)
• Secure Deallocation Lifetime zero on
  deallocation
• first write after allocation ? deallocation (when
  it is zeroed)

9
Lifetime Experiment

• Linux machine, 1GB RAM
• Windows machine, 1GB RAM
• Linux server, 256MB RAM
• Linux version of experiment
• 64MB buffer filled with 20 byte stamps
• Allocate and release
• Each stamp
• magic number, serial number, checksum
• Windows version
• 4MB buffer transmitted between machines

10
Lifetime Experiment (2)

• Immediately after allocation
• 2-4MB of stamps remain
• 14 days later
• 23KB 3MB of stamps remain
• Additional 14 days on Linux server
• 7KB of stamps remain
• Most remaining stamps trapped in blocks of
  memory in the kernel slab allocator
• Reboot on Thinkpad T30 laptop stamps remain
  after 30 seconds w/o power (!!)

11
Lifetime Experiment (3)
12
Constraints

• Minimize changes and artifacts
• analogy to physical data
• must balance tool footprint to usefulness
• Timely evidence acquisition and analysis
• Need access to the system
• Need to minimize impact on system, particularly
  if its mission-critical

13
Live Forensics Dangers

• Most live forensics tools rely on the OS to
  provide some basic services
• Reads of physical memory
• Disk dumping
• User-level rootkits
• Shallow trickery
• Modify system commands (e.g., ls, ps, du, df)
• Change disk space statistics, list of running
  processes, etc.

14
Live Forensics Dangers (2)

• Kernel-level rootkits
• Deep trickery
• Modify OS to produce arbitrary results
• Allows files, blocks of physical memory, etc. to
  be hidden even if trusted executables are used
• Disk drivers
• Hacking virtual memory system
• Replacement of shared libraries
• Affects even your trusted executables unless
  theyre statically linked!

15
Minimize Changes and Artifacts

• Small footprint (using trusted software)
• Try not to change anything
• But everything changes, all the time
• Minimize changes to evidence
• Record all steps taken and artifacts
• Low profile, minimize detection
• Artifacts can be explained
• analogy detectives finger-prints on ransom note

16
Timely Evidence Acquisitionand Analysis

• First response/triage
• Looking for evidence, or for what computers,
  disks, directories, etc. may contain evidence.
  Examples
• 30 servers, search warrant says image 1
• search warrant for quick hash scan to find
  sufficient cause to get a broader warrant
• Looking for context for static analysis

17
Need Access to the System

• Before investigation
• Use preinstalled agents
• Requires prior access (plan to be attacked?)
• Agents must not have become compromised
• During investigation
• Need credentials for log-in
• Must use known good binaries
• See previous on artifacts

18
Section Overview

• Goals
• Constraints
• Typical scenario
• Information available
• Analysis
• Wrap up

19
Typical Scenario

• Triage/quick peek
• Justify larger warrant
• Ongoing crime (in progress)
• Running an illegal server
• Trojan horse defense (support/refute)
• Whats going on with this machine?
• Machine running slow
• Lots of suspicious disk or network activity

20
Scenario Triage

• Limited time. Want to answer
• Is there a problem?
• What machine(s) are affected?
• What disks need to be imaged?
• What is running on the system?
• Focus on where the problem is the worst and the
  evidence is the most abundant

21
Scenario Justify Warrant

• Non-disruptive, quick hash search
• Look for the presence of a file or tools
• Match against known hashes
• Look for email addresses, etc.
• Investigator does not take machine down, disrupt
  service, or raise suspicion
• Investigator does not see the actual files
• If justified, can return for full investigation
• Risk of evidence damage, must be careful

22
Scenario Ongoing Crime

• Want to catch them in the act
• Show how things change (web pages, file access
  times, registry, memory, etc.)
• Want to understand
• How they got in
• What they compromised
• Where they are
• Who they are

23
Scenario Illegal Server

• Compare whats running on the machine and what
  ports are open to a network scan (e.g., nmap et
  al)
• What services does the world see visible on the
  computer?
• Some techniques are subtle (port knocking), but
  most are not
• Look at network traces to see who is talking to
  the computer (be careful of legal issues)

24
Scenario Trojan Defense

• Its not my fault, someone else is controlling
  my computer!
• Support or refute the claim
• Are there traces of (known) Trojans?
• Are there unusual network connections?
• What is running on the machine (that can be
  seen)?
• Any indication that something is hidden?
• Enough other evidence/activities to corroborate?
• Still new (1st used in court in UK in 2003)
• Still tricky (often Trojans/viruses are present)

25
Section Overview

• Goals
• Constraints
• Typical scenario
• Information available
• Analysis
• Wrap up

26
Information Available

• Running processes
• open DLLs
• registry
• file handles
• Open files
• Network connections
• Memory
• Regular disk files
• Images of entire disk
• Live disk imaging
• (a.k.a. shooting a moving target)
• Deleted files
• Live file carving

27
Information Available (2)

• Unencrypted document fragments
• Encryption keys for whole-disk encryption schemes
• Copies of volatile-only malware (for
  disassembly/investigation)

28
Running Processes

• Windows
• Open DLLs
• File handles
• Network connections
• Registry
• Unix
• Open files
• Network connections
• Access to corresponding EXE, even if deleted
• Command line

29
Open Files, Network Connections

• On Unix, lsof is a useful tool
• On Linux, /proc/nnnn has information on process
  nnnn, including open files, executable, and more
• Netstat under Windows for watching open network
  connections
• More details on these in a few slides

30
Memory

• Process memory
• Can yield passwords
• Can yield document fragments
• Can yield unencrypted documents
• Kernel memory
• Search for hidden processes
• String searches
• Most brute force technique

31
Disk

• Regular files
• Live imaging (moving target?)
• Live file carving to retrieve deleted files

32
Virtual Machines

• Freeze VM
• Copy/snapshot disks, memory, even screen
• Resume execution

33
Section Overview

• Goals
• Constraints
• Typical scenario
• Information available
• Analysis
• Wrap up

34
Analysis

• What's going on?
• Are things not right, processes or files hidden,
  or disk encryption in use?
• What's lingering in memory (a lot) and process
  memory dumps?
• Dumping OS structures
• (e.g., determining which areas of swap space
  correspond to a particular process)

35
Analyzing Processes

• Finer-grained then dumping entire RAM
• Easier to make sense of virtual address space for
  a process than physical memory
• More likely to find contiguous application
  structures
• Open files
• Open network connections
• Reliance on DLLs

36
Process/System Analysis Tools

• sysinternals tools for Windows
• FileMon shows filesystem activity in real time
• PSMon watch process/thread creation in real
  time
• PsFile - shows files opened remotely
• PsKill - kill processes by name or process ID
• PsInfo - list information about a system
• PsList - list detailed information about
  processes
• PsLoggedOn - see who's logged on locally and via
  sharing
• PsLogList - dump event log records
• PsPasswd - changes account passwords
• PsService - view and control services
• PsSuspend - suspends processes
• Handle shows which files are opened by which
  processes
• RegMon see registry activity in real time
• ListDLLs show loaded DLLs

37
Process Analysis Tools (2)

• ntsecurity.nu
• Pmdump dump process memory, given pid
• ListDrivers list loaded kernel drivers
• ListModules list EXEs / DLLs associated with a
  process
• PromiscDetect determine if network adaptors are
  running in promiscuous mode
• There are many others

38
psinfo
39
pslist
40
handle
41
filemon
42
psfile
43
promiscdetect
44
promiscdetect (Wireshark running!)
45
psloggedon
46
Dump of pgptray
Process info
Mem dump
Plaintext
From E. Casey, Practical Approaches to
Recovering Digital Evidence
47
netstat -a
Heart failure?!
48
netstat a -b
Whew!
49
Linux Process Listing
Partial output of ps aux less
50
Linux lsof
Partial output of lsof less
51
Linux Detailed Process Information
52
Linux Detailed Process Information (2)
53
Linux Detailed Process Information (3)
54
Linux Detailed Process Information (4)
55
Linux Detailed Process Information (5)
56
Integrated Live Analysis Toolset

• OnLine Digital Forensic Suite
• Tools for live investigation, data acquisition,
  and analysis
• Web-based interface

57
Live Forensics
58
Create Inquiry
59
Create (2)
60
Target Information
61
Confirm Information
62
Target Password
63
Initial Acquire
64
Initial Acquire (2)
65
Initial Acquire Complete
66
General System Information
67
IP Interface Information
68
Data Analysis
69
Network Ports
SMB file server
vmware phoning home to check for updates
70
Running Processes
71
Detailed Process Information
72
Network Connections by Process
73
Another Process
74
Open File Handles
75
Other Tools and Techniques
76
Software-based Acquisition of Physical Memory

• dd command
• Under Windows, use \\.\PhysicalMemory device
• (Not usable in user-space in XP SP2 )
• Under Linux
• /dev/mem
• Physical memory
• /proc/kcore
• Kernel virtual memory
• Problem must rely on OS to provide physical
  memory dump
• OS might be compromised
• After acquisition, interpretation is another
  issue!

77
Software-Based (2)

• Virtual machines (e.g., VMWare)
• State of a virtual machine, including memory and
  disk, can be extracted
• Primary drawback is that the machine under
• investigation must be a virtual machine
• Hibernation files
• Snapshot of physical memory
• Can hibernation process (e.g., writing of
  physical memory) be subverted?

78
Problems with Software-based What Else?

• Software-based solutions for memory imaging
  typically require loading software
• Probably erases some evidence
• Requires at least limited trust of the OS
• Hardware-based solutions
• Tribble
• Carrier (see references)
• Firewire hacks
• Maximillian Dornseif (see references)
• USB hacks?

79
Memory Dump Analysis

• Assuming that a trusted dump of system memory
  has been obtained, now what?
• Analyze dump to extract information about
  processes, threads, open files, sockets, etc.
• Most interesting things in the kernel are
  objects (e.g., structures)
• These objects likely have many pointers
  hanging off
• First method analyze lists/tables of kernel
  structures
• Second method do carving for interesting
  objects

80
Following Lists

• Windows Memory Forensics Toolkit (wmft)
• http//strony.aster.pl/forensics/
• kntlist
• Not released?
• MemParser
• Not released?
• Basic idea
• From symbol table for kernel, determine location
  of interesting tables/lists and enumerate
• e.g., for Linux, System.map file created when
  kernel is compiled
• Can locate system call table, first process, etc.
• One challenge DKOM

81
Direct Kernel Object Manipulation

• Idea kernel components have access to kernel
  memory
• (at least in non-microkernel OSs!)
• Malicious kernel component can modify kernel
  structures to hide, e.g., processes
• Good discussion of DKOM here
• http//www.blackhat.com/presentations/win-usa-04/b
  h-win-04-butler.pdf

82
Rootkits Episode IV

• A New Hope?

83
FU Rootkit and Descendents
pid 17
Doubly-linked process list in Windows kernel
Processes continue to run because
Windows scheduler handles threads, not processes
C\gt fu ph 17
pid 17
84
Lists and DKOM

• Not hopeless
• For process hiding case, can look deeper than the
  process list
• Look at lists of threads, make sure they match up
  with processes in the process list
• More difficult, because offsets of important
  kernel structures for this effort are
  version-specific?
• Walk all process lists, including those used by
  the scheduler
• Walk handle lists

85
Shadow Walker

• Details fairly complicated
• Look away if necessary ?
• Relies on split translation lookaside buffer
  (TLB) on 32bit Intel processors
• TLB for data accesses
• TLB for instruction accesses
• Poisons TLB using a hacked page fault handler
• Means can hide processes
• Hidden processes execute w/ no problem
• Read accesses to (virtual) memory are diverted
• Cant see code for executing process!
• Again, not hopelessmay be able to, e.g.,
  validate page fault handler

86
Schuster Approach Carving

• Want to find all processes/threads in memory dump
• Normal
• Hidden
• Terminated
• Dont rely on kernel lists/tables
• Search memory dump for objects that look like
  processes/threads

87
Schuster (2)

• Important ideas
• Memory is needed to store kernel objects
• Use info about how kernel performs allocation to
  find blocks of allocated memory
• Kernel objects have an OBJECT_HEADER structure
• Further, processes and threads have a
  DISPATCH_HEADER, used for scheduling/synchronizati
  on
• Use these ideas to develop templates for
  discovering interesting structures in a Windows
  memory dump
• Walk memory dump in 4K steps

88
Schuster POOL_TAG
PoolTag 0xe36f7250 for processes PoolTag
0xe5726854 for threads
89
Schuster OBJECT_HEADER
Known values for live/dead processes and threads!
Also know information about lengths associated
with name.
90
Schuster Additional Tests

• Know some characteristics of DISPATCH_HEADER
• Know some characteristics of ETHREAD structures
  (e.g., pointers to owning process,
  DISPATCH_HEADER w/ type thread, )

For a certain Windows version, size field is
constant for a particular object type.
91
Schuster Results

• Perl-based PTFinder
• Visualization using Graphviz
• More on thiswill play a role in the demo later

92
(No Transcript)
93
(No Transcript)
94
RAM Carving

• Process dump of MSN Messenger yields chat message
  fragments
• Content-Type text/plain charsetUTF-8
• X-MMS-IM-Format FNMS20Shell20Dlg EF CO0
  CS0 PF0
• Are you enjoying Mardi Gras this year? I hear
  that the crowds are smaller, but that the general
  spirit is high
• Can construct patterns based on these fragments
  and apply file carving techniques to discover
  fragments of chat sessions days or weeks old in
  memory dumps

95
Live Disk Imaging

• Can image disks live using essentially the same
  tools as for preservation step in dead analysis
• Problem Moving target, files changing
• On a relatively quiet system, image may be a
  reasonable representation
• Win dd if\\.\PhysicalDrive0 ofe\pd0.dd
• Linux dd if/dev/hdc of/mnt/images/hdc.dd

96
Live File Carving

• Similarly, can perform file carving against live
  block devices using standard tools
• e.g., Scalpel, Foremost
• Beyond consistency problem, need sufficient
  available storage
• Next Generation ? In-place, live carving

97
In-Place File Carving
client applications
scalpel_fs
FUSE
preview database
nbd client
local drive
network
nbd server
remote drive
Scalpel
G. G. Richard III, V. Roussev, V. Marziale,
In-Place File Carving, submitted to the Third
Annual IFIP WG 11.9 International Conference on
Digital Forensics, 2007.
98
FUSE (Filesystem in User Space)
Filesystem Implementation
dd if/evidence/DEC/img.dd ofcopy.dd
FUSE library
read()
user space
C library
C library
kernel space
Linux Virtual File System Interface (VFS)
FUSE
ext3
reiserFS
99
Section Overview

• Goals
• Constraints
• Typical scenario
• Information available
• Analysis
• Wrap up

100
(Very) Hard Problems

• Can you trust the O/S?
• kernel level rootkits
• can you get around itor under it?
• can you know when you can trust?
• Whole disk encryption
• BitLocker, EFS, CFS, TCFS, sfs, etc.
• pull the plug and then, ooops
• What do you do with a memory dump?
• beyond string searches
• reconstruct processes (running and dead?)

101
The Future (maybe)

• Live forensics will be broadlyaccepted (in
  court)
• Traditional forensics, as performed today, will
  not be practical due to huge disks
• Live forensics will provide increasingly
  essential information for investigations
• No one will be able to capture all details of a
  digital crime (just like in the physical world)
• Memory will become more than just string
  extraction
• Rootkits will be mostly a solved problem or will
  destroy all digital forensic integrity

102
References

• Links and other
• resources

103
Traditional Forensics

• Books
• Digital Evidence and Computer Crime (E. Casey,
  Academic Press)
• Computer Forensics and Privacy (M. Caloyannides,
  Artech House)
• File System Forensic Analysis (B. Carrier,
  Addison-Wesley)
• Forensic Discovery (D. Farmer, W. Venema,
  Addison-Wesley)
• Websites
• http//www.dfrws.org
• Lots of references related to digital forensics,
  including a link to an interesting e-journal
• http//www.ijde.org/
• International Journal of Digital Evidence
• http//www.tucofs.com/tucofs/tucofs.asp?modemainm
  enu
• Collection of forensics-related software
• http//www.digitalforensicssolutions.com
• Home of Scalpel (file carving software)

104
Traditional (2)

• Commercial and open-source digital forensics
  software
• Sleuthkit / Autopsy
• Scalpel
• Foremost
• Encase
• FTK (Forensics Tool Kit)
• ILook (law enforcement only)
• WinHex
• lots more
• Open source digital forensics software project
• http//www.opensourceforensics.org/

105
Live Forensics

• Adelstein, F. 2006. Live forensics diagnosing
  your system without killing it first. Commun. ACM
  49, 2 (Feb. 2006), 63-66. DOI http//doi.acm.org/
  10.1145/1113034.1113070
• Carrier, B. D. 2006. Risks of live digital
  forensic analysis. Commun. ACM 49, 2 (Feb. 2006),
  56-61. DOI http//doi.acm.org/10.1145/1113034.111
  3069
• Carrier, B., J. Grand, A Hardware-based Memory
  Acquisition Procedure for Digital Investigations,
  Digital Investigation (2004)1.
• Carvey, H., Windows Forensics and Incident
  Recovery, Pearson Publications, July 2004, ISBN
  0321200985
• Casey, E., Practical Approaches to Recovering
  Encrypted Digital Evidence, International
  Journal of Digital Evidence, (2002) 13.
• Jim Chow, Ben Pfaff, Tal Garfinkel, and Mendel
  Rosenblum, Shredding Your Garbage Reducing Data
  Lifetime Through Secure Deallocation, Proceedings
  of the 14th USENIX Security Symposium, 2005.
• M. Dornseif, FireWire - all your memory are
  belong to us, http//md.hudora.de/presentations/.
  
• Garfinkel, S., Forensic Feature Extraction and
  Cross-Drive Analysis, 6th Annual Digital Forensic
  Research Workshop (DFRWS 2005), West Lafayette,
  IN, 2006
• Schuster, A., Searching for Processes and Threads
  in Microsoft Windows Memory Dumps, 6th Annual
  Digital Forensic Research Workshop (DFRWS 2006),
  West Lafayette, IN, 2006.

106
Live Forensics

• http//www.vidstrom.net/
• http//www.usenix.org/events/sec05/tech/full_paper
  s/chow/chow.pdf (14th Usenix Security)
• http//www.security-assessment.com/Presentations/A
  uscert_2006_-_Defeating_Live_Windows_Forensics_DB_
  v1.8.ppt
• http//md.hudora.de/presentations/firewire/2005-fi
  rewire-cansecwest.pdf
• http//forensic.seccure.net/
• http//www.knoppix.net
• http//www.gcn.com/print/25_22/41502-1.html
  (Special Report, Live forensics is the future
  for law enforcement)
• http//news.com.com/2100-7349_3-5092781.html
  (U.K. teen acquitted with Trojan defense, Oct.
  17, 2003)
• http//www.newsmax.com/archives/articles/2003/8/12
  /204345.shtml (The Trojan Horse Defense in Child
  Pornography, Aug. 13, 2003)

107
Network Forensics

• The Tao of Network Security Monitoring Beyond
  Intrusion Detection, Richard Bejtlich,
  Addison-Wesley, 2004
• Intrusion Signatures and Analysis, Mark Cooper et
  al, SAMS, 2001
• Network Intrusion Detection, Stephen Northcutt
  and Judy Novak, 2002
• Books by W. Richard Stevens
• TCP/IP Illustrated, Volume 3 TCP for
  Transactions, HTTP, NNTP, and the UNIX Domain
  Protocols, Addison-Wesley, 1996
• TCP/IP Illustrated, Volume 2 The Implementation,
  Addison-Wesley, 1995
• TCP/IP Illustrated, Volume 1 The Protocols,
  Addison-Wesley, 1994
• tcpdump, www.tcpdump.org
• Wireshark (aka Ethereal), www.wireshark.org
• WinHex, www.winhex.com

108
END OF PART 3NEXT Putting it all together
(demo)