Automated Worm Fingerprinting - PowerPoint PPT Presentation

About This Presentation
Title:

Automated Worm Fingerprinting

Description:

Automated Worm Fingerprinting. Sumeet Singh, Cristian Estan, George Varghese, and Stefan Savage ... with ~500,000 bins (8 bits/bin) 2MB total. Address ... – PowerPoint PPT presentation

Number of Views:195
Avg rating:3.0/5.0
Slides: 88
Provided by: csF2
Learn more at: http://www.cs.fsu.edu
Category:

less

Transcript and Presenter's Notes

Title: Automated Worm Fingerprinting


1
Automated Worm Fingerprinting
  • Sumeet Singh, Cristian Estan, George Varghese,
    and Stefan Savage

2
Introduction
  • Problem how to react quickly to worms?
  • CodeRed 2001
  • Infected 360,000 hosts within 11 hours
  • Sapphire/Slammer (376 bytes) 2002
  • Infected 75,000 hosts within 10 minutes

3
Existing Approaches
  • Detection
  • Ad hoc intrusion detection
  • Characterization
  • Manual signature extraction
  • Isolates and decompiles a new worm
  • Look for and test unique signatures
  • Can take hours or days

4
Existing Approaches
  • Containment
  • Updates to anti-virus and network filtering
    products

5
Earlybird
  • Automatically detect and contain new worms
  • Two observations
  • Some portion of the content in existing worms is
    invariant
  • Rare to see the same string recurring from many
    sources to many destinations

6
Earlybird
  • Automatically extract the signature of all known
    worms
  • Also Blaster, MyDoom, and Kibuv.B hours or days
    before any public signatures were distributed
  • Few false positives

7
Background and Related Work
  • Almost all IPs were scanned by Slammer lt 10
    minutes
  • Limited only by bandwidth constraints

8
The SQL Slammer Worm30 Minutes After Release
- Infections doubled every 8.5 seconds - Spread
100X faster than Code Red - At peak, scanned 55
million hosts per second.

9
Network Effects Of The SQL Slammer Worm
  • At the height of infections
  • Several ISPs noted significant bandwidth
    consumption at peering points
  • Average packet loss approached 20
  • South Korea lost almost all Internet service for
    period of time
  • Financial ATMs were affected
  • Some airline ticketing systems overwhelmed

10
Signature-Based Methods
  • Pretty effective if signatures can be generated
    quickly
  • For CodeRed, 60 minutes
  • For Slammer, 1 5 minutes

11
Worm Detection
  • Three classes of methods
  • Scan detection
  • Honeypots
  • Behavioral techniques

12
Scan Detection
  • Look for unusual frequency and distribution of
    address scanning
  • Limitations
  • Not suited to worms that spread in a non-random
    fashion (i.e. emails, IM, P2P apps)
  • Based on a target list
  • Spread topologically

13
Scan Detection
  • More limitations
  • Detects infected sites
  • Does not produce a signature

14
Honeypots
  • Monitored idle hosts with untreated
    vulnerabilities
  • Used to isolate worms
  • Limitations
  • Manual extraction of
  • signatures
  • Depend on quick infections

15
Behavioral Detection
  • Looks for unusual system call patterns
  • Sending a packet from the same buffer containing
    a received packet
  • Can detect slow moving worms
  • Limitations
  • Needs application-specific knowledge
  • Cannot infer a large-scale outbreak

16
Characterization
  • Process of analyzing and identifying a new worm
  • Current approaches
  • Use a priori vulnerability signatures
  • Automated signature extraction

17
Vulnerability Signatures
  • Example
  • Slammer Worm
  • UDP traffic on port 1434 that is longer than 100
    bytes (buffer overflow)
  • Can be deployed before the outbreak
  • Can only be applied to well-known vulnerabilities

18
Some Automated Signature Extraction Techniques
  • Allows viruses to infect decoy programs
  • Extracts the modified regions of the decoy
  • Uses heuristics to identify invariant code
    strings across infected instances

19
Some Automated Signature Extraction Techniques
  • Limitation
  • Assumes the presence of a virus in a controlled
    environment

20
Some Automated Signature Extraction Techniques
  • Honeycomb
  • Find longest common subsequences among sets of
    strings found in messages
  • Autograph
  • Uses network-level data to infer worm signatures
  • Limitations
  • Scale and full distributed deployments

21
Containment
  • Mechanism used to deter the spread of an active
    worm
  • Host quarantine
  • Via IP ACLs on routers or firewalls
  • String-matching
  • Connection throttling
  • On all outgoing connections

22
Host Quarantine
  • Preventing an infected host from talking to other
    hosts
  • Via IP ACLs on routers or firewalls

23
Defining Worm Behavior
  • Content invariance
  • Portions of a worm are invariant (e.g. the
    decryption routine)
  • Content prevalence
  • Appears frequently on the network
  • Address dispersion
  • Distribution of destination addresses more
    uniform to spread fast

24
Finding Worm Signatures
  • Traffic pattern is sufficient for detecting worms
  • Relatively straightforward
  • Extract all possible substrings
  • Raise an alarm when
  • FrequencyCountersubstring gt threshold1
  • SourceCountersubstring gt threshold2
  • DestCountersubstring gt threshold3

25
Practical Content Sifting
  • Characteristics
  • Small processing requirements
  • Small memory requirements
  • Allows arbitrary deployment strategies

26
Estimating Content Prevalence
  • Finding the packet payloads that appear at least
    x times among the N packets sent
  • During a given interval

27
Estimating Content Prevalence
  • Tablepayload
  • 1 GB table filled in 10 seconds
  • Tablehashpayload
  • 1 GB table filled in 4 minutes
  • Tracking millions of ants to track a few
    elephants
  • Collisions...false positives

28
Multistage Filters
stream memory
Array of counters
Hash(Pink)
29
Multistage Filters
packet memory
Array of counters
Hash(Green)
30
Multistage Filters
packet memory
Array of counters
Hash(Green)
31
Multistage Filters
packet memory
32
Multistage Filters
packet memory
Collisions are OK
33
Multistage Filters
Reached threshold
packet memory
packet1 1
Insert
34
Multistage Filters
packet memory
packet1 1
35
Multistage Filters
packet memory
packet1 1
packet2 1
36
Multistage Filters
packet memory
Stage 1
packet1 1
No false negatives!
37
Conservative Updates
Gray all prior packets
38
Conservative Updates
39
Conservative Updates
40
Detecting Common Strings
  • Cannot afford to detect all substrings
  • Maybe can afford to detect all strings with a
    small fixed length

41
Detecting Common Strings
  • Cannot afford to detect all substrings
  • Maybe can afford to detect all strings with a
    small fixed length
  • A horse is a horse, of course, of course

F1 (c1p4 c2p3 c3p2 c4p1 c5) mod M
42
Detecting Common Strings
  • Cannot afford to detect all substrings
  • Maybe can afford to detect all strings with a
    small fixed length
  • A horse is a horse, of course, of course

F2 (c2p4 c3p3 c4p2 c5p1 c6) mod M
F1 (c1p4 c2p3 c3p2 c4p1 c5) mod M
43
Detecting Common Strings
  • Cannot afford to detect all substrings
  • Maybe can afford to detect all strings with a
    small fixed length

F2 (c2p4 c3p3 c4p2 c5p1 c6) mod M
(c1p5 c2p4 c3p3 c4p2 c5p1 c6 - c1p5)
mod M
(pF1 c6 - c1p5) mod M
44
Detecting Common Strings
  • Cannot afford to detect all substrings
  • Maybe can afford to detect all strings with a
    small fixed length
  • Still too expensive

45
Estimating Address Dispersion
  • Not sufficient to count the number of source and
    destination pairs
  • e.g. send a mail to a mailing list
  • Two sourcesmail server and the sender
  • Many destinations
  • Need to track the distinct source and destination
    IP addresses
  • For each substring

46
Bitmap counting direct bitmap
Set bits in the bitmap using hash of the flow ID
of incoming packets
HASH(green)10001001
47
Bitmap counting direct bitmap
Different flows have different hash values
HASH(blue)00100100
48
Bitmap counting direct bitmap
Packets from the same flow always hash to the
same bit
HASH(green)10001001
49
Bitmap counting direct bitmap
Collisions OK, estimates compensate for them
HASH(violet)10010101
50
Bitmap counting direct bitmap
HASH(orange)11110011
51
Bitmap counting direct bitmap
HASH(pink)11100000
52
Bitmap counting direct bitmap
As the bitmap fills up, estimates get inaccurate
HASH(yellow)01100011
53
Bitmap counting direct bitmap
Solution use more bits
HASH(green)10001001
54
Bitmap counting direct bitmap
Solution use more bits
Problem memory scales with the number of flows
HASH(blue)00100100
55
Bitmap counting virtual bitmap
Solution a) store only a portion of the bitmap
b) multiply estimate by scaling
factor
56
Bitmap counting virtual bitmap
HASH(pink)11100000
57
Bitmap counting virtual bitmap
Problem estimate inaccurate when few flows active
HASH(yellow)01100011
58
Bitmap counting multiple bmps
Solution use many bitmaps, each accurate
for a different range
59
Bitmap counting multiple bmps
HASH(pink)11100000
60
Bitmap counting multiple bmps
HASH(yellow)01100011
61
Bitmap counting multiple bmps
Use this bitmap to estimate number of flows
62
Bitmap counting multiple bmps
Use this bitmap to estimate number of flows
63
Bitmap counting multires. bmp
Problem must update up to three bitmaps
per packet
Solution combine bitmaps into one
64
Bitmap counting multires. bmp
HASH(pink)11100000
65
Bitmap counting multires. bmp
HASH(yellow)01100011
66
Multiresolution Bitmaps
  • Still too expensive to scale
  • Scaled bitmap
  • Recycles the hash space with too many bits set
  • Adjusts the scaling factor according

67
Too CPU-Intensive
  • A packet with 1,000 bytes of payload
  • Needs 960 fingerprints for string length of 40
  • Prone to Denial-of-Service attacks

68
CPU Scaling
  • Obvious approach sampling
  • - Random sampling may miss many substrings
  • Solution value sampling
  • Track only certain substrings
  • e.g. last 6 bits of fingerprint are 0
  • P(not tracking a worm)
  • P(not tracking any of its substrings)

69
CPU Scaling
  • Example
  • Track only substrings with last 6 bits 0s
  • String length 40
  • P(finding a 100-byte signature) 55
  • P(finding a 200-byte signature) 92
  • P(finding a 400-byte signature) 99.64

70
Putting It Together
Address Dispersion Table
key src cnt dest cnt




key cnt




Content Prevalence Table
71
Putting It Together
  • Sample frequency 1/64
  • String length 40
  • Use 4 hash functions to update prevalence table
  • Multistage filter reset every 60 seconds

72
System Design
  • Two major components
  • Sensors
  • Sift through traffic for a given address space
  • Report signatures
  • An aggregator
  • Coordinates real-time updates
  • Distributes signatures

73
Implementation and Environment
  • Written in C and MySQL (5,000 lines)
  • rrd-tools library for graphical reporting
  • PHP scripting for administrative control
  • Prototype executes on a 1.6Ghz AMD Opteron 242 1U
    Server
  • Linux 2.6 kernel

74
EarlyBird
  • Processes 1TB of traffic per day
  • Can keep up with 200Mbps of continuous traffic

75
Parameter Tuning
  • Prevalence threshold 3
  • Very few signatures repeat
  • Address dispersion threshold
  • 30 sources and 30 destinations
  • Reset every few hours
  • Reduces the number of reported signatures down to
    25,000

76
Parameter Tuning
  • Tradeoff between and speed and accuracy
  • Can detect Slammer in 1 second as opposed to 5
    seconds
  • With 100x more reported signatures

77
Performance
  • 200Mbps
  • Can be pipelined and parallelized for achieve
    40Gbps

78
Memory Consumption
  • Prevalence table
  • 4 stages
  • Each with 500,000 bins (8 bits/bin)
  • 2MB total
  • Address dispersion table
  • 25K entries (28 bytes each)
  • lt 1 MB
  • Total lt 4MB

79
Trace-Based Verification
  • Two main sources of false positives
  • 2,000 common protocol headers
  • e.g. HTTP, SMTP
  • Whitelisted
  • SPAM e-mails
  • BitTorrent
  • Many-to-many download

80
False Negatives
  • So far none
  • Detected every worm outbreak

81
Inter-Packet Signatures
  • An attacker might evade detection by splitting an
    invariant string across packets
  • With 7MB extra, EarlyBird can keep per flow
    states and fingerprint across packets

82
Live Experience with EarlyBird
  • Detected precise signatures
  • CodeRed variants
  • MyDoom mail worm
  • Sasser
  • Kibvu.B

83
Variant Content
  • Polymorphic viruses
  • Semantically equivalent but textually distinct
    code
  • Invariant decoding routine

84
Extensions
  • Self configuration
  • Slow worms

85
Containment
  • How to handle false positives?
  • If too aggressive, EarlyBird becomes a target for
    DoS attacks
  • An attacker can fool the system to block a target
    message

86
Coordination
  • Trust of deployed servers
  • Validation
  • Policy

87
Conclusions
  • EarlyBird is a promising approach
  • To detect unknown worms real-time
  • To extract signatures automatically
  • To detect SPAMs with minor changes
  • Wire-speed signature learning is viable
Write a Comment
User Comments (0)
About PowerShow.com