Freaky Friday Joe Morris ATN Web Systems

1
Freaky FridayJoe MorrisATN Web Systems
2
Tutorials

• Building a syslog Infrastructure
• Tina Bird
• Counterpane Internet Security
• System and Network Monitoring
• John Sellens
• Certainty Solutions

3
Technical Sessions

• Keynote Google
• Load Balancers
• The Constitutional Financial Argument Against
  SPAM
• Making Backups Easier With Disk

4
Guru Sessions and BOFs

• Infrastructure Guru
• University Information Systems
• System Monitoring
• Red Hat Linux Users

5
Building a syslog Infrastructure

• Need to watch systems and network
• Exploits
• Worms/viruses
• Unauthorized access
• Hardware failures
• Security no good without logging
• Good stuff gets in the way
• 99 generated by authorized activity
• Successful attacks look normal

6
Building a syslog Infrastructure

• Logs for large systems overwhelming
• Must automate log processing
• Status vs. critical information
• Must establish normal behavior
• Find what deviates from normal

7
Building a syslog Infrastructure

• syslogd
• Common to UNIX and other apps
• Local or remote storage
• No authentication
• UDP based
• Data sent unencrypted
• MS Windows alternatives
• Some TCP alternatives

8
Building a syslog Infrastructure

• Windows logging
• Event Log
• Difficult to parse or script due to GUI
• Auditing is disabled by default
• Vague startup/shutdown messages
• Color coding and categorized
• Only registered applications can log

9
Building a syslog Infrastructure

• Building a central log system
• Create good data
• Collect good data
• Watch for critical events real-time (swatch)
• Batch data for trend analysis
• Make attacks noisier
• Integrity checking
• Perimeter security controls

10
Building a syslog Infrastructure

• Log management
• Time synchronization
• Rotating files
• Long term storage
• Encryption and integrity (protection)

11
Building a syslog Infrastructure

• Legal considerations
• Logs generated as standard business procedures
  carry more weight
• Must assert integrity of logging
• Ease of tampering often used as a reason for
  discarding computer records

12
System and Network Monitoring

• It isnt a service if it isnt monitored. If
  there is no monitoring then youre just running
  software. Tom Limoncelli
• Historical data
• Trends
• Monitor what is important to you
• Too much data can hide real problems

13
System and Network Monitoring

• Minimize false alarms
• Start small and grow as required
• Can monitor anything with right tools
• Your business will dictate kinds of data
• Bosses like pretty pictures

14
System and Network Monitoring

• SNMP
• Simple Network Based Protocol
• Query and control devices
• UDP based
• Implemented on almost any IP device
• First defined in 1980s
• Network management
• Computer system management

15
System and Network Monitoring

• SNMP (continued)
• Idea and protocol are simple
• MIB hierarchies are complex
• v1 lacks security
• v2 offers encryption
• Categories of monitoring packages
• Trackers
• Alarmers
• Other

16
System and Network Monitoring

• Trackers
• MRTG
• Cricket
• RRDTool
• Orca (Orcallator for Sun)
• Tkined

17
System and Network Monitoring

• Alarmers
• SNIPS
• Big Brother (shell based)
• Big Sister (perl based)
• Mon
• NetSaint/Nagios

18
Keynote Google

• Jim Reese, Chief Ops Engineer
• 3,000,000,000 web documents
• 2000 queries/second
• 10,000 Linux servers
• Terabytes of data
• Sites scattered throughout US

19
Keynote Google

• Early infrastructure
• Computer parts on corkboard
• Drives mounted to plastic on top of boards
• Endless spaghetti of cables
• Systems reset when crashed

20
Keynote Google

• Todays infrastructure
• ½ footprint boards, cased
• Cable management
• Lots of fans
• HVAC challenges (HEAT!)
• Power challenges (sequencing)
• Very high density

21
Keynote Google

• Database consistency checks
• Caching
• Redundant Internet links
• Cool screensaver

22
Load Balancers

• Widespread use
• Code stabilizing (more standards)
• Competitive market
• Hardware-based offers high performance
• Faster CPUs helping software-based solutions
  catch up
• Current hardware-based supports about 800
  hits/second

23
Load Balancers

• Client persistence
• Keeps client attached to same server
• Cookies most reliable metric
• AOL mega proxies prevent IP metrics
• Microsoft broke SSLID (2 minute rollover)
• Need to use GB Ethernet
• Can do filtering/firewall

24
Load Balancers

• Source of network problems?
• Easily blamed for unexplained issues
• Yet, very stable and fast
• Document configuration for CYA
• Not understood by everyone

25
The Constitutional Financial Argument Against
SPAM

• BEST presentation at LISA
• Online soon and will present later
• Example of spam numbers
• Hotmail.com
• 1,000,000,000 messages/day
• 80 of it is spam (AFTER filtering)
• Must make spamming expensive
• Steals time and money from everyone

26
Making Backups Easier with Disk

• Tape drives are getting too fast
• More difficult to stream data
• Applications are implementing multiplexing
• Multiplexing impacts restores negatively
• Disks keep getting cheaper
• Large raid arrays simulate tape libraries
• People are still not making off-site copies

27
Infrastructure Guru

• Large organizations still face problems
• Challenge is to keep all systems consistent and
  standardized

28
BOF University Information Systems

• Implementing LDAP
• Securing LDAP
• Kerberizing services
• AFS still popular

29
BOF System Monitoring

• People starting to move away from Big Brother to
  other tools
• BB does not monitor less than 5 minutes
• BB is mostly shell based
• Does not scale very well
• Big Sister proving to be more versatile and will
  work with existing BB setups
• SNMP becoming backbone of monitoring due to
  standards and growing application support
• Nagios better for more complex networks

30
BOF Red Hat Linux Users

• Support issues
• Reasons for using RH Advanced Server
• 8.0 very stable

31
Miscellaneous tidbits

• Attendance of 1200 (low)
• 2 from UNC-CH
• 2 from NC State University
• 1 from Wake Forest University
• Apple Titaniums all over the place
• Expanded wireless coverage
• Student stipends

32
LISA 2003

• 17th Systems Administration Conference
• October 26-31, 2003
• Town and Country Hotel
• San Diego, California