Practical Applications in Security Lessons learned from recent security incidents in the Computer Science Department

1
Practical Applications in SecurityLessons
learned from recent security incidents in the
Computer Science Department
CSCI 6268
December 6, 2005

• Chris Schenk
• Mark Dehus
• Michael Oberg

2
Outline

• History and Background of CS Attacks (Chris)
• Fall 2004 Compromise
• Fall 2005 Compromise
• Details on the Recent Compromise (Mark)
• Detection
• Recovery
• Mitigation
• Computational Science Center Perspective
  (Michael)
• Systems Inspection
• Security Tools Used By The CSC

3
You know its a bad day when
4
It is possible

• Pre-attack on the CSEL, Fall 2004
• Request for user login name change
• Password file distribution from central source
• Rdist hosts.allow
• poof
• Password file corruption
• 600 accounts lost
• Empty root password
• Restore from single nightly backup copy (luckily!)

5
That someone may be

• Reload of machine duos to support ldap (with
  help from Dirk Grunwald)
• All machines on external network accessible via
  ssh
• toucan, cardinal, woodpecker, etc
• ssh woodpecker Remote host key has changed!
• Not initially thought of as a threat (woops)
• Host ID in base64 form, not in hex (strange)
• Two days later, logged into duos
• who command (some.company.ro)
• The Romanians are upon us

6
Doing something nasty!!

• freeshell.org email account shutdown
• Contact ITS to stop mail forwarding
• Attempt to email sysadmin at freeshell.org
• His response

Chris - Thanks for writing. Your colorado.edu
account may been compromised as well. Please
change your password. There is currently an
investigation being done on you by the FBI and
you may be contacted by a special agent Matthew
Rowald concerning some efnet IRC activity. I'm
not even sure if you will get this, but I
certainly wasn't going to email it to your SDF
account. I only saw connections from comcast and
colorado.edu, which leeds me to believe that the
individual (or in the FBI's case, you) are using
your colorado.edu acct. Can you please change
your passwords? Also, you might want to look
into your ssh versions used at colorado.edu.
7
Response

• Almost all machines compromised (90)
• Entire network bogged down with flooding traffic
  from compromised machines
• ls command taking 10 seconds to complete
• Single 100Mbit copper line to connect all CSEL
  machines to the world
• Very unhappy databases class (some were glad
  homework couldnt get done, others were not)
• Contacts from a few places around the world about
  attempted dictionary attacks from CSEL machines.

8
Remediation

• Bad Mojo
• No logs saved
• Unsure of how many accounts were logged (anyone
  using ssh that week)
• All account passwords expired to force password
  changes (didnt completely work)
• Good Mojo
• Reload of machines with Fedora Core 2
• Replacing RedHat 9 (1 ½ years after support
  dropped for OS)
• All machines moved to internal subnet
• No machines externally accessible except server
  csel.cs
• No further incidences seen after reload and
  NATing of machines

9
John the Ripper

• Password cracking utility
• Ran on password file after remediation
• 22 account passwords discovered after 2 hours
• 6 found immediately (one dead guy)
• 2800 accounts leftover in the lab
• Only 300 active users for the year

10
Fall 2005 Compromise

• Dirk Grunwalds machines compromised
• Local user with full sudo access logged in from
  Amsterdam
• Sshd immediately replaced on all machines
• Took one full day to figure out something was
  wrong

11
CSEL Second Compromise Overview

• CSEL Background
• Attack Timeline
• Detection
• Lockups and Errors
• Log Auditing
• Recovery
• Data migration
• Server Recovery
• Workstation Recovery
• Mitigation
• Locking Down OS
• Locking Down Dameons
• Automated Break-in detection

12
csel.cs.colorado.edu

• Primary public server
• Used for public ssh shell access
• Hosts the CSEL website student websites
• Has e-mail capabilities

13
duos.cs.colorado.edu

• Duos is the LDAP File Server
• This means it contains authentication data and
  home directories
• It also manages 'Self-Account Creation' from
  PLUS
• It has access to Uniquid (Unique User ID system)

14
lesc.cs.colorado.edu

• Lesc is used as a backup mirror of duos
• At the time Lesc was being used to phase out
  duos
• Important information was on it.. such as
• Recent LDAP Directory Dump (Contained password
  hashes)
• Workstation Image (The one I used that same day)
• Recent Backup of Home directories

15
Csel.cs.colorado.edu Locked up!
.. We were oblivious ..
16
CSEL Attack Timeline (What we saw)

• Friday, August 19th
• csel.cs.colorado.edu locked up at 1030am
• Computer did not want to reboot (Hanging on
  Shorewall)
• Shorewall was disabled and computer was booted
  with basic iptables
• Shrugged off as bad luck
• Sunday, August 21st
• Chris is concerned and checks sshd on servers by
  executing.. ps aux grep sshdroot 2092
  0.0 0.0 4668 264 ? Ss Aug21 000
  /usr/sbin/sshd
• Everything looks OK!
• Monday, August 22nd
• Check of log files shows a compromised account
  logged in on Friday!!
• Tuesday, August 23rd
• We determined that all the attacker was able to
  do was crash csel.cs

17
Lesc.cs.colorado.edu
18
Finding out what happened (Log Auditing)

• First sign.. csel.cs.colorado.edu was locked up
  for no reason
• Question Who How - gt /var/log/messages
  /var/log/secure
• Two ways to find answers
• Find by Hand
• Automated Log Analysis
• This question was answered by a mix of both.. but
  mostly by hand (I love Grep!)

Dec 5 154330 csel OTRS-PM-1012240
NoticeKernelSystemPostMasterFollowUpRun
FollowUp Article to Ticket 2005120510000011
created (TicketID128, ArticleID410). Dec 5
155003 csel crond(pam_unix)5839 session
closed for user otrs Dec 5 155005 csel
crond(pam_unix)5837 session closed for user
root Dec 5 155041 csel unix_chkpwd5906
password check failed for user (buschk) Dec 5
155041 csel sshd(pam_unix)5904
authentication failure logname uid0 euid0
ttyssh ruser rhostcamras.colorado.edu
userbuschk Dec 5 155041 csel sshd5904
pam_ldap error trying to bind as user
"uidbuschk,ouPeople,dccsel,dccs,dccolorado,dc
edu" (Invalid credentials) Dec 5 155050 csel
unix_chkpwd5909 password check failed for user
(trumpler) Dec 5 155050 csel
sshd(pam_unix)5907 authentication failure
logname uid0 euid0 ttyssh ruser
rhostcamras.colorado.edu usertrumpler Dec 5
155050 csel sshd(pam_unix)5910 session
opened for user trumpler by (uid0) Dec 5
155051 csel automount5937 mount(nfs) nfs
mount failure duos/home/nsa on /home/nsa
19
CSEL Second Compromise (What the logs told us)

• Friday, August 19th
• Attempted logins using previously collected
  usernames started at 5AM
• Csel.cs Lesc.cs were attacked simultaneously
• Attacker was io.physics.tamu.edu (obviously
  compromised)
• Payback.. Buffs beat Texas AM 40 21! That'll
  teach um!
• 35 Usernames were attempted, 4 were successful
• When io discovered successful login it alerted
  attacker
• Attacker then used login information from
  81.181.251.2
• 81.181.251.2 Belongs to RIPE.net (Large ISP) in
  Amsterdam

20
Lesc Damage

• Attackers got into Lesc using root password
• SSHD was replaced with chroot jailed trojan
• Contents of /root
• rhel4-workstation-image.tgz
• ldap-data-backup.ldif
• All attacker did was replace SSHD!

21
Duos.cs.colorado.edu

• Hackers did not bother with Duos
• We got lucky.. Would have been very easy to gain
  root access
• PHP was setuid root due to Uniquid requirements
• Who cares? Duos has access to Mailing Addresses!

cat 0wnDuos.php !/usr/local/sbin/php lt?php exec
(passwd root) ?gt ./0wnDuos.php Changing
password for user root New password i0wnU
su Password i0wnu rm / -rf ( or worse )
22
Recovery

• Copied important data to another computer
• Reloaded all three servers
• Reloaded all workstations
• Easy but time consuming (30 hours)

23
Mitigation (General)

• Don't use weak passwords and change them often!
• SSH Restrictions and Rate Limiting on incoming
  connections
• Use nosuid, nodev, and noexec to restrict files
• Monitor log files frequently with automatic
  analyzer
• Logcheck
• Logwatch
• Considering using RSA/DSA keys instead of
  passwords

24
Mitigation (duos.cs.colorado.edu)

• Self-Account Creation scripts still need setuid
• So-So Solution.. C setuid wrapper designed to
  call uniquid scripts only
• Better solution.. rewrite all scripts in perl
• Prefect solution.. fix uniquid so it does not
  need setuid
• Passwords are no longer allowed.
• Fine grained sudo controls and su restrictions
• SSH restrictions
• Blocked Internet Traffic Remote Logins
• Installed Intrusion Detection Software

25
Mitigation (csel.cs.colorado.edu)

• Passwords are allowed but restrictions are put on
  pam_cracklib
• Still cant restrict passwords from SAC.. Yet
• Fine grained sudo controls and su restrictions
• SSH Restrictions
• Rate Limiting on incoming connections
• Installed Intrusion Detection Software
• Put restrictions on file system

26
Mitigation (lesc.cs.colorado.edu)

• Removed from Internet completely
• Applied same restrictions as Duos
• Installed Intrusion Detection Software

27
Outline

• Computational Science Center Perspective on the
  Compromise
• Systems Inspection immediately following
  notification of the compromise
• Security Tools Used By The CSC

28
Computational Science Center

• 100 systems under Dr. Henry Tufo
• http//csc.cs.colorado.edu/
• Three clusters, plus several additional machines
• 65 node Intel based Computational Cluster
  (hemisphere)
• 28 node PowerPC based Computational Cluster
  (occam)
• 3 node Storage Cluster
• 1 x86_64 Big RAM (8GB) machine
• Management Server
• License Server
• Log / Security Server
• 8 externally accessible, remainder are on a
  private LAN

29
CSC Systems Inspection

• This work done by Sean McCreary, Matthew
  Woitaszek and myself
• Received notification of compromise of unknown
  number of systems in the CS department
• Manually audited logs to determine if anyone
  successfully logged in to our servers, found
  this
• Aug 19 060426 hemisphere sshd4212 Accepted
  password for grunwald from 165.91.181.6 port
  39256 ssh2
• Aug 19 061132 hemisphere sshd6929 Accepted
  password for grunwald from 81.181.251.2 port 2412
  ssh2
• Aug 19 063627 hemisphere sshd7861 Accepted
  password for grunwald from 81.181.251.2 port 2441
  ssh2

30
CSC Systems Inspection

• Hacker Successfully logged into our system, now
  what?
• Treat all machines as if they are hacked, full
  inspection
• If any machines are found to be compromised,
  remove from network and do full recovery
• Management server
• Only Administrator accounts
• Persistent connections to other main servers
  under GNU screen, a terminal multiplexer
• All Administrators use password protected RSA
  keys on known client machines to connect to CSC
  machines
• Trojan SSHD cant capture administrator passwords
• Administrators use sudo exclusively

31
CSC Systems Inspection

• Attempted logins from all 32 compromised accounts
• Only user grunwald was successful
• First, make sure grunwald is not logged in and
  lock the account
• sudo mkpasswd grunwald
• Change shell to /sbin/nologin
• Examine the following looking for any suspicious
  files
• /home/grunwald
• /tmp
• /var/tmp
• Examine the process table
• ps auxw

32
CSC Systems Inspection

• Manually Run Tripwire, Osiris, Samhain
• These are Filesystem Integrity Scanners
• Example Tripwire Output (Osiris is similar)

Host name hemisphere Host
IP address 128.138.207.210 Policy
file used /etc/tripwire/tw.pol Conf
iguration file used /etc/tripwire/tw.cfg Datab
ase file used /var/lib/tripwire/hemisp
here.twd Command line used
/usr/sbin/tripwire --check --------------------
--------------------------------------------------
--------- Rule Name Critical configuration
files Severity Level 100 ------------------------
--------------------------------------------------
----- Modified "/etc/crontab" "/etc/fstab" "/etc/
motd"
33
CSC Systems Inspection

• These Filesystem Integrity Scanners verify that
  the system binaries and configuration files are
  not compromised
• Attacker would have to modify the Filesystem
  Integrity Scanner database to prevent
  notification
• Could also disable the cron entry
• If you monitor systems, be sure to watch for
  missing emails
• Tripwire and Osiris use cron entries to run daily
  checks, which send out a single summary email
• Samhain has a client-server architecture, can
  send out alerts within minutes of finding an
  issue
• We highly recommend Samhain for new installations

34
CSC Systems Inspection

• "Hemisphere was the only system to survive the
  attacks
• Why?
• Luck
• We do what we can to limit vulnerabilities but
  focus most of our attention on early detection
  and the ability to recover
• Have taken the time to focus on security
• Is not easy, or quick
• So what have we done?

35
CSC Security Tools and Policies

• In addition to the Filesystem/Configuration
  Integrity Scanning and Reporting
• System Hardening
• Foreign login detection and reporting (custom
  scripts)
• Rootkit scanning and reporting (custom wrapper
  scripts)
• We have and follow Incident Response Procedures

36
CSC Security Tools and Policies

• Centralized Log Server
• We were in the process of building this server
  when the CS compromise happened
• All syslog, Samhain, chkrootkit, rkhunter logs to
  this server
• Extremely secure configuration
• Only three Administrators have accounts, use
  different passwords from any other system
• Root cannot login
• EAL4 Common Criteria Installation
• IBM RPM/script which hardens SuSE Enterprise
  Linux 9

37
CSC Security Tools and Policies

• System Hardening
• Common Service Audit
• Remove extraneous installed packages
• Only core, necessary services running
• Configuration of required software
• PAM ssh, sudo, su
• Host based firewalling
• Shorewall
• User Policies
• We run John-the-Ripper Password cracking utility
• Users are created with account aging

38
CSC Security Tools and Policies

• chkrootkit - a tool to scan for rootkits
• We have a wrapper to consolidate the output from
  the many servers into a single email
• Lack of scalability in security tools is an issue!

chkrootkit-0.45 found the following warnings,
vulnerabilities, and infections on
hosts node001-64,hemisphere transputer occam
,blade001-27 loaner,store01,store02,store03 Err
ors occured on the follwing nodes that prevented
chkrootkit from completing NONE The following
hosts were not scanned node008 blade003 blade0
16 All logs generated for this run are located
at /home/admin/logs/chkrootkit/20051206/ Quick
Stats warnings 0 vulnerabilities
0 infections 0
39
CSC Security Tools and Policies

• Foreign Login Detection custom script
• Detects login activity from new machines
• Short summary describes new logins
• Administrator follow-up
• Look for accesses from outside of known activity
• System administrators contact Advisor/PI to
  verify students location (this gets their
  attention)
• We have caught several compromised passwords this
  way

(Source CSC Group Slide by Matthew Woitaszek)
40
CSC Security Tools and Policies
Foreign Login Detection Example
-------------------------------------- Host
report times hemisphere 2005-12-01
000215 -------------------------------------- F
irst logins for each user from a host oberg
2005-11-30 140550 hemisphere
ec240-175-dhcp.colorado.edu 2005-11-30
154743 hemisphere ec240-250-dhcp.colorado.edu
sliu 2005-11-30 234433 hemisphere
ae203-053.resnet.stonybrook.edu 2005-11-30
234622 hemisphere ae203-053.resnet.stonybrook
.edu
Why is sliu, usually at a desk in ECCS122,
connecting from a dorm in Long Island at 145AM
Eastern Time?

• User training Dont type your Hemisphere
  password in on a remote machine, it might have a
  trojan SSH! Connect directly from your client!

(Source CSC Group Slide by Matthew Woitaszek)
41
CSC Security Tools and Policies

• Incident Response Procedure
• Secure email Commercial PGP, GPG for the rest
• Cell phones
• Upon any notification of compromise, immediately
  disconnect all machines from the internet, and
  conduct onsite analysis
• Start to download offsite backups for restore

42
Questions?