Server management and security

1
Server management and security
ccTLD name server training

• September 10, 2002
• Ko, YangWoo
• yw_at_mrko.pe.kr

2
Note

• Contents are NOT mine. Most of them are from the
  wonderful book Practical Unix and Internet
  Security and Real World Linux Security.
• Others are extracted from various good resources
  including
• Linux Security FAQ
• Solaris Security FAQ
• Sun Solaris / HP-UX / Tru64 Unix man pages

3
Table of contents

• Before we start
• Security basics
• Unix / Linux sever security
• System setup guide
• Detection
• Recovery

4
Module 1 Before we start
5
Welcome to wild Internet !

• Quote from Crypto-Gram (June 15, 2001 )

A random computer on the Internet is scanned
dozens of times a day. The life expectancy of a
default installation of Red Hat 6.2 server, or
the time before someone successfully hacks it, is
less than 72 hours. A common home user setup,with
Windows 98 and file sharing enabled, was hacked
five times in four days. Systems are subjected to
NetBIOS scans an average of 17 times a day. And
the fastest time for a server being hacked 15
minutes after plugging it into the network.
6

• No system is ever perfectly secure.

7
But, still we need security.

• Any number of toolkits exist that allow total
  amateurs to become holy terrors.
• The good news is that if you can beat the popular
  intrusion toolkits, 90 percent of the bad guys
  will go bother somebody else who's less secure.

8
System security in a page

• The Seven Most Deadly Sins
• Weak Passwords
• Open Network Ports
• Old Software Version
• Poor Physical Security
• Insecure CGIs
• Stale and Unnecessary Accounts
• Procrastination

9
Module 2 Security basics
10
Security requirements

• Confidentiality
• Integrity
• Authentication
• Non-repudiation
• Availability
• Access control
• Combined
• User authentication used for access control
• Non-repudiation combined with authentication

11
Some terminologies

• System security / network security
• Passive attack / active attack
• sniffing / spoofing
• Two models
• Access control
• discretionary access control vs. mandatory access
  control
• Audit

12
Security policy

• Simple and generic policy for system which users
  can readily understand and follow.
• Starting point
• That which is not permitted is prohibited.
• Setup steps
• (1) Identify what you are trying to protect.
• (2) Determine what you are trying to protect it
  from.
• (3) Determine how likely the threats are.
• (4) Implement measures which will protect your
  assets in a cost-effective manner.
• (5) Review improve the process continuously

13
Security policy (continued)

• References
• rfc2196 Site Security Handbook
• Samples
• ftp//coast.cs.purdue.edu/pub/doc/policy

14
Module 3 Unix / Linux server security

• Password
• Superuser
• File system
• Account
• Integrity
• Log and Audit
• Programmed threats
• TCP/IP

15
Module 3-1 Password
16
Bad passwords

• Your name, spouses name, partners name, pets
  name, childs name, friends name, bosss name
• Operating system, hostname, username
• Phone number, license plate number, birth date,
  social security number
• Words in the dictionary
• Simple patterns of letters on the keyboard
  (qwerty)
• Passwords of all the same letter
• Any of above spelled backwards
• Any of above followed or prepended by a single
  digit

Password
17
Good passwords

• Have both uppercase and lowercase letters.
• Have digits and/or punctuation characters as well
  as letters.
• May include some control characters and/or
  spaces.
• Are easy to remember, so they do not have to be
  written down.
• Are seven or eight characters long.

Password
18
The Thompson Test

• Devised by Ken Thompson
• Cracking algorithm
• One to six ASCII characters
• Seven or eight lowercase letters
• Any word from a large dictionary such as
  hangman-words, or a word spelled backward or with
  the digit 1 instead of the letter l, with the
  digit 0 instead of the letter o, or with the
  digit 3 instead of the letter e.
• Any pair of words from a large dictionary or
  words spelled backwards.

Password
19
Module 3-2 Superuser
20
Who is superuser ?

• UID of 0
• Any username can be the superuser.
• Normal security checks and constraints are
  ignored for the superuser.
• Superuser is not for casual use.
• Do not login as superuser, use /bin/su with -
  option instead.

Superuser
21
Simple trap to steal superuser

• Premise
• Roots PATH starts with .
• Contents of shell script ls
• !/bin/sh
• cp /bin/sh ./junk/.ss
• chmod 4555 ./junk/.ss
• rm f 0
• exec /bin/ls 1_at_

• Set a trap
• cd
• chmod 700 .
• touch ./-f
• To do is just say to administrator. I have a
  funny file in my directory I cant seem to
  delete.

Superuser
22
Several tricks for superusers

• Test complex commands in a non-destructive way
  before running it.
• rm foo.bar after echo foo.bar
• alias rmrm i
• Only become root to do single specific task. Stay
  normal user shell until you are sure what needs
  to be done by root.
• Command path
• Minimum and trusted directories only
• Never include .
• No writable directories

Superuser
23
Several tricks for superusers (continued)

• Never use r-utilities (e.g. rlogin, rsh). Never
  create .rhosts for for the root.
• No login from the remote
• Linux, HPUX /etc/securetty
• file which lists ttys from which root can log in
• Solaris /etc/default/login
• CONSOLE/dev/console
• Always be slow and deliberate running as root.
  Think before you type.

Superuser
24
Module 3-3 File system
25
File permission
File type - plain file d directory c
character device (tty, printer) b block device
(disk, CD-ROM) l symbolic link s socket , p
FIFO
Access granted to others
-rwxr--r--
Access granted to group member
Access granted to owner r read / w write / x
execute
File system
26
SUID/SGID/sticky bits

• SUID (set uid)
• Processes are granted access to system resources
  based on user who owns the file.
• SGID (set gid)
• (For file) Same with SUID except group is
  affected.
• (For directory) Files created in that directory
  will have their group set to the directory's
  group.
• sticky bit
• If set on a directory, then a user may only
  delete files that the he owns or for which he has
  explicit write permission granted, even when he
  has write access to the directory. (e.g. /tmp )

File system
27
File system tips

• Finding SUID and SGID Files
• find / \( -local -o -prune \) \( -perm -004000
  -o -perm -002000 \) -type f -print
• ( xdev can be used in place of local/prune)
• Files without associated owner/group can be a
  signal of compromise.
• find / -nouser o nogroup print
• Users are not allowed to have .rhosts file.
• find /home name .rhosts -print

File system
28
File system tips (continued)

• Turning off SUID / SGID in mounted file system
• use nosuid (and nodev if possible) when mounting
  remote file system or allowing users to mount
  floppies or CD-ROMs
• Device file can be created as a backdoor after
  compromise.
• find / \( -local -o -prune \) \( -type c -o
  -type b \) -exec ls -l \

File system
29
Critical system files

• These files should be backed up and compared with
  saved version frequently.
• /etc/passwd, /etc/shadow, /etc/group
• /etc/rc
• /etc/ttys, /etc/ttytab, /etc/inittab
• /usr/lib/crontab, /usr/spool/cron/crontabs/,
  /etc/crontab
• /usr/lib/aliases
• /etc/exports, /etc/dfs/dfstab
• /etc/netgroups
• /etc/fstab, /etc/vfstab
• /etc/inetd.conf
• UUCP related files

File system
30
Module 3-4 Account
31
Dangerous accounts

• Accounts without passwords
• cat /etc/passwd awk -F 'length(2)lt1 print
  1'
• Default accounts
• Just remove them !
• Shared accounts
• Less accountability, less security.
• Create several accounts in a group.
• e-mail ID and account
• Do not use e-mail ID as an account, utilized
  alias feature instead.

Account
32
Dormant account

• Risks
• Intruder can use dormant account without being
  noticed.
• Owner of dormant account cannot follow your
  policy or order. (e.g. Dear every users, please
  change your passwords right now.)
• How to handle
• Disabling dormant account automatically (SVR4)
• usermod f 10 newcat (locked if no login in 10
  days)
• Freeze it
• Put in password field
• chmod 0 /home/newcat
• find / -user newcat -ls

Account
33
Dormant account (continued)

• How to find
• !/bin/sh
• PATH/bin/usr/binexport PATH
• umask 077
• THIS_MONTHdate awk print 2
• /bin/last /bin/grep THIS_MONTH awk print
  1 /bin/sort u gt /tmp/users1
• cat-passwd /bin/awk F print 1
  /bin/sort u /tmp/users2
• /bin/comm 13 /tmp/users12
• /bin/rm f /tmp/users12

Account
34
Module 3-5 Integrity
35
Simple examples

• By metadata
• cat /usr/adm/filelist xargs ls -ilds gt
  /tmp/now
• diff -b /usr/adm/savelist /tmp/now
• By checksum
• find cat /usr/adm/filelist -ls -type f -exec
  md4 \ gt /tmp/now
• diff -b /usr/adm/savelist /tmp/now

Integrity
36
Tripwire

• Tripwire is a tool that checks to see what has
  changed on your system. The program monitors key
  attributes of files that should not change,
  including binary signature, size, expected change
  of size, etc.
• Where is it ?
• Commercial version http//www.tripwire.com/
• For Linux user http//www.tripwire.org/
• For Unix user ftp//coast.cs.purdue.edu/pub/COAS
  T/Tripwire/tripwire-1.2.tar.Z

Integrity
37
Tripwire tutorial in a slide

• Initial setup
• download / build / install it
• modify policy file (e.g. remove unnecessary
  files)
• vi /etc/tripwire/twpol.txt
• generate policy file
• twadmin create-polfile /etc/tripwire/twpol.txt
• build initial database
• tripwire init
• check periodically
• tripwire check
• reconcile differences (e.g. software
  installation)
• tripwire update accept-all twrfile
  report_file

Integrity
38
Module 3-6 Log and audit
39
Basics

• Consider remote logging to secure log data.
• List of log files
• acct / pacct Commands run by users
• aculog Dial-out modem (acu automatic call
  unit)
• lastlog Most recent login success/fail times
• loginlog Bad login attempts
• messages Console / syslog facility
• sulog su command
• utmp / utmpx Each user currently logged in
• wtmp / wtmpx Login/out, shutdown/startup
• xferlog FTP access

Log and audit
40
Files and commands

• lastlog file
• lastlog (Linux only)
• Displays last login time and location.
• u/wtmp file
• last
• Displays login and logout information about users
  and terminals
• acct/pacct file
• (Solaris 5.8) /usr/lib/acct/startup , shutacct
• Starts or stop accounting.
• (Solaris 5.8) acctcom, lastcom
• Displays the recent commands executed.

Log and audit
41
Monitoring logs

• logcheck (logsentry)
• Extracts anything that might indicate a security
  violation or other abnormality, and informs via
  e-mail.
• http//www.psionic.com/products/logsentry.html

Log and audit
42
Module 3-7 Programmed threats
43
Basic terms

• Bug vs. malware (or malicious software)
• Kinds of malwares
• Security tools and toolkits
• Back doors and trap doors
• Logic bombs
• Viruses
• Worms
• Trojan horses
• Bacteria and rabbits

Programmed threats
44
Against programmed threats

• Back door
• Do regular integrity check.
• Install software only from well-known sources.
• Separate test bed and production system.
• Trojan horse
• Never execute anything until youre sure of
  program or inputs to program.
• Never run anything as root unless you absolutely
  must.

Programmed threats
45
Against programmed threats (continued)

• Viruses
• Use same techniques used against back doors and
  Trojan horse.
• Dont include nonstandard directories (including
  .) in your PATH.
• Dont leave common binary directories unprotected
  and set permission of commands to 555 or 511.
• Make sure your own directories are writable only
  by you not by your group or world.

Programmed threats
46
Against programmed threats (continued)

• Worm
• Prevention
• If an intruder can enter your machine, so can a
  worm program.
• If under attack,
• Call computer incident response center to se if
  other sites have made similar reports.
• Isolate your server to prevent spread.

Programmed threats
47
Module 3-8 TCP/IP
48
Vulnerabilities

• ftp
• Passwords are sent in plain text.
• /etc/ftpusers
• List of accounts that are NOT allowed to use ftp.
• telnet
• Passwords are sent in plain text.
• Attacker can hijack the session.

TCP/IP
49
Vulnerabilities (continued)

• smtp (sendmail)
• Must be upgraded 8.9.3 or higher. Current version
  is 8.12.6.
• Check permission of /var/spool/mqueue,
  sendmail.cf, /etc/aliases, /etc/mail/mailertable
  (owned by root, writable by owner only)

TCP/IP
50
Vulnerabilities (continued)

• Sun RPC portmapper
• Assigns the TCP/UDP ports used for RPC.
• To improve security, turn it off if possible. Or,
• Replace it with Wietse Venemas version.
• Block packets on port 111.
• rexec, rsh, rlogin
• Executes remote program or login.
• rexec transmits plain text password and
  rsh/rlogin use trusted host/user concept.
• Disable rexec, and replace rsh/rlogin with ssh.

TCP/IP
51
Vulnerabilities (continued)

• web
• Yet another BIG topic. See references
• Lincoln D. Steins WWW Security FAQ
• http//www-genome.wi.mit.edu/WWW/faqs/www-security
  -faq.html
• Paul Phillips CGI security FAQ
• http//www.primus.com/staff/paulp/cgi-security
• NCSAs CGI security documentation
• http//hoohoo.ncsa.uiuc.edu/cgi/security.html

TCP/IP
52
Vulnerabilities (continued)

• NFS
• Limit exported and mounted file systems
• Export read-only and use root ownership
• Remove group-write permission for files and
  directories
• Do not export server executables and home
  directories
• Do not allow users to log into server
• Use fsirand and set the portmon variable
• Use showmount e
• Use secure NFS

TCP/IP
53
Vulnerabilities (continued)

• tftp (UDP 69)
• No security at all.
• finger ( 79 )
• Provides user information.
• POP ( 109, 110 )
• Username/password is sent in plain text.

TCP/IP
54
Module 4 System setup guide
55
Useful links for system setup

• Solaris
• Solaris/Unix Security Checklist Version 1.0
• http//www.geocities.com/losttoy2000/solarissec.rt
  f
• The Solaris Security FAQ
• http//www.itworld.com/Comp/2377/security-faq/
• Linux
• Securing Debian Manual
• http//www.debian.org/doc/manuals/securing-debian-
  howto/

56
System setup steps (1/2)

• Disconnect system from network.
• Install a minimal Operating System.
• Install the recommended patches.
• Use BIOS/EEPROM security.
• Securing root account
• Force root to login through su.
• Check environments
• default mask (027), PATH
• Apply hardening script if available.
• Direct syslog to loghost

57
System setup steps (2/2)

• Create minimal accounts and disallow login.
• Let minimal services run
• /etc/rc, /etc/inet.d
• Use tcpwrapper for network services.
• Install Secure Shell and encourage its use.
• Install integrity checker (e.g. Tripwire).
• Test it periodically
• e.g. Nessus, COPS, Tiger,
• Monitor it forever
• Check logs, login/outs, commands

58
Module 5 Detection

• Monitoring
• Scanning
• Handling

59
Monitoring (1/2)

• Log (logcheck)
• Propagate it using loghost and e-mail.
• Check it.
• Network port (netstat)
• Trojan horse may use network ports.
• http//www.glocksoft.com/trojan_port.htm
• Network (tcpdump)

Monitoring
60
Monitoring (2/2)

• Process (ps)
• Check suspicious processes, e.g. compiler.
• Record typical size of daemons and important
  programs to detect Trojan horse.
• Load (uptime)

Monitoring
61
Scanning

• Find suspicious files.
• Run Tripwire.
• Detect promiscuous network interfaces.
• (see next page)

Scanning
62
Perl script to detect sniffer

• !/usr/bin/perl
• my ifconfig /sbin/ifconfig
• my recips admin_at_my.admin.host
• my PROMISC ()
• my interface
• open( IFCONFIG, ifconfig ) die( Error
  cannot run ifconfig! )
• while( ltINCONFIGgt )
• interface 1 if m/(\S)/
• PROMISCinterface 1 if m/promisc/I
• close( IFCONFIG )
• if( PROMISC )
• open( MAIL, Mail s Promisc mode recips
  ) die( Error cannot send mail )
• print MAIL Interfaces in Promisc mode ,
  join( , sort keys PRMISC), \n
• close MAIL

Scanning
63
Handling incidents

• Dont panic
• Is it really a security incident ?
• Was any damage really done ?
• Evidence or normal operation, that is the
  question.
• Document
• Write down everything you find, always noting the
  date and time.
• Plan ahead !!!

Handling
64
Module 6 Recovery

• Regaining control of system
• Finding and repairing the damage
• Tracing attacker

65
Regaining control of system

• Operate as an unprivileged user.
• Check integrity of commands used.
• Have stealth version of crucial commands (ps / ls
  / tar / )
• Build from open source. Or,
• Rename from existing binary
• cd /home/larry/bin
• cp /bin/ls monthly
• cat text_file gtgt monthly
• (echo ls is monthly md5sum monthly) lpr
• Process must be kill by 9.

TCP/IP
Regaining control of system
66
Analyze Trojan horse

• Save suspicious executables on (removable) media.
• Analyze
• strings Trojan
• file Trojan
• if not stripped
• nm Trojan (see function names, syscalls)
• run debugger (see stack trace)
• Check files opened by Trojan
• (Linux) /proc/pid/fd
• (Solaris) pfile pid

Regaining control of system
67
Prevent further damage

1. Drop connection (unplug LAN, modem)
2. Shutdown abruptly
3. Close database
4. Run sync (from non privileged user)
5. Press reset (or power) button
6. Boot again
7. Remove the system disk from the compromised
   system and connect it as second disk to a secure
   system. (Or, boot from secure boot floppy.)
8. Run fsck
9. Before coming up multi-user mode, check cracker
   generated email.

Regaining control of system
68
Checking logs

• Log files
• /var/log/
• Shell history files (esp. for root)
• Mailboxes (mbox, /?/spool/mail, /?/spool/mqueue)
• Firewall logs, ISPs log
• tcpwrapper log (denied log only)
• Other files
• /tmp/
• Hidden directories (e.g. /home//.??)
• Other files started with .

Finding and repairing the damage
69
Finding cracker-altered files

• Use file integrity tools (e.g. Tripwire)
• Compare file system with backups.
• GNU tar -d option is very useful.
• Rename any Trojan horse found something obvious.
• mv /mnt2/tmp/ls /mnt/tmp/ls-CRACKED
• chmod 0 /mnt/tmp/ls-CRACKED
• Find normal files hidden in /dev
• find /dev type f ls
• Find set UID programs

Finding and repairing the damage
70
Useful commands

• With IP address (A.B.C.D)
• nslookup typeany D.C.B.A.in-addr.arpa
• dig x A.B.C.D
• With domain name
• whois
• Using ping
• See the distance
• Using traceroute

Tracing hacker
71
Module 7 D.I.Y.

• Requirement
• Analysis
• Plan and Do

72
What assets do I have ?

• Classification of assets
• Hardware
• Server / PC / Storage device / Printer
• Network
• Network distribution component (e.g. router, hub,
  switch)
• Network service host (e.g. directory, NMS)
• Network connection / Cabling
• Data (e.g. database, agreement, policy,
  guideline)
• Software
• Human
• Environment (e.g. UPS, air conditioner, cabinet)

Requirement
73
How valuable they are ? (1/4)

• Review documentations
• List of all servers
• List of all security products in place
• Operation guidelines
• Interview with operational personnel
• Valuation methods
• CIA
• Confidentiality / Integrity / Availability
• Cost of loss

Requirement
74
How valuable they are ? (2/4)

• Confidentiality
• 5 Top secret
• 4 Secret
• 3 Limited
• 2 Limited within organization
• Ordinary documents
• 1 Open

Requirement
75
How valuable they are ? (3/4)

• Integrity
• 5 Critical damage to operation
• 1 No (or very least) damage to operation
• Availability
• 5 Non stop
• 4 Recovery within 4 hours
• 3 Recovery within 8 hours
• 2 Recovery within 12 hours
• 1 Recovery within 24 hours

Requirement
76
How valuable they are ? (4/4)

• Cost of loss
• 5 Serious loss (e.g. Bankruptcy)
• 4 Major loss (e.g. Discontinuance of some
  businesses)
• 3 Significant loss (e.g. Discontinuance of some
  tasks)
• 2 Loss (e.g. lt U 10,000)
• 1 Trivial loss (e.g. lt U 1,000)

Requirement
77
Define analysis areas

• Network / system security
• Service daemons
• Backdoors, vulnerable files
• Misuse by users
• User accounts
• Log management
• Network configuration
• Network device management
• Database security

• Physical security
• Security management
• Compliance assessment
• Security policy assessment
• Contingency planning

Requirement
78
Analysis

• Automated analysis
• e.g. Nessus
• Manual analysis
• OS checklists

Analysis
79
Sample results

• Service daemons
• Problems
• Some old-version daemons have buffer overflow
  vulnerabilities.
• Unnecessary daemons are running.
• To do
• Remove unnecessary daemons.
• Keep necessary daemon up to date.
• Run security scanner periodically.

Analysis
80
Sample results

• Backdoors, vulnerable files
• Problems
• Backdoor is not found, but there is no counter
  measure for future backdoors.
• To do
• Install and run Tripwire periodically.

Analysis
81
Sample results

• Misuse by users
• Problems
• Sendmails vulnerability can lead to root
  compromise.
• To do
• Remove if unnecessary.
• Keep it up to date if necessary.

Analysis
82
Sample results

• User accounts
• Problems
• Super user accounts are shared by administrators
  and developers.
• Weak passwords are found.
• To do
• Define each systems usages clearly.
• Define each users role according to usage of
  system.
• Apply password control (including aging).

Analysis
83
Sample results

• Log management
• Problems
• No log management.
• To do
• Setup a loghost, and all logs are configured to
  be sent to it.
• Write a log management guideline and apply it.

Analysis
84
Sample results

• Network configuration
• Problems
• Database servers are exposed to Internet.
• To do
• Set up a DMZ.
• Put external service servers at DMZ.
• Put Database servers at internal network

Analysis
85
Categories of reaction

• Configuration issue
• Issues are solved by configuring servers and
  network equipments properly.
• Usually done within a week.
• Infra structure issue
• Issues are solved by investing on infrastructure.
• Usually outsourced in long time period.
• Management issue
• Several units within organization work together
  to handle these issues.
• Plan gt Do gt See cycle

Plan
86
Categorize To Dos

• Configuration issue
• Remove unnecessary daemons.
• Apply password control (including aging).
• Management issue
• Write a log management guideline and apply it.
• Define each systems usages clearly.
• Define each users role according to usage of
  system.
• Infra structure issue
• Run security scanner periodically.
• Set up a DMZ.

Plan