Selling an Idea or a Product

1
91.580.203 Computer Network Forensics
Xinwen Fu Linux Logging Mechanisms
2
Outline

• Log files
• What need to be logged
• Logging policies
• Finding log files
• Syslog the system event logger

3
Who logs data?

• The accounting system
• The kernel
• Various utilities
• All produce data that need to be logged
• Most of the data has a limited useful lifetime,
  and needs to be summarized, compressed, archived
  and eventually thrown away

4
Logging policies

• Throw away all data immediately
• Reset log files at periodic intervals
• Rotate log files, keeping data for a fixed amount
  of time
• Compress and archive to tape or other permanent
  media

5
Which policy to choose

• Depends on
• how much disk space you have
• how security-conscious you are
• Whatever scheme you select, regular maintenance
  of log files should be automated using cron

6
1. Throwing away log files

• Not recommend
• Security problems (accounting data and log files
  provide important evidence of break-ins)
• Helpful for alerting you to hardware and software
  problems
• In general, keep one or two months
• In a real world, it may take one or two weeks for
  SA to realize that site has been compromised by a
  hacker and need to review the logs

7
2. Reset log files at periodic intervals

• Most sites store each days log info on disk,
  sometimes in a compressed format
• These daily files are kept for a specific period
  of time and then deleted
• One common way to implement this policy is called
  rotation

8
3. Rotating log files

• Keep backup files that are one day old, two days
  old, and so on.
• logfile, logfile.1 , logfile.2, logfile.6
• Linux /etc/logrotate.conf
• Specify the frequency with which the files are
  reused
• Each day rename the files to push older data
  toward the end of the chain

9
Script to archive 4 days files
! /bin/sh cd /var/log mv logfile.2 logfile.3 mv
logfile.1 logfile.2 mv logfile logfile.1 cat
/dev/null gt logfile

• Some daemons keep their log files open all the
  time, this script cant be used with them. To
  install a new log file, you must either signal
  the daemon, or kill and restart it.
• In Unix-like operating systems, /dev/null or the
  null device is a special file that discards all
  data written to it, and provides no data to any
  process that reads from it. In Unix programmer
  jargon, it may also be called the bit bucket or
  black hole.

10
4. Archiving log files

• Some sites must archive all accounting data and
  log files as a matter of policy, to provide data
  for a potential audit
• Log files should be first rotated on disk, then
  written to tape or other permanent media

11
Finding log files

• To locate log files, read the system startup
  scripts /etc/rc or /etc/init.d/
• If logging is turned on when daemons are run
• Where messages are sent
• Some programs handle logging via syslog (syslogd
  or rsyslogd)
• Check /etc/syslog.conf (or rsyslog.conf on Fedora
  Core 9) to find out where this data goes

12
Finding log files (default configuration)

• Different operating systems put log files in
  different places
• /var/log/
• /var/cron/log
• /usr/adm
• /var/adm
• On Linux, all the log files are in /var/log
  directory

13
Outline

• Log files
• Syslog the system event logger
• how syslog works
• its configuration file
• debugging syslog
• the software that uses syslog

14
What is syslog

• A comprehensive logging system, used to manage
  information generated by the kernel and system
  utilities
• Allow messages to be sorted by their sources and
  importance, and routed to a variety of
  destinations
• Log files, users terminals, or even other
  machines

15
Syslog three parts

• Syslogd daemon that does the actual logging
• Configuration file /etc/syslog.conf
• API openlog, syslog, closelog
• Library routines that programs use to send data
  to syslogd
• logger
• User-level command for submitting log entries

16
Using syslog library routines write log entries
to a special file
/dev/log
/dev/klog

reads
consults
syslogd
/etc/syslog.conf

Most system logging daemons listen on one or more
Unix sockets, the most typical being /dev/log
/dev/klog is kernel log socket
dispatches
Other machines
Log files
Userss terminals
http//www.calpoly.edu/cgi-bin/man-cgi?syslogd
17
Configuring syslogd

• The configuration file /etc/syslog.conf controls
  syslogds behavior
• It is a text file with simple format, blank lines
  and lines beginning with are ignored
  (comment).
• selector ltTABgt action
• for example mail.info /var/log/maillog

18
Configuration file - selector

• Identifies
• Program facility that is sending a log message
  
• Messagess severity level
• eg. mail.info
• Syntax
• facility.level
• Facility names and severity levels must be chosen
  from a list of defined values

19
Configuration file - Facility Names
FACILITY PROGRAMS THAT USE IT kern the
kernel user User process, default if not
specified mail The mail system daemon System
daemons auth Security and authorization related
commands lpr the BSD line printer spooling
system news The Usenet news system
20
Configuration file - Facility names (Cont.)
FACILITY PROGRAMS THAT USE IT uucp Reserved
for UUCP cron the cron daemon mark Timestamps
generated at regular intervals local0-7 Eight
flavors of local message syslog syslog internal
messages authpriv Private or system
authorization messages ftp the ftp daemon,
ftpd All facilities except mark
UUCP stands for Unix to Unix CoPy.
21
Configuration file - Facility names (Cont.)

• Facility - Mark Timestamps can be used to log
  time at regular intervals (by default, every 20
  minutes), so you can figure out that your machine
  crashed between 300 and 320 am, not just
  sometime last night. This can be a big help if
  debugging problems occur on a regular basis
• Start at command line syslogd m 1
• Use syslog.conf
• Start syslog daemon syslogd
• Add the line to syslog.conf mark. /var/log/messa
  ges

22
Configuration file - severity level
LEVEL APPROXIMATE MEANING emerg (panic) Panic
situation alert Urgent situation crit Critical
condition err Other error conditions warning W
arning messages notice Unusual things that may
need investigation info Informational
messages debug For debugging
severe
not severe
23
Configuration file - selector

• Levels indicate the minimum importance that a
  message must have in order to be logged
• mail.warning - would match all the messages from
  the mail system, at the minimum level of warning
• Level of none will exclude the listed
  facilities regardless of what other selectors on
  the same line may say.
• .infomail.none action All the facilities,
  except mail, at the minimum level info will
  subject to action

24
Configuration file selector (Cont.)

• Can include multiple facilities separated with
  , commas
• e.g., daemon,auth,mail.info action
• Multiple selectors can be combined with
• e.g. daemon.level1mail.level2 action
• Selectors are -- ORed together, a message
  matching any selector will be subject to the
  action
• Can contain
• - meaning all
• none - meaning nothing

25
Configuration file - action

(Tells what to do with a message)
ACTION MEANING filename Write message to a
file on the local machine _at_hostname Forward
messages to the syslogd on hostname _at_ipaddres
s Forward messages to the host at IP address
user1, user2, Write messages to users screens
if they are logged in Write messages to
all users logged in
26
Configuration file - action (Cont.)

• If a filename action used, the filename must be
  absolute path. The file must exist since syslogd
  will not create it
• e.g. /var/log/messages
• If a hostname is used, it must be resolved via a
  translation mechanism such as DNS or NIS
• While multiple facilities and levels are allowed
  in a selector, multiple actions are not allowed.

27
Config file examples (1)
Small network or stand-alone syslog.conf file
emergencies tell everyone who is logged
on .emerg important messages .warningdaemo
n,auth.info /var/adm/messages printer
errors lpr.debug /var/adm/lpd-errs
28
Config file examples (2)
network client, typically forwards serious
messages to a central logging machine
emergencies tell everyone who is logged
on .emerguser.none important messages,
forward to central logger .warninglpr,local1.non
e _at_netloghost daemon,auth.info _at_netloghost
local stuff to central logger too local0,local2,lo
cal7.debug _at_netloghost card syslogs to local1
- to boulder local1.debug _at_ialab.cs.uml.edu
printer errors, keep them local lpr.debug /var/
adm/lpd-errs sudo logs to local2 - keep a copy
here local2.info /var/adm/sudolog
29
Sample syslog output

• Mar 27 091002 tcb-ia-lab-inst sshd4100
  Accepted password for cis418 from
  ffff216.254.235.105 port 61940 ssh2
• Mar 27 181000 tcb-ia-lab-inst sshd9332
  Failed password for root from ffff216.254.235.1
  05 port 62817 ssh2
• Mar 27 181008 tcb-ia-lab-inst sshd9332
  Accepted password for root from
  ffff216.254.235.105 port 62817 ssh2
• Mar 27 200827 tcb-ia-lab-inst sshd10629
  Accepted password for root from ffff10.0.0.111
  port 42172 ssh2
• Mar 27 200948 tcb-ia-lab-inst sshd10649
  Failed password for root from ffff10.0.0.111
  port 48233 ssh2

30
Syslogd

• A hangup signal (HUP, signal 1) cause syslogd to
  close its log files, reread its configuration
  file, and start logging again
• If you modify the syslog.conf file, you must HUP
  syslogd to make your changes take effect
• ps -ef grep syslogd
• Kill -1 pid-of-syslogd

31
Software that uses syslog
PROGRAM FACILITY LEVELS DESCRIPTION amd auth err
-info NFS automounter date auth notice Display
and set date ftpd daemon err-debug ftp
daemon gated daemon alert-info Routing
daemon gopher daemon err Internet info
server halt/reboot auth crit Shutdown
programs login/rlogind auth crit-info Login
programs lpd lpr err-info BSD line printer
daemon
32
Software that uses syslog
PROGRAM FACILITY LEVELS DESCRIPTION named daemon
err-info Name sever (DNS) passwd auth err Passwo
rd setting programs sendmail mail debug-ale
rt Mail transport system rwho daemon err-notice r
omote who daemon su auth crit,
notice substitute UID prog. sudo local2 notice,
alert Limited su program syslogd syslog,
mark err-info internet errors, timestamps
33
Syslog 's functions

• Liberate programmers from the tedious mechanics
  of writing log files
• Put SA in control of logging
• Before syslog, SA had no control over what
  information was kept or where it was stored
• Can centralize the logging for a network system

34
Debugging syslog -- logger

• Useful for submitting log entries from shell
  scripts
• Can also use it to test changes in syslogds
  configuration file.
• For example..

35
Add line to syslog.conf local5.info /var/log/te
st.log verify it is working, run logger -p
local5.info test messages a line containing
test messages should be written to
/tmp/test.log If this doesnt happen forgot
to create the test.log file or forgot to
send syslogd a hangup signal
36
Remote logging

• On a central logging server 10.0.0.192
• syslogd -r
• On a local server 10.0.0.45
• authpriv.auth. _at_10.0.0.192
• Question where are those events written?

37
Process Accounting

• accton is used to turn on or turn off process
  accounting
• lastcomm tracks commands each user uses
• touch /var/log/pacct
• /sbin/accton /var/log/pacct
• lastcomm -f /var/log/pacct
• ac prints out statistics about users' connection
  times in hours based on the logins and logouts in
  the current /var/log/wtmp file
• ac -p -d
• sa summarizes accounting information from
  previously executed commands, software I/O
  operation times, and CPU times, as recorded in
  the accounting record file /var/log/pacct
• sa /var/log/pacct

38
Process Accounting (Cont.)

• last goes through the /var/log/wtmp file and
  prints out information about users' connection
  times
• lastb is the same as last, except that by default
  it shows a log of the file /var/log/btmp, which
  contains all the bad login attempts.

39
Using syslog in programs

• openlog( ident, logopt, facility)
• Messages logged with the options specified by
  logopt begin with the identification string
  ident.
• syslog( priority, messge, parameters)
• Send message to syslogd, which logs it at the
  sepecified priority level
• close( )

40
/ c program syslog using openlog and closelog
/ include ltsyslog.hgt main ( ) openlog (
SA-BOOK, LOG_PID, LOG_USER) syslog (
LOG_WARNING, Testing . ) closelog ( )
On the host, this code produce the following log
entry Apr 4 152157 tcb-ia-lab-inst
SA-BOOK7762 Testing ...
41
Summary

• On linux, check following files
• /etc/syslog.conf syslog configuration file
• /etc/logrotate.conf logging policy, rotate
• /etc/logrotate.d/
• /var/log/ log files
• try following commands to find out more...
• man logrotate
• man syslogd

42
References

• Chris Prosise, Kevin Mandia, Matt Pepe, Incident
  Response and Computer Forensics, Second Edition
  (Paperback), ISBN 007222696X
• Brian Hatch, Preventing Syslog Denial of Service
  attacks, http//www.hackinglinuxexposed.com/articl
  es/20030220.html
• Albert M.C. Tam, Enabling Process Accounting on
  Linux HOWTO, 02/09/2001, http//www.faqs.org/docs
  /Linux-mini/Process-Accounting.html
• Keith Gilbertson, Process Accounting, 12/01/2002,
  http//www.linuxjournal.com/article/6144

43
Notes

• Change host name
• /etc/hosts add the host to the end of 127.0.0.1
• /etc/sysconfig/network

44
! /bin/sh cd /var/log mv logfile.2.Z
logfile.3.Z mv logfile.1.Z logfile.2.Z mv
logfile logfile.1 cat /dev/null gt logfile kill
-signal pid compress logfile.1
signal - appropriate signal for the program
writing the log file pid - process id