Unix Files, IO Plumbing and Filters

1
Unix Files, IO Plumbing and Filters

• The file system and pathnames
• Files with more than one link
• Shell wildcards
• Characters special to the shell
• Pipes and IO redirection
• Commands used with redirection
• Some useful filters awk, grep, sed etc.

2
The file system and pathnames

• The filesystem is a set of conventions and the
  specific data design for organising files and
  folders, or directories on a hard disk, USB
  drive, CD or other media. E.G. NTFS, FAT
  (Windows), EXT3 (Linux)?
• Files on a filesystem include plain files,
  directory files (or folders), links (Unix/Linux
  only) and device files.
• All modern OSs map files on a filesystem as an
• inverted tree.

3
Inverted tree with root dir as /

• / (root directory)?
• ---------------------------------------------
  ---------
• 
  
• dir1 file1 file2 dir2
  home
• 
  
• ---------------- -------------------
  ----------
• 
  
• file3 dir3 file4 file5
  fred bill
• 
  
• -------------------------------
  ----------
• 
  
• file6 file7 file8
  cupboard wardrobe
• 
  
• 
  ----------------------
• 
  
• socks
  vests shirts
• Absolute pathnames start with a / and relative
  pathnames don't.

4
Some pathname examples
5
Differences between Windows and Unix

• Pathname delimiters Unix forward / and Windows
  back \
• Windows has drive letters e.g. a
• Unix trees mount filesystems onto empty
• directories.
• So, a\reports\report.txt on Windows becomes
  /mnt/floppy/reports/report.txt on Unix

6
Links

• Multiple paths to the same file saves space and
  keeps
• objects updated
• Hard link
• ln /sales/personnel/simon /bonuses/recipients/simo
  n
• Soft link
• ln -s /usr/wizard/wands/magic/software/binaries
  /wiz
• Hard links only possible within same physical
  device, soft links needed to link between mounted
  devices/partitions.

7
Shell Wildcards

• Known as "regular expressions", REs or regexes.
• Used in shell commands to match or "glob"
• multiple files.
• Similar language used in Grep, Awk, Perl, Java,
  Python etc. to find/replace expressions in
  strings, e.g. tags in HTML

8
Shell Wildcards
9
Shell Wildcards can be combined, giving greater
flexibility
10
REs and special characters

• . at the beginning must match explicitly, and /
  must always match explicitly.
• Avoid putting ,?()\/ltgt"'! and spaces in
  filenames ?
• If you can't the backslash special character \
  will usually escape the special character
  following it
• rm \fred deletes the file called fred
• Other special characters
• . at start of name hides a system file
• at start of path means home directory

11
Pipes and redirection
12
Redirections can be combined e.g

• grep xxx lt f1 awk ' print 2 ' 2gt f2 \
• tee f3 wc -l gt f4
• \ is used to escape the newline so a single
  command line can be typed using more than one
  line on the terminal.

13
Commands often used with redirection
14
Some useful filters

• A filter is a program which reads from stdin and
  writes to stdout. Unix has many useful programs
  which filter data based on contents of text
  lines, or row positions or filtering on columns.
  We can also sort a file, obtain unique keys or
  chop the top or tail of it.
• These filters will make our pipelines into a
  general purpose and easy to use text-processing
  toolkit, once a few examples have been mastered.

15
awk

• Replaced as a scripting language by Perl, but
  still useful as a simple column filter.
• Example 1
• cat file1 awk ' print 5, 2 '
• outputs columns 5 then 2 of file1 which has to
  have spaces or tabs between columns. Use
  -Fltchargt to delimit input using ltchargt e.g. -F
  for colon seperated columns. You might prefer cut
  for this job.

16
awk example 2

• Put an awk program of any complexity into a
  separate file. Use the -f flag to give filename
  on shell command. This awkprog file has 3 lines
• BEGIN total0
• if ( 3 "usr19999" ) totaltotal4
• END print total/1024
• Run awk program with command
• ls -l /scratch awk -f awkprog
• Counts the number of 1K blocks used by files
  belonging to usr19999 in the /scratch directory.

17
cut

• For awk's usual purpose as a column filter, you
  might prefer to learn cut(1) as an alternative.
  The following command extracts columns 1, 3, 6
  and 7 from /etc/passwd, with colon () as input
  delimiter, and space as output delimiter.
• cat /etc/passwd cut -d --output-delimiter" "
  -f1,3,6,7

18
Input and output of cut command

• input sample from /etc/passwd
• bertx10031003Bert Trusted User,,,/home/bert/
  bin/bash
• avahi-autoipdx111111Avahi autoip
  daemon,,,/var/lib/avahi-autoipd/bin/false
• avahix116121Avahi mDNS daemon,,,/var/run/avah
  i-daemon/bin/false
• snortx117125Snort IDS/var/log/snort/bin/fals
  e
• pulsex119126PulseAudio daemon,,,/var/run/puls
  e/bin/false
• Output sample
• bert 1003 /home/bert /bin/bash
• avahi-autoipd 111 /var/lib/avahi-autoipd
  /bin/false
• avahi 116 /var/run/avahi-daemon /bin/false
• snort 117 /var/log/snort /bin/false
• pulse 119 /var/run/pulse /bin/false

19
grep

• grep extracts lines containing a particular
  pattern from a file, using regexes. Similar but
  not
• identical to shell REs.
• fgrep is faster and does not interpret REs
• egrep interprets more REs than grep

20
grep examples
21
Sed is used for automated "stream" editing.
Another language best used for one liners.
22
Another sed example

• cat f1 sed '1,s/(//g' sed '1,s/)//g' gt f2
• copies all lines from f1 to f2, removing all (
  and ) brackets. sed is used twice in the same
  pipeline and "/" is the search-replace delimiter.

23
sort

• Has many options and parameters for ascending or
  descending, string or numeric, field and column
  sort key selection etc. See sort(1) for further
  details. Example
• cat /etc/passwd sort -t 0n -1 more
• Gives paged output of passwd database, sorted by
  userid.

24
uniq

• Having sorted our input data we may find repeated
  lines. The following example only outputs unique
  lines once
• cat file.txt sort uniq
• The next example precedes all unique output lines
  with a count showing how many times each line
  occurred
• cat file.txt sort uniq -c

25
head and tail

• Head copies the first part and tail copies the
  last part of stdin file, see head(1) and tail(1)
  manual pages.
• Example
• head -5 index
• displays first 5 lines of index file

26
tr and colrm filters

• tr is used to translate characters
• e.g. convert a text from lower to upper case
• richard_at_saturn/misc cat keller_security tr
  a-z A-Z
• "SECURITY IS MOSTLY A SUPERSTITION. IT DOES NOT
  EXIST IN NATURE... LIFE IS EITHER A DARING
  ADVENTURE OR NOTHING."
• -- HELEN KELLER
• colrm similar to cut but filters based only on
  character positions, and doesn't understand field
  delimiters.
• richard_at_saturn/misc cat keller_security
  colrm 36
• "Security is mostly a superstition.
• either a daring adventure or nothin
• -- Helen Keller

27
Conclusions

• Redirections of input, output and error files
  combined with the use of some filter programs
  allows for us to manipulate the text data used
  for system configurations. We will be able to
  extract needles of security information from the
  system log data haystacks.
• Combining a few powerful one liners into short
  scripts will allow for fully customised
  administration and security monitoring of
  open-source servers, routers and firewalls.