File Systems and Storage

1
File Systems and Storage

• Rich Sudlow and Paul Brenner
• University of Notre Dame
• Center for Research Computing

2
(No Transcript)
3
Overview

• File System Concepts
• Aspects and Types
• Why do we have so many?
• Redundancy and Performance
• RAID
• Examples on X4500 thumper
• CRC Supported File Systems
• Centralized (ext3, fat32/ntfs)
• Distributed (AFS, NFS)
• Comparision capabilities
• AFS crc.nd.edu, nd.edu cells
• Backup Storage
• Software, Cache, Tape Silo
• Using CRC Storage
• Scratch space, User workspace, Backup

4
Disclaimer

• This is
• A broad overview
• Operational viewpoint
• Starting point
• This is not
• Comprehensive
• Authoritative analysis of the industry/technology
• For more info consider
• ND courses relevant to this topic
• Contacting CRC for specific individual
  requirements

5
File Systems Concepts

• Aspects
• Filenames
• Meta data (size, blocks, time, security)
• Hierarchical vs Flat
• Secure access
• Capabilities/Facilities (move, delete, append)
• Why so many? Why not use just one?
• Types
• Disk/Flash
• Database Transactional
• Network/Distributed
• Special Purpose

6

• Wikipedia List of File Systems
• http//en.wikipedia.org/wiki/List_of_file_systems

7
Redundancy and Performance

• File system design is strongly influenced by the
  target feature set
• Bandwidth, security, latency, distributed access,
  fault tolerance, etc...
• File systems can be tuned and tiered to exploit
  feature sets of Operating Systems and
  filesystems. E.g. Solaris vs Linux
• Underlying ZFS or ext3 filesystem for NFS/AFS
• Scalability Ability to run across multiple
  nodes by multiple users local vs distributed

8

• Wikipedia Comparison of File Systems
• http//en.wikipedia.org/wiki/Comparison_of_file_sy
  stems

9
RAID

• Redundant Array(s) of Inexpensive/Independent
  Disks
• Utilize multiple/many disks to provide
• Capacity
• Reliability/Redundancy
• Performance configurable based on user
  requirements
• IOPS bandwidth various block sizes
• Hardware and Software Implementations
• Performance, Flexibility, Boot chicken before
  egg
• RAID Levels
• Disk utilization configurations to optimize cost
  vs capability
• Tiered/Nested RAID Levels
• One RAID level on top another raid 01 vs 10

10
RAID Levels

• RAID 0
• Striped Performance and Capacity
• RAID 1
• Mirrored Read Performance and Fault Tolerance
  (FT)
• RAID 3 4
• Striped with Dedicated Parity
• RAID 5
• Striped with Distributed Parity Performance,
  Capacity, N1 FT
• RAID 6
• Striped with Distributed Parity Performance,
  Capacity, N2 FT
• RAID 01
• Striped sets in a mirrored set
• RAID 1 0 generally just called RAID 10
• Mirrored sets in a striped set
• RAID 50
• Striped (0) Across Distributed Parity RAID (5)s

11

• RAID Reference
• http//en.wikipedia.org/wiki/RAID

12
Raid example - Sun Thumper X4500

13
c5
vicepb single disk vicepc 2 disk
stripe vicepd 3 disk stripe vicepe 4 disk
stripe vicepf 6 disk stripe vicepg single
disk mirror viceph 2 disk stripe mirror vicepi
3 disk stripe mirror vicep j, k, l 3 disk
stripe only used for read problem encountered
in multiclient test.
C5 C4 C7 C6 C1 C0
14
Links to RAID examples Solaris/Red Hat on Sun
X4500 (thumper)

• UFS tests on Solaris 10 using Sun Volume Manager
• http//www.nd.edu/rich/afsbpw2007/thumper01-solar
  is-ufs-tests
• UFS tests on Linux (RH4U4)
• http//www.nd.edu/rich/afsbpw2007/thumper02-linux
  -ext3-tests

15
CRC Supported File Systems

• Why do we have so many?
• Centralized
• Ext3 (Linux) Red Hat 4 5
• FAT32/NTFS (Windows)
• Distributed
• NFS
• AFS
• Others
• ZFS

16

• CRC Supported File Systems http//www.nd.edu/rich
  /CRC_filesystems.html
• Scratch Space
• User Workspace
• Storage Backup Available for backup of CRC and
  research machines on campus.

17
Scalability

• Scalability of filesystem
• Bottlenecks
• Scalability of network
• Scalability of codes
• Simple testing tools
• http//ndt.hpcc.nd.edu7123 (simple but not
  always accurate)
• nuttcp - /opt/und/local/bin/nuttcp - firewalls
• nuttcp t (-r) opteron.hpcc.nd.edu (Dont abuse)
• diskrate diskrate n 10m f trash

18
File Permissions (Linux)

• What the heck does this mean?
• drwxr-xr--
• File permissions for user, group, and all
• 10 spaces the first indicates if directory
• Triples of rwx indicate read, write, and execute
  for user, group, and all
• Change file permissions with chmod
• Examples
• chmod ar filename
• chmod gow filename
• chmod 1777 directory
• chmod 700 directory

19
File Permissions (AFS)

• fs setacl -dir HOME -acl pat all terry none
• fs is the command suite.
• setacl is the operation code, which directs the
  File Server process to set an access control
  list.
• -dir HOME and -acl pat all terry none are
  arguments.
• Implies that terry previously had access
• -dir and -acl are switches -dir indicates the
  name of the directory on which to set the ACL,
  and -acl defines the entries to set on it.
• HOME and pat all terry none are instances of the
  arguments. HOME defines a specific directory for
  the directory argument. The -acl argument has two
  instances specifying two ACL entries pat all and
  terry none.
• Command abbreviations
• fs listacl (full command) , fs lista
  (abbreviation), fs la (alias)

20
File Permissions (AFS)

• AFS gives each user the permission to create
  their own groups
• Common to use syntax ownergroup
• pts creategroup cvrlcvrl_group
• pts adduser rich cvrlcvrl_group
• pts membership cvrlcvrl_group
• To recursively set permissions
• find ./ -type d print exec fs setacl
  cvrlcvrl_group read \
• Special groups systemanyuser, systemauthuser,
  nd_campus
• IP based Access Control Lists

21
AFS References

• AFS Reference Links
• http//crcmedia.hpcc.nd.edu/wiki/index.php/AFS_Ref
  erences_and_Resources
• Some AFS / NFS Storage comparisons
• http//crcmedia.hpcc.nd.edu/wiki/index.php/CRC_Sto
  rage_Comparisons
• Sometimes the system is more than just storage
  features are important but need to be the ones
  users use.

22
AFS crc.nd.edu nd.edu cell

• nd.edu is the campus legacy OpenAFS cell
  started May 1990 uses ND.EDU Kerberos realm
  run by OIT staff
• Currently the default cell for most CRC logins
  and batch system opteron, opterona, stats
• crc.nd.edu is the new cell run by CRC staff
• Started October 2007 uses CRC.ND.EDU
• The future cell for CRC logins and batch system
  target for rollout June 2008 hardware, and
  administrative differences.
• Kerb 4 EOL scheduled for 12 / 2008 for nd.edu
  cell

23
AFS crc.nd.edu nd.edu cell

• CRC Wiki Links
• http//crcmedia.hpcc.nd.edu/wiki/index.php/CRC_AFS
  _Cell
• Accessing multiple cells
• http//crcmedia.hpcc.nd.edu/wiki/index.php/Automat
  ic_CRC/ND_AFS_cell_setup
• Recommendations on cells for primary access
  interactive use
• Methods to migrating data tar, up, cp, vos
  dump/restore, start fresh.
• Issues with interactive use references to
  nd.edu that you dont know about e.g. mozilla,
  etc

24
CRC Storage Backup B023 Malloy Hall

• Software - Teradactyl Inc. True Incremental
  Backup System TiBS http//www.teradactyl.com
• Available for backup of CRC and any research
  machines in colleges On-site training June
  16-20th, 2008.
• Supported architectures include OpenAFS, Solaris,
  Linux, Windows, MacOSX.
• Hardware
• Backup server Dell Power Edge 6950 server
  utilizing 10 Gb ethernet fiber channel
  interfaces.
• Cache 16 TB Infortrend Fibre Channel Array

25
Storage Backup

• Sony Consolidated Storage Management System
  (CSM 200)
• Capacity of 604 tapes 3 LTO4 drives with 1 TB
  tapes 2 TB per tape with 21 compression-
  Library will hold gt 1 PB without reloading tapes
  expands to 2,988 tapes with 96 drives.

26
References

• Wikipedia FileSystems
• Advanced File Systems Issues-Andy Wang FSU
• http//www.cs.fsu.edu/awang/courses/cop5611_s2004
  /
• ND CRC wiki
• http//crc.nd.edu/wiki
• OpenAFS User Guide
• http//www.openafs.org/doc/index.htm
• OpenAFS Best Practices 2007 Sudlow
• http//crc.nd.edu/facilities/documents/afsbpw2007
  .pdf

27
Questions ?

• How can we improve this class?
• Additional topics?
• Cover one topic more thoroughly?
• Remove topics?
• Thanks for the feedback?