Advanced Operating Systems CS 518 Chad Bohannan November 28, 2006

1
Advanced Operating Systems CS 518 Chad
BohannanNovember 28, 2006
2
Motivation

• Embedded systems require economic use of memory.
• The Maia satellite systems needed to store as
  much data as it could.
• An embedded solution was developed for Maia,
  known as MaiaFS.

3
Why Linux?

• Embedded Linux is a new frontier for university
  built satellite systems.
• If MaiaFS could be ported to Linux, the embedded
  community could benefit from its advantages.

4
Implementing MaiaFS

• The Virtual File System
• The FUSE API
• MaiaFS Implementation Details
• Demonstration/Questions

5
The Virtual File System

• Everything is a file

Directories Regular Files IP Sockets Pipes Devic
es Ports Keyboards Consoles/Displays Error
Histories CPU Status Memory Blocks Program
Control Random Number Generators This is NOT a
complete List.
6
What makes the it virtual?

• It is virtual because it is not real.
• The Virtual File System allows multiple real
  file systems to be connected into a single
  virtual file system.

7
Mounting File Systems

• At Boot
• An empty Root File System is set up as /
• A physical device (hda) is mounted on /
• /dev and /proc exist within the kernel, and
  must be mounted onto / formally

/ VFS starts here
/ EXT2 is mounted to root from hda
The EXT2 file system can have subdirectories
Empty subdirectories can be used to mount new
file systems
/proc
8
The Directory Tree

• File System separation is hidden in the directory
  structure
• Directory structures (internal to Linux)
• Could contain Files
• Files may be user data or devices, anything that
  is a file
• Could contain a directory
• Could contain a File System
• A file system has a Superblock (containing
  information on what the file system can and cant
  do)
• There must be a root directory, which contains at
  least the files . and ..

9
The Focus Regular Files

• A regular file is a file that contains a block of
  data persistently.
• Persistent files allow actions like
• Read
• Write
• Seek
• Truncate

• Compare this to a Socket, which
• Can Read and Write
• Cannot Seek
• Cannot Truncate

10
In Linux

• A file can essentially allow anything
• So a Socket is a type of file for moving data.
• A regular file is type of file that stores data.
• In Java and C, streams do everything
• So a socket is a type of stream.
• A file is then special stream from a static
  source.
• (seems really complicated)
• Regardless, everything is blocks of data
• Some of it moves around.
• Some of it stays put.

11
So if a regular file is actually a special type
of file, how do we clarify the distinction

• We map generic function calls to specific tasks.

12
The File System API

• Motivation for Using the File System API
• Your service is something that can be accessed by
  opening a name.
• You would like your service to be browsable in a
  directory tree.
• Anti Motivations
• Your service is generally managing blocks that
  are too short lived or anonymous to be browsable,
  or even uniquely named.

13
FUSE (File systems in USEr space)

• A Kernel module that maps function calls from the
  Virtual File System to user level programs.
• Fast development, and allows user debugging
  tools, like printf().
• printf() is your friend

14
The FUSE diagram
MaiaFS
Needy Program
FUSE
Kernel Space
Virtual File System
Kernel
15
Fuse Initialization
static struct fuse_operations maia_oper
.init maia_init, .getattr
maia_getattr, .readdir maia_readdir,
.truncate maia_truncate, .open
maia_open, .release maia_release,
.read maia_read, .write
maia_write, .chmod maia_chmod,
.chown maia_chown, int main(int argc,
char argv) return fuse_main(argc, argv,
maia_oper, NULL)
16
Regular Files have names.

• The getattr() function describes a file at a
  given path location.

static int maia_getattr(const char path, struct
stat stbuf) if(strcmp(path, "/") 0)
stbuf-gtst_mode S_IFDIR 0755
stbuf-gtst_nlink 2 else
if(strcmp(path,/low_priority) 0)
stbuf-gtst_mode S_IFREG 0755
stbuf-gtst_nlink 1 stbuf-gtst_size
FAT000.file_size_bytes else
return -ENOENT return 0
17
Regular Files can be directories.

• The readdir() function gets all the file names
  that are contained in a directory.

static int maia_readdir(const char path, void
buf, fuse_fill_dir_t filler, off_t offset,
struct fuse_file_info fi) if(strcmp(path,
"/") ! 0) return -ENOENT
filler(buf, ".", NULL, 0) filler(buf, "..",
NULL, 0) filler(buf, "flag_low", NULL, 0)
filler(buf, flag_high", NULL, 0)
return 0
18
Regular Files have data.

• The read() function gets data from a file.

static int maia_read(const char path, char buf,
size_t size, off_t offset,
struct fuse_file_info fi) if( file
name_lookup(path, name_map) ) /ltsnipgt error
checkinglt/snipgt/ memcpy(buf,
file.dataoffset, size) return
size else return -ENOENT
19
Regular Files store data.

• The write() function stores data into a file.

static int maia_write(const char path, char
buf, size_t size, off_t offset,
struct fuse_file_info fi) if( file
name_lookup(path, name_map) ) /ltsnipgt error
checkinglt/snipgt/ memcpy( file.dataoffset,
buf, size) return size else return
-ENOENT
20
What does ls -lh do?

• -l is the flag to display more information.
• -h is to make the info human readable.
• ls calls two functions
• readdir(), which lists the files in a directory
• getattr(), which describes file attributes
• Only the directory file is ever read

21
With those functions.

• A primitive, mountable file system can exist and
  function.
• You could get the data from anywhere!
• This is the secret power of the VFS.

22
Implementing MaiaFS

• MaiaFS was designed for different operating
  system and hardware.
• MaiaFS has a strange way of deleting data.
• MaiaFS has a priority structure.

23
MaiaFS Specifications

• Data is deleted without asking.
• Data is deleted in whole blocks.
• Data is deleted from the file start.
• Data writes are not split over blocks.

24
MaiaFS Design

• Steal from the poor, give to the rich!

High priority file
First Block

Last Block
Reallocated Block
Low priority file
Deallocated Block
New First Block

Last Block
Oldest file block
25
MaiaFS Design

• File Allocation Table (FAT) is a prioritized list
  of entries, each with
• first_block_index
• last_block_index
• file_size_blocks
• file_size_bytes
• A FAT entry holds the head and tail of a Linked
  List of blocks.

26
MaiaFS Storage Block

• File Block Array (FBA) is a linear block of
  addressable bytes segmented at regular intervals
  in to FILE_BLOCKS.

27
Return of the linked list

• A linked list makes storing the file easily
  scalable, but random access is slow.
• New blocks are attached to the end of the file to
  hold new writes.
• Old blocks are deleted from the front of the list
  if they are needed elsewhere.

28
A in the File Block Array, with 3 blocks
EDAC is like a federal agency 1)It is there to
help 2)Noone actually knows its there 3)It
really just slows things down
29
Results
30

• chad_at_jhat cat flag_low
• something for flag_low
• chad_at_jhat cat flag_high
• This is the official and dramatically longer than
  ever test text for flag_high
• This is the official and dramatically longer than
  ever test text for flag_high
• This is the official and dramatically longer than
  ever test text for flag_high
• This is the official and dramatically longer than
  ever test text for flag_high
• This is the official and dramatically longer than
  ever test text for flag_high

31

• chad_at_jhat ls lh
• total 0
• -rwxrwxrwx 1 root root 1.4K Jan 1 1970
  flag_high
• -rwxrwxrwx 1 root root 23 Jan 1 1970 flag_low
• chad_at_jhat echo Completely new text that goes
  into flag_high, and destroys the contents of
  flag_low gtgt flag_high
• chad_at_jhat ls lh
• total 0
• -rwxrwxrwx 1 root root 1.5K Jan 1 1970
  flag_high
• -rwxrwxrwx 1 root root 23 Jan 1 1970 flag_low
• chad_at_jhat echo Completely new text that goes
  into flag_high, and destroys the contents of
  flag_low gtgt flag_high
• mchad_at_jhat ls lh
• total 0
• -rwxrwxrwx 1 root root 1.6K Jan 1 1970
  flag_high
• -rwxrwxrwx 1 root root 0 Jan 1 1970 flag_low
• chad_at_jhat cat flag_low

32
Conclusions

• The VFS is both flexible and powerful.
• The Maia File System works.
• Writing applications that use FUSE is fun.