Programming Project

1
Programming Project 4Kernel Message Passing
System

• A-term 2008

2
Project

• To build and test a message-passing system for
  Inter-process Communication among separate
  address spaces
• Implement API for message-passing functions
• Kernel system calls to handle messages
• Test program to exercise the message passing
  system

3
Objective

• To gain experience programming and testing
  synchronization and IPC operations
• To gain experience with synchronization memory
  management in the Linux Kernel

4
Overview

• Add a mailbox to each process
• Abstract object capable of holding messages
• Messages are of bounded length, undefined
  structure
• Mailboxes may be of bounded size
• All threads of a process share a mailbox
• Any Linux task can send a message to any mailbox
• Including own mailbox
• Addressed by pid_t

5
Overview (continued)

• Task may receive message only from own mailbox
• Any thread of task may receive!
• Mailbox created during fork
• Mailbox deleted during process termination
• Mailbox may be stopped
• I.e., no more message are accepted
• Optionally, mailbox may be flushed

6
This Project

• Predefined API (Application Program Interface) at
• mailbox.h in this directory
• All students must implement same API
• User space interface program
• Kernel implementation
• Test program

7
API

• int SendMsg(pid_t dest, void msg, int len, bool
  block)
• Sends a message body at msg to process dest
• Length len not more than MAX_MSG_SIZE
• Blocks if mailbox full and block TRUE
• Returns zero if successful, error code if not
• int RcvMsg(pid_t sender, void msg,int len,
  bool block)
• Gets a message from own mailbox, puts in msg
• Sender process ID returned in sender
• Blocks if mailbox empty and block TRUE
• Returns zero if successful, error code if not
• Messages in FIFO order

8
API (continued)

• int ManageMailbox(bool stop, int count)
• Gets number of message currently queued in
  mailbox
• If stop TRUE, prevents mailbox from receiving
  more messages
• Unblocks all waiting SendMsg and RcvMsg
• Future RcvMsg calls can still retrieve remaining
  queued messages
• Returns zero if successful, error code if not

9
API Documented Error Codes

• MAILBOX_FULL
• Non-blocking send
• MAILBOX_EMPTY
• Non-blocking receive
• MAILBOX_STOPPED
• On any send
• Also after blocked send or receive
• MAILBOX_INVALID
• On any call

• MSG_TOO_LONG
• On send
• MSG_ARG_ERROR
• Invalid argument or pointer
• copy_to_user or copy_from_user fails
• MAILBOX_ERROR
• Any other kind of error
• You may add other error codes as needed

10
Kernel Implementation

• Start with prePatch-Project4
• I.e., so we all have same task_struct and header
  files
• Three system calls
• .long sys_mailbox_send.long sys_mailbox_rcv.lon
  g sys_mailbox_manage
• Create and delete mailboxes
• Memory allocation within kernel
• All messages of fixed size
• Synchronization within kernel
• Suggest simulating a monitor per mailbox

11
Kernel Implementation

• Start with prePatch-Project4
• I.e., so we all have same task_struct
• Three system calls
• .long sys_mailbox_send.long sys_mailbox_rcv.lon
  g sys_mailbox_manage
• Create and delete mailboxes
• Memory allocation within kernel
• All messages of fixed size
• Synchronization within kernel

12
Pre-Patch your Kernel

• Apply prePatch-Project4 to a clean kernel tree
• Adds to task_struct
• struct cs3013_mailbox mailbox
• Defines syscalls in syscall.S and unistd.h
• .long sys_mailbox_send / 324 /.long
  sys_mailbox_rcv / 325 / .long
  sys_mailbox_manage / 326 /
• Reason
• Graders time in grading
• Dont modify other commonly used header files!

13
Kernel Implementation

• Start with prePatch-Project4
• I.e., so we all have same task_struct
• Three system calls
• .long sys_mailbox_send.long sys_mailbox_rcv.lon
  g sys_mailbox_manage
• Create and delete mailboxes
• Memory allocation within kernel
• All messages of fixed size
• Synchronization within kernel

14
Creating a Mailbox

• During do_fork()
• In kernel/fork.c
• If this is a new process
• Use kmalloc() to allocate a new data structure
  for a new mailbox
• Initialize
• Set pointer in task_struct-gtmailbox
• If this is just a new thread
• Keep existing mailbox of process
• Indicated by CLONE_THREAD argument to do_fork()

15
Creating a Mailbox (continued)

• No mailbox for kernel threads, etc.
• Indicated by null task_struct-gtmm
• Set task_struct-gtmailbox to null

16
Kernel Implementation of Mailbox Operations

• Use Project 3 as a model (if applicable)
• Monitor model per mailbox
• Three system calls for user-visible functions
• sys_mailbox_sendsys_mailbox_rcvsys_mailbox_mana
  ge
• Two other functions for kernel support
• mailbox_createmailbox_destroy
• Contents of monitor data structure
• Monitor lock
• spinlock_t initialized to SPIN_LOCK_UNLOCKED
• Semaphore and count to simulate condition
  variable
• Both initialized to zero
• List head for linked list of messages flag for
  stopped mailbox
• Other fields as needed

17
Kernel Implementation (continued)

• RcvMsg
• Grab monitor lock
• While linked list is empty
• Simulate wait on condition variable
• Unlink first message from mailbox linked list
• Release monitor lock
• copy_to_user to copy message body and other
  information to caller
• Free kernel space for unlinked message (see
  below)
• Non-blocking receive
• Exercise for student

18
Kernel Implementation (continued)

• RcvMsg
• Grab monitor lock
• While linked list is empty
• Simulate wait on condition variable
• Unlink first message from mailbox linked list
• Release monitor lock
• copy_to_user to copy message body and other
  information to caller
• Free kernel space for unlinked message (see
  below)
• Non-blocking receive
• Exercise for student

19
Kernel Implementation (continued)

• RcvMsg
• Grab monitor lock
• While linked list is empty
• Simulate wait on condition variable
• Unlink first message from mailbox linked list
• Release monitor lock
• copy_to_user to copy message body and other
  information to caller
• Free kernel space for unlinked message (see
  below)
• Non-blocking receive
• Exercise for student

20
Kernel Implementation (continued)

• RcvMsg
• Grab monitor lock
• While linked list is empty
• Simulate wait on condition variable
• Unlink first message from mailbox linked list
• Release monitor lock
• copy_to_user to copy message body and other
  information to caller
• Free kernel space for unlinked message (see
  below)
• Non-blocking receive
• Exercise for student

21
Kernel Implementation (continued)

• SendMsg
• Allocate kernel space for new message (see below)
• copy_from_user to copy message body into kernel
• Fill in other details (sender, length, etc.)
• Grab monitor lock
• Link new message to end of mailbox linked list
• Simulate signal to condition variable if any
  receivers waiting
• Release monitor lock
• Blocking send
• Exercise for student
• Must simulate wait on condition variable if
  mailbox is full

22
Kernel Implementation (continued)

• ManageMailbox
• Grab monitor lock
• Count number of messages
• If stop TRUE, determine if any tasks waiting
• If so, trick them into unblocking and returning
  error
• Release monitor lock
• Revisit blocking SendMsg, RcvMsg
• When unblocked, be sure mailbox has not been
  stopped in meantime.

23
Deleting a Mailbox

• In do_exit()
• Only if the task group is dead!
• Stop the mailbox to be sure that blocked send
  operations can complete!
• Flush messages (to free their space)
• Make sure all blocked operations are done
• Free the mailbox data structure
• kfree()
• Zero out task_struct-gtmailbox

24
Memory Allocation Resources

• kmalloc(), kfree()
• linux/slab.h
• Similar to malloc() free(), but with flags
• Slab allocator
• kmem_cache_t kmem_cache_create()
• void kmem_cache_alloc()
• void kmem_cache_free()
• int kmem_cache_destroy()

25
Memory Allocation Resources (continued)

• Use slab allocator
• All message bodies the same size in kernel
• Highly optimized for rapid allocation and free
• Low fragmentation
• Initialization
• Establish cache and any static variables during
  fork_init() (in kernel/fork.c)

26
Locking Tools in the Kernel

• Mailbox
• Needs to be locked to link and unlink messages
• Also to change state (START, STOP)
• Remember the Linux Kernel is fully preemptive!
• System call may be preempted before completion
• Interrupt may schedule another process at any
  time
• Interrupt may manage shared resources
• But not while holding a spinlock
• Due to support for symmetric multi-processing

27
Robert Love says

• It is a major bug if
• An interrupt occurs to access a resource while
  kernel code is also manipulating that resource
• Kernel code is preempted while accessing a shared
  resource
• Kernel code sleeps while in the middle of a
  critical section
• Two processors access same data at same time
• Implementing locking is not hard
• Tricky part is identifying what to lock.

28
Tools for Simulating Monitors

• Spin locks (for monitor locking)
• spin_lock_init()
• spin_lock(), spin_unlock()
• See linux/spinlock.h
• Must be used to provide mutual exclusion for all
  monitor functions
• Also to release and acquire lock around condition
  variable waits

29
Condition Variable (simulated)

• Condition variable representation
• Struct semaphore, int wait_count, both
  initialized to zero
• See asm/semaphore.h
• Wait on condition variable
• / while holding the monitor lock
  /wait_count / incr of waiting tasks
  /spin_unlock(mr_lock) /release lock
  /down_interruptible(sem) / wait
  /spin_lock(mr_lock) / reacquire lock /
• Signal condition variable
• / while holding the monitor lock /if
  (wait_count gt 0) up(sem) / unblock a task
  / wait_count-- / decr of waiting tasks /

30
Monitor Example
/ function implementations / FIFOMessageQueue(vo
id) / constructor/head tail
NULL void addMsg(msg_t newMsg) qItem new
malloc(qItem)new?prev tailnew?next
NULL if (tailNULL) head newelse tail?next
new tail new signal nonEmpty
monitor FIFOMessageQueue struct qItem struct
qItem next,prev msg_t msg / internal
data of queue/ struct qItem head,
tailcondition nonEmpty / function prototypes
/ void addMsg(msg_t newMsg)msg_t
removeMsg(void) / constructor/destructor
/ FIFOMessageQueue(void)FIFOMessageQueue(void)

Adapted from Kleiman, Shah, and Smaalders
31
Monitor Example
/ function implementations concluded/ FIFOMes
sageQueue(void) / destructor/while (head ltgt
NULL) struct qItem top headhead
top?nextfree(top) / what is missing here?
/
/ function implementations continued/ msg_t
removeMsg(void) while (head
NULL) wait(nonEmpty) struct qItem old
head if (old?next NULL) tail NULL /last
element/else old?next?prev NULL head
old?next msg_t msg old?msg free(old) return(m
sg)
32
Monitor Example
/ function implementations concluded/ FIFOMes
sageQueue(void) / destructor/while (head ltgt
NULL) struct qItem top headhead
top?nextfree(top) / what is missing here?
// Answer- need to unblock waiting threads in
destructor! /
/ function implementations continued/ msg_t
removeMsg(void) while (head
NULL) wait(nonEmpty) struct qItem old
head if (old?next NULL) tail NULL /last
element/else old?next?prev NULL head
old?next msg_t msg old?msg free(old) return(m
sg)
33
Deleting mailbox

• Must stop mailbox first
• In a loop, unblock all waiting tasks
• Be sure any waiting task has had time to complete
  operation
• Potential race condition
• Flush remaining messages
• Delete mailbox data structure

34
Testing

• Based on Project 3, but
• Multiple processes
• Each process with multiple threads
• Fork multiple processes
• Create mailboxes, exchange mailbox IDs
• Randomly send messages to each other
• Payload must be self identifying
• Acknowledge received messages
• Test extreme conditions
• E.g., fill up mailbox

35
Due Dates

• Project due at Sunday, October 12, 600 PM
• Pace yourself
• By September 29
• Creation deletion of mailboxes in do_fork(),
  do_exit()
• Design of mailbox data structure for
  producer-consumer
• By October 5
• Sending and receiving messages
• User space support and simple test program
• October 12 (final submission)
• Stopping mailbox with blocked calls on SendMsg or
  RcvMsg
• Flushing messages
• Comprehensive test program

36
Final Submission

• Submit using web-based turnin program
• http//web.cs.wpi.edu/kfisler/turnin.html
• Include
• One patch file for kernel implementation
• Difference from prePatch-Project4 and your
  implementation
• User space implementation of mailbox.h interface
• Test program(s) and results
• Makefile
• Write-up must explain synchronization
  programming invariants
• Sending and receiving messages
• Shutting down mailboxes
• Put your name on all documents at top of every
  edited file!

37
Individual vs. Teams

• This may be an individual project or a 2-person
  team project
• If two people wish to be a team, register with
  TAs by 500 PM on September 26
• Even if you have worked together on previous
  projects

38
Extra Credit (10 points)

• There is a lingering race condition in the
  simulation of condition variables
• Find it
• Propose a solution
• Argue/explain that the solution fixes the race
  condition (and does not introduce others)

39
Questions?