Reliable Distributed Systems

1
Reliable Distributed Systems

• RPC and Client-Server Computing

2
Remote Procedure Call

• Basic concepts
• Implementation issues, usual optimizations
• Where are the costs?
• Reliability and consistency
• Multithreading debate

3
A brief history of RPC

• Introduced by Birrell and Nelson in 1985
• Pre-RPC Most applications were built directly
  over the Internet primitives
• Their idea mask distributed computing system
  using a transparent abstraction
• Looks like normal procedure call
• Hides all aspects of distributed interaction
• Supports an easy programming model
• Today, RPC is the core of many distributed systems

4
More history

• Early focus was on RPC environments
• Culminated in DCE (Distributed Computing
  Environment), standardizes many aspects of RPC
• Then emphasis shifted to performance, many
  systems improved by a factor of 10 to 20
• Today, RPC often used from object-oriented
  systems employing CORBA or COM standards.
  Reliability issues are more evident than in the
  past.

5
The basic RPC protocol
client
server
binds to server
registers with name service
6
The basic RPC protocol
client
server
binds to server prepares, sends request
registers with name service receives request
7
The basic RPC protocol
client
server
binds to server prepares, sends request
registers with name service receives
requestinvokes handler
8
The basic RPC protocol
client
server
binds to server prepares, sends request
registers with name service receives
requestinvokes handlersends reply
9
The basic RPC protocol
client
server
binds to server prepares, sends
request unpacks reply
registers with name service receives
requestinvokes handlersends reply
10
Compilation stage

• Server defines and exports a header file giving
  interfaces it supports and arguments expected.
  Uses interface definition language (IDL)
• Client includes this information
• Client invokes server procedures through stubs
• provides interface identical to the server
  version
• responsible for building the messages and
  interpreting the reply messages
• passes arguments by value, never by reference
• may limit total size of arguments, in bytes

11
Binding stage

• Occurs when client and server program first start
  execution
• Server registers its network address with name
  directory, perhaps with other information
• Client scans directory to find appropriate server
• Depending on how RPC protocol is implemented, may
  make a connection to the server, but this is
  not mandatory

12
Data in messages

• We say that data is marshalled into a message
  and demarshalled from it
• Representation needs to deal with byte ordering
  issues (big-endian versus little endian), strings
  (some CPUs require padding), alignment, etc
• Goal is to be as fast as possible on the most
  common architectures, yet must also be very
  general

13
Request marshalling

• Client builds a message containing arguments,
  indicates what procedure to invoke
• Due to the need for generality, data
  representation a potentially costly issue!
• Performs a send I/O operation to send the message
• Performs a receive I/O operation to accept the
  reply
• Unpacks the reply from the reply message
• Returns result to the client program

14
Costs in basic protocol?

• Allocation and marshalling data into message (can
  reduce costs if you are certain client, server
  have identical data representations)
• Two system calls, one to send, one to receive,
  hence context switching
• Much copying all through the O/S application to
  UDP, UDP to IP, IP to ethernet interface, and
  back up to application

15
Schroeder and Burroughs

• Studied RPC performance in O/S kernel
• Suggested a series of major optimizations
• Resulted in performance improvments of about
  10-fold for Xerox firefly workstation (from 10ms
  to below 1ms)

16
Typical optimizations?

• Compile the stub inline to put arguments
  directly into message
• Two versions of stub if (at bind time) sender
  and dest. found to have same data
  representations, use host-specific rep.
• Use a special send, then receive system call
  (requires O/S extension)
• Optimize the O/S kernel path itself to eliminate
  copying treat RPC as the most important task
  the kernel will do

17
Fancy argument passing

• RPC is transparent for simple calls with a small
  amount of data passed
• Transparent in the sense that the interface to
  the procedure is unchanged
• But exceptions thrown will include new exceptions
  associated with network
• What about complex structures, pointers, big
  arrays? These will be very costly, and perhaps
  impractical to pass as arguments
• Most implementations limit size, types of RPC
  arguments. Very general systems less limited but
  much more costly.

18
Overcoming lost packets
client
server
sends request
19
Overcoming lost packets
client
server
sends request
Timeout!
retransmit
duplicate request ignored
ack for request
20
Overcoming lost packets
client
server
sends request
Timeout!
retransmit
ack for request
reply
21
Overcoming lost packets
client
server
sends request
Timeout!
retransmit
ack for request
reply
ack for reply
22
Costs in fault-tolerant version?

• Acks are expensive. Try and avoid them, e.g. if
  the reply will be sent quickly supress the
  initial ack
• Retransmission is costly. Try and tune the delay
  to be optimal
• For big messages, send packets in bursts and ack
  a burst at a time, not one by one

23
Big packets
client
server
sends request as a burst
ack entire burst
reply
ack for reply
24
RPC semantics

• At most once request is processed 0 or 1 times
• Exactly once request is always processed 1 time
• At least once request processed 1 or more times
• ... but exactly once is impossible because we
  cant distinguish packet loss from true failures!
  In both cases, RPC protocol simply times out.

25
Implementing at most/least once

• Use a timer (clock) value and a unique id, plus
  sender address
• Server remembers recent ids and replies with
  same data if a request is repeated
• Also uses id to identify duplicates and reject
  them
• Very old requests detected and ignored by
  checking time
• Assumes that the clocks are working
• In particular, requires synchronized clocks

26
RPC versus local procedure call

• Restrictions on argument sizes and types
• New error cases
• Bind operation failed
• Request timed out
• Argument too large can occur if, e.g., a table
  grows
• Costs may be very high
• ... so RPC is actually not very transparent!

27
RPC costs in case of local destination process

• Often, the destination is right on the callers
  machine!
• Caller builds message
• Issues send system call, blocks, context switch
• Message copied into kernel, then out to dest.
• Dest is blocked... wake it up, context switch
• Dest computes result
• Entire sequence repeated in reverse direction
• If scheduler is a process, context switch 6 times!

28
RPC example
Dest on same site
O/S
Source does xyz(a, b, c)
29
RPC in normal case
Destination and O/S are blocked
Dest on same site
O/S
Source does xyz(a, b, c)
30
RPC in normal case
Source, dest both block. O/S runs its scheduler,
copies message from source out-queue to dest
in-queue
Dest on same site
O/S
Source does xyz(a, b, c)
31
RPC in normal case
Dest runs, copies in message
Dest on same site
O/S
Source does xyz(a, b, c)
Same sequence needed to return results
32
Broad comments on RPC

• RPC is not very transparent
• Failure handling is not evident at all if an RPC
  times out, what should the developer do?
• Reissuing the request only makes sense if there
  is another server available
• Anyhow, what if the request was finished but the
  reply was lost? Do it twice? Try to duplicate
  the lost reply?
• Performance work is producing enormous gains
  from the old 75ms RPC to RPC over U/Net with a
  75usec round-trip time a factor of 1000!

33
Contents of an RPC environment

• Standards for data representation
• Stub compilers, IDL databases
• Services to manage server directory, clock
  synchronization
• Tools for visualizing system state and managing
  servers and applications

34
Closely Related Topic

• Multithreading is a common performance-enhancing
  technique
• Idea is that server is often idle while doing I/O
  for one client, so use extra threads to allow
  concurrent request processing
• In the limit, leads to database transactional
  concurrency model, but many non-transactional
  servers use threads for enhanced performance

35
Multithreading debate

• Three major options
• Single-threaded server only does one thing at a
  time, uses send/recv system calls and blocks
  while waiting
• Multi-threaded server internally concurrent,
  each request spawns a new thread to handle it
• Upcalls event dispatch loop does a procedure
  call for each incoming event, like for X11 or
  PCs running Windows.

36
Single threading drawbacks

• Applications can deadlock if a request cycle
  forms Im waiting for you and you send me a
  request, which I cant handle
• Much of system may be idle waiting for replies to
  pending requests
• Harder to implement RPC protocol itself (need to
  use a timer interrupt to trigger acks,
  retransmission, which is awkward)

37
Multithreading

• Idea is to support internal concurrency as if
  each process was really multiple processes that
  share one address space
• Thread scheduler uses timer interrupts and
  context switching to mimic a physical
  multiprocessor using the smaller number of CPUs
  actually available

38
Multithreaded RPC

• Each incoming request is handled by spawning a
  new thread
• Designer must implement appropriate mutual
  exclusion to guard against race conditions and
  other concurrency problems
• Ideally, server is more active because it can
  process new requests while waiting for its own
  RPCs to complete on other pending requests

39
Negatives to multithreading

• Users may have little experience with concurrency
  and will then make mistakes
• Concurrency bugs are very hard to find due to
  non-reproducible scheduling orders
• Reentrancy can come as an undesired surprise
• Threads need stacks hence consumption of memory
  can be very high
• Deadlock remains a risk, now associated with
  concurrency control
• Stacks for threads must be finite and can
  overflow, corrupting the address space

40
Threads can spawn too many
SCHED
event
41
Threads can spawn too many
Thread spawned, but blocks
SCHED
event
42
Threads can spawn too many
SCHED
Eventually, application becomes bloated, begins
to thrash. Performance drops and clients may
think the server has failed
event
43
Upcall model

• Common in windowing systems
• Each incoming event is encoded as a small
  descriptive data structure
• User registers event handling procedures
• Dispatch loop calls the procedures as new events
  arrive, waits for the call to finish, then
  dispatches a new event

44
Upcalls combined with threads

• Perhaps the best model for RPC programming
• Each handler can be tagged needs thread, or can
  be executed unthreaded
• Developer must still be very careful where
  threads are used

45
Recent RPC history

• RPC was once touted as the transparent answer to
  distributed computing
• Today the protocol is very widely used
• ... but it isnt very transparent, and
  reliability issues can be a major problem
• Today the strongest interest is in Web Services
  and CORBA, which use RPC as the mechanism to
  implement object invocation