Message Passing Distributed Systems with examples from Socket based Client/Server Systems

1
Message Passing Distributed Systemswith
examples from Socket based Client/Server Systems
2
Overview

1. Distributed Computing Topologies
2. Critical Points in C/S Topologies
3. The Message Passing Model
4. Delivery Guarantees
5. Request Ordering and Reliable Broadcast
6. Programming C/S Systems with Sockets

Starting with the common C/S model, we will
develop core properties for distributed protocols.
3
Distributed Computing Topologies
- Client/Server - Hierarchical - Totally
Distributed - Bus Topologies
4
Client/Server Systems
server
client
request
Request processing
response
Clients initiate communication and (typically)
block while waiting for the server to process the
request. Still the most common DS topology.
5
Hierarchical Systems
Client and Server
Every node can be both client and server but some
play a special role, e.g. are Domain Name System
(DNS) server. A reduction of communication
overhead and central control options are some of
the attractive features of this topology.
6
Totally distributed
Client and Server
Every node IS both client and server. Watch out
peer to peer systems need not be totally
distributed!
7
Bus Systems
Client and Server
Every node listens for data and posts data in
response. This achieves a high degree of
separation and indepencence. Event-driven systems
follow this topology.
8
Critical Points in C/S Topologies
- Load Management, Delays and Bottlenecks -
Security - Queuing Theory Aspects - Sync. vs.
Async. Processing Patterns
9
Critical Points in C/S Systems
server
client
Processing clustering?, single/multithr.?
Session state? Authentication?authorization?
Privacy?
Requester locate server? Authenticate?
Synchronous vs asynchr. call?
request
response
Bandwidth/latency?
Scalability, Availability and Security all depend
on the answers to those questions.
10
Queuing Theory for Multi-tier Process Networks
A modern application servers performance largely
relies on the proper configuration of several
queues from network listening to threadpools etc.
Queuing theory lets us determine the proper
configurations (see resources). In general,
architectures like above are very sensitive for
saturated queues. Good architectures create a
funnel from left to right and limit resources
like max. threads. Caching an batching are
directly derived from queuing theory. Picture
from http//publib.boulder.ibm.com/infocenter/was
info/v5r1//index.jsp?topic/com.ibm.websphere.base
.doc/info/aes/ae/rprf_queue.html
11
Synchronous I/O (blocking calls)
FileSys
Output Channel
Input Channel
Thread
Switch
Wait for client cmd.
Process client cmd, e.g. get file
Switch
Send response to client
Switch
Wait for client cmd.
Many threads are required to stay responsive.
Many context switches occur and each thread needs
extra memory. see Aruna Kalaqanan et.al.
http//www-128.ibm.com/developerworks/java/library
/j-javaio
12
Non-Blocking Reactor Pattern
Server applications in a distributed system must
handle multiple clients that send them service
requests. Before invoking a specific service,
however, the server application must demultiplex
and dispatch each incoming request to its
corresponding service provider. The Reactor
pattern serves precisely this function. It allows
event-driven applications to demultiplex and
dispatch service requests, which are then
delivered concurrently to an application from one
or more clients.
The Reactor pattern is closely related to the
Observer pattern in this aspect all dependents
are informed when a single subject changes. The
Observer pattern is associated with a single
source of events, however, whereas the Reactor
pattern is associated with multiple sources of
events.
From Aruna Kalaqanan et.al. http//www-128.ibm.co
m/developerworks/java/library/j-javaio. The
downside all processing needs to be non-blocking
and the threads need to maintain the state of the
processing between handler calls (explicit state
management vs. implicit in normal multi-threaded
designs).
13
Reactor Pattern
From Doug Lea (see resources). Note that this is
the single threaded version. Symbian OS uses a
similiar concept called Active Objects. Handlers
are not allowed to block. In many cases a single
thread may be enough.
14
The Message Passing Model
- Modelling and Automata - Async. vs. Sync
Systems - Protocol Properties Correctness,
Liveness, Fairness,... - Complexity - Failure
Types
15
Modeling of Distributed Systems
Ptp primitives
Process I State variables
Inbuf
lt - Receive
Outbuf
Send -gt
(After J.Aspnes) A processing function takes
the Inbuf Data from other processes, the internal
state variables and computes a new internal state
and new Outbuf data Communication ist
point-to-point and deterministic. A configuration
is the state vector for all processes. Events
change configurations into new ones. An execution
is a sequence of Configurations and Events C0
e0, C1 e1, C2 e2 ...
16
Synchronous vs. Asynchronous Systems
Synchronous (lockstep) e event, t time e0t0
---gt delivery at t01, e1t1 ---gt deliv. T11, ..
Asynchronous (delayed) e0t0 ---gt delivery at
?, e1t1 ---gt deliv. T1?, .. Reqs infinitely
many comp. Steps possible, events will be
eventually delivered.
Synchronous systems have simpler distributed
algorithms, but are harder to build. The reality
is async. Systems with additonal help from
failure detectors, randomization etc.
17
Protocol Properties
- Correctness invariant properties are shown to
hold throughout executions - Liveness/Termination
the protocol is shown to make progress in the
context of certain failures and in a bounded
number of rounds - Fairness no starvation for
anybody - Agreement e.g. all processes agree
output the same decision - Validity for the same
input x, all processes output according to x (Or
there is a possible execution for every possible
output value)
(after Aspnes).
18
Complexity
- Time complexity the time of the last event
before all processes finish (Aspens) - Message
complexity the number of messages sent
Message size and the number of rounds needed for
termination are important for the scalability of
protocols
19
Failure Types
- Crash failure a process stops working and
stays down - Connectivity failures network
failures e.g. causing split brain situations with
two separate networks or node isolation.
Typically the time for message propagation is
affected. - Message loss single messages are
lost, nodes are up. - Byzantine Failures Evil
nodes violating protocol assumptions and
promises. E.g. breaking a promise due to disk
failure, configuration failer etc.
All protocols are validated with respect to
certain failure scenarios!!
20
The Role of Delivery Guarantees
- Problem Scenario Shop order - TCP
Communication properties - Message complexity
the number of messages sent
21
Crash During Shop Order
shop
Message lost
Order
Order processing
Shop
user
Order confirmation
Browser crashed
Server crashed
Lost in transmit
What happens to the order when certain failure
types apply? What kind of guarantees do you have?
Does TCP Help? What outcomes to you expect?
22
TCP communication properties

• lost messages retransmitted
• Re-sequencing of out of order messages
• Sender choke back (flow control)
• No message boundary protection

These features form a reliable communication
channel. This does not include proper behavior
in case of connection failures! (timeout
problem). (Ken Birman, building secure and
reliable network applications, chapter 1)
23
Communication Failure timeout
Case Client sends request and receives timeout

Failure Cases a) network problem, server did not
receive request b) Server problem OS did receive
request but server crashed during work c)
OS/Network problem Server finished request but
response got lost or OS crashed during send.
Client options drop request (ok in c), resend
request (ok in a and b), send request to
different server (ok in a and b). Other client
actions lead either to lost or duplicated
requests.
24
Delivery Guarantees

• Best effort (doesnt guarantee anything)
• At least once (same request several times
  received)
• At most once (not more than once but possibly
  not at all)
• Once and only once (simply the best)

In case of channel break-down TCP does NOT make
ANY delivery guarantees. This becomes your job
then (or better a job for your middleware)
25
At most once implementation for non-idempotent
requests
server
A response is stored until client confirms
client
Request
request
Response (ack)
By adding a request number to each request the
server can detect duplicate requests and throw
them away. The server itself needs to store a
response until the client acknowledges that it
was received. This creates state on the server!
26
Idempotent Requests?

• Get bank account balance
• Push elevator button
• Get /index.html .
• Book flight
• Cancel flight
• .

What kind of delivery guarantee do you need for
idempotent service requests?
27
Idempotency

• Is not a medical condition (Pat Helland)
• The first request needs to be idempotent
• The last request can only be best effort
• Messages may be reordered.
• Your partner may experience amnesia as a result
  of failures, poorly managed durable state, or
  load-balancing switchover to its evil twin.
• .

Pat Helland, http//queue.acm.org/detail.cfm?id21
87821
28
Request Ordering with Multiple Nodes
- Reliable Broadcast - FifoCast - Causal Cast -
Absolute Ordered Casts
Taken from C. Karamanoulis and K.Birman
29
Fault-tolerant Broadcast Model
Process I
Process II
Broadcast primitives
Bcast (m,)
Deliver (m,)
Del. (m,)
Middle-Ware
Middle-Ware
Ptp primitives
Send (m,)
Receive (m,)
Communication Layer
Watch out messages can be delivered without
respect to some order. Or they can be sorted,
kept back at the middleware layer and only
delivered when a certain order can be guaranteed.
Notice the self-delivery of messages by the
sending process.
30
Reliable Broadcast with no Order
order
client
cancel
rebate
server
???? A cancel request without previous order
Taken from C. Karamanoulis, Reliable Broadcasts
31
Reliable Broadcast with FiFo-Order
M2
M1
M3
client
M3 gets FiFO delvd. Here
server
M1 Recv. Rel.delivery FiFo delivery
M3 Recv. Rel.delivery FIFO del. Delayed!!!
Taken from C. Karamanoulis,
32
Causal Violation with FiFO Order
M2
M1
M3
Stud1
Lecture cancelled!
Let's go somewhere!
Stud2
But we have a lecture???
Stud3
M1 FiFo delivered!
Taken from C. Karamanoulis, Local Order If a
process delivers a message m before broadcasting
a message m, then no correct process delivers m
unless it has previously delivered m.
33
Solutions for Causal Ordered Broadcasts
- Piggyback every message sent with privious
messages Processes which missed a message can
learn about it with The next incoming message and
then deliver correctly - Send event history with
every message (e.g. using vector Clocks. Delay
delivery until order is correct.
Taken from C. Karamanoulis and K.Birman. What
are the advantages/disadvantages of both
solutions?
34
Causal Re-Ordering
M1
M3
p1
M2
p2
p3
M2 delivered!
M3 delayed, until M1 delivered!
Taken from C. Karamanoulis, P3 has delivered M2
to itself, before delivering M1. Is this a
problem? Think about causal dependency and what
causes it!
35
Replication Anomalies with Causal Order
State100
State200
State2000
Replica 1
Add 100
Multiply by 10
Replica 2
State100
State1000
State1100
Taken from C. Karamanoulis, Total Order If
correct processes p and q both deliver messages m
and m, then p delivers m before m if and only
if q delivers m before m.
36
Solutions for Atomic Broadcasts (Total Order)
- All nodes send messages to every other node. -
All nodes receive messages, but wait with
delivery - One node has been selected to organize
total order. - This node orders all messages into
a total order - This node sends the total order
to all nodes - All nodes receive the total order
and deliver their messages According to this
order.
Taken from K.Birman. What are the
advantages/disadvantages of this solution?
37
Programming Client/Server Systems with Sockets
and different I/O Models
38
Overview

• Socket primitives
• Process Model with sockets
• Example of server side socket use
• Transparency and socket programming?
• Security, Performance, Availability, Flexibility
  etc. of socket based C/S.
• Typical C/S infrastructure (Proxies, Firewalls,
  LDAP)

39
Protocol Stack for Sockets
Socket host B, port 80, tcp-conn
Socket host A, port 3500, tcp-conn
Reliable comm. channel
Transport/Session
Transport/Session
Tcp connection
Udp connection
Network
Network
Data Link
Data Link
Physical
Physical
40
Socket Properties

• Using either tcp or udp connections
• Serving as a programming interface
• A specification of Host, Port, Connection
  type
• A unique address of a channel endpoint.

41
Berkeley Sockets (1)
Primitive Meaning
Socket Create a new communication endpoint
Bind Attach a local address to a socket
Listen Announce willingness to accept connections
Accept Block caller until a connection request arrives
Connect Actively attempt to establish a connection
Send Send some data over the connection
Receive Receive some data over the connection
Close Release the connection

• Socket primitives for TCP/IP.

From van Steen, Tanenbaum, Distributed Systems
42
Berkeley Sockets (2)

• Connection-oriented communication pattern using
  sockets.

From van Steen, Tanenbaum, Distributed Systems
43
Server Side Processing using Processes
Connecting on arbitrary port C
Listening on port X
Client
Server Dispatcher Process
Accept and spawn process on Port Y
Connection established between client on port C
and server on port Y
Server (process)
After spawning a new process the dispatcher goes
back to listening for new connection requests.
This model scales to some degree (process
creation is expensive and only few processes are
possible). Example traditional CGI processing in
web-server
44
Server Side Processing using Threads
Connecting on arbitrary port C
Listening on port X
Client
Server Dispatcher Process
Accept and spawn thread on Port Y
Connection established between client on port C
and server on port Y
Server (thread)
After spawning a new thread the dispatcher goes
back to listening for new connection requests.
This model scales well (thread creation is
expensive but they can be pooled) and a larger
number of threads are possible). Example servlet
request processing in servlet engine (aka
web-container)
45
Server Side Concurrency
Threaded server
Process per request
Thread
Thread
addMoney(account, value)
addMoney(account, value)
In the case of the threaded server the function
needs to be re-entrant. No unprotected global
variables. Keep state per thread on stack.
46
Designing a socket based service

• Design the message formats to be exchanged (e.g.
  http1.0 200 OK ). Try to avoid data
  representation problems on different hardware.
• Design the protocol between clients and server
• - Will client wait for answer? (asynchronous vs.
  synchr. Comm.)
• - Can server call back? ( client has server
  functionality)
• - Will connection be permanent or closed after
  request?
• - Will server hold client related state (aka
  session)?
• - Will server allow concurrent requests?

47
Stateless or Stateful Service?

• Stateless
• Scales extremely well
• Makes denial of service attacks harder
• Forces new authentication and authorization per
  request

• Stateful
• Allows transactions and delivery guarantees
• Can lead to resource exhaustion (e.g. out of
  sockets) on a server
• Needs somehow reliable hardware and networks to
  succeed.

48
Server Dangers Keeping State and expecting
clients to behave -TCP SYN flooding
client
server
server
client
SYN
SYN
Client info stored
Client info stored
SYN
SYN,ACK(SYN)
Client info stored
request
Client info stored
Client never sends request, only SYN, Server
buffer gets filled and other clients cannot
connect
49
A Client using sockets

1. Define hostname and port number of server host
2. Allocate a socket with host and port parameters
3. Get the input channel from the socket (messages
   from server)
4. Get output channel from socket (this is where
   the messages to the server will go)
5. Create a message for the server, e.g. GET
   /somefile.html HTTP/1.0
6. Write message into output channel (message is
   sent to server)
7. Read response from input channel and display it.

A multithreaded client would use one thread to
read e.g. from the console and write to the
output channel while the other thread reads from
the input channel and displays the server
messages on the console (or writes to a file)
50
A server using sockets

1. Define port number of service (e.g. 80 for http
   server)
2. Allocate a server socket with port parameter.
   Server socket does bind and listen for new
   connections.
3. Accept an incoming connection, get a new socket
   for the client connection
4. Get the input channel from the socket and parse
   client message
5. Get output channel from socket (this is where
   the messages to the client will go)
6. Do request processing (or create a new thread to
   do it)
7. Create a response message e.g. HTTP/1.0 2000
   \n
8. Write message into output channel (message is
   sent to client)
9. Read new message from client channel or close the
   connection

A bare bone server. Could be extended through
e.g. a command pattern to match requests with
processing dynamically. New commands could get
loaded dynamically as well. (Application Server)
51
Distribution Transparency with Sockets?

• Invocation The server side function cannot be
  called on the client side. Instead, socket
  operations must be used and messages defined.
• Location/Relocation/Migration If service moves,
  client breaks.
• Replication/Concurrency No support yet
• Failure No support yet
• Persistence No support yet

To be fair socket based services need to deal
with all that but they are still fairly simple to
write!
52
Infrastructure of C/S Systems
Directory
Proxy
Reverse Proxy
Authent. server
Firewalls
Load Balancer
Client (initiate)
Server (process)
Reverse Proxy cache results, end SSL session,
authenticate client Authentication server store
client data, authorize client
Load Balancer distribute
requests across servers
Directory help locate server Proxy check client
authorization, route via firewall
firewall allow outgoing calls only
53
I/O Models

• Thread per Connection (classic, stack-based,
• latency-hiding with threads)
• Non- Blocking (select() types), NIO, stackless
• Async (AIO, NIO2, node.js), stackless
• Callback Style
• Future/Promise Style

54
Non-Blocking

• Single-selector thread doing all work (no CPU
• intensive stuff, no blocking anywhere, scale
  through
• more instances
• Single selector thread with delegation of
• work to worker thread-pool. Worker threads can
• block.

Except for key (event) change semantics this
model is quite easy to understand. Originator
Berkeley Unix See Rox NIO Tutorial
http//rox-xmlrpc.sourceforge.net/niotut/
55
Asynchronous I/O

• Single thread doing all work (no CPU
• intensive stuff, no blocking anywhere, scale
  through
• more instances
• Potentially blocking calls need to be handled
• by hidden threads
• Callbacks called asynchronously. Ugly source
• Sequential illusion with futures/promises which
• represent unfinished call.
• AIO kernel interfaces used

This model requires full support of async.
Interfaces by the kernel/vm. Examples is node.js.
See An NIO.2 primer, Part 1 The asynchronous
channelAPIs, Catherine Hope, Oliver Deakin,
http//www.ibm.com/developerworks/library/j-nio2-1
/
56
Things to consider with I/O Models

• Service Requests/sec
• Memory and context switching costs of threads
• Number of cores
• Do services perform CPU intensive tasks?
• Do services need to block?

57
Exercises
Using code pieces from the Java examples book we
will

• Build a generic client
• Build an echo server
• Build a http client and server
• Build a proxy/firewall

We will discuss general design issues as well!
(patterns etc)
58
Resources

• David Flanagan, Java Examples in a Nutshell,
  OReilly, chapter 5. Code www.davidflanagan.com/j
  avaexamples3
• Ted Neward, Server Based Java Programming chapter
  10, Codewww.manning.com/neward3
• Doug Lea, Concurrent Programming in Java
• Pitt, Fundamental Java Networking (Springer).
  Good theory and sources (secure sockets, server
  queuing theory etc.)
• Queuing Theory Portal http//www2.uwindsor.ca/7E
  hlynka/queue.html
• Performance Analysis of networks
  http//www2.sis.pitt.edu/jkabara/syllabus2120.htm
  (with simulation tools etc.)
• Meet the experts Stacy Joines and Gary Hunt on
  WebSphere performance (performance tools, queue
  theory etc.) http//www-128.ibm.com/developerworks
  /websphere/library/techarticles/0507_joines/0507_j
  oines.html
• Doug Lea, Java NIO http//gee.cs.oswego.edu/dl/cpj
  slides/nio.pdf Learn how to handle thousands of
  requests per second in Java with a smaller number
  of threads. Event driven programming, Design
  patterns like reactor, proactor etc.
• Abhijit Belapurkar, CSP for Java programmers part
  1-3. Explains the concept of communicating
  sequential processes used in JCSP library. Learn
  how to avoid shared state multithreading and its
  associated dangers.
• Core tips to Java NIO http//www.javaperformancet
  uning.com/tips/nio.shtml
• Schmidt et.al. POSA2 book on design patterns for
  concurrent systems.
• Nuno Santos, High Peformance servers with Java
  NIO http//www.onjava.com/pub/a/onjava/2004/09/01
  /nio.html?page3 . Explains design alternatives
  for NIO. Gives numbers of requests per second
  possible.
• James Aspnes, Notes on Theory of Distributed
  Systems, Spring 2014, www.cs.yale.edu/homes/aspnes
  /classes/465/notes.pdf?