Handling Priority Inversion in ORB Middleware - PowerPoint PPT Presentation

1 / 41
About This Presentation
Title:

Handling Priority Inversion in ORB Middleware

Description:

... Priority Inversion in ORB Middleware. Bryan Thrall & Venkita Subramonian. Presented towards partial fulfillment of. CS520 Advanced Real-Time Systems, Fall 2003 ... – PowerPoint PPT presentation

Number of Views:87
Avg rating:3.0/5.0
Slides: 42
Provided by: venkat6
Category:

less

Transcript and Presenter's Notes

Title: Handling Priority Inversion in ORB Middleware


1
Handling Priority Inversion in ORB Middleware
  • Bryan Thrall Venkita Subramonian
  • Presented towards partial fulfillment of
  • CS520 Advanced Real-Time Systems, Fall 2003
  • 30 Sept 2003

2
Sources of Priority Inversion
  • Competition for resources
  • Processor Resources
  • Causes synchronization overhead
  • Communication Resources
  • Context switching
  • Connection demultiplexing
  • Connection cache
  • Memory resources
  • Memory accesses
  • Cache misses

3
Priority Inversion in the CORBA ORB
  • Connection Architecture (1)
  • Connection multiplexing
  • Dynamic connection management
  • Memory Freestore (2)
  • Serialized access
  • Concurrency Architecture (3)
  • Thread Pools
  • POA
  • Layered demultiplexing (4)
  • Active Object Table synchronization (5)

4
Real-time CORBA 1.0
  • Key Features to handle priority preservation
  • Portable Priorities
  • End-to-end priority propagation
  • Protocol properties
  • Key Features to handle priority inversion
  • Thread pools and buffering
  • Explicit binding
  • Standard Synchronizers

5
Portable Priorities
Pluggable Mapping to convert RTCORBA priority to
native priority
  • How can we map global priorities onto
    heterogeneous native OS host thread priorities
    consistently end-to-end?
  • Use Standard RT CORBA priority mapping interfaces

OS-independent design supports heterogeneous
real-time platforms
CORBA priorities are globally unique values
that range from 0 to 32767
6
End-to-end Priority Preservation
  • How can we ensure requests dont run at the wrong
    priority on the server?
  • Use RT CORBA priority model policies

Server handles requests at the priority declared
when object was created
Request is executed at the priority requested by
client. Priority is encoded as part of client
request
7
Client Propagated Priorities
  • Localized priority determination on each end
    system.

8
Priority Inversion - Request Processing
  • Server concurrency limited during request
    processing
  • High priority clients may not get sufficient
    resources held by low priority requests

Client
2
LO
1
Server
obj1.foo()
Client
3
ME
obj2.boo()
Client
5
4
HI
No thread to serve high priority client
Priority Inversion
9
Solution - Thread Pools
  • How can we pre-allocate threading resources on
    the server portably efficiently?
  • Use RT CORBA thread pools
  • Pre-allocation of threads
  • Partitioning of threads
  • Bounding of thread usage
  • Buffering of additional requests

10
Thread Pools - Implementation Strategies
  • Use the Half-Sync/Half-Async pattern to have I/O
    thread(s) buffer client requests in a queue
    then have worker threads in the pool process the
    requests
  • Use the Leader/Followers pattern to demultiplex
    I/O events into threads in the pool without
    requiring additional I/O threads

11
Thread Pools - Queue-per-Lane Design
  • Single acceptor endpoint
  • One reactor for each priority level
  • Each lane has a queue
  • I/O application-level request processing are in
    different threads

12
Thread Pools - Reactor-per-Lane Design
  • Each lane has its own set of
  • reactor
  • acceptor
  • endpoint
  • I/O application-level request processing are
    done in the same thread

13
Thread-Pool designs - comparison
14
Priority Inversion Resource Usage
1
Server
LO
Connection Cache
2
ME
3
HI
Client
  • Each lane has its own set of resources
  • Memory pool
  • Connection pool

15
Explicit Binding
  • How can we achieve the following?
  • Control jitter due to connection setup
  • Minimize thread-level priority inversions
  • Avoid request-level (head-of-line) priority
    inversions
  • Use RT CORBA explicit binding mechanisms
  • Connection pre-allocation
  • Priority Banded Connection Policy
  • Private Connection Policy

Note the priority inversion above since the
stop(), turn(), and query_state() requests all
share the same connection
16
Explicit Binding - Connection Banding
  • How can we minimize priority inversions, so that
    high-priority operations are not queued behind
    low-priority operations?
  • Program the client to use different connections
    for different priority ranges via RT CORBA
    priority bands

Note how the stop() and turn() requests no longer
share the same connection as query_state()
requests
17
Explicit Binding - Preallocate Connections
  • How can we prevent connection establishment
    between the client station and server from
    resulting in unacceptable jitter?
  • Jitter is detrimental to time-critical
    applications
  • Pre-allocate one or more connections using the
    Object_validate_connection() operation

The _validate_connection() operation must be
invoked before making any other operation calls
18
Explicit Binding Private Connections
  • How can we minimize priority inversions by
    ensuring applications dont share a connection
    between multiple objects running at different
    priorities?
  • e.g., sending a stop() request should use
    exclusive, pre-allocated resources
  • Use the RT CORBA PrivateConnectionPolicy to
    guarantee non-multiplexed connections

Note how the stop() and turn() requests no longer
share the same connection from client to server
19
POA Active Demultiplexing
20
POA - Active Demultiplexing (continued)
Work at Nuclear Plant
Run Nuclear Plant
Play PoohSticks
Play Saxophone
Make Money
Impress Girls
Sell Propane
Drink Beer
Eat Honey
Sell Beer
Bounce
Doh!
Skeleton Layer
Tigger
Pooh
Homer
Lisa
Mr. Burns
Moe
Hank
Bobby
Servant Layer
Fox / Simpsons
Disney / Winnie the Pooh
Fox / Simpsons / Family
Fox / Simpsons / Townspeople
Fox
Disney
Fox / King of the Hill
Root POA
POA Layer
ORB I/O
ORB Core Layer
21
Priority Inversion - POA dispatching
  • An object cannot be destroyed while it is still
    registered in the dispatching table
  • Monitor lock used both for dispatching and
    modifying the table
  • Other threads cannot delete an object that is in
    the midst of being dispatched
  • Unbounded priority inversion because upcall time
    could be unbounded

POA
2
1
3
obj1
LO
obj2
4
Block
HI
Active Object Map
22
Solution - Reference Counting
  • Reference Count objects in Active Object Map
  • Bounded blocking factor for high priority thread
  • time to search the object entry time to
    increment the reference count locking overhead

Acquire
Release
POA
2
5
1
7
obj1
4
LO
6
obj2
3
HI
23
Experiments Baseline
Increasing work
Establishing system capacity
Increasing invocation rate
24
Experiments - Classic CORBA
Client
Server
HI 75Hz
ME 50Hz
LO 25Hz
System capacity at 150 (755025) invocations/sec
25
Experiments RTCORBA with TP lanes 1/2
Higher rate thread assigned higher priority
Server
Client
HI 75Hz
HI
ME 50Hz
ME
LO 25Hz
LO
26
Experiments RTCORBA with TP lanes 2/2
Higher rate thread assigned lower priority
Server
Client
LO 75Hz
LO
ME 50Hz
ME
HI 25Hz
HI
27
Best Effort with classic CORBA
Client
Server
HI 75Hz
ME 50Hz
LO 25Hz
Best effort
Continuous workers
28
Experiments Best Effort with RTCORBA 1/3
Server
Client
HI 75Hz
HI
ME 50Hz
ME
LO 25Hz
LO
Best effort
Best effort
29
Experiments Best Effort with RTCORBA 2/3
Server running at capacity (work30)
30
Experiments Best Effort with RTCORBA 3/3
Server running under capacity (work28)
31
Conclusion
32
Critiques Research ideas
  • It would be nice to compare the performance of
    buffered request handling vs non-buffered request
    handling at the server
  • In real-time embedded systems, footprint could be
    a big issue. With isolation of resources, the
    runtime footprint could be high
  • Isolation of resources lead to more areas for
    replication for supporting fault-tolerance, which
    means more overhead and hence would be a concern
    for real-time
  • Convergence of aspects like real-time,
    fault-tolerance and security and how each one
    affects the other is an interesting area of
    research
  • Experimental baseline does not specify whether
    collocated or remote

33
Request Buffering
  • How can we support real-time applications that
    need more buffering than is provided by the OS
    I/O subsystem
  • Buffer client requests in ORB

34
Standard Synchronizers
  • An ORB application may need to use the same
    type of mutex to avoid priority inversions
  • e.g., using priority ceiling or priority
    inheritance protocols
  • Use the RTCORBAMutex synchronizer

35
Protocol properties
  • Selecting communication protocol(s) is crucial to
    obtaining QoS
  • TCP/IP is inadequate to provide end-to-end
    real-time response
  • Use RT-CORBA Protocol Policies to select and/or
    configure communication protocols

36
Network Priority Enforcement
  • How will the RTCORBA priority be enforced by the
    network?
  • Using RTCORBA protocol properties to enable
    priority enforcement mechanisms in the network
  • e.g., DiffServ
  • Pluggable Network Priority Mapping to convert
    RTCORBA priority to DiffServ TOS

IP Datagram Header
37
Experiments RTCORBA with TP lanes - remote
Higher rate thread assigned higher priority -
Remote
Collocated
Remote
38
Experiments - RTCORBA
Thread Pools with no lanes LO priority
Client
HI 75Hz
LO
ME 50Hz
LO 25Hz
Best effort
39
Thread Pools with no lanes ME priority
Client
HI 75Hz
ME
ME 50Hz
LO 25Hz
Best effort
40
Thread Pools with no lanes HI priority
Client
HI 75Hz
ME
ME 50Hz
LO 25Hz
Best effort
41
Locking Mechanisms A comparison
Write a Comment
User Comments (0)
About PowerShow.com