Reusable Models for Timing and Liveness Analysis of Middleware for Distributed Real-Time and Embedded Systems

1
Reusable Models for Timing and Liveness Analysis
of Middleware for Distributed Real-Time and
Embedded Systems

• Venkita Subramonian
• and Christopher Gill
• venkita,cdgill_at_cse.wustl.edu
• Dept. of Computer Science and Engineering
• Washington University, St. Louis, MO

César Sánchez and Henny Sipma cesar,
henny_at_cs.stanford.edu Department of Computer
Science Stanford University, Stanford, CA
EMSOFT 2006 October 23-25, 2006, Seoul, Korea
Research supported in part by NSF CAREER award
CCF-0448562 Research supported in part by NSF
grants CCR-01-21403, CCR-02-20134, CCR-02-09237,
CNS-0411363, and CCF-0430102, ARO grant
DAAD19-01-1-0723, and NAVY/ONR contract
N00014-03-1-0939 Thanks also to Joseph Sifakis,
Marius Bozga and Iulian Ober for valuable advice
about the IF tool set, and to Doug Schmidt and
the hundreds of ACETAOCIAO developers/users
world-wide
2
A Motivating Example Real-Time Image Transmission

• Soft real-time distributed system (over
  wireless)
• Many end-to-end activities must be
  managed at once
• image (de-)compression, transmission,
  analysis, display
• Needs adaptive enforcement
• Task scheduling, rate adaptation, feedback
  control
• Many applications are like this
• Is correct reuse of middleware possible?

Console
Gill et al., Integrated Adaptive QoS Management
in Middleware An Empirical Case Study (RTAS
04) Wang et al., CAMRIT Control-based
Adaptive Middleware for Real-time Image
Transmission (RTAS 04)
Camera
3
Motivation for Formal Models of Middleware
a main obstacle to the application of rigorous
development techniques is the lack of methodology
for relating application software and functional
design to physical architecture and
implementation
Joseph Sifakis, Modeling real-time
systems-challenges and work directions, EMSOFT
2001

• The formal verification community needs accurate
  middleware models

ORB configurations have a substantial impact on
performance, e.g., TAO's fastest concurrency
model is thread-per-connection, rather than
thread pool
what's ultimately most important is to determine
how things perform for applications that are
representative of actual user configurations,
rather than micro-benchmarks alone
Doug Schmidt, ICE/CORBA newsgroup discussion
3/29/05

• The real-time embedded systems community needs
  formal ways to evaluate the semantics of
  different middleware configurations

4
A Brief Survey of Other Related Work

• Vanderbilt University and UC Irvine
• GME, CoSMIC, PICML, DREAM
• UC Berkeley
• Ptolemy, E-machine, Giotto
• Kansas State University and University of
  Nebraska
• Bogor, Cadena
• University of Pennsylvania
• VERSA
• University of Brest
• Cheddar

5
ACETAOCIAO COTS Middleware Architecture
CODESISSS (Tuesday, session A10) system
software engineering focus
A Distributed System Software Stack
EMSOFT (Monday, session B1) protocol design and
proof focus
EMSOFT (Wednesday, session B9) modeling and
verification focus
6
Goal Correct (Re-)use of DRE Middleware
Thesis formal models of fine-grain middleware
building blocks provide a principled approach to
analyze timing and liveness properties of
middleware-based DRE systems
System Implementation
System Modeling
CADENA DREAM RMA Ptolemy GME CoSMIC UPPAAL IF-tool
kit SPIN Bogor
Formal approach to composition
Reusable software building blocks
7
Another Gap Between High Level Models and
Lower-Level Consequences of their Configuration
Choices

• Modeling at a high-level of abstraction is
  helpful
• Focus on nuances of the application
• Low-level design choices matter too
• May invalidate (implicit) assumptions
• We focus on low-level frameworks
• Capture fundamental concurrency and timing
  details of the middleware

Application Models
Implementation Platform
CORBA
CCM
Collocation strategies Buffer flushing
strategies Communication strategies Connection
strategies Concurrency strategies
EJB
Pattern-based frameworks
OS
8
Model Architecture in IF for ACE

• Network/OS layer interprocess communication
  abstractions
• Middleware layer ACE pattern-oriented
  abstractions
• Application layer application-specific semantics
  within ACE event handlers

9
Modeling Threads

• Challenge
• No native constructs for threads in model
  checkers that currently support timed automata
• Option 1 model all thread actions as a single
  automaton
• Suitable for high level modeling of application
  semantics
• Option 2 model a thread as multiple interacting
  automata
• Interactions model the flow of control
• This option better abstracts the nuances of
  ACE-level mechanisms

Foo
Bar
input method_request
output method_request
input method_result
output method_result
10
Modeling Thread Scheduling Semantics (1/4)

• Easy to achieve with one automaton per thread
• Specify to model checker directly
• E.g., using IF priority rules
• More difficult with more than one automaton per
  thread
• Thread of control spans interactions among
  automata
• Need to express thread scheduling in terms of
  execution control primitives provided by the
  model checker

Activity1
Activity2
Update Display
Control Flow Rate
1 automaton per thread
prio_rule pid1 lt pid2 if pid1 instanceof
Activity1 and pid2 instanceof Activity2
Foo
Bar
input m_req
output m_req
input m_result
output m_result
11
Modeling Thread Scheduling Semantics (2/4)

• Solution
• Introduce a thread id that is propagated along
  automata interactions
• Thread id acts as an index to a storage area
  which holds each threads scheduling parameters

Resulting Behavior
Bar1
Foo1
Foo1
Bar1
1
2
Bar2
Foo2
Bar1
Foo1
Prio5
Prio8
Foo2
Bar2
2
1
thread_schedule pid1 lt pid2 if pid1 instanceof
Foo1 and pid2 instanceof Bar1 and (Foo1pid1).th
readid ltgt (Bar1pid2).threadid
and (Thread((Foo1pid1).threadid)).prio
lt (Thread((Bar1pid2).threadid)).prio )
Hint to the model checker
Give higher preference to the automaton whose
thread (pointed to by thread id) has higher
priority
12
Modeling Thread Scheduling Semantics (3/4)

• What if two threads have the same priority?
• In an actual implementation, run-to-completion
  (SCHED_FIFO) may control the possible
  interleavings
• How can we model run-to-completion?

Bar1
Foo1
Foo2
Foo3
Foo1
Bar2
Foo2
Bar1
Foo1
Bar3
Foo3
Bar1
Bar2
Bar3
Bar1
Foo1
Bar1
How do we prune out this space?
Bar2
Foo2
Foo3
Bar3
13
Modeling Thread Scheduling Semantics (4/4)

• Solution
• Record id of currently executing thread
• Update when executing actions in each automaton

Currentnil
Bar1
Foo1
Current1
Current2
Foo2
Bar1
Current1
Bar1
Foo3
Current1
Bar1
Hint to the model checker
Give higher preference to the automaton whose
thread is the currently running thread.
Non-deterministic choice if Current is nil
Current2
Bar2
Current2
Bar3
Current2
14
Problem Over-constraining Concurrency
Hint to the model checker
Currentnil
Give higher preference to the automaton whose
thread is the currently running thread.
Non-deterministic choice if Current is nil
Bar1
Foo1
Current1
Current2
Foo2
Bar1
Current1
Bar1
Foo3
Current1
Bar1
Current2
Bar2
Current2
Bar3 always chosen to run
Bar3
Bar3
Current2
Current2
Time progresses
Foo3
15
Solution Idle Catcher Automaton
Foo3, Bar3 blocked at this point
Current2
Idle catcher runs

• Key idea lowest priority catcher runs when all
  others are blocked
• E.g., catcher thread in middleware group
  scheduling (RTAS 05)
• Here, idle catcher automaton
• runs when all other automata are idle (not
  enabled), but before time progresses
• Resets value of current id to nil

Currentnil
Time progress
Currentnil
Foo3
Bar3
Foo3 or Bar3 could be chosen to run.
Over-constraining eliminated
16
Problem Tractability
right away
in a minute
get coffee
go for an espresso
maybe tomorrow?

• Model checking can suffer from state space
  explosion
• State space reduction, live variable analysis can
  help
• But even good model checkers dont fully solve
  this
• Need to think of modeling as a design issue, too
• Does the model represent what it needs to
  represent?
• Can the model be re-factored to help the checker?
• Can domain specific information help avoid
  unnecessary checking?

17
Optimization 1 Leader Election

• Leader/Followers concurrency
• Threads in a reactor thread pool take turns
  waiting on the reactor
• One thread gets the token to access the reactor
  - leader
• All other threads wait for the token followers
• It does not matter which thread gets selected as
  leader in a threadpool
• Model checker not aware of this domain specific
  semantics
• For BASIC-P protocol example, saved factor of 50
  in state space, and factor of 20 in time

Token to access the reactor is available
T1
T3
T2
T2
T3
T3
Prune this out
18
Optimization 2 System Initialization

• Similar idea, but different technique
• Iff ok to establish initial object relations in
  any order, can optimize away
• E.g., 2 server automata, each of which creates a
  reactor automaton
• Useful when modeling object systems in model
  checkers with dynamic automaton creation
  capability (e.g., IF)
• State space reduction depends on application
• Factor of 250 for a deadlock scenario with 2
  reactors and 3 threads in each reactor

S1 creates R
S2 creates R
1
1
S1
R
S2
R
S2 creates R
S1 creates R
1
1
S1
R
S2
R
2
2
S2
R
S1
R
Prune this out
19
Verification of a Real-Time Gateway
Consumer1
Supplier1
Consumer2
Gateway
Consumer3
Supplier2
Consumer4

• An exemplar of many realistic ACE-based
  applications
• We modified the Gateway example to add new
  capabilities
• E.g., Real time, Reliability, Control-Push-Data-Pu
  ll
• Value added service in Gateway before forwarding
  a to consumer
• E.g. Using consumer specific information to
  customize data stream
• Different design, configuration choices become
  important
• E.g., number of threads, dispatch lanes, reply
  wait strategies

20
Model Checking/Experiment Configuration
C1
100ms
Gateway
20
C2
100ms
Period
20
100ms
S1
C3
10
50ms
10
S2
50ms
C4
50ms
Relative Deadline
Value-added execution (and its cost)

• Gateway is theoretically schedulable under RMA
• Utilization 80
• Schedulable utilization 100 for harmonic
  periods
• Assumption Messages from 50ms supplier is given
  higher preference than 100ms supplier
• ACE models let us verify scheduling enforcement
• IN THE ACTUAL SYSTEM IMPLEMENTATION

Deadline Exec time
C1 100ms 20ms
C2 100ms 20ms
C3 50ms 10ms
C4 50ms 10ms
21
Real-time Gateway Single Thread
Gateway
ConsumerHandler
SupplierHandler
ConsumerHandler
SupplierHandler
Consumer
ConsumerHandler
Supplier
ConsumerHandler
Reactor

• Single reactor thread dispatches incoming events
• I/O (reactor) thread same as dispatch thread
• I/O thread responsible for value added service

22
Real-time Gateway Dispatch Lanes
Gateway
ConsumerHandler
SupplierHandler
ConsumerHandler
SupplierHandler
Consumer
ConsumerHandler
Supplier
ConsumerHandler
Reactor

• Single reactor thread again dispatches events to
  gateway handlers
• I/O (reactor) thread puts message into dispatch
  lanes
• Lane threads perform value added service,
  dispatch to consumers
• DO QUEUES HELP OR HURT TIMING PREDICTABILITY?

23
Model/Actual Traces for Real-time Gateway
Execution in the context of lane threads
Execution in the context of reactor thread
Single threaded Gateway
Gateway with dispatch lanes
S1,S2
S2
S1,S2
S2
Deadline miss for Consumer4 because of blocking
delay at reactor
C1
C2
C3
C4
C3
C4
C1
C2
C3
C4
C2
Model
C1
C2
C3
C4
Actual
C3
C4
C1
C2
C3
C4
C2
Time
Time
20
40
60
10
30
50
20
40
60
10
30
50
70
80
90
100
Expected execution timeline with RMS
Period Exec time Deadline
C1 100ms 20ms 100ms
C2 100ms 20ms 100ms
C3 50ms 10ms 50ms
C4 50ms 10ms 50ms
C1, C2, C3, C4
C3, C4
C3
C4
C1
C2
C3
C4
C2
20
40
60
10
30
50
70
80
90
100
24
Concluding Remarks

• Timed automata models of middleware building
  blocks
• Can be used to verify middleware concurrency and
  timing semantics
• Domain specific model checking optimizations
• Can achieve significant reductions in state space
• We have run multiple case studies using these
  models
• Allow comparison of model checking software
  execution traces
• Demonstrate reusability and fidelity of our
  models
• Future work
• Continue to develop reusable models and
  optimizations
• Incorporate domain information directly into
  model checking tools
• Extend our approach beyond real-time concerns
• Model mechanism faults and failure modes
• Power usage (yet another real valued clock)

25
Questions?
26
Backup Slides
27
Timing Traces for BASIC-P Protocol
EH21
EH11
R1
R2
EH31
Flow1
EH22
EH12
R1
R2
EH32
Flow2
Model checking actual timing traces show
BASIC-P protocols regulation of threads use of
resources (no deadlock)
EH23
EH13
R1
R2
EH33
Flow3
28
BASIC-P Blocking Delay Comparison
Actual Execution
Model Execution
Blocking delay for Client2
Blocking delay for Client3
29
Overhead of ACE TP/DA reactor with BASIC-P
Negligible overhead with no DA protocol
Overhead increases linearly with of event
handlers due suspend/resume actions on handlers
at BASIC-P entry/exit