Run Time Performance Modeling and Measurement of Adaptive Distributed Object Application DOA 2002

1
Run Time Performance Modeling and Measurement of
Adaptive Distributed Object ApplicationDOA 2002

• Oct 30, 2002
• John Zinky
• Joe Loyall
• Richard Shapiro
• BBN Technologies

2
QoS Adaptation Needs to be Supported at Every
Layer in a Distributed Application
Applications
Applications
Logical Method Calls
Client
Object
Middleware
Middleware
QoS Adaptive Layer
QoS Adaptive Layer
Network Based Services
Distributed Objects COTS ORB Schedulers
Distributed Objects COTS ORB Schedulers
Replication Protocols Group Communications
Replication Protocols Group Communications
Operating System
Operating System
Resource Managers
Resource Managers
Client Host
Network
Servant Host
3
QuO Adds Specification, Measurement, and
Adaptation into the Distributed Object Model
Application Developer
CORBA DOC MODEL
Mechanism Developer
Application Developer
CLIENT
CLIENT
operation()
OBJ REF
out args return value
Delegate
Delegate
QuO Developer
SysCond
SysCond
SysCond
QUO/CORBA DOC MODEL
SysCond
IDL SKELETON
MECHANISM/PROPERTY MANAGER
IDL STUBS
OBJECT ADAPTER
Mechanism Developer
4
QoS Parameters are Measured bothIn-band and
Out-of-Band
5
Key Points

• QoS Adaptation can be separated from Functional
  behavior
• The running example will be extended from simple
  QoS adaptation to comprehensive adaptation
• The base application is unchanged, just different
  QoS adaptations are inserted between the client
  and server
• Predicted QoS behavior complements Measured QoS
• They can be compared to diagnose performance
  problems
• They can be combined for more robust QoS metrics
• A Model of Application is implicate in all QoS
  Adaptation
• Model resources, application and how they are
  bound together
• QuO Resource Status Service (RSS) can make a Run
  Time performance model of the application and the
  resources it uses

6
An Example User Can Adapt by Changing the
Amount of Data and the Level of Processing
Processed
UnProcessed
Check Box For User Specification
Big
Small
7
Quo Delegate Can Change the Method Called Based
on the Value of the Contract and Sysconds
Application

• The User can change the desired QoS, by setting
  the value of a SysCond
• The Contract integrates the Users desires with
  other information to form a Region
• The Delegate uses the Region to dispatch to a
  behavior. In this case call a different method on
  the object

User Sets Expectation
Client Code
Call Back
Reference
SysCond
QuO Kernel
Delegate
Contract
Proxy
ORB
ORB
Network
Control
Manager
QuO Kernel
ORB
SC
SC
Proxy
Del.
Contract
Object
8
QuO Contract Regions Summarizes the Users
Desires
Contract Description Language
contract UserAdapt ( // Expectation from
users application syscond ValueSC ValueSCImpl
userEnableSmall, syscond ValueSC ValueSCImpl
userEnableUnprocessed ) region Big (not
userEnableSmall) region Processed (not
userEnableUnprocessed) // Use Big Processed
Pictures region Unprocessed
(userEnableUnprocessed) // Use Big
UnProcessed Pictures region Small
(userEnableSmall) region Processed (not
userEnableUnprocessed) // Use Small
Processed Pictures region
Unprocessed (userEnableUnprocessed) // Use
Small UnProcessed Pictures
Generated Code
Abundant Resources
Panel From QuO GUI
9
QuO Components Are Packaged Into Reusable Bundles
of Systemic Behavior Called Qoskets

• The Qosket encapsulates a set of contracts (CDL),
  system condition objects (IDL), and QoS adaptive
  behavior (ASL)
• The Qosket exposes interfaces to access QuO
  controls and information (specified in IDL)
• The Qosket separates the functional adaptive
  behavior from the QoS adaptive behavior and the
  middleware controls from the QoS mechanisms

Client Code
Qosket
Delegate

QuO Contract
Functional Behavior Adaptation
QoS Apdative Behavior
SysCond
QoS Mechanisms
Object
10
Application Specific Behavior Is Bound to the
Qosket to Achieve the Desired QoS Adaptively
Aspect Specification Language
behavior UserAdapt () remote_object
slideSlideShow server void
slideSlideShowread(in long gifNumber,
out string size,
out octetArray buf)
inplaceof METHODCALL region Big
region Processed
server.readBigProcessed(gifNumber, size, buf)
region Unprocessed
server.readBig(gifNumber, size, buf)
region Small
region Processed
server.readSmallProcessed(gifNumber, size, buf)
region Unprocessed
server.readSmall(gifNumber, size, buf)

Delegate Behaviors
BigProcessed()
BigUnprocessed()
SmallProcessed()
SmallUnprocessed()
11
Suppose When the Call Latency Is Too High,Get
Lower Quality Picture Which Use Less Resources
Delegate Behaviors (ASL)
DownShift Contract (CDL)
Fast
BigProcessed()
Latency gt HighThreshold
Latency lt LowThreshold
BigUnprocessed() or SmallProcessed()
Med
Latency gt HighThreshold
Latency lt LowThreshold
SmallUnprocessed()
Slow

• Contracts Support State machine representation of
  regions
• QoS Designer has to decide which behavior to use
  for each region

12
Probe Sysconds Can Measure the Latency of a Call
by Observing the Internal Contract Evaluations
Call
Return
Pre-eval
Delegate
Post-eval
SysCond

• Observation of Contract evaluation is supported
  as part of the CDL language
• Latency Sysconds are part of the QuO Library of
  Sysconds.

13
The Call Latency Changes Depending on the Method
Call (User Selected Region)
BigProcessed()
Instrumentation Settling Time
SmallUnprocessed()
SysCond Panel from QuO GUI
Note The Latency will also change if the
underlying resources change
14
Suppose The Contribution to the Latency can be
Determined for the Network and Server
Delegate Behaviors (ASL)
Component Latency Contract (CDL)
Fast Network
BigProcessed()
Fast Server
Slow Server
BigUnprocessed()
SmallProcessed()
Slow Network
Fast Server
Slow Server
SmallUnprocessed()

• Contract Region predicate compare the time spent
  in the Network (or Server) to a threshold
• Contract can hold in a region until the statistic
  have converged

15
Detailed Timing Measurement can be made by adding
code to update a trace record
Call Trace Record
TS1
TS2
TS3
TS4
Excpt
size
Client Delegate

• ASL is used to weave in calls to update the trace
  record
• The trace record is sent to the server-side via
  an additional parameter in the interface

Server Delgate
16
Suppose When there is Excess Resources Choose
High Quality Picture
Delegate Behaviors (ASL)
Resource Capacity Contract (CDL)
Excess Network
BigProcessed()
Excess Server
Constrained Server
BigUnprocessed()
SmallProcessed()
Constrained Network
Excess Server
Constrained Server
SmallUnprocessed()

• Contract Region predicate compare the Fitted
  Resources capacity for the Network (or Server) to
  a threshold

17
System Conditions Hold the Processed Summary
Statistics which are used by the Contract
Kernel
SysCond
Signal
Delegate
probes
Summary Fits
Trace Records
Statistics

• SysConds have to deal with issues concerning
  startup, loss of information, invalid
  information.
• SysConds collect only the system properties need
  by contracts
• SysConds have a CORBA interface so that they can
  be queried by centralize management systems
• SysConds can detect thresholds and alert its
  clients to changes

18
Cross Traffic can Change theEffective Capacity
of the Network

• Fits are extremely sensitive to changes in the
  underlying process
• The middle graph show points from both before and
  after the cross traffic started
• Contracts support hysteresis, so that the
  threshold to enter a region is different then the
  threshold to level

19
Suppose The Resource Capacity Comes from
Out-of-Band Measurements
Delegate Behaviors (ASL)
Component Latency Contract (CDL)
Rich Network
BigProcessed()
Rich Server
Constrained Server
BigUnprocessed()
SmallProcessed()
Constrained Network
Rich Server
Constrained Server
SmallUnprocessed()

• Contract Region predicate compare the Expected
  Resources Capacity of the Network (or Server) to
  a threshold
• Many sources of out of band measurement must be
  integrated at each Client.

20
Resource Status Service Integrates External Data
about Remote Resources to Infer Expected QoS
Expected QoS
Isolation The Quosketeer will work with a high
level description of available resources Integrati
on Conflicting measurements will be resolved to
always give the best guess. Translation
different standards for Resources MIBs will be
translated into a QuO Resource Ontology
Collection interfacing details will be handled
by Data Feed
Resource Ontology
Resource Status Service
Model Level Resource Level Integration Level
Data Scopes Formulae
Translate Store
Translate Store
Translate Store
Data Feeds
Collect
Collect
Collect
http
Custom
CORBA
Configuration
Remos
StatusTEC
(Base-Line)
(Network)
(Host)
21
Out-of Band Measurements are Organized into a
QoS MIB
Host_10.23.12.1_CPU_LoadAverage_Expected Host
_10.23.12.1_Process_2232_Socket_5223 Host
_10.23.12.1_Process_2232_Object_Key_tao/foo/2 IPFL
OW_ 10.23.12.1_ 10.23.12.1_Capacity_Max

• Keys form a hierarchy of names
• Root is an entity
• Leaves are QoS Properties
• Data Values have multiple attributes
• Value (dynamic type, such a boolean, int, double,
  string)
• Credibility (belief on how good the measurement
  is)
• Units, Timestamp, collector name, etc)
• Subscribe to Changes on QoS Property values
• Based on Existing Standard MIBs,
• WBEM/CIM, SNMP, QoSMIB,

22
Each Collection Service has its own
Dissemination Infrastructure
When there are multiple values, The most
Creditable is used.
STEC
Host Probes
Channel Mesh
push
Resource Status Service
Integration Level
Network Management System eg. Remos
Network Routers
Collection Host
Translate Store
Translate Store
Translate Store
Data Feeds
poll
Collect
Collect
Collect
Host
Configuration
Network
CORBA
Web pages
Host
Sites
Appl
Test
Custom
http
23
Suppose Pick the Server with the Lowest Latency
to the Client
Delegate Behaviors (ASL)
Sever Selection Contract (CDL)
Select Latencyi
Server1.BigProcessed()
Server1
Server2.BigProcessed()
Server2
Server3.BigProcessed()
Server3

• Contracts support selecting the best region out
  of a pool of regions, based on a selection
  criteria
• Many factors need to be added to model for
  predicting latency between a client and server

24
RSS Data Representation Shadows the Real World
with Objects and Relationships

• Data Scopes represent objects with attributes and
  relationships
• Containment relation allows a child to inherit
  all the attributes of its parent.
• Specialized relations can be followed to inherit
  other attributes from peers.
• Query Functions help find Scopes
• Symbol represents a link to another Scope or an
  attribute value.
• Symbols are Static, value changes are not pushed.
• Symbols are used to find formulas
• Formula is a calculation, which depends on the
  values of other Formulas.
• Value Changes are Pushed to Subscribers, such as
  other formulas or SysConds.
• Raw Data from Feeds are published as formulas

Host
Process
Object
Method
Method
Object
Object Key
Interface
Calls
Latency Load/Capacity
25
Scopes Represent the Relationship between Real
World Objects

• Scopes are the foundation on which to find
  formulas
• Scopes Represent different layers of resources
• Queries can find scopes by following
  relationships.
• Paths follow the containment relationship
• Host(lemon)-gtProcess(1243)-gtObject(773)
• Symbols keep the value of scope attributes
• Pointers to other scopes
• Identifiers
• Random facts

Host
Process
Object
Class
Method
Method
Method
Method
IP Flow
TCP
IIOP
Host
Process
Object
Class
Method
Method
Method
Method
26
Formulas form a Publish and Subscribe Network
which Traces and Transforms Data

• Formulas Subscribe to values relative to the
  scopes for which they are attached
• Formula Setup time uses scopes to find the
  sub-formulas.
• The expensive setup happens only once.
• Value Calculations are done continuously as raw
  data changes
• Both forward chaining and backward chaining modes
  are supported,
• Pull vs Push
• Formulas are extremely Reusable
• Attach them to a scope and they will calculate a
  value

27
Micro-Benchmarks are used to Calibrate CPU
Resources.

• Java performance depends on many factors
• CPU Clock speed, Number of CPUs,OS, Java VM, CPU
  architecture,
• Micro-benchmarks are run when the application
  process starts up.
• Dynamic Factors, such as load average, are also
  used to model the effective CPU capacity at
  runtime

28
QuO Components were Used to Implement Distributed
Denial of Service Architecture
Cougaar Distributed Agent Architecture
GUI
NodeAgentA
Attack Detector Plugin
AttackStatus Listener Plugin
Dos Detectors
Syscond
SSL Aspect
Traffic Mask Aspect
QuO RSS
29
Metric Service Customizes the QuO RSS for the
Cougaar Environment
Sensor Components
Metrics Servlets
Adaptivity Engine
Node-level adaptation
SysCond
(Keys)
(Paths)
QuO/RSS
Data Scopes Data Formulas
Metrics Writer Service
Node/Agent Models
Metrics Reader Service
Integration Formulas
Cougaar Data Feed
STEC Data Feed
Config Data Feed
NMS Config Files
STEC Channel
STEC Channel Mesh
STEC Host Probe
Optional Sharing
30
QuO-Based Actuators Control Behavior Using Both
Blackboard Objects and Metric Services
Blackboard
Attack Status
Actuator Aspect
Attack Status Listener Plugin
SysCond
QuO Delegate In-band Adaptive Behavior
QuO Contract
SysCond
Attack Status Service
Metrics Service
Resource Control Services
Control Aspects
Measurement Aspects
Resource Configuration
31
Conclusions

• QuO Code generators inserts measurement probes
  in-band to the invocation of method calls going
  from the client to the server.
• QuO RSS integrates Out-of-Band measurements to
  predicted the expected resource capacity
• QuO RSS includes support for evaluating
  performance models at run time.
• QuO supports reuse of the QoS measurement code,
  e.g. libraries of Sysconds, Data Scopes, Data
  Formulas, and Data Feeds.