RealTime Fault Tolerant CORBA RFP

1
Real-Time Fault Tolerant CORBARFP

• Tom Bracewell
• Raytheon
• 978-440-2539
• bracewell_at_raytheon.com

2
Core issues this middleware must address

• Handling conflicts between real-time and fault
  tolerance
• Non-determinism, both real-time and fault
  tolerant
• Configuring systems for real-time fault tolerance
• Policy-driven fault recovery

3
Key distinguishing features

• Application-transparent fault tolerance in real
  time environments
• Resource-aware adaptation to crash, communication
  and timing faults
• Fast, scalable fault detection and fault recovery
• Bounded, predictable recovery times
• Scalability
• End-to-end predictability in fault-free operation

4
Mandatory Requirements

• Application-transparent fault tolerance
• Basic fault model
• object faults
• process and processor crash faults
• lost messages, corrupted messages, spurious
  messages
• timing faults, missed deadlines
• Component-level fault tolerance
• provide fault tolerance to entities and groups of
  entities consisting of software components

5
Mandatory Requirements

• Bounded, predictable recovery times
• times can be platform-specific
• plan for worst-case fault recovery times
• End-to-end predictability in fault free
  conditions
• bounded, predictable end-to-end latencies
• avoid impacting real-time operation
• Scalability
• multi-host locally networked systems
• low-footprint, performance-conservative, embedded
  systems

6
Mandatory Requirements

• Policy-driven dependability / configurable RT FT
• systems must make tradeoffs between their
  real-time and fault tolerance requirements in
  order to meet mission objectives
• if real-time and fault tolerant requirements
  conflict, RT FT CORBA shall respond according to
  predetermined policies
• tradeoffs depend on system's current resources
  and mission
• resources may be allocated to better guarantee
  fast transactions or faster fault recovery times
  should a fault occur
• reliability may be allowed to vary with replica
  consistency, or a replica might even be
  temporarily eliminated, based on policy

7
Mandatory Requirements

• Assignment of fault tolerance properties
• developer assigns FT properties to software
  components.
• properties include - but are not limited to -
  replica count, the physical distribution of
  replicas, replication style (active, passive,
  etc.), checkpointing frequency and
  fault-detection frequency.
• Fast fault detection/isolation
• fast, more predictable fault detection and
  isolation times than can be achieved with RT
  CORBA implementations - which have unpredictable
  fault detection times ranging in seconds.
• Ordering of tasks, events, operations
• support the ordering of tasks, events and
  operations to reduce non-determinism - both
  missed deadlines and weak replica consistency -
  via resource management and scheduling services.

8
Mandatory Requirements

• Support multithreading
• support multithreading while working to limit it
  as a source of non-determinism.
• Support nested operations
• address the need to better support nested
  operations in distributed real-time fault
  tolerant systems.
• Nested operations are sources of cascading faults
  and timeouts in many systems.
• It is hard to set appropriate deadlines to
  individual operations that are chained. When a
  fault occurs, it may cascade.
• For example, RT FT CORBA might not always need to
  restart an entire nested operation in order to
  mask a fault that occurred within the chain.

9
Mandatory Requirements

• Proactive dependability
• provide services for proactive dependability.
• predict when faults might occur, e.g. resource
  exhaustion, and take action to prevent them, e.g.
  move a replica.
• Most processor crashes are predictable, for
  example.
• use advance information to prevent many faults
  from occurring.
• mitigate to some degree the unpredictability of
  faults.
• Clock synchronization
• reduce clocks as a source of non-determinism.
• provide clock synchronization and global clock
  services for middleware and applications to use

10
Mandatory Requirements

• Support platform heterogeneity
• Employ incremental checkpointing
• Support ORB interoperability
• Specify how vendor ORB interoperability is to be
  achieved
• Support multiple replication styles
• including but not limited to active, passive and
  semi-active
• can leverage different replication styles to
  achieve desired levels of dependability and
  performance
• Self-healing
• avoid becoming a single point of failure in a
  system
• replicate critical ORB components

11
Optional Requirements

• Support software rejuvenation
• Support for Real-Time Java
• provide transparent support for Real-Time Java
  applications.
• handle RT-Java-specific real-time / fault
  tolerance conflicts such as distributed garbage
  collection and lease timeouts.
• Extendable fault model
• support fault types not in the basic fault model.
• Hooks for security and survivability
• provide ways to work cooperatively with system
  security and survivability services.

12
Optional Requirements

• Live software upgrades
• include methods to sustain continuous operation
  of during live software upgrades and downgrades.
• This includes replicable applications and
  middleware.
• Partition tolerance
• provide means to support continuous degraded
  system operation during a network partition
• provide facilities to support re-merging and
  recovery when partition is healed.

13
Optional Requirements

• Reliability Advisor
• Offline tools may be proposed to support RT FT
  CORBA and help the developer model and automate
  the configuration of dependable real-time fault
  tolerant systems.
• Tools should help developers configure systems to
  meet their dependability and performance
  requirements using available options. These
  options include, for example, replica count,
  checkpoint frequency, replication styles, fault
  detection frequency, and the physical and logical
  distribution of replicas.
• Tools should also help the developer determine in
  a resource-aware manner how to assign fault
  tolerance properties to software components based
  on an object's state size and recovery time
  requirements.

14
RFP Timetable

• April 2004
• June 29, 2004
• August 20, 2004
• October 25, 2004
• November 1, 2004
• February 9, 2005
• March 2, 2005
• May 2005
• June 2005

• TC votes to issue RFP
• LOI to submit to RFP due
• Initial Submissions due
• Voter registration closes
• Initial Submission presentations
• Preliminary evaluation by TF
• Revised Submissions on OMG server
• Revised Submission presentations
• Final evaluation and selection by TF
• Recommendation to AB and TC
• Approval by Architecture Board
• Review by TC
• TC votes to recommend specification
• BoD votes to adopt specification