SWARM Simulation of MultiAgent Fault Mitigation in LargeScale RealTime Embedded Systems

1
SWARM Simulation of Multi-Agent Fault Mitigation
in Large-Scale Real-Time Embedded Systems

• Derek Messie
• Jae C. Oh
• Department of Electrical Engineering and Computer
  Science
• Syracuse University

2
BTeV

• A particle accelerator-based High Energy Physics
  (HEP) experiment at Fermi National Laboratory
  studying matter-antimatter asymmetries in the
  decays of particles containing the bottom quark.

3
Real-Time Embedded Systems Collaboration (RTES)

• Design real-time embedded intelligent software to
  ensure data integrity and fault-tolerance within
  the BTeV data acquisition system.
• Fermi National Accelerator Laboratory (BTeV)
• Syracuse University (Very Lightweight Agents /
  Load Balancing)
• Vanderbilt University (Graphical Modeling
  Environment)
• University of Illinois (Adaptive, Reconfigurable,
  and Mobile Objects for Reliability)
• Sponsored by NSF

4
BTeV Triggering and Data Acquisition System
5
Very Lightweight Agents (VLAs)

• Monitor hardware / software integrity
• Intelligent / Adaptive
• Reactive / Proactive
• Small footprint
• Fast
• Hierarchical

6
VLA Hierarchy
Regional VLA
Farmlet
Buffer Manager
Farmlet Manager
Farmlet VLA
Worker
Worker VLA
PA
Local Manager
7
SC2003 Prototype

• 7-slot VME crate with 4 fully populated
  motherboards.
• 16 DSPs
• TI C6711 with 64 M of RAM at 166 MHz
• Graphical Modeling Environment (GME) to model the
  RTES/BTeV data acquisition system.

8
BTeV Error Scenarios Modeled
9
Challenges

• Scalability ? (16 lt gt 2500)
• Centralized expert system approach not
  effective given
• of real-time components / states
• Failover redundancy

10
VLA Design Motivation

• Subsumption Architecture
• (Brooks 1986)
• Mobile robot design
• Multiple layers of distributed sensors
• Decentralized, bottom-up approach
• Higher levels subsume lower levels
• Increasing levels of competence

11
Mobile robot layers of competence

• 0. Avoid contact with objects (moving or
  stationary).
• Wander aimlessly around without hitting things.
• Explore the world by seeing places in the
  distance which look reachable and heading for
  them.
• Build a map of the environment and plan routes
  from one place to another.
• Notice changes in the static environment.
• Reason about the world in terms of identifiable
  objects and perform tasks related to certain
  objects.
• Formulate and execute plans which involve
  changing changing the state of the world in some
  desirable way.
• Reason about the behavior of objects in the world
  and modify plans accordingly.

12
VLA layers of competence

• 0. Act on local reactive rules.
• Monitor local components and activate local
  proactive rules.
• Analyze and act on statistics gathered from rule
  firing.
• Map the local state of the environment.
• Reason about the system in terms of accumulated
  statistics and the state of connected components.
• Formulate and execute actions which involve
  changing the state of the system in some
  desirable way.
• Reason about the behavior of components in the
  system and modify plans accordingly.

13
SWARM (www.swarm.org)

• Multi-agent simulation of complex systems.
• GNU General Public License
• Java or Objective-C development kit

14
BTeV Components Modeled
Regional VLA
Farmlet
Buffer Manager
Farmlet Manager
Farmlet VLA
Worker
Worker VLA
PA
Local Manager
15
SWARM Simulation

• Buffer Manager
• queue volume
• Physics Application (PA)
• Injects error scenarios at individual DSPs.
  (random and manual)
• VLA (Worker, Farmlet, Regional)
• Reactive and proactive rule set
• Local Manager
• Additional DSP fault handling

16
SWARM Simulation (cont.)

• Component message logs
• Global behavior tracking
• Error rates
• Recovery rates
• Buffer queue levels

17
(No Transcript)
18
Lessons Learned / Future Work

• Subsumption Architecture vs. Traditional
  Hierarchical Systems
• More focus on adaptive qualities of VLA
• Self-organizing VLAs
• Cooperation within scheduling algorithm

19
Questions ?