RuleBased Runtime Verification

1
Rule-Based Runtime Verification

• Howard Barringer
• Allen Goldberg
• Klaus Havelund Koushik Sen

2
Overview

• Run-time Monitoring
• About EAGLE
• Enhanced Formal Testing
• Summary

3
NASA Mars Missions

• Missions are becoming frequent
• MER1 (landed) and MER2 (24 Jan)
• By 2009 mobile science lab to land on / near
  polar icecap.
• Long-range and long-duration
• Demonstrate hazard avoidance

• Smart system
• Autonomous for long periods
• Mustnt die!

4
Motivation for Runtime Verification

• Model checking and Theorem Proving are rigorous,
  but
• Not fully automated
• model creation is often manual
• Abstraction is often manual in the case of model
  checking
• Lemma discovery is often manual in the case of
  theorem proving
• Therefore not very scalable
• Applied Testing is scalable and widely used, but
  is ad hoc
• Lack of formal coverage
• Lack of formal conformance checking
• Combine Formal Methods and Testing

5
Run-time Verification

• Combine temporal logic specification and testing
• Specify properties in some temporal logic.
• Instrument program to generate events.
• Monitor properties against a trace of events
  emitted by the running program.
• Pros
• Formal conformance checking
• Automated
• Scalable
• Cons
• Formulation of properties is hard
• To quote NASA software engineer
• I have absolutely no idea what this
  system should satisfy.
• Lack of Coverage

6
A Model-Based Verification Architecture
Model
test property generation
test inputs
behavioural properties
Implemented system under test
instrumentation
Rqmts
event stream
reports
dispatch
Deadlock
Dataraces
Observer
instrumentation
7
Other Work on Monitoring Logics

• Our own previous work propositional future and
  past time LTL
• MaC Tool (UPenn) past time interval logic
  state
• Temporal Rover (commercial tool) (future and past
  time LTL real-time and some data handling).
• Finkbeiner and Sipma statistical future time
  LTL.
• Simmons et al. (CMU) interval logic.
• See ENTCS online-proceedings for the three
  previous
• runtime verification workshops RV01, RV02,
  RV03.
• RV04 Barcelona, Spain (satellite workshop to
  ETAPS 2004)
• http//ase.arc.nasa.gov/rv2004

8
So many logics

• What is the most basic, yet, general
  specification language suitable for monitoring?

EAGLE is our answer. Based on recursive rules
over three temporal connectives next, previous
and concatenation. Can encode future time
temporal logic, past-time logic, ERE, µ-calculus,
real-time, data-binding, statistics.
9
Introducing EAGLE

• Rule-based finite trace monitoring logic
• User defines
• a set of combinators
• a set of monitoring formulas
• Monitoring formulas are evaluated over a given
  input trace, on a state by state basis
• Evaluation proceeds by checking facts about the
  past and generating obligations about the future.

10
EAGLE by example LTL
max Always(Form F) F /\ ? Always(F) . min
Eventually(Form F) F \/ ? Eventually(F) . min
EventuallyP(Form F) F \/ ? EventuallyP(F) . To
monitor the LTL formula ?(xgt0 ? ? y3),
write mon M Always(x gt 0 -gt Eventually(y3)) .
11
EAGLE by example data binding

• ?(x gt 0 ? ?k. k x /\ ? y k)
• or written as
• ?(x gt 0 ? let k x in ? y k)
• is expressed in Eagle as
• min R(int k) Eventually(yk) .
• mon M Always(xgt0 ? R(x)) .

12
EAGLE by example metric LTL

• Timed operators, such as ?t1,t2
• assume events are time-stamped state variable
  clock
• min EventuallyAbs(Form F, float t1, float t2)
• clock lt t2 /\
• (F ? t1 lt clock) /\
• ( F ? ? EventuallyAbs(F, t1, t2)) .
• min EventuallyRel(Form F, float t1, float t2)
• EventuallyAbs(F, t1clock, t2clock) .

13
EAGLE by example timed automata
in
y 0
x 0
N0
N1
out
xlt5
xlt15 ylt12
x 0

• max N0(double x)
• label in /\ clock lt x5 /\ ?
  N1(x,clock) .
• max N1(double x,y)
• label out /\ clock lt y12 /\ clock lt
  x15 /\ ? N0(clock) .

14
EAGLE by example statistical logics

• Monitor that state property F holds with at least
  probability p
• max Empty() false .
• min Last() ?Empty() .
• min A(Form F, float p, int f, int t)
• ( Last() /\ (( F /\ (1 f/t) gt p) \/
• (F /\ (1 (f1)/t)
  gt p)) )
• \/
• (Last() /\ (( F ? ?A(F, p, f, t1)) /\ (F ?
  ?A(F, p, f1, t1))) .
• min AtLeast (Form F, float p) A(F, p, 0, 1) .

15
EAGLE by example beyond regular

• Monitor a sequence of login and logout events
  at no point should there be more logouts than
  logins and they should match at the end of the
  trace.
• min Match (Form F1, Form F2)
• Empty() \/
• F1 ? Match(F1, F2) ? F2 ? Match(F1, F2)
• mon M1 Match(login, logout)

16
Syntax
17
Semantics
18
Some EAGLE facts

• EAGLE-LTL (past and future). Monitoring formula
  of size m has space complexity bounded by m2 2m
  log m
• EAGLE with data binding has worst case dependent
  on length of input trace
• EAGLE without data is at least Context Free
• EAGLE logic currently implemented by rewriting as
  a Java application

19
EAGLE Internal Calculus

• Uses four functions
• init Form X Form X Form -gt Form
• transforms a monitor formula (1st arg) for
  evaluation, in particular the primitive ? and ?
  are replaced by rules Next and Previous with
  history parameters introduced to past-time rules
• eval Form X State -gt Form
• applies the given state to the formula yielding
  the obligation for the future
• update Form X State X Form X Form -gt Form
• updates the past time components in the formula
  (1st arg)
• value Form -gt Bool
• yields the value of the given formula at the end
  of monitoring

20
EAGLE Internal Calculus eval - II

• evalNext(F), s update F, s, null, null
• Evaluation of a next time formula Next(F) yields
  the obligation to evaluate F in the next state.
  Note that any past time args are updated by
  application of update
• evalPrevious(F, past), s evalpast, s
• Since past is the (possibly partial) evaluation
  of F from the previous state, the evaluation of a
  previous time formula must just re-evaluate past
  in the current state
• The cases of eval for rule definitions are
  synthesised from the rules

21
Correctness of EAGLE calculus

• Theorem
• s1,s2,sn, 1 D F
• iff
• value(eval(eval(eval(init(F,null, null), s1),
  s2), sn)) true
• for all state sequences s1..sn and formulas F

22
EAGLE Simplification rules

• For efficiency, we use the propositional decision
  of Hsiang,
• where formulas are represented in Exclusive Or
  normal form.
• We use the following rewrite rules
• F /\ F F .
• false /\ F false .
• true /\ F F .
• F true ? F .
• false ? F F .
• F1 /\ (F2 ? F3) (F1 /\ F2) ? (F1 /\ F3) .
• F1 \/ F2 (F1 /\ F2) ? F1 ? F2 .

23
Example Execution
?(x gt 0 ? ? x 0)
Specification max A(Term f) f /\ _at_ A(f) . min
E(Term f) f \/ _at_ E(f) . mon M A( x gt 0 ?
E(x 0)).
Trace x1 x2 x0 x3
24
Trace Evaluation
?(x gt 0 ? ? x 0) Formulas A(((x gt 0) /\ E(((x
0) (x 0) /\ Next(E(rec))
Next(E(rec)))) /\ Next(A(rec)) (x gt 0) /\
Next(A(rec)) Next(A(rec)))) state x1 ? x
0 ??(x gt 0 ? ? x 0) E(((x 0) (x 0)
/\ Next(E(rec)) Next(E(rec)))) /\ A(((x gt 0)
/\ E(((x 0) (x 0) /\ Next(E(rec))
Next(E(rec)))) /\ Next(A(rec)) (x gt 0) /\
Next(A(rec)) Next(A(rec)))) state x2 ? x
0 ??(x gt 0 ? ? x 0) A(((x gt 0) /\ E(((x 0)
(x 0) /\ Next(E(rec)) Next(E(rec)))) /\
Next(A(rec)) (x gt 0) /\ Next(A(rec))
Next(A(rec)))) /\ E(((x 0) (x 0) /\
Next(E(rec)) Next(E(rec)))) state x0 ?(x
gt 0 ? ? x 0) A(((x gt 0) /\ E(((x 0) (x
0) /\ Next(E(rec)) Next(E(rec)))) /\
Next(A(rec)) (x gt 0) /\ Next(A(rec))
Next(A(rec)))) state x3 ? x 0 ??(x gt 0 ? ?
x 0) E(((x 0) (x 0) /\ Next(E(rec))
Next(E(rec)))) /\ A(((x gt 0) /\ E(((x 0) (x
0) /\ Next(E(rec)) Next(E(rec)))) /\
Next(A(rec)) (x gt 0) /\ Next(A(rec))
Next(A(rec)))) Warning Property M violated.
25
EAGLE Implementation

• Initial implementation as a Java application
• Two phases
• System compiles the rule and monitor
  specification file to generate a set of Java
  classes, one for each rule and monitor
• System then compiles the generated class files to
  Java bytecode and runs the monitoring engine on a
  given input trace

26
EAGLE interface
class State int x,y update(Event e)
x e.x y e.y
User defines these classes
class Observer Monitors mons State state
eventHandler(Event e) state.update(e)
mons.apply(state)
e1 e2 e3
class Monitors Formula M1, M2
apply(State s) M1.apply(s)
M2.apply(s)
class Event int x,y
27
K9 Planetary Rover Executive

• Executive receives flexible plans from a planner
  for direct execution
• Plan is a hierarchical structure of actions
• Branching based on dynamic conditions flexible
  start and stop times
• Multi-threaded system (35K lines of C)
• Excellent vehicle for case studies on
  validation/verification technology essential
  for autonomy insertion

28
Running X9 Generates Web-Based Output
Model of Plans
plan property generation
plan inputs
behavioural properties
Manual Instrumentation
K9-Rover Executive
EAGLE engine
event stream
reports
filter
Observer
instrumentation
29
Future Work

• Implementation
• Optimisation of implementation especially
  regarding partial evaluation (past time logic
  data bindings).
• Specification
• Support user-defined surface syntax.
• Consider integration of EAGLE with algebraic
  specs, or other spec language, such as ASM,
  SpecWare, , Java.
• Better means for associating actions with
  formulas towards aspect oriented programming,
  fault-tolerance.
• Experiment with different logics in Eagle
  real-time, statistics, ERE, .
• Instrumentation
• Incorporate automated program instrumentation.
• Other Case studies
• Analyze log files generated during tests from
  planning tool (NASA)
• Hardware monitoring for Rainbow (Manchester
  University)

30
Summary

• EAGLE is a succinct but highly expressive finite
  trace monitoring logic. Can elegantly encode any
  monitoring logic we have investigated.
• EAGLE can be efficiently implemented, but users
  must remain aware of expensive features.
• EAGLE demonstrated once by integration within a
  formal test environment, showing the benefit of
  novel combinations of formal methods and test.
• EAGLE can handle very limited form of action in
  current version.

31
Other EAGLE papers

• Barringer, Goldberg, Havelund, Sen
• EAGLE does Space Efficient LTL Monitoring
• Submitted for publication
• (also http//www.cs.man.ac.uk/cspreprints/PrePrint
  s/cspp25.pdf)
• EAGLE Monitors by Collecting Facts and Generating
  Obligations
• (also http//www.cs.man.ac.uk/cspreprints/PrePrint
  s/cspp26.pdf)
• EAGLE for Real (Real-Time and Data Handling) in
  preparation.

If youre interested in flying with EAGLE Ask us