The structure of events

1
The structure of events

• Srini Narayanan
• snarayan_at_icsi.berkeley.edu
• http//www.icsi.berkeley.edu/snarayan
• FN and NTL groups

2
Commonplace language about events

• Day by day, we are moving closer to victory.
• US forces ready to resume final push into
  Baghdad.
• If not crippled, the republican guard is at least
  walking with a limp.
• US Economy on the verge of falling back into
  recession after moving forward on an anemic
  recovery.
• Indian Government stumbling in implementing
  Liberalization plan.
• Moving forward on all fronts, we are going to be
  ongoing and relentless as we tighten the net of
  justice.
• The Government is taking bold new steps. We are
  loosening the stranglehold on business, slashing
  tariffs and removing obstacles to international
  trade.

3
Event Structure in Language

• Fine-grained
• Rich Notion of Contingency Relationships.
• Phenomena Aspect, Tense, Force-dynamics, Modals,
  Counterfactuals
• Event Structure Metaphor
• Phenomena Abstract Actions are conceptualized in
  Motion and Manipulation terms. Schematic
  Inferences are preserved.

4
Aspect

• The term aspect refers to a variety of lexical
  and grammatical devices that languages use to
  specify the structure of events.
• Aspect describes the internal temporal
  constitution of a situation.
• In this respect aspect differs from tense.
• Aspectual categories are invoked by two different
  types of linguistic objects
• Verb-argument structures (situation aspect)
  states, processes, events, e.g., I see the
  mountains vs. I saw a flash.
• Grammatical and word-formation patterns
  (grammatical aspect) type-selecting and
  type-shifting constructions, e.g., Progressive in
  English, Imparfait in French, temporal modifiers
  like in an hour, twice, for ten minutes.

5
Phases, Viewpoints, and Aspects

• John is walking to the store.
• John is about to walk to the store.
• John walked to the store.
• John started walking to the store.
• John is starting to walk to the store.
• John has walked to the store.
• John has started to walk to the store.
• John is about to start walking to the store.
• John resumed walking to the store.
• John has been walking to the store.
• John has finished walking to the store.
• John almost walked to the store.

6
Aspect

• Aspect is the name given to the ways languages
  describe the structure of events using a variety
  of lexical and grammatical devices.
• Viewpoints
• is walking, walk
• Phases of events
• Starting to walk, walking, finish walking
• Inherent Aspect
• run vs cough vs. rub
• Composition with
• Temporal modifiers, tense..
• Noun Phrases (count vs. mass) etc..

7
Inherent Aspect

• Much richer than traditional Linguistic
  Characterizations (VDT (durative/atomic,
  telic/atelic))
• Action patterns
• one-shot, repeated, periodic, punctual
• decomposition concurrent, alternatives,
  sequential
• Goal based schema enabling/disabling
• Generic control features
• interruption, suspension, resumption
• Resource usage

8
Aspectual types

• Mappings describe the effect of grammatical
  aspect on situation type
• Progressive process ? state, e.g., She was
  running home.
• Perfect event ? state, e.g., Ive had a
  wonderful evening but this wasnt it.
• Inceptive process ? event, e.g., She started
  knitting.
• Prospective event ? state, e.g., Shes about to
  leave.
• They use a single mechanism to capture both
  implicit and explicit transitions
• For example, the inceptive transition is used to
  model not only inceptive grammatical aspect but
  also inceptive coercion, as in The program ran
  within a few seconds.

9
Background Primate Motor Control

• Relevant requirements (Stromberg, Latash, Kandel,
  Arbib, Rizzolatti)
• Should model coordinated, distributed,
  parameterized control programs required for motor
  action and perception.
• Should be an active structure.
• Should be able to model concurrent actions and
  interrupts.
• Model
• The NTL project has developed a computational
  model based on that satisfies these requirements
  (x- schemas).
• Details, papers, etc. can be obtained on the web
  at http//www.icsi.berkeley.edu/NTL

10
Basic Primitives

• An fine-grained executing model of action and
  events (X-schemas)
• A factorized representation of state (DBNs)
• A model of metaphor maps that project bindings
  from source to target domains.

11
An Active Model of Events

• Computationally, actions and events are coded in
  active representations called x-schemas which are
  extensions to Stochastic Petri nets.
• x-schemas are fine-grained action and event
  representations that can be used for monitoring
  and control as well as for inference.

12
Model Review
Basic Mechanism
1
1
13
Model Review
Firing Semantics
14
Model Review
Result of Firing
15
X-Schema Extensions to Petri Nets

• Parameterization
• x-schemas take parameter values (speed, force)
• Walk(speed slow, dest store1)
• Dynamic Binding
• X-schemas allow run-time binding to different
  objects/entities
• Grasp(cup1), push(cart1)
• Hierarchical control and durative transitions
• Walk is composed of steps which are composed of
  stance and swing phases
• Stochasticity and Inhibition
• Uncertainties in world evolution and in action
  selection

16
Active representations

• Many inferences about actions derive from what we
  know about executing them
• Representation based on extending stochastic
  Petri nets captures dynamic, parameterized nature
  of actions

Walking bound to a specific walker with a
direction or goal consumes resources (e.g.,
energy) may have termination condition(e.g.,
walker at goal) ongoing, iterative action
17
States

• Factorized Representation of State uses Dynamic
  Belief Nets (DBNs)
• Probabilistic Semantics
• Structured Representation

18
States and Domain Knowledge

• Factorized Representation using Dynamic Belief
  Nets (DBNs)
• Probabilistic Semantics
• Structured Representation

19
Belief Networks

• Expoits conditional independence requiring only
  local conditional beliefs.
• Basic operation is conditioning in the presence
  of evidence.
• Supports Multiple inference types

Forward
Inter-causal
Backward
20
(No Transcript)
21
Features of Representation

• Inherently action based, with fine grained
  distinctions in resource usage, and temporal
  evolutions.
• Can deal with concurrent actions, durations,
  hierarchical action sets, and stochastic actions
  (selection and effects).
• Highly responsive to a changing environment with
  uncertain evolutions.
• Can model complex domain constraints in a
  factorized representation that can compute
  complex ramifications as well as prior beliefs
  and possible predictions.

22
Aspectual Types
23
(No Transcript)
24
(No Transcript)
25
(No Transcript)
26
(No Transcript)
27
Metaphor Maps

• Static Structures that project bindings from
  source domain f- struct to target domain Belief
  net nodes by setting evidence on the target
  network.
• Different types of maps
• PMAPS project X- schema Parameters to abstract
  domains
• OMAPS connect roles between source and target
  domain
• SMAPS connect schemas from source to target
  domains.
• ASPECT is an invariant in projection.

28
Talk Outline

• Introduction and Background
• Basic Computational Result on Metaphor
• Background and Basic Result
• Basic Primitives Events, Processes, States and
  Maps
• Aspect and Language about Events
• Simulation Semantics
• The Metaphor Interpretation System
• Extensions/Scaling Up
• Embodied Construction Grammar
• FrameNet
• Linking to Simulation Semantics
• Conclusion

29
A Walk X-schema
30
A Climb X-schema
31
Common Patterns
START
FINISH
32
(No Transcript)
33
A Schema Controller
iterate
Ready
Done
Start
Process
Finish
interrupt
resume
Cancel
Suspend

• An active controller that sends signals to the
  embedded schema and transitions based on signals
  from the embedded schema.
• Useful for higher level coordination of actions.

34
A Generic Process Schema
iterate
Ready
Done
Start
Process
Finish
interrupt
resume
Cancel
Suspend

• Part of Conceptual Structure.
• Generalizes over actions and events. Has
  internal state and models evolution of processes.

35
Aspects of (Climb)
Iterate
Ready
Done
Start
Process
Finish
resume
Suspend
interrupt
Cancel
BINDINGS
Energy Ready Standing
On top
36
About to (Climb) (Prospective)
Iterate
Ready
Done
Start
Process
Finish
resume
Suspend
interrupt
Cancel
BINDINGS
Energy Ready Standing
On top
37
Be (Climb)-ING (Progressive)
Iterate
Ready
Done
Start
Process
Finish
resume
Suspend
interrupt
Cancel
BINDINGS
Energy Ready Standing
On top
38
Have (Climb)-ed (Perfect)
Energy Ready Standing
On top
39
Phasal Aspect Maps to the Controller
Iterative (repeat)
Inceptive (start, begin)
Iterate
Ready
Done
Start
Process
Finish
interrupt
resume
Cancel
Suspend
Completive (finish, end)
Resumptive(resume)
40
Aspect

• Aspect is the name given to the ways languages
  describe the structure of events using a variety
  of lexical and grammatical devices.
• Viewpoints
• is walking, walk
• Phases of events
• Starting to walk, walking, finish walking
• Inherent Aspect
• run vs cough vs. rub
• Composition with
• Temporal modifiers, tense..
• Noun Phrases (count vs. mass) etc..

41
Inference using the Controller
Different Bindings give rise to different
interpretations.
Dowtys Imperfective Paradox He was walking to
the store. He was walking. does not
imply does
imply He walked to the store.
He walked.
42
Embedding About to start (X)
Ready
Done
Start
Process
Finish
resume
Suspend
interrupt
R
D
S
P
F
r
i
S
C
X-Schema for X with bindings
43
Embedding Has Started (to X)
Ready
Done
Start
Process
Finish
resume
Suspend
interrupt

X-Schema for X with bindings
44
Embedding The end of the beginning
Ready
Done
Start
Process
Finish
resume
Suspend
interrupt
R
D
S
P
F
r
i
S
C

X-Schema for X with bindings
45
Embedding The beginning of the end
Ongoing
Finish
Done
S
R
D
P
F
r
i
S
C

X-Schema for X with bindings
46
Inherent Aspect

• Much richer than traditional Linguistic
  Characterizations (VDT)
• Action patterns
• one-shot, repeated, periodic, punctual
• decomposition concurrent, alternatives,
  sequential
• Goal based schema enabling/disabling
• Generic control features
• interruption, suspension, resumption
• Resource usage

47
Inherent Aspect Selects/Disables Controller
Transitions
48
(No Transcript)
49
Viewpoint Aspect (Perfective/Imperfective)
50
States and the Controller
51
Composition with nominals
52
Duration Modifiers
53
Problems?

• She walked into the room for an hour.
• She walked through the room for an hour.
• He ran onto the football field for an hour.

54
Other Transitions in the Controller may be coded

• Lexical items may code interrupts
• Stumble is an interrupt to an ongoing walk
• A combination of grammatical and aktionsart may
  code of the controller phases
• Ready to walk Prospective
• Resuming his run Resumptive
• Has been running Embedded progressive
• About to Finish the painting Embedded
  Completive.
• Canceling the meeting vs. Aborting the meeting.

55
Embedding The beginning of the end
Ongoing
Finish
Done
S
R
D
P
F
r
i
S
C

X-Schema for X with bindings
56
Evidence for Simulation Semantics

• BASIC ASSUMPTION SAME REPRESENTATION FOR
  PLANNING AND SIMULATIVE INFERENCE
• Evidence for common mechanisms for recognition
  and action (mirror neurons) in the F5 area
  (Rizzolatti et al (1996), Gallese 96, Boccino
  2002) and from motor imagery (Jeannerod 1996)
• IMPLEMENTATION
• x-schemas affect each other by enabling,
  disabling or modifying execution trajectories.
  Whenever the CONTROLLER schema makes a transition
  it may set, get, or modify state leading to
  triggering or modification of other x-schemas.
  State is completely distributed (a graph marking)
  over the network.
• RESULT INTERPRETATION IS IMAGINATIVE SIMULATION!

57
(No Transcript)
58
(No Transcript)
59
(No Transcript)
60
(No Transcript)
61
(No Transcript)
62
(No Transcript)
63
A Precise Notion of Contingency Relations
Activation Executing one schema causes the
enabling, start or continued execution of another
schema. Concurrent and sequential
activation. Inhibition Inhibitory links prevent
execution of the inhibited x-schema by activating
an inhibitory arc. The model distinguishes
between concurrent and sequential inhibition,
mutual inhibition and aperiodicity. Modification
The modifying x-schema results in control
transition of the modified xschema. The execution
of the modifying x-schema could result in
the interruption, termination, resumption of the
modified x-schema.
64
Combination with temporal connectives

• I bought stock when the market crashed.
• The market crashed when I bought stock.

• When they built the 39th Street bridge...
• a local architect drew up the plans.
• they used the best materials.
• they solved most of their traffic problems.

65
Inter-Schema relations
66
Combination with temporal primitives

• When the built the bridge,
• they lost the plans.
• they forgot to give the commuting public adequate
  warning.
• they ran out of materials
• they had a great opening event.
• they solved the traffic problem.
• When they were building the bridge .

67
Zoom-Out and Habituality
SHE SMOKES
R
D
S
F
P
r
i
S
C
68
Structural vs. Actual

• Does the habitual zoom-out operation result in a
  structural/actual distinction. In the model this
  would be outside the X-schema and an assertion in
  the PRM/Belief State?
• The process model affords both possibilities
• The Aspectual data doesnt support the
  distinction wrt. to habituals (Michaelis02).

69
Interaction of Aspect with Tense

• Reichenbachs system uses three pointers
• Speech Time (S)
• Reference Time (R)
• Event Time (E)
• Tense is a partial ordering relation between the
  pointers
• Simple Past E lt R, E lt S
• Perfect E lt R lt S

70
(No Transcript)
71
The Present Tense

• The Present tense is a state selector.
• It therefore selects a rest from an input event,
  resulting in various coercion effects
• Habitual and generic readings of iterated-event
  sentences, e.g., She smokes, Oil floats on water
• Progressive-style readings of event sentences
  in languages other than English, e.g., French Eh
  bien, à present, je me sens mieux. Le morale
  revient. Now Im feeling better. My morale is
  coming back. (Binet, Bidochon 8 42)
• Perfect-style readings of state-phase sentences
  in languages other than English, e.g., Ca fait
  dix minutes quelle nous parle de la moquette!
  Shes been telling us about the carpet for 10
  minutes. (Binet, Bidochon 1017)

72
The Present Triumvirate
JAN RUNS
JAN IS RUNNING
R
S
P
F
r
i
S
C
R
S
P
F
r
i
S
C
JAN HAS BEEN RUNNING
73
Other Present Tense Affordances

• Of course, we can extend through embedding ANY of
  the available states in the CONTROLLER.
• John is starting his run.
• John starts his run (every morning).
• John stops his run after 3 miles. (He never
  achieves his goal of running 5).
• John has been canceling his run.
• John cancels his run (twice a week).
• We have been restarting this Harley for the last
  5 mins.
• The meeting is about to resume.
• My morale is returning (Michaelis 02).
• Question Do (which) languages have constructions
  for these states?

74
Perfective/Imperfective
Perfective
Imperfective
75
Simulation and Reference Interval
Perfective
Imperfective
76
Two types of past tense

• Two types of past tense
• Imperfective
• Selects a state.
• States contain their reference interval
• Perfective
• Selects an whole event
• Events are contained within their reference
  interval

77
Events and Past tense coercions

• John ran yesterday.
• Episodic
• I glanced at her. she didnt notice. She looked
  elated.
• Stative
• When the bookie came to collect, John ran
  away.
• Inceptive.

78
Events and Past tense coercions

• John ran yesterday.
• Episodic
• I glanced at her. she didnt notice. She looked
  elated.
• Stative
• When the bookie came to collect, John ran
  away.
• Inceptive.

79
Summary of Aspect Results

• Controller mediates between linguistic markings
  and individual event/verbal x-schemas (Cogsci99)
• Captures regular event structure inspired by
  biological control theory
• Flexible specific events may require only a
  subset of controller interaction of underlying
  x-schemas, linguistic markers and hierarchical
  abstraction/ decomposition of controller accounts
  for wide range of aspectual phenomena.
• Important aspectual distinctions, both
  traditional and novel, can be precisely specified
  in terms of the interaction of x-schemas with the
  controller (Cogsci97,98, AAAI99)
• stative/dynamic, durative/punctual natural in
  x-schemas
• telic processes depletion of resources
• continuous processes consumption of resources
• temporary/effortful states habituals
• dynamic interactions with tense, nominals,
  temporal modifiers
• incorporation of world knowledge, pragmatics

80
Logical Action Theories

• Connection to ARD (or other Action Languages)
• The representation can be used to encode a causal
  model for a domain description D (in the Syntax
  of ARD) in that it satisfies all the causal laws
  in D. Furthermore, a value proposition of the
  form C after A is entailed by D iff all the terms
  in C are in Si the state that results after
  running the projection algorithm on the action
  set A. (IJCAI 99)
• Executing representation,
• frame axioms are encoded in the topology of the
  network and transition firing rules respect them.
• Planning as backward reachability or computing
  downward closure (IJCAI 99, WWW2002)
• Links to linear logic. Perhaps a model of
  stochastic linear logic? (SRI CSL TR 2001).

81
Current Work

• How does analysis provide the right reference
  interval properties for simulation?
• Aspectually sensitive tenses
• Granularity
• Temporal Connectives
• Hypothesis
• A simulation/enactment framework with rich
  inter-event relations (through activation,
  inhibition, interruption, termination, etc.)
  provides the right framework.

82
Features of Simulation Semantics

• Captures fine grained distinctions needed for
  interpretation
• Frame-based Inferences (COLING02)
• Aspectual Inferences (Cogsci98, IJCAI 99,
  COLING02)
• Metaphoric Inferences (AAAI 99)
• Sufficient Inductive bias for verb learning
  (Bailey97, CogSci99), construction learning
  (Chang02, to Appear)
• Captures essential features of neural computation
  (FeldmanBallard82, Feldman89, Valiant 94)
• Active, context sensitive knowledge
  representation.
• Natural model of concurrent and distributed
  computation at the knowledge level.
• Proposition Simulation Semantics is Biologically
  Motivated. (Boccino et al. 2001, NBL01, CNS02)

83
Connectionist Implementation

• x- schemas have been implemented in a
  connectionist network.
• Two main issues arise in the implementation.
• 1) Dynamic Binding.
• 2) Belief Propagation.
• Dynamic binding is modeled through temporal
  synchrony in SHRUTI.
• Purely local belief propagation requires
  restricting the topology of the domain models?

84
Experimental Verification of the Simulation
Hypothesis

• Behavioral Image First
• Does shared effector slow negative response?
• Pilot results (Bergen and Shweta Narayan)
• Imaging Simple sentence using verb first
• Does verb evoke activity in pre-motor effector
  area?
• Collaborators at Parma and Milan have obtained
  preliminary results.
• Berkeley Experiment under way (Shweta Narayan,
  Rich Ivry)
• Metaphor follow-on experiment
• Will kick the idea around evoke motor activity?
• Investigate the finer details of the simulation
  hypothesis.

85
Conclusions

• Embodiment can provide crucial insights for NLU
• Non-trivial action and interaction requires
  representations of events, states and domain
  relations.
• Representation of events based on motor control
  and imaginative simulation
• Substantial Progress in exploiting results in NLU
• We have built a pilot system that uses some of
  the key technologies in a proof of concept
  implementation.
• We are currently extending the pilot system to
• Use richer probabilistic representation and
  inference techniques that are able to scale to
  large domains and ontologies.
• Formalize and employ a compositional set of
  embodied conceptual primitives and grammatical
  constructions.
• Perform both behavioral and fMRI imaging
  experiments to test the predictions of the
  simulation hypothesis