Modeling Human Reasoning About MetaInformation

1
Modeling Human Reasoning About Meta-Information

• Presented By
• Scott Langevin
• Jingsong Wang

2
Introduction

• Human decision-making in real-time, dynamic
  environments is becoming more complex
• Decision-makers must manage large amounts of
  incoming information and integrate it with
  previous knowledge to develop a situational
  awareness
• Relies on domain-knowledge but also on the
  qualifiers (meta-information) describing the
  information
• Problem To replicate human reasoning or
  behavior, need to model both information and
  meta-information
• Most approaches have focused on representing the
  information, but little discussion of the
  meta-information

3
What is Meta-Information?

• Definitions
• Data is output from a system that may or may not
  be useful to decision-making (radar reports storm
  is coming)
• Information is recognized inputs that are useable
  to decision-making (storm is coming that may
  affect UAVs)
• Meta-data is qualifiers of data that may or may
  not be useful to decision-making (radar can
  locate aircraft with error of /-1.5m)
• Meta-information is qualifiers of information
  that affect decision-making, reasoning, or
  behavior
• Information processing
• Situation awareness
• Decision-making
• Definitions serve to explicitly identify the
  critical role of meta-information in human
  decision-making

4
Human Behavioral Models

• Attempt to replicate human cognitive processes
• Attempt to model human behaviors must capture the
  impact of meta-information
• HBM have wide variety of applications
• Developing and testing theories of human
  cognition
• Representing realistic human behavior in training
• Expert and Decision Support Systems
• Modelers typically do not address
  meta-information because of challenges acquiring,
  aggregating and integrating
• Focus of this research is on modeling
  meta-information in Bayesian Belief Networks
  (BBNs)

5
Uncertainty and Human Decision-Making

• Human decision-making under uncertainty deviates
  from logical decision-making and largely based on
  experience-based heuristic methods
• Often the heuristics represent how experts reason
  about the meta-information
• Uncertainty of information is one type of
  meta-information
• Different methods of classifying uncertainty
• Executional uncertainty
• Goal uncertainty
• Environment uncertainty
• Lack of information, etc
• While these classifications of uncertainty and an
  understanding of their impacts on decision-making
  have been useful, they may not generalize to
  other types of meta-information not based on
  uncertainty (recency, reliability, trust)

6
Computational Approaches to Uncertainty

• Probability Measures
• DempsterShafer belief functions
• Extensions to first-order logic (e.g., defeasible
  reasoning, argumentation)
• Ranking functions
• plausibility measures
• Fuzzy set theory
• Causal network methods (e.g., Bayesian belief
  networks, similarity networks, influence
  diagrams)

7
Types and Sources of Meta-Information

• Identified the main types of meta-information
  that impact the decision-making process
• Research from over 30 domain experts, and over
  500h of interviews, observations and evaluations
• From this developed a list of sources and types
  of meta-information that was consistently
  encountered across application domains
• Believe this approach developed an understanding
  of expert reasoning and behavior sufficient to
  understand the impact of meta-information at a
  level that supports modeling

8
Types and Sources of Meta-Information
9
Modeling Human Reasoning and Behavior

• Computational Representation of human reasoning
  and behavior
• Model based on recognition-primed decision-making
• Experts do not do significant amounts of
  reasoning and problem solving, but rather have
  been trained to recognize critical elements of a
  situation and act accordingly
• Domain independent, modeling situation
  awareness-centered decision-making in
  high-stress, time-critical environments
• SAMPLE is a general use HBM
• Defined modules Information Processing,
  Situation Assessment, Decision Making
• Inputs processed by information processing module
• Processed data (detected events) passed to
  situation assessment module
• Assessed situation is passed to decision-making
  module
• Rules, or lookup table of actions after
  situational assessment performed

10
SAMPLE Model
11
Bayesian Modeling about and with Meta-Information

• Difficult aspect of modeling human cognition and
  behavioral processes is the need to reflect the
  known impacts of meta-information on those
  processes
• Identified five features of reasoning that need
  representation within human behavior models
• Should succeed or fail to recognize relevant
  meta-information based on attentional and
  cognitive demands
• Should support the representation of successful
  or unsuccessful human strategies to process
  information according to meta-information
• Should represent the aggregation of
  meta-information
• Should capture how effectively meta-information
  is understood relative to any prior understanding
  or knowledge
• Should succeed and fail at incorporating
  meta-information-mediated situation assessments
  into behavior or decisions

12
Methods for Representing Human Reasoning

• Bayesian belief networks
• Fuzzy set theory
• Rule-based production systems
• Case-based reasoning
• BBNs address multiple types of modeling
  requirements
• Two types of meta-information reasoning
• Deductive reasoning
• Abductive reasoning
• BBNs support both types of reasoning

13
Two Types of Reasoning
14
Modeling the Recognition and Aggregation of
Meta-Information

• In many cases, human decision-makers will have to
  compute meta-information from multiple factors
• Data and meta-data can map to meta-information in
  the following ways
• One-to-one mappings
• Many-to-one mappings
• One-to-many mappings
• Many-to-many mappings
• Once meta-information is calculated, it can
  influence the information gathering, situation
  assessment, and decision-making process

15
Applying BBNs to Model Congnitive Computation of
Meta Information
16
Sensor Type as Node in Network Sensor Type 3
17
Sensor Type as Node in Network Sensor Type 1
18
Aggregating Meta-Information to Compute Overall
Confidence
19
Modeling the Recognition and Aggregation of
Meta-Information

• Knowing the best means to aggregate
  meta-information is challenging
• Observation and study of human decision-making
  amongst subject matter experts may provide some
  justification, but will often unavoidably result
  in inclusion of biases
• Using engineering data about sources may not
  adequately represent how a human would reason
  about meta-information, resulting in less
  reflective human behavior models

20
Modeling the Impact of Meta-Information on
Situation Assessment

• Three Approaches
• Simply filter or prioritize information based on
  meta-information
• Include meta-information within BBN models of
  information gathering, situation assessment, and
  decision-making processes
• Use the meta-information in a specific parameter

21
Incorporating Meta-Information Explicitly into a
BBN No Confidence
22
Incorporating Meta-Information Explicitly into a
BBN Low Confidence
23
Examples of Computing the Probability of a
Discrete Value for a BBN Node
24
Conclusion

• We described the application of meta-information
  and BBNs in modeling each of the following types
  of cognitive tasks
• Recognition of relevant meta-information based on
  aggregation of available data, meta-data,
  information, and meta-information into types of
  meta-information.
• Filtering and prioritization of information based
  on meta-information.
• Aggregation of different types of
  meta-information to acquire their combined
  impact.
• Understanding of the impact of meta-information
  on existing knowledge
• Incorporation of meta-information into mediation
  of situation assessment and decision-making.

25
Questions?