Increasing Trust in Answers from the Web: the Inference Web Approach - PowerPoint PPT Presentation

Loading...

PPT – Increasing Trust in Answers from the Web: the Inference Web Approach PowerPoint presentation | free to download - id: 23e44-OTM3N



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Increasing Trust in Answers from the Web: the Inference Web Approach

Description:

WineAgent: www.ksl.stanford.edu/people/dlm/webont/wineAgent ... KSL Wine Agent: Semantic Web Integration (Toy) Example ... from www-ksl-svc.stanford.edu ... – PowerPoint PPT presentation

Number of Views:104
Avg rating:3.0/5.0
Slides: 57
Provided by: debor118
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Increasing Trust in Answers from the Web: the Inference Web Approach


1
Increasing Trust in Answers from the Webthe
Inference Web Approach
  • Deborah McGuinness
  • Co-Director and Senior Research Scientist
  • Knowledge Systems Laboratory
  • Stanford University
  • dlm_at_ksl.stanford.edu
  • http//www.ksl.stanford.edu/people/dlm
  • Inference Web is joint work with Pinheiro da
    Silva, Fikes, Chang, Deshwal, Narayanan, Glass,
    Makarios, Jenkins, Millar,

2
Semantic Web Layers
  • Ontology Level
  • Languages (CLASSIC, DAML-ONT, DAMLOIL, OWL, )
  • Environments (FindUR, Chimaera,
    OntoBuilder/Server, Sandpiper Tools, )
  • Standards (NAPLPS, , W3Cs WebOnt, W3Cs
    Semantic Web Best Practices, EU/US Joint
    Committee, OMG ODM,
  • Rules
  • SWRL (previously CLASSIC Rules, explanation
    environment, extensibility issues, contracts,
    )
  • Logic
  • Description Logics
  • Proof
  • PML, Inference Web Services and Infrastructure
  • Trust
  • IWTrust, NSF with W3C/MIT

http//www.w3.org/2004/Talks/0412-RDF-functions/sl
ide4-0.html
3
Motivation Trust and Understanding
  • If users (humans and agents) are to use, reuse,
    and integrate system answers, they must trust
    them.
  • System transparency supports understanding and
    trust.
  • Even simple lookup systems benefit from
    providing information about their sources.
  • Systems that manipulate information (with sound
    deduction or potentially unsound heuristics)
    benefit from providing information about their
    manipulations.

Goal Provide interoperable infrastructure that
supports explanations of sources, assumptions,
and answers as an enabler for trust.
4
Requirements gathered from
  • DARPA Agent Markup Language (DAML)
  • Enable the next generation of the web
  • DARPA Personal Assistant that Learns (PAL)
  • Enable computer systems that can reason, learn,
    be told what to do, explain what they are doing,
    reflect on their experience, respond robustly
    to surprise
  • DARPA Rapid Knowledge Formation (RKF)
  • Allow distributed teams of subject matter experts
    to quickly and easily build, maintain, and use
    knowledge bases without need for specialized
    training
  • DARPA High Performance Knowledge Base (HPKB)
  • Advance the technology of how computers acquire,
    represent manipulate knowledge
  • ARDA Novel Intelligence for Massive Data (NIMD)
  • Avoid strategic surprise by helping analysts be
    more effective (focus attention on critical
    information and help analyze/prune/refine/explain/
    reuse/)
  • ARDA Advanced Question Answering for
    Intelligence (AQUAINT)
  • Advance QA against structured and unstructured
    info
  • Consulting including search, ecommerce,
    configuration,

5
Requirements
  • Information Manipulation Traces
  • hybrid, distributed, portable, shareable,
    combinable encoding of proof fragments supporting
    multiple justifications
  • Presentation
  • multiple display formats supporting browsing,
    visualization, summaries,
  • Abstraction
  • understandable summaries
  • Interaction
  • multi-modal mixed initiative options including
    natural-language and GUI dialogues, adaptive,
    context-sensitive interaction
  • Trust
  • source and reasoning provenance, automated trust
    inference
  • McGuinness Pinheiro da Silva, ISWC 2003, J.
    Journal of Web Semantics 2004

6
Selected History
  • Historical explanation research motivated by
    explaining theorem provers in practice
  • Web version originally aimed at explaining hybrid
    (FOL / special purpose) reasoners in a
    distributed environment like the web.
  • User demand drove focus on provenance extensions
  • Current web environment and programs, such as
    NIMD, drove connections with extraction engines
  • Current view Any question answering system can
    be viewed as some kind of information manipulator
    that may benefit from and/or require explanation

7
Inference Web
  • Framework for explaining question answering tasks
    by abstracting, storing, exchanging, combining,
    annotating, filtering, segmenting, comparing, and
    rendering proofs and proof fragments provided by
    question answerers
  • IWs Proof Markup Language (PML) is an
    interlingua for proof interchange. It is written
    in W3Cs recommended Ontology web language (OWL)
  • IWBase is a distributed repository of
    meta-information related to proofs and their
    explanations
  • IW Registration services provide support for
    proof generation and checking
  • IW Browser provides display capabilities for PML
    documents containing proofs and explanations
    (possibly from multiple inference engines)
  • IW Abstractor provides rewriting capabilities
    enabling more understandable presentations
  • IW Explainer provides multi-modal dialogue
    options including alternative strategies for
    presenting explanations, visualizations, and
    summaries
  • Work with Pinheiro da Silva

8
Explainable System Structure
Trust
Explanation
Interaction
Presentation
Abstraction
PML
InferenceML
Proof Interlingua (PML)
Inference Rule Specs
Source Provenance Data
Information Manipulation Data
Source Provenance Data
Information Manipulation Data
9
Registry Information
  • IWBase has core and domain-specific repositories
    of meta-data useful for disclosing knowledge
    provenance and reasoning information such as
    descriptions of
  • Question answering systems (Inference Engines,
    Extractors, ) along with their supported
    inference rules
  • Information sources such as organizations,
    publications and ontologies
  • Representation languages along with their axioms

10

11

12
Browsing Proofs
  • Enable the visualization of proofs (and
    abstracted proofs)
  • Proofs can be extracted and browsed from both
    local and remote PML node sets and can be
    combined
  • Links provide access to proof-related
    meta-information

select
select
13
Browsing Proofs
14
Explainer
  • Present
  • Query
  • Answer
  • Abstraction of justification (PML information)
  • Limited meta information
  • Suggests drill down options (also provides
    feedback options)

15
UIMA Explanation
16
Follow-up Metadata
17
Follow-up Assumptions
18
Explaining Extracted Entities (Techies)
Sentences in English
Sentences in annotated English
Sentences in logical format, i.e., KIF
19
Further Observations on Explaining Extracted
Entities
Source fbi_01.txt Source Usage span from 01 to
78
Same conclusion from multiple extractors
conflicting conclusion from one extractor
This extractor decided that Person_fbi-01.txt_46
is a Person and not Occupation
20
Search / Configuration
21
KSL Wine AgentSemantic Web Integration Example
  • Uses emerging web standards to enable smart web
    applications
  • Given a meal description
  • Deborahs Specialty
  • Describe matching wines
  • White, Dry, Full bodied
  • Retrieve some specific options from web
  • Forman Chardonnay from DLMs cellar, ThreeSteps
    from wine.com, .
  • Info http//www.ksl.stanford.edu/people/dlm/webo
    nt/wineAgent/

22
KSL Wine Agent Semantic Web Integration
Technology
OWL for representing a domain ontology of
foods, wines, their properties, and relationships
between them JTP theorem prover for deriving
appropriate pairings DQL/OWL QL for querying
a knowledge base Inference Web for explaining
and validating answers (descriptions or
instances) Web Services for interfacing with
vendors Connections to online web
agents/information services Utilities for
conducting and caching the above transactions
23

24
Knowledge Provenance Elicitation
XYZ says AB is the answer for my question.
Provenance information may be essential for users
to trust answers.
Data provenance (aka data lineage) is defined and
studied in the database literature. Buneman et
al., ICDT 2001 Cui and Widom, VLDB 2001
Why should I believe this?
Knowledge provenance extends data provenance by
adding data derivation provenance
information Pinheiro da Silva, McGuinness
McCool, Data Eng. Bulletin, 2003
25
IWTrust Trust in Action
Google-2.0 says AB is the answer for my
question.
Trust can be inferred from a Web of Trust.

Why should I trust the answer?
0

?

IWTrust provides infrastructure for building webs
of trust.
The infrastructure includes a trust component
responsible for computing trust values for
answers.
IWTrust is described in Zaihrayeu, Pinheiro da
Silva McGuinness, iTrust 2005
AB
0



0
?
?
26
Explanation Application Areas
  • Theorem proving
  • First-Order Theorem Provers Stanford (JTP
    (KIF/OWL/)) SRI (SNARK) University of Texas,
    Austin (KM)
  • SATisfiability Solvers University of Trento
    (JSAT)
  • Information extraction IBM (UIMA), Stanford
    (TAP)
  • Information integration/aggregation USC ISI
    (Prometheus,Mediator - Fetch) Rutgers ,
    Stanford (TAP)
  • Task processing SRI International (SPARK)
  • Service composition Stanford, U. of Toronto,
    UCSF (SDS)
  • Semantic matching University of Trento
    (S-MATCH)
  • Debugging ontologies U of Maryland, College
    Park (SWOOP/Pellet)
  • Problem solving University of Fortaleza
  • Trust Networks U. of Trento (IWTrust), UMd

27
Inference Web Contributions
  • Language for encoding hybrid, distributed proof
    fragments based on web technologies. Support for
    both formal and informal proofs (information
    manipulation traces).

Trust
Explanation
Interaction
2. Support (registry, language, services) for
knowledge provenance
Presentation
Abstraction
3. Declarative inference rule representation for
checking hybrid, distributed proofs.
Inference Meta-Language
Proof Markup Language
Inference Rule Specs
Provenance Meta-data
Information Manipulation Data
4. Multiple strategies for proof abstraction,
presentation and interaction.
5. End-to-end trust value computation for
answers.
6. Comprehensive solution for explainable systems
28
Status
  • Inference Web infrastructure (PML, browser,
    explainer, registry, toolkit) being used in
    government programs such as PAL and NIMD,
    commercial research labs IBM, Boeing, SRI,
    Universities USC, U MD,
  • Integration and registration process underway
    with extraction community
  • Useful now for helping decide if information is
    trustworthy, comes from authoritative sources,
    consistent, reliable
  • Benefits from more meta data and more information
    population but is useful in an incremental nature

29
Technical Status
  • Some focus areas
  • Follow-up question support
  • Trust
  • Contradiction support
  • Abstraction techniques
  • Extraction extensions
  • Task-oriented reasoning support
  • Query manager explanation support
  • Toolkit for embedding
  • Open issues for explanation
  • Granularity of explanations
  • Meta information filtering
  • Abstraction techniques
  • Requests / Suggestions?

30
More Info
  • Inference Web http//iw.stanford.edu
  • OWL http//www.w3.org/TR/owl-features/
    http//www.w3.org/TR/owl-guide/
  • DAMLOIL http//www.daml.org/
  • WineAgent www.ksl.stanford.edu/people/dlm/webont/
    wineAgent/
  • Chimaera http//www.ksl.stanford.edu/software/c
    himaera/
  • OWL-QL/DQL http//www.ksl.stanford.edu/projec
    ts/dql/
  • UIMA http//www.research.ibm.com/UIMA/
  • dlm_at_ksl.stanford.edu

31
Extras

32
Background
  • ATT Bell Labs AI Principles Dept
  • Description Logics, CLASSIC, explanation,
    ontology environments
  • Semantic Search, FindUR, Collaborative Ontology
    Building Env
  • Apps Configurators, PROSE/Questar, Data Mining,
  • Stanford Knowledge Systems, Artificial
    Intelligence Lab
  • Ontology Evolution Environments (Diagnostics and
    Merging) Chimaera
  • Explanation and Trust, Inference Web
  • Semantic Web Representation and Reasoning
    Languages, DAML-ONT, DAMLOIL, OWL,
  • Rules and Services SWRL, OWL-S, Explainable SDS,
    KSL Wine Agent
  • McGuinness Associates
  • Ontology Environments Sandpiper, VerticalNet,
    Cisco
  • Knowledge Acquisition and Ontology Building
    VSTO, GeON, ImEp,
  • Applications GM Search, etc. CISCO meta
    data org, etc.
  • Boards Network Inference, Sandpiper,
    Buildfolio, Tumri, Katalytik

33
KSL Wine Agent Semantic Web Integration (Toy)
Example
  • Uses emerging web standards to enable smart web
    application
  • Given a meal description
  • Deborahs Specialty, a crab dish,
  • Describe matching wines
  • White, Dry, Full bodied
  • Retrieve some specific options from web
  • Forman Chardonnay from DLMs cellar, ThreeSteps
    from wine.com, .
  • Explain description or specific suggestion
  • Info http//www.ksl.stanford.edu/people/dlm/webo
    nt/wineAgent/

34
KSL Wine Agent Semantic Web Integration
Technology
  • OWL for representing a domain ontology of
    foods, wines, their properties, and relationships
    between them
  • JTP theorem prover for deriving appropriate
    pairings
  • Chimaera ontology diagnostics and ontology
    merging
  • DQL/OWL QL for querying a knowledge base
  • Inference Web for explaining and validating
    answers (descriptions or instances)
  • Web Services for interfacing with vendors
  • Connections to online web agents/information
    services
  • Utilities for conducting and caching the above
    transactions

35
Inference Web in KANI Context
36
Summary
  • Tools are emerging that support understanding
    information
  • Understanding/Evaluating information can help
    focus a users attention and enable trust, reuse,
    and filtering
  • Semantic Web infrastructure (OWL, Structured
    query languages, Semantic Search, Extractors,
    Reasoners, Explanation Infrastructure, .) is
    ready for use and a growing trend

37
Knowledge Provenance Multiple Sources
Answer
Source
Source
38
Extra
39
Chimaera Trend Ontology Environment
  • An interactive web-based tool aimed at
    supporting
  • Ontology analysis (correctness, completeness,
    style, )
  • Merging of ontological terms from varied sources
  • Maintaining ontologies over time
  • Validation of input
  • Features multiple I/O languages, loading and
    merging into multiple namespaces, collaborative
    distributed environment support, integrated
    browsing/editing environment, extensible
    diagnostic rule language
  • Used in commercial and academic environments,
    basis of some commercial re-implementations
    (Ontobuilder/Ontoserver, Sandpiper, CISCO MDF,
    GM)
  • Available as a hosted service from
    www-ksl-svc.stanford.edu
  • Information www.ksl.stanford.edu/software/chima
    era
  • More features required versioning, diffs, .

40
Layer Cake Foundation
Current work
41
FindUR Architecture
Content to Search
CLASSIC Knowledge
Representation System
Research Site Technical Memorandum Calendars
(Summit 2005, Research) Yellow
Pages (Directory Westfield) Newspapers (Leader)
Internal Sites (Rapid Prototyping) ATT
Solutions Worldnet Customer Care
Content (Web Pages or Databases
Content Classification
Domain Knowledge
Domain Knowledge
Search Technology
Search Engine
Verity (and topic sets)
GUI supporting browsing and selection
Collaborative Topic Set Tool
User Interface
Verity SearchScript, Javascript, HTML, CGI,
CLASSIC
Results (domain specific)
Results (standard format)
42
Trend Semantic Search
43
(No Transcript)
44
Inferences drawn by Information Extraction
45
Infrastructure Core IWBase
Statistics for relevant domain independent
meta-data
29
Inference Engines

56
Axioms
38
Declarative Rules
select

10
Method Rules
6
Derived Rules
12
Languages
select
46
Explaining Answers GUI Explainer
Users can exit the explainer providing feedback
about their satisfiability with explanation(s)
Select action
Users can ask for alternative explanations or
summaries
47
Follow-up PML Abstraction(Techies only)
48
Browsing
  • Present
  • Query
  • Answer
  • Alternate formats (KIF, English, Raw, )
  • Graph Structure (with lens view)
  • Annotations

49
Knowledge Provenance Multiple Sources
Answer
Source
Source
50
IWTrustImproving User
Trust in Answers from the Web
  • Ilya Zaihrayeu
  • ITC-IRST
  • Paulo Pinheiro da Silva
  • Deborah L. McGuinness
  • Stanford University

51
Trusting Answers
  • It may be challenging for users to establish
    their degrees of trust, untrust, mistrust and
    distrust in a web application answer if the
    answer is provided without any kind of
    justification
  • Knowledge Provenance (KP) is a description of
    both the origins of knowledge and the reasoning
    process to produce an answer
  • Users may need KP to establish a degree of trust
    in the answer
  • Which sources were used?
  • Who are the authors of such sources?
  • Which engines were used?
  • What are the assumptions of the engines? Are the
    engines rules sound?
  • KP itself may not be enough for trusting the
    answer
  • I may not know anything about one or more sources
    in the KP
  • I may have no information about the reliability
    of one or more of then engines in the KP

52
Trusting Answers from the Web
  • The overall process of establishing a degree of
    trust in answers from web applications is
    particularly complex since applications may rely
    on
  • Hybrid and distributed processing, e.g., web
    services, the Grid
  • Large number of heterogeneous, distributed
    information sources, e.g., the Web
  • information sources with more variation in their
    reliability, e.g., information extraction
  • Sophisticated information integration methods,
    e.g., SIMS, TSIMMIS
  • The definition of trust is a significant part of
    the process
  • The task of keeping, encoding, sharing and
    gathering KP for partial answers towards the
    generation of the KP for answers is another part
    of the process
  • The use of KP to derive trust values for answers
    is yet another part of the process

53
The Inference Web
  • The Inference Web is an infrastructure supporting
    explanations for answers from the web
  • The Proof Markup Language (PML) is used to encode
    answer justification, i.e., information
    manipulation traces, proofs
  • IWBase is used to annotate PML documents with
    proof-related data, i.e., trust values for
    sources and engines
  • User U1 asks question Q
  • A question answering system returns the set of
    answers A1,A2,,An

PML Documents
IWBase
S1
(A1, t11, t12,...)
IE1
Q(U1)
S2
(A2, t21, t22,...)
...
S3
...
(An, tn1, tn2,...)
IE2
54
Inference Web and KP
  • Inference Web is an infrastructure supporting
    KP for answers derived by multiple methods
  • Information extraction IBM (UIMA), Stanford
    (TAP)
  • Information integration USC ISI
    (Prometheus/Mediator) Rutgers University
    (Prolog/Datalog)
  • Task processing SRI International (SPARK)
  • Theorem proving
  • First-Order Theorem Provers SRI International
    (SNARK) Stanford (JTP) University of Texas,
    Austin (KM)
  • SATisfiability Solvers University of Trento
    (J-SAT)
  • Expert Systems University of Fortaleza (JEOPS)
  • Service composition Stanford, University of
    Toronto, UCSF (SDS)
  • Semantic matching University of Trento
    (S-Match)
  • Debugging ontologies University of Maryland,
    College Park (SWOOP/Pellet)
  • Problem solving University of Fortaleza
    (ExpertCop)

55
The Inference Web Trust (IWTrust)
  • IWTrust extends the Inference Web to support
    trust computation
  • IW TrustNet is a social network of source
    recommenders
  • A trust component implementing an algorithm to
    compute trust values for answers
  • Trust values are used to rank answers and answer
    justifications
  • User U1 trusts U3 to a degree t1-3

PML Documents
IWBase
S1
(A1, t11, t12,...)
IE1
Q(U1)
S2
(A2, t21, t22,...)
...
S3
...
(An, tn1, tn2,...)
IE2
56
Conclusions
  • IWTrust provides infrastructure for building a
    trust graph from a user asking a question to the
    answers for the question
  • Knowledge Provenance is a key element of the
    trust graph and a requirement for trusting
    answers in general
  • Inference Web is a Semantic Web solution for
    Knowledge Provenance
  • iw.stanford.edu
  • Adaptive explanations based on user modeling
  • IWBase registration of a large set of software
    systems
  • Registration of a comprehensive set of primitive
    rules
  • Established library of explanation tactics
  • IWTrust intends to be a solution for the Semantic
    Web trust layer
  • Inference Web is a solution for the Semantic Web
    proof layer
About PowerShow.com