Evaluations in information retrieval - PowerPoint PPT Presentation

Loading...

PPT – Evaluations in information retrieval PowerPoint presentation | free to view - id: 72f4aa-NWViN



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Evaluations in information retrieval

Description:

Evaluations in information retrieval: summary The following gives an overview of approaches that are applied to assess the quality of information retrieval systems ... – PowerPoint PPT presentation

Number of Views:99
Avg rating:3.0/5.0
Slides: 24
Provided by: Nieu5
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Evaluations in information retrieval


1
Evaluations in information retrieval
2
Evaluations in information retrieval summary
  • The following gives an overview of approaches
    that are applied to assess the quality of
  • information retrieval systems, and more
    concretely of search systems
  • the resulting set of records obtained after
    performing a query in an information retrieval
    system
  • Note This should not be confused with assessing
    the quality and value of the content of an
    information source.

3
Evaluations in information retrieval introduction
  • The quality of the results, the outcome of any
    search using any retrieval system depends on many
    components / factors.
  • These components can be evaluated and modified to
    increase the quality of the results more or less
    independently.

4
Evaluations in information retrieval important
factors
  • The information retrieval system ( contents
    system)
  • The user of the retrieval system and the search
    strategy applied to the system

Result of a search
5
Evaluations in information retrieval why? (Part
1)
  • To study the differences in outcome/results when
    a component of a retrieval system is changed,
    such as
  • the user interface
  • the retrieval algorithm
  • addition by the database of uncontrolled, natural
    language keywords versus keywords selected from a
    more rigid, controlled vocabulary

6
Evaluations in information retrieval why? (Part
2)
  • To study the differences in outcome/results when
    a search strategy is changed.
  • To study the differences in outcome/results when
    searches are performed by different groups of
    users, such as
  • children versus adults
  • inexperienced users versus more experienced,
    professional information intermediaries/professio
    nals

7
Evaluations in information retrieval the simple
Boolean model
  • Boolean model items in database items
    selected items not selected
  • Items selected
  • relevant items irrelevant items

Relevant Yes 1 In
Irrelevant No 0 Out
8
Relevant items in a database scheme
Relevant items! (In most cases the small
subset) Irrelevant / NOT relevant items (In
most cases the large subset)
Dependent on the aims, independent of the search
strategy
9
Selecting relevant items by searching a database
scheme
Dependent on the aims and dependent on the
search strategy
Selected and relevant! Selected but not
relevant
Not selected but relevant Not selected and
not relevant
Dependent on the aims, independent of the search
strategy
10
Recall definition and meaning
  • Definition
  • Of selected
    relevant items
  • Recall -------------------------------------
    ------------ 100
  • Total of relevant items
    in database
  • Aim high recall
  • Problem in most practical cases, the total of
    relevant items in a database cannot be measured.

11
Selecting relevant items recall
Selected and relevant! Selected but not
relevant
Not selected but relevant Not selected and
not relevant
12
!? Question !? Task !? Problem !?
How to use of the concept recall, when you do
not know the total number of relevant items in
the database ?
13
Recall how to use the concept of recall
  • Using the same database, variations in recall
    express the effect of search variations
  • Variations in search terms
  • Use of a classification scheme
  • Use of a thesaurus
  • ...

14
!? Question !? Task !? Problem !?
How can you change your search strategy to
increase the recall?
15
Precision definition and meaning
  • Definition Of
    selected relevant items Precision
    --------------------------------------- 100
    Total of selected
    items
  • Aim high precision

16
Selecting relevant items precision
Selected and relevant! Selected but not
relevant
Not selected but relevant Not selected and
not relevant
17
!? Question !? Task !? Problem !?
How can you change your search strategy to
increase the precision?
18
!? Question !? Task !? Problem !?
When you change your search strategy to increase
the precision, which consequence do you expect
for the recall, in most cases?
19
Relation between recall and precision of searches
Ideal Impossible to reach in most systems
100 Recall 0
0 Precision 100
Search (results)
20
!? Question !? Task !? Problem !?
Indicate on the figure that a user improves a
search.
21
!? Question !? Task !? Problem !?
Indicate on the figure that a database producer
and / or the retrieval system improves the
retrieval quality.
22
!? Question !? Task !? Problem !?
Indicate the relation between the recall and
precision in a classical information retrieval
system in the form of a figure. Indicate in
that figure a good and a bad search.
23
Recall and precision should be considered
together
  • Examples
  • Increase in retrieved number of relevant items
    may be accompanied by an impractical decrease in
    precision.
  • Precision of a search close to 100 may NOT be
    ideal, because the recall of the search may be
    too low. Make search / query broader to increase
    recall !
  • Poor (low) precision is more noticeable than bad
    (low) recall.

24
Evaluation in the case of systems offering
relevance ranking
  • Many modern information retrieval systems offer
    output with relevance ranking.
  • This is more complicated than simple Boolean
    retrieval, and the simple concepts of recall and
    precision cannot be applied.
  • To compare retrieval systems or search
    strategies, decide to consider for comparison a
    particular number of items ranked highest in each
    output. This brings us to for instance first-20
    precision.

25
!? Question !? Task !? Problem !?
Give examples of retrieval systems that offer
relevance ranking.
About PowerShow.com