Enduser Interaction with Thesauri An Evaluation of Cognitive Overlap in Search Term Selection

1
End-user Interaction with ThesauriAn Evaluation
of Cognitive Overlap in Search Term Selection

• Ali Shiri Crawford Revie

University of Alberta Edmonton,
Canada University of Strathclyde Glasgow,
Scotland
2
Outline

• Motivation
• Research Method
• Results
• Conclusions / Implications

3
Motivation

• Increasing number of commercial search interfaces
  have thesaurus-enhanced features
• Previous research focused on professional
  searchers ? End-user revolution
• Little research on how end-users interact with
  thesaurus-enhanced search interfaces
• ?
• Study end-user interaction with thesauri for
  search term selection and query expansion

4
Research questions

• How well are users initial search terms matched
  within the thesaurus?
• In what ways were additional terms supplied by
  the thesaurus useful?
• What caused users to select certain additional
  terms from the thesaurus?
• What were users perceptions of overall closeness
  of matched terms and their general level of
  satisfaction with the thesaurus expansion?

5
Research Method

• System CAB Abstracts / Thesaurus using OVID
  interface on Web
• Participants 30 veterinary workers
• Search tasks real needs
• Data gathering techniques
• Experimental procedure

6
Research Method

• System CAB Abstracts / Thesaurus using OVID
  interface on Web
• Participants 30 veterinary workers
• Search tasks real needs
• Data gathering techniques
• Experimental procedure

7
OVID - main search page
8
OVID thesaurus interface (Hierarchical mapping)
9
Research Method

• System CAB Abstract / Thesaurus using OVID
  interface on Web
• Participants 30 veterinary workers
• Search tasks real needs
• Data gathering techniques
• Experimental procedure

10
Participants
Veterinary workers grouped by gender and status
11
Research Method

• System CAB Abstract / Thesaurus using OVID
  interface on Web
• Participants 30 veterinary workers
• Search tasks real needs
• Data gathering techniques
• Experimental procedure

12
Search tasks

• Two main approaches to search task
• Assigned search topics
• Search topics elicited from users
• Search tasks in this study
• topics were all based on genuine information
  needs defined by users themselves

13
Topic types and users intention
14
Research Method

• System CAB Abstract / Thesaurus using OVID
  interface on Web
• Participants 30 veterinary workers
• Search tasks real needs
• Data gathering techniques
• Experimental procedure

15
Data gathering techniques

• Pre-search questionnaire user background and
  search topics
• Screen capturing software Lotus ScreenCam
• Post-search questionnaire search term and search
  results satisfaction
• Post-session interview
• thesaurus usability
• interface usability

16
Research Method

• System CAB Abstract / Thesaurus using OVID
  interface on Web
• Participants 30 veterinary workers
• Search tasks real needs
• Data gathering techniques
• Experimental procedure

17
Experimental procedure

• Brief description of the research objectives and
  the stages of the experiment
• Short written tutorial about the system
• Carry out a practice search
• Perform searches on each of three specified
  topics fill out a post-search questionnaire
• Post-session interview

18
Key variables

• Search moves
• Terms (initial, browsed and selected)
• Subject knowledge and topic familiarity
• Prior topic search experience
• User search behaviour
• Users perceptions

19
Results

• Term matching
• User perceptions on additional terms
• usefulness of terms
• awareness of new vocabulary
• reasons for selection
• Closeness of additional thesaurus terms and
  overall level of satisfaction

20
Results 1 - Term Matching
21
Term Matching Illustrated
22
Results 2 - Additional Terms

• These were found in 74 of the 90 searches
• Of the 74, users judged the additional terms to
  have been useful in 88 of cases
• In 37 of the searches users noted that terms of
  which they were not previously aware had been
  suggested (62 of these, PGs)

23
Results 2 - (cont.)

• In the 73 of searches where users chose to
  incorporate one or more additional terms they
  were ask to classify why
• narrowing down - 47
• broadening - 22
• term relatedness - 18
• others - 13

24
Analysis of selection patterns

• Relationships between users initial query terms
  and the terms they selected from the thesaurus
  for query expansion were
• Narrower terms (38)
• Related terms (36)
• Broader terms (13)
• Synonymous terms (13)
• Academic staff chose narrower and synonymous
  terms more than did postgraduates who tended to
  select related and broader terms.

25
Results 3 - closeness / satisfaction

• When asked to rate the closeness of suggested
  thesaurus terms to their original terms, 94
  stated very or fairly close
• Users selected around 15 of the terms they
  browsed in the thesaurus
• Only 45 gave an overall very satisfied rating,
  with another 45 fairly satisfied and 10 not
  satisfied
• Those evaluating thesaurus terms as being useful
  selected on average around two more search terms
  than did those who did not find the terms useful

26
Conclusions

• End-users are able to interact with and make use
  of thesaurus-enhanced search interfaces
• CAB provides a pretty comprehensive thesaurus
  given our user profile - 82 found an exact or
  partial match (c.f. Fidel91 who stated that a
  good thesaurus will give 80 match)
• Users have a reasonable understanding of the
  types of expansion (cognitive and structural
  models show some degree of concordance)
• Users need a very close level of match to be
  highly satisfied (aka, users are hard to
  please!)

27
Implications

• Showing hierarchical structures would appear to
  be a useful mechanism for query expansion
• Users with varying levels of domain knowledge
  make use of thesauri in different ways
• User characteristics and prior search behaviour
  are elements that can be used within interface
  design in both directions
• design of personalised interaction
• use interactions to categorise new users

28
Thank you
Ali Shiri Crawford Revie
29
(No Transcript)
30
Search moves

• Cognitive moves users perform some kind of
  conceptual analysis of terms and/or documents
• Physical moves those associated with the use of
  system features

31
Cognitive moves
 
Mean number of cognitive moves per search and
per user
32
Physical moves
 
Mean number of physical moves per search and per
user
33
Results

• Search moves - effect of subject knowledge
• Terms browsed and selected
• Effect of subject knowledge
• Effect of topic familiarity
• Usefulness of additional thesaurus terms - effect
  of prior search experience

34
Results 1 - Moves

• Subject knowledge, as measured by user status
  (academic staff or PG student) had NO effect on
  either number of cognitive or physical moves
  executed

35
Results 2 - Terms browsed and selected

• Academic staff browsed 11 more terms than did PG
  students - ultimately no difference in the number
  of terms selected
• Those familiar with topic also browsed
  significantly more terms - again no difference in
  the number of terms selected
• Process vs Outcome

36
Topic familiarity prior topic search experience
 
37
Topic familiarity

• Those who were very familiar with the topic
  browsed significantly more thesaurus terms (18
  terms more)
• Those with no prior topic search experience found
  more additional and useful thesaurus terms
• One-way ANOVA indicated that there was no
  significant difference in either the number of
  physical (P0.07) or cognitive (P0.3) moves.
  However topics identified as being moderately or
  very familiar were associated with around five
  more physical and two more cognitive moves than
  those topics identified as unfamiliar

38
Results 3 - Prior search experience

• Those with no prior search experience found more
  additional terms
• They also judged the terms to be more useful that
  did those with prior topic search experience

39
Usefulness of thesaurus terms

• Users selected around 15 of the terms they
  browsed in the thesaurus
• In 50 of the cases where users found additional
  thesaurus term they had not been aware of those
  terms at the beginning of the search
• Users who evaluated the thesaurus terms as being
  useful selected on average around two more search
  terms than did those who did not find the terms
  useful

40
Conclusions

• End-users are able to interact with and make use
  of thesaurus-enhanced search interfaces
• Thesauri are capable of providing end-users with
  additional, useful terms for query formulation
  and expansion
• Variables such as topic familiarity and topic
  complexity have effects on search behaviour
• Users with varying levels of domain knowledge
  make use of thesauri in different ways

41
Implications

• Topic familiarity and complexity are useful
  elements to consider when studying end-users
  search behaviour
• These aspects and prior search behaviour are
  elements that can be used within interface design
  in both directions
• design of personalised interaction
• use interactions to categorise new users