Title: Using Words to Search a Thousand Images Hierarchical Faceted Metadata in Search
1Using Words to Search a Thousand
ImagesHierarchical Faceted Metadata in Search
Browsing
- Marti Hearst
- SIMS, UC Berkeley
- Research funded by
- NSF CAREER Grant IIS-9984741
2Outline
- How do people search for images?
- Current approaches
- Spatial similarity
- Keywords
- Our approach
- Hierarchical Faceted Metadata
- Very careful UI design and testing
- Usability Study
- Conclusions
3How do people want to search and browse images?
- Ethnographic studies of people who use images
intensely find - Find specific objects is easy
- Find images of the Empire State Building
- Browsing is hard, and people want to use rich
descriptors.
4Ethnographic Studies
- Garber Grunes 92
- Art directors, art buyers, stock photo
researchers - Search for appropriate images is iterative
- After specifying and weighting criteria,
searchers view retrieved images, then - Add restrictions
- Change criteria
- Redefine Search
- Concept starts out loosely defined, then becomes
more refined.
5Ethnographic Studies
- Markkula Sormunen 00
- Journalists and newspaper editors
- Choosing photos from a digital archive
- Stressed a need for browsing
- Searching for specific objects is trivial
- Photos need to deal with themes, places, types of
objects, views - Had access to a powerful interface, but it had 40
entry forms and was generally hard to use no one
used it.
6Query Study
- Armitage Enser 97
- Analyzed 1,749 queries submitted to 7 image and
film archives - Classified queries into a 3x4 facet matrix
- Rio Carnivals Geo Location x Kind of Event
- Conclude that users want to search images
according to combinations of topical categories.
7Ethnographic Study
- Ame Elliot 02
- Architects
- Common activities
- Use images for inspiration
- Browsing during early stages of design
- Collage making, sketching, pinning up on walls
- This is different than illustrating powerpoint
- Maintain sketchbooks shoeboxes of images
- Young professionals have 500, older 5k
- No formal organization scheme
- None of 10 architects interviewed about their
image collections used indexes - Do not like to use computers to find images
8Current Approaches to Image Search
- Using Visual Content
- Extract color, texture, shape
- QBIC (Flickner et al. 95)
- Blobworld (Carson et al. 99)
- Body Plans (Forsyth Fleck 00)
- Piction images text (Srihari et al. 91 99)
- Two uses
- Show a clustered similarity space
- Show those images similar to a selected one
- Usability studies
- Rodden et al. a series of studies
- Clusters dont work showing textual labels is
promising.
9Rodden et al., CHI 2001
10Rodden et al., CHI 2001
11Rodden et al., CHI 2001
12Current Approaches to Image Search
- Keyword based
- WebSeek (Smith and Jain 97)
- Commercial image vendors (Corbis, Getty)
- Commercial web image search systems
- Museum web sites
13A Disconnect
- Why are image search systems built so
differently from what people want? - An image is worth a thousand words.
- But the converse has merit too!
14Some Challenges
- Users dont like new search interfaces.
- How to show lots more information without
overwhelming or confusing?
15Our Approach
- Integrate the search seamlessly into the
information architecture. - Use proper HCI methodologies.
- Use faceted metadata
- More flexible than canned hyperlinks
- Less complex than full search
- Help users see where to go next and return to
what happened previously
16Faceted Metadata
17Metadata data about dataFacets orthogonal
categories
18Faceted Metadata Biomedical MeSH (Medical
Subject Headings)www.nlm.nih.org/mesh
19Mesh Facets (one level expanded)
20Questions we are trying to answer
- How many facets are allowable?
- Should facets be mixed and matched?
- How much is too much?
- Should hierarchies be progressively revealed,
tabbed, some combination? - How should free-text search be integrated?
21An Important Trend in Information Architecture
Design
- Generating web pages from databases
- Implications
- Web sites can adapt to user actions
- Web sites can be instrumented
22A Taxonomy of WebSites
high
Complexity of Data
low
low
high
Complexity of Applications
From The (Short) Araneus Guide to Website
development, by Mecca, et al, Proceedings of
WebDB99, http//www-rocq.inria.fr/cluet/WEBDB/pr
ocwebdb99.html
23The Flamenco Interface
- Nine hierarchical facets
- Matrix
- SingleTree
- Chess metaphor
- Opening
- Middle game
- End game
- Tightly Integrated Search
- Expand as well as Refine
- Intermediate pages for large categories
24(No Transcript)
25(No Transcript)
26(No Transcript)
27(No Transcript)
28(No Transcript)
29(No Transcript)
30(No Transcript)
31(No Transcript)
32(No Transcript)
33What is Tricky About This?
- It is easy to do it poorly
- See Yahoo example
- It is hard to be not overwhelming
- Most users prefer simplicity unless complexity
really makes a difference - It is hard to make it flow
- Can it feel like browsing the shelves?
34How NOT to do it
- Yahoo uses faceted metadata poorly in both their
search results and in their top-level directory - They combine region other hierarchical facets
in awkward ways
35Yahoos use of facets
36Yahoos use of facets
37Yahoos use of facets
38Yahoos use of facets
- Where is Berkeley?
- College and University gt Colleges and
Universities gtUnited States gt U gt University of
California gt Campuses gt Berkeley - U.S. States gt California gt Cities gtBerkeley gt
Education gt College and University gt Public gt UC
Berkeley
39Problem with Metadata Previews as Currently Used
- Hand edited, predefined
- Not tailored to task as it develops
- Not personalized
- Often not systematically integrated with search,
or within the information architecture in general
40HCI Methodology
- Identify Target Population
- Needs assessment.
- What to people want how to they work?
- Lo-fi prototyping.
- Produce cheap (throw-away) prototypes
- Get feedback from target population
- Design / Study Round 1.
- Simple interactive version. See if main ideas
work. - Design / Study Round 2
- More thorough interactive version more graphics.
Begin to fine-tune, fix remaining major problems - Design / Study Round 3
- Continue to fine-tune. Introduce more advanced
features.
41Our Project History
- Identify Target Population
- Architects, city planners
- Needs assessment.
- Interviewed architects and conducted contextual
inquiries. - Lo-fi prototyping.
- Showed paper prototype to 3 professional
architects. - Design / Study Round 1.
- Simple interactive version. Users liked metadata
idea. - Design / Study Round 2
- Developed 4 different detailed versions
evaluated with 11 architects results somewhat
positive but many problems identified. Matrix
emerged as a good idea. - Metadata revision.
- Compressed and simplified the metadata
hierarchies
42Our Project History
- Design / Study Round 3.
- New version based on results of Round 2
- Highly positive user response
- Identified new user population/collection
- Students and scholars of art history
- Fine arts images
- Study Round 4
- Compare the metadata system to a strong,
representative baseline
43New Usability Study
- Participants Collection
- 32 Art History Students
- 35,000 images from SF Fine Arts Museum
- Study Design
- Within-subjects
- Each participant sees both interfaces
- Balanced in terms of order and tasks
- Participants assess each interface after use
- Afterwards they compare them directly
- Data recorded in behavior logs, server logs,
paper-surveys one or two experienced testers at
each trial. - Used 9 point Likert scales.
- Session took about 1.5 hours pay was 15/hour
44The Baseline System
- Floogle
- Take the best of the existing keyword-based image
search systems
45Comparison of Common Image Search Systems
46sword
47(No Transcript)
48(No Transcript)
49(No Transcript)
50Evaluation Quandary
- How to assess the success of browsing?
- Timing is usually not a good indicator
- People often spend longer when browsing is going
well. - Not the case for directed search
- Can look for comprehensiveness and correctness
(precision and recall) - But subjective measures seem to be most
important here.
51Hypotheses
- We attempted to design tasks to test the
following hypotheses - Participants will experience greater search
satisfaction, feel greater confidence in the
results, produce higher recall, and encounter
fewer dead ends using FC over Baseline - FC will perceived to be more useful and flexible
than Baseline - Participants will feel more familiar with the
contents of the collection after using FC - Participants will use FC to create multi-faceted
queries
52Four Types of Tasks
- Unstructured (3) Search for images of interest
- Structured Task (11-14) Gather materials for an
art history essay on a given topic, e.g. - Find all woodcuts created in the US
- Choose the decade with the most
- Select one of the artists in this periods and
show all of their woodcuts - Choose a subject depicted in these works and find
another artist who treated the same subject in a
different way. - Structured Task (10) compare related images
- Find images by artists from 2 different countries
that depict conflict between groups. - Unstructured (5) search for images of interest
53Other Points
- Participants were NOT walked through the
interfaces. - The wording of Task 2 reflected the metadata not
the case for Task 3 - Within tasks, queries were not different in
difficulty (tslt1.7, p gt0.05 according to
post-task questions) - Flamenco is and order of magnitude slower than
Floogle on average. - In task 2 users were allowed 3 more minutes in FC
than in Baseline. - Time spent in tasks 2 and 3 were significantly
longer in FC (about 2 min more).
54Results
- Participants felt significantly more confident
they had found all relevant images using FC (Task
2 t(62)2.18, plt.05 Task 3 t(62)2.03, plt.05) - Participants felt significantly more satisfied
with the results - (Task 2 t(62)3.78, plt.001 Task 3 t(62)2.03,
plt.05) - Recall scores
- Task2a In Baseline 57 of participants found all
relevant results, in FC 81 found all. - Task 2b In Baseline 21 found all relevant, in
FC 77 found all.
55Post-Interface Assessments
All significant at plt.05 except simple and
overwhelming
56Perceived Uses of Interfaces
Baseline
FC
57Post-Test Comparison
FC
Baseline
Which Interface Preferable For
Find images of roses Find all works from a given
period Find pictures by 2 artists in same media
Overall Assessment
More useful for your tasks Easiest to use Most
flexible More likely to result in dead
ends Helped you learn more Overall preference
58Facet Usage
- Facets driven largely by task content
- Multiple facets 45 of time in structured tasks
- For unstructured tasks,
- Artists (17)
- Date (15)
- Location (15)
- Others ranged from 5-12
- Multiple facets 19 of time
- From end game, expansion from
- Artists (39)
- Media (29)
- Shapes (19)
59Qualitative Observations
- Baseline
- Simplicity, similarity to Google a plus
- Also noted the usefulness of the category links
- FC
- Starting page well-organized, gave ideas for
what to search for - Query previews were commented on explicitly by 9
participants - Commented on matrix prompting where to go next
- 3 were confused about what the matrix shows
- Generally liked the grouping and organizing
- End game links seemed useful 9 explicitly
remarked positively on the guidance provided
there. - Often get requests to use the system in future
60Study Results Summary
- Overwhelmingly positive results for the faceted
metadata interface. - Somewhat heavy use of multiple facets.
- Strong preference over the current state of the
art. - This result not seen in similarity-based image
search interfaces. - Hypotheses are supported.
61Other Domains
- Applying this to
- Text
- Tobacco Documents Archives
- Medline biomedical texts
- Products/Catalogs
- Dont have a collection would like one
62Implementation
- All open source code
- Mysql database
- Python web server (Webkit)
- Python code
- Lucene search engine (java)
63Summary and Conclusions
64Summary
- We have addressed several interface problems
- How to seamlessly integrate metadata previews
with search - Show search results in metadata context
- Disambiguate search terms
- How to show hierarchical metadata from several
facets - The matrix view
- Show one level of depth in the matrix view
- How to handle large metadata categories
- Use intermediate pages
- How to support expanding as well as refining
65Summary
- Usability studies done on 3 collections
- Recipes 13,000 items
- Architecture Images 40,000 items
- Fine Arts Images 35,000 items
- Conclusions
- Users like and are successful with the dynamic
faceted hierarchical metadata, especially for
browsing tasks - Very positive results, in contrast with studies
on earlier iterations - Note it seems you have to care about the
contents of the collection to like the interface
66Summary
- Validating an approach to web site search
- Use hierarchical faceted metadata dynamically,
integrated with search - Many difficult design decisions
- Iterating and testing was key
- Bits and pieces were there in industry
- The approach is being picked up too
- One is very similar now endeca.com
67Advantages of the Approach
- Supports different search types
- Highly constrained known-item searches
- Open-ended, browsing tasks
- Can easily switch from one mode to the other
midstream - Can both expand and refine
- Allows different people to add content without
breaking things - Can make use of standard technology
68Some Unanswered Questions
- How to integrate with relevance feedback (more
like this)? - Would like to use blobworld-like features
- How to incorporate user preferences and past
behavior? - How to combine facets to reflect tasks?
69The Flamenco Project Team
- Kevin Chen
- Ame Elliott
- Jennifer English
- Kevin Li
- Rashmi Sinha
- Kirsten Swearingen
- Ping Yee
- http//flamenco.berkeley.edu
70Thank you!
For more information