Jaime Teevan

1

• Amazing

The
ReSearch Engine

• Jaime Teevan
• MIT, CSAIL

2
Pick a card, any card.
3

• Abracadabra!

Case 1 Case 2 Case 3 Case 4 Case 5 Case 6
4
Your Card is GONE!
5
People Forget a Lot
6
Change Blindness
http//www.usd.edu/psyc301/ChangeBlindness.htm
7
Change Blindness
http//www.usd.edu/psyc301/ChangeBlindness.htm
8
ReSearch Engine
?
9
Merge Old and New Results
Old
Merged
New
10
We still need magic!
11
Overview

• Memorability study
• Recognition study
• Assumptions
• Implementation issues
• Evaluation issues
• Choose your own adventure

12
Memorability Study

• Participants issued self-selected query
• After an hour, asked to fill out a survey
• 129 people remembered something

13
Data Analysis

• Probability of being remembered
• Anything? of words? of fields?
• Features
• Result features clicked, not clicked, last
  clicked, rank, dwell time, frequency of visit,
  etc.
• Query features query type, query length, of
  search in session, elapsed time, etc.
• Remembered rank v. real rank
• Map remembered rank to real rank

14
Memorability
15
Remembered Results Ranked High
16
Recognition Study

• Same set-up as Memorability Study
• Follow-up survey Results the same?
• Case 1 New results
• Case 2 Random 4 same
• Case 3 Clicked to top
• Case 4 Same results
• Case 5 Intelligent merging
• 165 people completed both steps

19
38
41
66
81
17
Assumptions

• Re-search v. search
• Memorable v. relevant
• Results change v. stay the same
• Hide change v. show change
• Forget v. remember as forgettable
• Merge v. identify old or new

Why?
How to test?
What if Im wrong?
18
Implementation Issues

• Page of cached result may disappear
• Multiple result pages
• Identifying repeat queries
• Exact query may be forgotten
• User identified
• Search sessions are not repeat queries

19
Evaluation Issues

• Various goals to test
• Does a merged list look like the original?
• Does merging make re-finding easier?
• Is search improved overall?
• Lab study
• How to set up re-finding task?
• Timing differences significant enough?
• Longitudinal study What to measure?
• What are good baselines?

20
Choose Your Own Adventure

• Re-search v. search
• Memorable v. relevant
• Results change v. stay the same
• Hide change v. show change
• Forget v. remember as forgettable
• Merge v. identify old or new
• Implementation issues
• Evaluation issues

21
Choose Your Own Adventure

• Re-search v. search
• Memorable v. relevant
• Results change v. stay the same
• Hide change v. show change
• Forget v. remember as forgettable
• Merge v. identify old or new
• Implementation issues
• Evaluation issues

(Done)
22
Hide Change v. Show Change

• Why I think change should be hidden
• Example dynamic menus
• How to prove
• New results better, called the same or worse
• Baseline for testing 2 lists, change explicit
• What if we should show change?
• Memorability suggests changes to highlight
• Other applications where want to hide change

(Done)
23
Memorable v. Relevant

• Why I think memorability is important
• Relevance at a future date is what matters
• Necessary to hide change
• How to prove
• Baseline for lab study with target first
• What if relevance is whats important?
• Mapping between memorable and relevant
• Useful related work on implicit feedback

(Done)
24
Re-search v. Search

• Why I think people repeat searches
• Information seeking literature
• Re-finding consistently reported as a problem
• How to prove
• Study shows prefer to follow known paths
• Search log analysis
• What if people just want to search?
• Memorable results ranked first
• Other domains where list consistency matters

(Done)
25
Merge v. Identify Old and New

• Why I think results should be merged
• Information need not necessarily one or other
• People dont like to do extra work
• How to prove
• Search log analysis
• Look at what people do in longitudinal study
• Lab study timing becomes an issue
• What if people want to identify query type?
• Other applications where merging is useful

(Done)
26
Results Change v. Stay the Same

• Why I think results change
• How search engines work
• Personalization and dynamic content
• How to prove
• Track query results
• What if results dont change?
• Probably will in future applications
• Existing applications where lists change

(Done)
27
Forget v. Remember as Forgettable

• Why I think people forget
• Visual analogy
• How to prove
• Lab study Do people find new information?
• Longitudinal study Ever click on new result?
• What if remember as forgettable?
• Build better model of memorability
• Highlight important changes

(Done)
28
Implementation Issues

• Page of cached result may disappear
• Multiple result pages
• Identifying repeat queries
• User identified
• Search sessions are not repeat queries
• Exact query may be forgotten

(Done)
29
Evaluation Issues

• Various goals to test
• Does a merged list look like the original?
• Does merging make re-finding easier?
• Is search improved overall?
• Lab study
• How to set up re-finding task?
• Timing differences significant enough?
• Longitudinal study What to measure?
• What are good baselines?

(Done)
30

• Thank you!

• Jaime Teevan
• teevan_at_mit.edu

31
Strategies for Finding
Teleporting
Orienteering
32
Why Do People Orienteer?

• Easier than saying what you want
• You know where you are
• You know what you find

• The tools dont work

33
Structural Consistency Important
All must be the same to re-find the information!
New name
34
Absolute Consistency Unnecessary
New name
Focus on search result lists
35
Query Changes

• Most changes are simple
• Capitalization
• Phrasing
• Word ordering
• Word form
• New queries shorter
• What about longer time horizons?
• Recognition v. recall

36
Result List Changes

• Tracked 10 queries on Google for a year
• 1.18 of top 10 disappear each week
• Rate of change likely to increase, e.g.
• Raw personalization
• Relevance feedback
• People forget their queries
• 28 of queries forgotten within an hour

37
Example neon signs