The Infocious Web Search Engine: Improving Web Searching Through Linguistic Analysis

1
The Infocious Web Search Engine Improving Web
Searching Through Linguistic Analysis

• Alexandros Ntoulas1,2 Gerald Chao1 Junghoo Cho2
• 1 Infocious Inc.ntoulas, gerald_at_infocious.co
  m
• 2 University of California Los Angeles
• ntoulas, cho_at_cs.ucla.edu

2
Motivation

• Current Web search engines identify relevant
  pages based on keyword matching
• Example jaguar

Jaguar Cars Official worldwide web site of
Jaguar Cars. www.jaguar.com/
3
Motivation

• Is keyword matching enough ?
• Natural languages are inherently ambiguous
• Example jaguar
• The car brand ?
• Apple Mac OS X 10.2 ?
• The animal ?
• Chemical software

4
The Infocious Web Search Engine

• Uses Language Analysis techniques to
• Resolve ambiguities inside Web pages
• Rank the Web pages based on the coherence
  (quality) of the text
• Help users organize the results in intuitive ways
  through categorization
• Provide suggestions for query refinement

5
What is different about Infocious ?

• Search Engines today do not apply Language
  Analysis to the level Infocious does
• It is not simply a matter of applying existing
  algorithms need optimizations for Web scale
• Features made possible only through language
  analysis
• Makes Language Analysis features intuitive (yet
  powerful) for the user

6
Architecture


7
Architecture

• Crawler
• Follows links to discover Web pages
• Refreshes changed pages using sampling VLDB02
• Can download pages from the Hidden Web JCDL05

8
Architecture

• Linguistic Processing
• Resolves language ambiguities COLING02
• Annotates Web pages
• Extracts concepts
• Extracts named entities
• Operates at crawl speed

9
Linguistic Processing Disambiguation

• Part-of-speech (POS) tagging
• Example house plants
• Done probabilistically
• Given sentence S, set of tags T find
• Tbest(S) arg maxT P(T S)

... most house plants are hybrids of plant
species ... garden built to house our most
valuable plants ...
Adj Noun Noun Verb Noun Prep Noun
Noun Noun VerbD Inf Verb PronP Adv
Adj Noun
10
Linguistic Processing Disambiguation

• POS information stored inside the index
• User can manually specify POS at query time (or
  click on examples)

Query Nhouse Nplants GreenPatio.Com Tips
for buying house plants.Why keep natural indoor
plants.... Tips for buying house plants. Care for
indoor plants....www.greenpatio.com/tips.shtml L
ow Light Plants for the HouseIs a common name
for plants in the species Dieffenbachia.... As
with most house plants www.plantsgalore.com/arti
cles/houseplants/houseplants-low-light-plantfacts.
htm
11
Linguistic Processing Disambiguation

• POS information stored inside the index
• User can manually specify POS at query time (or
  click on examples)

Query Vhouse Nplants Over Wintering Bonsai
One method is to build a cold frame to house
your plants in the winter. ...www.evergreengarden
works.com/overwint.htm Keeping Your Sunroom Cozy
And if you want to house a hot tub or plants,
think about enclosing the doityourself.com/sunro
om/sunroomcozy.htm
12
Linguistic Processing Disambiguation

• Word-sense disambiguation
• Previous Example jaguar
• Approach through Web page categorization
• Use the categories of DMOZ (600,000)
• Given set of categories C and a page d
• Find maxc ? C P(cd)
• In Infocious a page may belong to multiple
  categories

13
Categorization
Computers Recreation/Autos
Computers

• The category of a result is highlighted
  onMouseOver()
• Allow users to restrict search within a category
• jaguar catComputers
• Can also be done by clicking on a category

• Jaguar CarsOfficial worldwide site of jaguar
  cars
• www.jaguar.com/
• Apple Mac OS X
• The Apple Mac OS Product page
• www.apple.com/macosx/

Apple Mac OS X
14
Linguistic Processing Concept Extraction

• More accurate phrase identification
• Identify concepts through a set of rules
  (pre-specified or automatically learned)
• Example VerbPhrase-PrepPhrase-NounPhras
• lightly tossed with salad dressing
• tossed with oil and vinegar dressing
• tossed immediately with blue-cheese dressing
• Reduced to Concept tossed with dressing

In the profession of cooking oil is an important
ingredient
15
Answering a query

• Default is AND-semantics
• Query disambiguation (e.g. in query train a pet
  Infocious knows train has to be a verb)
• Ranking takes into account a variety of factors
• Presence of keywords, Proximity
• Title, URL, formatting, font size, coloring etc.
• Popularity of a page measured by in/out links
• TextQuality

16
Architecture

• TextQuality
• Summarize probabilities from Linguistic
  Processing into one metric
• Promote coherent text
• Demote incoherent text

17
TextQuality (disabled)

• Promotes well-written pages (preferable from the
  user perspective)

Britney Spears Pictures britney spears pictures
picture of britney spears, hot pictures of
britney spears britney-spears-pictures.hotyoungs
tars.com/nude/ Hot Britney Spears Pics - hot
britney spears pics,...britney spears, new hot
pics of britney spears,...hot-britney-spears-pics
.hotyoungstars.com/nude/ Britney Spears Photos
britney spears photos spears, britney spears
nude photos, nude photos of britney-spears-photo
s.hotyoungstars.com/nude/ TextQuality DISABLED
18
TextQuality (enabled)

• Promotes well-written pages (preferable from the
  user perspective)

Is Britney Spears over the edge? Is Britney
Spears over the edge? Britney Spears is a
singer azwestern.edu/modern_lang/esl/cjones/mag/
spring2004/britney.htm IMPERSONATORS BRITNEY
SPEARS Is Proud to Present! Contact Gary
Shortall Back www.impersonators.com/brittany/bri
t.html Britney Spears Coke HabitBritney Spears
Coke Habit Destroys Herwww.emptyv.org/britney_sp
ears.htm TextQuality ENABLED
19
Other Language Analysis-Enhanced Features

• Key phrases Present a list of the salient
  concepts within the results
• Related topics Concepts related to the present
  query
• Hone your search Suggestion of more specific
  queries
• Spell Checking
• Personalization I like Sports but not Politics

20
Evaluation of Categorization

• Using Naïve Bayes classifiers for illustration
  Language Analysis improves accuracy
• Infocious actually employs an improved
  classification technique (76 accuracy)
• We used four different flavors of NB on 100,000
  Web pages
• C1 Words
• C2 Words POS tags
• C3 Words extracted concepts
• C4 Words POS extracted concepts

21
Evaluation of Categorization
3 accurary increase 8 error reduction

• C1 Words only
• C2 Words POS tags

• C3 Words extracted concepts
• C4 Words POS extracted concepts

22
User Interface
23
Conclusion

• Infocious uses language analysis to improve Web
  search
• Resolves language ambiguities
• Incorporates text coherence in the ranking
• Provides query suggestions and refinements
• Organizes information intuitively through
  categorization

24
Related Work

• Web Search Engines
• Google, Yahoo!, MSNSearch, Ask/Teoma, Altavista,
  Looksmart, Vivisimo,
• Enterprise Search
• Autonomy, Inquira, Inxight, iPhrase,
• Answer Engines
• START_at_MIT, BrainBoost,

25
Ongoing work

• Increase index size (currently 1 billion pages)
  through surface hidden Web-crawls
• Apply our Language Analysis algorithms to
  additional languages
• Leverage our Language-annotated repository for
  additional features (e.g. summarization, machine
  translation,)
• Investigate how to use Language Analysis to
  improve relevance in advertisements

26
Thank you !
You can check out our Search Engine
atwww.infocious.com