MetaSearch Engine

1
Lecture 9 Rank Aggregation in MetaSearch

• MetaSearch Engine
• Social Choice Rules
• Rank Aggregation

2
Requirement Rank aggregation protocols

• Majority rule
• Bordas rule
• Optimal Spearmans footrule distance solution
• Kemeny optimal aggregation.
• Kendall tau distance

3
Choices of Search Engines

• Many search engines exist to compete for users
• The results are not necessarily the same
• Different users prefer different search engines
• Search results may, in the future, be biased
  towards paid advertisements.

4
GOOGLE City University

• City University London - Top ranking UK
  university for employment www.city.ac.uk/
• City University Home www.cityu.edu
• City University of Hong Kong www.cityu.edu.hk/
• Welcome to Dublin City University - DCU
  www.dcu.ie/
• Oklahoma City University www.okcu.edu/ -
• The University of Iowa www.uiowa.edu/
• UMKC The University of Missouri-Kansas City
  (UMKC) www.umkc.edu
• The City College of New York www.ccny.cuny.edu/
• New Jersey City University NJCU www.njcu.edu/

5
YAHOO

• City University London - Top ranking UK
  university for employment www.city.ac.uk
• City University City University - www.cityu.edu
  
• Dublin City University www.dcu.ie
• City University of Hong Kong www.cityu.edu.hk
• UMKC www.umkc.edu
• New Jersey City University www.njcu.edu
• Oklahoma City University www.okcu.edu
• Department of Music www.city.ac.uk/music
• Oakland City University www.oak.edu
• City University of New York www.cuny.edu

6
AllTheWeb

• Apartment Listings in University
  City  (sponsored)St. Louis, Missouri area
  apartment listings at Apartments.com. Free
  nationwide apartment search with visual rental
  listings online.http//www.apartments.com
• City University - Washington  (sponsored)Contact
  information and resources such as yellow page
  information, phone number, address, maps and
  directions as provided by QwestDex.http//service
  .bfast.com
• City University Apartments - Rent.com  (sponsored)
  Rent.com has millions of free apartment listings
  nationwide. Get 100 when you sign a lease near
  your school - it's easy.http//www.rent.com
• City University London - the University for
  business and the professions... Contact Us
  About City University Maps Directions A-Z
  Index Site Map Help ... Description
  Official site with information about courses,
  research, schools, and departments. Includes
  details of news and events.more hits from 
  http//www.city.ac.uk/  -  13 KB
• Flash UpgradeDescription A University which
  believes in forward-thinking business and
  leadership skills, exposing students to the
  latest technology in all courses of
  study.http//www.cityu.edu/  -  27 KB
• Welcome to Dublin City University...
  KnowledgeWorks, wins DCU Mallin-invent award Full
  text you can go anywhere in the world from
  here... Dublin City University, Dublin 9,
  Ireland. Tel. [phone], Fax. 353 (0)
  1 836 0830. Page updated 02/09/03 legal search
  ... Description Information on facilities,
  services, degree courses, research, the campus,
  student life, the library and DCU news.

7
MSN Search

• City University of New York Details the CUNY
  system's campuses and admissions requirements.
  Eye photos of notable graduates such as actor
  Judd Hirsch.www.cuny.edu
• City University Offers prospectus, admissions,
  academic calendar, courses and degrees, and
  news.www.cityuniversity.net
• City University Private, nonprofit institution
  located in Belleview, Washington, offers higher
  education opportunities to working professionals
  and lifelong learners.www.cityu.edu
• Valley City University Admissions Site covers
  financial aid and tuition and offers a virtual
  campus tour. Site also provides access to an
  online application.www.vcsu.nodak.edu/admissions
• Oklahoma City University The Oklahoma City
  University site contains information about
  admissions, academic departments and student
  life. The site also provides library resources,
  faculty information and news.frodo.okcu.edu
• Dublin City University, Dublin - TripAdvisor
  Browse reviews of the Dublin City University in
  Dublin, County Dublin, Ireland. Check consumer
  reviews, articles and opinions about the Dublin
  City University.www.tripadvisor.com/Hotel_Review-
  g186605-d209093-Reviews-Dublin_City_U...

8
AOL Search

• City University London - the University for
  business and the ... - ... The University for
  business and the professions. Contact Us About
  City University Maps Directions, AZ Index
  Site Map Help. Prospective Students. ...
  http//www.city.ac.uk/
• City University Home - A University which
  believes in forward-thinking business and
  leadership skills, exposing students to the
  latest technology in all courses of study.
  http//www.cityu.edu/
• Welcome to Dublin City University - ... here...
  Dublin City University, Dublin 9, Ireland. Tel.
  353 (0) 1 700 5000, Fax. [phone].
  Page updated 02/09/03 legal. ...
  http//www.dcu.ie/
• http//www.cuny.edu/ - The University's Main
  Website.http//www.cuny.edu/
• City University of Hong Kong - AD Working Group
  lends staff, students an ear. Several issues
  still evoked strong emotions as staff members and
  students voiced their ... http//www.cityu.edu.hk
  /
• Welcome to Oklahoma City University - ... At
  Oklahoma City University, our students come
  first. Our ... Oklahoma City University offers a
  quality, values-centered education. United ...
  http//www.okcu.edu/

9
Ask Jeeves

• City University of Hong Kong Formerly the City
  Polytechnic of Hong Kong. Includes information on
  university, links to learning resources, research
  and departmental information.....Fromwww.cityu.ed
  u.hk/
• City University Text Version. The University for
  business and the professions ... Contact Us
  About City University Maps Directions A-Z
  Index Site Map Help...Fromwww.city.ac.uk/
• New Jersey City University Official site includes
  graduate and undergraduate information, news,
  events, online learning, calendars, and
  athletics.Fromwww.njcu.edu/
• Elizabeth City State University Admission
  Application Introduction 2004-2005 APPLICATION
  FOR ADMISSION. Welcome to the Elizabeth City
  State University Online Application. New
  Freshmen...Fromwww.ncmentor.org/applications/unc/
  apply/elizabeth_city_state_univ...
• City University A University which believes in
  forward-thinking business and leadership skills,
  exposing students to the latest technology in all
  courses of study.Fromwww.cityu.edu/
• The City College of New York City College of New
  York (CUNY) A senior college of The City
  University of New York. Located in Manhattan.
  138th Street at Convent Avenue. A Harlem
  masterpiece in the neo-Gothic...Fromwww.ccny.cuny
  .edu/

10
HotBot

• 1. City University of New YorkDetails the CUNY
  system's campuses and admissions requirements.
  Eye photos of notable graduates such as actor
  Judd Hirsch.www.cuny.edu/ - August 10, 2003 - 25
  KB
• 2. City UniversityPrivate, nonprofit institution
  located in Belleview, Washington, offers higher
  education opportunities to working professionals
  and lifelong learners.www.cityu.edu/ - August
  23, 2003 - 27 KB
• 3. New Jersey City UniversityExplore the course
  requirements for this liberal arts institution.
  Link to admissions and financial aid information.
  ... New Jersey City University. 2039 Kennedy
  Boulevard Jersey City, New Jersey 07305-1597 ...
  www.njcu.edu/ - November 1, 2003 - 20 KB
• 4. City University London - the University for
  business and the...... Contact Us About City
  University Maps Directions. A-Z Index Site
  Map Help ... www.city.ac.uk/ - September 28,
  2003 - 14 KB
• 5. Oklahoma City UniversityTake a tour of the
  campus, and explore a roster of academic
  programs. ... At Oklahoma City University, our
  students come first. ... www.okcu.edu/ - October
  26, 2003 - 20 KB
• 6. University of Missouri, Kansas CityUniversity
  based in Kansas City, Missouri, presents and
  overview of its academic programs, and offers
  campus news. ... UNIVERSITY OF MISSOURI-KANSAS
  CITY. Bulletin. Winter 2004 Fee Update, more...
  ... www.umkc.edu/ - October 23, 2003 - 16 KB

11
Lycos

• 1. City University London - the University for
  business and the... Contact Us About City
  University Maps Directions A-Z Index Site
  Map Help More results from www.city.ac.uk  
  September 16, 2003 - 14 KB
• 2. Flash UpgradeA University which believes in
  forward-thinking business and leadership skills,
  exposing students to the latest technology in all
  courses of study.www.cityu.edu   November 1,
  2003 - 27 KB
• 3. The City University of New YorkThe
  University's Main Website.www.cuny.edu  
  December 31, 1969 - 121 B
• 4. Welcome to Dublin City University
  KnowledgeWorks, wins DCU Mallin-invent award Full
  text you can go anywhere in the world from
  here... Dublin City University , Dublin 9,
  Ireland. Tel. [phone], Fax. 353 (0)
  1 836...More results from www.dcu.ie   October
  29, 2003 - 5 KB
• 5. City University of Hong KongFormerly the City
  Polytechnic of Hong Kong. Includes information on
  university, links to learning resources, research
  and departmental information and student
  information.More results from www.cityu.edu.hk
    October 29, 2003 - 26 KB
• 6. HCU HomePage -English- contact us For any
  suggestion and requests to this web site,
  www-admin_at_hiroshima-cu.ac.jp Feel free to link
  this web site. Hiroshima City University More
  results from www.hiroshima-cu.ac.jp   May 11,
  2003 - 26 KB

12
Teoma

• City University of Hong Kong Formerly the City
  Polytechnic of Hong Kong. Includes information on
  university, links to learning resources, research
  and departmental information...www.cityu.edu.hk/
  Related PagesMore Results from
  www.cityu.edu.hk
• City University Text Version. The University for
  business and the professions ... Contact Us
  About City University Maps Directions A-Z
  Index Site Map Help...www.city.ac.uk/More
  Results from www.city.ac.uk
• New Jersey City University Official site includes
  graduate and undergraduate information, news,
  events, online learning, calendars, and
  athletics.www.njcu.edu/Related PagesMore
  Results from www.njcu.edu
• Elizabeth City State University Admission
  Application Introduction 2004-2005 APPLICATION
  FOR ADMISSION. Welcome to the Elizabeth City
  State University Online Application. New
  Freshmen...www.ncmentor.org/applications/unc/apply
  /el...
• City University A University which believes in
  forward-thinking business and leadership skills,
  exposing students to the latest technology in all
  courses of study.www.cityu.edu/
• The City College of New York City College of New
  York (CUNY) A senior college of The City
  University of New York. Located in Manhattan.
  138th Street at Convent Avenue. A Harlem
  masterpiece in the neo-Gothic...www.ccny.cuny.edu/
  More Results from www.ccny.cuny.edu

13
MetaSearch Engine

• Metasearch Engines are designed to increase the
  coverage of web by forwarding users queries to
  multiple search engines
• Users requests are sent to multiple search
  engines such as AlltheWeb, Google, MSN.
• Then the results from the individual search
  engine are combined into a single result set to
  present to users.

14
Different Forms of MetaSearch

• Submit different representations of the same
  query to the same search engine, then combine the
  results.
• Submit the same query to several search engine
  adopting different information retrieval models,
  then combine the results.

15
Issues

• How to combine the results retrieved by different
  source search engines is crucial for the success
  of a metasearch engine.
• And this is the problem that social choice theory
  has been trying to answer.

16
Search Engine Watch

• Interesting metat search engines are listed at
• http//www.searchenginewatch.com/links/article.php
  /2156241

17
Nielsen NetRatings Search Engine Ratings Source
http//searchenginewatch.com/reports/article.php/2
156451

• Google Shows searches at any Google-branded web
  site such as Google.com or Google Images. 
• Yahoo Shows searches at any Yahoo-branded web
  site such as Yahoo.com or Yahoo Local. Does NOT
  include searches at the Yahoo-owned sites of
  AltaVista, AllTheWeb and Overture. 
• MSN Shows searches at any MSN-branded web site
  such as MSN Search. 
• AOL Shows searches at any AOL-branded web site
  such as AOL Search. Does NOT include searches at
  AOL-owned Netscape Search (this is on the chart
  separately) or probably other AOL-owned sites. 
• Ask Shows searches at Ask Jeeves but not Ask
  Jeeves-owned Teoma. Does NOT show searches at the
  Ask Jeeves-owned sites of MyWay.com, iWon and My
  Search. These are shown on the chart
  separately. 
• Earthlink Shows searches at Earthlink. 
• Other Shows searches that occur at other search
  sites not named on the chart. Any site not listed
  on the chart can be assumed to have a share that
  is less than the smallest named site (IE, My
  Search is the lowest named search site with a
  0.7 share. Other sites will have shares less
  than this).

18
Award WinnersSource http//searchenginewatch.c
om/awards/article.php/3494141

• Outstanding Search ServiceWinner YahooBest
  Meta Search EngineWinner Jux2Best News Search
  EngineWinner Google NewsBest Blog/Feed Search
  EngineWinner BloglinesBest Image Search
  EngineWinner Yahoo ImagesBest US Shopping
  Search EngineWinner Google's FroogleMost
  SEO/Webmaster Friendly Search ProviderWinner
  GoogleBest Search Ads ProviderWinners Google
  AdWordsBest Search Feature Honorable Mention
  Clusty's Clustering, MSN's Search Builder, Ask
  Jeeves Binoculars Site Preview, Surfwax
  LookAhead, Pinpoint Shopping Suggestions, A9
  Search History, My Jeeves, My Yahoo Search,
  Google Desktop Web History Feature, AOL
  Snapshots, Ask Jeeves Smart Search, Google OneBox
  Results, Yahoo Shortcuts
• Best Specialty Search EngineHonorable Mention
  Google Local, Yahoo Local, Google Scholar,
  Scirus, Citeseer, Librarians' Index To The
  Internet

19
Best Meta Search Engine Source
www.searchenginewatch.com

• Winner Jux2
• Just 2 percent of the 490 votes cast in this
  category went to Jux2, but we thought it deserved
  to win over the more popular choices. It provides
  a clean, simple way to see all the results from
  the most important major search engines. You can
  easily view what the big three crawlers of
  Google, Yahoo and Ask Jeeves agree on -- plus the
  results that are specific to each search engine,
  as well. There's no better way to understand the
  unique voices that each search engine speaks
  with.
• Jux2 takes its name from initially juxtaposing
  results from Google and Yahoo. Ask Jeeves was
  later added, and we hope that MSN will be
  included down the line. We also wouldn't mind
  seeing paid listings being added to the service,
  perhaps in a right-hand column display. Paid
  listings can often be relevant to searchers as
  well, so we'd like to have them.

20
Best Meta Search Engine Second Place Dogpile
Source www.searchenginewatch.com

• Dogpile was the clear winner in the popular vote,
  gaining 37 percent of the votes. Add in votes for
  sister-site MetaCrawler, which essentially is
  Dogpile in all but name, and the share goes up to
  52 percent.
• However, we thought a winning meta search engine
  should make it easy to pick and choose exactly
  which search engines you want to search against.
  At Jux2, this can easily be done right from the
  home page or saved as a permanent choice. At
  Dogpile, it's not an option at all, whether you
  use the advanced search page or the preferences
  option.
• In addition, we also dislike the fact that
  sponsored listings are mixed in among the regular
  results. The paid listings are all clearly
  labeled as "Sponsored By," so there's no
  confusion on that front. But we'd like to see
  them physically segregated from the editorial
  results. For these reasons, Dogpile didn't make
  the winning spot this year.

21
Best Meta Search Engine Honorable Mention
Clusty Mamma Source www.searchenginewatch.com

• Clusty was a newcomer to meta search in 2004,
  though it comes from an old hand in the area,
  Vivisimo. Vivisimo has long run meta search on
  its site to demo its clustering technology.
  Clusty was designed to move past being a
  demonstration and into a site that can better
  help searchers every day.
• We like the clustering options that Clusty takes
  its name from, as well as the clear distinction
  between sponsored and editorial results. We also
  love that you can permanently pick and choose
  which web search engines you wish to search
  against. Unfortunately, the collection doesn't
  include either Google or Yahoo, something we
  think would be essential to a winner in this
  category.
• In terms of popular voting, Clusty drew 14
  percent of the votes following Dogpile and
  MetaCrawler, a very strong showing for a new
  service.
• Mamma lacks the clustering that Clusty offers,
  but it does have all the other things we like
  sponsored listings segregated from editorial
  results and the ability to permanently pick and
  choose search engines. But like Clusty, it also
  lacks access to results from either Google or
  Yahoo. In the popular voting, it came after
  Clusty, picking up 9 percent of those cast. We
  thought that was strong enough to warrant an
  honorable mention, as well.

22
Social Choice Theory

• Studies on protocols that help a group of people
  make collective decisions, such as vote.

23
A Fundamental problem

• Given a collection of agents (voters)
• with preferences over different alternatives
  (allocations, outcomes),
• How should society evaluate these alternatives
  and make a decision for all?
• that may be for the will of some voters but
  against that of others.

24
Applications

• Voters elect president from several candidates.
• National polls for economic or political policy
  of the government
• The procedure or rule of election
• The rank of metasearch engine obtained from those
  of search engines

25
Group Descisions

• How do we make decisions
• Flip a coin?
• Dictatorship?
• Democracy (Majority rule)?
• in a group with
• Heterogeneity of individual preference

26
Group Decision Rules

• Majority rule ,
• Condorcet paradox (voting cycle)
• Borda rule

27
Mathematical model

• A set of voters Vv1,v2,v3,,Vn
• A set of alternatives or outcomes
  Ss1,s2,s3,Sm, with Sm and
• A set of preference relation PR1,R2,R3Rn,
  called a preference profile,
• the preference relation Ri for each voter i is a
  permutation (order) of elements in S.

28
Example 1 Majority Rule

• 3 rational people have rational preferences over
  2 alternatives x,y
• Person
• 1 2 3
• 1st X Y X
  1 XgtY
• Pref. i.e.Person 2
  YgtX
• 2nd Y X Y
  3 XgtY
• How to Aggregate their preferences? How to choose?

29

• Using majority rule.
• Since more than ½ people (two out of three)
  prefer x to y.
• Then the group prefers x to y

30
Example 2 Condorcet Paradox

• 3 rational people have rational preferences over
  3 alternatives x,y,z
• Person
• 1 2 3
• 1st X Y Z
  1 XgtYgtZ
• Pref. 2nd Y Z X i.e. Person 2
  YgtZgtX
• 3rd Z X Y
  3 ZgtXgtY

31
Binary/paired Comparison With Majority rule

• Person
• 1 2 3
• 1st X Y Z 1
  XgtY
• Pref. 2nd Y Z X for (x,y) 2
  YgtX? XgtY
• 3rd Z X Y 3
  XgtY
• Similarly, for (Y,Z) we can get YgtZ for (Z,X) we
  can get ZgtX.
• Then XgtYgtZgtX (cycling) , Intransitive ? Not
  rational

32

• It was noted by Condorcet in the 18 century that
  no alternative can win a majority against all
  other alternatives.
• Pairwise majority is not satisfactory in all
  cases.

33
Example 3 Borda Rule

• For each voter,
• associate the number 1 with the most preferred
  alternative,
• 2 with the second and so on,
• Assign to each alternative the number equal to
• the sum of the numbers the individual voters
  assigned to the alternative.

34

• Person
• 1 2 3
• 1st X(1) Y(1) X(1) X(4)
  X
• Pref. 2nd Y(2) X(2) W(2) ? Y(7) ?
  Y
• 3rd Z(3) W(3) Z(3)
  Z(10) W
• 4th W(4) Z(4) Y(4)
  W(9) Z
• Then We get choice XgtYgtWgtZ

35

• For above example, if we use binary/paired
  comparison With majority rule . We can get
• XgtY in 2 out of 3, YgtW in 2 out of 3,
• WgtZ in 2 out of 3, XgtW in 3 out of 3,
• XgtZ in 3 out of 3, YgtZ in 2 out of 3
• Then we can achieve same choice
• XgtYgtWgtZ

36

• For the previous example we had trouble with
  majority rule via binary/paired comparison, we
  get a tie between all three alternatives with the
  Bordas rule
• All three alternatives get a sum of 6.

37

• Some variations
• 1 with relevant scores available
• allotting each input system a point p to be
  distributed according to relevance scores of the
  documents.
• 2 Weighted Borda-rule
• Each voter may not have equal effectiveness to
  the final result. We may set more weight to good
  quality input systems.

38
Rank Aggregation in MetaSearch

• Here we discussed two cases which using
  algorithm rooted at social choice theory for
  MetaSearch rank aggregation.
• Data fusion track in TREC
• Javed A. Aslam, Mark Montague 2001 Models for
  Metasearch
• in SIGIR2001
• Rank aggregation for web search engine
• Cynthia Dwork, Ravi Kumar, Moni Naor,
  D.Sivakumar 2001
• Rank Aggregation Methods for the Web in WWW10

39
Data fusion track in TREC

• TREC (Text Retrieval Conference ,see
  http//trec.nist.gov/) maintains about 6Gb of
  SGML tagged text, queries and respective answers
  for evaluation purposes.
• The TREC organizers distribute data sets in
  advance and 50 new queries each year.
• The competing teams then submit ranked lists of
  documents that their system gave in response to
  each query. And these retrieval systems will be
  evaluated.

40

• These ranked lists are available for metasearch
  researchers to download and fuse.
• For each query, every retrieval system will
  return top 1000 documents and relevant score is
  available.
• Then given these results retrieved by many
  different retrieval systems, how to aggregate
  them for better performance?

41
Previous algorithms

• Min, Max and Average Models
• Fox and Shaw,1995
• Linear Combination Model
• Bartell 1995
• Logistic Regression Model

42
Example

• Min, Max and Average model
• The final score of each document d is based on
  the scores given to d by each input systems
  (voters).
• Algorithm Final
  score
• CombMin minimum of individual
  relevance scores
• CombMed median of individual relevance
  scores
• CombMax maximum of individual relevance
  scores
• CombSum sum of individual relevance
  scores
• CombANZ CombSum / num non-zero relevance
  scores
• CombMNZ CombSum num non-zero relevance
  scores

43

• Linear Combination Model
• The final score of document d is a simply
  linearly (each weighted differently) combining
  the normalized relevance scores given to each
  document.

44
Experiment result on TREC Model

• The performance of rank aggregation is evaluated
  by average precision over the queries
• Score-based borda-fuse is usually the best method
  among several borda variant algorithms.
• It is better than best input system over most of
  data collection. Such as TREC3, TREC5

45
Experiment result II

• The performance of rank aggregation is evaluated
  by average precision over the queries.
• Condorcet-fusion is the only algorithm that ,
  without training data, ever matches the
  performance of the best input system over TREC 9.
• Condorcet-fusion seems particularly sensitive to
  the dependence of input systems. If the input
  systems (voters) are too similar, the performance
  will decrease.

46
Rank aggregation methods for web

• New Challenges Different from the case in TREC
  data fusion,
• The coverage of various search engine is
  different
• Thus some highly relevant web pages may not be
  ranked by some search engines.
• Therefore, each voter ranks a partial candidate
  list

47
Preliminaries

• Given a universe U, an ordered list with
  respect to U is an ordering of a subset S U,
  i.e., ,with each
  and is some ordering relation on S.
• If contains
• all the elements in U, then it is said to be a
  full list,
• otherwise it is called partial list.

48

• Distance measures between two full lists with
  respect to a set S
• The Kendall tau distance
• It counts the number of pairwise disagreements
  between two lists.
• The distance is given by
• Normalize it by dividing the maximum possible
  value

49

• Spearman footrule distance
• Given two full lists and , the distance
  is given by
• Normalize it by dividing the maximum value

50

• Distance measures for more than 2 list
• Given several full lists ,
  for instance, the normalized Footrule distance of
  to s is given by
• An optimal Spearmans footrule distance
  solution s is one that minimizes the above
  normalized footrule distance.

51
Kemeny optimal aggregation

• The question is
• Given (full or partial) lists ,
  find a such that is a full list with
  respect to the union of the elements of
• and minimizes
• The aggregation obtained by optimizing Kendall
  distance is called Kemeny optimal aggregation.

52

• When kgt4,computing the Kemeny optimal
  aggregation is NP-hard.
• (please refer to Cynthia Dwork, Ravi Kumar,
  Moni Naor, D.Sivakumar 2001 for detailed proof )
• We can use Spearman footrule distance to
  approximate the Kendall distance.

53

• Proposition 1 For any two full lists , ,
• Proposition 2 if is the Kemeny optimal
  aggregation of full lists ,
  and optimizes the footrule aggregation, then

54

• Proposition 3 Given full lists
  if the median positions of the candidates in the
  lists form a permutation, then this permutation
  is a footrule optimal aggregation.
• Proposition 4 Footrule optimal aggregation of
  full lists can be computed in polynomial time,
  specifically, the time to find a minimum cost
  perfect matching in a bipartite graph.