Learning to Rank

1
Learning to Rank

• From Pairwise Approach to Listwise Approach

2
Agenda

• Introduction
• Ranking Problem
• Ranking
• Pairwise Ranking (Brief)
• Listwise Ranking
• Probability Models
• Permutation Probability
• Top one Probability
• Results

3
Introduction

• Construct model or method that learns to rank
• Area of use
• Anti Spam
• Product Rating
• Expert Finding
• ...

4
Introduction

• Ranking Problem Document retrieval

Documents d_1, d_2, ... d_n
Ranking of documents
Ranking System
Query q
5
Pairwise Ranking

• Classification of objects

Relevance label
Classification Model
6
Pairwise Ranking

• Support Vector Machine
• Ranking SVM
• Boosting
• RankBoost
• Neural Network
• RankNet

7
Listwise Ranking

• Training

(1)
(m)
Q q , ...., q Queries d d ,
...., d Documents y y , ...., y
Judgements x x , ...., x
Features f (x ) Score Func. z f(x )
, ...., f(x ) Scores
(i)
(i)
(i)
(i)
1
n
(i)
(i)
(i)
(i)
m
(i)
(i)
1
n
T (x , y )
(i)
i1
(i)
(i)
(i)
1
n
(i)
j
Listwise loss function
(i)
(i)
(i)
(i)
1
n
8
Listwise Ranking

• Ranking
• New Query q
• Associated Docs. d
• Feature vectors x
• Trained rank. Func. f (x )
• Rank documents in descending order

( i )
( i )
( i )
( i )
j
9
Permutation Probability

• f s probability distribution

pi ltpi(1), pi(2), ...., pi(n)gt s (s , s ,
.... s )
1
2
n
10
Top one Probability

• Probability of being ranked on top of list

11
ListNet

• Optimizing loss function
• Neural Network as model
• Gradient Descent as optimization alg.
• w neural network

12
Results

• TREC
• Web pages from .gov domain
• OSHUMED
• Documents and queries on medicine
• CSearch
• Data from commerciel search engine

13
Results

• NDCG Normalized Discounted Cumulative Gain
• Relevance judgements gt 2
• Korrekt Delvist korrekt - Ukorrekt
• MAP Mean Average Precision
• Relevance judgements 2
• Korrekt - Ukorrekt

14
Results

• NDCG on TREC

15
Results

• NDCG on OSHUMED