Using Graphical Probabilistic Generative Models for Recommendation

1
Using(Graphical) Probabilistic Generative
Modelsfor Recommendation

• Omid Madani
• David Pennock
• Sumit Sanghai

2
Overview

• What is a probabilistic generative model?
• What are graphical models?
• ExampleLatent Dirichelet Allocation (LDA)
• Discussion

3
What is a Probabilistic Generative Model?

• 1. Assume some parameterized random process
  generates the data we see
• 1b. (Optional) Learn the parameter values that
  best explain the data
• 2. Use the model to predict (infer) new data
  based on data seen so far
• Benefits
• Can focus on modeling forward process more
  natural, easier to do -- known algs for learning,
  inference, value of information, etc.
• Assumptions are explicit, up front

4
Simple (useless) example

• Assume there are 2 ( only 2) types of people
  easily pleased and cynical
• They occur with equal frequency
• They rate movieswith probability
• We see data user1 AAFAAAAAAFAA
• Infer probability user is easily pleased
• Predict likely next rating (A)

A F
easily pleased cynical
0.1
0.9
0.9
0.1
5
Simple (useless) examplewith learning

• Assume there are 2 ( only 2) types of people,
  occurring with equal frequency
• They rate movieswith probability
• We see data user1, user2, user3, ...
• Learn p1, p2 (e.g., max likelihood)
• Infer probability a given user is type 1/2
• Predict likely next rating

A F
type 1 type 2
1-p1
p1
1-p2
p2
6
Example Personality Diagnosis

• User is personality type isritrue
  
• Assumption 1. User is reported rating for title
  j is drawn from an independent normal
  distribution with mean ritrue Pr(rijx
  rijtruey) ? e-(x-y)2/2?2where ? is a free
  parameter.

7
PD Model

• Assumption 2. The distribution of personality
  types in the database is representative of the
  target population.
• Pr(ratrueri) 1/n

8
PD Model Inference

• Probability that the active user is of the same
  personality type as any other user, given the
  active users ratings.
• Pr(ratrueri ra1x1, , ramxm)
• ? Pr(ra1x1 ra1trueri1) ???
• Pr(ramxm ramtruerim) ? Pr(ratrueri)

9
PD Model Inference

• Probability distribution for the active users
  ratings of unseen titles, j?NR.
• Pr(rajxj ra1x1, , ramxm)
• ?i1 Pr(rajxj ratrueri) ?
• Pr(ratrueri ra1x1, , ramxm)
• Prediction most probable rating
• Time and space complexity O(n m)

n
10
Graphical Models

• Standardized way to depict probabilistic
  generative models graphically, incl params
• Easier to hand-create models easier to
  understand/interpret learned models
• Can build quite complex models Off the shelf
  algorithms exist for learning inference that
  can work efficiently

11
Other Graphical Models
Hofmann aspect model probabilistic LSI
Popescul et al. 3-wayaspect model
user
user
style
style
movie
movie
actor
12
Personality Diagnosis Graphical Model
ritrue
N(ra1true,?2)
ra1
ri2
rij
rim
...
...
13
Internet Movie Database (IMDb)
Source Jensen 2004

• The Internet Movie Database (www.imdb.com)
• Questions
• What predicts box office receipts?
• Are awards important?
• What about previouscommercial success?
• Do ticket buyers careabout studios, or
  onlyabout actors anddirectors?

14
Example Learned Model
Source Jensen 2004
15
More on Graphical Models

• Structures that allow
• Modularity ? Inference
• Interpretability ? Extensibility
• Examples
• Markov models Bayes nets
• Markov random fields
• LSI (PCA), probabilistic LSI, LDA, ...

16
Example Dirichelet Allocation

• Given,
• V words, e.g., win, game, vote
• k topics, e.g., politics, sports
• Matrix of word-given-topic probabilities
  (e.g. P(winsports).5)
• Dirichelet density D over topics, specified by
  , e.g. uniform where

17
Topics are multinomials over words
win game vote
Sports Politics
0.5
0.5
0
0.05
0.35
0.6
18
DA Model

• To generate a document
• Sample doc length N from Poisson
• Sample from D (e.g., )
• Repeat N times
• Draw a topic z with 1/3 probability zsports,
  2/3 zpolitics
• Draw a word from the topic z

19
The DA Graph
20
Latent Dirichelet Allocation

• Task given k and a corpus, learn
• Matrix
• Dirichelet parameters
• Method EM

21
Types of Inference/Uses

• Dimensionality reduction
• Document similarity
• Word similarity
• Clustering

22
Applications

• Blei et. al. (2003) applied it to EachMovie
  dataset
• User as document, movie as a word
• topic as ..
• 10-20 latent topics found
• Perplexity significantly better than pLSI and
  document as single mixture

23
Benefits of Graphical Models

• Structures that allow
• Modularity, interpretability, extensibility
• Inference
• known semantics
• provide probabilities
• handle missing values
• value of information (e.g. use in active
  learning)

24
Active Research

• Tractability of inference
• Sparse data and over-fitting
• Fidelity of the models to real world
• E.g., multinomial appropriate?
• More sophisticated models
• nested Chinese process, author-topic,
  hierarchical, modeling word order, relation
  models, ...
• Active learning