Topic models for corpora and for graphs

1
Topic models for corpora and for graphs
2
Motivation

• Social graphs seem to have
• some aspects of randomness
• small diameter, giant connected components,..
• some structure
• homophily, scale-free degree dist?

3
More terms

• Stochastic block model, aka Block-stochastic
  matrix
• Draw ni nodes in block i
• With probability pij, connect pairs (u,v) where u
  is in block i, v is in block j
• Special, simple case piiqi, and pijs for all
  i?j
• Question can you fit this model to a graph?
• find each pij and latent node?block mapping

4
Not? football
5
Not? books
6
Outline

• Stochastic block models inference question
• Review of text models
• Mixture of multinomials EM
• LDA and Gibbs (or variational EM)
• Block models and inference
• Mixed-membership block models
• Multinomial block models and inference w/ Gibbs

7
Review supervised Naïve Bayes

• Naïve Bayes Model Compact representation

?
?
C
C
..
WN
W1
W2
W3
W
M
N
b
M
b
8
Review supervised Naïve Bayes

• Multinomial Naïve Bayes

?

• For each document d 1,?, M
• Generate Cd Mult( ?)
• For each position n 1,?, Nd
• Generate wn Mult( ?,Cd)

C
..
WN
W1
W2
W3
M
b
9
Review supervised Naïve Bayes

• Multinomial naïve Bayes Learning
• Maximize the log-likelihood of observed variables
  w.r.t. the parameters
• Convex function global optimum
• Solution

10
Review unsupervised Naïve Bayes

• Mixture model unsupervised naïve Bayes model

• Joint probability of words and classes
• But classes are not visible

?
C
Z
W
N
M
b
11
LDA
12
Review - LDA

• Motivation

• Assumptions 1) documents are i.i.d 2) within a
  document, words are i.i.d. (bag of words)
• For each document d 1,?,M
• Generate ?d D1()
• For each word n 1,?, Nd
• generate wn D2( ?dn)
• Now pick your favorite distributions for D1, D2

?
w
N
M
13
Mixed membership

• Latent Dirichlet Allocation

?

• For each document d 1,?,M
• Generate ?d Dir( ?)
• For each position n 1,?, Nd
• generate zn Mult( ?d)
• generate wn Mult( ?zn)

a
z
w
N
M
?
K
14

• vs Naïve Bayes

?
a
z
w
N
M
?
K
15

• LDAs view of a document

16

• LDA topics

17
Review - LDA

• Latent Dirichlet Allocation
• Parameter learning
• Variational EM
• Numerical approximation using lower-bounds
• Results in biased solutions
• Convergence has numerical guarantees
• Gibbs Sampling
• Stochastic simulation
• unbiased solutions
• Stochastic convergence

18
Review - LDA

• Gibbs sampling
• Applicable when joint distribution is hard to
  evaluate but conditional distribution is known
• Sequence of samples comprises a Markov Chain
• Stationary distribution of the chain is the joint
  distribution

Key capability estimate distribution of one
latent variables given the other latent variables
and observed variables.
19
Why does Gibbs sampling work?

• Whats the fixed point?
• Stationary distribution of the chain is the joint
  distribution
• When will it converge (in the limit)?
• Graph defined by the chain is connected
• How long will it take to converge?
• Depends on second eigenvector of that graph

20
(No Transcript)
21
Called collapsed Gibbs sampling since youve
marginalized away some variables
Fr Parameter estimation for text analysis -
Gregor Heinrich
22
Review - LDA
Mixed membership

• Latent Dirichlet Allocation

?

• Randomly initialize each zm,n
• Repeat for t1,.
• For each doc m, word n
• Find Pr(zmnkother zs)
• Sample zmn according to that distr.

a
z
w
N
M
?
23
Outline

• Stochastic block models inference question
• Review of text models
• Mixture of multinomials EM
• LDA and Gibbs (or variational EM)
• Block models and inference
• Mixed-membership block models
• Multinomial block models and inference w/ Gibbs
• Beastiary of other probabilistic graph models
• Latent-space models, exchangeable graphs, p1,
  ERGM

24
Review - LDA

• Motivation

• Assumptions 1) documents are i.i.d 2) within a
  document, words are i.i.d. (bag of words)
• For each document d 1,?,M
• Generate ?d D1()
• For each word n 1,?, Nd
• generate wn D2( ?dn)
• Docs and words are exchangeable.

?
w
N
M
25
Stochastic Block models assume 1) nodes w/in a
block z and 2) edges between blocks zp,zq are
exchangeable
a
b
zp
zp
zq
p
apq
N
N2
26
Stochastic Block models assume 1) nodes w/in a
block z and 2) edges between blocks zp,zq are
exchangeable
a

• Gibbs sampling
• Randomly initialize zp for each node p.
• For t 1
• For each node p
• Compute zp given other zs
• Sample zp

b
zp
zp
zq
p
apq
N
N2
See Snijders Nowicki, 1997, Estimation and
Prediction for Stochastic Blockmodels for Groups
with Latent Graph Structure
27
Mixed Membership Stochastic Block models
a
b
?p
?q
?p
zp?.
z.?q
p
apq
N
N2
Airoldi et al, JMLR 2008
28
Parkkinen et al paper
29
Another mixed membership block model
30
Another mixed membership block model
z(zi,zj) is a pair of block ids nz pairs
z qz1,i links to i from block z1 qz1,.
outlinks in block z1 d indicator for
diagonal M nodes
31
Another mixed membership block model
32
Experiments
lots of synthetic data
Balasubramanyan, Lin, Cohen, NIPS w/s 2010
33
(No Transcript)
34
(No Transcript)
35
Experiments
Balasubramanyan, Lin, Cohen, NIPS w/s 2010
36
Experiments
Balasubramanyan, Lin, Cohen, NIPS w/s 2010