Language Model Methods and Metrics

1
Language Model Methods and Metrics

• Gary Luu
• Ryan Fortune

2
Skip N-grams

• Interpolated with Bigram
• Get Influence of words further away without
  increasing dimensionality
• Learning Curve

3
Skip N-gram Learning Curve
4
Content Word Language Model

• Help predict next word using last uncommon word,
  try to capture context
• Found list of 250 most common words
• Tried different sizes for common words
• Interpolated with language models, since this
  wouldnt maintain grammar
• P(wC)

5
Content Word Model
6
Bag Generation Metrics

• Bag Generation NP-Hard
• Random Restart Greedy Hill-Climbing
• Stability Metric
• Give model correct sentence, does it maintain it
  as an optima?
• A percentage of sentences that remain stable
• Reconstruction Metric
• Needs to be compared against lucky/random

7
Bag Generation Metrics
8
Clustering -IBMFullPredict

• Clustering overview
• Perplexity down to 107 with million sentence
  corpus
• Pibmfullpredict(wiwi-2wi-1)
  
  ?P(Wwi-2wi-1) (1-?)P(WWi-1Wi-2)
  µP(wwi-1wi-2,W) (1-µ)P(wWi-2,Wi-1,W)

9
Learning Curve for IBMFullPredict