Unsupervised Sentiment Classification Across Domains

1
Unsupervised Sentiment Classification Across
Domains

• Soumya Ghosh
• Matthew Koch

2
Sentiment Classification

• What is sentiment classification?
• In this study we are primarily interested in
  polarity.

3
Motivation

• Classifying product reviews into recommended or
  not recommended
• Web search engine can allow a user to narrow
  search to pages with positive or negative
  comments
• Email Filtering
• Categorizing news articles into positive and
  negative views

4
Background

• Turney (2002) uses unsupervised techniques (PMI)
  for review polarity classification.
• Accuracy 84 (automobile reviews) to 66 (movie
  reviews)
• Pang (2002/2004) uses supervised learning (SVMs)
  to achieve 82.9 accuracy on the movie domain.

5
Turneys Approach

• Select a bunch of reference words likely to
  indicate sentiment orientation.
• excellent and poor.
• Calculate Pointwise mutual information between
  words in the document to be classified and the
  reference words.

6
Pointwise Mutual Information

• Ratio between the observed co-occurrence
  probability for the two words and the
  co-occurrence probability one would expect to see
  if the two words were independent .
• Measure of statistical dependence.
• PMI 0 if the two words occur together more
  frequently than expected by chance.
• PMI frequently than expected by chance.

7
Turneys Approach

• Compute semantic orientations of document
  phrases
• SO is positive if the word (phrase) is strongly
  associated to excellent and negative when it
  is associated with poor.

8
Caveats

• Word statistics are captured using altavista.
• SO is computed over phrases rather than
  individual words.

9
Supervised Approaches

• Pang et al. compare a bunch of supervised
  learning algorithms for polarity classification.
• They explore a wide variety of feature vectors
  for this purpose.
• SVM performs the best, with word presence
  features.

10
Supervised Approaches
11
Problems with previous approaches

• Domain dependent (Engstrom 2004).
• Supervised techniques -- document (review) level
  classifications. These work really poorly when
  faced with sentence level sentiment
  classification.

12
Our Work

• We are interested in classifying quotes on the
  basis of their polarity and hence infer a
  figures stand on a particular issue.
• Sentence Level Classification problem.
• Try to make our classification model domain
  independent.

13
Our Approach

• Use Pointwise Mutual Information to score word
  sentiments in a non-sparse corpus.
• Select the top p percentile and the bottom q
  percentile words.
• Most sentimentally oriented words. Lets call this
  SO.
• Extract most sentimentally oriented words from
  the sparse target corpus ( bunch of sentences to
  be classified).
• This is our development set.
• We simply select all adjectives from this corpus.

14
Our Approach

• For each word wi in the dev set do
• Use wordnet to find its synset.
• Assign to every synonym its score from SO if
  synonym is not present in SO a score of 0 is
  assigned to it.
• Score of wi mean scores of the synset of wi.
• Sentence scores are the average scores of their
  words.

15
Our Approach

• Determine the pos/neg score decision boundary
  from the distribution of the development set.
• Assume that the development set is representative
  of the whole sample.
• In our case this boils down to using the median
  value of the sorted sentence-level scores.

16
Data

• 2000 (1000 pos/1000 neg) labeled document level
  movie review data. (Cornell -- IMDB)
• 3020 labeled sentence level movie review data.
  (Yahoo movies)
• 100 labeled sentence level data from WSJ.

17
Experiments

• Step 1
• Learn using the IMDB data and classify the movie
  review data.
• Step 2
• Classify the sentences from WSJ using the same
  model.

18
Results

• As a baseline we follow Pangs supervised
  learning method.
• We randomly split the Yahoo corpus into a test
  set of 420 sentences and a train set of 2600
  sentences.
• Accuracy over 10 runs 63.34, std 1.78

19
Results

• Next we trained up the same SVM model on the IMDB
  corpus. (2000 labeled documents) and tested on
  the random set of 420 sentences.
• Accuracy over 10 runs 62.89, std 1.57
• Difference is not statistically significant.
• Implies high correlation between the domains.

20
Results

• Turneys algorithm on the Yahoo movie corpus
• Accuracy over 10 runs 56.86, std 0.04
• Results are statistically worse than the
  supervised techniques at 95 confidence.

21
Results

• Our Results on the Yahoo movie corpus
• Averaged over 10 runs
• Accuracy 69.80, std0.02
• Statistically significant over previous
  techniques at 95 confidence
• Our Results on the WSJ corpus
• Averaged over 10 runs
• Accuracy 60.00, std 0.001

22
Our Results (Correct Classifications).

• the dollar strengthened Friday as stocks rocketed
  positive.
• Kidder Peabody reiterated its buy recommendation
  on the stock positive.
• But the thing it's supposed to measure --
  manufacturing strength -- it missed altogether
  last month - negative.
• I just don't feel the extra yield you're getting
  is worth it to justify all the risks you're
  taking - negative

23
Misclassifications.

• On past occasions, her finely textured singing
  has been ample compensation for her mannered
  gestures, but on this big night -- her first Met
  opening -- only some of her pianissimos were
  skillfully deployed. we predict this is
  positive.
• IBM is five times the size of Digital and is
  awesome we predict this is negative.

24
Conclusion

• Our unsupervised algorithm outperforms standard
  sentiment classification techniques.
• Encouraging results for limited cross-domain
  sentiment classification using knowledge
  transfer.

25
Future Work

• Look into semantic vocabulary mining.
• Explore other wordnet relationships for knowledge
  transfer.
• Take a look at the algorithm from a more
  theoretical point of view.

26
References

• Peter D. Turney Thumbs Up or Thumbs Down?
  Semantic Orientation Applied to Unsupervised
  Classification of Reviews, ACL 2002.
• Bo Pang, Lillian Lee, Shivakumar Vaithyanathan
  Thumbs up? Sentiment Classification using Machine
  Learning Techniques, CoRR cs.CL/0205070, 2002.
• Bo Pang, Lillian Lee A Sentimental Education
  Sentiment Analysis Using Subjectivity
  Summarization Based on Minimum Cuts. ACL 2004.
• Charlotta Engstrom Topic Dependence in Sentiment
  Classification. Masters thesis, University of.
  Cambridge, July 2004.