Subjectivity Analysis and Recognizing Contextual Polarity

1
Subjectivity Analysis and Recognizing Contextual
Polarity

• Theresa Wilson
• University of Pittsburgh

2
Collaborators

• Jan Wiebe
• Paul Hoffmann

3
Subjectivity

• Expression of
• opinions, emotions, evaluations,
• sentiments, speculations, uncertainty
• in natural language

Banfield (1982), Fludernik (1993)
4
Subjectivity Analysis

• Identify
• Opinions, emotions in NL
• Recognize/Extract
• Components
• Properties

The birth centennial of Anna May Wong has brought
much popular attention to the Chinese American
film and stage actress, who was active in
American and Europe through the 1940s. Thus it
appears to be apt moment to recover Wongs legacy
from the stereotypes of Madame Butterfly and
Dragon Lady. Wongs dragon lady costume was
condemned by Chinese Nationalists . . .
Source Chinese Nationalists Attitude was
condemned by Target Wongs dragon lady costume
5
Motivations

• Classifying reviews as positive/negative
• Analyzing product reputations
• Tracking sentiments toward topics and events
• Recognizing hostile messages
• Genre classification
• Opinion-oriented question answering
• Improving information extraction
• Improving word-sense disambiguation

6
Question Answering

• Fact-based question answering
• When is the first day of spring?
• Do Lipton employees take coffee breaks?
• Opinion-oriented question answering
• How do the Chinese regard the human rights record
  of the United States?
• Did America support the Venezuelan foreign policy
  followed by Chavez?

7
Question Answering Example
Q What is the international reaction to the
reelection of Robert Mugabe as President of
Zimbabwe?
African observers generally approved of his
victory while Western Governments denounced it.
positive/negative/neutral ? polarity
8
Polarity

• Key for subjectivity analysis
• Focus on positive/negative emotions, evaluations,
  stances

? sentiment analysis
Im happy the Steelers won! Shes against the
bill.
9
Prior and Contextual Polarity
Lexicon abhor negative acrimony negative .
. . cheers positive . . . beautiful
positive . . . horrid negative . . . woe
negative wonderfully positive
Cheers to Timothy Whitfield for the wonderfully
horrid visuals.
Cheers to Timothy Whitfield for the wonderfully
horrid visuals.
Prior polarity (out of context)
10
Contextual Polarity

• Polarity expressed by a word or phrase in context

11
Example

• Philip Clap, President of the National
  Environment Trust, sums up well the general
  thrust of the reaction of environmental
  movements there is no reason at all to believe
  that the polluters are suddenly going to become
  reasonable.

12
Example

• Philip Clap, President of the National
  Environment Trust, sums up well the general
  thrust of the reaction of environmental
  movements there is no reason at all to believe
  that the polluters are suddenly going to become
  reasonable.

13
Example

• Philip Clap, President of the National
  Environment Trust, sums up well the general
  thrust of the reaction of environmental
  movements there is no reason at all to believe
  that the polluters are suddenly going to become
  reasonable.

contextual polarity
prior polarity
14
Goal of This Work

• Automatically distinguish prior and contextual
  polarity in sentiment expressions

15
Approach

• Use machine learning and variety of features
• Achieve significant results for a large subset of
  sentiment expressions

16
Outline

• Introduction
• Corpus and Manual Annotations
• Contextual Polarity Influencers
• Experiments
• Related Work

17
Manual Annotations

• Needed contextual polarity of sentiment
  expressions
• sentiment expressions ? subjective expressions
• Had Multi-Perspective Question Answering (MPQA)
  Opinion Corpus
• Includes annotations of subjective expressions

18
MPQA Corpus of Opinion AnnotationsWiebe, Wilson,
Cardie Language Resources and Evaluation 39(1-2),
2005

• Fine-grained expression-level annotations rather
  than sentence or document level
• Annotations
• expressions of opinions, evaluations, emotions,
• material attributed to a source, but presented
  objectively

19
Direct Subjective Annotations

• Direct mentions of opinions, emotions,
• The United States fears a spill-over from the
  anti-terrorist campaign.
• Opinions expressed in speech events
• We foresaw electoral fraud but not daylight
  robbery, Tsvangirai said.

20
Expressive Subjective Elements Banfield 1982

• Indirectly express an opinion through the wording
  or the way something is described
• We foresaw electoral fraud but not daylight
  robbery, Tsvangirai said
• The part of the US human rights report about
  China is full of absurdities and fabrications

21
Objective Speech Events

• Material attributed to a source, but presented as
  objective fact
• The government, it added, has amended the
  Pakistan Citizenship Act 10 of 1951 to enable
  women of Pakistani descent to claim Pakistani
  nationality for their children born to foreign
  husbands.

22
Given the Annotations in the MPQA Corpus

• How to annotate contextual polarity?
• Which are subjective expressions?
• Which of those are sentiment expressions?
• What is our scheme for contextual polarity?

23
Which are subjective expressions?
Direct subjective anchor said source
ltwriter, Xirao-Nimagt intensity high
expression intensity neutral
Expressive subjective element anchor full of
absurdities source ltwriter, Xirao-Nimagt
intensity high
subjective expressions
Direct subjective anchor denounced source
ltwriter, Xirao-Nimagt intensity high
expression intensity high
Expressive subjective element anchor pack of
lies source ltwriter, Xirao-Nimagt
intensity high
24
Subjective Expressions in MPQA Corpus

• Direct subjective annotations with
  expression-intensity ? neutral
• All expressive subjective elements

25
Which subjective expressions are sentiment
expressions?

• Sentiment Expressions positive and negative
• emotions, evaluations, stances
• Not a sentiment expression

Im happy the Steelers won! Shes against the
bill.
I believe Santorum will be defeated in the next
election.
26
Scheme for Contextual Polarity Annotations

• Mark sentiment expressions as
  positive, negative, or both

positive
African observers generally approved of his
victory while Western governments denounced it.
negative
Besides, politicians refer to good and evil
both
27
Everything Else Neutral

• Neutral emotions, evaluations, stances
• All other subjective expressions

Jerome says the hospital feels no different than
a hospital in the states.
neutral
I believe Santorum will be defeated in the next
election.
28
Note on Annotation Scheme

• Judge the contextual polarity of sentiment
  ultimately being conveyed
• They have not succeeded, and will never succeed,
  in breaking the will of this valiant people.

positive
29
Agreement Study

• 10 documents with 447 subjective expressions
• Kappa 0.72 (82)
• Remove uncertain cases ? at least one annotator
  marked uncertain (18)
• Kappa 0.84 (90)
• (But all data included in experiments)

30
Corpus

• 425 documents from MPQA Opinion Corpus
• 15,991 subjective expressions in 8,984 sentences
• Divided into two sets
• Development set
• 66 docs / 2,808 subjective expressions
• Experiment set
• 359 docs / 13,183 subjective expressions
• Divided into 10 folds for cross-validation

31
Corpus Statistics

• Sentences and subjective expressions
• 28 none, 25 one, 47 two or more
• Sentences with two or more subjective
  expressions
• 17 mixtures of positive and negative
• 62 mixtures of polar and neutral

32
Outline

• Introduction
• Corpus and Manual Annotations
• Contextual Polarity Influencers
• Experiments
• Related Work

33
Negation

• Local
• not good
• Longer-distance dependencies
• does not look very good
• no one thinks that its good
• Some negation phrases intensify rather than flip
  polarity
• not only good but amazing
• nothing if not entertaining

34
Polarity Shifters

• Diminishers can flip polarity
• little truth, little threat
• Typically shift to negative
• lack of understanding
• affront to human decency
• Typically shift to positive
• abate the damage

35
Word Sense

• Trust
• Environmental Trust
• He has won the peoples trust
• Condemned
• Gavin Elementary School was condemned in April
  2004 after many of its roof supports
• Zimbabwes presidential election was condemned by
  some Western Countries

36
Syntactic Role of a Word

• Consider
• Polluters as subject
• Under the application shield, polluters are
  allowed to operate if they have a permit.
• Polluters as object of copular
• "The big-city folks are pointing at the farmers
  and saying you are polluters ...

37
Dependency Relationships between Words

• Modifiers
• wonderfully horrid
• Conjunctions
• good and evil
• rich and handsome
• rich and snobbish

38
Polanyi Zaenen (2004)

• Detailed discussion of contextual polarity
  influencers
• Inspired many of features used in experiments

39
Outline

• Introduction
• Corpus and Manual Annotations
• Contextual Polarity Influencers
• Experiments
• Wilson, Wiebe, Hoffmann HLT-EMNLP-2005
• Related Work

40
Experiments

• Both Steps
• BoosTexter AdaBoost.HM 5000 rounds boosting
• 10-fold cross validation

• Give each instance its own label

41
Definition of Gold Standard

• Given an instance inst from the lexicon
• if inst not in a subjective expression
• goldclass(inst) neutral
• else if inst in at least one positive and one
  negative subjective expression
• goldclass(inst) both
• else if inst in a mixture of negative and
  neutral
• goldclass(inst) negative
• else if inst in a mixture of positive and
  neutral
• goldclass(inst) positive
• else goldclass(inst) contextual polarity of
  subjective expression

42
Prior-Polarity Subjectivity Lexicon

• Over 8,000 words from a variety of sources
• Both manually and automatically identified
• Positive/negative words from General Inquirer and
  Hatzivassiloglou and McKeown (1997)
• All words in lexicon tagged with
• Prior polarity positive, negative, both, neutral
• Reliability strongly subjective (strongsubj),
  weakly subjective (weaksubj)

43
Motivation for 2-Step Approach

• Consider a prior-polarity classifier
• Assume contextual polarity prior polarity
• 48 accuracy
• 76 errors
• Words with positive/negative/both prior polarity
  but neutral contextual polarity

44

• Word features
• Modification features
• Structure features
• Sentence features
• Document feature

45

• Word token
• terrifies
• Word part-of-speech
• VB
• Context
• that terrifies me
• Prior Polarity
• negative
• Reliability
• strongsubj

• Word features
• Modification features
• Structure features
• Sentence features
• Document feature

46

• Word features
• Modification features
• Structure features
• Sentence features
• Document feature

• Binary features
• Preceded by
• adjective
• adverb (other than not)
• intensifier
• Self intensifier
• Modifies
• strongsubj clue
• weaksubj clue
• Modified by
• strongsubj clue
• weaksubj clue

Dependency Parse Tree
47

• Word features
• Modification features
• Structure features
• Sentence features
• Document feature

• Binary features
• In subject
• The human rights report poses
• In copular
• I am confident
• In passive voice
• must be regarded

48

• Word features
• Modification features
• Structure features
• Sentence features
• Document feature

• Count of strongsubj clues in
• previous, current, next sentence
• Count of weaksubj clues in
• previous, current, next sentence
• Counts of various parts of speech

49

• Document topic (15)
• economics
• health
• Kyoto protocol
• presidential election in Zimbabwe

• Word features
• Modification features
• Structure features
• Sentence features
• Document feature

Example The disease can be contracted if a
person is bitten by a certain tick or if a person
comes into contact with the blood of a congo
fever sufferer.
50
Results 1a

• Improvements over baselines significant
• Prior polarity only 52 accuracy

51
Results 1b
52
Step 2 Polarity Classification
19,506
5,671

• Classes
• positive, negative, both, neutral

53

• Word token
• Word prior polarity
• Negated
• Negated subject
• Modifies polarity
• Modified by polarity
• Conjunction polarity
• General polarity shifter
• Negative polarity shifter
• Positive polarity shifter

54

• Word token
• Word prior polarity
• Negated
• Negated subject
• Modifies polarity
• Modified by polarity
• Conjunction polarity
• General polarity shifter
• Negative polarity shifter
• Positive polarity shifter

• Word token
• terrifies
• Word prior polarity
• negative

55

• Word token
• Word prior polarity
• Negated
• Negated subject
• Modifies polarity
• Modified by polarity
• Conjunction polarity
• General polarity shifter
• Negative polarity shifter
• Positive polarity shifter

• Binary features
• Negated
• For example
• not good
• does not look very good
• not only good but amazing
• Negated subject
• No politically prudent Israeli could support
  either of them.

56

• Word token
• Word prior polarity
• Negated
• Negated subject
• Modifies polarity
• Modified by polarity
• Conjunction polarity
• General polarity shifter
• Negative polarity shifter
• Positive polarity shifter

• Modifies polarity
• 5 values positive, negative, neutral, both, not
  mod
• substantial negative
• Modified by polarity
• 5 values positive, negative, neutral, both, not
  mod
• challenge positive

57

• Word token
• Word prior polarity
• Negated
• Negated subject
• Modifies polarity
• Modified by polarity
• Conjunction polarity
• General polarity shifter
• Negative polarity shifter
• Positive polarity shifter

• Conjunction polarity
• 5 values positive, negative, neutral, both, not
  mod
• good negative

58

• Word token
• Word prior polarity
• Negated
• Negated subject
• Modifies polarity
• Modified by polarity
• Conjunction polarity
• General polarity shifter
• Negative polarity shifter
• Positive polarity shifter

• General polarity shifter
• pose little threat
• contains little truth
• Negative polarity shifter
• lack of understanding
• Positive polarity shifter
• abate the damage

59
Results 2a

• Improvements over baselines significant
• Prior polarity only 61 accuracy

60

• Ablation experiments removing features
• Negated, negated subject
• Modifies polarity, modified by polarity
• Conjunction polarity
• General, negative, positive polarity shifters

61
Summary

• Added a layer of contextual polarity judgments to
  the MPQA Corpus www.cs.pitt.edu/mqpa/databaserelea
  se (version 1.2)
• Acknowledgement ARDA AQUAINT
• Introduced a two-step approach to phrase-level
  sentiment analysis
• Combined linguistically-motivated features and
  machine learning to successfully recognize the
  contextual polarity of a large subset of
  sentiment expressions

62
Related Work

• Learn prior polarity of words and phrases
• e.g., Hatzivassiloglou McKeown (1997), Turney
  (2002)
• Sentence-level sentiment analysis
• e.g., Yu Hatzivassiloglou (2003), Kim Hovy
  (2004)
• Phrase-level contextual polarity classification
• e.g., Yi et al. (2003), Popescu and Etzioni (2005)

63
Recognizing IntensityWilson, Wiebe, Hwa
AAAI-2004, Computational Intelligence
(forthcoming)

• Identifying intensity
• New syntactic features learned from dependency
  parses of training data
• really quite nice
• Classify intensity at different levels
• Sentences and nested clauses

64
Attitude types and targetsWilson and Wiebe
Frontiers in Corpus Annotation II ACL Workshop
2005

• More attitudes than positive/negative sentiment
• Agreement/Disagreement
• Positive/Negative Arguing and Belief
• Positive/Negative Intention
• Speculation
• Annotating in MPQA corpus
• Developing recognition tools for attitude types

65
Example

• Agreement

Republicans concede that at this point it could
be his only option.
66
Examples

• Arguing

67
Examples

• Intention

68
Attitude types and targets

• I think people are happy because Chavez has
  fallen.

direct subjective span are happy source
ltwriter, I, Peoplegt attitude
direct subjective span think source
ltwriter, Igt attitude
inferred attitude span are happy because
Chavez has fallen type neg sentiment
intensity medium target
attitude span are happy type pos sentiment
intensity medium target
attitude span think type positive arguing
intensity medium target
target span people are happy because
Chavez has fallen
target span Chavez has fallen
target span Chavez
69
Future Work

• Add more structure to features
• Automate target identification
• Explore domain dependence of subjective language
• Explore subjectivity across languages
• Add discourse level connecting components and
  properties of opinions

70
Thank you!

• Questions?