Designing Ranking Systems for Consumer Reviews: The Economic Impact of Customer Sentiment in Electro

1
Designing Ranking Systems for Consumer Reviews
The Economic Impact of Customer Sentiment in
Electronic Markets Anindya GhosePanagiotis
IpeirotisStern School, New York University
ICEC 2007
2
Democratization of content creation

• Opinions in user-generated content
• The Web has changed the way that people express
  their views and opinions
• One can express opinions on almost anything at
• review sites, forums, reputation profiles, blogs
• Opinions Matter
• Businesses marketing intelligence, product
  design and service improvement.
• Firms invest a lot to find consumer opinions
• Consultants, Surveys and focus groups, etc
• Individuals interested in others opinions on
  products, services, topics, events
• Online word-of-mouth

3
Selected Prior Research

• Analyze and classify sentiments in online
  opinions
• Pang and Lee (2004), Hu and Liu (2004), Kim and
  Hovy (2004), Liu, Hu and Cheng (2005)
• Volume and valence in online reviews influences
  product sales
• Dellarocas et al. (2005), Chevalier and Mayzlin
  (2006), Liu (2006), Clemons and Gao (2006)
• Reviewers social information influences product
  sales and review helpfulness over and above
  volume and valence
• Forman, Ghose and Wiesenfeld (2006)

• What about the impact of textual sentiments on
• Product sales
• Review Helpfulness

4
Focus of this work

• Our Questions
• How sentiment in the text affects product demand?
• How sentiment in the text affects informativeness
  of reviews?

• Searching for reviews is different from general
  Web search.
• Reading only the review ranked at the top can be
  erroneous because it is only opinion of one
  person.
• Improve the design of electronic markets
• Propose two ranking mechanisms for ranking
  product reviews
• a consumer-oriented ranking mechanism ranks the
  reviews according to their expected helpfulness.
• a manufacturer-oriented ranking mechanism ranks
  the reviews according to their expected effect on
  sales.

5
Data
6
Data
7
Measuring Subjectivity

• Pang and Lee (2004)-technique that identifies
  which sentences in a text convey objective or
  subjective elements.
• Sentences in product description Objective
• Sentences in reviews Subjective
• A training set with two classes of documents
• A set of objective documents that contains the
  product descriptions of each of the 1,000
  products in our data set.
• A set of subjective documents that contains
  randomly retrieved reviews.
• Train a classifier (n-gram based) using Dynamic
  Language Model classifier from LingPipe toolkit
  to distinguish between subjective and objective
  sentences in each review.
• Average Probability of Subjectivity for a
  review
• Standard deviation of Subjectivity for each
  review
• In our context Subjectivity measures deviation
  of review from manufacturer-provided product
  information

8
Measuring Readability

• Readability measures amount of information in
  the review, and difficulty of reading the
  provided information
• Current work
• Measured number of sentences
• Measured length of review in words and characters
• Ratio of length in characters (or words) to
  number of sentences.
• Ongoing work
• Use Readability statistics of the reviews
• ColemanLiau
• Flesh Kinkaid Level
• Gunning
• SMOG Index

9
Impact of Sentiments and Sales

• Derived the stylistic characteristics of each
  review.
• Proceed to examine the economic impact of the
  subjectivity (or objectivity) of the review.

• Product level fixed effects
• We also ran first differences model using
  variations in the unit of time analysis (weekly,
  biweekly and monthly)
• Qualitative nature of results are robust to
  these specifications

10
Estimates for effect on sales rank
11
Effect of Sentiments on Informativeness
12
Validation with Content Analysis

• Asked two coders to classify 1,000 reviews (test
  set)
• Is the review informative or not?
• As influencing their purchasing decisions or not?
• Good inter-rater agreement (Kappa statistic of
  0.73)
• For second question, using polychoric correlation
  agreement was strong (p
• Estimated regression coefficients using rest of
  reviews (training set)
• Predicted helpfulness and influence of the test
  reviews, using only text

13
F-measure
F-measure combines precision and recall in a
single metric
14
Findings and Conclusion

• Show that textual sentiment influences sales over
  and above the numeric and self-descriptive
  information that consumer reviews and reviewers
  display.
• Increasingly subjective reviews can increase
  product sales.
• A mixture of objective and subjective opinions
  with extreme subjective content can increase
  sales.
• A mixture of objective and subjective opinions
  with extreme subjective content can increases
  informativeness of reviews
• Increased Readability can increases product
  sales and informativeness of reviews
• Our method can quickly identify reviews that are
  expected to be helpful to the users, and display
  them first.
• Improving the usefulness of the reviewing
  mechanism to the consumer in an electronic
  market.
• Manufacturers can understand which reviews are
  most impactful and use those product features for
  marketing promotions

15
Ongoing Work

• Combine the sentiments (positive, mixed or
  negative) with subjectivity analysis.
• Negative reviews may increase sales if the
  reviews are informative
• Incorporate the impact of all the past reviews of
  a reviewer.
• Examine more text-based variables, in detail
  (e.g. readability metrics).