Design and Evaluation of an Adaptive Incentive Mechanism for Sustained Educational Communities

1
Design and Evaluation of an Adaptive Incentive
Mechanism for Sustained Educational Communities

• Ran Cheng and Julita Vassileva
• User Modeling User-Adapted Interaction
• Presented by Rosta Farzan
• PAWS Group Meeting
• April 13, 2007

2
Problem
Insufficient user participation in online
communities
Provide incentives to encourage participation
Too much participation especially of low-quality
3
Goal

• Proposing an incentive mechanism regulating
  quantity and quality of user contribution and
  ensuring a sustainable user participation

4
Outline

• Related work
• Collaborative quality evaluation mechanism
• Brief introduction to Comtella
• Proposed mechanism
• Implementation of proposed mechanism in Comtella
• Evaluation
• Discussion

5
Collaborative Quality Evaluation Mechanism

• Active and sustained user participation while
  avinding information overload requires quality
  control
• Decentralized moderation
• Real world example
• Measuring the quality of journals or papers by
  counting the time they were cited
• Online communities
• Counting the number of clicks on each item
• Problem
• Not always a click means a positive attitude
• Explicit user rating
• E.g. rating process in Slashdot
• Problem rich get richer
• Unfair score for items with insufficient
  attention
• Lower initial rating
• Contributed late in discussion

6
Requirement of Incentives

• Time based community need
• New contributions in the early period of
  discussion
• Ratings of the contribution when many
  contributions are collected
• Different users have different contribution
  patterns
• Encourage higher participation for users who
  contributed few high-quality resources
• Inhibit contributions from users who contributed
  many low-quality resources
• Overall community need

7
Comtella

• Developed at University of Saskatchewan
• Online community for sharing URLs of
  class-related web-resources (bookmarks)

8
Proposed Incentive Mechanism

• Mechanism encouraging users to rate resources
• Adaptive reward mechanism

9
Encouraging Users to Rate

• All users can rate others contribution
• Each user receives a limited number of rating
  points to give out
• Users with higher level membership receives more
  points
• Initial rating of zero independent of level of
  resource provider
• Ensuring all contributions have equal chance to
  be read and rated
• Summative rating
• Incentive Virtual currency (c-points)
• Limited initial c-points
• Awarding users for rating resources
• Depending on users reputation
• Can be invested to promote the initial visibility

10
Sorted by c-points
Summative Ratings
Report duplicate, broken, or special permission
links
11
Adaptive Reward Mechanism

• Hierarchical membership
• Adapt rewards for different forms of
  participation
• Quality of users participation so far
• Community need
• Personalized motivational message
• Stating specific performance goals
• Calculating adaptive rewards
• Community Model
• Individual Model

12
Community Model

• Expected of total contributions
• Set by community admin
• Community Reward Factor
• Usefulness of new resources

13
Individual Model

• Contribution reputation
• Sharing
• Average summative rating of all shared resources
• Rating
• Quality Difference between the specific rating
  and the average of all ratings the resource gets
  eventually
• Smaller difference ? Higher quality
• Current membership level

14
Individual Model

• Expected of resource contribution
• Reward Factor

15
Adaptive Reward Mechanism
16
Implementation of proposed mechanism in Comtella
17
Evaluation

• Questions to be answered
• Will the users in the test group rate articles
  more actively?
• How well will the summative ratings reflect the
  real quality of the articles?
• Will the users tend to share resources earlier in
  the week?
• Will the actual number of contributions be close
  to the desired one?
• Will the users share the number of articles that
  is expected from them?
• Will the users contribute a higher percentage of
  high-quality articles?
• Will there be information overload?

18
Participant

• Class on Ethics and Information Technology
• Share web-articles related to the topic of each
  week
• 31 4th year undergraduate students
• Test group 15
• Control group 16
• Randomly assigned while controlling gender and
  nationality

19
Result

• Users in the test group were more active in
  rating articles
• The articles with higher ratings were more likely
  to be chosen by users to summarize
• The users in the test group were more satisfied
  with the summative ratings received by their
  articles
• The users in the test group tended to share
  resources earlier in the week
• No big difference between total number of shared
  articles across the two groups
• In both groups, the users attitude towards the
  quality of the articles were generally neutral
• No information overload problem the overall
  contribution did not exceed the community need
• The users in the test group were more active in
  terms of logging on the system and reading
  articles sustainability

20
Discussion/Limitations

• Choosing the Parameters for the mechanism
• Expected sum of contributions for each topic
• Threshold for reputation values
• Community reward function
• Narrow scale of rating (1 or -1)
• Less cognitive load
• No option to rate different aspect of a paper
• Popularity of topic, originality, interestingness
  
• Measure of quality
• Average rating means average taste rules
• In educational context tends to be superficial,
  easy to read articles

21
Discussion/Connection to Our Work

• Trying personalized motivational message in
  CourseAgent or CoPE
• Good evaluation questions
• Some people are easy to be motivated by glory
  and recognition
• Is there any cognitive tool to measure this?
• Compare Comtella with CoPE
• CoPE has the option to write summary and read
  summary written by others
• More personal benefit