AntiPhishing Phil: The Design and Evaluation of a Game that Teaches People not to fall for Phish

1
Anti-Phishing Phil The Design and Evaluation of
a Game that Teaches People not to fall for Phish

• Steve Sheng, Bryant Magnien, Ponnurangam
  Kumaraguru,
• Alessandro Acquisti, Lorrie Faith Cranor, Jason
  Hong, Elizabeth Nunge
• Presented by Corey White

2
Outline

• Introduction
• Background/Related Work
• Design of Anti-Phishing Phil
• Evaluation Methodology
• Results
• Effect of Training Learning versus Increasing
  Alertness
• Conclusions and Future Work

3
Introduction

• What is Phishing?
• Phishing is an attack in which criminals use
  spoofed e-mails and fraudulent web sites to trick
  people into giving up personal information.
• Phishing is part of a known class of attacks
  known as semantic attacks. Semantic attacks take
  advantage of the way humans interact with
  computers or interpret messages. Also exploiting
  differences between the system model and the user
  model.
• Anti-Phising Phils goal is to provide users with
  knowledge and information that will help them
  avoid phishing web sites.

4
Background and Related Work

• Phishing falls into three categories
• Why do people fall for phishing?
• Lack of awareness of vulnerabilities or a defense
  strategy
• Ignorant in how to handle unfamiliar risks.
• Ignoring of phishing clues and security warnings
• Tools to protect people from phishing
• Anti-phishing services
• Development tools and browser add-ons (e.g.
  Trust Bar, PassPet and WebWallet)
• Anti-phishing education
• Well designed phishing security training is
  proven to be effective
• Contextual training (users handle simulated
  phishing attacks)
• Embedded training.(trains while user does regular
  email use.)

5
Design of Anti-phishing Phil

• The developers objective of the game
• How to identify phishing URLs
• Where to look for cues for trustworthy or
  untrustworthy sites in web browsers
• How to use search engines to find legitimate
  sites.
• The game is goal-oriented, challenging,
  contextual, and interactive.
• Training is most effective if the material is in
  context, interactive, and relatable to the users.
• Conceptual and procedural knowledge
• conceptual knowledge is knowledge about concepts
  or relationships that can be expressed as
  propositions (e.g. there are protocol and domain
  name parts of a URL).
• Procedural knowledge is step-by-step knowledge to
  solve a given problem.(e.g. check the address bar
  to see if it contains an IP address which signals
  a phishing site.)

6
Design of Anti-phishing Phil continued

• Three learning science principles applied to the
  game
• Reflection principle
• Opportunity for learners to reflect on the new
  knowledge they gained after every round showing
  results of correct and incorrectly identified web
  sites.
• Story-based agent environment principle
• Agents help guide learners through the learning
  game. Phil in this game is controlled by the user
  through a anti-phishing survival story which
  are organized in a stimulating cognitive reading
  framework.
• Conceptual-Procedural principle
• Conceptual and procedural knowledge influence
  each other to build an iterative process of
  learning.

7
Design of Anti-phishing Phil continued

• Game Description The game is described in three
  parts.
• Story
• A fish named Phil who has to eat real
  worms(legitimate URL web sites) and reject all
  bait (phishing URLs) before time ends in the
  rounds. Phils father also provides tips
  (training messages) to help out Phil.
• Mechanics
• 4 rounds
• two minute rounds
• 8 worms a round containing concealed real or
  faulty URLs
• 100 points for correctly eaten or rejected worms,
  -10 seconds off the clock for rejecting a good
  worm(false positive).
• Severe penality for eating a bad worm and being
  caught by phishers (false, negative) and loses
  one of his three lives
• The player must identify six URLs correctly to
  advance to the round, if they still have lives
  they can repeat until completing this goal, but
  if they run out of lives the game is over.
• Technology
• The game is implemented in Flash 8.

8
Design of Anti-phishing Phil continued

• Training Messages
• What to teach
• How to identify URLs,
• Where to look for cues in the web browsers
• How to use search engines to find legit sites.
• Where to teach them
• Feedback during the game
• Help messages during the game
• End of round score sheets
• Anti-phishing tips between rounds

9
Design of Anti-phishing Phil continued

• Pilot Test
• 8 users from Carnegie Mellon University
  participated in a pilot test of the game
• Results
• Users looked at the address bar 14 prior to
  playing and 41 after playing the game
• The false negative rate decreased from 31 to 17
  after playing the game
• However, the false positive rate increased from
  37 to 48 in part due to misinterpreting URLs
  they examined.

10
Design of Anti-phishing Phil continued

• Difficulties Observed
• many users could not properly parse a long URL
  and did not seem to understand that the most
  important part of the URL is the right hand side
  of the domain name. This led them to
  misidentifying of many URL sites.
• Misinterpreting of lessons
• one common strategy consisted of checking whether
  the web site was designed professionally.
  However, this is not a useful strategy as many
  phishing sites are exact replicas of
  professionally designed legitimate sites.

11
(No Transcript)
12
(No Transcript)
13
(No Transcript)
14
(No Transcript)
15
(No Transcript)
16
.
http//cups.cs.cmu.edu/soups/2007/proceedings/p88_
sheng.pdf
17
Evaluation Methodology

• Study Design
• Participants were given a scenario of e-mail link
  to tell if it was a legitimate or spoofed website
• Then the participants were given 20 websites to
  evaluate for legitimate or maliciousness and to
  tell on a scale of 1 to 5 their confidence in
  their choices
• They were finally given an exit survey at the end
  of the study.
• The 20 websites were half legit and half phishing
  and they were divided up into two groups that
  were half and half.
• The participants could use an additional web
  browser and search engines if needed to aid their
  evaluations of the web sites.
• All phishing web sites were hosted DNS file by
  the developers on a local computer ,so no one was
  actually at risk and if they picked a phishing
  site, they were instantly informed about it.

18
Evaluation Methodology

• Between subjects experiment
• Existing training material condition
• Participants spent 15 minutes reading tutorials
  on phishing and spoofed emails from eBay,
  Microsoft, etc.
• Tutorial condition
• Participants spent 15 minutes reading
  anti-phishing materials based on the
  Anti-Phishing Phil game. It concluded all the
  hints and training messaged and was printed in
  color.
• Game condition
• Participants played the Anti-Phishing Phil game
  for 15 minutes in this condition.
• The game was conducted in two phases separated by
  five months and compared to a group that spent 15
  minutes playing solitaire.

19
Evaluation Methodology

• Demographics
• The developer recruited 14 people for each
  condition via flyers posted around campus, and
  with recruitment email on university bulletin
  boards, and on craigslist.com.
• They screened participants with respect to their
  knowledge of computers in general, in-order to
  recruit only participants who could be considered
  non-experts.
• They recruited users who answered no to two or
  more of the following screening questions
• 1) Whether they had ever changed preferences or
  settings in their web browser
• 2) Whether they had ever created a web page
• 3) Whether they had ever helped someone fix a
  computer problem

20
.
Participant Demographic
21
Results

• Results
• They found that participants in the game
  condition performed better than the other two
  conditions in correctly identifying the web
  sites.
• They also found that there was no significant
  difference in false negatives among the three
  groups. However, the participants in the game
  group performed better overall than the other two
  groups.
• A false positive is when a legitimate site is
  mistakenly judged as a phishing site.
• A false negative is when a phishing site is
  incorrectly judged to be a legitimate site.

22
False Negative Rate
False negative rates. The existing training
material performed best on false negatives.
However, the difference is not statistically
significant.
23
False Positive Rate
False Positive Rate. The false positives
increased in the existing materials condition,
and decreased in both the tutorial and game
condition, with the game condition showing the
highest reduction.
24
Total Correctness
Total correctness for the test groups. The game
condition shows the greatest improvements.
25
Results continued

• User Confidence rating
• Users became more confident about their decisions
  after the game or tutorial conditions, but no
  improvement on existing training material
  improving user confidence in a significant way.
• The game condition increased user confidence
  rating from 3.72 to 4.42. The overall average
  confidence rating was 4.18 pre test to 4.32 post
  test.

26
Results continued

• User Feedback
• 93 of the users either agreed or strongly agreed
  that they had learned a lot, and 100 of them
  agreed or strongly agreed that they had learned a
  lot of important information.
• On a five point scale, they were also asked to
  rate the educational and fun levels of the game.
  93 of the user felt the educational value of the
  game was very good or excellent, 50 of the users
  considered the fun level of the game as very good
  or excellent.
• Similar questions were asked about educational
  value and fun level in the existing training
  material condition. 93 percent of the users also
  felt the educational value of the existing
  training material was very good or excellent ,
  where as only 29 percent of the users considered
  the fun level of the existing training materials
  to be very good or excellent.

27
Results continued

• Where the game is failing
• The developers showed users a PayPal website with
  the address bar spoofed. 6 of the users in the
  game condition were unable to identify this
  attack in the post test, whereas only 3 users in
  the existing training material condition fell for
  it. This is could be because users are more prone
  to this kind of attacks because, after the
  training, they look specifically for clues in the
  URL, and if the clues confirm their assumptions,
  they dont look further. (current browsers now
  address this kind of problem).
• 2 users also fell for the similar domain attack
  after the game condition, in which they showed
  them myaol.com for account updates. This is an
  easy attack to identify if users notice the large
  amount of information requested, because of this
  reason, none of the users fall for it in the pre
  test. This problem brings up two issues first,
  some users still have problems with phishing
  domains that are similar to the real ones
  second, they tend to look less for other clues
  other than the URL, and if the URL does not raise
  suspicion, they do not look further.

28
Effect of training Learning versus
increasing alertness

• Signal Detection Theory
• SDT helps to gain insights into whether the game
  educated users about detecting phishing websites
  or increased their alertness.
• SDT quantifies the ability to discern between
  signal (phishing websites) and noise (legitimate
  websites). To gain this insight, two measures are
  used sensitivity (d) and criterion (C).
• Sensitivity is defined as how hard or easy it is
  to detect if that target stimulus is present from
  a background event. Which is the distance between
  the mean of signal and noise distributions. The
  larger the parameter (d), the better is the user
  at separating the signal from the noise.
• Results from the Signal Detection Theory analysis
  showed that users had a greater sensitivity with
  Anti-Phishing Phil, meaning that they were better
  able to distinguish between phishing and
  legitimate sites. Also users were able to make
  better decisions in the game condition compared
  to the users becoming conservative in the other
  condition.

29
Conclusions and future work

• The authors objective in developing the
  anti-phishing game was to teach users (1) how to
  identify phishing URLs, (2) where to look for
  cues in web browsers, and (3) how to use search
  engines to find legitimate sites. It also teaches
  users about identifying three types of phishing
  URLs IP based URLs, sub domain, and deceptive.
• They found that participants who played the game
  were better at identifying phishing websites than
  those who completed the two other types of
  training. Using signal detection theory, showed
  that while existing online training materials
  increase awareness about phishing ,however the
  game condition makes users more knowledgeable
  about techniques they can use to identify
  phishing web sites.
• The results show that interactive games can be a
  promising way of teaching people about strategies
  to avoid falling for phishing attacks. The
  results suggest that applying learning science
  principles to training materials can stimulate
  effective learning. Also, the results strongly
  suggest that educating users about security can
  be a standard education.