AntiPhishing Phil: The Design and Evaluation of a Game that Teaches People not to fall for Phish - PowerPoint PPT Presentation


PPT – AntiPhishing Phil: The Design and Evaluation of a Game that Teaches People not to fall for Phish PowerPoint presentation | free to view - id: 24ca67-ZDc1Z


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation

AntiPhishing Phil: The Design and Evaluation of a Game that Teaches People not to fall for Phish


Three learning science principles applied to the game. Reflection principle ... one common strategy consisted of checking whether the web site was designed ... – PowerPoint PPT presentation

Number of Views:105
Avg rating:3.0/5.0
Slides: 30
Provided by: c1sdo


Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: AntiPhishing Phil: The Design and Evaluation of a Game that Teaches People not to fall for Phish

Anti-Phishing Phil The Design and Evaluation of
a Game that Teaches People not to fall for Phish
  • Steve Sheng, Bryant Magnien, Ponnurangam
  • Alessandro Acquisti, Lorrie Faith Cranor, Jason
    Hong, Elizabeth Nunge
  • Presented by Corey White

  • Introduction
  • Background/Related Work
  • Design of Anti-Phishing Phil
  • Evaluation Methodology
  • Results
  • Effect of Training Learning versus Increasing
  • Conclusions and Future Work

  • What is Phishing?
  • Phishing is an attack in which criminals use
    spoofed e-mails and fraudulent web sites to trick
    people into giving up personal information.
  • Phishing is part of a known class of attacks
    known as semantic attacks. Semantic attacks take
    advantage of the way humans interact with
    computers or interpret messages. Also exploiting
    differences between the system model and the user
  • Anti-Phising Phils goal is to provide users with
    knowledge and information that will help them
    avoid phishing web sites.

Background and Related Work
  • Phishing falls into three categories
  • Why do people fall for phishing?
  • Lack of awareness of vulnerabilities or a defense
  • Ignorant in how to handle unfamiliar risks.
  • Ignoring of phishing clues and security warnings
  • Tools to protect people from phishing
  • Anti-phishing services
  • Development tools and browser add-ons (e.g.
    Trust Bar, PassPet and WebWallet)
  • Anti-phishing education
  • Well designed phishing security training is
    proven to be effective
  • Contextual training (users handle simulated
    phishing attacks)
  • Embedded training.(trains while user does regular
    email use.)

Design of Anti-phishing Phil
  • The developers objective of the game
  • How to identify phishing URLs
  • Where to look for cues for trustworthy or
    untrustworthy sites in web browsers
  • How to use search engines to find legitimate
  • The game is goal-oriented, challenging,
    contextual, and interactive.
  • Training is most effective if the material is in
    context, interactive, and relatable to the users.
  • Conceptual and procedural knowledge
  • conceptual knowledge is knowledge about concepts
    or relationships that can be expressed as
    propositions (e.g. there are protocol and domain
    name parts of a URL).
  • Procedural knowledge is step-by-step knowledge to
    solve a given problem.(e.g. check the address bar
    to see if it contains an IP address which signals
    a phishing site.)

Design of Anti-phishing Phil continued
  • Three learning science principles applied to the
  • Reflection principle
  • Opportunity for learners to reflect on the new
    knowledge they gained after every round showing
    results of correct and incorrectly identified web
  • Story-based agent environment principle
  • Agents help guide learners through the learning
    game. Phil in this game is controlled by the user
    through a anti-phishing survival story which
    are organized in a stimulating cognitive reading
  • Conceptual-Procedural principle
  • Conceptual and procedural knowledge influence
    each other to build an iterative process of

Design of Anti-phishing Phil continued
  • Game Description The game is described in three
  • Story
  • A fish named Phil who has to eat real
    worms(legitimate URL web sites) and reject all
    bait (phishing URLs) before time ends in the
    rounds. Phils father also provides tips
    (training messages) to help out Phil.
  • Mechanics
  • 4 rounds
  • two minute rounds
  • 8 worms a round containing concealed real or
    faulty URLs
  • 100 points for correctly eaten or rejected worms,
    -10 seconds off the clock for rejecting a good
    worm(false positive).
  • Severe penality for eating a bad worm and being
    caught by phishers (false, negative) and loses
    one of his three lives
  • The player must identify six URLs correctly to
    advance to the round, if they still have lives
    they can repeat until completing this goal, but
    if they run out of lives the game is over.
  • Technology
  • The game is implemented in Flash 8.

Design of Anti-phishing Phil continued
  • Training Messages
  • What to teach
  • How to identify URLs,
  • Where to look for cues in the web browsers
  • How to use search engines to find legit sites.
  • Where to teach them
  • Feedback during the game
  • Help messages during the game
  • End of round score sheets
  • Anti-phishing tips between rounds

Design of Anti-phishing Phil continued
  • Pilot Test
  • 8 users from Carnegie Mellon University
    participated in a pilot test of the game
  • Results
  • Users looked at the address bar 14 prior to
    playing and 41 after playing the game
  • The false negative rate decreased from 31 to 17
    after playing the game
  • However, the false positive rate increased from
    37 to 48 in part due to misinterpreting URLs
    they examined.

Design of Anti-phishing Phil continued
  • Difficulties Observed
  • many users could not properly parse a long URL
    and did not seem to understand that the most
    important part of the URL is the right hand side
    of the domain name. This led them to
    misidentifying of many URL sites.
  • Misinterpreting of lessons
  • one common strategy consisted of checking whether
    the web site was designed professionally.
    However, this is not a useful strategy as many
    phishing sites are exact replicas of
    professionally designed legitimate sites.

(No Transcript)
(No Transcript)
(No Transcript)
(No Transcript)
(No Transcript)
Evaluation Methodology
  • Study Design
  • Participants were given a scenario of e-mail link
    to tell if it was a legitimate or spoofed website
  • Then the participants were given 20 websites to
    evaluate for legitimate or maliciousness and to
    tell on a scale of 1 to 5 their confidence in
    their choices
  • They were finally given an exit survey at the end
    of the study.
  • The 20 websites were half legit and half phishing
    and they were divided up into two groups that
    were half and half.
  • The participants could use an additional web
    browser and search engines if needed to aid their
    evaluations of the web sites.
  • All phishing web sites were hosted DNS file by
    the developers on a local computer ,so no one was
    actually at risk and if they picked a phishing
    site, they were instantly informed about it.

Evaluation Methodology
  • Between subjects experiment
  • Existing training material condition
  • Participants spent 15 minutes reading tutorials
    on phishing and spoofed emails from eBay,
    Microsoft, etc.
  • Tutorial condition
  • Participants spent 15 minutes reading
    anti-phishing materials based on the
    Anti-Phishing Phil game. It concluded all the
    hints and training messaged and was printed in
  • Game condition
  • Participants played the Anti-Phishing Phil game
    for 15 minutes in this condition.
  • The game was conducted in two phases separated by
    five months and compared to a group that spent 15
    minutes playing solitaire.

Evaluation Methodology
  • Demographics
  • The developer recruited 14 people for each
    condition via flyers posted around campus, and
    with recruitment email on university bulletin
    boards, and on
  • They screened participants with respect to their
    knowledge of computers in general, in-order to
    recruit only participants who could be considered
  • They recruited users who answered no to two or
    more of the following screening questions
  • 1) Whether they had ever changed preferences or
    settings in their web browser
  • 2) Whether they had ever created a web page
  • 3) Whether they had ever helped someone fix a
    computer problem

Participant Demographic
  • Results
  • They found that participants in the game
    condition performed better than the other two
    conditions in correctly identifying the web
  • They also found that there was no significant
    difference in false negatives among the three
    groups. However, the participants in the game
    group performed better overall than the other two
  • A false positive is when a legitimate site is
    mistakenly judged as a phishing site.
  • A false negative is when a phishing site is
    incorrectly judged to be a legitimate site.

False Negative Rate
False negative rates. The existing training
material performed best on false negatives.
However, the difference is not statistically
False Positive Rate
False Positive Rate. The false positives
increased in the existing materials condition,
and decreased in both the tutorial and game
condition, with the game condition showing the
highest reduction.
Total Correctness
Total correctness for the test groups. The game
condition shows the greatest improvements.
Results continued
  • User Confidence rating
  • Users became more confident about their decisions
    after the game or tutorial conditions, but no
    improvement on existing training material
    improving user confidence in a significant way.
  • The game condition increased user confidence
    rating from 3.72 to 4.42. The overall average
    confidence rating was 4.18 pre test to 4.32 post

Results continued
  • User Feedback
  • 93 of the users either agreed or strongly agreed
    that they had learned a lot, and 100 of them
    agreed or strongly agreed that they had learned a
    lot of important information.
  • On a five point scale, they were also asked to
    rate the educational and fun levels of the game.
    93 of the user felt the educational value of the
    game was very good or excellent, 50 of the users
    considered the fun level of the game as very good
    or excellent.
  • Similar questions were asked about educational
    value and fun level in the existing training
    material condition. 93 percent of the users also
    felt the educational value of the existing
    training material was very good or excellent ,
    where as only 29 percent of the users considered
    the fun level of the existing training materials
    to be very good or excellent.

Results continued
  • Where the game is failing
  • The developers showed users a PayPal website with
    the address bar spoofed. 6 of the users in the
    game condition were unable to identify this
    attack in the post test, whereas only 3 users in
    the existing training material condition fell for
    it. This is could be because users are more prone
    to this kind of attacks because, after the
    training, they look specifically for clues in the
    URL, and if the clues confirm their assumptions,
    they dont look further. (current browsers now
    address this kind of problem).
  • 2 users also fell for the similar domain attack
    after the game condition, in which they showed
    them for account updates. This is an
    easy attack to identify if users notice the large
    amount of information requested, because of this
    reason, none of the users fall for it in the pre
    test. This problem brings up two issues first,
    some users still have problems with phishing
    domains that are similar to the real ones
    second, they tend to look less for other clues
    other than the URL, and if the URL does not raise
    suspicion, they do not look further.

Effect of training Learning versus
increasing alertness
  • Signal Detection Theory
  • SDT helps to gain insights into whether the game
    educated users about detecting phishing websites
    or increased their alertness.
  • SDT quantifies the ability to discern between
    signal (phishing websites) and noise (legitimate
    websites). To gain this insight, two measures are
    used sensitivity (d) and criterion (C).
  • Sensitivity is defined as how hard or easy it is
    to detect if that target stimulus is present from
    a background event. Which is the distance between
    the mean of signal and noise distributions. The
    larger the parameter (d), the better is the user
    at separating the signal from the noise.
  • Results from the Signal Detection Theory analysis
    showed that users had a greater sensitivity with
    Anti-Phishing Phil, meaning that they were better
    able to distinguish between phishing and
    legitimate sites. Also users were able to make
    better decisions in the game condition compared
    to the users becoming conservative in the other

Conclusions and future work
  • The authors objective in developing the
    anti-phishing game was to teach users (1) how to
    identify phishing URLs, (2) where to look for
    cues in web browsers, and (3) how to use search
    engines to find legitimate sites. It also teaches
    users about identifying three types of phishing
    URLs IP based URLs, sub domain, and deceptive.
  • They found that participants who played the game
    were better at identifying phishing websites than
    those who completed the two other types of
    training. Using signal detection theory, showed
    that while existing online training materials
    increase awareness about phishing ,however the
    game condition makes users more knowledgeable
    about techniques they can use to identify
    phishing web sites.
  • The results show that interactive games can be a
    promising way of teaching people about strategies
    to avoid falling for phishing attacks. The
    results suggest that applying learning science
    principles to training materials can stimulate
    effective learning. Also, the results strongly
    suggest that educating users about security can
    be a standard education.