The Turing Test

1
The Turing Test
Conversational AI and the Loebner Prize
Competition

• Jen Brandner
• for CSCI 405

2
What is a Turing Test

• It all started with A. M. Turings 1950 paper
  Computing Machinery and Intelligence.
• Turing described an imitation game in which a
  man and a woman both try to convince an
  interrogator that he/she is the woman.
• Expands this to a computer convincing an
  interrogator that it is human.

3
What is conversational AI

• Machines are programmed to carry on a
  conversation with the user.
• chatbots
• Requires natural language processing.
• Examples
• ELIZA (the Rogerian psychotherapist)
• AOL Messengers Smarter Child

4
What is the Loebner Prize Contest

• Sponsored by Hugh Loebner
• Annual event held sense 1991
• 1991 10 judges and 8 contestants (6 computers
  and 2 humans)
• Judges had short conversations with each
  contestant and rated their human-ness.
• To give the computers a fighting chance
  contestants were allowed to select a single topic
  to converse on.

5
Results of the First Contest in 1991

• Five judges rated the top contestant as human.
• Eight cases in which a computer was misclassified
  as human.
• Winning programmer Joseph Weintraubs program PC
  Therapist III
• His topic whimsical conversation
• Relied on non sequiturs in conversation
• Awarded 1500

6
The 1996 Contest

• Jason Hutchens entered two programs
• HeX (primary entry)
• MegaHAL
• HeX was a simple one-month hack. Hutchenss
  intent was to show the futility of Loebners
  contest.
• If I can beat those other systems with a program
  which took only a month to make then there is
  something wrong with the way the contest is
  structured. - Jason
• HeX was more complex than MegaHAL and actually
  used MegaHAL as just a part of its programming.
• Hutchenss HeX won the contest in 1996 but
  neither of his creations won again after that
  year.

7
HeXs Algorithm

• Iterate roughly in this order
• Parse sentences one-by-one convert to words.
  Look for keywords in a database of hardwired
  replies (and use one only if hadnt been used
  before).
• If a stored reply could not be located evaluate
  for a trick question and if detected give a
  witty reply.
• Call MegaHAL and generate psychobabble.
• Reformulate the users input according to one of
  several hundred templates and spit it back.
• Give a humorous response to silence.
• Accuse the user of being ungrammatical etc.
• As a last resort generate more psychobabble with
  MegaHAL.

8
MegaHALs Algorithm

• Constructs reply sentences using Markov models
  (sophisticated state machines) to predict what
  word should go next in MegaHALs reply based on
  the previous four words in the sentence.
• The information of a word is the surprise it
  causes the Markov model a function of the
  probability of the word
• I(ws) -log2P(ws)
• Read the users input and segment it into an
  alternating sequence of words and non-words.
• From this sequence find an array of keywords and
  use it to generate many candidate replies.
• Display the reply with the highest information to
  the user.
• Use the users input to update the Markov models
  so that MegaHAL can learn from what the user
  types.

9
About HeX and MegaHAL

• Strengths
• HeX was easy to implement (only took one month to
  develop).
• Weaknesses
• MegaHAL sometimes generated sentences that did
  not make sense. Since HeX used MegaHAL in its
  algorithm it had the same problem.
• Just a glorified random sentence generator.

10
Most recent Loebner Prize contest

• Winner in 2005 Rollo Carpenters Jabberwacky
• Uses a unique learning algorithm that stores
  previous conversations and uses them as guides in
  future conversations.
• You can talk to Jabberwacky on the web
• www.jabberwacky.com
• Awarded 3000

11
Jabberwacky learns from what you say and when
you say it. Then if the right moment comes up
some time in the future it says what you said
... and learns what someone ELSE says in
response. So its a giant feedback loop and an
imitator ... if you like its an unusually
clever parrot. To really get how it works you
have to think about it in a rather backwards sort
of way. Theres no programming to make it claim
to be human yet it does so often - because most
of the people speaking to it claim to be human.
And it will often accuse users of being a
robot... - from the website of icogno an AI
company founded by Rollo Carpenter
12
About Jabberwacky

• Strengths
• Learns by experience in past conversations.
• Gets its replies from what humans have said to it
  before so its replies are sure to be human-like.
• Weaknesses
• Requires a huge database to store previous
  conversations.
• Still has no concept of situation (what exactly
  it is talking about).

13
2005 Sample Transcripts

• Transcripts of conversations at 2005 Loebner
  Prize contest
• http//loebner.net/Prizef/2005_Contest/Transcript
  s.html

Excerpt from one of the conversations with
Jabberwacky
PROGRAM Ok then it was nice to share your
hair brush. JUDGE What do mean by that How
have you shared my hair brush PROGRAM
Good. JUDGE Good PROGRAM Yes. JUDGE This has
become a very zen like conversation. PROGRAM
And worse yet I gave it to
charity. JUDGE What WHat did you give to
charity This conversation PROGRAM Er.. 1
million Dollars
HUMAN How is the other entity treating
you JUDGE The other entity just told me
that it was nice to share my hair
brush. HUMAN Share your hair brush You
have a mad perverted bot... JUDGE I dont
even have a hair brush. Otherwise I would
have thought it quite forward indeed.
14
Animated George(Jabberwackys character)

• http//www.biscit.biz/data/s/george_jabberwacky_t
  elevirtual.wmv
• http//www.televirtual.com/movies/george_ITV.wmv

15
References

• Computing Machinery and Intelligence A. M.
  Turing http//loebner.net/Prizef/TuringArticle.ht
  ml provided by Hugh Loebner
• Icogno Icogno Ltd http//www.icogno.com/
• Lessons from a Restricted Turing Test Stuart
  M. Shieber 1993 http//www.eecs.harvard.edu/shie
  ber/Biblio/Papers/loebner-rev-html/loebner-rev-htm
  l.html
• MegaHAL Jason Hutchens http//megahal.alioth.d
  ebian.org/
• Home Page of the Loebner Prize in Artificial
  Intelligence 2003 http//loebner.net/Prizef/loe
  bner-prize.html
• How to Pass the Turing Test by Cheating Jason
  L. Hutchens 1997 http//www.agent.ai/doc/upload
  /200403/hutc97_1.pdf