Programming by Example

1
Programming by Example

• Henry Lieberman
• Media Laboratory
• Massachusetts Institute of Technology
• Cambridge, MA, USA
• lieber_at_media.mit.edu
• http//www.media.mit.edu/lieber
• PBE Web site http//www.media.mit.edu/lieber
  /PBE/

2
Your Wish is My CommandProgramming by Example

• Morgan Kaufmann, 2001

3
Programming should be like teaching

• User plays the role of a teacher
• Computer plays the role of a student

4
Programming by Example

• (Also called "Programming by Demonstration")
• Agent "watches what you do" in an interactive
  interface
• Records sequence of operations, data
• Generalizes program so that you can use an
  analogous procedure in new examples
• "Macros on Steroids"

5
Programming in the User Interface

• The same interface used by end-users can be used
  for programming
• Empowers end-users to do programming
• No switching between user and programmer
  interfaces
• Empowers "visual thinkers"

6
PBE and the "Intelligent Agent" movement

• There's been a recent movement toward
  "intelligent agents" -- proactive, assistive,
  (possibly anthropomorphic) interfaces
• PBE can be used to make agents instructible
• Agent personalities
• Butler/Servant/Secretary
• Teacher/Student, Master/Apprentice

7
Issues in Programming by Example

• The Generalization Problem ( Data Description)
• How do I describe actions and data?
• How similar do the actions/data have to be next
  time?
• Feedback
• How does the system show the user what it's
  learned?
• Advice
• How can the user influence or modify what the
  system learns?

8
Issues in Programming by Example

• How should we record the user's actions?
• At what granularity should things be recorded?
• Can the user interact with the agent during
  recording? If so, how?
• Do we represent the agent directly in the
  interface?
• Do we anthropomorphize the agent?
• Can users give advice during recording?
• Turning recording on and off

9
Issues in Programming by Example

• How should the recorded program be represented?
• Should the user see the recorded program?
• What should it look like?
• Should the user interact directly with the
  recorded program? If so, how?
• How to represent examples?
• How to represent generalizations?

10
Issues in Programming by Example

• How should the user edit an already-recorded
  program?
• How do we keep the representations consistent
  during editing?
• Can the user edit previous generalizations?
• How can users debug programs?
• Can users step/trace/break programs?

11
Machine learning in Programming by Example

• Lots of work in machine learning
• Careful Watch for hidden assumptions
• Few examples, learns fast, explainable
• Explanation-based Generalization
• Case Based reasoning
• Lots of examples, slow but more robust
• Neural networks, PAC learning, data-mining

12
Users of the world, unite!

• You have nothing to lose but your error messages!
• Almost "political" message of empowerment of
  users
• Users should not be at the mercy of software
  providers
• User procedures and data should not be held
  hostage by software companies

13
"Poor man's"Programming by Example

• Macro recorders in applications, AppleScript
• Emacs keyboard macros
• Interface builders / Scripting languages
• Quickeys / OneClick

14
Hooking PBE systems to conventional applications

• PBE systems would like to use conventional
  interfaces as tools rather than re-implement apps
• Conventional interfaces aren't set up to be
  operated by an external program
• Issues Access to application data, sharing the
  interface, parallelism
• Article Lieberman, IUI 98 "Attaching Interface
  Agents to Conventional Applications"

15
Watch What I Do

• Allen Cypher, ed. MIT Press 1993

16
A Small Matter of Programming

• Bonnie Nardi, MIT Press, 1994.
• End-User Programming
• Scripting languages
• Application-specific languages
• "Gardeners"

17
Eager - Allen Cypher

• Programming by example for Hypercard
• Notices patterns in user actions
• Anticipation highlighting of predicted actions
• Anthropomorphization of learning agent
• Example Collect subject lines from messages

18
Eager
19
Mondrian An Instructible Graphical Editor

• Mondrian
• An object-oriented graphical editor
• An agent that records user interface actions
• Learning and generalization heuristics

20
Mondrian An Instructible Graphical Editor
21
Wizard of Oz Experiment

• Human simulates agent
• What kinds of communication between human and
  agent are plausible?
• What kind of knowledge does the agent need to
  know to perform the task?

22
Wizard of Oz experiment task

• Reformat a Bibliography
• Lieberman, H., The One True Path to Intelligent
  Agents, International Conference on Buzzwords and
  Hype, Tahiti, 1996.
• Lieberman 96 The One True Path to Intelligent
  Agents, Int. Conf. on Buzzwords and Hype, Tahiti,
  (1996).
• Oz, Wizard Of, Pay No Attention to the Man Behind
  the Curtain, Journal of Smoke and Mirrors, August
  1942, Emerald City, Oz.

23
Wizard of Oz results

• Users can make mistakes
• Information can be incomplete
• Agents can ask the user if they encounter an
  error situation
• Agent could ask the user to supply missing
  information
• Agent could ask the user to edit the example
• Skip steps that don't work

24
Wizard of Oz experiment method

• User performs task once
• Thinking aloud goals and methods
• Agent (secretly) constructs description of
  actions
• In terms of simple implementable operations
• Agent offers help
• Agent performs help according to action
  description
• Agent gives user feedback
• Agent accepts advice from user

25
Survey of "Your Wish" systems

• Astonishingly wide variety of application
  domains
• Kids, Games, CAD, GIS, Text editing, Graphical
  editing, Web browsing, Animation, Databases, etc.
• Illustrates variety of design decisions in PBE
  systems
• Different reps for programs, agents, etc.
• How much inference, generalization, initiative?

26
Stagecast Creator - Smith, Cypher

• Programming for kids, grid game world
• If you see this pattern of squares, replace it
  with that pattern of squares
• Viz rep of rules, editor/stepper
• Generalization on object types, introduce
  variables
• Commercially available, success with thousands of
  users

27
A Stagecast Creator Rule
28
A Stagecast Creator Game
29
Toontalk - Ken Kahn

• Programming language is a video game
• Most radical programming model
• Concurrent logic programming
• Abstraction by removing detail
• Commercially available, success with kids in
  schools

30
Toontalk's Visual Language
31
Generalizing by Removing Detail
32
Toontalk Sort program
33
How much intelligence?- Brad Myers, Rich McDaniel

• CMU group one of most active and consistent in
  PBE
• Multitude of systems, "gems" of PBE
• Peridot, Lapidary, Jade, Gilt, Tourmaline, C32,
  Pursuit, Gold, Marquise, Katie, Turquoise, Topaz,
  Gamut
• Experience with range of conservative to radical
  PBE systems
• "Spectrum of Instructibility"
• Tradeoff between conveience and control

34
Pursuit's Visual Language
35
Selecting Generalizations in Marquise
36
Internet Scrapbook - Sugiura

• Everybody's favorite application area - the Web!
• Automates Browse Cut Paste applications
• How do you describe a Web page selection?
• Partial matching, Web design heuristics
• Commercially available - preloaded on NEC PCs

37
Internet Scrapbook
38
TriAS - Bauer, Meyer, Dengler, Paul

• Automating Browse Copy Paste, like Scrapbook
• Collaborative, mixed-initiative dialog
• At any time, agent or user can propose
  selections, generalizations
• What Next? Can I Stop?
• Integrates wrapper description language HyQL,
  InfoBeans dynamic Web sources

39
Collaborative Wrapper Definition
40
PBE for GIS CSprl - Traynor, Williams

• Study of how end users use Geographical
  Information Systems
• "Gardeners", "Surrogate Users"
• Comic-strip PBE language

41
A Geographical Info Systems PBE Program
42
Visual Language for Geographic Information Systems
43
PBE for Computer Aided Design -Girard, Pierra

• True "Industrial Strength" PBE, probably the
  champ
• CAD users are sophisticated in application
  domain, willing to specify a lot
• Provides complete editing and debugging
• Key issues Naming, Ambiguity in spec/solutions,
  general control structures conditionals, loops

44
Example-Based Programming in Computer-Aided
Design
45
Demonstrating "Hidden Features" - McDaniel, Myers

• PBE for animated computer games
• WYSIWYG - "What you get is all you have"
• Demonstrate "hidden features" by reifying them
• Guides, e.g. Arrows for directions
• Sophisticated heuristics, lots of inference
• Stop This!
• Do Something!

46
Demonstrating Hidden Features
47
Report Generation - Masuishi

• Mundane, but hey, could save a lot of time
• Format generalization
• Application-specific iteration
• Distributed to wide user community

48
Report Generation by Example
49
Dynamic Macro - Masui

• Notices loops in text editing commands
• Simple, automatic No start/stop recording
• Wide community of users
• Also POBox predictive input for PDAs, phones,
  etc.

50
Dynamic Macro
51
SmartEdit - Lau, Domingos, Wolfman, Weld

• Automates repetitive text-editing procedures
• Predictive, anticipatory, loop detection
• Uses version-space learning

52
SmartEdit Version Space
53
SmartEdit Screens
54
Grammex "Grammars by Example"- Lieberman,
Nardi, Wright

• Pick out "needles" of structured information in
  "haystack" of unstructured information
• First direct-manipulation user interface to
  powerful grammar and parsing technology
• Show example strings, explain components
  recursively
• Iterative System parses, you explain

55
Defining Recursive Grammars
56
Visual representation of regular expressions -
Blackwell

• SWYN "See what you need"
• Representation of positive and negative examples
• Anticipation of selections updated dynamically
• Representation of grammars as Finite-State
  machines
• User study of understandability of
  representations

57
Visual representation of Regular Expressions
58
APE PBE for Programming Environments - Ruvini,
Dony

• Combining predictive interfaces with PBE
• Conventional programming environment Smalltalk
• Automates coding, debugging procedures
• What-set, When-set use different learning
  algorithms
• C4.5, new algorithm IDHYS can learn
  non-consecutive patterns

59
Repetitive code in protocols
60
Familiar - PBE across applications - Paynter

• PBE for Applescript
• Application-independent, no mods to applications,
  uses commercial OS scripting language
• Limited by shortcomings of Applescript and
  application implementations

61
Familiar arranging files
62
Tatlin - Gaxiola, Lieberman

• Examinability Some applications will let you
  look at their data even if you can't record them
  or use API
• PBE by comparing states and inducing
  transformations rather than recording and
  generalizing actions

63
Tatlin - PBE via Examinability
64
Demonstrating What vs. When- Wolber, Myers

• Saying "when" you want something to happen
• Stimulus-Response PBE
• Pavlov Animation scripting system like Macromind
  Director

65
Pavlov driving game
66
Programming by Analogous Examples - Repenning,
Perrone

• Agentsheets - Grid world like Stagecast Creator
• If-then rules by example
• Make generalizations by analogy to
  generalizations of previous examples

67
Programming by Analogous Examples
68
Visual Generalization - St. Amant, Zettlemoyer,
Lieberman, Potter

• Actually parse the screen!
• Simulate human user's visual system
• Finesses data access problem, hooking to
  applications
• Fragile - if viz rep changes, you're screwed
• Permits unique visual generalizations not
  possible by accessing data

69
Triggers - "Just the pixels, ma'am"
70
Solitaire by "Parsing the screen"
71
Short order programming exercise

• Task Add similarity-based learning to an
  action-based learning system
• Ingredients Graphical editor, action recording,
  relation inference

72
Things to think about

• How could we integrate the paradigms of Mondrian
  and Eager?
• What sort of information do we need to get from
  the graphical editor?
• What should the learning algorithm learn?
• How should the result of learning affect the
  interface?