Visualization and Interaction ----------------------- towards intelligent scientific media

1
Visualization and Interaction--------------------
--- towards intelligent scientific media

• Jürgen Richter-Gebert
• Technical University Munich

2
Interacting with Computers
The problem with software is not that it needs
a good user interface, it needs
to have no user interface.
Randal Davis (2002), in a Position Statement of
Sketch Recognition at MIT
3
Input devices Keyboard
Keyboard very precise input, everything must
be sequentialized no parallel geometric
perception
Consequences Programs are written with
keyboards Programmed interfaces look static
4
Input devices Mouse
Mouse precise pointing device, positions can
be clicked exactly start/end-positions are
precise paths are difficult to control.
Consequences Easy positioning of items. Study
of Objects in start/end situations. Items to
interact with have to be present.
5
A shift of paradigms Pen

• Pen driven devices
• precise pointing,
• sensomotoric supports correct drawing
• difficult to change keyboard/pen
• Consequences
• Intuitive input
• Too intuitive input
• Artistic, creative, conceptual
• creating objects on the fly

6
Usage of pens
Pen Displays
PDAs
Artistic use Keyboard available Big screen
Small displays Making notes Assistant
Interactive Whiteboards
Tablet PCs
Must be reliable No keyboard Must be flexible
No keyboard Mobile use In-House use The future ?
7
Sketching
... informal, private drawings that
architectural designers mathematicians,
engineers, physicists,... use as a
medium for graphic thinking in the
exploratory stages of their work.
Dan Herbert, 1987
8
A personal story
Cinderella - the interactive geometry Software
1992-1995 Interactive proving
projective geometry (NeXT-Step)
1996-1999 Foundations of dynamic Geometry
serious implementation (Java) randomized
proving interactive exercises
2000-200? Interplay with other programs
more geometry (transformations/IFS)
physics simulation pen support
9
...and, Lecturing
10
...and, Lecturing
11
...and, Lecturing
12
The ultimate goal
A pen driven assistant that ... ...can be
passive as a sheet of paper, ...detects
inherent semantics of drawings, ...allows to
interact correctly with diagrams, ...knows
cultural backgrounds, ...is adaptable, ...enha
nces thinking,...
13
In this talk

• Brief overview of Cinderella
• Pen support for geometric diagrams
• Physics simulation (behaviours)
• Recognition of diagrams
• Stroke interpretation
• Micro semantics
• Graphic parsing
• A vision of advanced interaction techniques

14
Cinderella

• Dynamic Geometry
• General Methods
• Complex Analysis
• Automatic prover
• Java Based
• Multi View

Currently Transformations, Fractals, Physics,
Scribbling.
15
Principles
Beautification Report the recognition by
making the sketch better Create exact drawings
from sketches Fuzzyfication/Uglyfication Let
things appear conceptual Create appealing but
functional sketches Correctification Use
computer to give additional information Make
correct Pictures
16
UI Issues

• Make all your rules transparent
• Give instant feedback
• Provide easy UNDO
• Use cultural background
• Observe yourself

Intelligent interfaces
? Intelligent users
17
Diagram Types I
Tables
Knots/Weaving Diagrams
Easy adjustment of rows/columns/cells
Provide an easy way to draw crossings
18
Diagram Types II
Electric, Logic, Arithmetic Circuits
Evaluation on demand. Syntax highlighting
19
Diagram Types II
Electric, Logic, Arithmetic Circuits
Evaluation on demand. Syntax highlighting
20
Diagram Types III
Graphs

• Very Important
• Many things are graphs
• Flow charts, Circuits,
• Pipe to other programs

21
Diagram Types IV
User Interfaces, UML diagrams,
The SILK project (Berkeley, Carnegie Mellon)
22
Recognition process

• Stroke segmentation
• Subdivide a single stroke into semantic units
• Merge several strokes into a semantic unit
• Collect information
• What are the properties of new strokes
• Position, relative position, shape, speed, timing
• More information than in OCR!!
• Graphics parsing
• Use context sensitive parsing for interpretation
• Need for non-sequential parsing rules
• Parse frequently, use idle time

23
Micro Semantics
Recognition of single-stoke of primitives
24
Programming of behaviours

• 1. Possibility Hard coding
• Good for common cultural primitives
• Very flexible
• 2. Possibility Communication via API
• Good for second party users
• Use other peoples power
• 3. Programming by drawing diagrams
• Very personal adapting
• Good for non programmers
• Absolutely needed

25
Developing a marble run
A Real-world problem Designing a
marble run for a mathematical
exhibition
How to design a marble run that performs some
interesting calculations?
26
The result
A marble run that calculates binomial coefficients
27
How it works
Switches distribute marbles alternatingly
28
Geometric rewriting rules

• Draw replacement rules to define behaviour
• While running search for key-patterns
• Apply the rules

29
Programming the marble run

• Also possible
• Circuits,
• Logical gates,
• Lindenmayer systems,
• Finite automata
• Chip firing games
• Turing machines,
• Hasse diagrams,

Next Step Develop the right concepts for
graphic rewriting rules
30
Thanks for your attention