Introducing of handwritten icons

1
Introducingof handwritten icons
August 20th, 2008 ICFHR, Montreal

• Ralph Niels, Don Willems and Louis Vuurpijl

2
Introducing the NicIcon databaseof handwritten
icons
Ralph Niels Don Willems Louis Vuurpijl
Icon
3
Overview
Introducing the NicIcon databaseof handwritten
icons
Ralph Niels Don Willems Louis Vuurpijl

• Introduction
• Domain crisis management
• Data collection
• Icon design
• Method
• Data
• Classification experiment
• Method
• Results
• Conclusion and URL

NEW!
4
Crisis management (CM)
Introducing the NicIcon databaseof handwritten
icons
Ralph Niels Don Willems Louis Vuurpijl

• Scenario tunnel disaster
• Distributed computer systems to support CM
• Multiple modalities speech, gestures, and pen

5
Iconic pen gestures
Introducing the NicIcon databaseof handwritten
icons
Ralph Niels Don Willems Louis Vuurpijl

• Faster than handwriting
• Easy to learn remember
• Visual meaningful shape

6
Database
Introducing the NicIcon databaseof handwritten
icons
Ralph Niels Don Willems Louis Vuurpijl

• Public databases available for several
  handwriting applications
• Not for iconic pen gestures
• Based on symbology reference by USA Homeland
  Security Workgroup
• Used in e.g., USA, Australia, New-Zealand

7
Icons
Introducing the NicIcon databaseof handwritten
icons
Ralph Niels Don Willems Louis Vuurpijl
8
Data collection
Introducing the NicIcon databaseof handwritten
icons
Ralph Niels Don Willems Louis Vuurpijl

• 35 participants
• Online and offline
• Variation in size
• Per person
• 22 pages
• 55 instances / icon

9
Database content
Introducing the NicIcon databaseof handwritten
icons
Ralph Niels Don Willems Louis Vuurpijl

• Online 26,163 iconic gestures(24,144 reported
  in paper)

Tablet Wacom Intuos2 A4 oversize
10
Database content
Introducing the NicIcon databaseof handwritten
icons
Ralph Niels Don Willems Louis Vuurpijl

• Offline 770 scanned pages

Scanner HP Scanjet 7400C flatbed (_at_ 300dpi, 24
bit colour)
11
Classification experiment
Introducing the NicIcon databaseof handwritten
icons
Ralph Niels Don Willems Louis Vuurpijl

• Data sub sets
• Stratified
• Writer dependent (WD), writer independent (WI)

Complete data set (26,163 icons)
Evaluation set(40)
Train set(36)
Test set (24)
12
Classification experiment
Introducing the NicIcon databaseof handwritten
icons
Ralph Niels Don Willems Louis Vuurpijl
on online data

• Multi classifier system
• Feature sets
• 28 geometric features (Willems Vuurpijl)
• 30/60 coordinates running features (Schomaker
  Vuurpijl)
• 1185 features (Willems, Niels, Van Gerven,
  Vuurpijl)
• Feature classifiers
• Support Vector Machine
• Multi-Layered Perceptron
• Template matching
• Dynamic Time Warping (Niels Vuurpijl)

13
New feature set (m-fs)
Introducing the NicIcon databaseof handwritten
icons
Ralph Niels Don Willems Louis Vuurpijl

• Features from literature
• Geometrical, temporal, pressure

D. Rubine, Specifying gestures by example,
Computer Graphics. 25 (4) (1991) 329337. J.
LaViola Jr. R. Zeleznik, A practical approach
for writer-dependent symbol recognition using a
writer-independent symbol recognizer, IEEE
Transactions on pattern analysis and machine
intelligence. 29 (11) (2007) 19171926. L.
Zhang Z. Sun, An experimental comparison of
machine learning for adaptive sketch recognition,
Applied Mathematics and Computation. 185 (2)
(2007) 11381148. and many others
14
New feature set (m-fs)
Introducing the NicIcon databaseof handwritten
icons
Ralph Niels Don Willems Louis Vuurpijl

• Over complete icon, but also
• Mean over strokes
• Std. dev. over strokes

Feature definitions in technical report at website
15
New feature set (m-fs)
Introducing the NicIcon databaseof handwritten
icons
Ralph Niels Don Willems Louis Vuurpijl

• Feature selection
• 1185 features, which are the best?
• Sort them on best individual performance
• Add 1 by 1 to classifier
• Performance maximizes at
• 545 features for WI
• 660 features for WD
• Selected features
• 1/3 full icon
• 1/3 mean
• 1/3 standard deviation

The best features
16
Breaking news the best features
Introducing the NicIcon databaseof handwritten
icons
Ralph Niels Don Willems Louis Vuurpijl
Averagecentroidal radius
Sine first /last sample
Length ofdiagonal
Verticaloffsets
Area
17
Classification results
Introducing the NicIcon databaseof handwritten
icons
Ralph Niels Don Willems Louis Vuurpijl
g-28 28 geometric features(Willems
Vuurpijl) m-fs 1185 features(Willems, Niels,
Van Gerven, Vuurpijl) af-30/af-60 30/60 running
features (Schomaker Vuurpijl)DTW Dynamic
Time Warping(Niels Vuurpijl) MCS Multiple
classifier system (majority voting)
18
Misclassifications
Introducing the NicIcon databaseof handwritten
icons
Ralph Niels Don Willems Louis Vuurpijl

• Wrong box
• Sloppy drawing
• Retracing

19
Discussion
Introducing the NicIcon databaseof handwritten
icons
Ralph Niels Don Willems Louis Vuurpijl

• Results already quite good (WD 99.5, WI
  97.8),but gain is still possible
• Which features are important?
• For different domains
• Performance in interactive experiment
• Offline data still open
• Mapping online -gt offline

20
http//unipen.nici.ru.nl/NicIcon
Introducing the NicIcon databaseof handwritten
icons
Ralph Niels Don Willems Louis Vuurpijl

• Freely available
• Online (Unipen)
• Offline (PNG)
• Technical report about new feature set