Visual Programming Languages ICS 539 Icon System Visual Languages

1
Visual Programming LanguagesICS 539Icon
System Visual Languages Visual Programming,
Chapter 1, Editor Chang, 1990

• ICS Department
• KFUPM
• Sept. 1, 2007

2
Theory of generalized icon

• Based on work by Chang
• Definitions
• Icon (generalized icon) An object with the dual
  representation of
• logical part (the meaning)
• Physical part (the image)
• iconic system structured set of related icons
• iconic sentence (visual sentence) spatial
  arrangement of icons from iconic system

3
Theory (con.)

• visual language set of iconic sentences
  constructed with given syntax and semantics
• syntactic analysis (spatial parsing) analysis of
  an iconic sentence to determine the underlying
  structure
• semantic analysis (spatial interpretation)
  analysis of an iconic sentence to determine the
  underlying meaning

4
Theory (con.)

• A visual Language Models an Icon System.
• In the Icon System, a program (Visual Sentence)
  consists of a spatial arrangement of pictorial
  symbols (elementary icons).
• The spatial arrangement is the two-dimensional
  counterpart of the standard sequentialization in
  the construction of programs in the case of
  traditional programming languages.

5

• In traditional languages, a program is expressed
  by a string in which terminals are concatenated.
• As this operation (concatenation) is the only
  construction rule, it does not appear explicitly
  in the vocabulary of the language grammar.
• In the case of visual languages, three
  construction rules are used to spatially arrange
  icons
• Horizontal concatenation
• Vertical concatenation
• Spatial overlay

6

• For this reason, there is a need to include the
  spatial operators and elementary icons in the
  vocabulary of terminals of the visual language
  grammar.
• The elementary icons are partitioned into
• object icons
• process icons.
• Elementary object icons identify objects while
  process icons express computations.
• The meaning depends on the specific visual
  sentence in which it appears.

7
Example

• Consider the following primitive icons from the
  Heidelberg icon set.
• where
• the square denotes a character, and its
  interpretation is unique in the system.
• the arrow can be used to express insertion or
  moving operations in different visual sentences.

8

• A Visual Language (VL) is specified by the triple
  
• (ID, Go, B)
• where
• ID is the icon dictionary
• Go is a context-free grammar
• B is a domain-specific knowledge base.

9
The icon dictionary ID

• It is a set of triples of the form
• i (il , ip , type(i))
• Each triple i describes a primitive icon where
• il is the icon name (logical part or meaning)
• ip is the icon sketch (physical part)
• type(i) is the icon type (process or object)

10
The grammar Go

• It is a context-free grammar (N, T, S, P).
• Where
• N is the set of nonterminals
• T Tl U Tsp
• Tl il i ( il, ip, type(i)) ? ID
• Tsp , ,
• S is the start symbol of the grammar
• P is the set of productions of Go

11

• The grammar Go specifies how more complex icons
  can be constructed by spatially arranging
  elementary icons.
• Usually, nonterminals in Go the start symbol
  represent composite object icons.
• Composite object icons can be derived in one or
  more steps starting from a given nonterminals, N,
  which is different from S.
• In other words, composite object icons are
  obtained by spatial arrangement of elementary
  object icons only and are assumed to be of type
  object.

12

• Visual Sentences contain at least one process
  icon.
• Visual Sentences have no type.
• Here, object icons denote both elementary object
  icons composite object icons.
• It is easy to prove that all the icons involved
  in the icon system (elementary object, composite
  object, and visual sentences) are objects with
  dual representation of a logical and a physical
  part as follows

13

• Elementary object --- by definition in ID.
• Composite object and visual sentences ---
• The physical part is their image
• The logical part (the meaning) can be derived
  from the meaning of the elementary objects
  occurring in them.
• The meaning of a visual sentence is expressed in
  terms of a conceptual tree, which represents an
  intermediate code for later execution.

14
Conceptual Trees

• The meaning of a visual sentence is expressed in
  terms of a conceptual tree (CT), which represents
  an intermediate code for later execution.
• In CT notations a concept is denoted by ,
  and a relation is denoted by ( ).

15

• The concepts are expressed in the form
• OBJECT object-name
• EVENT event-name
• where
• object-name represents the logical part (meaning)
  of an object icon (elementary icon or composite
  object icon).
• event-name are event names of a process icon,
  i.e., procedure names, which accomplish the task
  expressed by the visual sentence.

16

• The following CTs are basic substructures
  occuring in all the CTs which represent the
  logical part of a visual sentence.
• EVENT event-name ? (rel1) ? OBJECT
  object-name1 ? (rel2) ? OBJECT
  object-name2
• EVENT event-name ? (rel) ? OBJECT
  object-name
• OBJECT object-name ? (rel) ? OBJECT
  object-name
• The meanings of these CTs are as follows

17

• 1) A process icon has two object icons as
  arguments
• Example
• consider the following icon from the Heidelberg
  icon set

18

• It is composed of the object icons
• Selected-string
• String
• and the process icon Up-arrow

19

• Its meaning is expressed by the CT
• EVENT replace (object) OBJECT
  String
• (place) OBJECT Selected-string

20

• 2) A process icon requires only one argument.
• Example

21

• It is obtained by composing in spatial overlay
• the object icon Selected-string
• and the process icon cross

22

• Its meaning is described by the CT
• EVENT delete (object) OBJECT
  Selected-string

23

• 3) Here we have two combined object icons. This
  is the case in which one of the two objects
  involved in the visual sentence represents a
  qualitative or quantitative specification of the
  other object.
• Example
• The object "name identifies the object "file"

??????????? ??????????? ???????????
File name
24

• The meaning is
• OBJECT file (identifier)
  OBJECT name

25

• Complex structures can be obtained by composing
  the basic substructures as shown below
• 1 2 3
• This visual sentence could mean append file1 to
  file2 and copy the result into file3

??????????? ??????????? ???????????
??????????? ??????????? ???????????
??????????? ??????????? ???????????
26

• The sentence could be decomposed into two
  subsentences
• 1 2
• and
• 2 3

??????????? ??????????? ???????????
??????????? ??????????? ???????????
??????????? ??????????? ???????????
??????????? ??????????? ???????????
27

• These subsentences are described, respectively,
  by the following CTs
• EVENT append (object) OBJECT
  file1
• (place) OBJECT file2
• EVENT copy (source)
  OBJECT file2
• (destination) OBJECT file3

28

• The whole visual sentence is described by
• EVENT copy (destination) OBJECT
  file3
• (source) EVENT append
  
• (object) OBJECT
  file1
• (object) OBJECT file2

29
The Knowledge Base B

• The Knowledge Base B contains domain-specific
  information necessary for constructing the
  meaning of a given visual sentence.
• It contains information regarding
• Event names
• Conceptual relations
• Names of the resulting objects
• References to the resulting objects.

30

• It is structured in the following seven tables
• 1) EVENT-NAME proc-name, obj-name1, obj-name2,
  op1, op2 This table returns the name of the
  event (procedure) to be associated to the process
  icon proc-name when the objects obj-name1,
  obj-name2 are spatially arranged (via op1, op2)
  to proc-name.
• 2) LEFT-REL proc-name, obj-name1, obj-name2,
  op1, op2
• 3) RIGHT-REL proc-name, obj-name1, obj-name2,
  op1, op2
• These two tables hold the conceptual relations
  existing between EVENT-NAME Proc-name,
  obj-name1, obj-name2, op1, op2 and obj-name1 or
  obj-name2 respectively.

31

• 4) RESULT-NAME Proc-name, obj-name1, obj-name2,
  op1, op2
• This table returns the name of the object
  resulting from the execution of the event.
• EVENT-NAME Proc-name, obj-name1, obj-name2,
  op1, op2. With arguments obj-name1 and
  obj-name2.
• 5) RESULT-REF Proc-name, obj-name1, obj-name2,
  op1, op2.
• This table returns a reference to the object
  representing the result.

32

• The following two tables take into account cases
  in which object icons are composed (composite
  object icons).
• 6) RELATION obj-name1, obj-name2, op.
• This returns the conceptual relation existing
  between obj-name1 and obj-name2 when they are
  spatially combined by means of the operator op.
• 7) RESULT-OBJECT obj-name1, obj-name2, op.
• This returns the name of the object resulting by
  the spatial combination (via op) of obj-name1 and
  obj-name2.
• In the previous tables, obj-namei refer to the
  logical part of the icons.

33

• Besides domain-specific relations the tables can
  also contain three special relations
• ltnewgt This relation means that the combination
• obj-name1 op obj-name2
• gives rise to a new object, different from
  obj-name1 and obj-name2.
• ltnullgtThis relation means that the combination
• obj-name1 op obj-name2
• generates an object that has the same logical
  part of obj-name1 or obj-name2.
• In other words, it points out that no
  additional information is derived from such a
  combination.

34

• ltl-objgt , ltr-objgt
• These two relations mean that the combination
• obj-name1 op1 proc-name op2 obj-name2
• generates an object that has the same reference
  of the icon whose name is obj-name1 or obj-name2
  respectively.

35
Example
file3
??????????? ??????????? ???????????
??????????? ??????????? ??????????? file2
Obj-name1
Op1
Obj-name2
Op2
Proc-name
36

• Table 1 returns the name of the event copy
• Table 2, 3
• LEFT-REL
• EVENT copy (source)
  OBJECT file2
• (destination) OBJECT file3
• RIGHT-REL

37

• Table 4 Return file 3
• Table 5 Return a reference (pointer) to file3
• Table 6
• OBJECT file (identifier)
  OBJECT file3
• returns this
• Table 7 Return the name of the resulting object
  in 6

38
The Attribute Grammar

• An attribute grammar consists of a semantics
  rules for synthesizing the meaning of a visual
  sentence
• AG consists of an underlying context-free
  grammar where
• i) each nonterminal in the context-free grammar
  has associated two attribute sets, namely,
  synthesized and inherited attributes.

39

• ii) each production has associated a set of
  semantics rules, used to determine the values of
  the attributes of the nonterminals depending on
  the values of the attributes of the remaining
  nonterminals appearing in the same production.
• The initial nonterminal has only synthesized
  attributes, and one of them is designated to hold
  the meaning of the derivation tree.

40

• The production rules in Go are of the following
  form
• a) X ?t t ? T
• b) X Y where type (X) type (Y)
• c) X M op L L op M M1 op M2
• d) X M1 op1 L op2 M2
• where
• type (L) PROCESS
• type (M) type (M1) type (M2) OBJECT.
• op, op1, op2 nonterminals for spatial operators
  ( , , )

41

• The type for a given nonterminal X is determined
  as follows
• Let
• X t be a derivation in Go and t ? T
• Then type (X) spatial operator if t ?? Tsp
• type (X) type (t) otherwise.
• The set of inherited attributes is empty for each
  nonterminal in Go.

42

• The set of synthesized attributes associated to
  each nonterminal X is defined as follows.
• For a given non-terminal X
• 1. if type (X) OBJECT
• NAME (X) is the name of the object icon
  represented by X.
• CG(X) is the CT associated to the subtree rooted
  in X.
• REF(X) is the reference number. The reference
  number is an index associated with each
  elementary object icon occurring in a visual
  sentence. It allows one to distinguish between
  different occurrences of the same elementary
  object icon in a visual sentence.

43

• 2. if type (X) PROCESS
• NAME(X) is the name of the process icon
  represented by X.
• 3. if type (X) "spatial operator"
• OP(X) is the spatial operator derived from X.

44

• The semantic rules associated with the four
  production rules are
• a) X t
• a1) If t icon-id
• where
• ( icon-id, icon-sk, type( i ) ) ? ID and
• type ( i ) OBJECT (i.e. t is a
  terminal for an elementary icon).
• Then
• NAME(X) icon-id
• REF (X) ref-set
• CG (X) OBJECT NAME(X), REF(X)

45

• a2) If t icon-id
• where
• ( icon-id, icon-sk, type( i ) ) ?? ID and
• type ( i ) PROCESS
• Then
• NAME(X) t
• a3) If t ??? , ,
• Then
• OP(X) t

46

• b) X Y
• b1) If type (Y) OBJECT
• Then
• NAME(X) NAME (Y)
• REF (X) REF (Y)
• CG (X) CG (Y)
• b2) If type (Y) PROCESS
• Then
• NAME(X) NAME (Y)
• b3) If Y op and op ??? , ,
  
• Then
• OP(X) OP(Y)

47

• c) X M op L
• c1) If type (M) OBJECT and type (L)
  PROCESS
• Then
• NAME(X) RESULT-NAME NAME(L), NAME (M),
  null-obj, OP(op), null-op
• REF(X) SET-REF (L, M, null-obj, op,
  null-op)
• CG (X) SELECTIVE-MERGE (L, M, null-obj, op,
  null-op)

48

• c2) If type (M) PROCESS and type (L)
  OBJECT
• Then
• NAME (X) RESULT-NAME NAME(M), null-obj,
  NAME(L), null-op, OP(op)
• REF(X) SET-REF (M, null-obj, L, null-op ,
  op)
• CG (X) SELECTIVE-MERGE (M, null-obj, L,
  null-op, op)

49

• c3) If type (M) type (L) OBJECT
• Then
• NAME (X) RESULT-OBJECT NAME (M),
  NAME(L), OP(op)
• REF(X) REF (M) U REF (L)
• CG (X) LINK (X, M, L, op)

50

• d) X M1 op1 L op2 M2
• If type (L) PROCESS and
• type (M1) type (M2) OBJECT
• Then
• NAME (X) RESULT-NAME NAME (L), NAME(M1),
  NAME (M2), OP(op1), OP (op2)
• REF(X) SET-REF (L, M1, M2, op1, op2)
• CG (X) SELECTIVE-MERGE (L, M1, M2, op1, op2)

51

• Semantic rules c and d make use of the following
  procedures
• Procedure LINK (X, M1, M2, op)
• rel RELATION NAME(M1), NAME (M2), OP(op)
• case rel of
• ltnewgt if NAME (X) ?? ID then
• - add the new composite object NAME(X) to
  ID
• - return OBJECT NAME (X), REF(X)
• endif
• ltnullgt return CG(M1)
• else return APPEND (CG(M1), CG(M2), rel)
• end case
• end LINK

52

• Procedure SELECTIVE-MERGE (L, M1, M2, op1, op2)
• C EVENTEVENT-NAME NAME(L),
• NAME(M1), NAME(M2), OP(op1), OP(op2)
• if OP(op1) null-op then G1 C
• else rel1 LEFT-REL NAME(L), NAME(M1),
• NAME (M2), OP(op1), OP(op2)
• G1 APPEND (C, CG(M1), rel1)
• endif
• if OP(op2) null-op then G2 C
• else rel2 RIGHT-REL NAME(L), NAME(M1),
• NAME (M2), OP(op1), OP(op2)
• G2 APPEND (C, CG(M2), rel2)
• endif
• return HEADER-MERGE (G1, G2)
• end SELECTIVE-MERGE

53

• Procedure SET-REF (L, M1, M2, op1, op2)
• ref RESULT-REF NAME(L), NAME(M1), NAME
  (M2), OP(op1), OP(op2)
• case ref of
• ltl-objgt return REF(M1)
• ltr-objgt return REF(M2)
• ltnew-objgt return NEXT-REF( )
• end case
• end SET-REF

54
The Icon Interpreter

• The system diagram of the icon interpreter is as
  follows

G0
ID
B
Visual Sentencce S
Construction of the Logical part
Pattern Analysis
Syntax Analysis
Meaning of S CG
Pattern String of S
Parse Tree of S
55

• The module Pattern Analysis" transforms the
  visual sentence into pattern string .
• Example
• The visual sentence
• is represented by the pattern string
• (char char char) cross

56

• The icon interpreter is logically divided into
  two phases
• 1. The syntax analysis is accomplished by a
  parsing algorithm for context-free grammars.
• 2. Constructing the meaning of the given visual
  sentence by evaluating the attributes of the
  nonterminal on the root of the parse tree using
  the ATTRIBUTE_EVALUATOR procedure
• The attribute evaluation is accomplished by means
  of a postorder visit of the parse tree t of the
  visual sentence S

57

• Procedure ATTRIBUTE-EVALUATOR (X).
• if X is a nonterminal in Go
• then / let p be the production applied to the
  node X in t, and let 1, ..., np be the indices
  of nonterminals in the right-hand side of p/
• for j 1 to np do
• call ATTRIBUTE-EVALUATOR (jth
  nonterminal son of X in t)
• end for
• apply the semantic rules associated to p
  so as to evaluate the attributes of X
• endif
• end ATTRIBUTE-EVALUATOR