Dimensions of

1
Dimensions of Media Object Compehensibility
Lawrie Hunter Kochi University of
Technology http//www.core.kochi-tech.ac.jp/hunter
/
2
Dimensions of Media Object Compehensibility
Island of Shikoku
Lawrie Hunter Kochi University of
Technology http//www.core.kochi-tech.ac.jp/hunter
/
KUT
3
Niigata
Osaka
Kochi
4
A pattern language for MMC
Source of insight language / language
learning Second language (L2) learning a
cognitive process? Comprehension of partially
acquired L2 revealing of the nature of
text/media. Language learning issues are germane
to MMC.
5
A pattern language for MMC
Intervention author's structural model of content
information (for second language learning
materials)
Background work towards a human-communication
paradigm for the guidance of machines
Frame the new multidisciplinary approach of
machine-mediated communication
Discussion of parameters of difficulty
Ground related issues in second language
learning materials
Objective development of a pattern language for
that multidisciplinary approach to MMC
Exemplars considerate text considerate
multimedia
Focus factors influencing the difficulty of
comprehension of media objects
Generating parameters of difficulty in media
object comprehension
Question how media objects carry information.
Tempering questions of significance and
applicability for machine automation.
6
L2 learning materials
The creation of second language (L2) learning
materials demands document transparency
1. document purpose 2. document content 3. target
behavior 4. target lexical items
must all be more immediately apparent to the
learner than in the case of materials for L1
medium learning scenarios
7
L2 learning materials
The creation of second language (L2) learning
materials demands document transparency
1. document purpose 2. document content 3. target
behavior 4. target lexical items
work towards transparency is informed by
difficulty-related issues
difficulty-related issues inform human
interaction with info media
8
Earlier work an EAP tool
David Kolb re using hypertext to present
scholarly text "...the easiest ways of making a
complex argument available in HT tend to move the
text toward linear structures that do not take
full advantage of the possibilities of linked
text."
in a talk to KMI at the Open University
9
Earlier work an EAP tool
David Kolb re using hypertext to present
scholarly text "...the easiest ways of making a
complex argument available in HT tend to move the
text toward linear structures that do not take
full advantage of the possibilities of linked
text." "...what the HT can do is present the
argument, but also use linkage and juxtaposition
to make the readers engagement with the
argument more creative, self-conscious, and
self-critical."
in a talk to KMI at the Open University
10
Earlier work an EAP tool
Lawrie Hunter re using hypertext to present
technical L2 text For the L2 reader,
engagement can only be enhanced if the
rhetorical and information structures are
articulated.
11
Earlier work an EAP tool
Lawrie Hunter re using hypertext to present
technical L2 text For the L2 reader,
engagement can only be enhanced if the
rhetorical and information structures are
articulated. What the HT can do for the NNR/W
is tp present simultaneously the various faces
of a research paper the rhetorical moves the
bits of structured information the text
necessary glosses. NNR/W EAP non-native
reader/writer of English for Academic Purposes
12
Earlier work an EAP tool
Lawrie Hunter re using hypertext to present
technical L2 text For the L2 reader,
engagement can only be enhanced if the
rhetorical and information structures are
articulated. What the HT can do for the NNR/W
is to present simultaneously the various faces
of a research paper the rhetorical moves the
bits of structured information the text
necessary glosses. And if the NNR/Ws design
their personal interface, a negotiated pattern
language of NNR/W EAP will emerge. NNR/W EAP
non-native reader/writer of English for
Academic Purposes
13
Arguably important direction "Tomorrow's
literacies... need to be process and systems
literacies. -John Thackara, In the Bubble
Designing in a complex world. MIT Press 2005.
14
Structural view of writing
Diversions
Train of argument
Rhetorical structures
This is the domain of structures up here.
Extension
Functional structures
Background
Knowledge structures
Cohesion devices
Rhetorical structure theory, systemic functional
linguistics and knowledge structure mapping
form a hierarchy of structures, whereas grammar
and sentence diagrams reflect rules for texture
management.
This is the domain of texture down here.
Grammar (sentence surface structure)
15
Structural view of writing
Diversions
Train of argument
Rhetorical structures
This is the domain of structures up here.
Extension
Functional structures
Background
Knowledge structures
False hierarchy the train stops here.
Cohesion devices
Rhetorical structure theory, systemic functional
linguistics and knowledge structure mapping
form a hierarchy of structures, whereas grammar
and sentence diagrams reflect rules for texture
management.
This is the domain of texture down here.
Grammar (sentence surface structure)
16
L2 reader needs analysis
Language skills Argument sequencing Info-struct
ured sentence generation Mimicry of model
language
Knowledge Niche grammar structures Niche
rhetorical structures General register
repertoires (distinguishing formal academic from
informal academic) Research Paper text
structure and information structure
Facilities Concordance collocation
resource Bank of model research
papers (annotated)
c.f. Brown and Browns annotation
17
L2 reader wants analysis

• In a technical hypertext, L2 reader/writers
  want
• 1. Glossing (of 'difficult' terms and phrases)
• 2. Moves indicator
• 3. Lexia position indicator
• 4. PDF-drawer-like phrase recurrence tab
• 5. Register converter
• (e.g. research paper ltgt presentation script)
• 6. Information structure maps for atomic
  utterances
• 7. Overall argument map on every lexia
• (similar to Horn's argument maps
• or Rhetorical Structure Analysis?)
• Based on a survey of 22 PhD engineering
  students

18
Technical hypertext design
NEEDS
A pattern language?
www.patternlanguage.com
WANTS
19
Technical hypertext design
NEEDS
A pattern language?
www.patternlanguage.com
WANTS

• The language, and the processes which stem from
  it, merely release the fundamental order which is
  native to us. They do not teach us, they only
  remind us of what we know already, and of what we
  shall discover time and time again, when we give
  up our ideas and opinions, and do exactly what
  emerges from ourselves.
• Christopher Alexander,
• The Timeless Way of Building

20
Do humans have a GRAPHIC THOUGHT FACILITY?
The knowledge structure map is a matrix
(confluence) for the situated learner and the
situated mentor to confirm context and the
nature of "stolen property." Jean
Lave Duguid and Brown
21
Hunters knowledge structure map links
Classification
Description
Degree comparison
Attribute comparison
Contrast
!
Sequence
Cause-effect
22
2005 project design level
EEAP students HT designs for the analysis of
technical academic papers. EEAP
Engineering English for Academic Purposes, a
subset of EAP, which is a subset of ESP
(English for Specific Purposes)
Hunter L. (2005) Technical Hypertext
Accessibility Information Structures and
Rhetorical Framing. Presentation at HyperText
2005, Salzburg. http//www.lawriehunter.com/presns
/20HT05poster0818.htm
23
TEXT STRUCTURE Introduction Background Ques
tion Methods and materials Results Observatio
ns Conclusion
RHETORICAL MOVES Common knowledge Cite Report Ex
plain Claim Question Qualify Evaluate Decide
Infer Project
INFO STRUCTURE Describe Classify Compare Sequ
ence Cause-effect Contrast
INFOMAP(s)
UTTERANCE(s)
In general, power plants boil some liquid to make
steam, which rotates turbines, which generate
electricity. Power plants boil a liquid to
produce steam, which is used to rotate turbines,
which in turn generate electricity.
24
TEXT STRUCTURE Introduction Background Ques
tion Methods and materials Results Observatio
ns Conclusion
RHETORICAL MOVES Common knowledge Cite Report Ex
plain Claim Question Qualify Evaluate Decide
Infer Project
INFO STRUCTURE Describe Classify Compare Sequ
ence Cause-effect Contrast
INFOMAP(s)
UTTERANCE(s)
Traditional power plants use fossil fuel heat or
heat from nuclear fission to boil water and
produce steam at 500C. Older type power plants
boil water with heat from fossil fuel combustion
or nuclear fission to produce steam with a
temperature of 500C.
25
TEXT STRUCTURE Introduction Background Ques
tion Methods and materials Results Observatio
ns Conclusion
RHETORICAL MOVES Common knowledge Cite Report Ex
plain Claim Question Qualify Evaluate Decide
Infer Project
INFO STRUCTURE Describe Classify Compare Sequ
ence Cause-effect Contrast
INFOMAP(s)
UTTERANCE(s)
OTEC power plants use seawater heat to boil
ammonia and produce steam at 20C. OTEC type
power plants boil ammonia with the heat of the
sea to produce steam with a temperature of 20C.
26
RHETORICAL MOVES Common knowledge Cite Report Ex
plain Claim Question Qualify Evaluate Decide
Infer Project
TEXT STRUCTURE Introduction Background Ques
tion Methods and materials Results Observatio
ns Conclusion
INFO STRUCTURE Describe Classify Compare Sequ
ence Cause-effect Contrast
INFOMAP(s)
UTTERANCE(s)
Traditional power plants use fossil fuel heat or
heat from nuclear fission to boil water and
produce steam at 500C, whereas OTEC type power
plants boil ammonia using the heat of the sea to
produce steam with a temperature of 20C. Older
type power plants boil water with heat from
fossil fuel combustion or nuclear fission to
produce steam with a temperature of 500C, while
OTEC power plants use seawater heat to boil
ammonia and produce steam at 20C.
27
Obstacle in 2005 project
Massive diversity in learner perception of
knowledge structures.
28
Obstacle in 2005 project
Massive diversity in learner perception of
knowledge structures.
29
Structural view of writing
Diversions
Train of argument
Rhetorical structures
Extension
This is the domain of structures up here.
Functional structures
Background
Knowledge structures
False hierarchy the train stops here.
Cohesion devices
Rhetorical structure theory, systemic functional
linguistics and knowledge structure mapping
form a hierarchy of structures, whereas grammar
and sentence diagrams reflect rules for texture
management.
This is the domain of texture down here.
Grammar (sentence surface structure)
30
Structural view of writing
31
2006 new layer READABILITY
The missing link in technical academic
writing Gopens readability -subject-verb
distance -topic position / stress
position -old/new information placement
32
Background readability work
In the design of traditional high-text language
learning materials, readability is a prominent
concern. Reading difficulty has for some time
been seen as depending on -word
length -sentence length -text length -number
of sentences per paragraph -vocabulary
difficulty
More recent work has extended this list to
include -subject-verb distance -adherence to
old/new position conventions -topic
position/stress position conventions
Treated extensively in Hunter L. (1998) Text
Nouveau Visible Structure in Text Presentation.
Computer Assisted Language Learning 11(4) pp.
363-379.
33
Background MM readability
Chun, D. M. and Plass, J. L. 1997. Research on
text comprehension in multimedia environments.
Language learning and technology 1(1) 60-81.
Treated extensively in Hunter L. (1998) Text
Nouveau Visible Structure in Text Presentation.
Computer Assisted Language Learning 11(4) pp.
363-379.
34
2006 new layer READABILITY
Hunters new TAW syllabus assume grammar
35
2006 new layer READABILITY
36
Readability
The creation of second language (L2) learning
materials demands appropriate readability.
1. understandable by the learner 2. stretching
learner knowledge/skill 3. contextualized to
support stretching 4. orchestrated with degrees
of scaffolding
37
Considerate text
One inroad to readability is considerate text
Original framing ?well-written, ?well-organized,
and ?signals the organization of its thought to
the reader
38
Considerate text
One inroad to readability is considerate text
Original framing ?well-written, ?well-organized,
and ?signals the organization of its thought to
the reader
More recent takes -glossing -phrase boundary
marking -de-idiomatizing -the Plain English
movement -graphic organizers -text nouveau
39
Text nouveau is still text
Text comprehension in multimedia environments is
a rich variant, BUT
Chun, D. M. and Plass, J. L. 1997. Research on
text comprehension in multimedia environments.
Language learning and technology 1(1) 60-81.
40
Text comprehension in multimedia environments is
a rich variant, BUT
Text nouveau is still text
Chun, D. M. and Plass, J. L. 1997. Research on
text comprehension in multimedia environments.
Language learning and technology 1(1) 60-81.
41
Sharing considerate text
L2 learning objects welcome to the TagTower of
Babel!
KUT English is a Moodle department.
Appropriateness of learning materials/tasks is
very complex. Tagging of these materials tasks
is daunting.
42
Sharing considerate text
Appropriateness of learning materials/tasks is
very complex. Tagging of these materials tasks
is daunting.
Fortunately, as David Weinberger points
out, there is a huge amount of metadata out
there, but this allows multiple simultaneous
organizations of content.
June 12, 2007 interview with IT
Conversations http//www.itconversations.com/shows
/detail1838.html Weinberger books The Cluetrain
Manifesto Small Pieces Loosely Joined Everything
is Miscellaneous
43
Considerate multimedia?
Considerate text in the context of M3C
suggests the notion of considerate multimedia
Tentative definition considerate multimodal
objects are those which contain few
non-essential obstacles to their comprehension.
44
Considerate multimedia?
Considerate text in the context of M3C
suggests the notion of considerate multimedia
Tentative definition considerate multimodal
objects are those which contain few
non-essential obstacles to their comprehension.
Tentative definition 2 considerate multimodal
objects are those which are tagged for various
forms of comprehension difficulty.
45
Considerate multimedia?
46
Multimedia comprehensibility?
Considerate multimedia confronts vastly more
complexity than considerate text
Alternative approach create a set of
parameters for multimedia comprehensibility
One approach to comprehensibility explore
obstacles to comprehensibility, as has been done
in readability work.
47
Multimodal equivalent of readability
In the domain of multimodal computer-mediated
communication, the question of readability
translates as ease of comprehension
How easy is it for a human to extract all the
information contained in a multimodal media
object (MMO)?
48
Parameters of media object function
To measure the ease of extraction of all the
INTENDED information contained in a MMO, we
need a characterization of the difficulty of
extraction Tentative list of sources of
information extraction difficulty, for
simplicitys sake limited here to text
objects graphic objects speech objects video
objects and combinations thereof.
This is a tentative, exploratory framing of MMO
comprehensibility,
49
Learning object tag shopping list
Parameters of media object function
utterance minimal spoken, written or graphical
communication unit
50
Concept density
Concept density in text space Concept density
in aural time Concept density in video
space concept density number of exophoric
references per sentence/minute/frame
Parameters of media object function
51
Metaphor density
Metaphors per sentence. Metaphors per
argument. Metaphors per minute. Idioms per
sentence. Idioms per argument. Idioms per minute.
Parameters of media object function
52
Phoneme density
Phonemes per unbroken utterance. Phonemes per
exhalation. Phonemes per word. Phonemes per
minute.
Parameters of media object function
e.g. Italian speech.
53
Noise density
Superfluous signals/utterance Readability
stoppages per sentence Asides per
sentence/message Cognitive dissonances per
utterance Facial expression-statement conflicts
Parameters of media object function
54
Symbol density
Numerals per page. Numerals per
sentence. Formulae per sentence/paragraph. Formula
e per rhetorical move.
Parameters of media object function
55
Channel imperfections
Imperfect audio channel Imperfect text
channel Imperfect visual channel
Parameters of media object function
56
Channel imperfections
Finding humans retain more info from video with
an imperfect audio channel
Parameters of media object function
Nass and Brave, Wired for speech Reeves and Nass,
The media equation
57
Channel-channel discord
Parameters of media object function
58
Channel-channel synchronicity
Parameters of media object function
59
Message-message harmony
Message-message agreements Message-message
discords
Parameters of media object function
60
manga vs. knowledge structure maps
Both involve reduced text density and spatial
highlighting of text, and suggest the question
of a 'graphic thought facility' in humans.
To illustrate the use of the parameter approach,
here is a comparison of two relatively similar
types of media objects, manga and knowledge
structure maps. Both involve reduced text density
and spatial highlighting of text, and suggest the
question of a 'graphic thought facility' in
humans.
61
Generating parameters of difficulty in media
object comprehension
Tempering questions of significance and
applicability for machine automation
Once a comprehensive set of parameters of MMO
comprehensibility has been developed, questions
of application will arise. How can (should?)
these parameters be situated among larger
semantic frameworks? Which of these parameters
are relevant to the development of
machine-mediated communication? How can they
be operationalized in computable form?
62
Daunting ontology-based readability
Work on ontology-based research writing
reforming how scientific research is
written/read. EXPO and the Robot
Scientist Does the ontology EXPO feed back from
a machine interface with a body of
knowledge/practice to a solidification of human
interface with that body of knowledge/practice?
EXPO An Ontology of Scientific Research. Ross D.
King Larisa N. Soldatova http//www-tsujii.is.s.
u-tokyo.ac.jp/jw-tmnlpo/RossKing.pdf
63
Daunting ontology-based readability
Work on ontology-based research writing
reforming how scientific research is
written/read. Use of Natural Language is a
great hindrance when using computers to store
and analyse data hence the growing importance of
text-mining. We argue that the content of
scientific papers should increasingly be
expressed in formal languages. Is writing a
scientific paper closer to writing poetry or a
computer program?
EXPO An Ontology of Scientific Research. Ross D.
King Larisa N. Soldatova http//www-tsujii.is.s.
u-tokyo.ac.jp/jw-tmnlpo/RossKing.pdf
64
Daunting ontology-based readability
Work on ontology-based research writing
reforming how scientific research is
written/read. Can humans now experience
knowledge differently, thanks to machine
interface work, i.e. through a formal language
imposed for the machines sake? Will this reform
how we read? how we think?
EXPO An Ontology of Scientific Research. Ross D.
King Larisa N. Soldatova http//www-tsujii.is.s.
u-tokyo.ac.jp/jw-tmnlpo/RossKing.pdf
65
References 1 Elsayed, A. (2007)
Machine-mediated communication the technology.
6th IEEE International Conference on Advanced
Learning Technologies, ICALT 2006, 5-7 July 2006,
Kerkrade, The Netherlands. 2 Hunter, L. (2005)
Technical hypertext accessibility information
structures and rhetorical framing. Proceedings of
the sixteenth ACM conference on Hypertext and
hypermedia, Salzburg, Austria. 3 Kalyuga, S.
(2006) Instructing and testing advanced learners
A cognitive approach. Nova Science
Publishers. 4 Mann, B. (1999) An introduction
to rhetorical structure theory (RST).
http//www.sil.org/mannb/rst/rintro99.htm 5
Mohan, B.A.M. (1986) Language and content.
Reading, MASS Addison-Wesley. 6 Nass, C. and
S. Brave. (2005) Wired for speech How voice
activates and advances the human-computer
relationship. MIT Press. Chun, D. M. and Plass,
J. L. 1997. Research on text comprehension in
multimedia environments. Language learning and
technology 1(1) 60-81. Grow, G. (1996) Serving
the strategic reader cognitive reading
theory?and its implications for ?the teaching of
writing. Viewed June 30, 2007 at
http//www.longleaf.net/ggrow/StrategicReader/inde
x.html Goldman, S.R., Rakestraw, J.A. (2000).
Structural aspects of constructing meaning from
text. In M.L. Kamil, P. B. Mosenthal, P. D.
Pearson, R. Barr (Eds.), Handbook of reading
research (Vol. II, pp. 311-335). Mahwah, NJ
Erlbaum. The Plain English movement
http//www.plainenglish.co.uk/index.htm
(de-idiomatizing)
66
References 2 Research via ontologies Ian
Horrocks http//www.cs.man.ac.uk/horrocks/ EXPO
Ontology of scientific experiments
http//expo.sourceforge.net/ Soldatova L.N.,
Clare A., Sparkes A. and King, R.D. (2006) An
ontology for a Robot Scientist. Bioinformatics
(Special issue ISMB) (in press). Soldatova, LN
King, RD. (2006) An Ontology of Scientific
Experiments. Journal of the Royal Society
Interface (in press). EXPO An Ontology of
Scientific Research by Ross D. King Larisa N.
Soldatova, Department of Computer Science,
University of Wales, Aberystwyth. Hunter Hunter
L. (2005) Technical Hypertext Accessibility
Information Structures and Rhetorical Framing.
Presentation at HyperText 2005, Salzburg.
http//www.lawriehunter.com/presns/20HT05poster08
18.htm Text Nouveau Visible Structure in Text
Presentation. Computer Assisted Language Learning
11(4) pp. 363-379. (text nouveau) WordbyWord
http//www.core.kochi-tech.ac.jp/hunter/WordByWord
/index.html (text nouveau) Text usability for
non-native readers of English. Ueta, R, Hunter,
L. Ren, X.?Proceedings, Information Processing
Society of Japan, Vol. 2003.7. Pp. 199-200.
(phrase boundary marking)
67
Thank you for your kind attention.
Dont hesitate to write to me.
Lawrie Hunter Kochi University of
Technology http//www.core.kochi-tech.ac.jp/hunter