Chapter 2: Light and Sound, Seeing and Listening

1
Chapter 2 Light and Sound, Seeing and Listening

• Perception
• Seeing and hearing
• Light and sound in the physical environment
• Physics of light and sound
• Initial transduction of light and sound
• Physiology eyes, ears, optic auditory nerves
• Additional transduction
• Psychology Perception, identification,
  interpretation etc.
• Cognition, thoughts can be influenced by
  perception
• Perception can be influenced by previous
  experience, cognition, thoughts

2
Interpretation Illusion - 1
What is this symbol?
3
Interpretation Illusion - 2
A
D
E
4
Interpretation Illusion - 3
T
C
What is it now?
5
Interpretation Illusion - 4

• Goldstone (1995). Effects of categorization on
  color perception. Psychological Science, 6,
  298-304.
• Adults judged hue of letters numbers
• Experimental manipulation letters tended to be
  redder than numbers (but not always)
• People judged a letter to be redder than a
  number, even when the specific letter and the
  specific number were exactly the same hue

6
Effects on Audition (Hearing)

• There are interpretation effects on hearing as
  well as sight
• E.g., perception of hearing a particular word can
  be influenced by the context of what is being
  said
• Text gives illusion examples from music

7
Emphasis is on Human as Receiver Interpreter of
the Communication
Multimedia system (i.e., computer software)
human
(information flow)
8
Considerations in Perception - 1

• Environment
• Containing animate inanimate objects
• The origination of energy, physical, and chemical
  properties
• Some energy or chemicals are are detected by
  human perceptual systems
• Light, sound energy
• Taste, smell chemical
• Touch physical
• Sensory (perceptual) systems
• Transduction of energy, physical, and chemical
  properties into a neural code

9
Considerations in Perception - 2

• Cognitive (brain) systems
• Interpretation, discrimination, identification,
  classification
• Further signal transduction
• Effector/motor systems
• Some stimuli directly trigger reflexes
• Interaction can be important in perception
  (perception is active!)
• E.g., eye saccades

10
Active Perception

• The perceiver is considered to act as an
  intuitive i.e., naïve scientist, making and
  testing hypotheses about the world on the basis
  of inadequate or distorted sensory evidence
  (Gordon, 1989, p. 12)
• Sensory information is ambiguous
• Multiple interpretations are the rule
• Other knowledge has to be applied
• Combining information from multiple media and
  modalities can help to reduce ambiguity

11
Waveform Terminology

• Cycle one complete unit of a periodic waveform
• Hz hertz unit of frequency
• Number of times an event occurs per second
• E.g., number of cycles per second
• Amplitude height of a waveform
• Phase relative timing of identical waves
• In Phase Same timing
• Out of Phase Different timing
• Harmonics
• Waveforms with frequencies that are whole number
  multiples of others

12
Waveform interaction

• When different waveforms are produced at the same
  time
• E.g., two sounds produced at the same time
• The waveforms interact and combine
• Observed result
• Sound or light is different
• Combined result of multiple waveforms

13
Waveform math

• Waveforms can be described by equations
• y f(x)
• Two waveforms
• y1 f1(x)
• y2 f2(x)
• When the two waveforms occur together, they
  produce a new waveform
• Mathematically, describe this by summing
• ynew f1(x) f2(x)

14
Examplex1.27 sin(x) .951 cos(x) .309
15
Examplex1.27 sin(x) .951 cos(x) .309 sin(x)
cos(x) 1.26
16
In Phase Waveforms

• What occurs?

17
Phase Cancellation

• Two waves with same frequency and amplitude
• Propagated at same time and place
• One waveform is 180 degrees (1/2 cycle) out of
  phase with other
• Get Phase Cancellation
• Waveforms cancel each other
• No sound or light

18
Electromagnetic Radiation

• Visible Light, X-rays, UV rays, etc.
• About 1840 Morse Telegraph
• 1861 Maxwell proposes that electric and
  magnetic forces same
• 1888 Hertz discovers radio waves
• A few years later Marconi Information
  transmitted over radio waves

19
Properties of Light

• Speed
• Light is slowed when entering some materials
• Slows about 33 when entering glass
• When exiting the material, the speed resumes!
• However In the case of light, the intensity is
  diminished when exiting
• Intensity power per unit area (unit candela)
• Strength decreases as the inverse square of
  distance
• Radiance total amount of energy (e.g., light
  plus heat) emitted from the source (unit watt)
• Luminance measure of light strength perceived by
  the human eye (unit lumen)
• Brightness subjective measure of relative
  brightness
• Temperature see p. 55

20
Interaction of Light with Materials

• Reflection waves bouncing off a material
• Some waves may enter others may reflect
• Variation by
• Angle of incidence
• Color of surfaces
• Glare
• Ambient reflecting light interfering with vision
• Fiber optics
• Refraction When light has direction changed by
  passing through a material
• Optics/lenses systematically make use of
  refraction
• Focusing/dispersing effect with convex/concave
  lenses

21
Color and Color Sensation

• Object color
• When object is not emitting light
• Color of the reflected light

White light
Subtractive process Incoming absorbed
reflected
Wavelengths not absorbed
22
Measurement Terms

• Emission (emitted source), remission
  (transmitted) curves of wavelength intensity
  composition of light
• HSB Color Model
• Hue single wavelength of light (Chroma similar)
• Saturation
• degree to which a hue differs from a neutral gray
  (0 - no color saturation- gray 100, full
  saturation- no gray).
• Brightness
• luminance (measure of perceived light strength)
  of object in context
• Lightness
• Level of illumination (100 - white, washes out
  color 0 black)
• Value
• amount of light emanating from the color

23
Subtractive Additive Principles

• Subtractive reflected light
• Objects, pigments, etc. absorb, or subtract
  certain wavelengths of light
• Mixing pigments to form a new pigment combines
  their subtractive qualities
• Mix pigment A (absorbs everything but red), with
  pigment B (absorbs red) should get black
• Additive emitted light
• Colored lights mix, combine to form new colors
• E.g., computer monitor, TV, color photographic
  enlarger

24
Trichromatic vision

• Human retina Different types of photoreceptive
  cells
• Rods low light/motion
• Cones different types of photopigments
• Absorbs 435 nm light (blue)
• Absorbs 540 nm light (green)
• Absorbs 656 nm light (yellow)
• Generation of color
• Perception of any color can be created by mixing,
  in varying proportions the three primary colors
  (red, green, blue)

25
Sensation of ColorAlso Depends On

• Physiology
• Cornea is yellow (adaptive for UV light)
• Filters out shorter wavelengths (green, blue,
  violet)
• Allows red through
• Perception
• (1) Rapid pulse or change in brightness
• Causes perception of different colors
• (2) Retinal cells adapt to common
  colors/intensities
• Context
• Color perception may depend on color context of a
  color
• Subjective aspects
• Colors tend to be associated with emotions

26
More On Color

• Color-wheel
• Red-blue-green triangle
• Yellow-cyan-magenta triangle (secondary colors
• Complimentary colors
• Colors on opposite sides of the color wheel
• When combined produce perception of white light
• Rule of thumb
• Choose colors that complement, rather than
  contrast
• Pantone Color System
• Popular computer method of comparing colors
• Collection of known color samples
• Colors identified also numerically (hexadecimal)
  proportions of red, green, blue

27
Opponent coding

• Complementary colors apparently arise from the
  opponent coding of the visual system. After
  staring at a single color for a prolonged period,
  the retinal cells that contribute to the
  definition of the color become fatigued. When
  sight is directed to a white page, these cells
  briefly can't contribute to the definition of
  white, leaving "white" to be represented by the
  unfatigued cells on the opposite side of the
  opponent contrasts. (http//www.handprint.com/)

28
Methods of Computer Color Display

• RGB (red, green, blue)
• Numeric (hexadecimal) codes for the intensities
  of the three primary colors
• 24 bit RGB 8 bits for R, 8 bits for G, 8 bits
  for B
• 256 x 256 x 256 16,777,216 colors
• HSV (hue, saturation, value), HLS (hue,
  lightness, saturation), CIE (any three colors
  can be used as primary)
• CIE combination of two cant produce third
• Algorithms to do conversions
• http//www.cs.rit.edu/ncs/color/t_convert.htm
  l
• Color lookup table (CLUT)
• Index into a table to find the appropriate color
  values
• Color representation depends on hardware
  (monitor, display adapter)

29
Communication

• Intentional unintentional
• Language-based e.g., speech, reading
• Visual-based
• Strongly tied to learning
• Important differences in amounts of learning
• 10 remembered hearing
• 30 remembered reading
• 80 remembered see do

30
Camera vs. Eye

• Typical Camera
• Passive optical system
• Visual Perception
• rays of light that constitute images that should
  be regarded as faithful representations of the
  external world (p. 68, main text)
• But what about levels of illumination, shadows,
  occlusion, dynamics of motion?
• Watt (1991) We would be nearer the mark if we
  were to look to the artist, painting first with
  his coarse brush and then building within and
  over these broad strokes with finer details The
  artist is an active device

31
Visual Media Literacy

• Visual literacy
• understanding and comprehension of visual images
• Media literacy
• an understanding of how images (and other media)
  are manipulated for effect
• Requires visual literacy
• Visual media literacy may
• Be applicable (transfer to) other domains of
  knowledge
• Make the viewer more aware of these manipulations
• Provide a basis for informed (e.g., aesthetic)
  appreciation
• Extend (create?) skills

32
Meaning

• Image optical sensations meaning
• Meaning
• Sense external reality denoted
• Feeling attitude or emotion conveyed
• Tone message transmitted
• Intention effect desired in the viewer
• People can only make inferences about intentions
  on the basis of their own subjective
  interpretations and personal experience
• Can you have more than this?
• Are all visual conventions arbitrary? (e.g.,
  powerful, powerlesss)
• meaning has broad interest
• http//www.lucs.lu.se/epigenetic-robotics/Papers/Z
  latev.pdf

33
Interesting Question

• See second paragraph, top of p. 72, text
• Can the lack of an event or object be
  characterized by only images viewed in isolation
  without other explanation (or interpretation)?
  (From Worth, 1981)

34
Pragmatic Image Interpretation

• Personal Gut reaction (subjective)
• Historical Analysis within previous similar work
• Technical Methods-based analysis
• Ethical Moral and ethical issues
• Cultural Analysis from view of a culture
• Critical Reasoned (objective) issues

35
Role of Previous Knowledge

• Used to help make sense of images
• What does a viewer have to know to make sense of
  an image?
• E.g., the assumed visual alphabet of the viewer
• Depends in part on socio-cultural background

36
Issues with Pictures vs. Reality

• Light and color variance
• Range of brightness levels
• Range of colors (e.g., b w only)
• Changes in brightness on surface of objects
• Depth/position information
• Parallax apparent differences of position of an
  object when seen from different viewing positions
• Apparent decrease in size with distance
• Object recognition
• Temporal view
• Things can only be viewed from one point of view
  at a time
• Distortions (commissions, omissions)
• Lesson People can adapt to various image
  features
• Generally, devote resources to substance
• But keep in mind audience Universal Access

37
Form

• Components dots, lines, shapes
• arise (mostly) from the real world
• Lines
• Especially relevant to Western cultures?
• Linear conception of the world
• Different interpretations depending on placement
  (see Figure 2.22)
• Western viewers scan from left to right

38
Shapes

• Balance
• Objects placed off center in a square create
  sense of unbalance
• Off center objects in rectangles ? balance
• Circles
• Draw the viewers attention
• Should be used sparingly
• Isosceles triangle (two sides equal length)
• Can be used as symbolic (pointing up), or
  directional (pointing other directions)

39
Graphics Primitives

• Graphics object used so often, it is essential to
  the creation of images
• Points Represented by one pixel
• Lines Represented by two or more pixels
  connected to each other
• Polygons Closed figure, more than two sides,
  exists in one plane

40
Depth Perception

• What cues enable depth to be perceived?
• Stereovision differences between images
• Field of vision 180 degrees
• Field in which depth is perceived 90 degrees
• Other cues to depth
• Space, object size, color (warm-close vs.
  cool-far), lighting, texture gradients,
  interposition (occlusion), time, perspective

41
Motion

• Real vs. apparent motion
• Real An object moving in the world
• Apparent Some kind of illusion of motion (e.g.,
  a video image on the TV)
• Critical flicker fusion frequency about 15 hz
• Perception of physical causality
• Michotte How do humans perceive that one event
  has physically caused another?
• Launching
• Entraining
• Object B starts as soon as object A passes by
• Graphic movement (eye motion), implied movement
  (illusion of movement)

42
Writing Important in Visual Communication

• Standardized or conventionalized systems of
  representation
• Morpheme Minimal meaningful language unit
• Gutenberg 1400s
• Advances in movable type printing the Bible
• Typography/fonts
• a message about the message
• Serif Small strokes or lines at the end of the
  main strokes
• Help guide the readers eye in larger blocks of
  text
• Sans serif Without these small strokes
• Display type Headings
• Measurement points (72 points to an inch)
• Alignment justified, flush left/ragged right etc
  (p. 95)
• Flush left/ragged right Best ease of reading

43
Sound

• Visual information is often primary
• Acoustic information is still relevant
• Sound generation, propagation, reception
• Generation, propagation physical
• Reception physiological/psychological
• Sound received is never the same as the sound
  produced, and in turn, may be different than the
  sound perceived
• Sounds is generated by vibrations
• Vibrations induce sound waves compression/rarefac
  tion in the air
• These rapid pressure changes are detected by the
  ear
• Sound is much slower than light (1/5 mile/s
  186,000 miles/s)
• High frequency, high pitch sound
• Amplitude intensity or volume

44
Fourier Analysis

• Analysis of a sound into its frequency components
• Sine wave components
• Aka FFT (Fast Fourier Transform)
• Not necessarily related to human perceptual
  analysis

45
Room Acoustics

• Direct sound
• Waves received that propagate directly from
  source
• Reflected sound
• Early sound (first reflection)
• Reverberations (later reflections)
• Diffraction
• Sound waves can propagate through holes
• Its like you have a new sound source starting at
  the hole

46
Doppler Effect

• An effect of the sound source moving relative to
  the listener
• For relative speeds 32-48 km/hour
• May want to recreate this in a multimedia
  presentation
• Source approaching
• I.e., Distance between sound source and listener
  decreasing
• Pitch of sound increases
• Source moving away
• I.e., Distance between sound source and listener
  increasing
• Pitch of sound decreases

47
Sound Perception-1

• Categorical perception
• a change in some variable along a continuum is
  perceived, not as gradual but as instances of
  discrete categories
• http//www.ling.umu.se/rand/KatPer/index.eng.htm
  l
• E.g., in phonemes of sound, a gradual change
  along one sound dimension suddenly causes
  perception of a categorically different sound
  (ba, da, ga)
• Above web site also has an active experiment that
  illustrates this

48
Sound Perception-2

• Hearing and listening
• Habituation
• The brain is interested mainly in change
• The longer one kind of stimulus occurs, the less
  sensitive perception becomes to that stimuli
• Adult human perception
• 20 20,000 Hz
• Infants and children have fullest range
• Perception of higher frequencies are first to be
  lost in aging

49
Perception-3

• We interpret sounds as the same despite
• Overall variations in volume, pitch, noise,
  accent
• Sound localization
• Determining the location of a sound source
• Cues
• Effects due to pinnae (outer ear) assists
• Ear disparity effects (Interaural differences)
• Identification
• Identifying the source of a sound (e.g., speech
  sounds you hear over a telephone)
• People use various cues to identify sounds
• Context, physical attributes of sound, subjective
  attributes

50
Sound Measurement

• Acoustic watt power
• Decibels (dB)
• Logarithmic scale
• 0 dB Hearing threshold
• 3 db change typically required to detect
  difference
• Corresponds to at least a doubling of power