The Action Language Compatibility Effect in American Sign Language - PowerPoint PPT Presentation

1 / 58
About This Presentation
Title:

The Action Language Compatibility Effect in American Sign Language

Description:

Phon-Signs encode phonological motion which has no relation to the semantics of the sign. ... Release 3-way ANOVA (phon, sem/met) Response 2-way ANOVA (sem) ... – PowerPoint PPT presentation

Number of Views:203
Avg rating:3.0/5.0
Slides: 59
Provided by: meyl
Category:

less

Transcript and Presenter's Notes

Title: The Action Language Compatibility Effect in American Sign Language


1
The Action Language Compatibility Effect in
American Sign Language
  • Meylysa Tseng
  • University of Hawaii at Manoa

2
Acknowledgments
  • Ben Bergen
  • Amy Schafer
  • Victoria Anderson
  • Kathryn Wheeler
  • Avis Chan
  • Michigan ASL Web Browser
  • Sign Language friends

3
What will be discussed
  • The Action-Sentence Compatibility Effect
  • Sign Language
  • Motor Simulation in Sign Language

4
Language affects hand motion
  • Action-Sentence Compatibility Effect (ACE)
    Glenberg and Kaschak (2002)
  • Find that hearing sentences which imply motion in
    a forward or a backward direction influence how
    quickly the hand moves in the same or opposite
    direction
  • Supports the theory that language uses some of
    the same brain circuitry as the movement it
    represents. Motion language might evoke a
    simulation of motion.

5

Mechanics of ACE Experiment
  • They used a button box with 3 buttons, one was in
    the middle, the other required movement forward,
    the other required movement backward
  • Subjects heard English sentences which implied
    forward or backward movement and had to decide if
    they were sensible or not.
  • Open the door. (concrete movement backward)
  • Liz told you a story. (metaphorical movement
    forward)

6
Finding the ACE
  • Lets say the forward button is sensible and
    the backward button is not sensible
  • Subject presses and holds their finger on the
    middle ready button
  • Subject hears Open the Door
  • Subject lifts finger and moves forward to hit the
    sensible button
  • ACE If the button marked sensible were the
    backward button, they would be slower in their
    release.

7
Results from the ACE Experiment
  • When subjects saw sensible sentences which
    implied action in one direction, they had
    difficulty if making the sensibility judgment
    required moving in the opposite direction.
  • The effect occurred for release time, from the
    middle button.

8
In other words
  • The meaning of the sentence primes hand motion.
    If the meaning implied is motion that is
    compatible with your hand movement, there will be
    a facilitation effect on release time.
  • The ACE findings seem to indicate that our
    movement is affected by language. This is
    consistent with the theory that some of the same
    cells activated to perform movement are also used
    to understand language about movement.

9
Motor control in Language
  • Language can affect our motor control
  • What if language itself involved movement?
  • Spoken Languages - When we talk we are moving our
    tongue and various parts of our vocal tract
  • Signed Languages offer an interesting twist,
    when we sign we move our hands and upper body

10
Implications of Sign Language on ACE
  • If direction implied in spoken language affects
    movement, then what about in a language where
    meaning about motion coincides with actual
    movement?
  • What about signs that dont imply movement in
    their semantics but phonologically consist of
    movement?

11
Research Questions
  • If the ACE exists in English, does it exist in
    American Sign Language (ASL)?
  • Does phonology yield ACE in ASL?

12
Sign Language Phonology
  • Visual Holds and Movements (Liddell and Johnson
    1985, 1986)

ARROGANT 1st Hold
ARROGANT 2nd Hold
13
Phonology in Terms of Iconicity Auditory
modality (sound)
  • Onomatopoeia Crash, Ding, Meow
  • Pitch, loudness, melody
  • NOTE the articulators used in speech are less
    likely to be used iconically than are the sounds
    they produce

14
Phonology in terms of Iconicity Visual modality
(body and motion)
AFFECTIONATE
15
The Implications of Phonological Iconicity
Sentences vs. Signs
  • English Sentences Glenberg and Kaschak (2002)
    used sentences where the phonology did not convey
    motion, only the sentence meaning did so.
  • ASL Sentences Not feasible to use in this study
    since the motions performed when signing a
    sentence cannot be kept constantTry signing a
    sentence of more than one sign with only one
    motion.
  • ASL Signs Since ASL signs can be better
    controlled for consistency of movement, this
    study uses signs instead of sentences.

16
Properties of Signs
  • Meaning For the purposes of this study, signs
    either convey hand/arm motion or do not convey
    motion.
  • Phonology signs of interest either had forward
    movement or backward movement
  • With these 2 factors we determined there to be 3
    types of signs, each of which use phonology and
    meaning in different ways.

17
Types of Signs Investigated
  • Phon-Signs encode phonological motion which has
    no relation to the semantics of the sign.
  • Sem-Signs encode phonological motion which
    iconically represents the concrete action meaning
    of the sign.
  • Met-Signs encode phonological motion which
    iconically represents the metaphorical action
    meaning of the sign.

18
Sign Language Phon-signs
  • The phonological motion has no relationship to
    the semantics of the sign.

19
Phonological Forward GIRL
20
Phonological Backward HOME
21
Sign Direction
  • Fosigns Signs that have phonological hand
    motion forward
  • Backsigns Signs that have phonological hand
    motion backward

22
Sign Language Sem-signs
  • The phonological motion iconically represents
    hand motion encoded into the semantics of the
    sign.

23
Semantic Forward THROW
24
Semantic Backward ACQUIRE
25
Sign Language Met-signs
  • The motion iconically represents metaphorical
    motion encoded into the semantics of the sign.

26
Metaphorical Forward BAWL-OUT
27
Metaphorical Backward BEFORE
28
The effects of Phonology and Semantics on Motor
Coordination
  • ACE was demonstrated to occur for comprehension
    of English sentences implying movement
  • Signs in American Sign Language implying movement
    have not yet been tested to see if they show ACE,
    which we must now rename Action-Language
    Compatibility Effects (as opposed to
    Action-Sentence)
  • Signs in ASL which do not imply movement, but
    which phonologically consist of movement have
    also have not been tested

29
Method
  • Reproduced Glenberg and Kaschak (2002) using ASL
    sign matching task this is more feasible than a
    sentence sensibility task
  • Pairs of signs were shown to subjects.
  • Matching condition (critical) phon/phon,
    sem/sem, met/met (22 each for 66 total)
  • Non-matching condition (filler) phon/sem or met,
    sem/phon or met, met/sem or phon (22 each for 66
    total) same sign types were matched
    (Fosign/Fosign and Backsign/Backsign)

30
Trial Procedure
  • Fixation Cross appears
  • Subject presses and holds the h key
  • Subject sees first ASL sign which is a movie
    consisting of four frames (450ms)
  • Subject sees visual mask (1s)
  • Subject sees second ASL sign (450ms)
  • Subject lifts finger and presses the SAME
    (critical matching condition) or DIFF
    (non-critical non-matching condition) key

31
4 Experiment Versions
Block A 1/2 of (22 phon 22 sem 22 met (66
total) matching 22 phon 22 sem 22 met (66
total) nonmatching pairs) Block B the remaining
half
32
Experiment Run Procedure
  • Practice session (80 required)
  • 1st half of 66 trials (33 matching (critical), 33
    nonmatch (filler))
  • Score given
  • Experimenter changes buttons so Response
    direction is now opposite
  • Second Practice Session
  • 2nd half of 66 trials
  • Score given

33
Hypothesis
  • Subjects Reaction Times (RTs) should yield ACE
    for Met-signs and Sem-Signs.
  • Subjects RTs may or may not show ACE effects for
    Phon-signs.

34
Subjects
  • Ages 19 70 yrs
  • Deaf or partially deaf
  • Most speak some English with a hearing aid
  • All can read English
  • Some had knowledge of other signed languages
    (BSL, JSL, etc.)
  • Live in Oahu
  • Out of 46 subjects used 29
  • Computer crashed
  • Left handed
  • ASL lt 5 years
  • RTs over 3 STDEVs from the mean
  • Accuracy rate lower than 80

35
Analysis
  • Release RT (time to release the h key) 3-way
    Repeated Measures ANOVA for Subjects
  • Response RT (time to hit the a or key)
    3-way Repeated Measures ANOVA for Subjects

36
Predictions
  • Since Glenberg and Kaschak (2002) found that
    sem-signs and met-signs both showed the ACE, we
    expect to find an interaction between sign dir
    and hand motion.
  • A 3 way interaction effect between Sign type x
    Sign dir x Hand motion could indicate that
    different sign types (phon, sem and met) act
    differently with respect to ACE.

37
Release Near Significant Effect of Sign Type x
Sign Direction x Hand Motion
Sign dir x Hand motion x Sign type F1 (2, 52)
1.92, p 0.16
38
Response Not so near significant Effect of Sign
Type x Sign Direction x Hand Motion
Sign dir x Hand motion x Sign type F1 (2, 52)
0.43, p 0.65
39
Discussion
  • Thus, the 3-way interaction between sign type,
    sign direction and hand motion was nearly
    significant.
  • By lumping sem-signs and met signs together we
    can now see if, as a group, they yield the ACE
    (Sign dir x Hand motion).
  • We can also see if phon-signs yield the ACE.

40
Release Near Significant interaction Phon vs
Sem/Met
  • signdir x motion x signtype F1(1, 26) 2.36, p
    0.14

41
Response ACE, phon vs. sem/met-signs
Sign dir x Hand motion F1(1, 26)4.12, p .05
42
Response ACE, Met-Signs and Sem-Signs
Sign dir x Hand motion F1(1, 26) 6.83, p
0.02
43
Response ACE Near Significant for Sem-Signs,
Sign dir x Hand motion
Sign dir x Hand motion F1 (1, 26) 1.89, p 0.18
44
Response No ACE for Phon-signs, sign dir x hand
motion
Sign dir x Hand motion F1 (1, 26) 0.53, p 0.48
45
Discussion
  • Response vs. Release Significant interaction
    effects were only found for the response.
  • Not enough subjects (Release ACE F value was
    over 2)
  • Sentence sensibility rating task requires
    thinking about semantics before release
  • Sign matching task involves a more automatic
    response, where meaning effects would possibly
    show up after the decision is made, in the
    response RT
  • Why no significant ACE effects for Sem-signs?
  • Probably not enough subjects (F was almost 2)
  • Perhaps subjects didnt recognize some of the
    Sem-signs (Hawaiian ASL dialect vs. Michigan
    State ASL Browser)

46
Phon-Signs different from Met-Signs and Sem-Signs
  • The experiment shows ACE only occurs for
    semantically meaningful phonological motion

47
Backsigns Have Longer Release RTs
Sign dir F1 (1, 26) 18.00, p 0
48
Why longer release RTs for Backsigns?
  • Fosigns have place and handshape encoded at the
    start as opposed to some Backsigns which encode
    this information at the end of the sign,
    therefore perhaps it took signers longer to
    recognize Backsigns.
  • Some of the Backsigns had shorter distances
    covered than Fosigns, therefore the movement
    between holds was more minute and harder to see,
    possibly making it more difficult for signers to
    recognize the sign.

49
Summary of Results
  • Three way interaction effect of Sign type x Sign
    dir x Hand motion was nearly significant for
    Release and Response RTs
  • ACE was almost significant in Release RTs for
    sem/met (in sem/met vs. phon)
  • Two-way interaction effect for Sign dir x Hand
    motion (ACE) for Response RTs was significant for
    sem/met (in sem/met vs. phon) and met
  • Sem-signs had Response RTs which show nearly
    significant ACE
  • Phon-signs had Response RTs which showed no ACE
  • Backsigns cause a significantly longer release RT

50
Conclusions
  • Evidence for an Action-Language Compatibility
    Effect (as opposed to Action-Sentence)
  • ACE only occur when motor actions represent
    meaning
  • Motor action which is phonological and not
    semantic perhaps activates areas of the brain
    different from motor action which has semantic
    significance

51
Future Research
  • Run the experiment on more subjects (deaf,
    partially hearing and hearing)
  • Conduct a norming study to determine that Phon,
    Met and Sem signs are all valid
  • Conduct a norming study to control for dialectal
    variation in signing
  • Test other types of signs (deictics, phonetic vs.
    phonological motion)

52
Thank you
53
References
  • Glenberg, A. M. and M. P. Kaschak. (2002).
    Grounding language in action. Psychonomic
    Bulletin Review.
  • Liddell, Scott and Robert Johnson. (1985).
    American Sign Language The phonological base.
    Unpublished ms.
  • Liddell, Scott and Robert Johnson. (1986).
    American Sign Language Compound Formation
    Processes, Lexicalization, and Lexical
    phonological remnants. Natural Language and
    Linguistic Theory 4. 445-513.

54
Conditions
  • 66 signs total
  • 11 Phon-Fosigns, 11 Phon-Backsigns
  • 11 Met-Fosigns, 11 Met-Backsigns
  • 11 Sem-Fosigns, 11 Sem-Backsigns
  • Matching
  • Critical trials
  • Either responded by moving FORWARD or BACKWARD
    the body
  • Nonmatching
  • Noncritical trials
  • Fosigns matching with Backsigns and vice versa
  • Different types matched together randomly
  • i.e. Sem-Fosign with Phon-Fosign

55
Other Results
  • Release 1-way ANOVA by Items F(2, 57) motion x
    signtype x signdir 1.42, p .25
  • Response 2-way ANOVA (met) by Subjects ACE
    signdir x motion F(1, 26) 9.29, p lt 0.01
  • Release 3-way ANOVA (phon, sem/met) Response
    2-way ANOVA (sem) signdir x motion F(1, 26)
    1.87, p0.18
  • Response 2-way ANOVA (semet) signdir x motion
    F(1, 26) 6.83, p 0.02
  • Release 2-way ANOVA (phon) signdir x motion F(1,
    26) 1.67, p 0.21
  • Release 2-way ANOVA (semet) signdir x motion F(1,
    26) 1.87, p 0.18
  • Release 2-way ANOVA (sem) signdir x motion F(1,
    26) 2.07, p 0.16
  • Release 3-way ANOVA by Items (p,s,m) Sign dir x
    Hand motion F2(1, 57) 3.78, p 0.06

56
Release Near Significant ACE for Sem/Met Signs
  • signdir x motion F1(1, 26) 1.87, p0.18

57
Method - Implemenation
  • E-Prime on laptop
  • Detachable keyboard placed perpendicular to the
    body, with the a key closest to you. The h
    key is the middle key and the key is the key
    farthest from you.
  • Locations office cubicles and secluded areas
    outside

58
Results
  • AFFECTIONATE, BINARY and GOODLUCK
  • Eliminated because RTs were beyond 3 STDEV from
    the mean for item RT means
  • RTs beyond 3 STDEV replaced with the Maximum
    value (mean3 x STDEV)
Write a Comment
User Comments (0)
About PowerShow.com