Effects of Web-based Oral Activities Enhanced by Automatic Speech Recognition

1
Effects of Web-based Oral Activities Enhanced by
Automatic Speech Recognition
Tsuo-Lin Chiu, Hsien-Chin Liou, Yuli Yeh National
Tsing Hua University Taiwan R.O.C.
2
Overview

• I. Literature review
• II. CandleTalk
• III. Methodology of the evaluation study
• IV. Results and discussion
• V. Implications and conclusion

3
I. Lit review Communicative Competence

• Communicative competence has been the goals for
  the teaching of English as a second language and
  as a foreign language
• One of the components of communicative competence
  by Canale and Swain (1980) is sociolinguistic
  competence

4
Speech Act

• Speech acts (Austin, 1962 Searle, 1969) are the
  units of words that speakers use for achieving
  communicative functions
• Cohen and Olshtain (1993) explicit teaching of
  speech acts could help benefit in accelerating
  the process of developing communicative
  competence
• Cohen (1994) two abilities for explaining
  successful speech act performance
• Sociolinguistic ability selection of proper
  linguistic forms for speech act performance
• Sociocultural ability selection of proper speech
  acts based on sociocultural factors

5

• A recent CALL application is Automatic Speech
  Recognition (ASR) technology.
• 4 Types of ASR Recognizers (Wachowicz and Scott,
  1999)
• Speaker-independent ASR systems
• Speaker-dependent ASR systems
• Discrete word ASR systems
• Continuous ASR systems

6
ASR Advantages

• For teaching oral skills.
• Being a private tutor (for speaking)
• Simulated face-to-face conversation
• Face-saving
• Freedom of practice
• Suitable for assisting those who rarely have
  chances to speak with native speakers such as EFL
  contexts in Taiwan

7
ASR Limitations

• Recognition Errors
• Give false feedback

8
Review of ASR Programs

• MY English Tutor (MY ET) (,Chen Chiu, 2005
  Tsai, 2003)
• Subarashii (Bernstein et al., 1999)
• Virtual Conversations (Harless, Zier, Duncan,
  1999)
• The Voice Interactive Training System (VILTS)
  (Rypa Price, 1999)
• The FLUENCY (Eskenazi, 1999)
• Project LISTENs Reading Tutor (Mostow Gregory,
  1999)
• NTNU pronunciation site (Chen, 2005)

9

• Reviews of the literature showed that most
  studies on ASR were description of the system,
  pilot testing on a small number of subjects, or
  relying on learners self-report as measures of
  the effectiveness of environments.
• There seems to lack empirical data from a larger
  scale of testing that examines the effectiveness
  of ASR on EFL learners language learning (cf.
  Zhao, 2003) .

10
Purposes of the study

• Therefore, the current study
• presented the environment of CandleTalk
• examined learners perceptions of CandleTalk and
  its effectiveness on teaching target language
  skills.

11
Research Questions

• What is students perceptions toward a
  self-developed system, CandleTalk?
• Will the application of ASR technology in
  CandleTalk improve EFL students oral production
  by using appropriate and intelligible utterances?
  

12
A self-devised ASR system, CandleTalk
(http//candle.cs.nthu.edu.tw)

• Based on dialogues with content on Tawains
  culture
• Contents of Four Units
• Unit 1 greeting and parting, famous singer
• Unit 2 requesting, local resort
• Unit 3 complaining and apologizing, online games
• Unit 4 complimenting, night market

13
An example of Flowchart in CandleTalk (unit1)
14
Introduction Page
15
Selection of voices
16
Main page
17
Review Page
18
Dialogue Record
19
Methodology

• Participants
• 49 college freshmen from a public university
• 29 English majors, 20 non-English majors
• Instruments
• Online Questionnaire
• Using 5-point Likert Scale
• Scores were given (Strongly agree5, A4, N3,
  D2, SD1)
• A spoken version of Discourse Completion Test
  (DCT) as pre- and posttest

20
Example of DCT

• An example of DCT test
• You are going to go abroad for a week. You are
  worried that nobody can feed your cat while you
  are abroad. You are going to ask your friend to
  take care of your cat. What would you say?
• A sample answer might be
• Chris, I am going to Japan this week. I was
  wondering if you could help me take care of my
  cat?

21
Data collection procedures
Week Date Procedures
1 3/7 Consent form and background questionnaire (15-20 min)
2 3/14 Pretest and Units 1-2 in-class practice (100 min)
3 3/21 Units 1-2 home practice
4 3/28 Units 3-4 in-class practice (50 min)
5 4/4 Units 3-4 home practice
6 4/11 Posttest and evaluation questionnaire (45 min)

22
Data Analysis

• The results of the questionnaire were coded
  according to the categorizes of items
• Results of DCT were rated based on two criteria
• Comprehensibility of the speech.
• Use of speech acts.

23
Evaluation Questionnaire

• Results of evaluation questionnaire were analyzed
  and ranked based on three categories
• Items concerning the 4 units
• Functionality of the site
• Overall effectiveness of CandleTalk

24
Items of the 4 units
Rank Item number Items Mean
1 7 The introduction page tells me the focus of the unit 4.07
2 8 The introduction page tells me the context of the dialogue 4.01
3 9 I learned how to perform appropriate speech acts from the unit 3.98
4 13 The review page tells me how to use the dialogues taught in the unit 3.94
5 11 The difficulty level of the dialogue in the unit is appropriate 3.87
6 10 The topic of the dialogue in the unit is interesting 3.65
7 12 The recognition rate in the unit is appropriate 3.02
25
Functionality of the site
Rank Item number Items Mean
1 16 The responses from the computer are clear 4.16
2 17 The directions of operating the environment are clear 4.10
3 22 The word color and font size are clear and easy to read 4
4 18 I like the selection of talking to a male or a female 3.96
5 21 The interface design in the environment is acceptable 3.86
6 23 The color and pictures in the environment are appropriate 3.84
7 20 The navigation in the environment is free. It is easy for me to move around in the environment 3.78
8 19 The recording procedure is easy 3.57
26
Overall Effectiveness
Rank Item number Items Mean
1 25 The environment helps me learn how to speak according to contexts 3.80
2 26 The recognizer helps me pay attention to my English conversation 3.73
3 27 The environment fits my expectation of learning English 3.59
4 15 The multiple choices in each dialogue make me feel like talking to a real person 3.55
5 24 Interacting with computer is helpful for me to learn English dialogues 3.51
6 14 The flow of the dialogues in each unit is fluent 3.37
27
Major Results from Questionnaires

• The first research question is well-supported by
  the results of the questionnaire that students in
  general held positive attitude toward CandleTalk
  regarding its pedagogical purposes, designing
  features, and interface
• The recognition rate and the recording interface
  require improvement

28
Rating of DCT Responses

• A total of ten questions (with different orders
  in the pre- and posttest)
• An English native-speaker professor helped with
  range finders by listening to 20 samples
  (49x2x10), 10 recorded sound files from pretest
  and 10 from posttest
• The data were rated by two raters who are both MA
  TEFL students, disagreements solved by discussion
  with interrater reliability of .87
• Full score of the test is 50 (3x10 2x10)

29
Rating of DCT Responses

• Comprehensibility (3-point scale,) is about how
  easy to understand an utterance (Munro Derwin,
  1995)
• 0 point totally incomprehensible utterance
• 1 point utterance which contains poor
  pronunciation or is difficult for comprehension
• 2 point utterance which is fully comprehensible
• Use of Speech acts (4-point scale)

30
Rating scale of Use of Speech Acts
Score Descriptions
0 No answer or fully misuse of a speech act
1 Correct use of speech act based on the question, but poorly formulated, inappropriate, or incomplete
2 Correct use of speech act, appropriate, but contains many non-native features.
3 Correct use of speech act. Comprehensive and appropriate
31
Comparison of pre- and posttest

• Paired T-Test was used to see whether there are
  significant differences in scores
• Results were analyzed from three categories
• Total score (502030)
• Comprehensibility (full score20)
• Use of speech act (full score30)

32
Comparison of pre- and posttest (total score)
Pretest Posttest Gain in scores
Mean 40.85 42.32 1.47
Number 49 49 49
SD 5.51 4.32 3.63
df 48
t Stat -2.84
P(Tltt) two-tail 0.01
T Critical two-tail 2.01
33
Comparison of pre- and posttest
(comprehensibility)
Pretest Posttest Gain in scores
Mean 17.80 17.92 o.1
Number 49 49 49
SD 2.48 2.28 1.53
Df 48
t Stat -0.57
P(Tltt) two-tail 0.57
T Critical two-tail 2.01
34
Comparison of pre-and posttest (use of speech
act)

Pretest Posttest Gain in scores
Mean 23.06 24.39 1.31
Number 49 49 49
SD 3.89 2.64 3.03
df 48
t Stat -3.06
P(Tltt) two-tail 0.003
T Critical two-tail 2.01
35
Comparison of two groups (total score)
FL FL FL FL NonFL NonFL NonFL NonFL
N Mean SD t N Mean SD t
Pretest 29 42.28 4.53 -0.17 20 38.78 6.24 -4.73
Posttest 29 42.38 4.55 -0.17 20 42.23 4.07 -4.73
Gain in scores 29 0.1 3.26 -0.17 20 3.45 3.26 -4.73
plt .05
36
Comparison of two groups (comprehensibility)
FL FL FL FL NonFL NonFL NonFL NonFL
N Mean SD t N Mean SD t
Pretest 29 18.60 1.30 1.30 20 16.63 3.52 -2.05
Posttest 29 18.29 2.02 1.30 20 17.38 2.58 -2.05
Gain in scores 29 -0.31 1.28 1.30 20 0.75 1.39 -2.05
plt .05
37
Comparison of two groups (use of speech act)
FL FL FL FL NonFL NonFL NonFL NonFL
N Mean SD t N Mean SD t
Pretest 29 23.67 3.94 -0.72 20 22.18 3.74 -4.22
Posttest 29 24.05 3.03 -0.72 20 24.88 1.91 -4.22
Gain in scores 29 0.38 2.80 -0.72 20 2.70 2.86 -4.22
plt .05
38
Major Results From the Comparison of DCT Results

• Students showed significant increase in total
  score and use of speech act, but not in
  comprehensibility
• CandleTalk is effective in helping students how
  to perform the six speech acts appropriately, but
  it did not significantly help the students
  improve the clarity of their speech

39
Major Results From the Comparison of DCT Results

• When group differences were concerned, the non-FL
  group showed significant improvement in the
  categories of total scores and use of speech act,
  but the FL group did not show any significant
  improvement in any of the three categories
• The pretest scores of non-FL group were lower
  than those of FL group, but in the posttest
  non-FL group caught up.

40

• Answer to the second research question
• the students did improve their oral skills in
  terms of the use of speech acts, but the salient
  improvement was mostly made by the non-FL group,
  which is also the relatively lower proficiency
  group

41
Students perceptions toward the environment

• Students were positive toward CandleTalk, and
  believed it met their expectation of learning
  English
• The results consistent with previous studies
  (Harless, et al., 1999, Hsia, et al., 2004, and
  Tsia, 2003)
• Use of native culture as topics of dialogues
  (Savignon Sysoyev, 2002) and the feature of
  simulated real-life conversation (e.g. Bernstein,
  et al., 1999, Wachowicz Scott, 1999) seem
  helpful.

42
Students perceptions toward the environment

• The value of feedback given by the speech
  recognizer (e.g., Eskenazi, 1999)
• The recognition rate/accuracy
• The recording interface design

43
Students learning outcome after the instruction

• An ASR-supported learning environment, like
  CandleTalk, is effective on enhancing the target
  language skills
• The result is similar to Harless, et al, (1999)
  in which pedagogy and ASR technology combined
  could enhance or sustain the language skills
• Students of lower language proficiency seem to
  benefit more from the instruction supported by
  ASR
• The result is similar to Tsia (2003) that it is
  the lower level group benefited more form the
  application of speech recognition technology
• Limited improvement of the FL group

44
Development Implications for an ASR supported
Conversation Environment

• Innovative formats of dialogue presentations are
  needed
• The interface design should be user-friendly
• The input provided by the environment could be
  diverse and relevant to learners

45
Development Implications for an ASR supported
Conversation Environment

• The feedback given by the recognizer should be a
  reliable and helpful indicator for learners
  performances in speaking
• It takes the collaboration between the subject
  matter experts and the programmers to construct a
  better recognizer and further explore the
  potentials of ASR on language learning

46
Pedagogical Implications

• Instructional materials supported with ASR could
  be an effective alternative for teaching and
  learning of speaking
• The explicit teaching of use of speech acts would
  help facilitate EFL learners development of
  communicative competence
• Clear orientations and directions during CALL
  instruction are still needed

47
Limitations of the Study

• No control group
• The use of Discourse Completion Test

48
Suggestions for Future Research

• To explore the application of ASR on other
  aspects of oral skills
• To investigate the effectiveness of ASR on
  learners in other contexts, of different ages,
  and of different proficiency levels

49
The End

• This project, Naitonal Science and Technology
  project for E-learning, under the grant number
  NSC 92-2524-S007-002, is supported by National
  Science Council, Taiwan

Thank you for your attention! Hsien-Chin Liou
hcliu_at_mx.nthu.edu.tw Tsuo-Lin Chiu
g925256_at_alumni.nthu.edu.tw
50
(No Transcript)
51
Principles of Sociocultural Strategies

• Savignon and Sysoyev (2002) sociocultural
  strategies
• Two main principles
• Dialogue of culture
• Acting as representatives of culture

52
Examples of use of speech acts

• Questions You completely forget a crucial
  meeting with your friend. An hour later you call
  him/her to apologize. The problem is that this is
  the second time youve forgotten such a meeting.
  What would you say to your friend?
• 3 points
• 2 points
• 1 Point