Helen Shen and Chen-hui Tsai

1
Evaluating Reading Proficiency Gain and Program
Effectiveness A Three-tiered Web-Based Reading
Assessment Approach

• Helen Shen and Chen-hui Tsai
• The University of Iowa

2
Introduction

• This presentation reports the evaluation
  system for The comprehensive web-based Chinese
  Reading Program supported by a DOE grant.
• The program
• The evaluation

3
The Program Purpose

• To remedy the acute dearth of appropriate Chinese
  reading materials (beyond the class textbook)
• To furnish a superior Chinese reading environment
  in which reading can take place anytime, anywhere
• To improve users vocabulary knowledge, reading
  skill, and cultural knowledge through extensive
  reading

4
The Program The Targeted Users

• College students who are taking Chinese courses
  at beginning, intermediate, and advanced levels
• High-school students who are taking Chinese AP
  courses
• Individuals who work in government agencies or
  private organizations and who wish to improve
  their reading skills in a systematic way

5
The Program Content

• Three levels of reading materials (900 Lessons)
• 90 achievement tests
• 15 proficiency tests

6
The Program Format

• Flash-embedded HTML files
• Image files
• NJStar files
• Audio HTML files

7
The Program Academic strength

• Proficiency-guided definable reading levels
• Cultural theme-based reading unit organization
• Fluency-oriented progressive reading skill
  building
• Emphasizing Intentional vocabulary gain
• Supporting Constructive responses
• Three-tiered web-based assessment system

8
The Program The Design
9
The Evaluation

• A Three-tiered web-based assessment system is
  developed to assess both individual reading
  achievement and the overall program effectiveness
• The flash-based built-in reading outcome
  assessment for individual users
• The FileMaker Pro reading database
• The Quia web survey tool

10
The Flash-based Reading Outcome Assessment for
Individual Reading Proficiency Gain

• Vocabulary pre- and post tests
• Multiple choice and open-ended Reading
  comprehension questions
• Reading speed and comprehension rate (REI) report
  
• Achievement tests
• Proficiency tests

11
The FileMaker Pro Database

• This reading database allows us to record
• User contact information
• Users reading activities (reading time, reading
  comprehension score, reading lessons completed,
  vocabulary tests score.)
• Built-in on-line survey

12
The Evaluation Quia Web Survey

• The Quia survey data allow us to collect users
  additional quantitative and qualitative feedback
  on certain arising issues during the use of the
  program.
• http//www.quia.com/sv/246903.html

13
The Evaluation Data Analysis

• Data from FileMaker Pro
• Distribution of Users in the world
• Distribution of Reading levels and topics among
  users

14
The Evaluation Data Analysis
Distribution of Users in the world (78
countries 1771 valid entries)
Africa America Asia Europe Oceania
South Africa 8 USA 1171 China 204 United Kingdom 84 Australia 78
Argentina 4 Canada 77 Thailand 48 Germany 46 New Zealand 11
Algeria 2 Brazil 25 Singapore 33 France 42 Fiji 1
15
The Evaluation Data Analysis
Distribution of Reading levels and topics
among US users (2741)
Reading Level L1 Beginning L2 Intermediate L3 Advanced
Frequency of Read lessons 1712 840 189
Popular units U1 / 720 ???????? U11/ 493 ???? U 10/ 125 ?????
U5/ 266 ????? U1/ 116 ???? U20/ 14 ???????
U13/ 114 ?? U3/ 42 ???????? U1/ 12 ????????
Units not read U8/18/22 U6/8/9/16-30 U2-9/12-16/18/24/26/28/30
16
The Evaluation Data analysis

• Data from The Quia web survey One month
  reading activity and survey (February 2009)
• Distribution of users and age groups in the world
• Qualitative assessment of program effectiveness
• Quantitative assessment of program effectiveness

17
The Evaluation Data analysis
Distribution of users nationality and age
groups (valid entries 87), Feb. 09
Africa America Asia Europe Oceania
Nationality 1 37 35 11 3
Age Range Age group ranking 21-30 Under 20-above 50 21-30 Under 20 31-40 Under 20 above 50 21-30 31-40 Under 20 Under 20-above 50 21-30 Under 20-40
18
The Evaluation Qualitative
Purpose Of helpfulness The part like most Problem Suggestion
Teacher Use Improve reading skills Articles in various topics The sound file does not work for me Include pinyin in the reading text
Student use Improve listening skills Articles are interesting very close to real life more beginning level topics
Self -study Expand vocabulary Audio files Make sound files downloadable
Pleasure reading Improve reading speed Njstar files
Improve cultural knowledge The interface is appealing and easy to use

• Qualitative data

19
The Evaluation Quantitative
Asia from a 7-point scale 35 users
images are clear.
20
The Evaluation Quantitative

• America From a 7-point scale 37 users

images are clear.
21
The Evaluation Quantitative
T-test on the comparison of the ratings between
Asia and US
Population Item number M SD t Sig.
Asia User 12 5.43 .22 -1.13 .272
USA User 12 5.54 .26
22
Conclusion

• The three-tiered web-based assessment system
  is an excellent approach conducting the formative
  assessment for the program, which helps us to
  know users needs, expectations, and problems they
  are encountering so that we can adjust the
  program based on users direct and indirect
  feedback. This system contributes to maintain a
  long-lasting high quality program.

23
The reading program is free to all users!!!

• http//chinesereadingworld.org
• http//collections.uiowa.edu/chinese
• Contact information
• Helen-Shen_at_uiowa.edu
• Chen-hui-Tsai_at_uiowa.edu

24
The End
25
HTML file
Back
26
NJStar
Back
27
Audio files
Back
28
FileMaker
Back
29
Pretest
Go to Post-test
30
Posttest
Back
31
multiple
Go to Open Q
32
Open-ended Questions
Back
33
Final Report
Back
34
Achievement Test
Back
35
Proficiency Test
Back
36
FileMaker
Back
37
Built-in Survey
Back
38
Quia Survey