DEVELOPMENT OF AN ONLINE READING TEST

1
DEVELOPMENT OF AN ONLINE READING TEST

• Associate Professor Dr. Alisa Vanijdee
• Dean, School of Liberal Arts
• Sukhothai Thammathirat Open University, Thailand
• Alisa.Van_at_stou.ac.thavanijdee_at_hotmail.com

2
OBJECTIVES

• (1) develop an online English reading test
• (2) validate the quality of the test
• (3) study the reading performance-level of the
  test-takers
• (4) analyse common weaknesses found in the test
  results, and
• (5) propose guidelines to improve the reading
  performance of test-takers.

3
INFORMANTS

• 31 academics and graduate students

4
RESEARCH INSTRUMENTS

• (1) 3 sets of English reading tests
• (2) test evaluation forms, and
• (3) a test-specifications crosscheck form.

5
RESULTS

• (1) 3 sets of online reading tests posted on the
  website of Sukhothai Thammathirat Open
  University

6
RESULTS

• (2) the tests are qualified as specified
• the tests were designed based on the reading
  process and the reading evaluation process, with
  the reading performance level based on CEFR
  level

7
the tests were modified according to the IOC
scores of lower than .50 and the experts and
pilot-test takers suggestions together with the
trial test scores of the pilot-test takers
8
the reliability of the 3 sets was analysed by
Kuder Richardson 20 formula, which yielded .86,
.88, and .61 respectively the difficulty and
discrimination was accepted at .20 lt p lt .80
the rest were modified
9
RESULTS

• (3) the reading performance level of the 3 sets
  of tests

marks level Number of test takers Number of test takers Number of test takers
marks level Version 1 Version 2 Version 3
76-100 C 1 (effective operational proficiency) 3 3 2
51-75 B 2 (independent user vantage) 12 14 11
26-50 B 1(independent user threshold) 16 14 18
0-25 Needs improvement 0 0 0
Total number 31 31 31
10
RESULTS

• (4) the test-takers weaknesses included
  insufficient understanding of complex structure
  and lack of vocabulary
• (5) the guidelines for reading improvement focus
  on reading strategies practice at the cognitive,
  metacognitive, and socio-affective level.

11
DEVELOPMENT PROCESS

• Theoretical Framework
• - Reading process framework
• (Khalifa Weir, 2009 62)
• - designed based on the mikroskills
• from Alderson (2000) and
• - test usefulness by Bachman
  Palmer
• (1996)

12
Test specifications for Reading

• CEFR
• Unaldis (2009) comprehensive framework
• Text difficulty
• -vocabulary profile of lexitutor
• -Flesch-Kincaid Grade Level

13
Test specifications for grammar

• selected from Grammar City Guilds ESOL

14
Test specifications for Vocabulary

• AWL list

15
TEST SPECIFICATIONS

• designed based on the mikroskills from Alderson
  (2000) and test usefulness by Bachman Palmer
  (1996).
• Unaldi (2009) detailed specification

16
Unaldis detailed specifications

• CONTEXTUAL
•  
• Response Method Gap filling
•  
• Text length up to 500 words

17
Genre everyday materials such as letters and
emails(e.g., enquiries, orders, letters of
confirmation etc.), public information leaflets
brochures, short official documents straightforwar
d instructions for equipment expository and
informative newspaper/magazine articles on
familiar subjects personal letters with
description of events, feelings and wishes simple
informational sources (e.g., junior
encyclopaedias, leaflets and brochures)
18
Rhetorical task narrative, descriptive,
instructive, expository Pattern of exposition
define, describe, elaborate, illustrate, compare
and contrast, classify Explicitness of text
structure explicit
19
Structural resources Words/sentence Ave
Flesch-Kincaid Grade Level Ave 8 The
complexity of sentence structure mostly simple
sentences (but some use of subordinate clauses
in PET) Cohesion explicit
20
Lexical resources 95 K1-20 words K1 84.7
K2 8.7 K3 2.3 AWL words 2.5
21
Nature of information (abstract/concrete)
concrete Content knowledge not required
everyday situations encountered in work, school,
leisure etc., (personal feelings, opinions and
experiences, hobbies and leisure familiar topics
in expository texts
22

• COGNITIVE
• Type of reading
• - understand the main points and/or relevant
  points though not necessarily in detail
  (description of events, feelings and wishes,
  significant and clearly signalled reasoning, and
  argumentation)
• identify unfamiliar words from the context
• extrapolate the meaning of occasional unknown
  words from the context and deduce sentence
  meaning

23
Text level word, sentence and across sentences
24
SAMPLES OF TEST SPECIFICATIONS ANALYSIS
? See word file
25
Components of the Test

• Grammar
• Vocabulary AWL words
• Reading
• ? Test on CD

26
The level of difficulty of the Vocabulary

• processed through lexitutor software
  (http//www.lextutor.ca/vp/). The analysis
  revealed frequencies of vocabulary, types and
  levels (K1, K2, K3, AWL) types, tokens,
  families, tokens per family, types per family).
  
• Flesch-Kincaid Grade Level was also taken into
  consideration when adapting reading texts to
  correspond the specifications in B1 B2 C1.

27
(No Transcript)
28

• Thank you.