Evaluating TaskBased Language Programs

1
NOTE This PowerPoint presentation has been
modified by removal of all high-resolution
graphics, to reduce the storage and downloading
requirements (sorry, no pretty photos!).
Evaluating Task-Based Language Programs Colloquium
TBLT 2009
2
The colloquium
?Why bother with TBLT program evaluation? ?Three
presentations clarification questions
(200-330) Re-framing the evaluation of
task-based language education Evaluating a TBLT
Spanish immersion program Evaluation of TBLT in
Flanders ?Open audience-panel discussion
(330-350)
3
John M. Norris University of Hawaii at
Manoa TBLT 2009
Please cite as Norris, J. M. (2009, September).
Reframing the evaluation of task-based language
education. Paper presented at the refereed
colloquium Evaluating task-based language
programs, at the 3rd International Conference on
Task-Based Language Teaching, Lancaster, UK
(September 14, 2009).
Re-framing the evaluation of task-based language
education
4
TBLL v. TBLT Disconnects between inquiry and
practice
5
What is task-based language learning (TBLL)?
1. Societal need for change in language
education ?Value ?Outcomes ?Methods
2. Emerging notions of L2 acquisition ?Processes
?Impediments ?Indicators
6
What is task-based language learning (TBLL)?
Practices
1. Societal need for change in language
education ?Value ?Outcomes ?Methods
Observations
Proposals
Opportunity for a researched language pedagogy
Discussions
2. Emerging notions of L2 acquisition ?Processes
?Impediments ?Indicators
Hypotheses
Findings
7
What is task-based language learning (TBLL)?
Pedagogic principles, such as
?Promote learning by doing, experiential learning
?Use task as the unit of analysis for instruction
assessment
?Provide rich L2 input
?Elaborate (rather than simplify) L2 input
?Respect learner-internal syllabuses
?Enable inductive/chunk learning
?Promote collaborative-cooperative interaction
?Provide focus on form, negative feedback
(e.g., Doughty Long, 2003 Ellis, 2003)
8
What is task-based language teaching (TBLT)?
Rationales and Principles
L2 Education Programs
Applied to
Philosophy of Education
Learners
Task-Based Language Teaching
Cognitive Psychology
Needs
Curriculum
Sociocultural theory
Instruction
SLA
Materials
Curriculum theory
Assessment
Planning and policy
Inform???
Teacher development
9
What is the role of task-based inquiry?
Test hypotheses
DISCRETE
GENERALIZABLE
Discover robust, if small, truths
Task-Based Language Learning
Generate theory
Task-Based Language Teaching
Understand what works, where, when, why
Inform curriculum, course design
HOLISTIC
SITUATED
Improve teaching practice
10
What is the role of task-based inquiry?
Test hypotheses
DISCRETE
GENERALIZABLE
Discover robust, if small, truths
Task-Based Language Learning
Generate theory
Task-Based Language Teaching
Understand what works, where, when, why
Inform curriculum, course design
HOLISTIC
SITUATED
Improve teaching practice
11
What is the role of task-based inquiry?
Test hypotheses
DISCRETE
GENERALIZABLE
Discover robust, if small, truths
Task-Based Language Learning
Generate theory
Task-Based Language Teaching
Understand what works, where, when, why
Inform curriculum, course design
HOLISTIC
SITUATED
Improve teaching practice
12
Challenges for task-based inquiry
1. The scope of task-based research does not
match the scope of our claims about (for or
against) TBLT.
2. The focus of theoretical task-based research
does not relate to the situated realities of
task-based teaching.
Needs Curriculum Materials Instruction Teachers Le
arners Assessment
Goals, outcomes Scope, sequence Resources Practice
s History, training Individual differences Intende
d uses, users
TBLT Education Programs
13
Framing TBLT inquiry through program evaluation
14
Inquiry through evaluation
Research emphasizes theoretical,
conclusion-oriented inquiry Evaluation
operationalizes decision-oriented inquiry
Cronbach Suppes (1969)
15
Inquiry through evaluation
Evaluation is the gathering of information about
any of the variety of elements that constitute
educational programs, for a variety of purposes
that include primarily understanding,
demonstrating, improving, and judging program
value evaluation brings evidence to bear on the
problems of programs, but the nature of that
evidence is not restricted to one particular
methodology. Norris (2006) MLJ Perspectives
Inquiry frame and focus
Inquiry impetus
Inquiry question prioritization
16
Inquiry through evaluation
Paradigms
Cronbach et al. (1980)
The evaluator will be wise not to declare
allegiance to either a quantitative-scientific-sum
mative methodology or a qualitative-naturalistic-d
escriptive methodology. (p. 7)
Epistemology 1
Epistemology 2
Methodology 1
Methodology 2
17
Inquiry through evaluation
Paradigms
Cronbach et al. (1980)
The evaluator will be wise not to declare
allegiance to either a quantitative-scientific-sum
mative methodology or a qualitative-naturalistic-d
escriptive methodology. (p. 7)
Epistemology 1
Epistemology 2
Methodology 1
Methodology 2
18
Inquiry through evaluation
Pragmatism
Cronbach et al. (1980)
The evaluator will be wise not to declare
allegiance to either a quantitative-scientific-sum
mative methodology or a qualitative-naturalistic-d
escriptive methodology. (p. 7)
Why?
Who?
What?
When?
Method 1
Method 2
Method 3
Method 4
Method 5
19
Inquiry through evaluation
1. Participation stakeholders, representatives,
primary intended users
2. Prioritization challenges, questions in
immediate need of answers
3. Instrumentation what data will answer the
questions?
4. Collection how can we get data in available
time/resources?
5. Interpretation what do findings mean in
context?
6. Utilization what decisions actions are
taken?
Participation by language educators is essential
throughout evaluation if contextual relevance is
sought.
A focus on specific intended uses for evaluation
findings is essential from the outset, if
evaluation is to make any difference.
Language educators are ultimately responsible for
what happens in language education.
20
Context Program features
Learner needs, institutional resources, program
goals and outcomes, curriculum, materials,
instruction, assessment, teachers, teacher
development, learners, etc.
Context Intended uses
Context Intended users
Evaluating TBLT Programs in situ
Understand Improve Educate Demonstrate worth Hold
accountable Empower (Test theory)
Teachers Administrators Curriculum
writers Learners Parents/public Funders (Researche
rs)
Questions Methods
Values clarification Implementation Process-produc
t Outcomes
Patton (1997) Utilization-focused evaluation
21
Learning from evaluation examples the brief
history of TBLT
22
Learning from TBLT evaluation
Evaluating the Communicational Teaching Project
Prabhus Bangalore Project (See Prabhu, 1987)
Context
Theory
Program
English L2 Education in Bangalore, India
Seeking Improvement via Innovation
L2 learning by processing meaning
Unconscious grammar construction by learners
Project/task- based work 4 experimental schools
Implemented 1979-1984
23
Learning from TBLT evaluation
Evaluating the Communicational Teaching Project
Phase 1 (See Beretta Davies, 1985)
Initial inquiry, final year of the project
To assess, through appropriate tests, whether
there is any demonstrable difference in terms of
attainment in English between classes of children
who have been taught on the CT project and their
peers who have received normal instruction in the
respective schools. Beretta Davies (1985)
24
Learning from TBLT evaluation
Evaluating the Communicational Teaching Project
Phase 1 (See Beretta Davies, 1985)
Purpose ?Test theory ?Demonstrate method
effectiveness
Findings ?Structures test Control gt
CTP ?Contextual grammar Control
CTP ?Dictation Control CTP ?List/Read comp
CTP gt Control ?Task-based test CTP gt Control
Claims??? ?Task-based learners achieved as much
or more than traditional on all but the least
functional outcomes ?Task-based instruction is
successful Warranted claims???
What do we really know???
Methods ?Quasi-experimentation ?Class/method
comparison ?Outcome achievement assessments
25
Learning from TBLT evaluation
Evaluating the Communicational Teaching Project
Phase 2 (See Beretta, 1986, 1990, 1992)
Purpose ?Understand program implementation ?Illumi
nate relation with apparent outcomes
Findings ?Lack of comparability (intact classes,
no baseline data) ?More qualified teachers in CTP
classes ?Implementation of CTP highly variable
(over time, between classes, with
structures) ?More confident teachers better
results
Methods ?Retrospective interview
protocols ?Teacher level of concern
questionnaires ?Document analysis
26
Learning from TBLT evaluation
Evaluating the Communicational Teaching Project
Lessons Learned (See Beretta, 1992)
?Theory testing, methods comparisons, what
works claims are rarely feasible in real
educational programs
?Apparent differences in learning achievement,
behaviors, etc. can only be explained by
observation of multiple factors as they are
experienced in real program contexts
?Even poorly executed evaluations (e.g.,
post-hoc) can shed light on how programs function
and help explain why learning does or does not
occur
?Teachers (beliefs, training, commitment, time)
play a key role in implementing programs what
they actually do must be understood
27
Learning from TBLT evaluation
Evaluating a university French curriculum
Student perspectives (See Towell Tomlinson,
1999)
Context
Theory
Program
French FL Education, UK university, Salford
Restructuring advanced FL teaching
Input, text, task Learning through
form- function mapping in tasks
Task-based syllabus Multiple levels at
university Implemented 1988-96
28
Learning from TBLT evaluation
Evaluating a university French curriculum
Student perspectives (See Towell Tomlinson,
1999)
Multiple iterations of development,
implementation, evaluation, revision
Curriculum design, evaluation, application and
enhancement is a slow process, and subject to a
number of extraneous influences which make it
impossible to measure with totally scientific
precisionuse of diaries and questionnaires on
the first occasion enabled a number of lessons to
be learned and these helped considerably in
creating a second application where the testimony
of the student population through a detailed
questionnaire shows the success of the
operation. Towell Tomlinson (1999)
29
Learning from TBLT evaluation
Program Initial TBLT ?long group projects
Program Revised TBLT ?staged, short projects

• Methods
• Learner diaries
• Learner surveys
• Assessments/exams

• Methods
• Learner surveys
• Focus groups
• Assessments/exams

STAGE 1
STAGE 2

• Findings
• Projects too long (6 wks)
• Training in group work
• Staged task objectives
• Gains in text/task learning
• Developing accuracy?

• Findings
• Increased satisfaction
• Higher learning of skills
• Improved oral translation
• Written translation?
• Developing accuracy?

30
Learning from TBLT evaluation
Evaluating a university French curriculum
Student perspectives (See Towell Tomlinson,
1999)
?Student/learner perspective on teaching with
tasks sheds important light on the realities of
implementation (how how well)
?Building evaluation activities into curricular
delivery from the outset (e.g., student diaries),
enables longitudinal insights about change,
development, response to instruction
?Triangulated learner feedback (diaries,
self-assessments, questionnaires, exams) can lead
to effective improvements in curriculum and task
design, and in turn to higher evaluations
?Learners can change how they learnacculturating
to TBLTespecially when tasks, instructions,
assessments are intentionally designed and staged
to do so
31
Learning from TBLT evaluation
Evaluating TBLT for EAP Developmental
evaluation in Thailand (See McDonough
Chaikitmongkol, 2007)
Context
Theory
Program
English FL education, Thai university
Improving EAP instruction
Integrated-skills, communication
Life-long learning Learner needs
interest orientation
Task-based syllabus English department Implemen
ted 12 months
32
Learning from TBLT evaluation
Evaluating TBLT for EAP Developmental
evaluation in Thailand (See McDonough
Chaikitmongkol, 2007)
Inquiry for developing and improving TBLT
experiences
relatively few empirical studies have
documented how teachers and learners react to
entirely task-based courses, as opposed to the
use of individual tasksThe purposes of this case
study were (a) to identify teacher and learner
reactions to the course and (b) to describe how
their concerns, if any, were addressed.
McDonough Chaikitmongkol (2007)
33
Learning from TBLT evaluation
Evaluating TBLT for EAP Developmental
evaluation in Thailand (See McDonough
Chaikitmongkol, 2007)

• Methods
• Learner task evaluations (repeated)
• Learning notebooks
• Class observations
• Student course evaluations
• Teacher/student interviews
• Observer field notes

• Findings
• Increased learner independence, language skills,
  learning strategies
• Decreased grammar obsession
• Non-specific real-world relevance
• Need time to adjust (TL)
• More support, guidance from teachers
• Too much to cover, disparate materials

• Uses
• Intro unit on language learning
• Teachers guide to instruction workshop
• Enhanced task guidelines, built-in feedback
  opportunities
• Reduced number of tasks
• Consolidated materials

34
Learning from TBLT evaluation
Evaluating TBLT for EAP Developmental
evaluation in Thailand (See McDonough
Chaikitmongkol, 2007)
?Cycles of evaluation planned into TBLT
innovation, and carried throughout, can lead to
increased likelihood of effectiveness
?Systematic evaluation (a) from multiple
stakeholder perspectives and (b) focused on
multiple program elements (materials,
preparedness, outcomes) enables balanced change
?Teachers and learners both require support in
implementing TBLT, especially during early phases
of introducing task-based instruction
?TBLT based on learner needs can work well in EAP
contexts, especially when evaluation is used to
support on-going effectiveness of delivery from
the outset
35
Learning from TBLT evaluation
Evaluating TBLT teacher training Cyclical
evaluation in Belgium (See Van den Branden, 2006)
Context
Theory
Program
Dutch SL Education in Flanders Nationwide
K-16 Ensuring Educational Access, Equity
Large-scale Task-Based LT Innovation Improving
Functional DSL Abilities
School-based Teacher- Training Programs Enabling
Change, 1994-2003
36
Learning from TBLT evaluation
Van den Branden (2006) the teacher tries to
act as a true interactional partner, negotiating
meaning and content with the students, eliciting
and encouraging their output, focusing on form
when appropriate and offering them a rich,
relevant and communicative input (p. 217).
Evaluation PROBLEMS
Evaluation USES
Teacher cognition
?Understand teachers ?Illuminate context ?Improve
T-dev program ?Encourage teacher agency ?Ensure
teaching success ?Enable TBLT learning ?Demonstrat
e outcomes

• What do they theorize about TBLT?
• How do they learn about TBLT?

Teacher action

• Are they willing to change with syllabus?
• How do they adopt/adapt TBLT in practice?

Teaching context

• What are the social constraints on T-Dev?
• How can T-Dev be optimized?

37
Learning from TBLT evaluation
Program Theoretical inservice training
Program TB training syllabus support
Program Training coaching agency

• Methods
• Teacher survey
• Training observation

• Methods
• Teacher logs, interviews, classroom observations

• Methods
• Coaching obs, classroom obs, coach/ teacher
  interviews

STAGE 1
STAGE 2
STAGE 3

• Findings
• Transmission model
• Short term (3 hrs.)
• Try that with my studentsPost-coursal
  depression!

Findings awareness of TBLT student enthusiasm
?teacher adoption -teacher control
-task complexity -groupwork
Findings conscious decisions TBLT adaptation
self-evaluation ?teacher control
-transfer -groupwork
38
Learning from TBLT evaluation
Program Sustained implementation of TBLT T-Dev
with coaching, support

• Methods
• Pre-post student learning outcomes, teacher
  surveys, classroom observations

STAGE 4
Findings incorporation of TBLT correlated with
higher Dutch L2 proficiency outcomes 3-year
gains in DSL higher in TBLT intensive adopting
schools ?mixed incorporation of TBLT across
schools, teachers
39
Learning from TBLT evaluation
Evaluating TBLT teacher training Cyclical
evaluation in Belgium (See Van den Branden, 2006)
?Long-term evaluation of TBLT sheds light on how
ideas are implemented, how participants change,
and what support is needed
?Multi-directional evaluation (political, social,
school, individual) increases our capacity to
explain why task-based ideas work or do not
?Persistent follow-through on evaluation findings
(use) underlies effective innovation
?Teachers can learn to engage with TBLT, but
change takes time, requires individualized
support, and must be valued
40
Learning from TBLT evaluation
What have we learned? Reframing evaluation in TBLT
?From summative to intentional
?From assessment-driven to multi-methodological
?From external to participatory
?From method-testing to program-illuminating
?From one-shot to longitudinal, cyclical
?From theoretical conclusions to educational
decisions
41
Research, evaluation, and the future of
task-based education
42
TBLL research
? Sociocultural, cognitive, and other theories
provide useful starting points for thinking about
language teaching and learning, and offer
principles for building educational programs
? Task-based language learning research helps in
that it raises our awareness about particular
factors that we should pay attention to in the
instructed L2 learning process
? Task-based language learning research cannot
tell us much about how or why language education
programs work findings from TBLL research should
not be interpreted as direct implications for
TBLT education
43
TBLT evaluation
Answers questions informs decisions of local
interest
Sheds light on how TBLT ideas work in practice
Provides truths situated in rich contexts of
programs
Intentional Evaluative Inquiry
Relates outcomes to TBLT delivery and other
factors
Focuses on scope that is meaningful to teachers,
learners
Tests and informs innovation on the ground, in
situ
Empowers participants to learn, and learn to
change
Forces an honest accounting of TBLT
44
TBLT evaluation
Resources It takes time and money to do
evaluation well and to sustain it within L2
educational programs.
Challenges For TBLT Evaluation
Training Effective evaluation calls upon skills
that may not be easily available among personnel
at hand.
Dissemination There are few venues for
publishing evaluation reports, thereby limiting
learning.
Actual uses There are many possible uses/needs
for evaluation that we are not sufficiently
attuned to, yet.
45
Cheers! (Mahalo!)
References Beretta, A. (1986). Program-fair
language teaching evaluation. TESOL Quarterly,
20, 431-445. Beretta, A. (1990). Implementation
of the Bangalore Project. Applied Linguistics,
11(4), 321-340. Beretta, A. (1992). What can be
learned from the Bangalore evaluation? In J. C.
Alderson and A. Beretta (eds.), Evaluating second
language education (pp. 250-273). Cambridge
Cambridge University Press. Beretta, A.,
Davies, A. (1985). Evaluation of the Bangalore
Project. ELT Journal, 39(2), 121-127. Cronbach,
L. J., Associates. (1980). Toward reform of
program evaluation. San Francisco
Jossey-Bass. Cronbach, L., Suppes, (1969).
Research for tomorrow's schools Disciplined
inquiry for education. New York Macmillan,
1969. Doughty, C., Long, M. H. (2003). Optimal
psycholinguistic environments for distance
foreign language learning. Language Learning
Technology, 7, 50-80. Ellis, R. (2003).
Task-based language learning and teaching. New
York Oxford University Press. McDonough, K.,
Chaikitmongkol, W. (2007). Teachers and
learners reactions to a task-based EFL course in
Thailand. TESOL Quarterly, 41(1),
107-132. Norris, J. M. (2006). The why (and how)
of student learning outcomes assessment in
college FL education. Modern Language Journal,
90(4), 590-597. Norris, J. M. (2009). Task-based
teaching and testing. In M. Long and C. Doughty
(Eds.), Handbook of language teaching (pp.
578-594). Cambridge Blackwell. Patton, M. Q.
(1997). Utilization-focused evaluation (3rd ed.).
Thousand Oaks, CA Sage. Prabhu, N. S. (1987).
Second language pedagogy. Oxford University
Press, Oxford. Towell, R., Tomlinson, P.
(1999). Language curriculum development research
at university level. Language Teaching Research,
3(1), 1-32. Van den Branden, K. (2006). Training
teachers Task-based as well? In K. Van den
Branden (ed.), Task-based language teaching in
practice (pp. 217-273). Cambridge Cambridge
University Press.
http//www2.hawaii.edu/jnorris