Two Categories of Test Accommodations for English Language Learners

1

Technology-Based Assessment for High-Performance
Learning
Eva L. Baker
UCLA Graduate School of Education Information
StudiesCenter for the Study of
EvaluationNational Center for Research on
Evaluation, Standards, and Student Testing
International Congress for School Effectiveness
and Improvement January 2003 Sydney, Australia
2
Premises and Goals

• Assessment is central to the effectiveness of
  classroom and distance learning and
  accountabilitysome basics
• What are advances in assessment? How does
  technology fit?
• How can technology weave instructional and
  external testing into a coherent system?
• Can assessment be cost-sensitive and valid?
• Many examples throughout

3
Order of Topics

• Basics about assessment
• CRESST research-based models
• Attributes, benefits
• Examples
• Template 1 Paper and pencil
• Template 2 Computer
• Template 3 Authoring
• Technology benefits and requirements

4
About Assessment

• Role of external testsaccountability,
  evaluation, and system monitoring
• Instructional uses of tests in classrooms and
  schools diagnosis, modeling, formative
  assessment as a core teaching strategy
• Variations in curriculum require strands
  supporting coherence among goals (standards),
  content, cognitive demands (what thinking skills
  are required?)
• Coherence perceived from student, teacher,
  administrator, and expert views

5
External Instructional Exams
Assessments

• Motivated performance
• Time-sensitive
• Standardized
• Shallow sampling
• Stand-alone

• Embedded in learning
• Adapted to learners
• Extended time
• Opportunity to revise
• Contextualized results

6
External and Instructional Assessments Must
Be More Coherent

• Horizontally and vertically
• Conceptual and psychological linkage
• For accountability systems, classroom measures
  may be good supplements

7
CRESST Assessment Models

• Do not start with content
• Focus on aspects of learning and assessment that
  transfer from content to content area
• Multi purpose both formative learning and
  outcomes
• Emphasize student-constructed answers
• Expert performance defines scoring
• Research substantiates these models across
  different subjects and people

8
Families of Cognitive DemandsModel-Based
Assessment
Content Understanding
Learning
Problem Solving
Teamwork and Collaboration
Metacognition
Communication
9
CRESST Assessment Models

• Research-based, and sites are exclusively
  classrooms, schools, and systems
• Focus on cognition and learning
• Combine content-independent and content-dependent
  knowledge and strategies
• Reusable and cost-sensitive

10
Examples in Knowledge Understanding and Problem
Solving

• Paper-pencil templates
• Technology-based administration, scoring, and
  reporting
• Technology-supported authoring templates and
  menus
• For teachers, curriculum experts, test makers

11
Assessment of Understanding

• Deep understanding of primary source materials or
  key processes
• Standard reading as part of task
• Standard directions
• Standard scoring rubrics based on experts
  performance

12
Hawaiian History Assessment Task Bayonet
Constitution
Imagine you are in a class that has been
studying Hawaiian history. One of your friends,
who is a new student in the class, has missed all
the classes. Recently, your class began studying
the Bayonet Constitution. Your friend is very
interested in this topic and asks you to explain
everything that you have learned about it.
Write an essay explaining the most important
ideas you want your friend to understand.
Include what you have already learned in class
about Hawaiian history, and what you have learned
from the texts you have just read. While you
write, think about what Thurston and Liliuokalani
said about the Bayonet Constitution, and what is
shown in the other materials.
Your essay should be based on two major sources
1. The general concepts and specific facts you
know about Hawaiian history, and especially
what you know about the period of the Bayonet
Constitution. 2. What you have learned from the
readings yesterday.
Be sure to show the relationships among your
ideas and facts.
13
Excerpts from Hawaiian HistoryPrimary Source
Documents
LILIUOKALANI
For many years our sovereigns had welcomed
the advice of American residents who had
established industries on the Islands. As they
became wealthy, their greed and their love of
power increased. Although settled among us, and
drawing their wealth from resources, they were
alien to us in their customs and ideas, and
desired above all things to secure their own
personal benefit. Kalakaua valued the
commercial and industrial prosperity of his
kingdom highly. He sought honestly to secure it
for every class of people, alien or native.
Kalakauas highest desire was to be a true
sovereign, the chief servant of a happy,
prosperous, and progressive people. And
now, without any provocation on the part of the
king, having matured their plans in secret, the
men of foreign birth rose one day en masse,
called a public meeting, and forced the king to
sign a constitution of their own preparation, a
document which deprived him of all power and
practically took away the franchise from the
Hawaiian race.
14
History ExplanationScoring Rubric
1. General impression of content quality 2.
Principles or concepts (DD) 3. Prior knowledge
(DD) 4. Use of available resources (DD) 5.
Misconceptions (DD) 6. Argument (DD?) 7.
English mechanics (DI)
15
Mathematics ExplanationTask (4th or 5th Grade)

Imagine a person from a television station has
asked you to give a demonstration on TV. You will
be on a show to help other students learn about
maths. You are asked to explain everything
10-year-old students should know about
fractions. Below are some questions you should
try to answer. These are questions that students
in the TV audience will ask you. For each
question you should draw as many pictures as you
can to show what you mean. Then write down what
you would say about your pictures on TV. Use as
many words and pictures as you need. What is a
fraction? Why are there two numbers in a
fraction? How many fractions are there between 0
and 1? How many fractions are equal to 1/2? What
other important ideas should students know about
fractions? Show how you would explain these
ideas. Use as many pictures and words as you need.
16
Scoring for Maths Task

• Principles
• Prior knowledge
• Resources
• Misconceptions
• Argument/explanation

17
Technology for What?

• Most available technology-based assessment
  promotes efficiency rather than expands the
  boundaries for measurement of learning
• Tendency to limit attention to data processing
  requirements
• We should use technology to extend our
  understanding of student accomplishment and
  program and school quality

18
Template 2 Knowledge Representation to Assess
Content Knowledge and Problem Solving

• Same tasks for content knowledge
• Responses not essay but representation of
  relationships, hierarchies
• Scored by expert maps
• Efficient but expands breadth and depth of
  knowledge measurement

19
History U.S. Depression (CK)
20
Genetics High Performance (CK)
21
Bicycle PumpHigh Performance (PS)
22
Integrating Knowledge Map with Web Search Strategy
23
MAIN Story Integrating Formative and External
Assessments

• Authoring systems (computer-supported guidance)
  to help teachers create and share various sorts
  of assessments intended to measure goals
  included, not covered by, or that need deeper
  attention than given in external measures

24
Strategy to Link External and Formative
Assessments

• Cognitive demands (domain independent) and
  content knowledge and strategies (domain
  dependent)
• Authoring system allows teachers to assess goals
  not externally measured, or to connect their
  assessments to external measures but in a more
  contextualized setting

25
How It Works

• Teacher will get guidance, using research-based
  templates
• What is the purpose of the test? What is to be
  measured? What content? What conditions? What
  intellectual skills?
• Scoring rubrics (based on expert performance)
  will be provided but can be edited
• Graphical representation, simulations, and essays
  are current options

26
Assessment Authoring Benefits

• Common floor on assessments created by teachers
• Systems start with easy, fixed formats related to
  learning demands, and as teacher sophistication
  develops, move to more choices (mix and match)
• Will allow summaries of student work bubbling up
  from teachers formative assessments to validate
  the external scores
• Supports collaboration across different teachers

27
Authoring Systems Issues

• Scored work made public (within the school, with
  privacy provisions, and among schools)
• Success depends upon teacher subject matter
  knowledge, access to needed information, and
  sharing
• Success may depend on the realistic link to
  external examinations
• Generation of paper- or computer-based tasks

28
Minimum Requirements

• Infrastructure
• Capacitysubject matter
• Orientation to learning and to results
• Congruence with external mandates
• Availability of smart tools
• Lead to a culture shift

29
Distance Learning Applications

• High-quality performance demanded
• No bottleneck in scoring
• Basis of comparing courses
• Generates online assessments with instant
  scoring, feedback to student and instructor
• Aggregates student work

30
Window on the Development of Problem-Solving
Template 2Author Screen

• Assessment Purpose(s)
• Diagnostic, readiness monitoring, certification
• Scenario
• Context, constraints, situation
• Problem Characteristics
• Fix, change usual sequence, improvise step(s),
  combination
• Problem Identification Menu
• Stated, embedded, multiply masked, barriers,
  inconsistent data from multiple sources, time
  bound, partial identification, prior knowledge
  requirements

31
Author Screen Template 2 (Contd)

• Macro Planning Menu
• Explicit courses of action, problem subdivision,
  backup strategies, help seeking, mix of
  domain-independent and domain-dependent cognitive
  strategies
• Trial and Feedback Menu
• Data capture of process, process feedback, help,
  iteration
• Solution(s)
• Convergent (right answer), multiple acceptable,
  partially acceptable, divergentwith scoring
  criteria, sequential, mixed

32
Sample Examinee Screen Components

• Scenario
• Problem
• Information acquisition
• Macro strategy
• Micro strategies (domain specific)
• Solution trials
• Report of performance

33
Sample Examiner Information

• Time spent on problem
• Trials to criterion
• Help access
• Solution paths
• Generalizability of solution
• Acceptability of solution
• Likely explanation for errors (e.g., lack of
  prior knowledge)
• Metric (standards for performance)

34
Summary

• Technology-based assessment needs to extend what
  we can do
• Authoring systems can help teachers design
  better, more sensitive tests and projects
• Technology can help us share findings
• Technology-based assessment requires the same
  evidence of technical quality
• Demand evidence, not business claims, before you
  buy