Model-Based Assessment: Why, What, How, How Good, and What Next?

1
Model-Based Assessment Why, What, How, How Good,
and What Next?

• Eva L. Baker

UCLA Graduate School of Education Information
StudiesCenter for the Study of
EvaluationNational Center for Research on
Evaluation, Standards, and Student Testing
National Research Council, Board on Testing and
Assessment Bridging the Gap Between Classroom and
Large-Scale Assessment Workshop January 23-24,
2003 Washington, DC
2
Why? Assessment Knowledge Usable and Useful

• Usable Knowledge
• In a form that can be understood
• In a form that can be applied
• Timed appropriately
• May cause rethinking of the problem
• Useful Knowledge
• Rethinking indicates a new solution path
• Adapted to situation
• Sufficient to guide solution
• Improved outcomes occur as a result

3
Why Are Some Schools Successful in Using
Assessment Knowledge?

• Focus on learning (students and adults)
• Constant use of appropriate information (formal
  and informal)
• Focus on feedback and change
• Public display and exchange
• Community pride in outcomes of students and place

4
Goals for CRESST Model-Based Assessment (MBA)

• Assessment components share a common framework.
  MBA starts with thinking skills and applies them
  to content domains to support
• Coherent, sustained learning
• Spiral teaching-common language
• Transfer (application to new situations)
• Multipurpose
• Learning organization

5
CRESST Model-Based Assessments (MBA) Features

• Research based
• Focus on cognition and learning
• Abstracted in models based on key learning
  elementsprinciples guiding test design and
  instruction
• Operationalized in templates
• Reusable and cost-sensitive design/training/scorin
  g

6
Model-Based AssessmentCognitive Families
Content Understanding
Learning
Problem Solving
Teamwork and Collaboration
Metacognition
Communication
7
Model-Based Assessment Design

• Models to templates (to specification) to tests
• Template contains domain-independent (transfer)
  and domain-specific (strategy and knowledge)
  components
• Templates that allow common domain-specific
  design approaches to be used, e.g., primary
  sources in history
• Scoring requirements

8
Expert ModelDeep Understanding of Content
(Domain Independent)

• Principles or themes (big ideas)
• Use of prior knowledge
• Explicit relationships
• Avoiding misconceptions
• Expert performance-based scoring

9
Specifications forLarge-Scale Use

• Standards reference
• Place in sequence
• Content domain (whats in and out)
• Proportion of effort
• Format options
• Interpretation rules
• Time

10
Template

• Task(s)
• Format(s)
• Prompt(s) and requirements
• Scoring
• Directions
• Sample

11
Three Templates for the Model of Deep Content
Understanding

• Explanation
• Explanation with explicit knowledge
• Graphical representation of relationships

12
Deep Content Understanding

• Primary source materials in each domain
• Student required to integrate prior knowledge and
  principles to succeed
• Scored by using expert model in subject matter

13
Content UnderstandingTemplate 1 Explanation

• An array of primary source materials
• A prompt that asks for an explanation in context
• Constructed (written) answer
• Evaluated by means of a scoring rubric that
  operationalizes learning model

14
Hawaiian History WritingAssignment
BayonetConstitution
Imagine you are in a class that has been
studying Hawaiian history. One of your friends,
who is a new student in the class, has missed all
the classes. Recently, your class began studying
the Bayonet Constitution. Your friend is very
interested in this topic and asks you to explain
everything that you have learned about it.
Write an essay explaining the most important
ideas you want your friend to understand.
Include what you have already learned in class
about Hawaiian history, and what you have learned
from the texts you have just read. While you
write, think about what Thurston and Liliuokalani
said about the Bayonet Constitution, and what is
shown in the other materials.
Your essay should be based on two major sources
1. The general concepts and specific facts you
know about Hawaiian history, and especially
what you know about the period of the Bayonet
Constitution. 2. What you have learned from the
readings yesterday.
Be sure to show the relationships among your
ideas and facts.
15
EXCERPTS from HAWAIIAN HISTORYPRIMARY SOURCE
DOCUMENTS
LILIUOKALANI
For many years our sovereigns had welcomed
the advice of American residents who had
established industries on the Islands. As they
became wealthy, their greed and their love of
power increased. Although settled among us, and
drawing their wealth from resources, they were
alien to us in their customs and ideas, and
desired above all things to secure their own
personal benefit. Kalakaua valued the
commercial and industrial prosperity of his
kingdom highly. He sought honestly to secure it
for every class of people, alien or native.
Kalakauas highest desire was to be a true
sovereign, the chief servant of a happy,
prosperous, and progressive people. And
now, without any provocation on the part of the
king, having matured their plans in secret, the
men of foreign birth rose one day en masse,
called a public meeting, and forced the king to
sign a constitution of their own preparation, a
document which deprived him of all power and
practically took away the franchise from the
Hawaiian race.
16
ExplanationScoring Rubric

• General impression of content quality
• Principles or concepts
• Prior knowledge
• Examples
• Misconceptions
• Argumentation

17
Template 2Prior Knowledge and Explanation

• Explicit measurement of knowledge domain before
  explanation
• Uses short answer or selected response
• Helps interprets explanation performance

18
Template 3Knowledge Representation

• Same prompts
• Key aspects of ideas, supporting facts and views
  and their relationships
• Relationship is explicit
• Organizational options
• Core and peripheral
• Hierarchical
• Cause-and-effect
• Chronological
• Expert scoring

19
History
20
Genetics
21
Bicycle Pump
22
Brief History of MBA in LAUSD

• Content understanding and problem-solving models
• Explanation templates
• 4 subjects, 3 grade levels, 2 languages
• Purposes (1) to clarify expectations (2) to
  provided instructionally embedded assessment (3)
  to get a measure of school performance
• CRESST-managed teacher involvement

23
LAUSD Process

• Teacher design teams
• LAUSD standards first
• Adapted to success standards
• Training cadre of scorers
• Training trainers
• Supervising scoring

24
LAUSD Process (contd)

• Shift in four-topic focus (capacity based) to two
  and then to one, now back to two
• Continual assaults
• Curriculum mandates
• Accountability pressure (API)
• Long-term embedded approach resurfacing

25
Present LA Situation

• Administered in 2, 3, 4, 5, 6, 7, 8, 9
• Purpose added regarding promotion
• Teacher scored with an audit reported to school
• Local sub-districts managing activity
• Ongoing validity studies
• District review of alternative assessments

26
(No Transcript)
27
CRESST Validation Studies

• Score reliability
• Task and rater generalizability
• Stability of student performance over time
• Relationships among measures
• Instructional sensitivity
• Opportunity to Learn (OTL)
• Effect of school composition on performance
• Cut-score modeling

28
LAUSD Grade 7 Students Achievement Levels
Comparison of 2002 California Standards Test and
Performance Assignment Scores
Evidence of Predictive Validity
29
LA Scale-Up

• Cost and time driven
• Maintained by board and union support
• Transfer of responsibility
• Reduction in technical quality
• Reduction in range of measures
• Positive evaluation from independent group
  focusing on changing teaching practices

30
Continuing RD Areas

• New contexts
• Trade-offs (limited number of templates vs. wide
  range of formats)
• Performance over time
• Scalability in the long run
• Authoring systems to support teacher-developed
  assessments linked to large- scale assessment

31
Summary of Assessment Knowledge Requirements

• Knowing why
• Knowing what to assess content plus cognitive
  demands (problem solving, communication, learning
  to learn, teamwork, content knowledge)
• Knowing how transfer (application to other
  topics and situations)
• Reflecting applying MBA to teaching

32
Model-Based AssessmentCognitive Families
Content Understanding
Learning
Problem Solving
Teamwork and Collaboration
Metacognition
Communication
33
Context for Success of Knowledge-Based Reform

• Local ownership of knowledge
• Infrastructure and stability
• Capacity to investigate
• Learning
• Congruence or peace with external mandates

34
Usable Knowledge and Support May Get to Useful
Knowledge

• For assessment knowledge to be useful, it depends
  upon the context, capacity, and communication of
  the teaching system
• For assessment knowledge to be useful to
  students, it must go to the heart of why, what,
  and how they learn