Building a Comprehensive Assessment System

1

• Building a Comprehensive Assessment System
• Edward Roeber
• Office of Educational Assessment and
  Accountability

2
This is the time of assessment.

• This is the era of standards-based reform, based
  on student assessment information
• Federal laws require much assessment at the state
  level
• District response more assessment!
• Using assessment information is now high stakes
  - school survival now depends on it
• Yet, educators are not always prepared to develop
  assessments nor use the information provided by
  them

3
Added Assessments

• New assessments will add even more external
  pressures
• Michigan Merit Examination
• End-of-course examinations
• With added state level assessment, as well as
  district-level assessments, our assessment
  systems may be out of balance - too much external
  testing, too little classroom-based assessment

4
Whats Needed

• We need a balanced assessment system - one that
  honors the work in classroom assessment and is
  aligned to the state and district assessments
• A variety of approaches to assessment are needed
  - perhaps more assessment, but of different kinds
• All assessments needs to work in the same
  direction - a coordinated system of assessment

5
What is a Balanced Assessment System?

• Formative
• Supports on-going learning in the classroom
• Interim
• Provides periodic snapshots of student
  achievement at the district level
• Summative
• Assesses student learning at the conclusion of a
  phase of learning
• All parts work together and are equally important

6
Balanced Assessment Systems

• Are all elements of the assessment system
  coordinated? Are all three types present?
• Formative
• Interim
• Summative
• Are the content and skills measured compatible?
• Are standards, curriculum, instruction, learning,
  assessment and professional development aligned

7
Balanced Assessment Systems

• Is assessment continuous - do the assessments
  measure student progress over time (days, weeks,
  months, school years, and school careers)?
• Are a variety of formal and less formal
  assessment methods used on an on-going basis?
• Is assessment integrated - does the assessment
  system fit in the larger educational system?
• Is assessment seamlessly integrated into
  instruction?

8
Balanced Assessment Systems

• Are a variety of high quality assessments being
  used?
• Do all assessments meet professional standards
  (reliability, validity, and so forth)?
• Are the assessment purposes clearly spelled out
  and are assessments used that can achieve these
  purposes?
• Do all students participate in assessments
  appropriate for them?
• Do assessments have positive consequences - for
  students, educators, parents, and others?

9
What are Summative Assessments?

• State Level
• Michigan Educational Assessment Program
• Michigan Merit Examination
• MI-Access
• English Language Proficiency Assessment
• Secondary Credit Assessments?
• District Level

10
Uses of Summative Assessments

• District
• Prioritize schools for improvement assistance
  efforts
• Provide data for the local school board to
  understand student and school achievement
• Report academic progress of the district to
  parents and other taxpayers, including key
  community decision-makers
• Increase community support for schools and for
  school improvement efforts
• Key Question Did students learn what they should
  have?

11
Uses of Summative Assessment

• State
• Determine priority schools for the interventions
  required by Federal (and state) law
• Guide state-level assistance to high priority
  schools - those continuing not to make adequate
  yearly progress
• Fund schools - could be general state aid and/or
  special programs to impact struggling schools
• Allocate human resources - such as providing
  staff or consultants to provide supportive
  management advice and strategies to schools

12
What are Interim Assessments?

• Periodic assessments given school- or district-
  wide at fixed times during the school year
• Measure the schools or districts curricula in
  important content areas
• Provide evidence that all students have been
  taught key skills
• Serve to reinforce a common curriculum and sets
  of learning experiences

13
Interim Assessments

• District-constructed assessments, or
• Commercially-available assessments
• Replicas of the MEAP assessments
• Pacing assessments that follow the curriculum
• Assessments that do not follow MEAP nor the
  district curriculum
• Key Question Are students in each school on
  track for proficiency?

14
Interim Assessments

• Examples
• Quarterly tests based on the instructional
  sequence(s) used to measure the pacing guides
  used instructionally
• Advantages
• May determine if students are learning the
  enabling skills
• Might catch students who are experiencing
  difficulties in learning before they fall far
  behind
• Challenges
• Instructional sequences are not always universal
  - are multiple forms of the assessments needed?
• Are the assessments of high quality and/or match
  instruction?

15
Uses of Interim Assessment

• Increase instructional alignment - horizontal and
  vertical
• Inform parents of all students about the level of
  student achievement and improvement efforts under
  way
• Assure consistency of instruction across schools
  and the district
• Program evaluation - evaluate the effectiveness
  of new initiatives

16
What is Formative Assessment?

• Classroom-based assessments used on an on-going
  basis in every classroom
• Carefully-thought-out strategies to engage
  students in learning in and outside of the
  classroom
• May encourage different ways of learning - moving
  from passive to active student learning
• May encourage student self-assessment/self-monitor
  ing

17
Formative Assessment

• Teachers are engaged actively in the assessment
  process by
• Sharing instructional targets with students
• Questioning students
• Observing students
• Examining and evaluating student work
• Providing feedback to students
• Determining instructional implications of the
  assessments
• Conferencing with students, other teachers and
  parents

18
Formative Assessments

• Students are actively engaged in the assessment
  process by
• Learning what is expected of them
• Taking responsibility for their own learning
• Actively participating in the learning process
• Participating in how students document what they
  have learned and how they learned it
• Demonstrating their learning to peers, educators
  and their parents

19
Uses of Formative Assessment

• Guide student learning on a daily basis by
  providing information about what critical skills
  were and were not learned
• Provide extra learning opportunities to students
  who are struggling academically
• Provide additional learning opportunities for
  students who are doing well academically
• Report student progress to students, parents, and
  other educators
• Key Question Has each student learned?

20
Characteristics of Both Large-Scale and Classroom
Assessments

• Shared model of student learning
• Should include learning progressions over time
• Shared concept of disciplinary knowledge and
  competence
• Focus assessment on the most valued knowledge and
  skills
• Focus on understanding and reasoning, not recall
• Assess enabling skills and procedural knowledge
• Signal to teachers and students what is most
  important
• Base assessment on clearly written standards

21
Characteristics of Both Large-Scale and Classroom
Assessments

• Measure a manageable body of knowledge and a
  limited number of the most important skills
• Target general forms of cognition
• problem solving and inductive reasoning
• skills that are more domain-specific
• Move from single-answer assessments towards
  greater use of tasks
• content based
• measure rich and well-structured knowledge
• are open to multiple approaches

22
Characteristics of Both Large-Scale and Classroom
Assessments

• Designed to be valid and useful to support
  decisions (large-scale and classroom)
• Are technically sound and timely
• Are designed in accordance with the purpose for
  which results will be used

23
Characteristics of Both Large-Scale and Classroom
Assessments

• Measure the knowledge and skills they purport to
  measure
• Are designed to be reliable, valid, and fair for
  the inferences that will be drawn from results
• Report results in enough detail to reveal needed
  instructional changes and highlight deficiencies
  in the system or in instruction
• Focus on student learning in school, rather than
  what students have learned outside of school

24
Characteristics of Both Large-Scale and Classroom
Assessments

• Provide opportunities for students of different
  backgrounds to connect their knowledge to
  relevant school expectations.
• A range of measurement approaches are used to
  provide a variety and range of evidence of
  student achievement
• Provide multiple opportunities for students to
  demonstrate competence
• Look at differences in how students perform in
  different assessments as rich data about
  students, not error

25
Unique Characteristics of Large-Scale Assessment

• Provide comparative data (normative or
  standards-based)
• judge the adequacy of performance
• determine specific curricular/instructional areas
  for improvement
• Provide quality feedback to educators about
  patterns of errors that could be target for
  instruction in the future

26
Unique Characteristics of Large-Scale Assessment

• Must be cost-effective and feasible
• instructional uses must be worth the time taken
  from instruction
• Results must be reported to stakeholders

27
Unique Characteristics of Classroom Assessments

• Must be on-going and integrated seamlessly into
  instruction so teachers and students receive
  frequent feedback on their progress
• Assess some desired proficiencies that cannot be
  effectively assessed on large-scale assessments
• Provide quality ongoing feedback to teachers
  about patterns of errors that could indicate the
  need for modification of future instructional
  strategies

28
Unique Characteristics of Classroom Assessments

• Help teachers to identify and reconstruct
  students misconceptions
• Provide quality feedback to students about their
  performance and specific guidance about how to
  improve
• Help students learn how to assess their current
  levels of understanding in relation to learning
  goals learn to recognize and produce quality
  work

29
What are ways we can assess students?

• Selected-response tests
• Forced-choice - multiple-choice, true-false,
  matching
• Written response - short-answer, extended
  response
• Observation
• Structured
• Unstructured
• Samples of Student Work

30
What are ways we can assess students?

• Performance Events
• Individual interviews
• Hands-on performance assessments
• Performance Tasks
• Individual assessments
• Group performance assessments
• Projects
• Datafolios
• Portfolios

31
Forced-Choice Items

• Examples
• multiple-choice, true-false, matching
• Advantages
• Can cover much content in little time
• Inexpensive
• Easy to score - Objective
• Challenges
• Difficult to write high-quality items
• May stress memorization over understanding and
  application
• Could encourage instruction that emphasizes
  content coverage over deep understanding of
  concepts

32
Written Response Items

• Examples
• essays, short-answer, solve math problems, fill
  in the blank
• Advantages
• Tap student understanding
• Can measure application of knowledge
• Can be administered in group settings
• Challenges
• More time-consuming, expensive to score
• Scoring may not be objective
• Students may not respond

33
Structured Observation

• Examples
• A topic for group discussion question by small
  groups of students
• Teamwork exercise with observation protocol
• Science laboratory exercise
• Advantages
• Assess application of skills
• Can observe students using skills in real
  contexts
• Challenges
• Difficult to observe multiple groups (or
  individuals) at the same time
• Time consuming labor intensive

34
Unstructured Observation

• Examples
• Student classroom participation
• Quality of student responses to teachers
  questions
• Advantages
• Naturalistic observation of students
• Collect data (systematically) not usually
  collected or used
• Challenges
• Lack of opportunity for all students to be
  observed
• Anecdotal in nature - how generalizable?

35
Samples of Student Work

• Examples
• Daily classroom work
• Term papers
• Advantages
• Captures work already being done in the classroom
• Could reinforce the on-going creation of products
  by students
• Challenges
• The information from each student may be
  different
• May be difficult to standardize across multiple
  students

36
Performance Events

• Examples -
• Musical performance sing, play a musical
  instrument
• Science experiment
• Solve mathematical problem
• Advantages
• Often, these assess the most important skills
• Challenges
• Time consuming to assess all student
• Expensive to develop, use, and score

37
Performance Tasks

• Examples
• Paper on an English topic
• Compose a song
• Plan for a school or community improvement
  project
• Advantages
• Sustain and extend classroom learning
• Encourage students to work deeply in the content
  area
• Challenges
• Time consuming - for students and teachers
• Students need support and assistance

38
Projects

• Examples
• Senior project on a topic of the students
  interest
• Implementation of a plan to improve the school or
  community
• Advantages
• Students can engage actual work that interests
  them
• Challenges
• Very time consuming for both students and
  teachers, to carry out, to present, and to score

39
Datafolios

• Examples
• Collection of data from one or more sources, such
  as observation, ratings of student work, test
  scores in one folio
• Advantages
• Captures and organizes information about a
  student in one place
• Challenges
• Time consuming for students and/or teachers to
  select and organize the information
• Data itself may not tell much of a story
  narrative commentary may also be needed

40
Portfolios

• Examples
• Showcase of best arts products
• Collection of essays written in English class
• Advantages
• Show student progress
• Capture best work of the student
• Encourage students to produce work
• Challenges
• Very time consuming to develop, collect,
  organize, and score
• May be idiosyncratic to the individual students

41
Questions to Consider First

• Do we know what students need to learn?
• Do we understand what students need to learn?
• How do students best learn?
• What instructional strategies are most effective?
• How do we know that students have learned?
• Can state, district, and classroom assessments
  work together to promote quality student
  learning?

42
Questions to Consider First

• What work on assessment is needed to improve how
  assessment contributes to student learning?
• What professional learning needs to occur for
  this work to happen?
• How can the state encourage work on all three
  parts of the balanced assessment system?

43
Choosing Assessment Methods

• Each educator and school should use a variety of
  assessment methods (multiple methods)
• The methods selected locally (district, school,
  and classroom) should complement and build on the
  assessments used at the state and national level
• Coordinated assessments are most useful, since
  they can present a more complete picture of
  student achievement and other accomplishments

44
Coordinated Assessment Systems

• Use both large-scale and classroom-based
  assessments
• Measure the same skills or related skills using
  different measures of related skills
• One possible model
• State or district end-skills
• Classroom learning progressions leading to the
  accomplishment of end skills

45
Coordinated Assessment Systems

• Developed to provide comprehensive information
  on learning and achievement
• Different assessment methods may or may not yield
  consistent information
• If the results are consistent, still use multiple
  methods
• If not consistent, look for reasons - may yield
  insights into student behavior and/or the quality
  of the measures used.
• Avoid using multiple assessments as a
  multiple-choice item - selecting one source and
  ignoring the others

46
Developing the Needed System

• Examine the academic content standards valued at
  the state level
• Format - end-skills or learning progressions?
• Breadth and depth - how thorough?
• Completeness - few or many in number?
• Determine how the state will assess these
• What types of measures will be used?
• What types of measures are missing?
• What other types of data do classroom teachers
  need?

47
Developing the Needed System

• If the skills being measured are end-skills, then
  create the desired learning progressions in
  grade-level expectations
• Work should involve educators from all levels to
  assure horizontal and vertical alignment
• Determine the assessments methods to be used to
  assess the learning progressions
• Determine assessment methods needed to gauge
  student learning
• Develop the needed assessments

48
Developing the Needed System

• Create and implement instructional plans to
  provide the instruction on learning progressions
• Make sure all teachers are on the same page
• Horizontal alignment
• Vertical alignment
• Use assessments to determine student learning
• Embed assessments in on-going instruction
• Review and use assessment results to improve
  instruction
• Examine assessment results to determine
  individual student and group instructional needs

49
Michigans Assessment System

• In Michigan, the assessment system needs to be
  more fully balanced
• Summative assessments play a major role
• Interim assessments are needed
• Formative assessments are also needed
• Teachers need to be given time to actively
  discuss instructional strategies
• Assessments need to be embedded in good
  instruction
• Teachers need to be formally prepared to work on
  the assessments to be used

50
Secondary Credit Assessment Program

• State and districts need to determine whether
  students have learned enough to receive credit in
  a variety of high school credit areas
• This could be done in conventional end-of-course
  exams or through a different manner
• We propose to do both - to develop a balanced
  assessment system in Michigan
• We invite interested districts, schools, and
  educators to become partners in this effort

51
An Opportunity to be Different

• The Secondary Credit Assessments provide an
  opportunity to develop something very different
• Re-balance the assessment system
• Develop assessments that encourage high levels of
  student learning, rather than simply determining
  pass-fail
• Build a system of interim and formative
  assessments to go with the summative assessments

52
A Modest Proposal

• We propose to develop a series of formative
  (classroom), interim, and summative assessments
  for each of the key high school credit areas
• Develop these assessments using interested local
  educators across the state
• The system will present good local educator
  work made available to others in the state

53
Areas to be Developed

• English Language Arts
• English 9, 10, 11, and 12
• Mathematics
• Algebra I
• Geometry
• Algebra II
• Pre-Calculus
• Data and Statistics

54
Areas to be Assessed

• Science
• Biology
• Chemistry
• Physics
• Earth Science
• Social Studies
• American History and Geography
• World History and Geography
• Economics
• Civics

55
How Will Work be Carried Out?

• Develop ISD/RESA teams to sponsor work groups of
  local educators
• Leadership team from one or more ISDs/RESAs
• Working groups of teachers and curriculum
  specialists in the four content areas
• Professional development assistance provided by
  MDE and others
• Use other existing resources with an interest in
  improved instruction and assessment

56
How Will Work be Carried Out?

• There are existing learning communities, formal
  and informal, across the state already engaged in
  thinking about
• How students learn best
• How teachers can draw out the best from students
• How we know whether students have learned
• How educators can learn more about learning,
  teaching and assessment
• More such learning communities are needed in the
  state

57
What Work Will be Done?

• Develop model instructional lessons that address
  parts of the high school content expectations
• Build a variety of assessments within these
  instructional ideas
• Try out these lessons and assessments with
  students
• MDE Collect, select and share the ideas

58
Target

• Provide instructional and assessment ideas across
  the state no later than Fall 2008
• Build an assessment system for use locally
• Deliver the assessments on an on-going basis, so
  that students can be assessed before they take a
  class, as they are taking it, or at the
  conclusion of it

59
Summary

• Our goal is to seek to improve how students are
  taught, as well as how they are assessed
• Emphasize improved student learning, not just
  higher test scores.
• Continue to seek to improve educator skills in
  instruction, as well as assessment
• Build a balanced assessment system with
  coordination among the parts

60
Summary

• The requirement to assess the High School Content
  Expectations will be used as the basis for
  modeling a balanced assessment system that
  includes formative and interim assessments
• Engage educators across the state in developing
  these formative and interim assessments
• Begin this activity this year and support it as
  it grows and spreads
• Recruit local and ISD educators who want to be
  part of building this new system

61
For Questions and Comments

• Dr. Edward Roeber
• Michigan Department of Education
• Office of Educational Assessment Accountability
• P.O. Box 30008
• Lansing, MI 48909
• [phone] voice
• RoeberE_at_michigan.gov