California Assessment Literacy Initiative for Improved Student Learning

1
California Assessment Literacy Initiative for
Improved Student Learning
Margaret Heritage
California Educational Research
AssociationAnnual Conference November 19,
2009 San Francisco, CA
2
Session Objectives

• 1- Provide background to the California
  Assessment Literacy Initiative (CALI)
• 2- Present goals and framework for CALI
• 3- Present content ideas and delivery system
  design ideas
• 4- Solicit feedback on the above

3
CALI Steering Committee

• Gina Koency ( LACOE)
• Paula Carroll (SJCOE)
• Kathryn Edwards (LACOE)
• Karen Greer (LACOE)
• Margaret Heritage (CRESST/UCLA)
• Mary Tribbey (BCOE)

4
The Question

• Cheshire , Alice began rather timidly, would
  you tell me please, which way I ought to go from
  here?
• That all depends a good deal on where you want
  to get to, said the Cat
• Lewis Carroll

5
The Answer

• Cheshire , Alice began rather timidly, would
  you tell me please, which way I ought to go from
  here?
• That all depends on what your data tell you,
  said the Cat
• Lewis Carroll

6
When you have a robust data system and
professional development system working hand in
hand, there is total alignment for the
kids. Chris Steinhauser, Long Beach Unified
School District Superintendent
7
Background
8
Increased Focus on Data Use

• The collection, analysis and use of educational
  data are central to the improvement of student
  outcomes envisioned by No Child Left Behind
  (NCLB)
• (U.S. Department of Education, 2009, p. vii).

9
Data Use

• Data use leads to improvements in student
  learning
• (Snipes, Doolittle Herlihy, 2002 Williams,
  Kirst, Haertel et al., 2005 Armstrong Anthes
  (2001) Cawelti Protheroe, 2001 LaRocque,
  2007 Symonds, 2004 Tognieri Anderson, 2003).

10
Increased Focus on Data Use

• Increased attention paid to data use
• (Heritage Yeagley, 2005 Ikemoto Marsh, 2007
  Mandinach Honey, 2008 Wayman Stringfield,
  2006).
• BUT
• Expansion of data gathering without a
  corresponding jump in data use
• (Data Quality Campaign, 2006 Hamilton et al.,
  2009 Rothman, 2008).

11
Lack of Data/Assessment Literacy

• Data literacy presumes an accumulating facility
  with the interpretation of data, not to mention a
  familiarity with data sources and creativity in
  assembling relevant data quickly and efficiently
• (Knapp, Swinnerton, Copland, Monpas-Huber,
  2006, p. 13).
• Administrators and teachers lack skills to use
  data effectively
• (Datnow, Park, Wohlstetter, 2007 Heritage,
  Lee, Chen, LaTorre, 2005 Stiggins, 2002).

12
What is Assessment Literacy? (HO)

• Assessment literate educators
• Design/evaluate assessments
• Identify learning targets
• Provide differentiated instruction assessments
• Monitor student progress
• Organize, analyze, interpret, and use data
• Evaluate reliability/validity of assessments
• Engage students in assessment process provide
  feedback

13
Survey

• Administer survey of assessment literacy skills
  and assessment data use

14
Project Goals (HO)

• Build county office capacity
• Create and disseminate Web-based assessment
  literacy professional development modules
• Conduct impact studies
• Provide guidance to teachers focused on using
  existing state resources
• Provide teachers with clear guidance on how to
  effectively utilize assessment results

15
Discussion

• Do the attributes of assessment literacy seem
  right to you?
• What would you change?
• Do the project goals seem right to you?
• What would you change?

16
Collaborative Model
17
CALI Professional Development Program Framework
18
Data Use Processes and Skills

• Research calls for
• A systematic process facilitating data use
• (Armstrong Anthes, 2001 Boudett, City,
  Murnane, 2005 Heritage Chen, 2005)
• Strong data-literate leadership at all levels of
  the system
• (Mason, 2002 Herman Gribbons, 2001)
• A data-driven culture
• (Datnow, Park, Wohlstetter, 2007 Lachat
  Smith, 2005)

19
(No Transcript)
20
Assessment System

• Coherent built on a well-structured conceptual
  base
• Comprehensive provides a variety of evidence
• Continuous provides indications of student
  growth over time
• (NRC, 2001, p.259)

21

• Different levels of granularity for different
  purposes

22
Annual State Tests Interpretation What students
have learned/ have they met the
standard? Differences among groups Strengths/wea
knesses in Individuals and groups
learning Strengths/weaknesses in curriculum/instr
uction/programs Are our improvement strategies
working?
Annual State Tests Action Adjustments to
curriculum, instruction, programs Monitoring/acco
untability Reporting Inform professional
development needs

23
Quarterly/Monthly Assessments Interpretation Pro
gress monitoring for individuals have students
reached the benchmark? Differences among
groups Strengths/weaknesses in individuals and
groups learning Strengths/weaknesses in
curriculum/instruction
Quarterly/Monthly Assessments Action Adjustments
to curriculum, instruction Progress monitoring
/ accountability Reporting
24
Minute-by-minute/Daily/Weekly Action What to do
to move learning forward Adjustments to ongoing
instruction/learning Feedback
Minute-by-minute/Daily/Weekly Interpretation
Identify the gap between current status and the
desired learning goals Identify individual
misconceptions/difficulties Identify missing
building blocks
25
Assessments in the System
26
Assessment Knowledge

• The PD modules will address specific knowledge
  about the assessments
• a) range of assessment task (constructed
  response, selected response, performance,
  instructional task)
• b) purpose of each of the assessments and their
  potential use
• c) assessment quality validity (including
  alignment), reliability, freedom from bias, and
  usability
• d) the need for multiple measures to inform
  decisions

27
Assessment Use
28
Assessment Skills

• The PD modules will address specific skills for
  assessment use
• a) establishing learning goals accessible to all
  students
• b) interpretation skills, including statistical
  knowledge
• c) matching learning opportunities to learners
  needs
• d) feedback to learners

29
Example
30
(No Transcript)
31
(No Transcript)
32
Guidelines for Writing Data Statements and
Summaries

• Each statement should
• Communicate a single idea about student
  achievement
• Present the facts objectively rather than state
  evaluative or explanatory comments
• Be short, clear sentences or phrases in everyday
  language that is easy to understand
• Be an independent statement, that is, its meaning
  should not be dependent on other statements
• Represent the data accurately by including
  relevant numerical data when needed for evidence
• Review all of the data statements and identify
  the most important ideas that convey the story
  about achievement
• (Adapted from Van Houten, L., Miyasaka, J.,
  Agullard, K., Zimmerman, J. (2006). Developing
  an Effective School Plan An Activity-Based Guide
  to Understanding Your School and Improving
  Student Outcomes. Oakland, CA WestED)

33
Writing Data Statements and Summaries Activity

• Write a paragraph of statements summarizing the
  major and important findings. The statements can
  be in a slightly more narrative style, but still
  tightly based on data. Important numerical
  results should be included to support the points
  made. Avoid including personal judgments and
  opinions. If you find you are describing why the
  results occurred, or using the word because in
  your summary, you have moved to interpretation
  and are no longer summarizing!
• (Adapted from Van Houten, L., Miyasaka, J.,
  Agullard, K., Zimmerman, J. (2006). Developing
  an Effective School Plan An Activity-Based Guide
  to Understanding Your School and Improving
  Student Outcomes. Oakland, CA WestED)

34
Student Involvement in Assessment

• Self-assessment (metacognition and
  self-regulation)
• Peer-assessment
• Use of feedback

35
Discussion

• What topics should the professional development
  program include to support educators effective
  use of data?

36
CALI Professional Development Program
37
Online Professional Development

• Meta-analyses comparing distance education
  classes vs. traditional classes highlight the
  need for deliberate course design for online
  delivery to be effective, specifically
  incorporation of interactivity
• (Bernard et al., 2004 Bernard et al., 2009).

38
Modules Design Elements

• Completed in 3-5 hours
• Designed to support collaboration
• Include assignments tied to practice
• Include pretest, checks for understanding, and
  posttest
• Include videos of related practice activities
• Include videos of experts speaking
• Links to related resources for additional
  information
• Inclusion of interactive tools that would enable
  the user to explore the impact of certain
  decisions

39
Virtual Professional Learning Communities

• Professional Learning Communities (PLCs) are
  structures for continuous learning and use of
  knowledge in the course of conducting the work of
  teaching
• (Hord and Sommers, 2008, cited in Mundry
  Stiles, 2009, p. 9).
• Teachers in every case are learning and working
  with their peers to situate their learning in
  real practice
• (Mundry Stiles, 2009, p.9)

40
Electronic Networking and Professional Development

• Review of research finds the following benefits
  to incorporating electronic networking into
  professional development
• 1) Reducing teacher isolation supporting
  sharing
• 2) Encouraging reflection on practice
• 3) Improving teaching practice
• 4) Encouraging professional learning communities
• (Barnett, 2002)

41
Electronic Networking and Professional
Development (cont.)

• Studies of professional development for secondary
  science teachers incorporating electronic
  networking (i.e. blogging, discussion forum, and
  an online community) found evidence of learning
  through interactions with others, increased
  levels of reflective teaching, and increased
  value placed on learning by participants
• (Luehmann Tinelli, 2008 Makinster, Barab,
  Harwood, Andersen, 2006)

42
Discussion

• What else should we be thinking about with regard
  to the online delivery system?

43
mheritag_at_ucla.edu