Assessment Design Issues

1

2
CATS Assessment Design

• Providing Parameters for a Request for Proposals
  (RFP) for Commonwealth Accountability Testing
  System
• Kentucky Board of Education
• April 6, 2005

3
Setting an Assessment and Accountability Context
At a Glance

• Requirements KRS 158.6453, Section 1
• The Kentucky Board of Education shall be
• responsible for creating and implementing a
• statewide assessment program to be known as
• the Commonwealth Accountability Testing
• System to ensure school accountability
• for student achievement of the goals set forth
• in KRS 158.645 and 158.6451.

4
Setting an Assessment and Accountability Context
At a Glance

• Requirements KRS 158.6453, Section 3(a)
• The assessments shall be designed to measure
  grade appropriate core academic content, basic
  skills, and higher-order thinking skills and
  their application. The assessment shall measure
  the core content for assessment.

5
Current State Components Implemented in CATS

• Kentucky Core Content Test
• grades 4, 5, 7, 8, 10, 11 and 12
• 7 content areas tested reading, mathematics,
  science, social studies, writing, practical
  living/vocational studies, arts and humanities,
  and writing
• complex thinking skills Multiple-choice (33)
  and open response (67)
• current 100 target of coverage of Core Content
  per year
• Norm-Referenced Test
• commercial shelf
• grades end of primary, 6 and 9
• 5 of accountability index
• Writing Portfolio
• Grades 4, 7 and 12

6
Additional Federal Requirements

• NCLB requirements
• annual testing Reading and Mathematics grades 3-8
• augmented NRT beginning 2006

7
Values and Priorities

• Respect the developmental needs of children.
• Maintain a valid and reliable program that is
  credible with educators, public, stakeholders
• Have tests that maintain strong support for
  instruction.
• content standards alignment
• basic skills assessment
• higher-order thinking skills through constructed
  response items
• more released items for instructional and student
  accountability purposes
• Strive to merge curriculum, instruction and
  assessment into a seamless system.

8
Values and Priorities

• Meet federal testing requirements of reading and
  mathematics annually and science per grade span
  per federal requirements
• Develop a longitudinal scale for reading and
  mathematics
• Maintain involvement of Kentucky teachers
• Enhance score reportingmore student information
  and less time

9
Values and Priorities

• Seek equitable distribution of assessment across
  grade levels
• Stay within a reasonable total testing time
• Maintain wise stewardship of funds
• Focus on relevant simplicity while adhering to
  values and priorities

10
Emerging National Assessment Issues Since 1998

• Increase in statewide testing programs focusing
  on state standards
• No Child Left Behind (NCLB) enacted
• annual testing of reading and mathematics grades
  3--8
• reporting required by beginning of school year
• consequences for schools and districts
• Increase in demand for customized testing
• Availability of new assessment technologies

11
Emerging State Assessment Issues Since 1998

• Current System
• A challenge of the current system is that it is
  designed for school accountability purposes only.
  
• Assessment and accountability system with added
  NCLB requirements requires more complex design.
• Testing time for students and schools will
  increase with NCLB requirements in 2006.
• 
  
• Results need to be received quicker to impact
  instruction and provide for student
  accountability and motivation.
• As changes are proposed, particular grades can be
  overburdened with testing requirements without
  careful placement.

12
Emerging State Assessment Issues Since 1998

• P-16 Issues
• Early assessments (grades 8 or 10) to predict
  college success
• Alignment with the American Diploma Project

13
Building the Assessment RFPStep 1 Reconfirming
the Primary Purpose of the CATS Assessment
System
School Accountability
Individual Student Results
Does the KBE wish to expand the purpose of CATS
beyond a school accountability to include
additional student-based measures?
The balance between school accountability and
student results directs test design options.

• A single assessment design cannot meet all goals
  equally.
• A single assessment can report results for
  schools and individual students, but for both
  types of results to be valid and reliable require
  technical guidelines be met.

14
Building the Assessment RFPStep 2 Determining
System ComponentsCore Content and
Writing Assessments

• Refined Core Content and Test Blueprint
• A test blueprint defines the relative emphasis of
  specific categories of content on an assessment.

Would the KBE approve a different model of core
content coverage moving from a 100 per year to a
model that would allow more flexibility?
(a.100-85 or b. one or two years)
15
Building the Assessment RFPStep 2 Determining
System ComponentsCore Content and
Writing Assessments

• Revised Writing Assessments
• On-demand and Portfolio
• Analytic scoring
• Multiple choice based
  language/mechanics
  measure

Does the KBE wish to consider a change to the
number of on-demand writing prompts or how we
assess on-demand writing?
16
Building the Assessment RFPStep 2 Determining
System ComponentsKCCT
Possible KCCT test design to include Common and
Matrix Items
Form 1
Common Items provide a core for low stakes
student accountability
Matrix Items provide complete coverage of CC for
school accountability
Text Booklet

• Common
• Every child takes the same core of common items.
• Common items
• allow for adequate comparability across student
  scores.
• are released (with support materials) after
  administration for student information and
  accountability and to impact instruction and
  curriculum.
• strengthen reporting to students, parents and
  teachers.

17
Building the Assessment RFP Step 2 Determining
System ComponentsKCCT

• Matrix
• Each form of the assessment contains the same
  common items but different matrix items,
  therefore only a percentage of students take
  each matrix item.
• Matrix items
• remain secure and are reused for several years
  to allow a means of equating from year to year.
• provide a good picture of how the school is
  addressing core content.
• allow for item development through pretesting.

Does the KBE prefer that the KCCT test design
include a core of common items to provide
additional student level results and matrix items
for coverage of core content, equating and
pretesting?
18
Building the Assessment RFP Step 2 Determining
System ComponentsKCCT

• Format of items for the KCCT
• Open-response items
• Multiple-choice items
• On-demand writing prompt
• Writing Portfolio
• Alternate Portfolio for students with severe and
  profound disabilities

Does the KBE wish to continue emphasizing higher
order skills by assigning greater weight to open
response items?
19
Building the Assessment RFPStep 2 Determining
System ComponentsNRT

• A norm-referenced test

• Allows a national comparison
• May be bid in the RFP
• as a stand alone assessment or
• as items to be embedded into other assessments
  such as the Kentucky Core Core Test

Questions regarding NRT will appear later.
20
Building the Assessment RFPStep 2 Determining
System Components Longitudinal Measure

• A technically sound longitudinal measure for
  Reading and Mathematics
• Can be accomplished with a KCCT, NRT or
  combination.
• Can use an NRT embedded model that places NRT
  items in the KCCT assessment. Items could serve
  more than one purposeNRT measure plus contribute
  to assessment of Core Content.
• Will be based on the assessment of all
  accountable students in Reading and Mathematics
  grades 38.

Does the KBE wish to pursue an embedded NRT for
a longitudinal measure in Reading and Mathematics?
21
Building the Assessment RFPStep 2 Determining
System ComponentsFuture Considerations

• Possible future design for assessing Practical
  Living, Vocational Studies and Arts and
  Humanities that continues to value the content
  and includes the measures in accountability
  calculations

Does the KBE wish staff to initiate pilot studies
to develop and/or identify assessment approaches
in Arts and Humanities and Practical
Living/Vocational Studies that will address what
students do as well as what they know in these
areas?
22
Building the Assessment RFPStep 2 Determining
System ComponentsFuture Considerations

• An assessment of student readiness for
  postsecondary education at end of middle school
  and grade 10 at the high school

Does the KBE wish staff to include in the RFP a
predictive measure of college success?
23
Building the Assessment RFPStep 3 Placing the
Components

• Requires that content area assessments are
  assigned to specific grades before the complete
  balance is reviewed. (For example, changes in
  writing impact grades 5 and 8.)
• Ensures that no single grade(s) is overburdened
  with requirements.

24
Building the Assessment RFPChallenges

• Assessing 7 core content area
• Assessing higher order thinking through multiple
  forms of expression
• Accommodating multiple purposes
• Meeting state statutory requirements
• Meeting NCLB requirements
• Securing quicker turnaround of results
• Maintaining reasonable per student testing time
• Operating within budget constraints

25
Questions for KBE

• Does the KBE wish to expand the purpose of CATS
  beyond a school accountability to include
  additional student-based measures?
• Would the KBE approve a different model of core
  content coverage moving from a 100 per year to a
  model that would allow more flexibility?
  (a.100-85 or b. one or two years)
• Does the KBE wish to consider a change to the
  number of on-demand writing prompts or how we
  assess on-demand writing?
• Does the KBE prefer that the KCCT test design
  include a core of common items to provide
  additional student level results and matrix items
  for coverage of core content, equating and
  pretesting?

26
Questions for KBE

• Does the KBE wish to continue emphasizing higher
  order skills by assigning greater weight to open
  response items?
• Does the KBE wish the state to pursue an embedded
  NRT for a longitudinal measure in Reading and
  Mathematics?
• Does the KBE wish staff to initiate pilot studies
  to develop and/or identify assessment approaches
  in Arts and Humanities and Practical
  Living/Vocational Studies that will address what
  students do as well as what they know in these
  areas?
• Does the KBE wish staff to include in the RFP a
  predictive measure of college success?

27
(No Transcript)
28
(No Transcript)
29
(No Transcript)
30
(No Transcript)
31
(No Transcript)
32
(No Transcript)
33
(No Transcript)
34
(No Transcript)
35
Timeline

• Current contract expires June 30, 2006 but
  includes fall 2006 reporting
• New contract will be bid for 2 years with
  2 year renewable extension
• Tentative RFP Calendar
• July/August 2005 RFP approved by KBE
• August/September 2005 Target RFP release