Assessment Director Meeting March 12, 2009 - PowerPoint PPT Presentation


PPT – Assessment Director Meeting March 12, 2009 PowerPoint presentation | free to view - id: 8dd4b-ZDc1Z


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation

Assessment Director Meeting March 12, 2009


Coming to State on March 30. Must get to LEAs by April 30. Must spend by ... fact, we've been 'bridging' from 2006 through 2008 because of the slow evolution ... – PowerPoint PPT presentation

Number of Views:45
Avg rating:3.0/5.0
Slides: 85
Provided by: tash5


Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Assessment Director Meeting March 12, 2009

Assessment Director Meeting March 12, 2009
  • Judy W. Park
  • Associate Superintendent
  • Data, Assessment Accountability
  • Utah State Office of Education

  • Introductions

(No Transcript)
(No Transcript)
(No Transcript)
(No Transcript)
(No Transcript)
Thank You
  • . . . for hanging in there.

Federal Stimulus Package
  • American Recovery and Reinvestment Act (ARRA)

Federal Stimulus Package American Recovery and
Reinvestment Act (ARRA)
  • Governor will use some for budget short fall
    (back fill) 279 M
  • HB 2 There is appropriated from Federal Funds -
    American Recovery and Reinvestment Act (H.R. 1,
    111th Congress) to the State Board of Education
    for fiscal year 2009-10 only, 207,176,000 for
    Social Security and retirement as provided in
    Section 53A-17a-125.

Federal Stimulus Package American Recovery and
Reinvestment Act (ARRA)
  • Spring Money
  • 50 of Total available funds
  • IDEA Title I
  • Coming to State on March 30
  • Must get to LEAs by April 30
  • Must spend by September 2011
  • Use for existing statutes with some flexibility
  • More information is coming in next 2 weeks

Federal Stimulus Package American Recovery and
Reinvestment Act (ARRA)
  • Four Principles
  • Spend quickly to save and create jobs
  • Improve student achievement
  • Transparency to the school level
  • Separate accountability for how money is spent
  • 4. Invest one time ARRA funds thoughtfully to
    minimize funding cliff

Federal Stimulus Package American Recovery and
Reinvestment Act (ARRA)
  • More money in July and in the Fall
  • Competitive grants
  • Technology
  • Statewide Data
  • Teacher quality

Federal Stimulus Package American Recovery and
Reinvestment Act (ARRA)
  • Fall race to the top
  • Focus on increased student achievement
  • Fall - Invest in what works innovation
  • States already showing high achievement and/or
    achievement gains
  • Money to replicate what you are already doing

2009 Legislative Session
1S SB 81 Senator Margaret Dayton
  • Concurrent Enrollment Program Amendments
  • Bill Status Passed
  • 53A-15-101. Higher education courses in the
    public schools Cooperation between public and
    higher education
  • to ensure that students are prepared for college
    level work, an appropriate assessment is given
  • Accuplacer
  • What is the role of Assessment Directors?

HB 334 Representative Merlynn Newbold
  • Writing Assessment and Instruction
  • Bill Status Passed House, Senate 1st reading
  • a direct an online writing assessment for
    grades 6 5 and 9 8,

HB 334 Representative Merlynn Newbold
  • Writing Assessment and Instruction
  • RFP
  • Separation of formative and summative
  • Formative
  • Grades 4 12
  • Implementation varies for each LEA
  • Summative
  • Secure prompts
  • Responsibility of assessment directors
  • Administration Training?

HB 334 Representative Merlynn Newbold
  • Writing Assessment and Instruction
  • (6) (a) A school district or charter school, as
    applicable, is encouraged to administer an online
    writing assessment to students in grade 11.
  • (b) The State Board of Education may award a
    grant to a school district or charter school to
    pay for an online writing assessment and
    instruction program that may be used to assess
    the writing of students in grade 11.

1S SB 159 Senator Howard Stephenson
  • Math Education Initiative
  • Bill Status
  • Passed Senate
  • House 2nd reading
  • Grants to pay for Singapore math

1S SB 159 Senator Howard Stephenson
  • Math Education Initiative
  • Award grants to
  • (i) school districts and charter schools to help
    pay for a school district's or charter school's
    costs of adopting Singapore math and
  • (ii) institutions of higher education or
    nonprofit educational organizations for math
    teacher training programs

1S SB 159 Senator Howard Stephenson
  • Math Education Initiative
  • A school district or charter school that
    participates in the Math Education Initiative
  • shall adopt Singapore math for grades
    kindergarten through six,
  • may adopt Singapore math for grades seven and
  • may phase in the adoption of Singapore math
  • in grades K-8 over two or more years.

1S SB 159 Senator Howard Stephenson
  • Math Education Initiative
  • shall at the secondary school level, offer
  • (i) honors and regular sections of algebra 1,
    geometry, and algebra 2
  • (ii) an honors pre-calculus course that
    corresponds to college algebra and
  • (iii) calculus and statistics courses that meet
    the standards for the Advanced Placement exams

1S SB 159 Senator Howard Stephenson
  • Math Education Initiative
  • Participation is Voluntary
  • Competitive Grant process
  • Grants up to 150 per student
  • Grant monies to pay for Singapore math
    instructional materials and/or professional
  • shall administer math achievement tests at the
    beginning and the end of the school year

100 Computer Based Testing Criterion-Referenced
  • 2007 8
  • 2008 50
  • 2009 65
  • 21 Districts and 42 Charter Schools have
    committed to test 85 or more of their CRTs via
  • 2010 ?

100 Computer Based Testing Criterion-Referenced
  • 2009 - Scale and Equate from CBT only
  • Innovative Item Types
  • Hardware Options
  • Technology funding - AARA

Computer Based Testing UBSCT
  • Optional only for those schools who choose to
    be 100 CBT for UBSCT
  • As early as October 2009?
  • Pre-Equated
  • Quick/immediate return of reading and math scores
  • Quicker return of writing scores?

Digital SAMS UTREx Update
Changes to Clearinghouse Procedures for SY 2008-09
  • USOE Data Assessment and Accountability
  • (801.538.7953)

  • Increase Clearinghouse data quality
  • Ensure on time (July 15) final data submissions
  • Increase confidence in accountability reports
  • Eliminate indecisions about resubmission of data

Major Changes for 2008-09
  • Elimination of November Resubmissions.
  • Data Quality Auditors will be available to help
    improve submitted data.
  • New Clearinghouse Validation Reports will be
  • More Formal Assurance from the LEAs that good
    data has been submitted.

LEA-USOE Major Data Flows
July - Sept
Local SIS
May- July
Summary Validation Reports
May- July
AYP, CRT, U-PASS Reports Cognos, SERF
USOE Clearinghouse
Enrollment DWA Supplemental Reading UAA IOWA Dis
July -Sept
July 15
USOE Warehouse
Sept 15
July 15
USOE Web Sites
USOE Partners
No More November Resubmissions
  • Less than three LEAs were taking advantage of
    this option in any given year.
  • The November option sometimes promoted
    insufficient attention to review of the data in
    June and July.
  • LEAs need to review and correct data in June and
    July when the data are new and before the new
    school year.
  • Resubmissions were very time consuming and risky
    for all concerned.

Data Quality Auditors
  • Beginning in the spring of 2009, the USOE will be
    assigning two data auditors/analysts to work with
    LEAs in an effort to improve the quality of data
    flowing between the LEAs and the USOE.
  • These analysts will help review Clearinghouse
    reports and the LEAs state data submission
    procedures. Their main focus will be helping to
    ensure the submitted data accurately represents
    the LEAs schools, students, personnel and
  • As the roles and operating procedures of these
    individuals become more well defined additional
    details will be available.
  • The USOE welcomes suggestions in regards to this

New Clearinghouse Validation Reports
  • There will be new Clearinghouse reports this
    spring to accompany the summary reports that are
    currently produced each time a LEA updates the
    USOE Clearinghouse.
  • These new validation reports will compare last
    school years counts/totals (e.g. membership and
    demographic) with this school years
    counts/totals and indicate any that appear to
    deviate significantly.
  • There are still some design details to be
    finalized, and the USOE is open for suggestions.

Current Clearinghouse Report
Current Clearinghouse Report
New Clearinghouse Validation Reports
  • It may not be feasible to compare every data
    element/count in the current years report to
    last years report.
  • There have been suggestions that some validation
    reports, based on new data, be added. One would
    be counts of the number of students by course
    type and the number of classes in that course,
    resulting in a preliminary class size statistic.
  • While not based on the more sophisticated
    business rules used in final U-PASS reports (e.g.
    compensation for multi-grade and multi-section
    classes), this report could assist LEAs in
    reviewing and correcting data that impacts this
    important statistic. The next slide shows a
    prototype of such a report.

New Clearinghouse Validation Report s New Class
Size Estimation for Validation Report
New Clearinghouse Validation Reports
  • With the new Clearinghouse Validation reports,
    modified procedures for the Clearinghouse
    submissions may need to be defined. One
    possibility is that every LEA must submit at
    least one Clearinghouse file that can pass the
    preliminary edit/error reports and thus be
    eligible for a full summary and verification
    report cycle prior to July 8.
  • This would give every LEA at least one week prior
    to the July 15 deadline to review the summary and
    validation reports and resubmit its Clearinghouse
    file if necessary. Modifying this procedure is
    important because LEAs will no longer be able to
    resubmit a clean-up Clearinghouse file in

More Formal Data Assurance
  • Consideration is being given to the idea of
    having each LEA sign an assurance letter
    indicating they have reviewed the Clearinghouse
    reports and believe their data to be correct.
  • If the LEA cannot do this, it must submit a
    substitute letter indicating they recognize
    possible errors in their data but are allowing
    the USOE process the data and produce official
    state and federal reports from it.
  • One of these two letters must be submitted no
    later than July 15.

Next Steps - Conclusions
Computer Based Testing Update
  • Julie Quinn

Computer Based Testing Hardware Options
  • Julie Quinn

Innovative Item Types
  • Kevin King

UALPA Update
  • Kevin King

Submission Dates
  • March 2nd/6th
  • Just passed
  • Thank you
  • Reports returned after April 20th
  • May 8th Close of Testing Window
  • The FINAL
  • For ALL schools, for ALL students
  • Reports returned after June 22

Recap of Procedures
  • Ordering of Materials
  • Thank you for being patient with some production
  • We are now fully stocked with all materials
  • IF LEAs need more materials, please use UTOS
  • - IF you are a multi-pilot district, please
    continue communicating as you have with Sarah

Production Errors
  • Affected Documents
  • B1aB2b TAM
  • B1aB2b Flipchart
  • E1aE2c Flipchart
  • These are isolated incidents of missing pages
  • Please let us know ASAP
  • Please send the materials back to the USOE

Recap of Procedures
  • Answer Documents Sharons News
  • Make sure all student files are in prior to
    sending in answer documents
  • Package by level, form, grade
  • - NOT by schools
  • Need to bubble in teacher AND section on header
  • Need header sheets between grades
  • Separate level 1 and level 2 forms

Recap of Procedures
  • Answer Documents
  • BUBBLE in FORM CODE!!!!!
  • THANK YOU to LEAs who have done a good job at all
    of this.

Item Writing Evaluation
RFP Status
Thank You
  • . . . for hanging in there.

Iowa Tests of Basic Skills
  • 3rd Grade Spring Reading Test

Iowa Tests of Basic Skills
  • 3rd Grade Spring Reading Test
  • Testing Window
  • Apr. 13 - May 14, 2009

Iowa Tests of Basic Skills
  • 2009 Analyses
  • http//
  • Documents/Results_Iowa_State_20090309.pdf

(No Transcript)
(No Transcript)
(No Transcript)
(No Transcript)
(No Transcript)
(No Transcript)
  • 2009
  • Completion of robust item banks
  • 2010
  • Pre Post Equate
  • Scale Scores only after post-equating
  • 2011
  • Pre-Equate
  • Immediate Scale Scores

Pre-Equating - 2010
  • Embedded pilot items
  • May have a slightly longer test (a few more items
    in the test section)
  • Reduced number of sections because there will no
    longer be a pilot section
  • Student level spiraling
  • Easy for CBT
  • All paper test booklets returned to USOE
  • Each booklet must be accounted for

Scaling and Equating Considerations for Utah
  • David Smith
  • Assessment Development Coordinator

  • Utah score scalewith 160 established as the
    proficient scorewas established in 2004
  • Subsequent years tests were equated back to this
    2004 base year scale
  • This works as long as the construct we are
    measuring (e.g., the Core) and the test blueprint
    remains stable

But things change…
  • As you all know, the secondary ELA Core and
    mathematics Core have both changed
  • ELA was made more specific at each grade level,
    while math was made somewhat more rigorous
  • As with most things in life, everything is a
    matter of degree with score equating

What are our options?
  • Depending on the degree of change, we have a
    variety of options. The greater the change, the
    fewer the options.

ELA Options
  • If changes are judged to be minor, we can simply
    equate back to the previous score scale. In
    fact, weve been bridging from 2006 through
    2008 because of the slow evolution of the ELA
    CRTs to follow the change in Core.
  • We can recognize that the Core/test has changed
    enough to warrant establishing a new baseline,
    but can still use the 160 scale with some sort of
    designation that the scale is new.
  • If we think the changes are major, we should
    reset standards and create a completely new score

Pros and Cons
  • For every complicated issue, there is usually a
    single simple solution….except it is almost
    always wrong
  • On the following slides, we discuss some of the
    pros and cons of each of the options

Option 1 Scale Equate
  • Pros
  • Can maintain score trends
  • Easy for people to understand and interpret
  • Cons
  • Could be seen as papering over changes
  • Could be psychometrically questionable if the
    construct did in fact change more than we thought

Option 2 Establish a new 160 scale
  • Pros
  • Communicates that we have a new scale, but not
    that new
  • Allows the proficiency scale to have similar
    meanings as now, and we can move back to 160
    relatively quickly (within a year or two)
  • Cons
  • Could be hard for people to recognize that 160n,
    for example, is really different than 160

Option 3 Create a completely new scale
  • Pros
  • Could be the most psychometrically sound approach
  • If the change was large enough, this would be
    truth in advertising
  • Cons
  • Would be hard to communicate since people just
    got used to 160
  • Would have tests with different score scales

How do we decide?
  • Empirical and judgmental analyses of the ELA core
    changes suggested that the changes were not
    major, which led the technical advisory committee
    to recommend standards validation rather than
    standard setting.
  • Validation presumes that things will not be that
    different and attempts to check that assumption

Secondary ELA Validation Results
  • Put on the 160 scale for comparison purposes, the
    recommended cutscores from the validation ranged
    from 158.5 to 163.5 depending on the grade level.
  • Taking into account both measurement error and,
    more importantly, the variation (e.g., error)
    in panelist judgment, these recommended cutscores
    are all well within chance differences from 160.
  • This suggests that both internal and external
    (the panelists) judgments support the fact that
    the new Core is not very different from the
    previous Core.

Conclusion and Input
  • These results and prior judgments suggest that we
    can eliminate option 3. How do you feel about
    options 1 and 2?
  • Option 1 Scale Equate
  • Option 2 Establish a new 160 scale
  • Option 3 Create a completely new scale

A Look Ahead to Math
  • It appears that the math Core changes are more
    significant that ELA
  • Currently, we are conducting the empirical and
    judgmental analyses to help us make these
  • We could possibly arrive at a different answer
    for math than ELA
  • How would you feel about that?

  • February 2009 Test Information
  • Nolan Fawcett

Tentative Report Schedule
  • SSID file cleanup
  • 3/16-3/18
  • Preliminary State results to USOE
  • 4/8-4/13
  • Slice Files and Reports released to website
  • 4/16
  • Paper reports shipped
  • 4/23

Reports on Web site
Questions Answers