NAEP SCIENCE INTERACTIVE - PowerPoint PPT Presentation

Loading...

PPT – NAEP SCIENCE INTERACTIVE PowerPoint presentation | free to view - id: 8f511-MTQ1M



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

NAEP SCIENCE INTERACTIVE

Description:

School hardware and connectivity: Wide variety of hardware, OS, network connectivity, security policies ... on school hardware, software, or connectivity ... – PowerPoint PPT presentation

Number of Views:52
Avg rating:3.0/5.0
Slides: 29
Provided by: mn71
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: NAEP SCIENCE INTERACTIVE


1
(No Transcript)
2
  • NAEP SCIENCE INTERACTIVE
  • COMPUTER TASKS - 2008

3
Measurement Challenges
  • Developing tasks that match the framework
  • Building tasks that measure both science content
    and process
  • Developing substantive tasks that can be
    completed within 30 minutes
  • Developing interfaces that are appropriate and
    engaging and that do not
  • introduce construct irrelevance
  • take too long to learn
  • Developing assessment, NOT teaching tasks

4
ICT Development and Review Process
  • Work at ETS, NCES, NESSI, Westat, Pearson,
    Fulcrum IT, and with external advisors
  • Tryouts with students
  • Various iterations of tasks

5
2008 Pilot
  • Two Goals (broader than most pilot tests)
  • Pilot test tasks
  • Evaluate student engagement in and understanding
    of tasks
  • Try out scoring processes
  • Evaluate item performance
  • Learn about issues related to delivery of
    simulation-based assessments on nationally
    representative samples

6
2008 Pilot Delivery Goals
  • To use TestNav, an existing system from Pearson
  • To use school equipment
  • To use schools internet connection as much as
    possible to deliver content and retrieve results

7
Initial Challenges Delivering ICTs
  • School hardware and connectivity
  • Wide variety of hardware, OS, network
    connectivity, security policies
  • Variety of applications running in background
  • Pre-Assessment Visits, including installation of
    software, longer and more complex than PP
  • Lack of incentive for school personnel to address
    these challenges

8
Problems With Initial Administrations
  • Students unable to begin test
  • Test sessions frequently aborted
  • Installation and technical support burden
    untenable

9
Response to Delivery Challenges
  • Adjusted desktop settings
  • Switched to completely disconnected mode
    utilizing USB flash drives
  • Use NAEP hardware instead of school computers as
    much as possible
  • Support field administrators with IT staff

10
Delivery Status Update
  • Significantly higher success rate
  • First 3 weeks approximately 50
  • Final 2 weeks approximately 90
  • Combination of NAEP hardware and disconnected
    mode create much more predictable environment
  • Issues of scalability

11
Looking Ahead to 2009, 2010, 2011
  • Short-term Use NAEP hardware as much as
    possible, change desktop settings, work in
    disconnected mode
  • Long-term Remove any untenable reliance on
    school hardware, software, or connectivity
  • Memorandum describing options under development

12
The Tasks Themselves
  • Field data not yet available
  • DAC raised questions about the constructs
    underlying the extended tasks
  • A few issues about interface comfort have surfaced

13
Next Steps for Task Development
  • Reconvene external advisory panel and DAC
    subcommittee, and involve experts in simulation
    development from places like MIT
  • Revamp tasks as needed over next 6 months
  • Lock-down content no later than September 1

14
What to Expect for 2009
  • Until we have full data, we will not be able to
    determine the most appropriate analysis approach
  • We would recommend a special report on these
    tasks and what we have learned after the main
    release of results

15
View from the Field
  • From January 28 to March 7, NAEP administered
    computer delivered assessments in 273 schools
  • Sample
  • 280 schools
  • Grades 4, 8 and 12
  • 7 schools declined
  • Method
  • Both online and flash drives

16
View from the Field
  • Operationally, what does it mean to administer a
    computer delivered assessment?
  • Staffing
  • Training
  • Resources at school

17
View from the Field
  • Staffing for computer delivered assessments
  • Skills needed for paper and pencil assessments
  • Understanding of and experience working in a
    school environment
  • Experience working with students
  • Skills needed for computer delivered assessments
  • Understanding of and experience working in a
    school environment
  • Experience working with students
  • Basic computer skills (flash drives)
  • Computer knowledge to understand basic network
    infrastructures (online)
  • Ability to transport and set up 15 laptops (NAEP
    provided equipment)

18
View from the Field
  • Training for Online Assessments
  • PowerPoint Demonstration
  • Live demonstration (online)
  • School computer lab simulation
  • Review of and resolution common failures
  • In addition
  • Help Desk
  • Troubleshooter field staff
  • Pearson technical professionals

19
View from the Field
  • Resources at School
  • At the planning visit (online only)
  • School/district computer specialist
  • Time in computer lab to test computers
  • On assessment day
  • Use computer lab
  • Computer specialist on standby (online)

20
View from the Field
  • Reaction to computer administered testing?
  • School Staff
  • Students
  • NAEP State Coordinators

21
View from the Field
  • School Staff Debriefing
  • School coordinators
  • School computer specialist
  • Overall reactions
  • Pleased with student engagement in computer
    assessment
  • Concerns about the amount of time that computer
    labs and classrooms were occupied generated the
    small number of negative reactions

22
View from the Field
  • School Coordinators
  • Brochure prepared them well
  • Computer laboratory occupied too long
  • Assessment process was time consuming

23
View from the Field
  • School computer specialist
  • Process is time consuming
  • District coordination required in many schools
  • Needed to grant access so students could write
    to the subfolder was complicated
  • Would like more advance notice in future
  • Need a back up plan
  • Liked idea of using flash drives

24
View from the Field
  • Administrator observations and student
    comments/questions
  • Students at all grades were engaged
  • Patient and understanding of technical
    difficulties
  • Students did not use scratch paper
  • Many students at all grades did not finish task
    in time allowed

25
View from the Field
  • Grade 4 students had questions about
  • Navigation
  • How to play videos
  • How to advance to the next page
  • Double clicked and could not go back to the
    previous section
  • Unsure about what was direction or part of task
  • Other questions from 4th graders
  • What does horizon, data mean
  • How do you make the degree sign , 97
  • Did not always understand directions

26
View from the Field
  • Grade 8 and 12
  • Could not advance until key questions were
    answered
  • Found directions difficult or unclear
  • Even older students did not complete tasks in
    time allowed

27
View from the Field
  • NAEP State Coordinator comments
  • Suggestions to improve planning visit
  • If NSC has all technical requirements, can help
    network administrators to prepare
  • More detailed instructions needed for school IT
    staff

28
View from the Field
  • NAEP State Coordinator comments about the
    sessions
  • Even with work done at planning visit, students
    could not access site or if access gained,
    students kicked off (online)
  • Should schedule in mid morning, so computers can
    be tested again before assessment starts
  • Too much time required of district/school IT
    staff
  • Send a techy to each school
  • Help desk support inadequate
About PowerShow.com