Digital Forms of Assessment Assessment of Practical Performance Using Digital Formats - PowerPoint PPT Presentation


PPT – Digital Forms of Assessment Assessment of Practical Performance Using Digital Formats PowerPoint presentation | free to download - id: ab897-NzBkN


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation

Digital Forms of Assessment Assessment of Practical Performance Using Digital Formats


Digital Forms of Assessment Assessment of Practical Performance Using Digital Formats – PowerPoint PPT presentation

Number of Views:88
Avg rating:3.0/5.0
Slides: 37
Provided by: curricul9


Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Digital Forms of Assessment Assessment of Practical Performance Using Digital Formats

Digital Forms of AssessmentAssessment of
Practical Performance Using Digital Formats
  • Paul Newhouse
  • Associate Professor - Director, CSaLT

A research project funded by a three year
Australian Research Council (ARC) Linkage grant
and the Curriculum Council of WA. Conducted by a
research team led by Paul Newhouse and John
Williams from the Centre for Schooling and
Learning Technologies (CSaLT) School of
Education, Edith Cowan University
The Problem - generally
  • Student performance in many areas is difficult to
    capture using paper and pen in 3 hours.
  • Students need to be provided with assessment
    opportunities that are
  • authentic
  • reliably and manageably judged
  • Good teaching must be encouraged and rewarded
  • All students need to be engaged with their study
  • In the developed world a very small proportion of
    tasks are done on paper.

  • What is assessed is critical because
  • (1) Students tend to focus on, and be motivated
    by assessment.
  • (2) Teachers tend to teach to the test.
  • (3) We are accountable to our society for the
    outcomes of the use of resources in education.
  • If tests designed to measure key learning in
    schools ignore some key areas because they are
    harder to measure and attention to those areas by
    teachers and schools is then reduced, then those
    responsible for the tests bear some
    responsibility for that (McGaw, 2006, p. 3).

Performance assessments that are locally managed
and involve multiple sources of evidence assist
students in learning and teachers in teaching for
higher order skills Large scale examples come
from states such as Nebraska, Wyoming,
Connecticut and New York, ... The evidence from
research on these and other systems indicate that
through using performance assessments schools can
focus instruction on higher order skills, provide
a more accurate measure of what students know and
can do, engage students more deeply in learning
... (Wood, Darling-Hammond, Neill and Roschewski,
2007) - Report to US Congress
Critical questions
  • Any performance assessment approach or strategy
    will not be perfect and requires compromises.
  • (1) What skills or knowledge are best
    demonstrated through practical performance?
  • (2) What are the critical components of that
    practical performance?
  • (3) What representations of performance
  • could be used?
  • (4) What level of compromise in reliability,
  • authentication and cost is acceptable in
  • preference to NOT assessing the
  • performance at all?

Learning from the Literature
  • Completed a wide ranging literature review into
  • Performance assessment
  • Computer-based assessment
  • Assessment
  • Created online database of papers Endnote
  • Created a wiki to begin analysis

Learning from the UK
Project e-scape (2004-2007) Technology
Education Research Unit, University of London
(Richard Kimbell Tony Wheeler) Assessment of
student performance on a design task.
2007 Pilot Project
  • Considered the assessment options of
  • Electronic Portfolio
  • Performance/Simulated Tasks Exam
  • Production Exam
  • Worked one-on-one with a group of teachers to
    develop assessment instruments (no common tasks).
  • Student work collected from 7 AIT classes and 7
    Engineering classes
  • Development of marking tools
  • Work marked using Scales of Achievement
  • Comparative pairs trial

  • ARC grant for three year investigation to involve
    FOUR courses.
  • How are digitally based representations of
    student work output on authentic tasks most
    effectively used to support highly reliable
    summative assessments of student performances for
    courses with a substantial practical component?

Proposed Research
  • 1 Proof of Concept The aim will be to trial
    each of the forms of assessment in one course in
    at least four schools. This will include placing
    student work in an online repository and marking
    using the comparative pairs technique.
  • 2 Prototype The aim will be to implement a
    prototype of each form of assessment in one
    course in at least four schools under normal or
    typical conditions. This will include the use of
    direct capture of student work into the online
    repository in at least one course.
  • 3 Scalable Product The aim will be to
    implement each of the forms of assessment in at
    least one course in a representative sample of
  • Four courses Applied Information Technology,
    Engineering Studies, Italian, and Physical
    Education Studies

Research Team
  • Paul Newhouse - coordination of the project
    leader AIT team.
  • John Williams - co-cordination leader
    Engineering team.
  • Dawn Penney (Uni of Tas) - leader Physical
    Education Studies team.
  • Cher Ping Lim - leader Italian team.
  • Jeremy Pagram - expertise in the use of digital
  • Andrew Jones - local leadership for PES team.
  • Mark Hackling - oversight for the design of
    assessment instruments.
  • Ron Oliver - quality assurance for research
  • Russell Waugh - expertise in Rasch analysis.
  • Specialist Consultants
  • - Professor Richard Kimbell (University of
    London) and TAG Learning.
  • - Professor David Andrich and Dr. Stephen
    Humphry (UWA)
  • An Advisory Group - representatives of school
    systems and Curric Council.
  • Working Parties - researchers, Council personnel
    and participating teachers.
  • Technical Support - ECU technical support
    contracted support.
  • Management and Research Assistants - project
    manager and two post-doctoral scholars Martin
    Cooper and Alistair Campbell.
  • Two Doctoral students

Key Features
  • (1) The representation in digital files of the
    performance of students doing practical work.
  • (2) The presentation of digital representations
    of student performance in an online repository so
    that they are easily accessible to markers.
  • (3) Assessing the digital representations of
    student performance using both analytic marking
    and the paired comparison judgement method with
    holistic and component criteria-based judgements.

Online Repository
  • Storage of digital representations of student
    work for online access.
  • Considered using MAPS as in pilot study
  • Difficulty accessing variety of file types
  • Marking tools separate from access to student
  • Decided to try accessing student work through
    folders/files on a server using Filemaker online
    to deliver through a browser

Database Driven Marking Tools
  • Developed two prototypes in Filemaker online
  • Standards-Reference Analytical marking tools
  • Comparative pairs marking tools
  • Allows the viewing of digital work and criteria
    and recording of judgements both quantitatively
    and qualitatively
  • Markers can be anywhere in the world
  • Easy transfer to spreadsheets and statistic
    packages for analysis.

Determining Assessment Tasks
  • For each course completed a situation analysis
  • Student characteristics
  • Course requirements
  • The Performance
  • (1) What skills or knowledge are best
    demonstrated through practical performance?
  • (2) What are the critical components of that
    practical performance?
  • (3) Why cant those components be demonstrated on
  • The Technologies
  • (1) What alternative representations other than
    paper could be used?
  • (2) What level of compromise is acceptable?
  • The Teachers
  • (1) What are key characteristics of typical
  • (2) How do these characteristics relate to
    assessment in the course?
  • (3) How do these characteristics relate to the
    use of ICT in the course?
  • Determined appropriate form of assessment
  • Described task(s), marking criteria, and created
    student materials.

Courses Assessment Tasks
  • AIT
  • Portfolio Performance Tasks Exam
  • Engineering Studies
  • Extended Production Exam
  • Italian
  • Oral Exam with Portfolio
  • Phys Ed Studies
  • Performance Tasks and Response Exam

Capturing Performance
  • AIT
  • USB Flash Drives, templates in Word, Powerpoint
  • Engineering Studies
  • eeePC and Filemaker Runtime template
  • Italian
  • Digital video camera with radio microphone
  • Digital still camera and desktop computers
  • Phys Ed Studies
  • PTZ remote control cameras
  • Combined video feeds with audio
  • Filemaker runtime template
  • USB Flash Drives

Physical Education Studies
PES Assessment Task
  • A challenge/problem was set for each sport
  • Rugby lineout
  • Soccer corner
  • Volleyball return serve
  • Swimming medley
  • (1) Develop response
  • (2) Video skills routines
  • (3) Video game (main and alternate strategies)
  • (4) Reflection evaluation

Physical Education - video
PES First Exam
  • Challenge set for the sport
  • Rugby lineout
  • Soccer corner
  • Volleyball return serve
  • Swimming medley
  • (1) Develop response

PES Final Exam
PES Marking 2
PES Comparative Pairs Marking
Engineering Studies
Engineering Assessment Task
  • A series of specified activities which will take
    students from a design brief to the construction
    of a model over a period of 3 hours.
  • Each activity was timed, so students had a
    specific time frame in which to complete each
  • Each student had mini computer (ASUS eeePC) on
    which they will compile their portfolio within a
    pre-formed template already loaded onto their
  • The Task Manager or Teacher coordinated the
    activity by showing a presentation which will be
    projected onto a screen for all the class to see.
    This was the mechanism for managing the set time
    students had on each activity, prompting the
    students to move onto the next page of their
    portfolio at the appropriate time.
  • The task involved the design and modelling of a
    solar water heater for a rural developing country

Engineering Studies
  • 3 hour practical exam supported by eeePC
    (built-in mic), webcam, USB drive and Filemaker

Marking the Portfolio
  • High quality radio microphone connected to a
    digital video camera.

Italian Assessment Task
  • Part A - Folio Andiamo!
  • Collect a series of tasks and activities
    including map activity, retrieval chart and
    question answers, brainstorm, fact sheet, word
    processed reflection on practice talk and one
    minute voice recording evaluating final
    performance, to show development of ideas and
    preparation for presentation (Part B).
  • Part B - Presentation Destinazione Western
  • Prepare a talk to be recorded on video/DVD and
    send to a school in Italy. Your talk will focus
    on your local area/WA destination, providing
    information such as, features of the area,
    recreational activities and cultural events. You
    should also consider the time of visit, and
    choose activities which are appropriate.
    Comparisons with the area you have chosen and
    region in Italy where your presentation will be
    shown should also be included. During your
    presentation you will be asked to respond to a
    question relating to the information youve

(No Transcript)
Applied Information Technology
AIT Assessment Task
  • Digital Reflective Process Portfolio
  • (1) A digital product designed as a prototype of
    an information solution.
  • (2) The design process document for that digital
  • (3) Two other digital artefacts that illustrate
    skills in areas from a specified list.
  • Performance Tasks Exam
  • Completed using a computer workstation and
    included a set of reflective questions
    (on-screen) concerning the digital portfolio and
    a set of short performance tasks.
  • Exam (Component 4) - Reflective Questions
  • Exam (Component 5) - Performance Task(s)
  • Students completed a set of scripted tasks at a
    computer workstation over a short period of time
    (2 hours) under exam conditions.
  • Produced a logo, graphs and brochure.

(No Transcript)
AIT - Pairs Marking Tool
Rasch Analysis of Analytical Marking
  • Exam
  • SI0.85
  • Portfolio Product
  • SI0.96

What now?
  • Refine assessment tasks
  • Improve use of digital technologies (e.g. MAPS)
  • Further improve implementation of assessment
  • Improve efficiency of comparative pairs marking
    (e.g. MAPS Pairs Engine)
  • Scale up in 2010 (more representative schools)
  • Report available
  • http//