Creating Confidence and Activity with eAssessment

1
a
Creating Confidence and Activity with
e-Assessment
Dr Richard Parsons Director of the Learning
Centre University of Dundee r.parsons_at_dundee.ac.u
k
2
University of Dundee

• More than 10 years substantial experience of
  online assessments
• Blackboard clients Academic Suite (virtually
  100 modules)
• Delivering up to 3500 formative assessment
  sessions a day with QM Perception
• Delivering 50 summative exams a year with QM
  Perception
• Range of online assessment activities recognised

3
E-Assessment Matters

• Efficient (gains are now documented)
• Effective
• Engages students
• Provides feedback to students
• Any place, any time, any pace
• Key tool for academic analytics Identify
  students who may be at risk, personalise
  individual student progress

4
The Reality

• For many reasons e-Assessment matters
• But
• Too recent to be sustained
• Changes are frequent and substantial
• Academic engagement is varied
• Is a complex activity to manage

5
Individual concerns within an online assessment
world
College Officer Can we book an IT suite for 240
students on Wednesday the 25th?Two students have
additional requirements.Will the results
transfer directly to the SRS?
Traditional Lecturer Who can electronically
author my questions?Can we use textbook
questions?Is this assessment driving learning?
eLearning Manager Are the staffing levels
correct?Is our software choice appropriate?Are
the server specifications correct to cope with
the expected load?Are staff education procedures
in place?
Lecturer Are my questions too simple?Is this
technology reliable?Do the students find it
tedious?Where can we obtain challenging
questions?Are these assessments aligned with our
learning outcomes?
IT Desktop Administrator Does the desktop
software operate correctly?Are the exam
accounts secure and available?Are the PCs well
maintained and reliable?
Timetabling Officer Can the proposed exams be
fitted into the best rooms?Are assessments
receiving appropriate prioritisation?
Student This is nice and privateI am getting
real feedback hereDo I know the answers?Can I
learn from this?I am at home in this online
environment
Distance Student Is my technology compatible?Is
my connection reliable?Is this a replacement for
F2F?
IT Server Administrator Is the server fully
patched, reliable and secure?Is the OA software
functioning correctly and reliablyDo we have
monitoring and is it working?Are the logging and
backup procedures working correctly?
Examinations Officer Are these results 100
reliable?secure?Do our procedures represent the
best possible practice?
Online Assessment Officer Are all assessments
up-to-date and correctly available?Is the
server-client functionality all compatible?Will
the next examination run smoothly?Are the
staff education programmes ready?
University Database Administrator Are the
databases supporting e-assessment optimised?Is
disk space available for the expected expansion
in activity?Are our archive policies
operationally appropriate?
Richard Parsons, University of Dundee
6
Creating Confidence

• Responsibility of the e-Assessment manager
• Attend to matters behind the scenes
• Resource and support the infrastructure
• Sustain with annual meetings
• What is plan B?

7
Online Assessment System
Assessment hardware design deployment
Assessment software design deployment
Clear policy and procedures
Central assessment support
Learning systems integration
Staff and student education
8
University of Dundee Questionmark Perception V4
Systems Model
Participant
Participant
Oracle 10G
Oracle 10G
QMP V4.2 CAA-02 Production Formative
Repository
Participant
QMP V4.2 CAA-01 Production Exam Repository
Participant
LC Administrator
QM / Bb integration
QMP V4 Authoring Manager
Oracle 10G
Oracle 10G
Participant
QMP V4.2 CAA-03 DevelopmentRepository
Blackboard 7.3 My Dundee
Local Author (Staff)
Participant
QMP V4 Authoring Manager
Participant
Local Author (Staff)
QMP V4 Authoring Manager
Local Author (Staff)
David Walker, University of Dundee
9
The Annual Cycle of Online Assessment
Sept.
Testing tuning
Modules begin
Online resit exams
Annual frameworkmeeting
Summer upgrade window
Online exams
Dec.
June
Mid semester window
Online exams

• Growth may occur each year
• Changes are made to the systems
• A framework allows the process to be sustained

OA staff education module
Richard Parsons, University of Dundee
March
10
Clear Policy and Procedures

• Central online assessment system required clear
  guidance
• Prepared draft Policy and Procedures
• Generally follows BS79882002
• Developed for about a year, inc. consultation
• Passed at University Senate
• Excellent for ensuring quality
• Runs to 17 pages
• Additional documentation for Online Assessment
  system administrators

11
Summative Procedures

• Assessment must be tested
• Assessment loaded to both servers, scheduled on
  one
• Central staff attend examinations
• Use special exam accounts
• Use Perception Secure Browser
• Candidates complete Examination Attendance form
• Immediately after the exam a backup of results is
  taken
• Use two way radios or mobiles
• No system is fullproof, 99 success target ?

12
Creating Activity

• Need to engage many academic staff
• But how?

13
Online Assessment System
Assessment hardware design deployment
Assessment software design deployment
Clear policy and procedures
Central assessment support
Learning systems integration
Staff and student education
14
The Hardest Task

• Remains writing quality questions
• Staff time is precious
• Students are more critical than ever before
• Assessments to be aligned
• Questions need to be prepared ahead of time
• Quality runs to feedback, distracters, variety,
  depth, currency
• Authoring Q within the software is secondary

15
Staff and Student Education

• For 5 years at Dundee we have run an Online
  Assessment course, online
• Covers technical and pedagogical aspects
• 5 weeks long, 3 to 6 hours a week.
• Formally accredited by our Education Faculty (10
  SC)
• Building a bank of staff skills
• Most students need little guidance

16
Examples of Assessment Strategies

• 15 Q from pool of 100, max mark taken (Genetics)
• Take assessment first, then repeat in an open
  fashion. Mark taken from first run. (Medicine)
• 10 numerical questions, each from a pool of 50
  randomly generated Q (Life Sciences)
• Student revision students invited to compose 2
  Q each with FB, all compiled automatically (Env
  Sci)
• Open assessment, one submission only (Law)
• Self assess your answer against a model answer
• Summative (range of departments)

17
Incentives for Academics

• Recognise and reward e-Assessment
• Use a question coding scheme for credit
• Questions/exercises are reusable
• Self and peer assessment no marking
• Adaptive release a useful tool
• Use templates / web tools to simplify
• Teamwork for advanced questions
• Ensure alignment with references in feedback
• Demonstrate LO and assessment achievement

18
New Resources

• Geoffrey Crisps Book
• The e-Assessment Handbook
• REAP Scottish Transformation Change Project,
  documentation to support e-Assessment
• Re-Engineering Assessment Practices in Scottish
  Higher Education
• Dundees self and peer assessment tool within
  Blackboard 8

http//www.reap.ac.uk/
19
Dundees Self Peer Assessment Tool in
Blackboard version 8

• Assessments are added and edited by the
  instructors in any course
• One or more questions, each with one or more
  marking criteria
• Marking criteria as discrete values or range
• Feedback provided as score and text if required
• Students automatically allocated to mark peers (1
  to many)
• Peer assessments are assigned automatically
• Self assessment is an option
• Marks and reports are visible to the Instructor
• All marks can be downloaded for analysis in excel
• Moderated results automatically passed to Bb
  Gradebook
• Exercise text is available for download and
  online plagiarism checks
• Questions and criteria are stored for reuse and
  exchange

20
(No Transcript)
21
(No Transcript)
22
http//www.dundee.ac.uk/learning/dol/templates.htm
23
Additional Online Assessment

• Self Peer
• Framework for plagiarism advice and management
• Procedures for online submission
• Discussion board assessments
• ePortfolio assessment rubrics and tools
• Classroom/personal response systems
• Business intelligence activities
• Overall framework of OA support at Dundee

24
Benchmarking Learning

• Why e-Assessment matters
• Assessment as a key component of L T
• Embed within new evolving pedagogic models
• Formalise the change from lecture-based framework
  to online framework
• Enhance student education

25
A Model for Understanding, Benchmarking and
Enhancing HE Discipline-Based Education,
Including the Impact of Learning Technologies
Analysis Programme review, market surveys, risk
analysis, financials, ideas, synthesis, external
influences
Strategy Decisions investment Staff
appointments promotions Staff development Staff
workload allocations New programmes
modules Support service resource
enhancement Support service resource
reduction Marketing investment
Environmental Influences External SFC QAA,
Institutional, Staff makeup, Traditions,
Discipline benchmarks, Research activity
Society
Students
Employers
Graduates
Learning and Teaching Approaches
1. Discipline pedagogies
New course opportunities ()
2. Learning activities
3. Assessment activities
Student completion statistics
4. Administration, information, co- and
extracurricular activities
Measures Resolution at discipline and
institutional level Financial position Peer
institutions Press league tables KPIs Student
satisfaction data HESA returns Student recruitment
Evaluation Benchmarking
Supporting Educational Services
Academic support
QA support
Staff education
Teaching facilities
eLearning support
Teaching workload
Library support
Evaluation Benchmarking
ITS support
Third party measurement
National Student Survey
26
Teaching styles
Level
Learning style
Discipline pedagogic tradition
Instructivist
Constructivist
Didactic
Facilitative
Problem-based
Blooms taxonomy, Eric Wignall, Governors State
University
Activities
Biggs Constructive Alignment
Assessment
Synthetic
Contextual
ILOs
Procedural
Objectives
Declarative
27
Law eLearning activities
Interaction
Low
High
28
Learning activities
29
Law 101 pedagogic approach We engage the
students with a range of learning activities that
include lectures, court visits and online
discussions. Didactic lectures are used to convey
substantial amounts of information and small
group tutorials are used to explore the
understanding of a range of key concepts.
30
Life Sciences 101 assessment approach This
module is assessed through a combination of in
course (50) and final examination (50)
assessments. The in course assessments include
fortnightly online assessments that test
knowledge (6 at 3 each), practical reports (8 at
3 each and a group project (8). The final
examinations consists of a mid-semester online
assessment of 10, a final online assessment of
20 and an essay paper of 3 questions (20).
31
Summary

• E-Assessment is efficient and effective
• Solid infrastructure is a central responsibility
• Generating questions and assessment is an
  academic responsibility
• Centre can make this as straightforward as
  possible
• Recognise pedagogic approaches are changing with
  an expectation for e-Assessment

32
University of Dundee

• More than 10 years substantial experience of
  online assessments
• Blackboard clients Academic Suite (virtually
  100 modules)
• Delivering up to 3500 formative assessment
  sessions a day with QM Perception
• Delivering 50 summative exams a year with QM
  Perception
• Range of online assessment activities recognised

33
(No Transcript)
34
(No Transcript)
35
(No Transcript)
36
Elements
Actors
Procedural
Each addressed by a tasked individual(s) within a
supported team, and each element able to cope
with growth and changes Revisit annually? The
entire operation has limited inertia, limited
history, sustainable staffing, space, budget.
Represents a framework
OA policy and procedures
ZZ AA
Student and staff training
XX WW
Examination practices
Academic staff XX WW
Room booking procedures
College staff
Student personal PCs
Web guidance VV
University student desktop
BB CC
Question creation
Academic staff XX WW
Question/assessment repositories
XX WW
Network
DD EE
OA application software support
XX WW
13 specialist staff academics college staff
Technical database considerations
FF GG
Operating system web server
XX HH
Hardware infrastructure
HH II
Technical
Richard Parsons, University of Dundee
37
Blackboard Server Configuration
Summer 2006 Bb upgrade 6.3 to 7.1 on Unix, QM
application from 3.4 to 4.1, Bb - QM bridge
software (twice), QM repositories for
assessments, Blackboard building block
deployments and upgrades, Solaris and Oracle
database upgrades, Annual module rollovers,
Enhanced PDP programmes, Medical School
curriculum map integration, University
restructuring configuration changes