Computer-based and computer-assisted tests to assess procedural and conceptual knowledge

1
Computer-based and computer-assisted tests to
assess procedural and conceptual knowledge
Richard C. Rayne and Glenn Baggott School of
Biological and Chemical Sciences Birkbeck
College, University of London, WC1E 7HX,
UK r.rayne_at_bbk.ac.uk
2
Pedagogical Problem

• Biosciences are essentially practical and
  experimental subjects
• Biosciences undergraduates therefore must develop
  abilities to
• acquire and apply procedural knowledge
• demonstrate understanding of concepts that
  underpin procedures
• use procedural knowledge in appropriate contexts
  i.e. to develop strategic knowledge within the
  domain

3
Knowledge framework
c.f. Shavelson et al. (2002) EARLI/Northumbria
Conference keynote address
4
Declarative knowledge

• Knowing that
• a fact-base, or knowledge store

5
Procedural knowledge

• Knowing how to do something
• Consists of
• if/then rules
• sequence of steps
• Calls upon
• declarative knowledge

6
Schematic knowledge

• Having a scientifically justifiable conception,
  i.e. knowing why
• May be used to
• interpret problems
• troubleshoot
• explain an outcome
• predict an outcome
• Depends on
• having an understanding of principles
• conceptual knowledge combination of
  declarative and procedural knowledge

7
Strategic knowledge

• Knowing when, where and how to use certain types
  of knowledge in a new situation
• conditional knowledge, strategies
• what strategic knowledge is applicable here?
• essential for solving new problems

8
Conceptual knowledge

• a subset of schematic knowledge
• taken to equate to understanding the basis for a
  particular procedure

as I have used in the abstract
9
Knowledge framework
c.f. Shavelson et al. (2002) EARLI/Northumbria
Conference keynote address
10
A key aim

• Exploit the strengths of CBA to promote
  development of procedural and conceptual
  knowledge in biosciences students
• Context
• first-year module Molecular Cell Biology
• using case-based approaches in different forms

11
Two Approaches

• TRIADS CBA focusing on a classic experiment
• Meselson-Stahl experiment outcome simple, but to
  appreciate its power requires understanding of
  several sophisticated principles
• CaseIT! simulation software
• to teach and assess procedural and conceptual
  knowledge of key molecular biology techniques
• http//www.uwrf.edu/caseit/caseit.html

12
CaseIT!

• Virtual molecular biology lab
• apparatus for performing separation of DNA
  fragments and for identification of specific DNA
  sequences
• apply molecular biology techniques in the context
  of a case study of human genetic disease
• must perform a Southern blot

13
Southern Blot Procedure

• Component tasks
• obtain DNA sample and select enzyme
• perform reaction to digest the DNA
• load sample onto electrophoresis gel
• run gel stain and destain
• transfer DNA from gel to a membrane
• select radioactive hybridisation probe and apply
  to membrane
• reveal specific DNA sequences by exposing
  membrane to X-ray film

14
Movie

• CaseIT in action

15
Computer-Assisted Test

• t0 Introduction to techniques and to CaseIT!
  (demonstrated in lecture)
• t1 Classroom session, hands-on (3 h)
• students work on cases involving Southern
  blotting as the key technique
• t5 Test session
• paper-based test with unseen case
• must use CaseIT to obtain results
• test items (short answer responses) depend on
  correctly performing Southern blot

16
Why did we do it?

• History
• In 2002, only 4 of 40 students correctly answered
  an item on a TRIADS exam which required
  understanding of the Southern blot procedure
• For 2003, we devised and used the CaseIT test it
  was used again in 2004
• CaseIT! Test Results
• 2003 2004 extremely poor on the whole
• 2004 mean 42
• 30 of 66 got lt35!
• but 61 of 66 (in 2004) were able to perform the
  Southern blot procedure correctly
• TRIADS Item, 2004
• 18 of 57 scored 100

17
Southern blot TRIADS item
18
Summary

• CaseIT! provides a reasonably realistic analogue
  of real lab techniques
• helps build understanding of rationale/ approach
  in fail-safe environment
• The CaseIT! test forced students to learn blot
  procedure!
• have to learn a logical sequence of procedures
  that mimics the learning required in the real
  situation
• CAA/CBA need not be a test given on a computer

19
Acknowledgements

• Funding
• Birkbeck College Development Fund
• Higher Education Funding Council of England
  (HEFCE) through FDTL4
• People at Birkbeck
• Authorware Programmers Dijana Maric, Ellen
  McCarthy, Caroline Pellet-Many
• People elsewhere
• Don Mackenzie and his team at the Centre for
  Interactive Assessment Development, University of
  Derby
• Mark Bergland, author of CaseIT!, University of
  Wisconsin-River Falls

20
References

• Bergland M., et al. (2004) DNA Electrophoresis
  Module for CaseIT!, version 4.0.2.
  http//www.uwrf.edu/caseit/caseit.html
• Shavelson, R.J., Li, M., and Ruiz-Primo, M.A.
  (2002) Evaluating new approaches to assessing
  learning. Keynote address to the Joint
  Northumbria/EARLI Assessment Conference,
  Newcastle, UK. August 2002.http//www.stanford.ed
  u/dept/SUSE/SEAL/Presentation/Presentation.htm

21
Contact

• Dr Richard Rayne
• Lecturer in Biology and Director of the OLAAF
  Project School of Biological Chemical
  SciencesBirkbeck CollegeUniversity of
  LondonMalet StreetLondon WC1E 7HX, United
  Kingdom44 (0)20 7631-6253r.rayne_at_bbk.ac.uk
• OLAAF web site
• http//www.bbk.ac.uk/olaaf/

22
Meselson-Stahl Experiment

• Outcome
• DNA replication is semi-conservative in bacteria
• Component knowledge/understanding required to
  understand the experiment
• what is an isotope?
• what is density?
• how does a centrifuge work?
• how to separate molecules based on their density?
  (density gradient centrifugation)

23
TRIADS CBA

• 2004 text and supplementary written materials
  provided
• 5 items on TRIADS exam 1 comprehension 2
  application 2 problem solving
• 2005 TRIADS tutorial to replace (or supplement)
  the paper materials
• uses animations to demonstrate concepts visually
• will test basic understanding at component
  level (i.e. concept-by-concept)
• Recall Qs precede Comprehension Qs
• Achievement must answer lower level to proceed
• same items as in 2004 to be used on exam to probe
  conceptual understanding

24
M-S 1.1
25
M-S 1.2
26
M-S 1.3
27
M-S 2.1
28
M-S 2.2
29
M-S 2.3
30
M-S 3.1
31
M-S 3.2
32
M-S 3.3
33
M-S 4.1
34
M-S 4.2
35
M-S 4.3
36
M-S 5.1
37
M-S 5.2
38
M-S 5.3
39
Assessment Development Model
40
Cognitive type inventory

• ReCAP
• Recall
• Comprehension
• Application
• Problem solving
• ( analysis synthesis evaluation)
• Imrie (1995) Assessment Evaluation in Higher
  Education 20 (2) 175-189.

41
Southern blot item 2