Designing multiple choice questions and feedback responses

1
Designing multiple choice questions and feedback
responses

• Phil Race

2
Designing multiple-choice questions and feedback
responses

• In this workshop, well explore how
  multiple-choice questions can be used to provide
  students with very rapid formative feedback on
  their on-going learning.
• It is important for feedback responses to address
  the two questions in students minds each time
  they pick an option was I right? and
  particularly if not, why not?.
• We will also explore some of the limitations of
  multiple-choice questions when merely used for
  testing rather than for formative feedback.

3
Intended learning outcomes

• To give you a short test!
• To look at how multiple-choice questions can be
  designed to allow learners to learn from rapid
  and direct feedback on their thinking.
• To illustrate hot potatoes.
• To get you designing some multiple-choice
  questions and feedback responses.

4
A content-free test!

• Please work on your own.
• Try out your exam technique to pick the most
  likely option for each of the eight questions.
• Then try to work out why you made your selection.
• If you feel you can make sense of the questions,
  please seek help!

5
Please jot down numbers 1-8

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8

6
1 The usual function of grunge prowkers is to
remove

• A grunges.
• B snarts.
• C trigs.
• D grods.

7
2 Antigrottification occurs
A on spring mornings. B on summer evenings
provided there is no rain before dusk. C on
autumn afternoons. D on winter nights.
8
3 Lurkies suffer from trangitis because
A their prads are always underdeveloped. B all
their brizes are horizontal. C their curnpieces
are usually imperfect. D none of their dringoes
can ever adapt.
9
4 Non-responsive frattling is usually found in
an
A gringle. B janket. C kloppie. D uckerpod.
10
5 Which are exceptions to the law of lompicality?
A The miltrip and the nattercup. B The bifid
pantrip. C The common queeter. D The flanged
ozzer.
11
6 Which must be present for parbling to take
place?
A Phlot and runge. B Runge. C Stuke and
runge. D Runge and trake.
12
7 One common disorder of an overspragged
uckerpod is
A copious vezzling. B intermittent
weggerment. C non-responsive frattling. D uneven
yerkation.
13
8 Which of these is the correct answer?
A B C D
14
1 The usual function of grunge prowkers is to
remove

• A grunges.
• B snarts.
• C trigs.
• D grods.

Dont include a word from the stem in one of the
options, or it will make people think it is the
correct option.
15
2 Antigrottification occurs
A on spring mornings. B on summer evenings
provided there is no rain before dusk. C on
autumn afternoons. D on winter nights.
This option looks more probable, because it is
longer, and because it is the only one which is
qualified.
16
3 Lurkies suffer from trangitis because
A their prads are always underdeveloped. B all
their brizes are horizontal. C their curnpieces
are usually imperfect. D none of their dringoes
can ever adapt.
Dont mix indefinites with definites
indefinites are more likely to be correct.
(Indefinites usually, sometimes, often,
seldom Definites all, never, none, and so on)
17
4 Non-responsive frattling is usually found in
an
A gringle. B janket. C kloppie. D uckerpod.
Dont allow the indefinite article (an) to give
away the right answer.
18
5 Which are exceptions to the law of lompicality?
A The miltrip and the nattercup. B The bifid
pantrip. C The common queeter. D The flanged
ozzer.
Dont ask for something plural, then make just
one option plural.
19
6 Which must be present for parbling to take
place?
A Phlot and runge. B Runge. C Stuke and
runge. D Runge and trake.
It its not this option, it cant be any of them
(one way of thinking about it!!)
20
7 One common disorder of an overspragged
uckerpod is
A copious vezzling. B intermittent
weggerment. C non-responsive frattling. D uneven
yerkation.
Be careful about overlapping questions!
Candidates should look at both questions, and get
maximum thinking.
21
8 Which of these is the correct answer?
A B C D
There was a pattern. D is due to come up again.
22
Correct answers
1 A 2 B 3 C 4 D 5 A 6 B 7 C 8 D
23
Some examples

• (I showed three examples of multiple-choice
  questions composed in PowerPoint. I havent
  included these in these web-based slides as the
  various hyperlinks dont transfer successfully
  email me if youd like me to email them direct to
  you).

24
Writing MCQs (multiple-choice questions) and
feedback responses

• Various software packages are available, which
  can be used for both formative and summative
  online or computer-based assessment.
• Once written, multiple-choice questions can save
  us a lot of time. Used formatively, they can be a
  useful vehicle for helping students to find out
  how their learning is progressing and where their
  weaker areas are.
• Used summatively (after due testing and piloting)
  they can save us time assessing some aspects of
  students learning.
• This session will not be about the technology but
  about the design of good questions.

25
Designing interactivity

• Think of something which you want your students
  to do working with the computer.
• Next think of the feedback you will provide to
  your students
• If they do it right
• If they dont!

26
Please work in groups of 3 or 4

• Choose one of you, whose ideas for doing and
  feedback will be worked up with the help of the
  other members of the group.
• Draft out on OHT film the words which will appear
  on screen to set the task
• and some of the feedback messages which will
  follow it.

27
Marking scheme

28
The task..

• Think of something which you want your students
  to do working with the computer.
• Next think of the feedback you will provide to
  your students
• If they do it right
• If they dont!
• Choose one of you, whose ideas for doing and
  feedback will be worked up with the help of the
  other members of the group.
• Draft out on paper the words which will appear on
  screen to set the task
• and some of the feedback messages which will
  follow it.

29
WIRMI

• What it really means is
• To help you decide the best possible wording for
  the question, and for the feedback responses.

30
Using multiple-choice for feedback

• The main point is that students get useful
  feedback on whichever options they choose.
• It can be worth saying which one of these do
  you think is best, and can you spot whats wrong
  with each of the other options? and allowing
  students to learn from the whole agenda behind
  the question.

31
MCQ for testing

• You need to pilot with (e.g.) 3000 learners.
• What seems like a good question isnt always a
  good one!
• Get the software to work out for you the
  discrimination index (how well the question
  rewards the more-able candidates compared to the
  less-able ones)
• And get the software to work out the facility
  value a measure of how easy the question is
  found to be, on average.

32
Tips

• Write questions in groups get plenty of
  feedback on your ideas.
• Students are often much better at thinking of
  good distractors than we are!
• They also know how best to respond to
  distractors.
• Its better to write 23 bad questions in an
  hour than to write one good one, - several of the
  23 will be able to be turned into good ones.

33
Action planning statements

• One thing Im going to do is
• One idea Im taking away is
• Im going to think more about
• I have found out that
• Id like to know
• In future, Im not going to

Monday, 16 November 2009
34
Thank you www.Phil-Race.com e-mail
p.race_at_leedsmet.ac.uk