Assessing ILOs in Large Classes John Biggs

1
Assessing ILOs in Large ClassesJohn Biggs
2
ILOs for this session

• You should have with you ILOs for at least one
  course.
• You explore some options for suitable large class
  assessment tasks. This is not directly assessed,
  but is necessary for ILO2
• You design an assessment strategy for at least
  one course (self- and peer-assessment)

3
Assessment Tasks for Large classes

• Task-format Useful for
• Multiple choice Recognition, strategy,
  comprehension, coverage
• Ordered outcome Hierarchies of understanding
• Poster Holistic understanding, integration,
  application
• Concept maps, Venn Coverage, relationships
• Diagrams
• Three minute essay Level of understanding, sense
  of relevance
• Gobbets Appreciating significant detail, why
• Short answer Recall units of information,
  coverage
• Letter to a friend Holistic understanding,
  application, reflection
• Cloze Comprehension of main ideas
• Final exam Ensuring work is students own,
  overview of whole course

4
Assessment Procedures Large Classes

• Self- and Peer-assessment
• Self and peer assessment especially valuable in
  OBTL because the students are made conscious of
  meeting the ILOs, not on doing a good
  assignment. Peer and self often used together,
  as a learning experience, and for achieving
  clarity on what the ILOs mean.
• Group assessment
• On-line assessment
• Random assessment

5
An ordered-outcome item for physiotherapy
students

• SEVERITY
• OF ASTHMA
• 6 am 12 md
  6 pm 12 mn 6 am
• DIURNAL VARIATIONS
  IN SYMPTOMS OF ASTHMA
• When is the asthma attack most severe during the
  day?
• Is an asthmatic patient physically fitter at 1
  p.m. or 8 p.m.?
• Do you expect an asthmatic patient to sleep well
  at night? Give your reasons.
• Advise an asthmatic patient how to cope with
  diurnal variation in symptoms.

6
Example from Chemistry

• 3.0gm of carbon are burnt in excess oxygen
• (The relative atomic mass of C 12.0, O 16.
  The molar mass of any gas under the experimental
  conditions is 24 dm3.)
• Which of the following types of reaction takes
  place?
• (addition, reduction, oxidation, elimination)
• What mass of oxygen reacts with 3.0 gm of carbon?
• What is the volume of the gaseous product?
• How can hydrochloric acid be shown to have the
  formula HCl rather than H2Cl or HCl2

7
Using ordered-outcome items

• If you are using only one item for an ILO, it
  will tell you directly how well the student meets
  that ILO.
• If you use several items, you can
• Make a holistic judgement as to how well the ILO
  in question is addressed.
• Score each item, for example 4 or 5 for highest,
  3 for next, then 2, and 1 if only the lowest item
  is correct.

8
Steps in creating ordered- outcome items

• Devise a rich situation, the stem, that can be
  responded to in increasingly complex ways.
• Think of (usually) four responses, from simplest
  to the most complex. SOLO is a useful guide for
  creating these responses.
• Make sure the items form a staircase it should
  be impossible to fail an item that is lower in
  level to one that has been passed.
• Note the highest level a student can reach.
• Scoring can be qualitative or quantitative.

9
A SOLO sequence for ordered-outcome items

• Use one obvious piece of information coming
  directly from the stem. (Unistructural)
• Use two or more discrete and separate pieces of
  information contained in the stem.
  (Multistructural)
• Use two or more pieces of information each
  directly related to an integrated understanding
  of the information in the stem. (Relational)
• Use a general principle or hypothesis which can
  be derived from, or suggested by, the information
  in the stem. (Extended Abstract). Often extended
  abstract items will be open-ended
• BUT you dont have to use SOLO. Any system of
  ordering items that makes sense in your
  discipline will do.

10
Now in your group devise an ordered outcome item

• Devise a rich stem context
• Simplest answer requires only one obvious piece
  of information coming directly from the stem.
• Next answer requires two or more discrete and
  separate pieces of information contained in the
  stem.
• Next highest form of answer requires two or more
  pieces of information to be integrated to answer
  correctly.
• Highest level generalises to a situation or
  principle not directly given. This level may be
  open-ended.

11
Posters/Web pages

• Posters are easy to read summaries with visual
  support of work done either individually or in
  groups. They are especially useful in large
  classes because they are prepared out of class
  time, and can be assessed at an exhibition of
  posters, either physically or on-line. Self- and
  peer-assessment can ease the teachers load
  considerably.

12
Concept maps, Venn diagrams

• Students are presented with a central concept and
  generate sub-concepts that relate to it. They
  arrange the sub-concepts in a way that makes best
  sense to them. Lines are drawn linking sub- and
  central concepts with a brief explanation of what
  the link or relationship is. Concept maps present
  an overall picture, and as holistic
  representations of a complex conceptual
  structure, are best evaluated by judging the
  complexity of the arrangement, and the
  correctness of the interrelations.
• Venn diagrams are a simple form of concept map,
  using circles or ellipses, inter-relations
  between concepts expressed by the intersection or
  overlap of the circles.

13
Using a Venn Diagram

Give examples of interactions
at 1. 2. 3. 4.
psychologist
student
1
4
2
3
school
14
Now you design a concept map or Venn Diagram in
your group
15
Three-minute Essay

• During or at the end of a lecture you can ask
  such questions as
• What do I most want to find out in the next
  class?
• What is the main point I learned today?
• These questions provide useful information for
  the teacher. Formatively, they tell us how the
  content is interpreted by students. Summatively,
  this information can be used for grading purposes
  The three minute essay can be answered in minutes
  in a large class.

16
Gobbets, critical incidents

• Gobbets are significant chunks of content to
  which the student has to respond
• A critical incident
• a paragraph from a standard text
• a brief passage of music
• an archeological artefact
• a photograph (a building, an engine part)
• The students task is to identify the gobbet,
  explain its context, say why it is important,
  what it reminds them of, or whatever else you
  would like them to comment on.
• Gobbets should access a bigger picture. Three
  gobbets can be completed in the time it takes one
  essay exam question, so that to an extent you can
  assess both coverage and depth.

17
Now you think up a few gobbets in your group

• What are your gobbets?
• What questions are you going to ask your
  students?

18
Letter-to-a-friend

• The student writes to a friend, who is thinking
  of enrolling in the course, about his/her
  experience. The student should reflect on the
  unit and report on it as it affects them. Letters
  at worst are lists of unit content, a rehash of
  the course outline. Good responses provide
  integrated accounts of how the topics fit
  together and form a useful whole, while the best
  show a high degree of reflection, perhaps
  describing a change in personal perspective as a
  result of doing the course.
• Letters-to-a-friend also provide a useful source
  of feedback to the teacher. Letters are about a
  page in length and are written and assessed in a
  few minutes.
• Letters supplement more fine-grained tasks with
  an over-view of the unit.

19
Cloze

• Every seventh (or so) word in a passage is
  deleted. The student has to fill in the space
  with the correct word or good synonym. A text is
  chosen that can only be understood if the topic
  under discussion is understood, rather like the
  gobbet. The omitted words are essential for
  making sense of the passage.

20
A Cloze Example

• It is useful to express ___ by using
  appropriate___. High level ___ would include
  reflect, hypothesise, ___ , generate new
  alternatives. Such would be used to define an A
  or B grade in meeting the ___. Low level___
  such as describe, identify,___ would be more
  frequent in defining C and D grades. What level
  of___is suitable for what___is however highly
  specific to each___taught. Each discipline and
  topic has its own appropriate ___that reflect
  different levels of___. The topic content is
  addressed as the objects the ___ take.

21
Self- and Peer-assessment

• Self/peer assessment can greatly reduce the
  teachers assessment load, even when conventional
  assessments such as exam or assignment are used.
  When posters are used, the assessment can be over
  in one session. The criteria have to be
  absolutely clear. If self/peer assessments agree
  within a specified range, the higher grade is
  best awarded (collusion can be mitigated by
  spot-checking).
• Self/peer assessment is also an important
  learning experience. The student has to learn
  what makes a good performance, then judge own
  work and someone elses work with very likely a
  different take on the topic.

22
Group Assessment

• Requiring a large assessment to be carried out in
  groups of 4, say, instead of individually, cuts
  the teachers assessment time drastically. Some
  issues
• Ideal for assessing social ILOs such as
  teamwork, communication.
• How do you know if a given student has met the
  ILOs in question? Frequently, the work is shared
  so that different students address different
  aspects of the task and hence different ILOs.
• Can be met in part by peer ratings.

23
Random Assessment

• In some situations involving frequent or
  continuing assessment, the accumulation of
  information becomes unmanageable. On the other
  hand, if students know they are not going to be
  assessed they may not participate (e.g. in online
  discussion). In such cases it useful to make
  summative assessments at random.
• Also, frequent summative assessments means that
  each is not worth very much, and students may
  not put in much effort. Random assessment means
  you only count a few but students dont know
  which assessments.

24
Typical ILO Possible Assessment Task

• Acquire content short answer, MCQ, 3-minute
  essay, ordered-outcome
• Explain 3-minute essay, ordered-outcome,
  gobbet
• Integrate ordered-outcome, letter-to-a-friend
• Apply gobbet, poster
• Solve problem set problems, case study
• Design, create poster
• Hypothesise gobbet, design an experiment
• Reflect letter-to-friend
• Teamwork group assessment

25
Devise an assessment strategy for a course you
are teaching.

• List your ILOs
• What verbs do you want to assess?
• Beside each verb list possible ATstasks whose
  performance requires students to use that verb,
  and that enable you to judge how well each ILO
  has been met.
• Now describe each task, the questions to be
  asked, and
• the items you will use.
• Make a start on designing the criteria or rubrics
  for the CityU grading levels (D, C, B, A) as they
  might apply to each task. (See handout).

26
Other Workshops relevant to assessing large
classes

• Achieving and assessing ILOs with online learning
• Group assessment
• Grading
• Suggestions for more workshops on feedback form,
  please!

27
Concluding Comments

• Any assessment, but especially large class, best
  achieved by using several different modes to
  address the ILOs rather than One Big Event
• The invigilated final exam contextclosed-door,
  timedcan be used for many other and better
  formats than written essays. Try gobbets,
  critical incidents (e.g. comment on a video
  segment), ordered outcome, letter-to-a-friend,
  and so on.

28
ILOs for this session

• You have explored some options for suitable large
  class assessment tasks.
• You (hopefully) have designed an assessment
  strategy for at least one course

29
References

• Biggs, J. Teaching for Quality Learning at
  University. Buckingham Open U Press, 2003
  Chapter 6.
• Gibbs, G., Jenkins, A Wisker, G. Assessing More
  Students. Oxford PCFC/Rewley Press, 1992
• http//www.tedi.uq.edu.au/largeclasses/