PowerPoint title

1
Its Judging the Evidencethat Counts
Ayesha Ahmed Alastair Pollitt Cambridge Exam
Research www.camexam.co.uk
2
Causes of difficulty

• Concept difficulty
• The difficulty of the ideas
• - complexity, abstractness, etc
• Process difficulty
• Name, describe, explain, justify, etc
• Presentation difficulty
• Language, graphics, layout, cues, etc

3
Validity
An exam question can only contribute to valid
assessment

• if the students minds are doing the things
• we want them to show us they can do

and if we give credit for, and only for, evidence
that shows us they can do it.
4
The trait - Geography
5
The trait - History
6
Outcome Space
Poor Good
Desired Outcome Space
The Desired Outcome Space is the range of
responses we want to see, as evidence of good and
poor understanding
7
Outcome Space
Poor Good
Desired Outcome Space
8
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
9
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
10
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
11
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
12
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
13
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
14
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
15
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
16
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
17
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
18
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
19
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
20
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
21
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
22
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
23
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
24
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
25
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
26
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
27
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
28
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
29
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
30
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
31
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
32
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
33
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
34
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
35
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
36
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
37
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
38
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
39
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
40
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
41
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
42
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
43
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
44
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
45
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
46
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
47
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
48
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
49
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
50
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
51
Outcome Space
Poor Good
Desired Outcome Space
Observed Outcome Space
52
Outcome Space
Poor Good
53
Validity
An exam question can only contribute to valid
assessment

• if the students minds are doing the things
• we want them to show us they can do

and if we give credit for, and only for, evidence
that shows us they can do it.
54
Outcome Space Control Assessment

• What does it mean to be good or poor?

Evidence for discriminating
Idea of a task
Desired outcome space
Mark scheme
Question
55
Classifying Questions

• Explorations
• Projects
• Unstructured essays
• Structured essays
• Structured questions
• Short answer questions
• Cloze/completion
• Selected response
• Multiple Choice

Unconstrained
Semi-Constrained
Very Constrained
56
Mark Schemes forUnconstrained questions

• UC Judging the quality
• UC.0 Model answer
• UC.1 Holistic implicit levels
• UC.2 Multiple levels
• - implicit
• - explicit
• UC.3 Specific trait interpretation

57
UC.0 Model answer

• 6. How convincing are the arguments in favour of
  granting independence to the Bank Of England?
  (June, 1995)

GENERAL

• Pennant-Rea (1993) argued that there was a link
  between the objective of eliminating inflation
  from the UK psyche and Bank independence.
  Consistency in the pursuit of targets would
  increase the credibility of policy-making (thus
  impressing financial markets and reducing the
  risk of exchange rate fluctuations) the
  inflation expectations of wage and price setters
  would be influenced in a downwards direction, and
  low inflation would be achieved without much
  output and employment loss in the longer run,
  interest rates would fall, thus boosting growth
  and employment. Inflation targets could be set by
  parliament, thus maintaining democratic
  accountability.
• The eventual establishment of a European Central
  Bank will require all national central banks to
  be granted independence (or, at least, greater
  autonomy). National governments will no longer
  have unlimited access to central bank credit,
  while there are already restrictions (Maastricht
  convergence criteria) on the size of national
  budget deficits and Debts.
• ARGUMENTS AGAINST
• The Bank might be left with too many functions -
  regulation of the banking system, note issue,
  government banker, operator in the foreign
  exchange market, etc. Policy conflict and policy
  overload are dangers.
• Policy effectiveness and democratic
  accountability are potentially at odds with each
  other. What inflation target should be set, and
  by whom? How frequently reviewed? How much
  attention should be given to other economic
  objectives? Can a Governor with a fixed-term
  contract really act independently?
  (Quasi-independence in Japan, for example,
  contributed to that country's worst ever
  recession, and persistent recriminations between
  the Ministry of Finance and the Bank of Japan).
• The arrangement might jeopardise the
  co-ordination of monetary and fiscal policy (a
  problem in the USA since the early 1980s, and in
  Germany since 1989). The Bank might blame the
  government's high PSBR for undermining monetary
  policy the government could claim that the PSBR
  was the outcome of monetary policy that was too
  tight. The government might over-compensate for
  restrictive monetary policy with expansionary
  fiscal policy.
• Much controversial monetary theory would provide
  the basis for policy decisions, which would rest
  on assumptions about the relationships between
  the money stock, interest rates, aggregate demand
  and nominal income. The long-run inflation -
  unemployment trade-off is a subject of much
  dispute. Efforts to estimate the size of the
  output gap (as a basis for forecasting future
  inflation) have not always been successful.
• Some critics argue that the Bank already has an
  unhealthy preoccupation with inflation targets,
  and has regularly over-estimated the risks to
  inflation. There is a danger of 'monetary
  overkill'. What of the 'real economy'? (output
  and jobs conditions for growth). Not all agree
  with Barro's contention (B of E Quarterly, May
  1995) that high inflation reduces long-run growth
  rates. The assumption that the sole purpose of
  monetary policy is inflation control is
  questionable..

• Candidates should outline the role of the Bank
  and establish the extent to which it is already
  effectively independent. Both sides of the
  argument should be reviewed, and credit should be
  given for introducing appropriate theoretical
  perspectives. International comparisons should be
  rewarded. The best answers will be characterised
  by evaluation - i.e. much depends upon policy
  priorities and upon the constitutional
  relationship between parliament and an
  independent bank.
• The present formal position is that the
  government decides on its fiscal stance, and then
  instructs its agent, the Bank of England, to
  undertake the corresponding monetary policy the
  Bank advises on the timing of interest rate
  changes. Since leaving the ERM, the emphasis has
  been on meeting an inflation target (currently
  1-21/2 for annual RPIX change), and there has
  been a shift towards transparency in
  policy-making. The Bank has produced a quarterly
  Inflation Report (not vetted by the Treasury)
  since February 1993, while minutes of the monthly
  meetings between Clarke and George have been
  published since April 1994. Interest rate policy
  has been discussed more openly, with 1/2
  increases in rates announced in September 1994,
  December 1994, and February 1995. Relations
  appeared to cool in May 1995, when George's call
  for a further rise (in the light of an apparently
  deteriorating inflation outlook) was rejected by
  Clarke.
• This change of approach has led some
  commentators to suggest that the Bank is already
  quasi-independent. The term 'independent' is
  itself ambiguous, as it leaves open the
  possibility that parliament will continue to set
  inflation targets, leaving the Bank to set the
  interest (or discount) rate unilaterally much
  would depend on the constitutional
  responsibilities granted to the Bank, and on the
  powers and duties of its governors. Blair (Mais
  Lecture, May 1995) called for the establishment
  of a monetary committee that would formulate
  inflation policy advice for the Chancellor, the
  extent to which the Bank was free to set interest
  rates would depend upon structural reform and
  upon its track record. Major has ruled out
  central bank independence, though three former
  Conservative Chancellors (Howe, Lawson, Lamont)
  appear to favour it.
• ARGUMENTS FOR
• A growing number of central banks (e.g. New
  Zealand, Canada, Japan) have become independent,
  and the track record in fighting inflation is
  generally impressive. (This may simply reflect
  the fact that countries with an independent Bank
  are those which value low inflation most.) The
  Bundesbank has a statutory obligation to pursue
  price stability, and its record is impressive
  (though not impeccable - M3 has been above its
  target range in 6 of the last 15 years).
• A single-minded objective of low inflation would
  introduce clarity into policy-making, and require
  fiscal policy to be consistent with monetary
  policy. Interest rate decisions would not be
  determined by short-term political considerations
  (e.g. pre-election cuts). Thus an independent
  Bank is more likely to be committed to long-run
  inflation control than are politicians.

58
UC.0 Model answer

• 6. How convincing are the arguments in favour of
  granting independence to the Bank Of England?
  (June, 1995)

Since leaving the ERM, the emphasis has been
on meeting an inflation target (currently 1-21/2
for annual RPIX change), and there has been a
shift towards transparency in policy-making. The
Bank has produced a quarterly Inflation Report
(not vetted by the Treasury) since February 1993,
while minutes of the monthly meetings between
Clarke and George have been published since April
1994. Interest rate policy has been discussed
more openly, with 1/2 increases in rates
announced in September 1994, December 1994, and
February 1995. Relations appeared to cool in May
1995, when George's call for a further rise (in
the light of an apparently deteriorating
inflation outlook) was rejected by Clarke.
59
UC.0 Model answer

• 6. How convincing are the arguments in favour of
  granting independence to the Bank Of England?
  (June, 1995)

Blair (Mais Lecture, May 1995) called for the
establishment of a monetary committee that would
formulate inflation policy advice for the
Chancellor,
Not all agree with Barro's contention (B of E
Quarterly, May 1995) that high inflation reduces
long-run growth rates.
60
UC.1 Holistic implicit levels
Levels of response mark scheme. Work upwards from
lowest level. Level 1 Choice of case study
applied reasonably well. Gives simple description
or explanation. Information is communicated by
brief statements. 1/2 marks Level 2 Choice of
case study applied well. Gives descriptive points
in more detail but little explanation.
Communication begins to show structure with
occasional use of specialist terms. Sentences
show some coherence but occasional errors in
spelling, punctuation and grammar. 3/4
marks Level 3 Appropriate choice of case study
applied well. Provides a balanced account which
gives descriptive detailed points with some
explanation. Communication has structure with
some use of specialist terms. Coherent sentences
with few errors in spelling, punctuation and
grammar 5/6 marks Level 4 Appropriate
choice of case study applied very well. Provides
a balanced account which includes specific
description and explanation. Communication is
logical and includes specialist terms. Spelling,
punctuation and grammar have considerable
accuracy. 7/8 marks Total 30
61
UC.1 Holistic implicit levels
Level 2 Choice of case study applied well. Gives
descriptive points in more detail but little
explanation. Communication begins to show
structure with occasional use of specialist
terms. Sentences show some coherence but
occasional errors in spelling, punctuation and
grammar. 3/4 marks
Level 2 Choice of case study applied well. Gives
descriptive points in more detail but little
explanation. Communication begins to show
structure with occasional use of specialist
terms. Sentences show some coherence but
occasional errors in spelling, punctuation and
grammar. 3/4 marks
62
UC.2 Multiple levels - implicit

• Explain the formation of a corrie and its lake.
• Add labels to the diagrams below to illustrate
  your answer. (6)

63
UC.2 Multiple levels - implicit
64
UC.2 Multiple levels - explicit

• (b) Discuss whether penetration pricing would be
  the most appropriate pricing strategy for CP to
  use for its new conference business customers. (8)

65
UC.2 Multiple levels - explicit
MS Possible areas for discussion For Against

66
UC.3 Specific trait interpretation

• (v) Cliff recession causes many problems for
  people who live in coastal areas.
• Choose a case study of a stretch of coastline or
  coastal area
• that is suffering from cliff recession.
• Chosen stretch of coastline or coastal area
  ..
• Explain the causes and effects of cliff
  recession in this area. (5)

67
UC.3 Specific trait interpretation
MS Levels mark Walton case study or other.
Section 1A of appendix Do not credit
management
68
Mark Schemes forVery Constrained questions

• VC Correct wrong answers
• VC.0 No guidance / Model answer
• VC.1 Complete list of right answers
• VC.2 List of right list of wrong
• - Examples
• - Complete
• VC.3 Rule/principle to differentiate answers

69
VC.0 No guidance/Model answer

• (g) Complete the following sentence about private
  sector businesses
• The capital of a private business is contributed
  by.
• ..... 1

MS (g) The capital of a private business is
contributed by the owners/shareholders.
70
VC.0 No guidance/Model answer

• (i) Name a hardwood commonly used in the
  manufacture of childrens toys. 1

MS Hardwood beech. 1
71
VC.1 Complete list ofright answers

• (iii) The location from which Photograph B was
  taken is shown on Figure 1a. In which direction
  was the camera pointing?

• MS
• Point mark
• south south west south south west S SW SSW
  (1)

72
Outcome Space and Multiple Choice

• Q But the outcome space is just A, B, C, D.
  Whats interesting in that?

A Thats the observed OS think about the
desired OS. What errors do you want poorer
students to make? What errors would
differentiate poorer from better students? These
errors should identify those who dont understand
the science, not those who dont understand the
question. These errors should become the
distractors.
73
VC.2 List of right list of wrong -
Examples or Complete
(ii) Name two other permanent joints which could
be used for the corner joint of the wooden frame.
1 .............................................
...................................... 2 ........
..................................................
......................... (2)
MS (ii) Mitre/dovetail/comb/finger/glue and
screw/glue and nail/glue and pin/rebate/butt/bis
cuit/dowel 2 x 1(2) (Do not accept KD
fittings/mortice and tenon) (do not accept just
glue / screw / nail / pin)
74
VC.3 Rule/principle todifferentiate answers

• On the above chart, label the break even point.
• (ii) How many garments does Badge Identity Ltd
  need to sell in order to breakeven?

75
VC.3 Rule/principle todifferentiate answers
76
VC.3 Rule/principle todifferentiate answers

• On the above chart, label the break even point.
• How many garments does Badge Identity Ltd need to
  sell in order to breakeven?
• MS
• (ii) 3000 (allow 3000-3200) 1

77
VC.3 Rule/principle todifferentiate answers

• (a) Draw a structural formula for octane.

78
(No Transcript)
79
VC.3 Rule/principle todifferentiate answers

• Draw a structural formula for octane.
• MS
• Structure correctly drawn with no bonds missing
  between carbon atoms.
• Allow 1 missing carbon to hydrogen bond or 1
  hydrogen missing.
• Correct shortened structural formula acceptable.

80
CH3CH2CH2CH(CH3)CH2CH2CH3
CH3CH2CH2CH2CH2CH2CH2CH3
CH3CH2CH2C(CH3)2CH2CH3
81
Mark Schemes forSemi-Constrained questions

• SC Points for content and/or quality
• SC.0 No guidance / Model answer
• SC.1 List of good responses - Examples
• - Complete
• SC.2 List of good list of poor
• - Examples
• - Complete
• SC.3 Rule/principle to differentiate responses

82
SC.0 No guidance/Model answer

• (i) What is a quality circle? (1)

• MS
• (i) a group of employees that meets to identify
  quality problems, thinks of solutions and makes
  recommendations

83
SC.1 List of good responses - Examples or
Complete
1 a iv) This valley has been created by a
glacier, which has changed the shape of the
land by a process known as glacial
abrasion. Explain in detail how this process
works. 4
MS Ice contains rocks (1) source of rock (1)
glacier moves (1) gravity (1) fragments scrape
land (1) striations cut (1) surface smoothed
(1) analogy (1) process continues through time
(1) (Max 4)
84
SC.1 List of good responses - Examples or
Complete
1 a iv) This valley has been created by a
glacier, which has changed the shape of the
land by a process known as glacial
abrasion. Explain in detail how this process
works. 4
Does this response deserve 4 marks?
Gravity makes the glacier move slowly, and
over time it smooths the surface.
Gravity makes the glacier move slowly, and
over time it smooths the surface.
85
SC.2 List of good list of poor - Examples
or Complete

• (ii) Describe how the handle could be made more
  comfortable to hold.
• .................................................
  ...........................................
• .................................................
  ........................................... (1)

MS (ii) Round off the ends/edges 1
Padding, fabric, foam, rubber etc NO mark
86
SC.3 Rule/principle todifferentiate responses

• (i) What is meant by a renewable source of
  energy? (1)

MS Credit a simple statement. Bottom line of
doesnt run out. No credit for
exemplification. 1
ii Calais has a warmer winter and a cooler summer
than Wroclaw. Explain why.
MS Looking for answers related to distance from
the sea therefore latitude is not credited. Land
heats up quicker than sea (1) A clear
distinction between land and sea heating. (2) (3)

87
OSCA worked example
In Business/Economics
88
a) Using an example, explain the meaning of the
term public sector. (4)
89
a) Using an example, explain the meaning of the
term public sector. (4)
90
Issues

• Is the mark scheme taxonomy useful? Do the
  levels 0-3 describe important properties of mark
  schemes?
• Are the six elements of a good mark scheme right?
• Does the multiple choice format give us better or
  poorer evidence of achievement?
• Should we separate QMS writing from paper
  production?
• It should improve the quality of QMS - but at a
  price.
• 5 OSCA emphasises evidence as the most important
  aspect of assessment. Is this helpful?

91
Components of a good Mark Scheme
A Good answer B Key idea C List of good
responses D Example of poor response, or
discussion of the boundary between good and
poor. E Scoring rule F Question-specific
interpretation of the trait
92
Contact us at camexam
ayesha_at_camexam.co.uk alastair_at_camexam.co.uk
93
An idea for a question (GCSE DT) Why is a new
material like carbon fibre used in making
something like a bicycle?
Key idea?
Evidence of Good learning? Evidence of Poor
learning?