Portfolio Committee on Basic Education Report on the Quality Assurance of the NSC

1
Portfolio Committee on Basic EducationReport
on the Quality Assurance of the NSC

• 12 February 2013
• Dr Mafu S Rakometsi

2

• WHAT CAN WE LEARN FROM THE NSC RESULTS?
• Introduction to context, principles, approaches
  and processes
• Dr Mafu Rakometsi - CEO of Umalusi

3

• Regulatory Framework
• Quality Assurance of Assessment
• NQF Act Section 27 (h)
• The QC must develop and implement policy
  and criteria for assessment for the
  qualifications on its sub-framework.

4

• Section17 of the GENFETQA Act
• (5) The Council must, with the concurrence of the
  Director-General and
• after consultation with the relevant
  assessment body or education
• institution, approve the publication of the
  results of learners if the
• Council is satisfied that the assessment
  body or education institution
• has
• (i) conducted the assessment free from any
  irregularity that may
• jeopardise the integrity of the assessment
  or its outcomes
• (ii) complied with the requirements prescribed by
  the Council for
• conducting assessments
• (iii) applied the standards prescribed by the
  Council which a learner is
• required to comply with in order to obtain
  a certificate and
• (iv) complied with every other condition
  determined by the Council.

5

• Framework for QA of Learner Achievement
• Based on established and existing practices in
  assessment for certification
• Prescribed components of External assessment
  (examinations) and Site based/ internal /
  continuous assessment
• Use of systems, processes, and procedures to
  evaluate, inspect, monitor and report on
  examination systems, processes and procedures of
  public and private assessment bodies.

6

• Framework for Quality Assurance of Assessment
• Evaluation and /or accreditation of assessment
  bodies
• Periodic inspection of assessment systems
• Ongoing monitoring of assessment systems
• Quality assurance of external examinations
  through
• Moderation of examination question papers
• Monitoring and moderation of SBA
• Monitoring the conduct of examinations
• Moderation of marking
• Standardization of assessment outcomes

7

• Approval for the release of Results
• Approval is based on the following requirements
• The examinations are conducted compliant to the
  applicable policies regulating the conduct and
  administration of the examinations
• At the time of approval, there is no serious
  irregularity which could undermine the
  credibility of the examinations.

8

• Quality Assurance of the DBE 2012 National Senior
  Certificate Examination
• Vijayen Naidoo Sen. Manager Quality Assurance
  of Assessments

9

• Moderation of question papers

• PURPOSE
• To ensure that the question papers are of the
  required standard
• (standard captured in the NCS and SAGs)
• To ensure that the question papers are
  relatively
• - fair
• - reliable
• - representative of an adequate sample of the
  curriculum
• - representative of relevant conceptual domains
• - representative of relevant levels of
  cognitive challenge
• External moderators

10
Moderation of the question papers

• Approach
• Question papers set by panel of examiners DBE
• Internally moderated by DBE
• Externally moderated by Umalusi
• Subsequent moderations and approval.

11
Moderation of Question papers

• Number of NSC 2012 question papers moderate

Number of subjects Number of papers Approved/ Conditionally approved at 1st moderation Approved/ Conditionally approved at 2nd moderation Approved/ Conditionally approved at 3rd moderation 4th moderation and beyond
62 (including the non-official languages and LO) Nov 2012 -132 27.3 (36) 56.8 (75) 15.2(20) 0.7 (1)
Mar 2013 -130 27,7 (36) 54.6 (71) 13.9 (18) 3.8 (5)
12
Moderation of the question papers

• Criteria
• Technical Criteria
• Internal moderation quality of internal
  moderator reports.
• Content coverage
• Cognitive Demand
• Marking Guidelines
• Language and Bias
• Adherence to Policies and guidelines
• Predictability

13
Moderation of the question papers

• Findings Areas of Good Practice
• Percentage of question papers and memoranda
  approved after first and second moderation ( Nov
  2012- 79,6 Mar 13 72,3)
• Simultaneous moderation of final and
  supplementary question papers.
• Standard of Internal moderation

14
Moderation of the question papers

• Findings Areas of Concern
• Adherence to timeframes and impact on quality of
  setting and moderation.
• Question papers requiring more than four
  moderations.
• Role of the external moderator.

15

• Moderation of internal /continuous assessment

• Internal assessment refers to any assessment
  conducted by the provider, the outcomes of which
  count towards the achievement of the
  qualification
• panels of moderators / subject specialists

16

• Moderation of internal assessment (cont.)

• Purpose of Umalusis verification
• To verify the rigour and appropriateness of the
  DBE moderation process linked to DBE plans
• Ascertain the degree to which assessment
  bodies/provinces are attempting to ensure
  standardisation across
• Ascertain the standard and quality of the tasks
• Determine the extent and quality of internal
  moderation and feedback.
• Determine the reliability and validity of the
  assessment outcomes

17

• Moderation of Internal Assessment
• Phase 1 (June/July 2012)- focus on assessment
  instruments (teacher files)

PROVINCIAL EDUCATION DEPARTMENT Subjects
Eastern Cape KwaZulu- Natal Limpopo Mpumalanga Mathematics, Life Sciences, Physical Sciences Accounting, English FAL, History, Geography, Life Orientation,
18

• Moderation of Internal Assessment
• Phase 1 Findings
• Quality and Standard of Assessed tasks
• Content Coverage mostly appropriate
• Cognitive demand and difficulty levels standards
  variable mainly at lower cognitive levels,
• Over reliance on previous question papers.
• Projects and practical investigations- not
  cognitively balanced, more theory than practice.
• Marking tools not applied consistently
• Teachers subject knowledge often inadequate
• Physical Education Task (PET) inappropriately
  assessed.

19

• Moderation of Internal Assessment
• Phase 1 Findings
• Internal Moderation
• Moderation instruments compliance vs. quality
• School moderation seriously neglected
• Monitoring vs. Moderation
• Feedback and Support general support by subject
  advisors. Very little developmental feedback.

20

• Moderation of Internal Assessment
• Phase 1 Areas of Good Practice
• Use of Common Tasks
• Diagnostic Analysis to improve performance
• Compliance with Internal Assessment protocols.

21

• Moderation of Internal Assessment
• Phase 2 (October/November 2012)- focus on learner
  evidence and teacher files.

PROVINCIAL EDUCATION DEPARTMENT Subjects
All 9 PEDs Mathematics, Life Sciences, Physical Sciences Accounting, English FAL, History, Geography, Life Orientation,
22

• Moderation of Internal Assessment
• Phase 2 Findings
• Quality and Standard of Moderation
• Brief unclear focus on marking only and not on
  quality of task.
• Shadow marking vs. moderation
• Marking of the Tasks
• Marking tools not applied consistently
• Teachers subject knowledge often inadequate
• Challenges with marking extended writing and
  research projects.
• Marking too lenient

23

• Moderation of Internal Assessment
• Phase 2 Findings
• Learner performance
• Difficulty with higher-order questions
• Ability to express/respond in English a
  determining factor
• Extremely poor performance in certain subjects
  e.g. Maths, LO Common Assessment Task,
• Evidence suggests that very little effective
  teaching and learning is taking place

24

• Moderation of Internal Assessment
• Phase 2 Findings
• Internal Moderation at School, Cluster and
  District Levels.
• Varying quality of Moderation
• Limited moderation of candidates work shadow
  marking
• Little evidence of moderation of tasks.
• Lack of mark sheets or incorrect recording of
  marks.

25

• Moderation of Internal Assessment
• Phase 2 Areas of Good Practice
• Use of Common Tasks
• Diagnostic Analysis to improve performance
• Compliance with Internal Assessment protocols.
• Evidence of moderation at the various levels.
• Introduction of National Moderation

26

• Monitoring of Examinations
• State of readiness
• Conduct of examinations
• Marking

State of Readiness
Monitored the DBE audit visits (March/April) and evaluation visits to all Comprehensive approach that monitors exam systems (including SBA)
27

• Monitoring of Examinations
• State of readiness
• Findings
• All provinces have examination systems in place
• Vacant posts and use of contract staff is a major
  concern.

28
Monitoring of the writing phase.

• Findings
• General Management of the examinations
• Generally examinations conducted in line with
  policy
• Isolated instances of non-compliance (suitability
  of venue, noise, security, contingency plans)
• Shortage of question papers and answer books
• Provincial Monitoring

No of exam centres No of exam centers monitored by Umalusi No of Umalusi monitors per province
6623 207 36
29
Monitoring of the marking phase
No of marking centers (2010) No of marking centers (2011) No of marking centers (2012) No of marking centers monitored by Umalusi
Total 127 123 118 87
30
Monitoring of the marking phase

• Findings
• In general marking processes in line with policy
• Concern with lack of security at some marking
  centres
• Concern with the appointment and training of
  markers
• Concern with the quality and standard of marking
  and internal moderation.
• Number of marking centres security risk

31

• Verification of marking

• PURPOSE
• Moderation of marking determines the standard and
  quality of marking and ensures that marking is
  conducted in accordance with agreed practices
• Umalusi engages the following during the
  moderation of marking
• 1. Pre-marking/memorandum discussion
  centralised memo discussions recommended - this
  will ensure consistency across marking centres
• 2. Moderation of marking (off-site and
  on-site)

32

• Verification of marking
• Memoranda discussion meetings
• Approved and signed off finalized memoranda/
  marking guidelines for all the NSC subjects
• Concern with non attendance and pre- memo
  discussion preparations
• Concern with timing and scheduling of memo
  discussions.
• Practice of marking and training in the memo
  discussions impacts positively on standard of
  marking.

33
Verification of marking

• Findings
• Findings
• Adherence to Marking memo
• Marking generally accurate problem with marking
  extended writing and essays

Provinces Subjects
Centralised All 9 Gateway subjects 4 others
On-site Eastern Cape Gauteng KZN Limpopo Northern Cape 13 in total
34
Verification of marking

• Quality of marking varied. Question marking
  improved consistency.
• Competency of markers to mark interpretation type
  questions.
• Internal moderation
• No unauthorised changes to marking guidelines
  reported
• Training of chief markers and internal moderators
  at memo discussions having a positive impact on
  marking.

35

• Standardisation and Verification of Resulting
• Provision of GENFETQA Council may adjust raw
  marks.
• International practice large scale assessment
  systems
• Standardisation process used to mitigate the
  effect of factors other than learners knowledge
  and aptitude on the learners performance.
• Sources of variability difficulty in question
  paper, undetected errors, learner interpretation
  of questions

36
Statistical moderation

• Standardisation
• 58 subjects standardised
• Raw marks accepted 41 subjects
• Moderated Upward 5 subjects
• Moderated Downward 12 subjects

37

• Verification of the Resulting Process

Planned Status
Subject structures Subject structures verified.
Candidate registration System verified during state of readiness visits.
Generation of mark Sheets Monitored
Capturing of Marks Monitored
Standardisation data Booklets Data sets received and verified
Capturing of adjustments Verified
Statistical moderation resulting Verified
38
Examination Irregularities

• The majority of irregularities were of a
  technical nature and these were reported to
  Umalusi according to the established channels.
• Some irregularities were as a result of
  registration-related problems, e.g. candidates
  nor appearing on mark sheets, some registered for
  incorrect subjects.
• Umalusi represented on NEIC
• Major irregularities
• Alleged leakage of Question Papers in KZN

39

• WHAT CAN WE LEARN FROM THE NSC RESULTS?
• STANDARDISATION DECISIONS
• DBE NSC 2012

40
SUBJECT ADJUSTMENT
Life Sciences CA( 0 at 0 scale to -8 at 23. scale to 0 at 70, scale to 3 at 119, scale to 0 at 171, scale to -7 at 239, scale to -1 at 291, scale to -7 at 300)
Physical Sciences 0 at 0, scale to -12 at 102, block -12 from 102 to 252, scale to 0 at 300
Mathematics Raw
Mathematical Literacy 0 at 0, raw to 114, scale 0 to 9 from 114 to 240, scale to 0 at 300
Mathematics Probability Data Handling Raw

41
SUBJECT ADJUSTMENT
Geography 0 at 0, scale to -6 at 96, block -6 from 96 to 180, scale to 0 at 240, raw to 300
Life Orientation Raw
Hospitality Studies Raw
Tourism ½ CA( 0 at 0, scale to -12 at 121, scale to 0 at 300)
Agricultural Science Raw
Consumer Studies Raw

42
SUBJECT ADJUSTMENT
Agricultural Management Practices Raw
Agricultural Technology Raw
Music Raw
Dance Studies Raw
Design 0 at 0, scale to -12 at 102, scale to -6 at 246, scale to 0 at 300
Dramatic Arts ½ CA(o at o, scale to -15 at 66, scale to 0 at 300)

43
SUBJECT ADJUSTMENT
Visual Arts Raw
Religion Studies Raw
Information Technology Raw
Computer Applications Technology Raw
Civil Technology 0 at 0, scale to -9 at 90, block -9 from 90 to 180, scale to 0 at 300
Electrical Technology ½ CA(0 at 0, scale to -15 at 211, scale to 0 at 300)

44
SUBJECT ADJUSTMENT
Mechanical Technology Raw
Engineering Graphics and Design Raw
Accounting 0 at 0, scale to -9 at 78, block -9 from 78 to 222, scale to 0 at 300
Business Studies Raw
Economics Raw

45
SUBJECT ADJUSTMENT
History Raw
Afrikaans HL 0 at 0, scale to 4 at 116, block of 4 from 116 to 247, scale to 0 at 300.
Afrikaans FAL Raw
Afrikaans SAL Raw
English HL 0 at 0, scale to 8 at 112, block of 8 from 112 to 240, scale to 0 at 300.

46
SUBJECT ADJUSTMENT
English FAL 0 at 0, scale to -6 at 51, block of -6 from 51 to 189, scale to 0 at 300.
English SAL Raw
IsiNdebele HL Raw
IsiNdebele FAL Raw
IsiZulu HL 0 at 0, scale to 8 at 112, block of 8 from 112 to 232, scale to 0 at 300.
IsiZulu FAL Raw
IsiZulu SAL Raw

47
SUBJECT ADJUSTMENT
Setswana HL Raw
Setswana FAL Raw
Setswana SAL Raw
Siswati HL Raw
Siswati FAL Raw
Siswati SAL Raw
IsiXhosa HL Raw
IsiXhosa FAL Raw
IsiXhosa SAL Raw

48
SUBJECT ADJUSTMENT
Xitsonga HL ½ CA (0 at 0, scale to -13 at 74, scale to 0 at 300)
Xitsonga FAL Raw
Tshivenda HL ½ CA(0 at 0, scale to -12 at 115, block -12 from 115 to 153, scale to 0 at 300)
Tshivenda FAL Raw

49
SUBJECT ADJUSTMENT
Sepedi HL Raw
Sepedi FAL Raw
Sesotho HL Raw
Sesotho FAL ½ CA(0 at 0, scale to -15 at 300)
Sesotho SAL Raw

50

• Quality Assurance of the IEB 2012 National Senior
  Certificate Examination
• Vijayen Naidoo Sen. Manager Quality Assurance
  of Assessments

51
Moderation of Question papers

• Number of IEB NSC 2012 question papers moderated

Number of subjects Number of papers Approved/ Conditionally approved at 1st moderation Approved/ Conditionally approved at 2nd moderation Approved/ Conditionally approved at 3rd moderation 4th moderation and beyond
39 Nov 2012 -69 39.7 (28) 54.4 (37) 4.4(3) 1.5 (1)
52
Moderation of the question papers

• Findings Areas of Good Practice
• Percentage of question papers and memoranda
  approved after first and second moderation ( Nov
  2012- 94.1 )
• Standard of Internal moderation

53

• Moderation of Internal Assessment
• Phase 2 (November 2012)- focus on learner
  evidence and teacher files.

IEB Subjects
Mathematics, Maths Lit, Life Sciences, Physical Sciences Accounting, English HL, History, Life Orientation, Business Studies, Economics
54

• Moderation of Internal Assessment
• Findings Areas of Good Practice
• Regional and national moderation is in place and
  well executed
• The marking of tasks of a highly acceptable
  standard
• Findings Areas of Concern
• Feedback to the learners
• Assessment of Physical Education Tasks in LO
  distance education learners

55
Monitoring of the writing phase.

• Findings
• General Management of the examinations
• Generally examinations conducted in line with
  policy
• Isolated instances of non-compliance (candidate
  identification)
• Lack of Irregularity Registers
• Assessment Body Monitoring

56
Verification of marking

• 8 Subjects
• Findings
• marking and moderation of scripts is extremely
  thorough
• Marking generally accurate
• practice of double marking is highly commendable
• Moderation doesnt always involve full moderation
  of a script

57
Statistical moderation

• Standardisation
• 58 subjects standardised
• Raw marks accepted 44 subjects
• Moderated Upward 9 subjects
• Moderated Downward 4 subjects

58

• WHAT CAN WE LEARN FROM THE NSC RESULTS?
• STANDARDISATION DECISIONS
• IEB NSC 2012

59
SUBJECT ADJUSTMENT
Accounting 0 at 0, scale to -6 at 96 block -6 from 96 to 212, scale to -5 at 245, back to 0 at 300)
Business Studies ½ CA(0 at 0, scale to -15 at 139, block of -15 from 139 to 208, scale back to 0 at 300)
Computer Application Technology ½ CA (Raw from 0 to 101, scaled from 0 at 101 to 7 at 219, scaled to 0 at 300)
Consumer Studies Raw
Design Raw
Dance Studies Raw

60
SUBJECT ADJUSTMENT
Dramatic Arts Raw
Economics ½ CA(0 at 0, scaled to 10 at 45, block 10 from 45 to 102, scaled to 0 at 300)
Engineering Graphics and Design ½ CA ( 0 at 0, scale to 6 at 41, block of 6 from 41 to 79, scale to 0 at 300)
Geography CA(0 at 0, scale to 12 at 36, scale to -7 at 59,scale to -1 at 121, scale to 8 at 173, block 8 from 173 to 200, scale to 0 at 300)

61
SUBJECT ADJUSTMENT
History CA (0 at 0, scale to -18 at 56, scaled to -25 at 109, scaled to 0 at 300
Information Technology Raw
Hospitality Studies Raw
Tourism Raw
Visual Arts ½ CA ( 0 at 0, scale to -15 at 89, block -15 from 89 to 222, scale to 0 at 300)
Music Raw

62
SUBJECT ADJUSTMENT
Religion Studies Raw
Life Orientation Raw
Life Sciences Raw
Mathematical Literacy Raw
Mathematics Raw
Mathematics P3 Raw

63
SUBJECT ADJUSTMENT
Physical Science ½ CA ( 0 at 0 scale to -7 at 28, scale to 0 at 68, block 1 from68 to 74, scale to 0 at110, scale to -7 at 241, scale to 0 at 300)
English FAL ½ CA (0 at 0, scale to 15 at64, scale to 6 at 221, scale to 10 at 258, scale to 0 at 300)
English HL Raw
Afrikaans FAL ½ CA (0 at 0, scale to -10 at 147, block of -10 from 147 to 175, scale back to 0 at 300)

64
SUBJECT ADJUSTMENT
Afrikaans HL Raw
Sepedi FAL Raw
Sesotho FAL Raw
Setswana FAL Raw
Siswati FAL Raw
Siswati HL Raw
IsiXhosa FAL Raw
IsiZulu HL Raw

65
SUBJECT ADJUSTMENT
IsiZulu FAL CA( 0 at 0, scale to -16 at 146, scale to -6 at 166, block -10 from 166 to 242, scale to 0 at 300)
IsiZulu SAL Raw
Arabic SAL Raw
French SAL ½ CA( 0 at 0, scale to -13 at 51, scale to -1 at 174, scale to -5 at 206, block -5 from 206 to 279, scale to 0 at 300)
German HL Raw
German SAL Raw

66
SUBJECT ADJUSTMENT
Gujurati SAL Raw
Hebrew SAL 0 at 0, raw to 180, scale to -6 at 246, scale to 0 at 300
Hindi FAL Raw
Hindi SAL Raw
Italian SAL Raw
Latin SAL Raw

67
SUBJECT ADJUSTMENT
Modern Greek SAL Raw
Portuguese HL Raw
Portuguese SAL Raw
Spanish SAL Raw
Tamil SAL Raw
Urdu FAL Raw
Urdu SAL Raw
Serbian HL Raw

68
SUBJECT ADJUSTMENT
Equine Studies Raw
Sport and Exercise Science Raw
Maritime Economics Raw
Nautical Science Raw
Advanced Programme Mathematics ½ CA( Block 5)
Advanced Programme English Raw

69
Annexure to Approval Letter to DBE

• Specific concerns regarding the quality assurance
  of the NSC examination and assessment
• Non-adherence to timeframes for the submission of
  question papers must be addressed. It is
  important to note that for the credibility of the
  NSC examination it is vitally important that
  every effort is made to adhere to agreed
  deadlines for setting and moderation of question
  papers. Any delays affect Umalusi in its quality
  assurance exercise.
• In addition to what is said above, every effort
  should be made to set question papers of an
  appropriate quality and standard so that they can
  be ideally approved at first moderation.

70

• Low levels of performance in key subjects. This
  issue need to be looked into and addressed so as
  not to compromise the standard of the NSC.
• Umalusi is concerned with the problem in Life
  Orientation with awarding of marks at the 80
  level where there is clear evidence of tampering
  with the marks. The DBE is urged to send a strong
  message of warning to the provinces indicating
  that should this problem persist in the future
  all learners at the 80 mark will be adjusted
  downward.

71

•  The quality of School-Based Assessment (SBA)
  continues to be a concern. There may be several
  plausible reasons for this poor quality of SBA,
  but it is clear that there is too little control
  over work that is done.
• It is evident from our reports that progress has
  been made over the years with regard to the
  standard of marking. There is however room for
  improvement in the selection and training of
  markers, chief markers, senior markers and
  internal moderators. Competency tests will go a
  long way in the identification of competent
  markers.

72
Conclusion

• The findings of the quality assurance processes
  are a clear indication of a maturing system that
  has, on the one hand, made positive strides
  towards improvement in certain areas of
  assessment and examination, but, on the other
  hand, still has a few challenges that need to be
  addressed.
• The quality assurance of each of these processes
  presented above was conducted based on Umalusi
  criteria. Umalusi uses criteria that are
  subjected to constant review and refinement, to
  ensure that they are in line with current trends
  in assessment and examinations.

73
Conclusion

• In general Umalusi is pleased with the manner in
  which the 2012 NSC examination was administered.
• Umalusi acknowledges that a number of technical
  irregularities were reported, but these were
  addressed in a fitting manner.
• Umalusi takes this opportunity to express
  appreciation to the national provincial
  departments of education for their concerted
  effort in ensuring a credible examination.
• Umalusi expresses appreciation also to all the
  relevant stakeholders for the necessary support
  given in line with Umalusi quality assurance
  initiatives.

74

• Thank you!