Portfolio%20Committee%20on%20Basic%20Education%20Report%20on%20the%20Quality%20Assurance%20of%20the%20NSC - PowerPoint PPT Presentation

View by Category
About This Presentation
Title:

Portfolio%20Committee%20on%20Basic%20Education%20Report%20on%20the%20Quality%20Assurance%20of%20the%20NSC

Description:

Portfolio Committee on Basic Education Report on the Quality Assurance of the NSC 19 January 2010 Dr Mafu S Rakometsi ... – PowerPoint PPT presentation

Number of Views:93
Avg rating:3.0/5.0
Slides: 67
Provided by: amaz211
Learn more at: http://pmg-assets.s3-website-eu-west-1.amazonaws.com
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Portfolio%20Committee%20on%20Basic%20Education%20Report%20on%20the%20Quality%20Assurance%20of%20the%20NSC


1
Portfolio Committee on Basic Education Report
on the Quality Assurance of the NSC
  • 19 January 2010
  • Dr Mafu S Rakometsi

2
  • WHAT CAN WE LEARN FROM THE NSC RESULTS?
  • Introduction to context, principles, approaches
    and processes
  • Dr Mafu Rakometsi - CEO of Umalusi

3
  • Regulatory Framework
  • Quality Assurance of Assessment
  • NQF Act Section 27 (h)
  • The QC must develop and implement policy
    and criteria for assessment for the
    qualifications on its sub-framework.

4
  • Section17 of the GENFETQA Act
  • (5) The Council must, with the concurrence of the
    Director-General and
  • after consultation with the relevant
    assessment body or education
  • institution, approve the publication of the
    results of learners if the
  • Council is satisfied that the assessment
    body or education institution
  • has
  • (i) conducted the assessment free from any
    irregularity that may
  • jeopardise the integrity of the assessment
    or its outcomes
  • (ii) complied with the requirements prescribed by
    the Council for
  • conducting assessments
  • (iii) applied the standards prescribed by the
    Council which a learner is
  • required to comply with in order to obtain
    a certificate and
  • (iv) complied with every other condition
    determined by the Council.

5
  • Framework for QA of Learner Achievement
  • Based on established and existing practices in
    assessment for certification
  • Prescribed components of External assessment
    (examinations) and Site based/ internal /
    continuous assessment
  • Use of systems, processes, and procedures to
    evaluate, inspect, monitor and report on
    examination systems, processes and procedures of
    public and private assessment bodies.

6
  • Framework for Quality Assurance of Assessment
  • Evaluation and /or accreditation of assessment
    bodies
  • Periodic inspection of assessment systems
  • Ongoing monitoring of assessment systems
  • Quality assurance of external examinations
    through
  • Moderation of examination question papers
  • Monitoring and moderation of SBA
  • Monitoring the conduct of examinations
  • Moderation of marking
  • Standardization of assessment outcomes

7
  • Approval of Results
  • Compliance with minimum requirements
  • Shift from Moderation to Verification (Moderation
    for verification)
  • Emphasis on Assessment Body reports
  • The monitoring of the conduct of examinations and
    the moderation of marking as well as the
    monitoring of the implementation of internal
    assessment and the moderation of internal
    assessment is undertaken by Umalusi to verify the
    veracity of the assessment body reports in
    respect of these process.
  • Broadened definition of irregularities (including
    assessment system processes)

8
  • WHAT CAN WE LEARN FROM THE NSC RESULTS?
  • Standardisation
  • Dr Sizwe Mabizela - Chair of Umalusi Council

9
  • Why Umalusi standardises results, and how
  • Provision of GENFETQA Council may adjust raw
    marks.
  • International practice large scale assessment
    systems
  • Standardisation process used to mitigate the
    effect of factors other than learners knowledge
    and aptitude on the learners performance.
  • Sources of variability difficulty in question
    paper, undetected errors, learner interpretation
    of questions

10
  • Why Umalusi standardises results, and how
  • Assumptions for large populations the
    distribution of aptitude and intelligence does
    not change appreciably
  • Process of standardisation
  • Moderation of question papers
  • Review of learner performance against historical
    performance of candidates in each subject.
  • Statistical moderation of Internal assessment

11
  • Why Umalusi standardises results, and how
  • Qualitative input
  • Reports (moderator, Chief marker and Internal
    Moderator)
  • Umalusi research (post exam analysis)
  • Responsibility of Assessment Standards Committee
  • Committee of Council
  • Responsible for setting and maintaining standards
  • Observers (SAQA, HESA, Teacher Unions)

12
  • General principles applied in the standardisation
    of examination marks
  • 1. In general no adjustment should exceed 10 or
    the historical average.
  • 2. Adjustments in excess of 10 could be
    considered at the upper end to increase the
    number of distinctions in a subject.
  • 3. In the case of the individual candidate, the
    adjustment effected should not exceed 50of the
    raw mark obtained by the candidate.
  • 4. If the distribution of the raw marks is
    below the historical average, the marks may be
    adjusted upwards subject to the limitations.

13
  • General principles applied in the standardisation
    of examination marks (cont.)
  • 5. If the distribution of the raw marks is above
    the historical average, the marks may be adjusted
    downwards subject to the limitations.
  • 6. The computer adjusted mark is calculated
    based on these principles.
  • 7. For those subjects with a practical
    component of 50, raw marks could be accepted.
  • 8. Umalusi however retains the right to amend
    these principles where and when deemed to be
    necessary based on sound educational principles.

14
  • Reasons why Umalusi does not disclose
    standardisation decisions
  • Process is complex and context driven
  • Sensitive information
  • Could lead to varied erroneous interpretations
  • Prejudicial to learners
  • International practice
  • Credibility of the process of arriving at the
    decision rather than the decision itself.

15
  • Standardisation Decisions for 2010
  • Way forward
  • Public interest vs. international practice
  • Composite report to be tabled with Portfolio
    Committee
  • Composite report to be made public

16
  • The Class of 2010
  • Third cohort of candidates to write the new
    National Senior Certificate (NSC)
  • Completely new qualification based on the
    National Curriculum Statement
  • Some 642 001 enrolled including (full time and
    part-time candidates)
  • Approximately 67,8 of all those who enrolled as
    full time and who wrote have met the requirements
    of a pass

17
Some criteria for comparison
  • The number of candidates passing
  • The quality of the results
  • The standardisation of the exam
  • The predictive quality of the exam for
    performance at HE level

18
The number of candidates passing
  • Massification is a natural outcome of an
    education system that is non-racist, non-sexist
    and democratic
  • It is the clear that there has been a steady
    increase over the last decade in the number of
    learners who enrolled for and wrote the SC and
    then NSC notable exception 2010
  • The underlying philosophy of the NCS had been
    designed to ensure that most people achieve the
    minimum requirement for a pass
  • The NSC determines a pass or fail by 40 in at
    least three subjects and 30 in three others

19
The number of candidates passing Trends in
SC/NSC enrolment and passes
Senior Certificate examination results, all
schools, 1994-2008
Year No. of Candidates No. of Candidates Passes Passes
Year Number Wrote Change in number () Number Change in passes ()
2002 471 309 5 324 752 17
2003 440 267 -7 322 492 -1
2004 467 985 6 330 717 3
2005 508 363 9 347 184 5
2006 528 525 4 351 503 1
2007 564 775 7 368 217 5
2008 533 561 -6 333 604 -9
2009 552 073 3.5 334718 0.33
2010 537 543 -2.7 364 513 8
Sources DoE, 2004, 2005a, 2005b, 2005c, 2007,
2008, 2009
20
SC/ NSC examination results and HE
SC/ NSC examination results, all schools,
1994-2010
Year No. of Candidates Passes Passes University Exemption University Exemption
Year No. of Candidates Number Rate () Number Rate ()
2004 467 985 330 717 71 85 117 18
2005 508 363 347 184 68 86 531 17
2006 528 525 351 503 66 85 830 16
2007 564 775 368 217 65 85 454 15
2008 533 561 333 604 63 107 642 20
2009 552 073 334 718 60 109 697 20
2010 537 543 364 513 67.8 126 371 23.5
Sources DoE, 2004, 2005a, 2005b, 2005c, 2007,
2008, 2009
21
The quality of the results
  • Concerns about quality has become a thorny issue
    and the subject of much debate albeit uninformed
    at most times.
  • SA public uses the matric results as the main
    indicators of quality of the education system and
    so there is an understandable concern about what
    these results of a relatively new qualification
    is saying about the current state of education
  • It is important however that we interrogate some
    of our assumptions that inform our beliefs about
    standards and quality

22
Assumptions that affect our perceptions
  • The first assumption is that more means less or
    more means worse
  • Stellenbosch University Website
  • In the light of phasing out of matriculation
    exemption and of the distinction between subjects
    on the higher and standard grade, a significant
    number of learners will probably qualify for
    basic admission to university on the basis of
    their subject combination..The implication is
    that many more learners may be able to apply for
    admission to Stellenbosch University. . With
    this background, it becomes clear why it is
    necessary for the University to have its own
    measure to interpret the meaning and value of the
    new NSC and also to generate additional
    information in order to make finer distinctions.
  • Higher pass rate is as a result of
  • Lowering standards
  • Upward adjustment of marks

23
Trustworthiness The standardisation of the exam
  • Another assumption is that we now use vastly
    different methods of standardising the results
  • Responsibility for matriculation results has
    changed hands
  • JMB (1918 to 1992)
  • SAFCERT (1992 2001)
  • Umalusi (2002 to date)
  • The one constant has been the standardisation
    process used for the examinations.
  • This is crucial to
  • Obtain equivalence of the standard of the SCE
    across years, subjects and examination
    authorities
  • It is also necessary in order to deliver a
    relatively constant product to the HE sector and
    to the workplace
  • Trust in the statutory institutions established
    with particular mandates.

24
Umalusis research on comparisons of NSC with
NATED 550
  • Another assumption is that the NCS is of a lower
    standard than the previous curriculum.
  • Research completed by Umalusi into the standard
    of the NCS curriculum confirms that in most cases
    the NCS presents a greater cognitive challenge
  • The NCS also represents modern, updated and more
    demanding versions of previous subjects
  • (Ref Maintaining Standards Reports 2008, 2009,
    2010)

25
SC /NSC examination results
Senior Certificate examination results, all
schools, 1994-2010
Sources DoE, 2004, 2005a, 2005b, 2005c, 2007,
2008, 2009
26
Senior Certificate examination results
Senior Certificate examination results, all
schools, 1994-2008
Sources DoE, 2004, 2005a, 2005b, 2005c, 2007,
2008, 2009
27
  • Quality Assurance of the 2010 National Senior
    Certificate Examination
  • Vijayen Naidoo Sen. Manager Quality Assurance
    of Assessments

28
Scope of the quality assurance processes
  • Moderation of Question papers
  • Number of NSC 2010 question papers moderated

Number of subjects Number of papers Approved/ Conditionally approved at 1st moderation Approved/ Conditionally approved at 2nd moderation Approved/ Conditionally approved at 3rd moderation
58 130 47 71 12
29
  • Moderation of question papers
  • PURPOSE
  • to ensure that the question papers are of the
    required standard
  • To ensure that the question papers are
    relatively
  • - fair
  • - reliable
  • - representative of an adequate sample of the
    curriculum
  • - representative of relevant conceptual domains
  • - representative of relevant levels of
    cognitive challenge
  • External moderators

30
Moderation of the question papers
  • Approach
  • Question papers set by panel of examiners DBE
  • Internally moderated by DBE
  • Externally moderated by Umalusi
  • Subsequent moderations and approval.

31
Moderation of the question papers
  • Examples
  • Approved at 3rd moderation (CAT History IT
    Life Sciences P1 Math Lit P2 Physical Sc P1
    etc.)
  • Approved at 2nd moderation (Business Studies
    Economics Life Sciences P2 Maths Physical Sc
    P2 etc.)
  • Isizulu FAL P3 Sesotho HL P1 rejected at 1st
    moderation.

32
Moderation of the question papers
  • Findings
  • Technical Criteria
  • Internal moderation quality of internal
    moderator reports.
  • Content coverage
  • Cognitive Demand
  • Marking Guidelines
  • Language and Bias
  • Predictability

33
Moderation of the question papers
  • Maths Lit P1- Content coverage

LO Norm Exam Panel (1st Moderation) Ext Moderator (1stModeration) Exam Panel (Approved) Ext Moderator (Approved)

LO1 25 29 35 27 27
LO2 25 22 15 23 25
LO3 25 23 21 24 22
LO4 25 25 29 26 27
34
Moderation of the question papers
  • Civil Technology- Cognitive Weighting

Cognitive Level (Domain) Norm Exam Panel (1st Moderation) Ext Moderator (1stModeration) Ext Moderator (Approved)

Lower 30 35 21.5 27
Middle 50 35.5 68.5 56.5
Higher 20 29.5 10 16.5

35
  • Scope of the quality assurance processes
  • Moderation of internal /continuous assessment
  • Internal assessment refers to any assessment
    conducted by the provider, the outcomes of which
    count towards the achievement of the
    qualification
  • panels of moderators / subject specialists
  • Moderation at all levels
  • Directives for internal assessment

36
  • Moderation of internal assessment (cont.)
  • Purpose
  • Ascertain the appropriateness and standard of the
    assessment tasks being developed within colleges
  • Ascertain the degree to which assessment
    bodies/provinces are attempting to ensure
    standardisation across
  • Determine the extent and quality of internal
    moderation and educator development
  • Determine the reliability and validity of the
    assessment outcomes

37
  • Moderation of SBA
  • Previous model - moderation of learner and
    educator portfolios at the end of the year.
  • Implementation of new model in 2010 focus on
    tasks rather than on portfolios
  • Ongoing 3 moderations across 9 PDEs
  • Feed back to educators
  • Improvement in SBA implementation

38
  • Moderation of SBA - Scope

Planned Completed
Term 2 Number of subjects 11 plus Life Orientation Subjects Accounting, Agricultural Science, Business Studies, Economics, Engineering Graphics Design, History, Geography, Life Orientation, Life Sciences, Mathematics, Mathematics Literacy, Physical Science (each assessment body was allocated 3 subjects) Term 2 Moderation was conducted 1-8 June 2010.
Term 3 Number of subjects 12 language subjects plus Life Orientation Subjects Afrikaans FAL, English HL, English FAL, Sepedi HL, Setswana HL, Sesotho HL, IsiNdebele HL, Tshivenda HL, IsiZulu, HL, IsiXhosa, HL, Xitsonga HL, SiSwati HL. (where each assessment body was allocated 2 language subjects plus Life Orientation) Term 3 Moderation was conducted 18-19 August 2010, and 25-26 August 2010.
Term 4 Number of subjects 15 Subjects plus Life Orientation Term 3 Moderation was conducted 12 Oct 2010 -12 November 2010
39
  • Moderation of SBA
  • Findings
  • Adherence to policy guidelines
  • Quality of Assessment tasks
  • Content Coverage
  • Cognitive demand and difficulty levels
  • Marking (tools, application and mark allocation)
  • Internal Moderation
  • Moderation instruments
  • Moderation at School level district level
  • Feedback and Support

40
  • Moderation of Life Orientation
  • Scope LO moderated across 9 PDEs

TERM SPECIFIED TASK
2 Source based task
3 Other task and the examination
4 Three tasks not previously moderated incl Physical Education Task.
41
  • Moderation of Life Orientation
  • Findings
  • Good practice common tasks
  • Concerns
  • Quality of tasks
  • moderation
  • Use of rubrics
  • Physical Education Tasks
  • Recommendations
  • At least one common task

42
Scope of the quality assurance processes
  • Focus of the monitoring visits
  • Policy development and implementation
  • Systems, processes and procedures
  • Plan for assessment and moderation
  • Availability and training of staff
  • Planning for the development of learner
    portfolios and educator portfolios

43
  • Monitoring of Examinations
  • State of Readiness
  • Avoid duplication
  • Establish veracity of DBE monitoring processes

Planned Completed Still to happen
State of readiness Monitored the following provinces with the DBE WC 6-8 July Mpumalanga 13-15 July KZN 27-29 July Eastern Cape 11-12 August Receive consolidated report from the DBE
44
Scope of the quality assurance processes
  • Monitoring of the NSC examination writing
  • Findings
  • General Management of the examinations
  • Candidate registration
  • Irregularities
  • Mainly technical
  • Packaging

No of exam centres No of candidates enrolled No of exam centers monitored by Umalusi No of Umalusi monitors per province No of exam centers monitored by Umalusi staff
6540 642 001 162 32 13
45
Scope of the quality assurance processes
  • Monitoring of the NSC examination marking
  • Findings
  • Delay in commencement of marking
  • Delay in capturing of marks
  • Concern with the appointment of markers
  • Large number of marking centers security risk

No of marking centres No of marking centers visited by Umalusi monitors No of marking centers visited by Umalusi staff
127 45 10
46
  • Verification of marking
  • PURPOSE
  • Moderation of marking determines the standard and
    quality of marking and ensures that marking is
    conducted in accordance with agreed practices
  • Umalusi engages the following during the
    moderation of marking
  • 1. Pre-marking/memorandum discussion
    centralised memo discussions recommended - this
    will ensure consistency across marking centres
  • 2. Off-site moderation of marking

47
Scope of the quality assurance processes
  • Verification of marking
  • Memoranda discussion meetings
  • Approved and signed off finalized memoranda/
    marking guidelines for all the NSC subjects
  • Concern with non attendance and pre- memo
    discussion preparations

48
Scope of the quality assurance processes
  • Centralized verification of marking

Non language subjects Large enrolments Language subjects Small enrolments and subject s with Practical Component
Accounting Afrikaans First Additional Language Computer Applications Technology
Agricultural Science English First Additional Language Consumer Studies
Business Studies IsiXhosa Home Language Engineering Graphics and Design
Economics Sesotho Home Language Tourism
Geography Setswana Home Language
History IsiZulu Home Language
Life Sciences IsiNdebele Home Language
Mathematics English Home language
Mathematical Literacy
Physical Science
49
Scope of the quality assurance processes
  • Centralized verification of marking
  • Findings
  • Adherence to marking memo
  • Provision of alternate answers
  • Consistency and accuracy
  • Internal moderation

50
Scope of the quality assurance processes
  • Centralized verification of marking
  • Concerns
  • Change to marking guidelines
  • Competence of markers
  • Sample of scripts verified by Umalusi

51
Scope of the quality assurance processes
  • Standardisation
  • 58 subjects standardised
  • Raw marks accepted 39 subjects
  • Upward adjustments 9 subjects
  • Downward adjustments 10 subjects

52
Scope of the quality assurance processes
  • Standardisation
  • Concerns
  • African Languages
  • Consumer Studies

53
  • Verification of the Resulting Process

Planned Status
Subject structures Subject structures verified.
Candidate registration System verified during state of readiness visits.
Generation of mark Sheets Monitored
Capturing of Marks Monitored
Standardisation data Booklets Data sets received and verified
Capturing of adjustments Verified
Statistical moderation resulting Verified
54
Areas of good practice
  • Generally papers catered for both the weaker
    high performing candidates,
  • In many cases the Nov 2010 March 2011 were
    presented at the same time for moderation, this
    ensured comparability of standards
  • External moderators and Umalusi staff signed the
    final papers off before dispatch to provinces for
    printing,
  • Great improvement observed in the monitoring of
    exams by provinces,
  • Security measures were intensified in the
    majority of centers across provinces,

55
Areas of concern
  • Registration of Candidates
  • Filtering of candidates
  • Moderation of Question Papers
  • Late submission of question papers for
    moderation.

56
Areas of concern
  • Moderation of SBA
  • Majority of provinces do not conduct actual
    moderation of SBA tasks learner evidence.
    Current moderation is a mere audit for
    compliance,
  • Majority of the SBA tasks reflect over reliance
    on past examination papers, educators lack
    creativity in developing suitable tasks,
  • SBA tasks developed by educators focus mainly on
    assessing the lower cognitive skills,
  • Educators battle with the development use of
    rubrics used for scoring learners,

57
Areas of concern
  • Implementation Assessment of Life Orientation
    (LO) requires serious attention. LO is 100
    internally assessed and standards vary from one
    province to the other, even within provinces.
  • Monitoring of exams
  • Incidents of improper registration of candidates
    resulted in duplications and some candidates not
    registered or registered for the incorrect
    subjects,
  • The appointment of suitable Chief Invigilators is
    encouraged. This will address problems relating
    to the flouting of even basic examination
    regulations.
  •  

58
Areas of concern
  • Criteria for the appointment of markers should be
    strictly adhered to so as not to disadvantage
    candidates.
  • All stakeholders need to be consulted when
    changes are made relating to the appointment of
    markers.
  • Moderation of marking
  • Memoranda discussion meetings for small subjects
    were generally a disservice to the external
    moderators due to the non-attendance of the
    relevant provincial delegates. Added to this was
    the absence of input from the provinces.

59
Areas of concern
  • Additions were made to the final memoranda
    without the consent of the DBE and Umalusi.
  • Some markers do not have the ability and
    experience to handle higher-order cognitive level
    questions that required insight and logical
    reasoning.
  • There were instances of inaccurate totaling of
    marks and incorrect transferring of marks to the
    cover page during marking.

60
Areas of concern
  • Potential administrative and management problems
    in Limpopo, KwaZulu Natal and Free State
  • Large number of marking centers security risk

61
Recommendations
  • Moderation of SBA
  • Introduction of a common (standardised task)
  • Moderation of the standard of SBA must be
    implemented at the school and cluster levels.
    Checks for compliance can still be part of the
    moderation process to ensure adherence to policy.
  • All the assessment bodies should ensure that all
    educators are familiar with the policy
    requirements to ensure the meeting of national
    standards.
  • .

62
Recommendations
  • Verification of marking
  • Criteria for the appointment of markers should be
    strictly adhered to so as not to disadvantage
    candidates.
  • All stakeholders need to be consulted when
    changes are made relating to the appointment of
    markers.
  • Standardization of marks
  • Common framework for the assessment of Languages.
  • Teaching and learning of Mathematics and Science
    needs to be strengthened.

63
Examination Irregularities
  • The majority of irregularities were of a
    technical nature and these were reported to
    Umalusi according to the established channels.
  • Some irregularities were as a result of
    registration-related problems, e.g. candidates
    nor appearing on mark sheets, some registered for
    incorrect subjects.
  • Umalusi represented on NEIC
  • Major irregularities
  • No major irregularities reported

64
Conclusion
  • The findings of the quality assurance processes
    are a clear indication of a maturing system that
    has, on the one hand, made positive strides
    towards improvement in certain areas of
    assessment and examination, but, on the other
    hand, still has a few challenges that need to be
    addressed.
  • The quality assurance of each of these processes
    presented above was conducted based on Umalusi
    criteria. Umalusi uses criteria that are
    subjected to constant review and refinement, to
    ensure that they are in line with current trends
    in assessment and examinations.

65
Conclusion
  • In general Umalusi is pleased with the manner in
    which the 2010 NSC examination was administered.
  • Umalusi acknowledges that a number of technical
    irregularities were reported, but these were
    addressed in a fitting manner.
  • Umalusi takes this opportunity to express
    appreciation to the national provincial
    departments of education for their concerted
    effort in ensuring a credible examination.
  • Umalusi expresses appreciation also to all the
    relevant stakeholders for the necessary support
    given in line with Umalusi quality assurance
    initiatives.

66
  • Thank you!
About PowerShow.com