Reducing the Monitoring Burden on Centres: Understanding the Current Environment

1
Reducing the Monitoring Burden on
CentresUnderstanding the Current Environment

• Summer 2006

2
Contents

• Objectives and approach
• Executive summary
• Centres and awarding bodies
• Understanding how learners access qualifications
• Understanding how awarding bodies monitor centres
• Understanding the impact of awarding body
  monitoring on centres
• Steps awarding bodies are taking to reduce
  monitoring activity
• Introducing other bodies that monitor or inspect
  centres / providers
• Inspectorates
• Funding and planning bodies
• Quality marks
• Identifying what makes a good centre
• Defining good
• Measuring good and the impact on monitoring
• Other monitoring approaches
• In the education environment
• In other sectors
• Appendix
• A Consulted organisations

3
This document analyses how centres are currently
monitored and reviews the approaches of other
monitors, both in and outside of the centre
environment
Objectives
Approach

• Understand how awarding bodies monitor centres
• Understand how other bodies monitor or inspect
  centres
• Analyse other monitoring approaches in and
  outside of the education sector to understand
  what can be learnt
• Scope
• The geographic areas in scope are England, Wales
  and Northern Ireland
• Scotland is looked at in order to understand how
  monitoring differs and what could be learnt
• The project scope includes only those
  qualifications in the National Qualifications
  Framework (NQF)
• In Wales, General Qualifications are excluded

• Secondary research
• Awarding body and other organisations websites
• Industry publications and reports
• Awarding body monitoring templates
• Regulatory documents
• Text line searches
• Primary research
• Interviews with organisations representing a wide
  sample of the stakeholders involved in the centre
  monitoring process

Note A list of stakeholder interviews/ meetings
and a bibliography is contained in the appendix.
4
Executive summary
5
Learners access qualifications in centres that
are accountable to awarding bodies that are, in
turn, accountable to regulators
REGULATORY BODY
AWARDING BODY
CENTRE
LEARNER

• Recognise awarding body
• Accredit awarding bodys qualification

• Seek recognition from the regulator
• Seek accreditation for qualification from the
  regulator
• Recognise centre
• Approve centre to offer a qualification

• Seek recognition from awarding body
• Seek approval to offer a qualification from
  awarding body
• Offer qualification to learner

• Seek qualifications at centre

Before a learner can be offered a qualification,
the centre needs to be recognised and approved
for that qualification by the awarding body, and
the awarding body needs to be recognised and the
qualification approved by the regulator.
Source www.QCA.org.uk Research into the
current centre recognition and centre
qualification approval arrangements, MORI,
September 2005 The market for qualifications in
the UK, PricewaterhouseCoopers, April 2005.
6
There are over 115 awarding bodies in the markets
and most centres (78) have between one and five
awarding body relationships
Number of awarding body relationships of centres
Average number of awarding body (AB)
relationships of centresa
Commentarya,b

• Learners receive training and / or teaching at
  their centre and usually assessed for their
  qualification(s) at their centre also
• Schools, colleges, training providers and
  employers can all apply to awarding bodies to be
  recognised as centres
• There are over 115 awarding bodies eligible to
  offer regulated qualifications in the UK
• Most centres (78) have between one and five
  awarding body relationships, but a significant
  minority (8) have over sixteen relationships
• The number of awarding body relationships a
  centre has is driven by factors such as the range
  of qualifications a centre offers, personal
  preference and local demand

Centre sample size 1,000
Centres state that qualification content and
reputation are more important when choosing
awarding bodies and qualifications than paperwork
and processes and fees.
Source (a) Research into the current centre
recognition and centre qualification approval
arrangements, MORI, September 2005 (b)
Capgemini analysis based on stakeholder
interviews, Summer 2006.
7
Awarding bodies monitor centres for regulatory,
quality and commercial purposes
Post-approval (or ongoing) monitoring of centres
by awarding bodies
Post-approval monitoring of centres is defined
as The review of, and reporting on, the
centres quality assurance arrangements by the
awarding bodiesa
DEFINITIONa

• The purpose of awarding bodies monitoring of
  centres is to
• Maintain quality and standards in their
  qualifications
• Ensure that they are able to satisfy the
  regulators
• Assure their own reputation

PURPOSEb

• Contact with centres that occurs through some
  monitoring can also result in
• Support for programme leaders
• Relationship building
• Ensuring opportunities for learners

OTHER OUTCOMESb
Awarding bodies and others in the environment,
including centres, stress that awarding bodies
can provide significant support to programme
leaders as part of the monitoring process.
Source (a) Stakeholder consultation Centre
recognition and centre qualification approval,
November 2005 January 2006, Centre Recognition
Project, 11/2005 (b) Capgemini analysis based on
stakeholder interviews, Summer 2006.
8
Awarding body monitoring activity relates either
to assessment or to a centres continued
adherence to the centre approval criteria
How awarding bodies monitor centres
Rationale for monitoring activity
Centre responsibilities
Awarding body monitoring activity

• To ensure that standards are aligned within and
  across centres

• Internal assessment and standardisation of
  learners written work

• External moderation of internally assessed work

• To ensure a centres conduct of examinations
  adheres to awarding body expectations

• Conduct of examinations in line with instructions

• Visits during examination periods
• Analysis of examination outcomes

ASSESSMENT RELATED

• To ensure quality and consistency of assessment
  practices and procedures within and across
  centres
• This ensures standards are aligned

• Internal assessment of learner competence
• Internal verification of internal assessment

• External verification of internal assessment and
  verification
• Monitoring of external verifiers

• To ensure that the centre is continuing to meet
  the requirements of the approved centre criteria
• By reviewing the centres quality assurance
  arrangements

• Systems audit

• Adherence to centre approval criteria

CENTRE APPROVAL RELATED
Awarding bodies tend to define their monitoring
by qualification type, but different
qualifications are subject to the same types of
monitoring if the assessment methods used are the
same.
Source (a) GCSE, GCE, VCE, GNVQ and AEA Code of
Practice, 2006/07, QCA, ACCAC (as was), CCEA,
2006 (b) NVQ Code of Practice, QCA, ACCAC (as
was), CCEA, 2001 (c) Proposed centre
recognition requirements, QCA, ACCAC (as was),
CCEA, SQA, 2005 (d) Capgemini analysis based on
stakeholder interviews, Summer 2006.
9
On the surface, awarding bodies collect very
similar information when monitoring centres
Sample of awarding body VRQ assessment means
Externally assessed
Internally assessed
Both internal and external
Centre responsibilities
Awarding body monitoring activity
GQ
NVQ
A
B
C
D
E
F
G
H
I
Internal assessment and standardisation of
written work
Sampling of internally assessed work
X
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
X
n/a
Conduct examinations in line with instructions
Checking exam conduct
X
n/a
X
X
X
n/a
n/a
n/a
X
X
X
Analysing exam outcomes
n/a
X
X
X
n/a
n/a
n/a
X
Internal assessment of learner competence Internal
verification of internal assessment
Sampling internal assessments
n/a
X
n/a
n/a
n/a
X
X
X
X
X
Checking internal verification processes, e.g.
sampling plan
n/a
X
n/a
n/a
n/a
X
X
X
X
X
X
Adherence to centre approval criteria
Recording details of centre, candidates and
qualifications
n/a
X
X
X
X
X
X
X
X
X
X
Understanding changes since the last monitoring
check
n/a
X
X
X
X
X
Understanding if actions have been implemented
n/a
X
X
X
X
X
X
X
X
X
Checking quality assurance records are maintained
n/a
X
X
X
X
X
X
X
X
X
X
Health and safety policy
n/a
X
X
X
X
X
X
Physical resources
n/a
X
X
X
X
X
X
X
X
X
X
Staff qualifications
n/a
X
X
X
X
X
X
X
X
X
X
QUALITY ASSURANCE CHECKS
Communications system
n/a
X
X
X
X
X
X
X
X
Particular requirements
n/a
X
X
X
X
X
X
X
Appeals process
n/a
X
X
X
X
X
X
X
Equal opportunities policy
n/a
X
X
X
X
X
X
X
X
Awarding bodies offering GQs or NVQs seek
information according to the code of practice
and, for VRQs, where they have more flexibility,
appear to seek very similar information.
Source (a) Capgemini analysis based on
stakeholder interviews and awarding body
documentation, Summer 2006
10
The level of monitoring a centre receives depends
on the qualifications it offers as well as on the
number of awarding body relationships it has
Monitoring activity
Centre scenario
External moderation1
Exam visits
External verification2
Systems audit
ABs3
Visits p.a.
A

• A school
• Offers GQs only
• Has 5 unitary body relationships

• Applies to each qualification with internally
  assessed work

• Maximum of one visit per exam cycle

• n/a

• n/a

5
0-1
B

• A small employer
• Offers one NVQ to 10 employees p.a.
• Has a relationship with one awarding body

• n/a

• n/a

• Two external verifier (EV) visits per year
  including a systems audit

• n/a

1
2
C

• A training provider
• Offers several NVQs and VRQs in engineering
• Has two awarding body relationships

• n/a

• Awarding body 1 may visit during its VRQ exams
• (All externally assessed)

• For NVQs, 4 EV visits per year 2 from each
  awarding body
• An additional EV visit from awarding body 2 for
  its VRQs
• (All internally assessed)

• Awarding body 1 does a systems audit as part of
  an EV visit
• Awarding body 2 conducts a separate systems audit
  visit

2
6-7
D

• A college
• Offers a wide range of GQs, NVQs and VRQs
• Has over 40 awarding body relationships

• Applies to each qualification with internally
  assessed work

• Receives one visit for all GQs p.a.
• May receive one visit per awarding body for other
  qualifications with exam elements

• Two EV visits p.a. per NVQ programme area for
  each awarding body
• Additional 1 or 2 visits for VRQs with internal
  assessment
• Some awarding bodies combine NVQ and VRQ visits

• Conducted separately by several of the awarding
  bodies
• Combined with external verifier visits for some
  others
• Some do not do audits

40
40
Each centre has a very different monitoring
experiences and, as a result, different
interpretations of the burden of this
monitoring.
Source (a) AEA, GCE, VCE, GCSE, GNVQ and ELC
Instructions for conducting examinations, 1
September 2005 to 31 August 2006, JCQ, 2005 (b)
NVQ Code of Practice, QCA, ACCAC (as was),
CCEA, 2001 (c) Capgemini analysis based on
stakeholder interviews, Summer 2006. Note (1)
External moderation of internally assessed and
standardised written work (2) External
verification of internally assessed and verified
learner competence EV external verifier (3)
ABs awarding bodies. Column indicates the
number of awarding body relationships the centre
has.
11
Centres commented on the benefits of awarding
body monitoring such as support, but feel that
awarding body approaches differ greatly
RANGE OF CENTRE VIEWS
Monitoring is a burden
Monitoring is a benefit
Awarding bodies can provide a lot of help to
struggling centres. Training provider
I am paying awarding bodies for a service and
they are conducing tick box exercises on me that
are of no benefit to me. Training provider
Some awarding bodies have a very supportive
approach whereas others are there to check only.
Training provider
Programme leaders find direct contact with an EV
useful. Examination officer
It is burdensome having lots of awarding body
relationships, but we wouldnt not have a
relationship purely on the basis of burden. We
want to offer learners the best quality.
Examinations officer
Some of the information they ask for does not
relate in any way to the quality and standards of
their qualifications Training provider
All awarding bodies do things differently. All
their forms are very different, their visit
cycles etc. It would help if these were aligned
College Director
All monitors not just awarding bodies ask
the same things Training provider
Centres commented that increasing the number of
awarding body relationships does increase burden,
but they want to offer the best quality
qualifications to learners and increased burden
can be an acceptable by-product.
Source (a) Capgemini analysis based on
stakeholder interviews, Summer 2006.
12
Many awarding bodies commented that the
monitoring they do is not burdensome, and the
support they provide can be very valuable
RANGE OF AWARDING BODY VIEWS
Monitoring is a burden
Monitoring is a benefit
There are times where we have helped a failing
centre to turn around and subsequently receive a
1 in their ALI inspection
Some centres feel burdened by awarding bodies
because we give them actions, but theres a
reason they need the actions!
I appreciate that we can only see the impact
from our point of view and dont have a feel for
the combined impact of awarding body monitoring.
What burden?
Efficient centres have no problem providing
awarding bodies with the information they need
therefore its not burdensome
The primary role of a verifier is to check
assessment set up is robust QA purpose rather
than support.
Centres have a choice about the number of
awarding body relationships they have
Some of the burden centres feel is as a result
of regulatory requirements.
Some awarding bodies felt that tweaks could be
made to reduce some of the burden centres feel.
Source (a) Capgemini analysis based on
stakeholder interviews, Summer 2006.
13
The burden that awarding bodies place on
centres as a result of ongoing monitoring could
be reduced by simplifying or integrating the
activity

• Simplify the monitoring process
• Awarding bodies could reduce the monitoring
  activity per monitoring interaction
• i.e. request less information for monitoring
  purposes or stop some monitoring
• Awarding bodies could reduce the frequency of
  their monitoring activity
• Integrate the monitoring process across awarding
  bodies
• Awarding bodies could standardise their
  monitoring activity
• For example, standardising forms they use or the
  format of documents they request
• Awarding bodies could join up their monitoring
  activity
• For example, combining visits or having one
  organisation monitor on behalf of others

These burden reduction methods are not mutually
exclusive and can all produce options for
consideration.
Source (a) Capgemini analysis, Summer 2006.
14
Awarding bodies are already taking steps to
reduce monitoring burden on centres, with most
initiatives being led by JCQ members
Simplify monitoring
Integrate monitoring
Reduce the monitoring activity per monitoring
interaction
Standardise monitoring across awarding bodies

• Common centre approval criteria
• Centres do not need seek approval more than once
• Consortium arrangements
• Moderation and verification of internal
  assessment is done across the consortium rather
  than on each centre
• Risk-based approaches monitoring good centres
  less (but struggling centres more)
• Conducting desk-based or telephone monitoring

• Common instructions for examination conduct
• Common access arrangements
• Common centre approval criteria
• Common external verifier reporting
• Common data standards
• Building the capacity to share data
• Common centre numbers
• Unique learner numbers

Reduce the frequency of monitoring activity
Join up monitoring across awarding bodies

• Risk based approaches
• For exam visits, for example, awarding bodies
  will target spot checks at the centres they feel
  pose the greatest risk
• The risk assessment is often qualitative (for
  example, the centre is new, therefore high risk)

• Combined exam visits for all qualifications
• One organisation monitors on behalf of others
• Common centre approval criteria
• Some awarding bodies effectively approve on
  behalf of others

All of the starred initiatives are part of the
JCQs Eight Pledges and, with the exception of
common exam instructions and access arrangements,
apply to JCQ awarding bodies (or some of them)
only at present.
Source (a) Consortium Arrangements for BTEC and
Edexcel NVQ Qualifications, Edexcel, 2004 (b)
Consortium Arrangements for General
Qualifications, Edexcel, 2004 (c) Application
for Centre Consortium Arrangements, JCQ, 2005
(d) Capgemini analysis based on stakeholder
interviews, Summer 2006.
15
In addition to awarding bodies, centres are
inspected or reviewed by inspectorates and
funding and planning bodies, each for different
purposes
Bodies that monitor centres
Funding and planning bodies

• Ensure that the provision they fund is meeting
  the demands of learners, employers and local
  communities for learning and skills
• Ensure that the provision they fund is of a high
  quality and represents value for money
• Also, they ensure that providers take ownership
  of quality and demonstrate their commitment to
  continuous improvement

• Maintain quality and standards in the
  qualifications
• Ensure that they are able to satisfy the
  regulators
• Assure their own reputation
• Provide support for programme leaders
• Ensure opportunities for learners

• Raise quality and standards of education and
  training by identifying and promoting good
  practice
• Provide an independent evaluation of the quality
  and standards achieved by a particular centre to
  inform the government and the public

Purpose of monitoring / inspections
Need assurance that Learners achieve awards that
are representative of their competence
Need assurance that Learners achievement levels
are high and education and training quality is
high
Need assurance that Provision meets the needs of
the community, it is high quality and represents
value for money
A centre may be providing education and training
to a low standard, but be maintaining the correct
level of standards in the assessment of
qualifications
To promote self-assessment and increase capacity
for self-improvement, some centres also choose to
adhere to certain external quality frameworks.
Source (a) www.ali.gov.uk (b) The Common
Inspection Framework for Education and Training
from 2005, Ofsted and ALI (c)
www.ofsted.gov.uk (d) The Common Inspection
Framework for Education and Training in Wales,
Estyn (d) COMMON FRAMEWORK FOR INSPECTION for
The Education and Training Inspectorate, ETI
(e) IMPROVING QUALITY RAISING STANDARDS,
Arrangements for the Inspection of Further
Education Programmes in Northern Ireland Revised
2003, ETI, 2003 (f) Capgemini analysis, summer
2006 (g) www.lsc.gov.uk (h) www.elwa.ac.uk
(i) www.delni.gov.uk (j) www.deni.gov.uk (k)
www.hefce.ac.uk.
16
Awarding bodies are the only monitors to check
the appropriateness of assessment decisions or
examination conduct
Bodies that monitor centres
Funding and planning bodies

• The providers self-assessment report and quality
  development/ improvement plans
• Inspection reports or reports from other visits
  by the inspectors
• Strategic plans
• Information from health and safety monitoring,
  and financial data
• Performance against national, regional and local
  benchmarks for learner success rates

• Moderation of internally assessed work
• Visits during examination periods to ensure
  adherence to instructions
• Analysis of examination outcomes
• Verification of internal assessment and
  verification through sampling and speaking with
  staff and learners
• Systems audit to check appropriateness of a
  centres quality assurance processes
• This may include conversations with staff and
  learners

• Observing learning
• Interviewing learners, staff and other
  stakeholders
• Examining learners' work and documents relating
  to training, assessment, verification and
  qualifications
• Reviewing self-assessment reports, quality
  assurance processes, and strategic and
  operational plans

Evidence gathered
However, all monitors appear to do a quality
assurance / systems audit type check.
Source (a) www.ali.gov.uk (b) The Common
Inspection Framework for Education and Training
from 2005, Ofsted and ALI (c)
www.ofsted.gov.uk (d) The Common Inspection
Framework for Education and Training in Wales,
Estyn (d) COMMON FRAMEWORK FOR INSPECTION for
The Education and Training Inspectorate, ETI
(e) IMPROVING QUALITY RAISING STANDARDS,
Arrangements for the Inspection of Further
Education Programmes in Northern Ireland Revised
2003, ETI, 2003 (f) Capgemini analysis, summer
2006 (g) www.lsc.gov.uk (h) www.elwa.ac.uk
(i) www.delni.gov.uk (j) www.deni.gov.uk (k)
www.hefce.ac.uk.
17
Most monitors, even some inspectorates, monitor
or review centres on a annual basis with any
visits lasting anything from a few hours to weeks
Bodies that monitor centres
Funding and planning bodies

• Length ranges from a few hours to a full day
• Frequency some centres will receive no
  monitoring contact, others will receive multiple
  visits from many awarding bodies per year

• Length often conducted remotely. Length of
  visits vary
• Frequency often annually more for providers
  with failings either in the quality of their
  provision, or the rigour of their own internal
  quality management systems

• Length ranges from two days to eight weeks
• Frequency ranges from three to seven years for
  inspections
• However, some inspectorates conduct more frequent
  visits for example Ofsted visits colleges to
  conduct a quality monitoring visit every year

Length and frequency monitoring / provider reviews
Inspectorates and funding and planning bodies and
a number of awarding bodies also use risk
assessments (sometimes informally) to determine
the level of contact a centre / provider requires.
Source (a) www.ali.gov.uk (b) The Common
Inspection Framework for Education and Training
from 2005, Ofsted and ALI (c)
www.ofsted.gov.uk (d) The Common Inspection
Framework for Education and Training in Wales,
Estyn (d) COMMON FRAMEWORK FOR INSPECTION for
The Education and Training Inspectorate, ETI
(e) IMPROVING QUALITY RAISING STANDARDS,
Arrangements for the Inspection of Further
Education Programmes in Northern Ireland Revised
2003, ETI, 2003 (f) Capgemini analysis, summer
2006 (g) www.lsc.gov.uk (h) www.elwa.ac.uk
(i) www.delni.gov.uk (j) www.deni.gov.uk (k)
www.hefce.ac.uk.
18
The monitoring scope varies between awarding
bodies and other monitorswith inspectorates and
funding bodies looking at total provision
Bodies that monitor centres
Funding and planning bodies

• Awarding bodies are concerned primarily with the
  programme areas delivering their qualifications
• This may include a quality assurance check at the
  programme level
• However, some awarding bodies check quality
  assurance systems across the centre

• Funding bodies are concerned with provision
  across an entire centre
• Providers that have demonstrated that adequate
  quality management systems are in place, may
  receive a very light touch funding body review

• Inspectorates are concerned with provision across
  an entire centre
• Inspectorates may only sample a number of
  programme areas in order to make a judgement on
  the quality and standards of education and
  training in a centre

Scope of monitoring / provider reviews
A centre that has received an inspection result
of satisfactory may have shortcomings or areas
for improvement in the one programme area that an
awarding body is concerned with.
Note that a small number of centres may not be
inspected or reviewed by inspectorates or funding
and planning bodies meaning awarding bodies may
be the only monitors looking at that centre
Source (a) www.ali.gov.uk (b) The Common
Inspection Framework for Education and Training
from 2005, Ofsted and ALI (c)
www.ofsted.gov.uk (d) The Common Inspection
Framework for Education and Training in Wales,
Estyn (d) COMMON FRAMEWORK FOR INSPECTION for
The Education and Training Inspectorate, ETI
(e) IMPROVING QUALITY RAISING STANDARDS,
Arrangements for the Inspection of Further
Education Programmes in Northern Ireland Revised
2003, ETI, 2003 (f) Capgemini analysis, summer
2006 (g) www.lsc.gov.uk (h) www.elwa.ac.uk
(i) www.delni.gov.uk (j) www.deni.gov.uk (k)
www.hefce.ac.uk.
19
Inspectorates and funding and planning bodies use
risk based approaches to reduce burden on
providers
Simplify monitoring
Integrate monitoring
Reduce the monitoring activity per monitoring
interaction
Standardise monitoring

• Risk based monitoring approaches
• The extent of the review is determined based on
  the previous review and more recent data and
  reports
• The inspectorates and funding bodies have
  agreements in place to share data
• Self-assessment is used to review providers
• Desk-based reviews occur rather than visits in
  person

• Providers are usually monitored by one
  inspectorate and one funding and planning body
• Therefore, unlike in the awarding body
  environment where centres are monitored by many
  of the same type of bodies, there is little scope
  for standardisation here
• The exception is in England where some providers
  are monitored by both ALI and Ofsted and LSC and
  another funding bodies (e.g. Jobcentre Plus)
• In these cases, the organisations are working
  together to standardise their approach

Reduce the frequency of monitoring activity
Join up monitoring across awarding bodies

• Risk based monitoring approaches
• Some reviews are removed for providers that
  perform well, for example
• ALI conducts quality monitoring visits about a
  year after inspections, but providers that
  perform well at inspection are not visited
• LSC conducts an interim review six months after
  the annual review, but providers that perform
  well are not reviewed again

• Some bodies inspect or review on behalf of others
• ETI conducts all inspections or surveys on behalf
  of DEL, the funding body in NI
• Co-ordinated visits to providers
• Where possible LSC conducts Provider Financial
  Assurance visits in line with inspectorates

Inspectorates visiting on behalf of other bodies
(as occurs in NI with DEL and ETI) and
coordinated visits also reduce the inspection and
review burden on providers.
Note (1) Providers in NI are reviewed by ETI,
the inspectorate, only and ETI informs the
funding body of outcomes.
20
If a centre can be identified as being good
then the monitoring activity it is subjected to
could be reduced
Identifying good and the potential impact on
monitoring

• In order to understand whether monitoring
  activity could be reduced per monitor (by seeking
  less evidence per monitoring interaction or
  reducing the frequency of monitoring), two
  questions need to be asked
• What makes a centre good?
• How can a good centre be identified?

What good looks like for each different centre
monitor will be explored in turn followed by how
good is measured to reduce monitoring activity.
21
When asked what a good centre looks like,
awarding bodies tended to comment on the
efficiency of the centre and its quality
assurance processes
Good centres for awarding bodies

• A good centre
• is one that returns all examination
  documentation on time and completed correctly
• has standard admin processes across the centre
• feels good when you walk through the door you
  can smell it
• has excellent quality management processes in
  place
• sounds good. Our experience means we just know
• has good internal verifier processes so that not
  all verification is end-loaded
• has a balance of experienced staff versus new
  staff and these new staff have training
  opportunities available to them

According to awarding body commentary,
identifying good tends to be a qualitative
measure identified as a result of the experience
of the awarding body staff or contractors
conducting the monitoring.
Source (a) Capgemini analysis based on
interviews with stakeholders, summer 2006.
22
For inspectorates, good centres are places
where learners achieve high standards, quality of
provision is high and management is effective and
efficient
Good providers for inspectorates

• The elements of good for inspectorates centre
  around the three elements of the common
  inspection framework
• Achievements of learners
• Quality of provision
• Quality of leadership and management
• Quality of leadership and management underpins
  the achievement of learners and quality of
  provision, for example
• One of Estyns key questions in part 3 of the
  framework isb
• How well do leaders and managers evaluate and
  improve quality and standards?
• Moreover, when grading, Estyn provides guidance
  to its inspectors that, if learner achievement
  and quality of teaching has been graded poorly,
  management and leadership cannot be rated highlyd
• One of ETIs questions in part three of the
  framework isc
• How well do leaders and managers set a clear
  direction leading to high quality education and
  training?
• ALI and Ofsteds key question in part 3 of the
  framework isa
• How effective are leadership and management in
  raising achievement and supporting all learners?

For the inspectorates, effectiveness of
management underpins learner achievement and
teaching quality and for Estyn, low grades for
the latter two mean a low grade must be given for
management.
Source (a) The Common Inspection Framework for
Education and Training from 2005, Ofsted and
ALI. (b) The Common Inspection Framework for
Education and Training in Wales, Estyn (c)
COMMON FRAMEWORK FOR INSPECTION for The
Education and Training Inspectorate, ETI (d)
Meeting with Ann Keane of Estyn,
08/08/2006. Note Ofsted and ALI publish one
inspection framework
23
For funding and planning bodies, good centres
have strong self-assessment and development
processes that enable them to self-improve
Good providers for funding and planning bodies
LSC (England)a,1
DELLS Planning and Funding (Wales)b,2
ETI (reviewing on behalf of DEL) (Northern
Ireland)c,3
funding bodies, inspectorates and quality
improvement The self-assessment guidance of the
bodies was published in the last year and it will
take time for all providers to achieve the
expected standard.
Source (a) Quality improvement and
self-assessment, LSC, 05/2005 (b) Quality
handbook, ELWa, 06/2006 (c) Together towards
improvement, ETI. Note (1) LSC Learning and
Skills Council (2) DELLS Department for
Education, Lifelong Learning and Skills. The
funding body was previously ELWa, but this has
merged with DELLs (3) DEL Department for
Employment and Learning.
24
In order to reduce monitoring activity, a monitor
must conduct one cycle of standard monitoring
or a pre-monitoring analysis in order to assess
good
Examples of activity to baseline in order to
reduce monitoring activity

• Awarding body A conducts annual monitoring visits
  (systems audits) to check that the required
  quality assurance processes are in place
• Centres are graded on a scale of 1-5 with 1 being
  the top grade
• The grade impacts the level of monitoring centres
  receive the following year
• Centres scoring 1 or 2 are offered the
  opportunity to self-assess the following year

• ALI is conducting one cycle of standard
  monitoring to gather baseline data
• After this, the extent of a providers inspection
  is determined by a pre-inspection analysis
• This analysis draws on a range of information
  including
• Previous inspection report or quality monitoring
  report
• Retention and achievement data and other
  performance information
• ALI conducts a Quality Monitoring Visit about a
  year after inspection
• Providers that perform well at inspection are
  exempt from this visit

• Prior to annual planning reviews, providers
  submit self assessment reports
• Providers that demonstrate, through
  self-assessment, effective management of the
  risks to outstanding or good quality receive less
  intensive Annual Planning Reviews from the LSC
• If the review identifies poor quality provision
  or lack of progress towards the achievement of
  headline performance measures, the LSC will
  discuss with providers options for support and
  intervention
• If the LSC considers the self-assessment to lack
  rigour it will work with the Quality Improvement
  Agency (QIA) to support the provider in the
  revision of its self-assessment

Whilst monitoring activity may reduce for those
centres that are identified as being good, it
will increase for struggling centres as the
monitor provides them with additional support and
contact.
Source (a) Capgemini analysis based on
interviews with stakeholder, summer 2006 (b)
Changes in inspection, What they mean for you,
ALI, 04/2005 (c) Quality improvement and
self-assessment, LSC, 05/2005. Note ALI
Adult Learning Inspectorate (England), LSC
Learning and Skills Council (England)
25
Even once a centre has been identified as being
good its performance is still often reviewed
frequently to check that performance has not
dropped
The risk associated with risk assessments

• Annual monitoring occurs for all centres
• Good centres self-assess, all others are
  visited
• If standards are suspected of slipping in a
  good centre as a result of the self-assessment,
  a visit is triggered
• Investigations are also triggered if the awarding
  body identifies any potential malpractice through
  its analysis of exam outcomes or the exam
  documentation submitted by the centre

• ALI conducts a pre-inspection analysis to
  determine the extent of an inspection
• If the quality of some aspect of provision is
  found to be unsatisfactory in reduced intensity
  inspections, ALI may allocate further days to the
  inspection

• The LSC reviews all providers annually
• Prior to annual planning reviews, provider submit
  self assessment reports that are used to
  determine the extent of the Annual Planning
  Review

Awarding bodies, inspectorates and funding bodies
mitigate against the risk of risk-based
monitoring by frequently reviewing performance
and having triggers in place that prompt
monitoring action.
Source (a) Capgemini analysis based on
interviews with stakeholder, summer 2006 (b)
Changes in inspection, What they mean for you,
ALI, 04/2005 (c) Quality improvement and
self-assessment, LSC, 05/2005. Note ALI
Adult Learning Inspectorate (England), LSC
Learning and Skills Council (England)
26
Other monitors in and outside of the education
sector are illustrating a number of methods that
they are using to reduce monitoring burden
Simplify monitoring
Integrate monitoring
Reduce the monitoring activity per monitoring
interaction
Standardise monitoring

• Risk based monitoring approaches
• The extent of the monitoring is determined based
  on the previous review and more recent data and
  reports
• Self-assessment is used to review organisations
• Self-assessment reports are often used as part of
  reviews
• SQA centres can be delegated more authority and
  are monitored only on their quality assurance
  processes
• Conducting desk-based reviews
• For example, Edexcel is no longer visiting HEIs
  but reviewing information submitted electronically

• The heath concordat is promoting standardisation
  through the tools it has developed for
  implementation, for example
• Baseline assessment, using gap analysis to map
  existing processes against recommended practice
• Evaluating costs and benefits against a best
  practice report
• Developing guidelines for information gathering
  and sharing
• Guidelines for making recommendations to
  providers, to allow co-ordination and prevent
  duplication

Reduce the frequency of monitoring activity
Join up monitoring across awarding bodies

• Risk based monitoring approaches
• For example, SQA conducts systems audits
  according to a risk system that means poor
  centres are monitored more and good centres less

• Some bodies inspect or review on behalf of others
• QAA assesses quality of teaching on behalf of all
  higher education funding bodies
• Co-ordinated visits
• For example, health concordat members are
  co-ordinating inspections with other reviews

Risk based approaches, monitoring via
self-assessment and conducting desk-based reviews
(rather than visits) are all common.
27
Centres and awarding bodies

• Understanding how learners access qualifications
• Understanding how awarding bodies monitor centres
• Understanding the impact of awarding body
  monitoring on centres
• Steps awarding bodies are taking to reduce
  monitoring activity

28
Centres and awarding bodies

• Understanding how learners access qualifications
• Understanding how awarding bodies monitor centres
• Understanding the impact of awarding body
  monitoring on centres
• Steps awarding bodies are taking to reduce
  monitoring activity

29
Learners have a wide range of regulated
qualifications available to them, ranging from
general to vocational to occupational
qualifications
Categories of regulated qualifications available
to learners
General qualifications
Occupational qualifications

• 24 of the qualifications in the National
  Qualifications Framework are General
• The true general qualifications are recognised
  as AEAs, GCSEs, GCEs and GCE AS
• In April 2005, there were 768 qualifications in
  this market
• GCSEs are at levels 2 and 3 and GCEs at level 3
• The General category also includes other
  general qualifications such as graded music and
  free-standing mathematics qualifications

• 37 of the qualifications in the NQF are
  Occupational
• Occupational qualifications comprise the largest
  group of qualifications in the National
  Qualification Framework
• They are primarily NVQs
• The majority of qualifications are at Levels 2
  and 3 (65)

Vocationally Related qualifications
Other qualifications

• 27 of the qualifications in the NQF are
  Vocationally Related
• These qualifications can be offered at all levels

• 18 of the qualifications in the NQF are in the
  other category
• This category includes Key Skills, Entry Level,
  Basic Skills and Higher Level qualifications.

In April 2005 there were over 4,500
qualifications in the National Qualifications
Framework the list of all regulated
qualifications.
Source (a) The market for qualifications in the
UK, PricewaterhouseCoopers, April 2005. Note
NQF National Qualifications Framework. Due to
their dominance, NVQs are used in this report as
a proxy for the occupational qualifications
market.
30
Learners access qualifications at centres which
can include schools, colleges, training providers
and employers
Types of centresa
Schools
Colleges

• Schools provide learning to children usually up
  to the age of 16, on behalf of the state
• Learning and assessment predominantly take place
  within the schools own building
• Principal qualifications offered are general
  qualifications such as GCSEs
• Schools may offer a broad range of qualifications

• Colleges include
• Further education and sixth form colleges
• Higher education colleges (outside project remit
  unless offering qualifications on the NQF)
• Sixth form colleges
• Independent schools
• Qualifications offered can be the same as at
  school but often include level 3 general
  qualifications and vocational courses

Employers
Training providers

• Many employers offer qualifications to their
  employees in house
• According to research by PricewaterhouseCoopersb,
  40 of employers surveyed provided internal
  training leading to a qualification
• Qualifications offered include both NVQs and VRQs
• Employers are most likely to provide job specific
  training (80) with 69 providing health and
  safety training and 53 providing training in new
  technologyc

• Training providers generally provide training in
  specific skills at designated centres or on an
  employers site
• For example, business skills workshops
• Qualifications offered often include VRQs
• Training providers often offer training in
  specific areas and therefore have a more focussed
  range of qualifications than, for example, a
  college

Normally, learners receive training and / or
teaching at their centre and are also assessed
for their qualification(s) there.
Source (a) Research into the current centre
recognition and centre qualification approval
arrangements, Mori, September 2005, (b) The
market for qualifications in the UK,
PricewaterhouseCoopers, April 2005 (c) National
Employer Skills Survey, 2003 (d) Capgemini
analysis based on stakeholder interviews, Summer
2006.
31
In order to offer qualifications, centres need a
relationship with an awarding body as it is these
organisations that develop and award
qualifications
Awarding body market and relationships with
centres
Awarding body market overviewb,c,d
Awarding body and centre relationshipa

• Awarding bodies develop and award qualifications
• They also assess and quality assure
  qualifications and provide customer service to
  centres and candidates
• There are over 115 awarding bodies that offer
  qualifications that are part of the National
  Qualifications Framework
• There are only five awarding bodies in England,
  Wales and NI that can offer General
  Qualifications
• These are AQA, Edexcel, OCR, WJEC (Wales) and
  CCEA (NI) and are known as the unitary bodies
• The awarding body market is dominated by the
  large players that make up the Joint Council for
  Qualifications (JCQ)
• Members of the JCQ are the unitary bodies listed
  above plus SQA, the unitary body in Scotland, and
  City Guilds, the largest vocational awarding
  body
• Of the 115 awarding bodies, the majority are
  members of the Federation of Awarding Bodies
  FABe,1

• A centre is defined as any organisation,
  establishment or individual entity accountable to
  an awarding body for the assessment arrangements
  leading to an accredited qualification or unit.a
• A centre must be recognised by an awarding body
  and approved to offer one of its qualifications
  before the qualification can be offered to
  learners
• Centre recognition is defined as a process
  through which a centre is recorded as having
  committed itself to maintain the required quality
  and consistency of assessment and comply with
  other expectations of the relevant awarding
  bodya
• Centre qualification approval is defined as a
  process through which a centre is confirmed as
  being able to maintain the required quality and
  consistency of assessment and comply with other
  expectations of the relevant awarding bodya
• Once approved, a centre is subjected to
  post-approval monitoring by the awarding body
• This is defined as the review of, and reporting
  on, the centres quality assurance arrangements
  by the awarding bodiesa

Centres are accountable to awarding bodies on an
ongoing basis for the quality and consistency of
assessment and any other expectations that
awarding bodies define.
Source (a) Stakeholder consultation Centre
recognition and centre qualification approval,
November 2005 January 2006, Centre Recognition
Project, 11/2005 (b) The market for
qualifications in the UK, PricewaterhouseCoopers,
April 2005 (c) www.JCQ.org.uk (d)
www.QCA.org.uk (e) www.awarding.org.uk Note
(1) For a list of FAB members, go to the FAB
website and click on the members tab
www.awarding.org.uk
32
Most centres (78) have between one and five
awarding body relationships, but a significant
minority (8) have over sixteen relationships
Number of awarding body relationships of centres
Average number of awarding body (AB)
relationships of centresa
Commentarya,b

• According to research by MORI, 80 of centres
  have formed relationships with only 1-5 awarding
  bodiesa
• However, 8 of centres have complex relationship
  networks, working with over 16 awarding bodies
• Colleges tend to have the most relationships with
  awarding bodies, and schools the least when
  considering applications for recognition and
  qualification approval
• The total number of awarding body relationships a
  centre has is determined by factors such as
• The range of qualifications the centre offers
• Personal preference of programme leaders
• Availability of funding for particular courses or
  qualifications
• Local demand
• Legal requirements, for example, health and
  safety training requirements for employers

Centre sample size 1,000
The number of awarding body relationships a
centre has is driven by factors such as the range
of qualifications a centre offers, personal
preference and local demand.
Source (a) Research into the current centre
recognition and centre qualification approval
arrangements, MORI, September 2005 (b)
Capgemini analysis based on stakeholder
interviews, Summer 2006.
33
Centres decisions about which awarding bodies
they have relationships with is affected by the
content of the qualifications and reputation
Factors affecting which awarding bodies centres
choosea
Factors affecting choice of awarding body
Commentary

• MORI research found that the content of
  qualification and the reputation of that
  qualification were seen as important by 96 of
  centres
• 95 of centres felt that the reputation of the
  awarding body is also important
• Fewer centres felt that considerations about the
  paperwork and processes involved were important
  when choosing an awarding body (73)
• Fewer still (66) felt fees were important
• Two thirds of centres also cited other factors
  affecting their choice
• This included the overall or ongoing support and
  guidance offered by an awarding body
• 14 of centres cited this factor and it was more
  important for training providers (19 citing the
  factor)

The content of a particular qualification/
fitness for purpose
The reputation of their qualifications
The reputation of the awarding body
The paperwork and process involved
The fees an awarding body charges
Centres state that qualification content and
reputation are more important when choosing
awarding bodies and qualifications than paperwork
and processes and fees.
Source (a) Research into the current centre
recognition and centre qualification approval
arrangements, MORI, September 2005.
34
In order to award regulated qualifications,
awarding bodies must seek recognition and then
approval for each qualification from the
regulators
The regulatory bodies and relationships with
awarding bodiesa
Regulatory bodies overview
Regulatory and awarding body relationships

• QCA is the regulatory body in England, DELLS in
  Wales and CCEA in Northern Ireland1
• These organisations work together to regulate
  awarding bodies and qualifications
• The purpose of regulation is to ensure that
  qualifications are fair, standards are secure,
  and public confidence is maintained
• They regulate by
• Developing and publishing criteria for the
  accreditation of qualifications
• There are three types of regulatory criteria
  statutory regulations subject criteria and
  codes of practice
• Accrediting qualifications against those criteria
  
• Keeping qualifications under review
• Publishing and sharing information relating to
  accredited qualifications

• In order to award qualifications that are part of
  the NQF, an awarding body must first be
  recognised by a regulator
• An organisation can also enter into partnership
  with an existing recognised awarding body
• Once recognised, awarding bodies can submit their
  qualifications to the regulators for
  accreditation
• Once accredited, the qualification is placed on
  the NQF
• The recognition process and qualification
  requirements are outlined in the regulatory
  document The statutory regulation of external
  qualifications
• Awarding bodies with qualifications on the NQF
  are subjected to ongoing monitoring from the
  regulators
• Monitoring findings inform decisions about
  re-accrediting qualifications or, if necessary,
  withdrawing accreditation

Awarding bodies are monitored by regulatory
bodies to ensure that they deliver high quality
qualifications and provide good customer service
in compliance with the regulatory criteria.
Source (a) www.QCA.org.uk. Note (1) QCA
Qualifications and Curriculum Authority DELLS
Department for Education, Lifelong Learning and
Skills CCEA Council for the Curriculum,
Examinations and Assessment.
35
Centres and awarding bodies

• Understanding how learners access qualifications
• Summary

36
Learners access qualifications in centres that
are accountable to awarding bodies that are
accountable to regulators
REGULATORY BODY
AWARDING BODY
CENTRE
LEARNER

• Recognise awarding body
• Accredit awarding bodys qualification

• Seek recognition from the regulator
• Seek accreditation for qualification from the
  regulator
• Recognise centre
• Approve centre to offer a qualification

• Seek recognition from awarding body
• Seek approval to offer a qualification from
  awarding body
• Offer qualification to learner

• Seek qualifications at centre

Before a learner can be offered a qualification,
the centre needs to be recognised and approved
for that qualification by the awarding body, and
the awarding body needs to be recognised and the
qualification approved by the regulator.
Source www.QCA.org.ukl Research into the
current centre recognition and centre
qualification approval arrangements, MORI,
September 2005 The market for qualifications in
the UK, PricewaterhouseCoopers, April 2005.
37
There are over 115 awarding bodies in the markets
and most centres (78) have between one and five
awarding body relationships
Number of awarding body relationships of centres
Average number of awarding body (AB)
relationships of centresa
Commentarya,b

• Learners are trained and / or taught and usually
  assessed for their qualification(s) at their
  centre
• Schools, colleges, training providers and
  employers can all apply to awarding bodies to be
  recognised as centres
• There are over 115 awarding bodies eligible to
  offer regulated qualifications in the UK
• Most centres (78) have between one and five
  awarding body relationships, but a significant
  minority (8) have over sixteen relationships
• The number of awarding body relationships a
  centre has is driven by factors such as the range
  of qualifications a centre offers, personal
  preference and local demand

Centre sample size 1,000
Centres state that qualification content and
reputation are more important when choosing
awarding bodies and qualifications than paperwork
and processes and fees.
Source (a) Research into the current centre
recognition and centre qualification approval
arrangements, MORI, September 2005 (b)
Capgemini analysis based on stakeholder
interviews, Summer 2006.
38
Centres and awarding bodies

• Understanding how learners access qualifications
• Understanding how awarding bodies monitor centres
• Understanding the impact of awarding body
  monitoring on centres
• Steps awarding bodies are taking to reduce
  monitoring activity

39
Awarding bodies monitor centres for regulatory,
quality and commercial purposes
Post-approval (or ongoing) monitoring of centres
by awarding bodies
Post-approval monitoring of centres is defined
as The review of, and reporting on, the
centres quality assurance arrangements by the
awarding bodiesa
DEFINITIONa

• The purpose of awarding bodies monitoring of
  centres is to
• Maintain quality and standards in their
  qualifications
• Ensure that they are able to satisfy the
  regulators
• Assure their own reputation

PURPOSEb

• Contact with centres that occurs through some
  monitoring can also result in
• Support for programme leaders
• Relationship building
• Ensuring opportunities for learners

OTHER OUTCOMESb
Awarding bodies and others in the environment,
including centres, stress that awarding bodies
can provide significant support to programme
leaders as part of the monitoring process.
Source (a) Stakeholder consultation Centre
recognition and centre qualification approval,
November 2005 January 2006, Centre Recognition
Project, 11/2005 (b) Capgemini analysis based on
stakeholder interviews, Summer 2006.
40
Awarding bodies tend to define their monitoring
by qualification type, but different
qualifications are often subject to the same
types of monitoring
Approaches of awarding bodies monitoring centres
by qualification types
Typical associated monitoring
Main types of qualification
Assessment of the qualification

• Visits during examinations to check centres
  conduct of exams
• Analysis of exam outcomes
• Moderation of internally marked work in line with
  the code of practice1

• A general qualification (GQ)

• External assessment of learner achievement
  through examinations / tests
• Internal assessment of written work

• External verification of internal assessment and
  verification conducted in line with the code of
  practice1

• A national vocational qualifications (NVQ)

• Internal assessment of learn