Finnish Higher Education Evaluation Council FINHEEC Dr. Seppo Saari, docent www.kka.fi

1
Finnish Higher Education Evaluation Council
FINHEEC Dr. Seppo Saari, docent www.kka.fi

• Seminar
• South-Africa
• October 2006

2
Finland in the World
3
(No Transcript)
4
Outline

• FINHEEC's role
• Background for QA in Finland
• Audit framework
• Feedback from Audits

5
FINHEEC (1/5)

• Duties (Decrees 1320/95 and 465/98)
• Initiate evaluations of higher education and
  promoting its development
• Engagement in international evaluation
  cooperation
• Promote research on evaluation of higher
  education and
• Evaluation and approval of professional courses
  offered by higher education insitutions, entering
  of courses into a register as stipulated in
  Article 14 of the Decree on the Higher Education
  System, and maintenance of such a register.

6
Aims of FINHEEC (2/5)

• The most fundamental aim of FINHEEC is a
  long-term development of higher education through
  evaluations. For this reason, FINHEEC
• Supports higher education institutions while they
  design their own quality assurance and evaluation
  systems
• Supports the HE institutions own evaluation work
• Financial aid for projects
• Benchmarking
• Produces national data enabling international
  comparison of higher education institutions for
  policy makers, students, trade and industry.

7
Evaluations (3/5)

• Evaluations focusing on individual universities
  (20)/polytechnics (29)
• Institutional evaluations of both HE sectors
• Evaluations of quality work in polytechnics
• ------gt to be substituted by the auditing of
  quality assurance systems of higher education
  institutions
• Programme evaluations
• PhD education in Universities
• Masters programmes
• Media and communications studies
• Open university education
• Regional impact of HEs in the Lahti region
• Selection of students into universities
• Civil engineering

8
Evaluations (4/5)

• Education policy and other thematic evaluations
• Student as a citizen in HEIs
• Education and themes that have social or
  educational importance or that are rapidly
  expanding, developing or problematic
• Evaluation and registration of professional
  courses (accreditation)
• Evaluation assignments implemented on the order
  of universities/polytechnics and the Ministry of
  Education as a separate commercial
  service

9
Evaluations (5/5)

• Selection of quality units and centres of
  excellence
• Polytechnics
• Quality units of education (2000, 2002, 2005)
• Regional centres of excellence of polytechnics
  (2001, 2003, 2006)
• Universities
• Quality units of basic and/or continuing
  education (1997, 2000, 2003, 2007)
• Quality units of adult education (1997, 2000,
  2003, 2007)

10
Background for QA and audits (1/3)

• International development in the higher education
  sector
• Bologna declaration 1999 and the objective to
  create coherent European higher education area by
  2010
• Communiqué of the Conference of Ministers
  responsible for Higher Education in Berlin 2003
• The quality of higher education is at the heart
  of the setting up of a European Higher Education
  Area (development of quality assurance at
  institutional, national and European level)
• The need to develop mutually shared criteria and
  methodologies on quality assurance

11
Background for QA and audits (2/3)

• By 2005 national quality assurance systems should
  include
• A definition of the responsibilities of the
  bodies and institutions involved
• Evaluation of programmes or institutions,
  including internal assessment, external review,
  participation of students and the publication of
  results
• A system of accreditation, certification or
  comparable procedures
• International participation, co-operation and
  networking

12
Background for QA and audits (3/3)

• The aims of the quality audits are to
• create an effective national quality assurance
  system
• demonstrate the higher education institutions
  preconditions for quality in their activities for
  the national and international stakeholders

13
Quality Assurance in Higher Education (1/3)

• Quality assurance of the Finnish higher education
  is constituted of three elements
• National HE policy
• National evaluations
• HEIs' own quality assurance

14
Quality Assurance in Higher Education (2/3)

• National quality assurance signifies for the
  procedures, processes or systems, which aim at
  assuring the quality of higher education at a
  national level
• The main responsibility for the quality of
  education, RD, scientific research and other
  operations lies on the individual higher
  education institutions

15
Quality Assurance in Higher Education (3/3)

• The Ministry of Education instituted a working
  group in 2003 to analyse the present stage and
  development requirements for quality assurance in
  Finnish higher education
• The working group proposed that in view of the
  communiqué of the ministerial meeting in Berlin,
  Finland will implement a system of evaluating the
  quality assurance of universities and polytechnics

16
Requirements for QAsystems (1/2)

• The quality assurance systems should
• a) meet the developing quality assurance criteria
  of the European Higher Education Area
• b) be part of the operational steering and
  management system
• c) cover the entire operation of the higher
  education institution

17
Requirements for QAsystems (2/2)

• d) be interrelated as part of the normal
  operations of the higher education institution
• e) be continuous
• f) be documented and
• g) enable the participation of all members of the
  higher education community in quality work

18
Audit Framework (1/5)

• Each higher education institution has a quality
  assurance system that best suits its own
  operations and set aims
• The objective is to audit the quality assurance
  systems assuring the quality of the higher
  education institutions' operations, not the
  operations themselves or their outcomes

19
Audit framework (2/5)

• The main responsibility of the quality assurance
  systems remains in the higher education
  institutions
• The approach in quality audits is fitness for
  purpose
• quality assurance systems are fit for purpose in
  advancing the institutions mission and aims
• Continuous quality development and enhancement
  

20
Audit framework (3/5)

• The aims of the quality audits are to
• Support higher education institutions in the
  development of quality assurance
• Evaluate, how the quality assurance systems
  function in relation to the institutions aims
• Quality assurance systems include the procedures
  processes and mechanisms by which the
  institutions manage and develop the quality of
  the higher education
• Show for the national and international
  stakeholders the quality level of higher
  education in Finland

21
Audit Framework (4/5)

• The general aim is the continuous developing of
  quality
• The audits of quality assurance systems will
  replace former evaluations of higher education
  institutions carried out by FINHEEC
  institutional evaluations on both higher
  education sectors and audits of quality work in
  the polytechnic sector

22
Audit Framework (5/5)

• Consequences of audits two options
• Audit processes and methodology
• self-evaluation documents are consisted of
  existing documents (not prepared just for the
  audit)
• audit visit
• criteria for the members of the audit panel
• public audit report

23
Quality assurance system level
Quality assurance whole
Strategic planning
Setting of objectives
Resourcing
Leadership and management
Quality assurance of basic tasks
Degree education
Research/ RD
Societal interaction, regional development
(continuing education, open HEI teaching)
Support and service functions
HR development
24
What is being audited?
Coverage, functionality, openness and
communicativeness of the QA system
25
Auditing targets
1. Objectives, overall structure and internal
coherence of the QA system
5. Interface between the QA system and
management/steering
2. Documentation including the formulation of Q
policy and the definition of procedures, actors
and responsibilities
6. Relevance of and access to, QA information

• 3. Comprehensiveness of QA focusing on basic
  tasks and support and service functions
• degree education
• research/RD
• - interaction with and impact on society and
  contribution to regional development
• support and other services
• staff development

7. Relevance of, and access to, quality assurance
information for external stakeholders
8. Efficiency of QA procedures and structures and
their effect on the development of activities
9. Use of information produced by the QA system
as tools for quality management and enhancement
in education and other activities
4. Participation of staff, students and external
stakeholders in QA
10. Monitoring, evaluation and continuous
development of the QAsystem
26
Scaled criteria of targets

• The selection of the criteria is based on their
  relevance for the evaluation of the functioning
  of the HEIs QA system, leaving the HEIs scope of
  movement and the freedom of choice in the use of
  their own QA system
• The evaluation of the QA system is based on a
  scaled set of criteria focusing on four levels of
  development (absent, emerging, developing and
  advanced)
• Rather than being mere descriptions of
  operations, the evaluation criteria go deeper
• The HEI should be able to prove the functioning
  of the QA system and convince the external panel
  of its impact.
• Process criteria - content criteria
• Process criteria good/bad quality itself is not
  defined, HEI responsible for quality definition,
  audit focuses on processes
• Content criteria high-quality operations are
  defined based on the criteria.

27
When is quality assurance system sufficient?

• Premise is constituted by the objectives and
  criteria
• Evaluation of sufficiency is based on
• HEIs own justifications and evidence
• Core questions
• - how can the HEI know that its QA is working
  and how can the HEI prove it?
• - is the QA system serving the continuous
  enhancement (in the line with the strategy and
  objectives)
• Expertise and experience of the audit group
  members (HEI representatives, student
  representatives, practical employment sector
  work life representatives)
• Quantity quality of the QA procedures?

28

Targets used in auditing

• What are the elements included in the QA system?
  Does it comprise all core areas of operations?
• Is there coherence between the various elements?
  Are the actors aware of the links between the
  various elements?
• How does the system work in practice? Has the
  system been cascaded down to the organisation?

• 1. Objectives and overall structure of the QA
  system and its internal coherence

2. Quality-related documentation, including the
formulation of quality policy and definition of
activities, actors and responsibilities

• How and where have the functions, actors and
  responsibilities of the QA system been defined?
• Are the documents easily accessible?
• Are factual operations and division of labour in
  line with descriptions in documentation? How
  efficient is the division of responsibilities?

29
Targets used in auditing
3. Comprehensiveness of QA in terms of different
activities and processes relating to basic
mission 3 a) Degree education 3 b) Research /
RD 3 c) Interaction with and impact on society
contribution to regional development 3 d)
Support and other services (e.g. library and
information services, career and recruitment
services, and international services) 3 e) Staff
development

• Does QA comprehend all areas of planning,
  implementation and evaluation of education?
• Does QA comprehend all areas of research/RD?
• Does QA comprehend all areas of societal
  interaction and impact and regional development?
• Does QA comprehend all areas of support and
  service functions?
• Does QA comprehend all areas of HR development?

30
Targets used in auditing

• In what way do the members of the HEI community
  participate in QA? Are they committed and active?
  How is participation promoted and supported?
• students
• teaching staff
• support service staff
• researchers
• administration
• management
• What is the role of external stakeholders in the
  operation of the QA system?

• 4. Participation of staff, students and external
  stakeholders in QA

31
Targets used in auditing

• In what way have the HEI objectives been
  considered in the construction of the QA system?
• What is the link between the operation of the QA
  system and strategic planning?
• In what way is the information provided by the
  QA system utilised in the steering of operations,
  follow-up of results and development? What are
  the proofs of such utilisation of information?
• What are the links between the QA system and
  resourcing decisions?
• In what way is the QA system used as a tool for
  the steering of operations?
• Is the HEI able to use the information produced
  by the QA system to form an overall picture of
  the quality of its operations?
• Management commitment?

5. Interface between the QA system and HEI
leadership and management
32
Targets used in auditing
6. Relevance of and access to information
produced by the QA system within the HEI

• In what way has the planning of the QA
  operations focused on the relevance of the
  information produced by the QA system from the
  perspective of the HEIs internal actors?
• Have the information needs of different actors
  been taken into account?
• Are the actors aware of the operation of the QA
  system and its core results? Are communications
  active and frequent?
• Have the external stakeholders been defined?
• In what way has the planning of the QA
  operations focused on the relevance of the
  information produced by the QA system from the
  perspective of cooperation partners and external
  stakeholders?
• Are the communications active? Are they targeted
  at precisely defined groups?

7. Relevance and availability of information
produced by the QA system from the perspective of
external stakeholders
33
Targets used in auditing
8. Efficiency of QA procedures and structures and
their effect on the development of activities

• Functioning and impact of the QA procedures and
  structures (cfr. items 1 and 3a-e)
• Does the QA system include modes of operation and
  structures that promote new ideas and change?
  What are they like? In what way do they support
  the development of operations?
• Does the operating culture support
  innovativeness?
• In what way does the QA system help to identify
  insufficient quality?
• In what way and how systematically is the
  information produced by the QA system used for
  the management and enhancement of the quality of
  the basic tasks? Can the HEI show proof of that?

9. Use of information produced by the QA system
as a tool for quality management and enhancement
in education and other activities
34
Targets used in auditing
10. Monitoring, evaluation and continuous
development of the QA system

• What is the overall view of the HEI management,
  staff and students of the operation and impacts
  of the system?
• What are the means to follow and develop the
  system?

35
Degree education - examples of processes and
procedures

• Planning, implementation and evaluation of degree
  education
• Planning and approval of new education and
  training
• Application procedures, student selection,
  training need
• Curriculum process
• Teaching methods
• Learning environments
• Practical training
• Thesis
• Assessment of learning results
• Course of study, degrees completed, drop-outs
• Education-focused follow-up and evaluation
  procedures and their development

36
Examples of QA procedures included in degree
education (3a)

• Instructions and documents Planning of
  education, student selection, study guidance,
  implementation of education (e.g., teaching
  methods, training, theses), evaluation and
  development
• Gathering and use of evaluation data and
  feedback assessment of learning results, student
  feedback system, staff feedback questionnaires,
  feedback from practical employment sector or from
  alumni, gathering and use of statistics (e.g.,
  progress of studies, degrees, drop-outs),
  performance appraisals, feedback sessions,
  self-evaluations by degree programmes/departments,
  internal audits or cross audits, external
  auditing
• Other procedures supporting QA and development
• ways of organising operations participation of
  students and work-life exponents in curriculum
  development, innovative new ways to construct and
  perceive study entities (incl. those improving
  understanding and communicativeness),
  organisation of steering discussions,
  construction of personal study plans (HOPS)
• RD projects development of teaching methods or
  new evaluation methods

37
Examples of QA procedures included in
research/RD (3b)

• Instructions and documents related to
  research/RD as well as gathering of evaluation
  and feedback data
• Linking research/RD with teaching students in
  projects (theses, practical training), students
  in projects (e.g., as trainers, experts,
  post-graduate students)
• Participation by practical employment sector
  exponents and their influence through advisory
  committees
• Linking of research topics and projects to the
  overall strategy of the HEI

38
Examples of procedures related to societal
interaction, impact and regional development (3c)

• Open university/polytechnic teaching and
  continuing education operating
  instructions/documents as well as gathering of
  evaluation and feedback data (incl. training
  needs, marketing, student selection,
  implementation of teaching)
• Cooperation between HEI and external stakeholders
  in implementation of teaching and research/RD
  instructions, documents as well as gathering of
  evaluation and feedback data (numbers of persons
  or organisations, feedback from external
  stakeholders)
• Anticipating training needs regional development
  studies, gathering of forecast information

39
Examples of QA procedures included in support and
service functions (3d)

• Instructions, documents, gathering of evaluation
  and feedback data as well as QA and development
  data regarding, for example, the following
  support and service function processes
• - library and information services
• - career and recruitment services
• - international services
• - information management services
• - marketing and communications
• - administrative services (incl. financial
  administration, real estate services)

40
Examples of QA procedures related to HR
development (3e)

• HR planning and resourcing operating
  instructions and selection criteria for
  recruitments
• Competence and level of education of staff
  additional and continuing education of staff
  (e.g., pedagogic training), organisation of
  various thematic and development meetings and
  seminars
• Resources and job satisfaction of staff
  gathering and use of feedback, activities for
  maintained working ability, performance
  appraisals
• Leadership and management skills additional and
  continuing education

41
Auditing targets and criteria
1. Objectives, overall structure and internal
coherence of the quality assurance system
ABSENT
EMERGING
The HEI has no QA procedures in place relating
to its activities.
There are QA procedures relating to some HEI
activities, but they are neither systematic in
structure nor interlinked.
DEVELOPING
ADVANCED
QA covers many of the HEI's activities and the QA
procedures form a fairly efficient system.
QA covers all or nearly all activities, and QA
procedures processes form a dynamic whole.
42
Auditing targets and criteria
2. Documentation, including the formulation of
quality policy and definition of procedures,
activities, actors and responsibilities
ABSENT
EMERGING
Quality policy, activities, actors and
responsibilities have not been defined or
documented.
The definition and docu-mentation of
responsibilities and activities included in the
QA system are inadequate, and the procedures are
inadequately organised.
DEVELOPING
ADVANCED
Activities, actors and responsibilities are
clearly, comprehensibly and concr-etely defined,
and the documentation is easily available to all.
The organ-isation is well-designed.
The activities and division of labour manifestly
comply with the documented processes. The
organisation is extremely well-designed and
reinforces QA.
43
Auditing targets and criteria
3. Comprehensiveness of QA in terms of different
activities and processes relating to basic
mission
ABSENT
EMERGING
No QA in the activities and processes relating to
basic mission.
The system covers isolated activities and
processes mainly relating to degree education.
DEVELOPING
ADVANCED
The system covers many activities and processes
relating to basic mission.
In the main, the system covers activities and
procedures relating to the basic mission.
44
Auditing targets and criteria
3 a) Degree education
ABSENT
EMERGING
No QA in degree education.
QA covers some isolated aspects of the planning,
implementation and evaluation of degree education.
DEVELOPING
ADVANCED
QA covers several aspects of the planning,
implementation and evaluation of degree
education.
QA covers all salient aspects of the planning,
implementation and evaluation of degree
education.
45
Auditing targets and criteria
6. Relevance of and access to information
produced by the QA system within the HEI
ABSENT
EMERGING
QA does not cater for internal stakeholders and
information is not communicated within the HEI.
The production of information is not systematic
and does not sufficiently cater for internal
stakeholders.
ADVANCED
DEVELOPING
The operation of the QA system is overt and
transparent. Internal communication relating to
QA is active and information is systematically
communicated and targeted to different parties
within the HEI. The relevance of information to
internal stakeholders is an important
consideration in the planning and continuous
development of the QA system.
The activities and key results of the QA system
are known to internal stakeholders. The system
produces information relevant to them.
46
Two levels of focus in auditing

• Starting point is the HEIs own system, developed
  on the basis of its premises.
• Through this approach, it is possible to
• promote the development of HEIs own quality
  culture, and
• promote an active and comprehensive participation
  of the HEI, and
• enhance the sense of QA ownership among the HEI
  actors.
• Object of audit constituted by QA processes and
  procedures (fitness for purpose perspective) -
  audit does not address HEIs objectives,
  operative contents and results per se.
• Two levels of focus
• QA as a whole
• QA of the basic tasks

47
Challenges

• Combining the two levels of focus and related
  challenges (cfr. total quality and quality of
  degree education).
• How to address the real quality of education?
• Quality cultures in administration and operative
  control as well as on the grass-root level
  understanding and combining different
  perspectives (cfr. objectives of second visit
  day)
• Links between the QA procedures included in the
  HEIs basic tasks
• Sufficient scope and depth of auditing,
  especially as concerns large HEIs, is a challenge
  for the audits. Audit efficacy is influenced by
• relevance of auditing criteria
• auditing methods (scope and depth of audit
  material, implementation of audit visit)

48
Three main areas in recommendations

• Functionality of QA system
• Comprehensiveness of QA systems
• Efficiency of QA system

49
Thank you for your attention!