National Developmental evaluation

1
National Developmental evaluation

• Kauko Hämälainen
• Heikki K. Lyytinen
• The Finnish Education Evaluation Council
• The Miracle of PISA Conference 10. -12-Sept. In
  Helsinki

2
Content

• 1.Where did we come from and where are we going
  to in the national evaluation?
• 2. Within what kind of administrative and
  organizational structure we act?
• 3. What kind of demands/requirements we make for
  the national evaluation?
• 4. What are the values and principles we set on
  the evaluation?
• 5. How do we carry out the developmental
  evaluations in practice ?
• 6. What have been our evaluation themes?
• 7. Examples of evaluation practices (evaluation
  projects in practice)

3
Change of Educational Steering System in Finland
Situation in 1970s and 1980s Situation in 1990s/2000
Centralised control and decision-making Centralised control and decision-making Devolution of power
centralised curricula long-term plans budgeting based on expenditures controlling follow up external evaluation inspections conducted by state authorities etc. self-governance school-based curricula distinctive educational profile of educational institutions self-direction and -regulation learning organisation as a model of functioning self-evaluation and own control performance-based funding etc. new evaluation organisations National evaluation plan
4
Evaluation instead of inspection

• no separate school inspectorate and inspection
  visits to schools
• the activitis of education providers are quided
  by objectives laid down in legislation and the
  national core curricula
• the system relies on the proficiency of teachers
  in their efforts to meet the objectives
• strong focus on self-evaluation and external
  evaluation

5
The Forms of Steering and Improving the Quality
of Education in the Finnish School System Focus
in steering by information, monitoring and
evaluation and steering by professional expertise

• Steering by information, monitoring and evaluation

Steering by professional expertise freeing the creative resources of the school Aim improved quality of education Market-driven steering competition between schools
Norm- and resources-based steering
6
Challenges for renewing evaluation in educational
services

• responsibility to organise educational services
  also includes evaluating them
• directing educational services requires goals
  and evaluating how well they have been met
  (so-called profit responsibility)
• evaluation seen as a comprehensive process (cf.
  e.g. quality assurance)
• evaluation as a tool for learning and
  development( developmental evaluation)

7

• evaluation adds to the active role of
  actors(empowerment)
• knowledge of the current state of affairs
  essential when setting goals for development
• controlled change when steering towards future
  goals (controlled change based on facts/evidence
  based cahnge management)

8
TOWARDS NEW EVALUATION CULTURE

• 1 Explorative/evaluative work community
  EVIDENCE BASED
• continuing evaluation transparency
• making use of research and evaluation knowledge
• 2 Reflective work community - towards culture of
  EMPOWERMENT
• independence, motivation, atmosphere,
  appreciation, security
• 3 Network-oriented operation cultures
• 4 Making use of parallelisms
• peer evaluation, benchmarking
• 5 From data management to knowledge management
  collective awareness
• 6 Demands for involvement, multiple perspectives,
  and consensus authentic dialog
• 7 Managing the whole of evaluation

9
Towards a new evaluation system
Change
Transition period 2-3 years
New evaluation awareness 2-3 years
Continuous development (new evaluation
culture) Change of evaluation culture Towards
developmental evaluation
External evaluation system of expert networks
Internal administrative evaluation system
Administrative dependence (internal evaluation)
Expert network based independence
(external/internal)
10
Benefitting from the network both in evaluation
and in project planning

• Evaluation projects are organised in two phases
  to maximise
• best expertise
• commitment of the field, partners and interest
  groups
• independence and
• impact

• 1. Planning group
• expertise of the field and interest groups
• draws up a plan and schedule for the project and
  its different stages
• proposes assessors to be appointed in the
  evaluation group to the Council
• 2. Evaluation group
• independent experts who work in assessment and
  research
• evaluate and draw conclusions with further
  recommendations
• write a report and are responsible for it

11
National production of evaluation information to
serve policy-making
Parliament
The Government
The Ministry of Education
The other evaluation organisations (national
and international)
The Finnish Higher Education Evaluation Secreteria
t 12 members
The Finnish Education Evaluation Council 14
members
The Education Evaluation Secreteriat
The Evaluation Secreteriat
National evaluation and information production
network
National Board of Education Education Providers
Private Evaluation Organisations
- Regional Authorities - Provincial State
Offices - Provincial Councils - Employment and
EconomicDevelopment Centres
Universities Polytechnics Statistics Finland Res
earch Institutes in Univeristies
Schools
12

• EducationEvaluation council in conjunction with
  the Ministry of Education since April 2003
• planning, co-ordinating, managing and
• developing the evaluation of basic education
• and upper secondary education and training
• The polytechnics and univeristies are responsible
  for the evaluation of their own operations and
  outcomes (support from the Higher Education
  Evaluation Council)

13
(No Transcript)
14
The Finnish Higher Education Evaluation Council
(FINHEEC)

• (FINHEEC) is an independent expert body assisting
  universities, polytechnics and the Ministry of
  Education in matters relating to evaluation.
• The evaluations conducted by FINHEEC can be
  divided into three rough categories
• Evaluations of higher education institutions
  Institutional evaluations (universities and
  polytechnics), audit of quality work, and
  evaluations for the accreditation of the
  polytechnics
• Programme and thematic evaluations
• Accreditation of professional courses offered by
  higher education institutions

15
NATIONAL EVALUATION SYSTEM
Indicators -quantitative -qualitative
Learning outcomes
Situation and system- evaluations
Thematic evaluations
International cooperation PISA Copenhagen process
Methods and models
Self-assessment Quality control External and
internal audits Paperless enviroment Indicators

• General qualifications
• learning to learn

General knowledge/ School subjects
Professional competence
Evaluation research
16
AIMS OF EVALUATION IN THE FINNISH EDUCATION SYSTEM
The aims of evaluation are to gather and analyse
information which serves educational
decision-making at the national policy and local
levels and contributes to the development of
education in general. In addition, the aim is
to support students learning and the work of
educators, as well as institutional development.
17
Educational evaluation Focus on many levels

• Individuals (pupils, students, teachers)
• Classrooms/courses
• Curriculum/programmes
• Organizations (schools, universities)
• Fields (all schools in a municipality, all
  programmes in a discipline)
• The national level (national quality development
  and quality assurance systems)
• The international level (PISA,IEA, EQUIS in the
  business school area)

18
Regional evaluationThe Provincial Government
Act (22/1997) obliges the provincial governments
to evaluate the accessibility of basic services
within their borders ? The Ministry of the
Interior and other minitstries define shared
evaluation targets for a number of years (i.e.
the term of the government) ? evaluations
should be carried out co-operatively,
consistently, and drawing primary on existing
statistics and department specific databases
19
DISTRIBUTION OF WORK AND CO-OPERATION IN
EVALUATION

• Ministry of Education and the Education
  Evaluation Council are responsible for external
  evaluation and its development. Education
  providers are primarily responsible for local
  evaluation and its development.
• The National Board of Education
• participates in evaluating syllabuses and
  learning
• achievements within degree regulations
• as a part of educational evaluation expert
  network
• Links between external evaluation, local
  evaluation and evaluation of basic services
  should be taken into account when drawing up
  evaluation programmes and plans.
• Educational institutions and providers who have
  participated in the evaluation must have access
  to the evaluation reports on their own
  activities.

20
EVALUATION OF LEARNING ACHIEVEMENTS

• The National Board of Education
• participates in evaluating syllabuses and
  learning
• achievements within degree regulations
• as a part of educational evaluation expert network

21
Evaluation of education in The Finnish National
Board of Education

• quantitative and qualitative indicators will be
  defined
• information in the form of indicators will be
  produced
• learning achievements will be compared
• learning achievements will be evaluated in
  different school types and levels as follows
  (sample-based approach)
• basic education mathematics and mother
  tongue, regularly
• basic education other subjects and
  themes (for ex. sustainable development)
• methods of evaluation will be developed

22
REQUIREMENTS FOR EVALUATION SET BY THE
PARLIAMANENT EVALUATION POLICY)

• strengthening the role of evaluation in
  developing education, adds the neutrality of
  national evaluation of education and
  promotesco-operation and communication between
  actors in evaluation and interest groups
• promotes equality
• main emphasis should be in supporting the
  development of teaching and learning at school
  and municipality level
• education providers should be given priority
• evaluation must concern the whole learning
  environment

23

• Organising evaluation
• evaluation adequately independent of the rest of
  the education administration
• independent and impartial evaluation actors
• increasing the impact of evaluation especially
  from the viewpoint of educational institutions
• a system must be established to monitor the
  impact of evaluation

24
VALUE BASIS OF EVALUATION

• Equality means promoting educational equality by
  means of evaluation.
• Fairness means creating an ethically sustainable
  foundation for evaluation and refraining from
  comparison which may be harmful for the target of
  evaluation.
• Truthfulness means high ethical responsibility
  and critical examination, as well as optimisation
  of the reliability of evaluation.

25
Principles for educational evaluation

• Evaluation is based on education policy decisions
  (development aspect)
• The legitimacy is founded on educational
  legislation
• Education evaluation should be transparent
• The essential outcomes need to be published
• Methods used are to be recorded and justified, as
  reliable, valid and comparable as possible

26

• In this regard, evaluation must provide
• central and local administration with valid
• data on how succesfully educational
• objectives are being achieved. Thus,
• evaluation should foster self-control and
• self regulation at the level of central and
• local administration and promote
• accountability to the various parties
• involved.

27

• From the perspective of educational policy,
• it is important to ensure that education
• providers deliver the educational services
• that they are intended to produce and that
• they do this in accordance with the goals
• set for these services and on the basis of
• professional skills of adequately high
• standard.

28
Evaluation process and decision making
Publishing a report
EEC
Quality assurance
Writing a report
Decision on evaluation
Evaluation visits Interpreting results
Appointing the planning group
1.
feedback
4.
EVALUATION GROUP
PLANNING GROUP
INSTITUTIONS Self-assessment, other material
Planning the evaluation
Accepting the plan Appointing the evaluation
group Accepting methods and measures and
self-assessment models
Evaluation group proposal Developing data
gathering and feedback systems Drawing up
instructions for self- assessment Evaluation
visits and panels
3.
2.
EEC
29
THE IMPLEMENTATION OF EVALUATION (PLANNING AND
EVALUATION NETWORK)
30
Implemented evaluation projects

• Basic education
• 1. The implementation and quality of basic
  security in education
• 2. The implementation, availability and quality
  of special needs education, remedial teaching and
  student welfare services in basic education
• Upper secondary education
• The need for special needs education in upper
  secondary school education

31
Adult education

• Liberal Adult Education Institutional Structure,
  Service Ability and Societal Effects
• Training Opportunities and Provider Network of
  Vocational Adult Education on the Secondary Level
• Both projects long lasting three years in two
  phases (2004-2005, and 2006-2007)
• Based on available data statistics, research
  reports, documents, the variety of material
  produced in various development projects etc no
  primary data gathered
• Process (1) multiperspective planning group
  comprised of education providers and relevant
  interest groups, (2) independent, scientific
  research oriented evaluation group

32

• Basic education
• Teaching methods in the comprehensive school from
  January 1st 2007 to December 31st 2007
• Upper secondary education
• Collaboration in upper secondary school education
  and vocational education from October 1st 2005 to
  October 10th 2006
• Vocational education
• Workplace learning from Sebtember 2004 to
  October 2006
• The regional effectiveness of vocational
  education from January 1st 2006 to December 31st
  2007
• Key competencies in vocational education from
  January 1st 2007 to December 31st 2007
• Adult education
• The competence-based examination system

33
Different evaluation frameworks
34
EFQM EXCELLENCE MODEL
Action 50
Results 50
Leadership Management 10
Processes 14
Key Performance Results 15
Personnel (HR) 9
Personnel Results 9
Customer Results 20
Policy and Strategy 8
Society Results 6
Partnership and Resources 9
Innovation and learning
35
A Common Quality Assurance Framework
Model (Common Core Criteria)
Gains Mutual Trust Trancparence Credit Transfer
Methodology (Self assement/Peer Review)
Measurement (Indicators)
Monitoring (External)
Quality assurance methods and process,
indicators, standards and norms, approach of
selfevaluation
36
QUALITY ASSURANCE MODEL EUROPEAN COMISSION
(LISBON STRATEGY)
PLANNING i.e. strategies, goals, objectives,
resources, anticipation and evaluation
mechanism
IMPLEMENTATION i.e. action plan, resources,
partnerships, leadership, process management,
training of trainers, didactical materials
METHODOLOGY quality management, self
evaluation, stakerholderss role management of
information external evaluation
REVIEW feed back and prosedures for
change continue improvement development
MONITORING AND EVALUATION i.e. self evaluation,
external inspection, internal quality control
Results of teaching, training and learning
staff-oriented results, key performance results,
sosietal results
37
Examples on evalaution projects
38
EVALUATION MODEL FOR LEARNING ON THE JOB LEARNING

• Quantitative indices as part of self-assessment

• 2 Self-assessment by education providers
• multisectoral evaluation
• consensus

• 4 Description of best practices
• - Portfolio

5. Stakeholder questionnaire education
committees
web-based evaluation environment
Backgound data
Initial assessment by an evaluator team
In-depth analysis of self-assessments, and
related visits to educational institutes by the
evaluator team
5. Authority interviews
Field-specific evaluation seminars
Provider feedback Final evaluation and evaluation
report
39
FROM GOALSTO INTERACTIONThe evaluation of
basic education pedagogy
40
What was evaluated?

1. Key principles of pedagogy
2. Teachers goal awareness
3. Range of teaching methods
4. Attendance to the pupils unique characteristics
5. Equality of learning processes
6. Functionality of the learning environments
   factors
7. Teaching, studying and ???

17.9.2015
40
41
How was information produced?

• Survey of teachers (N 2351)
• Survey of rectors (N 410)
• Visits to schools ( N 12)
• Hearings ( N 12)
• Data and research results from previous
  statistics, studies and evaluations

17.9.2015
41
42
TEACHERS IMPORTANTPEDAGOGIC PRINCIPLES

• Fairness
• Student oriented approach
• Goal awareness
• Importance of classroom piece
• Stressing action and variation
• The objectivity of pedagogic thinking
• Socio-constructivism has its opponents

17.9.2015
42
43
TEACHERS GOAL AWARENESS

• The status of curriculum is significant.
• Goal awareness highlights many aim fields of
  education and upbringing.
• Text book still has quite a significant role.
• Critical approaches to curriculum
• Wishes for time division (lesson plan)
• Goal awareness given contemplation on aims thin
  

17.9.2015
43
44
TAKING STUDENTSINTO ACCOUNT

• Students appreciated as individuals
• Equity central / significant
• Students differences and needs important
• Student oriented approach has increased
• Making use of students activity
• No new remedies for bullying.
• Few means of managing disruptive students.
• Special needs students cannot always be
  supported.
• Possibilities of differentiation limited in
  practice.

17.9.2015
44
45
RANGE OF TEACHING METHODS

• The range and variety of teaching methods is
  highlighted.
• Externally most often teacher oriented teaching
  with student oriented elements.
• Instrumental use of teaching materials is
  becoming more common.
• Information technologies have enriched working
  methods and materials.
• Examining and experimental methods of working not
  usual.
• Minor changes in teaching.

17.9.2015
45
46
PREREQUISITES FOR ACTION

• A positive atmosphere for development and shared
  perception of teaching positive (rectors view)
• The pedagogic know-how of teachers good in this
  country.
• Technical equipment and options for teaching
  material are usually of good quality.
• Great variation in the preconditions of action
  between schools.
• Large and heterogeneous teaching groups is a
  problem.
• Too much variation in the resources for
  complementary education.
• Haste and the feeling of haste in school work
  have increased
• Messages of problems with air-conditioning and
  schools with mold and mildew problems!

17.9.2015
46
47
THE IMMEDIATE CHALLENGES OF PEDAGOGY?

1. Large heterogeneous groups (taking into account
   the broad questions of differentiation)
2. The frequency of class-room disturbance
   remedies?
3. The needs of developing self-evaluation how?
4. The curriculum includes too much content, too
   demanding goals, and too high criteria for good
   learning outcomes, the instructions for
   evaluation call for clarifications.
5. Teachers supplementary education has to be taken
   more seriously more time and funding, more
   equity, more long-term planning, teacher oriented
   approach.

17.9.2015
47
48
LONG-TERM NEEDS IN THE DEVELOPMENT OF PEDAGOGY

1. Supporting pedagogic leadership at schools
2. General education in the future launching the
   discussion
3. What are the contents?
4. How do we learn?
5. What do we evaluate?
6. Enhancing goal awareness and pedagogic thinking
7. levels
8. steps of advancement
9. goals
10. Increasing the amount of the studying, evaluating
    and developing approach in the teachers work
    in the work communities
11. Enhancing the understanding and encountering of
    the students differences, and the pedagogy of
    developing.
12. Enhancing learning as collective construction of
    knowledge
13. Enhancing the appropriation of media, information
    technologies, and information flows as part of
    e.g. developing the learning to learn skills
14. Increasing the amount of interaction with the
    surrounding community /society (e.g. childrens
    and young peoples lifestyles and cultures)
15. Developing the evaluation skills (of the schools
    / teachers /students)

17.9.2015
48
49
DISSEMINATION OF EVALUATION INFORMATION

• Publication / announcements
• Report, feedback for organizers?
• National, regional, local events
• Articles etc.
• Developing the overall evaluation process from
  the point of view of making use of information
  (collaboration, networking, participation, client
  oriented approach)

17.9.2015
49
50
The process in short

• The evaluation group
• brings all information together
• describes the state of affairs
• defines the criteria
• evaluates the state of affairs
• points out the needs of development
• makes some general recommendations
• takes part in the distribution of the results