WP4: Validation

1
WP4 Validation

Andrea Kárpáti, ELU (karpatian_at_axelero.hu) In
cooperation with Roger Blaimire,
EUN(roger.blamire_at_eun.org)
2
Validation framework

• Levels of validation
• Systems
• Institutional - school level
• Pedagogical - teacher/student level, educational
  usability
• Technical - performance of project deliverables
  versus school insfastructure technical usability
• Economic sustainability, dissemination
• Cultural/Linguistic how content, technology,
  inherent teaching style style etc. Travel
• Methods of validation
• Usability tests National (pre)pilots, (inter)
  national workshops, (inter) national discipline
  based collaborative groups
• Interviews semi-structured individual, focus
  groups
• (Pre)pilots, school based usability tests
• Workshops (inter)national

3
Institutional / school level

• Assessment criteria
• e-maturity
• change in teaching and in collaboration
• dissemination
• involvement of wider community
• change in school ICT culture communication and
  management)

• Assessment instruments
• ICT Competence Inventory (ICT/CI)
• Educational Strategies Questionnaire (ESQ)
• Focus group interviews (teachers, heads, students
  of same school)

4
Pedagogical - teacher/student level

• Authoring, adapting, modifying, sharing of LO
• Implementation in classroom teaching
• Impact on practice e.g. the role of teachers,
  teaching strategies, teaching styles
• Discipline-related experiences Math, Science,
  Environmental Science, English as Foreign
  Language.

• Lesson plans online template (lesson type,
  content, ICT and traditional tools used,
  pedagogical strategy, evaluation methods etc.
• Blog online diary on working with Calibrate
  tools and their utilization experiences,
  problems, desires for further development
• National and international fore for
  discipline-based teacher groups
• Focus group interviews (teachers of the same
  discipline from different schools)

5
Technical - performance of Calibrate tools and
the LRE

• How deliverables work in practice
• Implications for ICT resourcing in the school
  Localization issues 1 technical aspects
• Technical support
• Integration with existing ICT technology
• Adaptability to school needs
• Economic
• Cost for schools integration with existing
  environment, training, technical support

• (Pre)pilots
• Blogs
• International forum for piloting teachers
• SUSTAINABILITY STUDY I
• school visits with interviews one year after
  project ends
• ICT infrastructure survey at the beginning of the
  project and one year after finish
• Interview with school leadership

6
Cultural/Linguistic

• Localization issues 2 cultural aspects
• Support for multiple languages
• Intercultural (what aspects of national
  educational culture does the project support and
  what not?)
• Trans-national collaboration
• Culture of education (attitude to innovation,
  ethos of profession, private and public sector)
• Attitude of stakeholders (students, researchers,
  support staff)

• International workshop for teachers with focus
  group interviews and collaborative LO design
  experiments
• Interviews with national educational policy
  makers experts for (ICT policy, European
  harmonization of curricula,
• SUSTAINABILITY STUDY II
• Attitude surveys (participating teachers, heads
  and peers at piloting schools, other
  stakeholders)
• On-site observations peer review visits

7
Timeline for Validation 1

• January 2006 disciplinary areas for validations
  selected, curriculum analyses
• January March 2006 selection of piloting
  school / teachers
• May 2006. 1. pre-pilot
• May-September 2006 elaboration / discussion of
  validation framework
• September November 2006 teacher training where
  needed, pilot of validation instruments
• October December 2006 elaboration of training
  material for validating teachers
• December 2006 2. pre-pilot, finalisation of
  validation plan and instruments

8
Timeline for Validation 2

• January 2007 teacher training courses end with
  module on CMS, with special focus on LRE and
  Toolbox
• February May 2007 Validation Round 1 school
  usability studies followed up through lesson
  plans, interviews and site visits
• June-July 2007 summary of VR1, further
  development / eventual improvement of tools,
  planning for VR2
• September December 2007 VR2
• January February 2008 summary of VR2,
  finalisation of Calibrate tools, suggestions for
  dissemination

9
Prepilot 1 First impressions

• Aim test basic functionalities of LRE
• Methods guided walkthrough, structured
  questionnaire for individual walkthrough followed
  by plenary discussion all video/audio-taped
• Participants 2 teachers / country from 6
  countries, LRE and Toolbox designers, WP4
  coordinators
• Results design and help improved, relevance of
  functions for teachers clarified, further
  functions planned co-evolutionary design

10
Prepilot 2 User test

• Experienced users (teachers of the 1th pilot) and
  newcomers to Calibrate tools piloted, 2x2
  teachers / 10 piloting countries
• Self-report questionnaire filled out while
  performing tasks in the LRE
• Topics
• Ease of use (navigation, performance of tasks)
• Visual aspects (pleasingness, communication)
• Pedagogical aspects (relevance of tools /
  content)
• Suggestions for tool design
• General attitudes / feelings / motivational level
  changes (moods) while using tools
• Summary by ELU will compare expert and novice
  users attitudes, usage patterns, levels of
  involvement, critical observations and
  satisfaction

11
Teacher Training for Calibrate

• Methods and content varies acc. to participating
  country / teachers selected. Common core using
  learning content management systems incl. LRE and
  Toolbox a unit developed by ELU
• Phase 1, preparation for pilot collaborative
  learning in school-based, national groups, theme
  ICT in education
• Phase 2, pilot knowledge building community of
  teachers discipline-based work in
  (inter)national groups
• Course types
• accredited national training courses
• crash course on LRE and Toolbox
• EPICT the European Pedagogical ICT Licence

12
Educational Strategies Questionnaire

• F1 Aims and objectives of school education
  present and future
• Values, attitudes and beliefs about teaching and
  education
• The Role of school
• F2 ICT at school function and mission
• Existing good practice / problems
• ICT development plan
• F3. The role / task / mission of the teacher
• F4 Own educational practice
• Teaching environment / infrastructure
• Paradigms, strategies, methods
• Preparation for teaching
• F5. Opinion / plans / wishes about in-service
  training
• F6. Innovation experiences
• F7. Personal efficiency / resilience / coping
  with change

13
ICT Competency Inventory (ICT-CI)

• A self report questionnaire piloted in a project
  measuring ICT readiness in about 150 schools of a
  district in Budapest. The ICT-SI is an index
  based measuring tool. (The concept of indexing is
  used for the data reduction method employed
  several variables are composed in one index.) Our
  test comprises the following indices
• Index (1) School access to ICT
• Index (2) Home access to ICT
• Index (3) ICT related attitudes
• Index (4) ICT related competencies
• Index (5) Educational use of ICT

14

• Four domains for every index
• low (value range 0,011),
• medium (value range 1,012)
• high (value range 2,013)
• 0 value, indicating the non-existence of any
  interpretable data for the index. (E. g., low
  level ICT access means sporadic, rare access
  while 0 level indicates no access to a PC).
• Items correlated with the presence / absence of
  certain personality traits measured by the
  California Personality Inventory (CPI) to
  identify
• teacher profiles that tend to success or fail
  with ICT
• Individual training / coaching needs

15

16
Personality tests for better validation results?

• Coaching -. A generally accepted performance
  enhancement method in industry and commerce, very
  rare in education
• EPICT (European Pedagogical ICT Licence),
  www.epict.org nationalization and adaptation
  usability at stake
• CPI was used to define basic personality traits
  relevant for the world of work (sociability,
  innovativeness, coopeartion, assertiveness etc.)
• CPI results were correlated with ICT Competence
  Inventory and Education Strategies Questionnaires
• LEARNER PROFILES defined facilitators could
  improve mentoring and customize assessment
• BFQ to be used for Calibrate in Hungary and
  offered for other participants for consideration

17
EPICT pilot in Hungary Strong correlation
between personality traits and ICT skills,
attitudes and competence
18
Pearson two-tailed correlation, significance
levels 0.05, 0,01
19

• Correlation analysis of CPI and ICI findings,
  questionnaires and semi-structured interviews,
  show personality features that are associated
  with success and failure to acquire ICT skills on
  a level sufficient for educational use
• Some of these variables may be affected by
  specially targeted mentoring methods, teaching
  environment course content (best practice
  examples) custom-made skills training.
• Based on EPICT adaptation results in Hungary,
  in-service training of teachers may be improved
  to suit not only the initial level of ICT skills
  but also the mindset and personality of teachers.