Best practices in continuous evaluation: Evidence-Based Decision Making and Admissions Models

1
Best practices in continuous evaluationEvidence
-Based Decision Making and Admissions Models

• Admissions criteria and success criteria
• ETS Europe Jenny Dalalakis jdalalakis_at_etseurope.
  org
• January 2006 Gerben van Lent gvanlent_at_etseurope.
  org

2
Key Messages

• Admissions Criteria are the outcome of a decision
  process which must be regularly evaluated
• The process has the following components
• Strategic objectives of the institution
• Instruments to identify student quality
• Student application model
• Admissions criteria

3
Objective of this presentation Introduce Best
Practices in Admission Evaluations
Why Admit?
4
Admissions Best Practices
Why Admit? Consensus on Strategic Objectives
5

Outline
Determining Institutes Strategic Objectives for
Admissions Deciding on Qualities of Student
Body Evaluating Application Models Deciding on
Key Admissions Criteria Wrap up Conclusions and
Questions
6
Strategic Objective Successful fit between
Policy and Admissions Process

• College Board research identified best practices
  as
• Consensus on admissions objectives
• Explicitly affirmed and revised periodically
• Quantitative and qualitative criteria of incoming
  student body
• Consensus on the qualities sought in student body
• Board, staff, alumni, current student body,
  labour market representatives
• Academic and non-academic qualities of admired
  students
• Acceptable instruments to measure these desired
  qualities
• Admissions criteria
• Institutional objectives supported by individual
  and group qualities of student body
• Reflected in application components (explicit)
  and underlying factors (role of informal
  priorities)
• Evaluations and processing models
• Institutional goals supported in transparent
  manner
• Institutions Policy affects weight given to
  student quality measurements
• Profile of student balanced among different
  qualities sought
• Fairness
• relevance of information, equal access,
  transparency of admission process
• Validity of admissions process

7
Admissions Best Practices
Why Admit? Consensus on Strategic Objectives
8
Strategic Objectives Why Admit?

• ACA Research identified 6 key objectives
• Policy
• Revenue
• Profile
• Sustainability
• Non-discrimination
• Demand
• (Hybrid needs typically exist)
• Source The Admission of International Students
  in Higher Education, ACA Paper on International
  Cooperation in Education (2004)

9
Why Admit? Key objective Development Policy

• Institution Policy or Government Policy requires
  foreign students
• France Prestige in educating other countries
  elite students and future leaders
• Netherlands Manner of providing aid to
  developing countries
• UK, US Foreign students become host countrys
  goodwill ambassadors when they return home
• EU Support for mobility (Bologna, Lisbon) means
  international student exchanges are vital at
  Higher Education level
• Source The Admission of International Students
  in Higher Education, ACA Paper on International
  Cooperation in Education (2004)

10
Why Admit? Key objective Revenue

• Market-driven need for funds related to foreign
  students
• Public and private schools may charge more for
  all or some international students directly so
  the extra income is welcome
• UK, US
• CH (MBA programme students specifically)
• NL (English-taught programme students
  specifically)
• Public university with foreign students may
  qualify for extra government funds
• Australia
• Programme survival in danger and loss of staff so
  foreign students help numbers
• SE, CH, NL
• Disincentives may apply if institution does not
  qualify for extra funds if they have foreign
  students
• CH
• Source The Admission of International Students
  in Higher Education, ACA Paper on International
  Cooperation in Education (2004)

11
Why Admit? Key objective Institutional Profile

• International profile is of intrinsic value for
  Institution
• Foreign students enhance innovation and
  modernization
• Global pool better for choosing world-scholar
  quality
• Minimum or maximum target may be set AU
• Global recognition of host university for its
  student quality is very competitive
• Competition at level of Institution/ Discipline
  or at level of Labour Market Demands
• Nature of programme requires international
  approach
• Research literature is in English (MBA,
  Engineering) FR
• Applied Languages, International Relations,
  International Law
• Foreign students bring the world home to remote
  areas
• AU, SE
• Source The Admission of International Students
  in Higher Education, ACA Paper on International
  Cooperation in Education (2004)

12
Why Admit? Key objective Sustainability

• Securing the future of the programme, staff,
  research
• Smaller countries do not have enough local
  students for keeping some disciplines alive at
  their universities
• NL, SE, CH need Science students
• Students coming for Masters programmes are
  encouraged to stay on for PhD programmes to
  secure enough teaching staff assistants and
  research assistants
• Source The Admission of International Students
  in Higher Education, ACA Paper on International
  Cooperation in Education (2004)

13
Why Admit? Key objective Non-discrimination

• No discrimination is tolerated between any
  student groups
• Quality of students comes first regardless of
  students origin
• Important objective for highly selective /
  private institutions US
• Some are more equal than others (informally or
  formally)
• CH evidence required that foreign student could
  have studied back home
• FR to protect local student quota, local
  students chosen with earlier deadline for public
  university places
• SE first choose local students and unfilled
  quota goes to foreign students
• Source The Admission of International Students
  in Higher Education, ACA Paper on International
  Cooperation in Education (2004)

14
Why Admit? Key objective Students Demand

• Large number of foreign students want to come to
  the Institution
• Local, regional and global mobility (economic,
  academic, professional) is a growing demand by
  students, governments, labour market so students
  expect international programmes in countries of
  interest to them
• Universities receive large number of foreign
  student applications and react by starting
  international programmes and (hopefully) support
  programmes
• Source The Admission of International Students
  in Higher Education, ACA Paper on International
  Cooperation in Education (2004)

15
Admissions Best Practices
Why Admit? Consensus on Strategic Objectives
16
How to Admit? Look for QualityWhat is Student
Quality?

• ACA research showed a variety of instruments for
  gathering evidence on student quality
• Secondary school/ Bachelor diploma
• Transcript of courses
• List of completed courses
• Motivation statement/personal statement
• Letter of recommendation
• Proof of work experience
• Proof of previous academic work
• Curriculum Vita
• Standardized tests for subject and skill
  competencies
• Proof of knowledge of the teaching language
• Right to access Higher Education in home country
• Source The Admission of International Students
  in Higher Education, ACA Paper on International
  Cooperation in Education (2004)

17
How to Admit? Look for QualityWhat is Student
Quality?

• College Board research on factors used in making
  admissions decisions
• Academic Achievement, Quality, and Potential
• Direct measures (coursework, grades, test scores)
• Caliber of high school/ previous study
  institution (rank of class, school)
• Evaluative measures (depth and breadth of
  interests, world awareness, communication skills)
• Non-academic Characteristics and Attributes
• Geographic (remote or disadvantaged region, local
  resident)
• Personal Background (minority, socio-economic
  status, age)
• Extra-curricular activities, service,
  leadership (awards, social service, work
  experience)
• Personal attributes (talent, character,
  commitment, social concern, creativity,
  determination)
• Extenuating circumstances (family problems,
  health issues, language difficulties)
• This may differ somewhat from what institutions
  tell students what they are looking for
• Source Admissions Decision-Making Models How
  US Institutions of Higher Education Select
  Undergraduate Students, College Board (2003)

18
Where do Student Quality Instruments fit?
University policy for educating students
Student background
Low score on TOEFL ITP or TOEIC
High score on GMAT
High score on TOEFL
19
Where do Student Quality Instruments fit?
University policy for educating students
Student background

• Motivation statement/personal statement

• List of completed courses

• Letter of recommendation

• Proof of work experience

• Secondary school/bachelor diploma

• Right to access HE in home country

• Proof of knowledge of the teaching language

• Proof of previous academic work

• Standardized tests

20
How to decide depends on which type of error you
tolerate more
21
Consider exceptional cases(a) Lottery systems
(Numerus Clausus)(b) Admissions and immigration
(NL, DE)
University policy for educating students
Student background
22
Capable Successful

• How do we recognize success?
• Popular indicators of evidence
• Student quality
• Attrition
• III. Time to degree

• Faculty ratings
• Grade point average
• Results of Exams

23
Which Student Quality?

• Competencies important for graduate school
  success

• Student Quality
• Some qualities are also important to the labour
  market

24
Student Quality?
Competencies important for graduate school
success

• Student Quality

• Three research studies
• Two studies asking faculty for competencies of
  successful students
• One scheme for classifying outcome or criterion
  variables taxonomy of higher order performance
  components.

25
Types of Competencies linked to Student Quality

• Job-specific, Study-specific task proficiency
• Non-job, Non-study-specific task proficiency
• Written Oral Communication task proficiency
• Demonstration of effort
• Maintenance of personal Discipline
• Facilitation of peer and team performance
• Supervision/ Leadership
• Management/ Administration
• Some are present when student is admitted
• Some are part of the curriculum that the
  institution provides (educational value added)
  and so are expected at graduation
• Source Standardized Letter of Recommendation,
  Educational Testing Service

26
Competencies

• Job/study Specific Task Proficiency
• 1. Research/scholarly experience (amount and
  quality)
• 5. Mastery of discipline
• 11. Ability to read and analyze research in the
  field
• 13. Breadth of perspective
• C. Explanation (Enright Gitomer)
• G. Synthesis (Enright Gitomer)

27
Competencies

• II. Non-Job/study specific Task Proficiency
• 7. Ability to teach (cross-listed in VII and
  VIII)
• 8. Independence (cross-listed in IV)
• 9. Creativity ability to think out of the box
• 14. Open mindedness
• 15. Opportunism/Resourcefulness
• 16. Critical thinking ability, logic, problem
  solving ability
• 18. Computer literacy/Understand Manage
  Technology
• 19. Skill in investigation
• 20. Ability to find the facts information
  gathering
• B. Creativity (Enright Gitomer)
• E. Planning (Enright Gitomer)

28
Competencies

• III. Written Oral Communication Task
  Proficiency
• 4. Communication (professional)
• 6. English-as-a-Second-Language ability
• A. Communication (Enright Gitomer)
• IV. Demonstration of Effort
• 2. Persistence/Tenacity
• 8. Independence
• 10. Enthusiasm
• 17. Drive/Commitment/Motivation/Zeal
• D. Motivation (Enright Gitomer)

29
Competencies

• V. Maintenance of Personal Discipline
• 12. Values/Character integrity,
  fairness/openness/honesty, trustworthiness/
  consistency personal and professional
• 21. Professional posture
• VI. Facilitation of Peer and Team Performance
• 3. Collegiality/Making professional
  connections/Networking (Cross-listed with VII)
• F. Professionalism
• VII. Supervision/Leadership
• 3. Collegiality/Making professional
  connections/Networking (Cross-listed with VI.)
• 7. Ability to teach (Cross-listed with II. and
  VIII.)
• VIII. Management/Administration
• 7. Ability to teach (Cross-listed with II. and
  VII.)

30
Cross reference to Tuning Project

• How are competences and learning outcomes
  related?
• Learning outcomes according to Tuning methodology
  should be formulated in terms of competences.
• Learning outcomes are requirements of a unit or a
  programme and are expressed in terms what the
  learner knows and is able to do at the end of the
  learning experience.
• Competences may be developed to a greater degree
  than the level required by the learning outcome.

31
Staff-centred vs. Student-centred?
32

• On completion of a first cycle degree in
  Mathematics, students should be able to
• Show knowledge and understanding of basic
  concepts, principles, theories and results of
  Mathematics
• Understand and explain the meaning of complex
  statements using mathematical notation and
  language
• Demonstrate skill in mathematical reasoning,
  manipulation and calculation
• Construct rigorous proofs
• Demonstrate proficiency in different methods of
  mathematical proof.

33
Admissions Best Practices
Why Admit? Consensus on Strategic Objectives
34
How to recognize competencies?
35
The Admission Challenge
What are the competencies you attribute to
successful students?
Which of them have to be present at the start?
Is the aim Inclusive or Exclusive selection?
36
The Admission Challenge
What evidence do you considerIs it fair,
valid, reliable?
Have you considered all costs?
What is the legal setting?
37
Evaluating an Application Different models of
meeting the Admissions challenge

• Admissions offices may be responsible for
• Combining all individually-rated components and
  weighing
• Application information
• Transcripts
• Essays
• Recommendations
• Combining academic and non-academic factors and
  weighing
• Some factors considered separately e.g.,
  communication
• Balance of academic and non-academic
• Many wide-scope factors vs. few key factors
• Source Admissions Decision-Making Models How
  US Institutions of Higher Education Select
  Undergraduate Students, College Board (2003)

38
Evaluating an Application Different model
examples

• Example 1
• Academic
• Personal characteristics
• Suitability for desired field of study
• Example 2
• Academic
• Communication
• Character, leadership, initiative
• Source Admissions Decision-Making Models How
  US Institutions of Higher Education Select
  Undergraduate Students, College Board (2003)

39
Evaluating an Application Different model
examples

• Example 3
• Exceptional academic Achievement
• Academic Promise
• Potential to Contribute
• Example 4
• Academic Achievement
• Academic Qualities
• Non-academic Achievement
• Personal Qualities
• Source Admissions Decision-Making Models How
  US Institutions of Higher Education Select
  Undergraduate Students, College Board (2003)

40
Evaluating an Application Different model
examples

• Example 5
• Quality of Courses
• Grades in Core Curriculum
• Test results
• Activities (extra-curricular or curricular)
• Essay
• Example 6
• Academic Performance
• Extra-curricular activities
• Teacher / counselor recommendations
• Interview
• Personal Inventory
• Essays
• Source Admissions Decision-Making Models How
  US Institutions of Higher Education Select
  Undergraduate Students, College Board (2003)

41
Evaluating an Application Different model
examples

• Example 7
• Application and Essay
• Academic Performance
• Level of Challenge of Academic Record
• Recommendation and Interview
• Personal Qualities
• Special Talents
• Source Admissions Decision-Making Models How
  US Institutions of Higher Education Select
  Undergraduate Students, College Board (2003)

• Example 8
• Academic Achievement
• Intellectual Curiosity
• Potential
• Commitment
• Communication
• Engagement with others
• Extra-curricular activities
• Initiative

42
Making Decisions Making sense of what
information comes in from Applications

• Single readers, multiple readers, group readings,
  automated readings
• Process x for collecting academic info (formulaic
  rating) into an index
• Process y for collecting non-academic info
  (holistic rating) into an index
• Clear minimum thresholds of entry per index or
  factor
• Clear criteria for strong, and weak candidates
• Level descriptors for each level of qualification
• Clear guidelines for multiple entryways
• Special consideration for minorities, shortage of
  students
• Lottery for some competitive programmes with
  limited places
• Work experience
• Final review of denials
• Conditional acceptance
• Additional information or interview requested

43
Quality Assurance for Application Models

• Setting key current criteria into application
  form
• Direct and indirect questions
• Setting dates for re-evaluation of application
  form criteria
• Selection of staff who represent key constituents
  affected by decision
• Adapting admissions criteria to institutional
  profile over time
• Respecting changes in strategic objectives
• Adapting to external changes in education,
  demographics, labour market
• Understanding where the student body comes from
  and goes to

44
Best Practices in Admissions
What to look for Qualities sought in student body
How to Select Admissions Criteria
Consensus on Admissions objectives

• How to Decide
• Application Processing models
• Fairness
• Validity

45
Key Messages

• Admissions Criteria are the outcome of a decision
  process which must be regularly evaluated
• The process has the following components
• Strategic objectives of the institution
• Instruments to identify student quality
• Student application model
• Admissions criteria

46
Discussion and insights

• Why do you admit students?
• What are the 3 main Strategic Objectives of your
  institution?
• What are your main 3 Admission objectives?
• What are your 3 most common profiles of admitted
  students?
• 2. What is student quality for you?
• What are the 3 main student-quality instruments
  that you use? Consider how your policy objectives
  affect the level of detail that you require from
  your instruments.
• What are the most common instruments of your
  peers?
• Where do your admissions instruments fall on this
  chart?

47
Where do your Student Quality Instruments fit?
University policy for educating students
Student background
48
Discussion and insights

• 3. What is the quality of your added value to the
  students education?
• Which 2 key competencies does your institution
  expect to be present at admission?
• Which 2 key competencies does your institution
  expect at the learning process?
• 4. How do you recognize quality?
• What evidence does your institution require?
• Evidence from the student directly or other
  institutions
• Student portfolio
• High-school/ previous programme transcripts
• Letters of recommendation
• Personal answers to questions on the application
• Candidates scores on standardized admissions
  tests
• Language standardized tests
• Which formal instruments do you use?
• How do you ensure fairness, validity, and
  reliability?

49
Discussion and insights

• 5. What type of student application do you use?
• What are the top 3 factors in your qualification
  evaluation model?
• What quality assurance principles to you use to
  keep your application model valid?
• How frequently do you hold re-evaluation?
• How do you ensure validity of criteria/
  benchmarking success factors
• What is your process of updating criteria?