Best practices in continuous evaluation: Evidence-Based Decision Making and Admissions Models - PowerPoint PPT Presentation

Loading...

PPT – Best practices in continuous evaluation: Evidence-Based Decision Making and Admissions Models PowerPoint presentation | free to download - id: 495a6e-MjkwZ



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Best practices in continuous evaluation: Evidence-Based Decision Making and Admissions Models

Description:

Best practices in continuous evaluation: Evidence-Based Decision Making and Admissions Models Admissions criteria and success criteria ETS Europe Jenny Dalalakis ... – PowerPoint PPT presentation

Number of Views:118
Avg rating:3.0/5.0
Slides: 50
Provided by: Jann84
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Best practices in continuous evaluation: Evidence-Based Decision Making and Admissions Models


1
Best practices in continuous evaluation Evidence
-Based Decision Making and Admissions Models
  • Admissions criteria and success criteria
  • ETS Europe Jenny Dalalakis jdalalakis_at_etseurope.
    org
  • January 2006 Gerben van Lent gvanlent_at_etseurope.
    org

2
Key Messages
  • Admissions Criteria are the outcome of a decision
    process which must be regularly evaluated
  • The process has the following components
  • Strategic objectives of the institution
  • Instruments to identify student quality
  • Student application model
  • Admissions criteria

3
Objective of this presentation Introduce Best
Practices in Admission Evaluations
Why Admit?
4
Admissions Best Practices
Why Admit? Consensus on Strategic Objectives
5

Outline
Determining Institutes Strategic Objectives for
Admissions Deciding on Qualities of Student
Body Evaluating Application Models Deciding on
Key Admissions Criteria Wrap up Conclusions and
Questions
6
Strategic Objective Successful fit between
Policy and Admissions Process
  • College Board research identified best practices
    as
  • Consensus on admissions objectives
  • Explicitly affirmed and revised periodically
  • Quantitative and qualitative criteria of incoming
    student body
  • Consensus on the qualities sought in student body
  • Board, staff, alumni, current student body,
    labour market representatives
  • Academic and non-academic qualities of admired
    students
  • Acceptable instruments to measure these desired
    qualities
  • Admissions criteria
  • Institutional objectives supported by individual
    and group qualities of student body
  • Reflected in application components (explicit)
    and underlying factors (role of informal
    priorities)
  • Evaluations and processing models
  • Institutional goals supported in transparent
    manner
  • Institutions Policy affects weight given to
    student quality measurements
  • Profile of student balanced among different
    qualities sought
  • Fairness
  • relevance of information, equal access,
    transparency of admission process
  • Validity of admissions process

7
Admissions Best Practices
Why Admit? Consensus on Strategic Objectives
8
Strategic Objectives Why Admit?
  • ACA Research identified 6 key objectives
  • Policy
  • Revenue
  • Profile
  • Sustainability
  • Non-discrimination
  • Demand
  • (Hybrid needs typically exist)
  • Source The Admission of International Students
    in Higher Education, ACA Paper on International
    Cooperation in Education (2004)

9
Why Admit? Key objective Development Policy
  • Institution Policy or Government Policy requires
    foreign students
  • France Prestige in educating other countries
    elite students and future leaders
  • Netherlands Manner of providing aid to
    developing countries
  • UK, US Foreign students become host countrys
    goodwill ambassadors when they return home
  • EU Support for mobility (Bologna, Lisbon) means
    international student exchanges are vital at
    Higher Education level
  • Source The Admission of International Students
    in Higher Education, ACA Paper on International
    Cooperation in Education (2004)

10
Why Admit? Key objective Revenue
  • Market-driven need for funds related to foreign
    students
  • Public and private schools may charge more for
    all or some international students directly so
    the extra income is welcome
  • UK, US
  • CH (MBA programme students specifically)
  • NL (English-taught programme students
    specifically)
  • Public university with foreign students may
    qualify for extra government funds
  • Australia
  • Programme survival in danger and loss of staff so
    foreign students help numbers
  • SE, CH, NL
  • Disincentives may apply if institution does not
    qualify for extra funds if they have foreign
    students
  • CH
  • Source The Admission of International Students
    in Higher Education, ACA Paper on International
    Cooperation in Education (2004)

11
Why Admit? Key objective Institutional Profile
  • International profile is of intrinsic value for
    Institution
  • Foreign students enhance innovation and
    modernization
  • Global pool better for choosing world-scholar
    quality
  • Minimum or maximum target may be set AU
  • Global recognition of host university for its
    student quality is very competitive
  • Competition at level of Institution/ Discipline
    or at level of Labour Market Demands
  • Nature of programme requires international
    approach
  • Research literature is in English (MBA,
    Engineering) FR
  • Applied Languages, International Relations,
    International Law
  • Foreign students bring the world home to remote
    areas
  • AU, SE
  • Source The Admission of International Students
    in Higher Education, ACA Paper on International
    Cooperation in Education (2004)

12
Why Admit? Key objective Sustainability
  • Securing the future of the programme, staff,
    research
  • Smaller countries do not have enough local
    students for keeping some disciplines alive at
    their universities
  • NL, SE, CH need Science students
  • Students coming for Masters programmes are
    encouraged to stay on for PhD programmes to
    secure enough teaching staff assistants and
    research assistants
  • Source The Admission of International Students
    in Higher Education, ACA Paper on International
    Cooperation in Education (2004)

13
Why Admit? Key objective Non-discrimination
  • No discrimination is tolerated between any
    student groups
  • Quality of students comes first regardless of
    students origin
  • Important objective for highly selective /
    private institutions US
  • Some are more equal than others (informally or
    formally)
  • CH evidence required that foreign student could
    have studied back home
  • FR to protect local student quota, local
    students chosen with earlier deadline for public
    university places
  • SE first choose local students and unfilled
    quota goes to foreign students
  • Source The Admission of International Students
    in Higher Education, ACA Paper on International
    Cooperation in Education (2004)

14
Why Admit? Key objective Students Demand
  • Large number of foreign students want to come to
    the Institution
  • Local, regional and global mobility (economic,
    academic, professional) is a growing demand by
    students, governments, labour market so students
    expect international programmes in countries of
    interest to them
  • Universities receive large number of foreign
    student applications and react by starting
    international programmes and (hopefully) support
    programmes
  • Source The Admission of International Students
    in Higher Education, ACA Paper on International
    Cooperation in Education (2004)

15
Admissions Best Practices
Why Admit? Consensus on Strategic Objectives
16
How to Admit? Look for Quality What is Student
Quality?
  • ACA research showed a variety of instruments for
    gathering evidence on student quality
  • Secondary school/ Bachelor diploma
  • Transcript of courses
  • List of completed courses
  • Motivation statement/personal statement
  • Letter of recommendation
  • Proof of work experience
  • Proof of previous academic work
  • Curriculum Vita
  • Standardized tests for subject and skill
    competencies
  • Proof of knowledge of the teaching language
  • Right to access Higher Education in home country
  • Source The Admission of International Students
    in Higher Education, ACA Paper on International
    Cooperation in Education (2004)

17
How to Admit? Look for Quality What is Student
Quality?
  • College Board research on factors used in making
    admissions decisions
  • Academic Achievement, Quality, and Potential
  • Direct measures (coursework, grades, test scores)
  • Caliber of high school/ previous study
    institution (rank of class, school)
  • Evaluative measures (depth and breadth of
    interests, world awareness, communication skills)
  • Non-academic Characteristics and Attributes
  • Geographic (remote or disadvantaged region, local
    resident)
  • Personal Background (minority, socio-economic
    status, age)
  • Extra-curricular activities, service,
    leadership (awards, social service, work
    experience)
  • Personal attributes (talent, character,
    commitment, social concern, creativity,
    determination)
  • Extenuating circumstances (family problems,
    health issues, language difficulties)
  • This may differ somewhat from what institutions
    tell students what they are looking for
  • Source Admissions Decision-Making Models How
    US Institutions of Higher Education Select
    Undergraduate Students, College Board (2003)

18
Where do Student Quality Instruments fit?
University policy for educating students
Student background
Low score on TOEFL ITP or TOEIC
High score on GMAT
High score on TOEFL
19
Where do Student Quality Instruments fit?
University policy for educating students
Student background
  • Motivation statement/personal statement
  • List of completed courses
  • Letter of recommendation
  • Proof of work experience
  • Secondary school/bachelor diploma
  • Right to access HE in home country
  • Proof of knowledge of the teaching language
  • Proof of previous academic work
  • Standardized tests

20
How to decide depends on which type of error you
tolerate more
21
Consider exceptional cases (a) Lottery systems
(Numerus Clausus) (b) Admissions and immigration
(NL, DE)
University policy for educating students
Student background
22
Capable Successful
  • How do we recognize success?
  • Popular indicators of evidence
  • Student quality
  • Attrition
  • III. Time to degree
  • Faculty ratings
  • Grade point average
  • Results of Exams

23
Which Student Quality?
  • Competencies important for graduate school
    success
  • Student Quality
  • Some qualities are also important to the labour
    market

24
Student Quality?
Competencies important for graduate school
success
  • Student Quality
  • Three research studies
  • Two studies asking faculty for competencies of
    successful students
  • One scheme for classifying outcome or criterion
    variables taxonomy of higher order performance
    components.

25
Types of Competencies linked to Student Quality
  • Job-specific, Study-specific task proficiency
  • Non-job, Non-study-specific task proficiency
  • Written Oral Communication task proficiency
  • Demonstration of effort
  • Maintenance of personal Discipline
  • Facilitation of peer and team performance
  • Supervision/ Leadership
  • Management/ Administration
  • Some are present when student is admitted
  • Some are part of the curriculum that the
    institution provides (educational value added)
    and so are expected at graduation
  • Source Standardized Letter of Recommendation,
    Educational Testing Service

26
Competencies
  • Job/study Specific Task Proficiency
  • 1. Research/scholarly experience (amount and
    quality)
  • 5. Mastery of discipline
  • 11. Ability to read and analyze research in the
    field
  • 13. Breadth of perspective
  • C. Explanation (Enright Gitomer)
  • G. Synthesis (Enright Gitomer)

27
Competencies
  • II. Non-Job/study specific Task Proficiency
  • 7. Ability to teach (cross-listed in VII and
    VIII)
  • 8. Independence (cross-listed in IV)
  • 9. Creativity ability to think out of the box
  • 14. Open mindedness
  • 15. Opportunism/Resourcefulness
  • 16. Critical thinking ability, logic, problem
    solving ability
  • 18. Computer literacy/Understand Manage
    Technology
  • 19. Skill in investigation
  • 20. Ability to find the facts information
    gathering
  • B. Creativity (Enright Gitomer)
  • E. Planning (Enright Gitomer)

28
Competencies
  • III. Written Oral Communication Task
    Proficiency
  • 4. Communication (professional)
  • 6. English-as-a-Second-Language ability
  • A. Communication (Enright Gitomer)
  • IV. Demonstration of Effort
  • 2. Persistence/Tenacity
  • 8. Independence
  • 10. Enthusiasm
  • 17. Drive/Commitment/Motivation/Zeal
  • D. Motivation (Enright Gitomer)

29
Competencies
  • V. Maintenance of Personal Discipline
  • 12. Values/Character integrity,
    fairness/openness/honesty, trustworthiness/
    consistency personal and professional
  • 21. Professional posture
  • VI. Facilitation of Peer and Team Performance
  • 3. Collegiality/Making professional
    connections/Networking (Cross-listed with VII)
  • F. Professionalism
  • VII. Supervision/Leadership
  • 3. Collegiality/Making professional
    connections/Networking (Cross-listed with VI.)
  • 7. Ability to teach (Cross-listed with II. and
    VIII.)
  • VIII. Management/Administration
  • 7. Ability to teach (Cross-listed with II. and
    VII.)

30
Cross reference to Tuning Project
  • How are competences and learning outcomes
    related?
  • Learning outcomes according to Tuning methodology
    should be formulated in terms of competences.
  • Learning outcomes are requirements of a unit or a
    programme and are expressed in terms what the
    learner knows and is able to do at the end of the
    learning experience.
  • Competences may be developed to a greater degree
    than the level required by the learning outcome.

31
Staff-centred vs. Student-centred?
32
  • On completion of a first cycle degree in
    Mathematics, students should be able to
  • Show knowledge and understanding of basic
    concepts, principles, theories and results of
    Mathematics
  • Understand and explain the meaning of complex
    statements using mathematical notation and
    language
  • Demonstrate skill in mathematical reasoning,
    manipulation and calculation
  • Construct rigorous proofs
  • Demonstrate proficiency in different methods of
    mathematical proof.

33
Admissions Best Practices
Why Admit? Consensus on Strategic Objectives
34
How to recognize competencies?
35
The Admission Challenge
What are the competencies you attribute to
successful students?
Which of them have to be present at the start?
Is the aim Inclusive or Exclusive selection?
36
The Admission Challenge
What evidence do you consider Is it fair,
valid, reliable?
Have you considered all costs?
What is the legal setting?
37
Evaluating an Application Different models of
meeting the Admissions challenge
  • Admissions offices may be responsible for
  • Combining all individually-rated components and
    weighing
  • Application information
  • Transcripts
  • Essays
  • Recommendations
  • Combining academic and non-academic factors and
    weighing
  • Some factors considered separately e.g.,
    communication
  • Balance of academic and non-academic
  • Many wide-scope factors vs. few key factors
  • Source Admissions Decision-Making Models How
    US Institutions of Higher Education Select
    Undergraduate Students, College Board (2003)

38
Evaluating an Application Different model
examples
  • Example 1
  • Academic
  • Personal characteristics
  • Suitability for desired field of study
  • Example 2
  • Academic
  • Communication
  • Character, leadership, initiative
  • Source Admissions Decision-Making Models How
    US Institutions of Higher Education Select
    Undergraduate Students, College Board (2003)

39
Evaluating an Application Different model
examples
  • Example 3
  • Exceptional academic Achievement
  • Academic Promise
  • Potential to Contribute
  • Example 4
  • Academic Achievement
  • Academic Qualities
  • Non-academic Achievement
  • Personal Qualities
  • Source Admissions Decision-Making Models How
    US Institutions of Higher Education Select
    Undergraduate Students, College Board (2003)

40
Evaluating an Application Different model
examples
  • Example 5
  • Quality of Courses
  • Grades in Core Curriculum
  • Test results
  • Activities (extra-curricular or curricular)
  • Essay
  • Example 6
  • Academic Performance
  • Extra-curricular activities
  • Teacher / counselor recommendations
  • Interview
  • Personal Inventory
  • Essays
  • Source Admissions Decision-Making Models How
    US Institutions of Higher Education Select
    Undergraduate Students, College Board (2003)

41
Evaluating an Application Different model
examples
  • Example 7
  • Application and Essay
  • Academic Performance
  • Level of Challenge of Academic Record
  • Recommendation and Interview
  • Personal Qualities
  • Special Talents
  • Source Admissions Decision-Making Models How
    US Institutions of Higher Education Select
    Undergraduate Students, College Board (2003)
  • Example 8
  • Academic Achievement
  • Intellectual Curiosity
  • Potential
  • Commitment
  • Communication
  • Engagement with others
  • Extra-curricular activities
  • Initiative

42
Making Decisions Making sense of what
information comes in from Applications
  • Single readers, multiple readers, group readings,
    automated readings
  • Process x for collecting academic info (formulaic
    rating) into an index
  • Process y for collecting non-academic info
    (holistic rating) into an index
  • Clear minimum thresholds of entry per index or
    factor
  • Clear criteria for strong, and weak candidates
  • Level descriptors for each level of qualification
  • Clear guidelines for multiple entryways
  • Special consideration for minorities, shortage of
    students
  • Lottery for some competitive programmes with
    limited places
  • Work experience
  • Final review of denials
  • Conditional acceptance
  • Additional information or interview requested

43
Quality Assurance for Application Models
  • Setting key current criteria into application
    form
  • Direct and indirect questions
  • Setting dates for re-evaluation of application
    form criteria
  • Selection of staff who represent key constituents
    affected by decision
  • Adapting admissions criteria to institutional
    profile over time
  • Respecting changes in strategic objectives
  • Adapting to external changes in education,
    demographics, labour market
  • Understanding where the student body comes from
    and goes to

44
Best Practices in Admissions
What to look for Qualities sought in student body
How to Select Admissions Criteria
Consensus on Admissions objectives
  • How to Decide
  • Application Processing models
  • Fairness
  • Validity

45
Key Messages
  • Admissions Criteria are the outcome of a decision
    process which must be regularly evaluated
  • The process has the following components
  • Strategic objectives of the institution
  • Instruments to identify student quality
  • Student application model
  • Admissions criteria

46
Discussion and insights
  • Why do you admit students?
  • What are the 3 main Strategic Objectives of your
    institution?
  • What are your main 3 Admission objectives?
  • What are your 3 most common profiles of admitted
    students?
  • 2. What is student quality for you?
  • What are the 3 main student-quality instruments
    that you use? Consider how your policy objectives
    affect the level of detail that you require from
    your instruments.
  • What are the most common instruments of your
    peers?
  • Where do your admissions instruments fall on this
    chart?

47
Where do your Student Quality Instruments fit?
University policy for educating students
Student background
48
Discussion and insights
  • 3. What is the quality of your added value to the
    students education?
  • Which 2 key competencies does your institution
    expect to be present at admission?
  • Which 2 key competencies does your institution
    expect at the learning process?
  • 4. How do you recognize quality?
  • What evidence does your institution require?
  • Evidence from the student directly or other
    institutions
  • Student portfolio
  • High-school/ previous programme transcripts
  • Letters of recommendation
  • Personal answers to questions on the application
  • Candidates scores on standardized admissions
    tests
  • Language standardized tests
  • Which formal instruments do you use?
  • How do you ensure fairness, validity, and
    reliability?

49
Discussion and insights
  • 5. What type of student application do you use?
  • What are the top 3 factors in your qualification
    evaluation model?
  • What quality assurance principles to you use to
    keep your application model valid?
  • How frequently do you hold re-evaluation?
  • How do you ensure validity of criteria/
    benchmarking success factors
  • What is your process of updating criteria?
About PowerShow.com