Finding the Right Students: The Search for Predictive Validity in Applicant Screening for the Health

1
Finding the Right StudentsThe Search for
Predictive Validity in Applicant Screening for
the Health Sciences

• Rosemary Lysaght, Ph.D.
• Catherine Donnelly, M.Sc.
• Michelle Villeneuve, M.Sc.
• School of Rehabilitation Therapy

2
Background

• Impetus for examination of our admissions process
  was the creation of the new masters entry-level
  program in 2004
• Goals for admission included
• An evidence-based admissions process
• Increase the applicant pool
• Efficiency in admission process

3
Overall Aim of Selection

• To select students who will
• Succeed in the academic program
• Perform credibly in professional practice
• Possess the traits of character and ethical
  values desired of a professional person
• (Nayer, 1992)

4
Commonly Used Selection Criteria in Health
Sciences

• Pre-admission academic grades
• Discipline-specific aptitude tests (MCAT, HOAE)
• Interviews
• Written submissions
• Letters of reference
• Prerequisites
• (Auriemma, 2002 Salvatori, 2001)

5
Predictors of academic success

• Evidence across disciplines supports the
  predictive validity of
• Pre-admission academic grades
• No clear support found for
• Discipline-specific aptitude tests
• Interviews
• Written submissions
• Letters of reference
• Prerequisites

(Bridle, 1987 Caplan et al, 1996 Howard
Jerosh-Herold, 2000 Lewis Smith, 2002
Kirchner et al., 2000 Kirchner Holm, 1997
Salvatori, 2001)
6
Predictors of clinical performance

• Unclear relationship between pre-admission
  performance and clinical performance
• Valid outcome measures difficult to identify
• Lack of consistent raters
• Variability across settings
• (Howard Jerosch-Herold, 2000 Kirchner
  Holm, 1997 Katz Mosey, 1980 Tan et al., 2004)

7
Research Questions

• What admissions screening tools best predict
  academic performance in a masters level OT
  program?
• Selected academic courses and
• Clinical performance
• Does undergraduate coursework predict success in
  topically-related coursework?

8
Method

• Analysis of existing data for 128 students
  admitted to the OT masters program
• Multiple regression
• Models created for each research question
• Project received approval by the Queens
  Universitys Research Ethics Board
• Sample included 3 cohorts of students (1st three
  years of the new professional masters program)

9
Factors Considered in OT Admissions

• GPA
• Academic transcript
• Letter of intent
• 2 referee rating forms/letters
• Supporting data (foreign/non-traditional
  applicants)

10
Predictor Variables

• Undergraduate Grade Point Average (GPA) from
  ORPAS
• Letter of Intent (LOI)
• Average rating, 2 raters
• Referee Rating
• 5 point scale, 12 items
• Average total score of 2 external referees

11
Predictor Variables (cont)

• Additional ratings were created for each student
  based on file review (1 weak to 5
  exceptional)
• Physical Sciences Experience/Preparation
• e.g. Anatomy, physiology, kinesiology (r for 2
  raters .98)
• Social Sciences Experience/Preparation
• e.g. Psychology, Sociology, Family Studies
  (r.91)
• Experience with People with Disabilities/Vulnerabl
  e Populations
• Rated for number, duration, and relevance (r.79)
• Information drawn from Letter of Intent, referee
  letters and experience questionnaire

12
Criterion Variables

• Program GPA (1st year)
• Communication Skills Practicum
• Targeted course grades in
• Physical Determinants of Occupation
• Cognitive-Neurological Determinants
• Psycho-Emotional Determinants

13
Descriptive Statistics - Predictor Variables
14
Descriptive Statistics - Criterion Variables
15
Results First Year GPA

• Predictor Variables GPA, Letter of Intent,
  Referee Rating
• Model is significant (R2 .101 F 4.65, p lt
  .001)
• GPA is only variable with significant beta score
  (p .005)
• Significant correlations between all three
  variables and program GPA

16
Results Communication Skills

• Predictor Variables GPA, Letter of Intent,
  Referee Rating, all experience ratings,
• Model is not significant (R2 .021 F .62)
• Referee rating is significantly correlated with
  performance rating
• No background experience ratings were correlated
  with performance rating

17
Results Physical Determinants

• Predictor Variables GPA, all experience ratings
• Result
• Model not significant (R2 .009 F.275)
• No significant correlations between any
  background experience rating and outcome

18
Results Cognitive-Neuro

• Predictor Variables GPA, all experience ratings
• Result
• Model is significant (R2 .14 F5.0, p .004)
• GPA only variable with significant beta score (p
  .00)
• Background in physical sciences was significantly
  negatively correlated with outcome

19
Results PsychoEmotional Det.

• Predictor Variables GPA, all experience ratings
• Result
• Model is significant (R2 .082 F 2.6, plt .05)
• Social science bkgd only variable with
  significant beta score (-.2 )
• GPA is significantly positively correlated with
  outcome, while SS bkgd is significantly
  negatively correlated.

20
Additional Observations

• The contribution of the undergraduate GPA to all
  models was reduced by the addition of the 3rd
  cohort, which had significantly higher GPA
  ratings than the first 2 cohorts and less spread
  in scores
• Referee ratings also had small but significant
  correlations with
• physical determinants grade (r .18, p .02)
• communication skills grade (r .25, p .003)

21
Conclusions

• Findings relative to GPA as a significant
  predictor of academic performance in a health
  science program supports previous research
• Other positive correlates with 1st year GPA
• Referee ratings
• LOI ratings
• suggest that these measures have some value
  in the admissions process.
• Referee ratings have even broader potential
  value, given positive correlations with
  performance in Physical Determinants
  Communications Skills courses.

22
Conclusions (cont)

• No support for the requirement of specific
  academic pre-requisites
• Previous experience with PWD/vulnerable
  populations did not affect academic grades or
  communication skills performance

23

• No support for the requirement of specific
  academic pre-requisites
• Previous experience with PWD/vulnerable
  populations did not affect academic grades or
  communication skills performance

24
Discussion

• Fairness of admissions process
• Pre-requisite requirements not justified if
  predictive validity of courses not substantiated
• No control over authorship of LOI
• Inherent biases in process
• LOI requirement may bias selection towards
  persons from the same culture who highlight
  issues salient to that culture, females, and
  strong writers
• Applicants with strong social science background
  may be over-represented if GPA is primary
  selection factor
• Impact on program
• Elimination of pre-requisites broadens applicant
  pool, may result in higher calibre class
• More diversity of students in program/field

25
Discussion

• Role of Pre-Requisites and other Screening Tools
• LOI, Referee ratings useful as screen out, rather
  than selection criteria?
• Certain pre-requisites may make academic progress
  easier for student
• Admissions requirements may have value beyond
  identification of best applicants
• Credibility of process and applicant
• Interviews may help form relationships, promote
  program

26
Limitations

• Findings limited to one health science program
  format within a Canadian context, and may not
  generalize
• Other potential screening tools not available for
  consideration in this study
• Course grades are subject to unsystematic scoring
  errors
• Ratings of background experience and LOI subject
  to rater error

27
References

• Auriemma, D. (2002). Admission methods of
  professional occupational therapy programs in the
  united states. Education Special Interest Section
  Quarterly, 12(3), 1-4.
• Bridle, M. J. (1987). Student selection A
  comparison of three methods... queen's university
  occupational therapy program. Canadian Journal of
  Occupational Therapy, 54(3), 113-117.
• Caplan, R.M, Kreiter, C., Albanese, M. (1996).
  Preclinical science course preludes taken by
  premedical students do they provide a
  competitive advantage? AMJ, 71 920-922.
• Howard, L., JeroschHerold, C. (2000). Can entry
  qualifications be used to predict fieldwork and
  academic outcomes in occupational therapy and
  physiotherapy students? British Journal of
  Occupational Therapy, 63(7), 329-334.
• Katz, G.M., Mosey, A.C. (1980). Fieldwork
  performance, academic grades, and pre-selection
  criteria of occupational therapy students.
  American Journal of Occupational Therapy, 34(12),
  794 800.

28

• Kirchner, G. L., Holm, M. B. (1997). Prediction
  of academic and clinical performance of
  occupational therapy students in an entry-level
  master's program. American Journal of
  Occupational Therapy, 51(9), 775-779.
• Kirchner, G. L., Stone, R. G., Holm, M. B.
  (2000). Use of admission criteria to predict
  performance of students in an entry-level
  master's program on fieldwork placements and in
  academic courses. Occupational Therapy in Health
  Care, 13(1), 1-10.
• Lewis, M., Smith, S. (2002). Selection of
  pre-registration physiotherapy students
  Changing to a more objective process.
  Physiotherapy, 88(11), 688 698.
• Nayer, M. (1992). Admission criteria for
  entrance to physiotherapy schools How to choose
  among many applicants. Physiotherapy Canada, 44,
  41 46.
• Salvatori, P. (2001). Reliability and validity of
  admissions tools used to select students for the
  health professions.see comment. Advances in
  Health Sciences Education, 6(2), 159-175.
• Tan, K. P., Meredith, P., McKenna, K. (2004).
  Predictors of occupational therapy students
  clinical performance An exploratory study.
  Australian Occupational Therapy Journal, 51(1),
  25-33.