Selection Process: The Multiple Mini Interview

1
Selection Process The Multiple Mini Interview

• Greg Eddy
• Joy Stuckless
• February
  14,2006

2
Objectives

• Selection, Interviews and Tests
• Overview of case
• Interview Preference Discussion
• MMI Research Strengths Weaknesses
• Does MMI increase validity?
• Traditional Interview Research
• Recommendations

3
Selection

• the process of making a hire or no hire
  decision regarding each applicant for a job
  (hiring those external to the organization,
  promotion, transfer, td)

4
Steps in the Selection Process
5
Initial Screening

• Identify applicants meeting minimum requirements
  for education and experience
• Able to short list applicants to make interview
  process more manageable

6
Types of Interviews

• Individual Interview interviewed by 2 or more
  interviewers, interview panel
• Group Interview a group is interviewed and
  interviewers observe who takes the lead role,
  monopolizes,
• answers questions, performs under pressure

7
Types of Interviews

• Stress Interviews Interviewer is assessing how
  the applicant acts and speaks under pressure,
  quick questions,
• challenges

8
Interview Styles

• Structured/Directive Series of questions, leads
• Meandering/Nondirective relies on your lead of
  discussion, open ended questions
• Situational describes situation and asks what
  you would do
• Behavioral asked to describe how you dealt with
  situations and used skills in the past
• Audition brief exercise to evaluate your skills

9
Traditional Interviews

• Grading answers with a preset plan
• Based on a job
• Specific applicant characteristics
• Judgements should have predefined criteria
• Rationalizes and standardizes selection
• Agreement among raters indicates reliablity

10
Employment Tests

• Aptitude assess how well a person can learn or
  acquire skills and abilities
• Achievement measures a persons existing
  knowledge and skills
• Physical ability tests physical strength and
  endurance
• Cognitive abilities measures verbal skills
  quantitative and reasoning ability

11
Employment Tests

• Personality tests
• Medical exams
• Job performance assessment centre a wide
  variety of specific selection programs that use
  multiple selection methods to rate applicants
• Combining several assessment methods increase the
  validity

12
Overview of Case A Dozen First Impressions

• McMaster University Hamilton, Ontario
• Third largest medical school
• 384 candidates
• Recruit 138 students yearly

13
Changes in Interviewing Process

• Although valued non-cognitive variables, such as
  interpersonal skills and professionalism, it
  wasnt clear current tools like the personal
  interview were capable of assessing ability in
  these areas
• Based on 3 year research study How to enhance
  quality of selection process
• University changed from traditional panel
  interviews
• Implemented Assessment Center Approach

14
Multiple Mini Interview (MMI)

• Applicants rotate through 12 mini
  interviews/stations, 8 minutes each in separate
  rooms
• Discuss one scenario and answer questions
• Each station has a different examiner
• Observers get insight into candidates qualities
• Differentiate those stronger personal qualities
  desired

15
MMI Rational

• Critical insight was that it is necessary
• to broadly sample an individuals competencies
  in order to gain an accurate picture of the
  individuals strengths and weaknesses

16
Variety of Issues

• Communication
• Collaboration
• Current Events
• Ethics
• Critical thinking
• Awareness of societal health issues
• Traditional interview questions
• Other areas of interest

17
Interviewee Evaluation Basis

• Content and method of response
• Body language, eye contact, voice modulation,
  tone, variety, speed of delivery
• Examples given of teamwork, leadership, problem
  based learning
• Responses related to issues

18
Research

• Kevin Eva BSc. PhD principle researcher
• Problem Challenge of medical educators in
  development of admissions protocol that provides
  valid information pertaining to non-cognitive
  qualities
• Participants in the MMI and enrolled in program
  45 candidates (Spring 2002)
• 54 candidates (Spring 2003) 112 candidates
• (Spring 2004)

19
Research (continued)

• Performance on this tool and on the traditional
  protocol was compared to performance on
  pre-clerkship evaluation exercises
• Results MMI was the best predictor of objective
  structured clinical exam performance and GPA best
  predictor for exam results
• Shown to be feasible, acceptable and reliable
• Further validity testing required

20
Additional Selection Methods

• Self assessment volunteer work, research, HC
  systems knowledge, first aid, CPR
• extra curricular activities
• MCAT Medical College Admissions Test
• verbal reasoning, physical sciences, biological
  sciences, writing skills
• Grades GPA more important than which undergrad
  program
• Personal statement
• References

21
Important Selection Principles

• Combine several sources of information
• about candidates
• Sources should relate to characteristics
  identified
• Variety of sources increase the validity of
  decision criteria
• More likely to make decisions that are fair and
  unbiased
• More likely to choose the best candidates

22
Alternatives

• No Interviews
• Continue with traditional structured panel
  interviews
• Implement MMI

23
Interview Process Preferences

• Traditional Interview
• Which do you prefer?
• 2 Groups
• Pro traditional
• Discussion
• Report back

• MMI Interview
• Pro MMI
• Discussion
• Report back

24
Does this selection method increase the
validity of selection process at McMasters
Medical School?

• Limitations
• Only one outcome measured
• Small sample size
• GPA was better predictor for medical knowledge
  exam performance
• Limited validity (No long term evaluation )
• Results discussed based on primary researchers
  opinions

25
Limitations Continued

• Participants aware participation (or lack of
  participation) would in no way influence chances
  of being accepted to program
• Offered candidates 40 in an attempt to make it
  clear that this initiative was completely
  separate from the regular admissions process.

26
Does this selection method increase the validity
of selection process at McMasters Medical School?

• Strengths
• Best predictor of clinical exam performance
• Another means of evaluation of applicants
• Assesses non-cognitive qualities which are
  important especially for bedside manner
• Traditional interviews dont necessarily capture
  this aspect
• Combining several selection methods increases the
  validity

27
Strengths Continued

• Dilutes the effect of chance and interviewer/
  situational biases.
• Ensures that the ratings assigned to the multiple
  points of discussion are given independently
  because interviews are in separate rooms.
• Provides flexibility for stations development.
  For any given station, the examiner might be an
  interviewer or an observer.

28
Strengths Continued

• Flexibility in station development reduces the
  likelihood that candidates will benefit from
  preparing and rehearsing responses
• If desire to query applicants regarding their
  personal experiences, challenges or beliefs
  traditional interview stations can be used
• potential to draw interviewers from diverse
  academic and community areas and allow them to
  assess topics that are consistent with their
  domain of expertise
• Fewer resources might be required

29
Strengths Continued

• Allows multiple samples of insight into a
  candidate's abilities
• Stations can be structured so that all candidates
  respond to the same questions
• Stations are designed with qualities they would
  like to select for in mind
• Staffing each station with a single interviewer
  has the potential to correct the imbalance in
  numbers between interviewers and candidates
• Candidates can recover from a disastrous station
  by moving on to a new independent interviewer

30
Advantages of MMI

• Potential of achieving 4 purposes of admissions
  interviews information gathering, decision
  making, verification and recruitment
• Demonstrates the value the institution places on
  human touch criterion
• Reliability increases with maximizing the number
  of stations

31
Participant Feedback - Positive

• Provide a more balanced view of the applicant's
  skills and experiences
• Opportunity to maintain a dialogue with the
  interviewer and the opportunity to 'solve and
  discuss REAL PROBLEMS

32
Participant Feedback - Negative

• Logistical issues, such as including 'a chair
  between stations', lengthening the amount of time
  for each interview
• Allow for some discussion at the end,
• Opportunity to go back to a point not adequately
  covered
• Would allow for a 'shorter interview day', but
  that a 'break half way through would help'.
• Lack of an opportunity to reveal group skills a
  domain that could potentially be built into
  future iterations of the MMI.

33
Participant Feedback - Negative

• Fun, but more tiring than the personal
  interview, because a single person is responsible
  for each interview. Suggested rotating the
  interviewers throughout the day so that a
  different station was assessed during each
  rotation.
• Examiners would have liked more training
  beforehand.

34
Single Interview Limitations

• Interviewer Bias
• candidate who is randomly assigned to a
  like-minded, 'easy' interviewer who influences
  the rest of the interview panel
• interviewers' backgrounds
• interviewers' expectations
• Context Specificity
• cognitive 'skills' are highly dependent on
  context
• performance is commonly less determined by
  'trait' and more determined by the 'state' (the
  context within which the performance is elicited).

35
Traditional Interview Research

• An interview may not provide and accurate
  portrayal of candidates abilities
• Reliability can be improved by standardizing
  questions training the interviewer
• Structuring interviews increases predictive
  validity

36
Structuring Interviews

• Use situational and behavioral questions
• Past oriented questions have higher validity then
  future oriented
• Reduces discrepancies between individual
  interviewers
• No proof than an individual with position
  knowledge, skills and abilities necessarily
  outperforms during an interview

37
Interview Research

• More structure is better
• Recruiter behaviors may affect applicants
• Interview validity may rival that of mental
  ability tests
• Previewing applicants information can harm
  relation effectiveness

38
Key to Selection Process

• Trained interviewers
• Structure to interview
• Interview questions
• Consistency of process
• Selection criteria
• Scoring
• Documentation requirements

39
Evaluation Criteria

• Reliability
• Research shows that the MMI is reliable method of
  assessing non cognitive abilities
• While shown to be reliable this might be enhanced
  with further improved examiner training
• Validity
• Performance of students who participated in MMI
  and entered medical school will allow comparison
  to program performance
• Feasibility
• Cost effectiveness
• Acceptability
• Participant feedback summarized as being
  positive/ constructive and worth pursing further

40
Recommendations

• MMI
• Participants accepted in Medical School should be
  followed for longer term to establish predictive
  validity
• Research be expanded to more medical schools
• Conduct concurrent validation with existing
  medical students and physicians
• Conduct research in non health sciences fields to
  determine if generalizable

41
Conclusion

• Traditional panel interviews is still the
  preferred interview option for student selection
  in medical schools.