Performance Reports for Failing Candidates - PowerPoint PPT Presentation

1 / 21
About This Presentation
Title:

Performance Reports for Failing Candidates

Description:

Presented at the 2005 CLEAR Annual Conference. September 15-17 ... Mean scores & alphas. Holistic Scales Coefficient. Communications .83. 12 stns - competency 4 ... – PowerPoint PPT presentation

Number of Views:139
Avg rating:3.0/5.0
Slides: 22
Provided by: Steph6
Category:

less

Transcript and Presenter's Notes

Title: Performance Reports for Failing Candidates


1
Performance Reports for Failing Candidates
  • Carol OByrne
  • Pharmacy Examining Board of Canada

Presented at the 2005 CLEAR Annual
Conference September 15-17 Phoenix,
Arizona
2
What failing candidates want to know
  • How close was I to passing?
  • What did I do wrong? What did I miss?
  • How many such errors and omissions lead to a
    failing result?
  • In which area(s) do I need to improve?
  • What does PEBC expect in these areas?
  • Why am I expected to perform at a higher level
    than what I see some pharmacists doing?

3
PEBC rationale for providing feedback to
candidates
  • Supports PEBCs mandate to certify candidates
    who demonstrate that they have the knowledge,
    skills, abilities and attitudes required for
    practice
  • Increases candidates awareness of practice
    requirements
  • Supports the cooperative but arms length
    relationship between credentialing bodies and
    training bodies

4
Rationale
  • Benefits all parties
  • Assists candidates to recognize and address their
    weaknesses
  • Improves efficiency of PEBC processes and lessens
    potential threat on exam security by reducing the
    number of retakes
  • Benefits the profession and the public by
    supporting further development of qualifications
    of those preparing to enter practice
  • Addresses manpower needs - guides remediation and
    bridging efforts, facilitating earlier entry to
    the profession of those who may not yet have
    received adequate training

5
Why only to failing candidates?
  • No demand from passing candidates
  • Resource issues
  • Issuance of reports
  • Failing candidates often retake the exam without
    appropriate preparation and plug the system

6
PEBC Qualifying Examination
  • Based on national competencies and standards
  • Offered in English and French
  • Must be PEBC certified to license in 9/10
    provinces
  • Mobility enabled by mutual recognition (if PEBC
    certified)
  • Part I (MCQ) 200 scored items
  • Part II (OSCE) 15 scored stations
  • 12 SP/HP interactions
  • 3 non-client stations
  • 7 minutes/station
  • 1 assessor/station
  • 2 sets of scores/station
  • Analytical checklist
  • Holistic scales

7
Competencies assessed
8
Test format
  • Interactive client stations
  • Standardized patients
  • Standardized health professionals
  • Non-client stations
  • Technical, e.g.
  • Screening prescriptions for appropriateness
  • Checking dispensed prescriptions
  • Written short answer, e.g.
  • Responding to drug information requests -
    evaluating and interpreting drug information from
    several / conflicting sources
  • Medication management - reviewing patient data
    and recommending therapeutic options, along with
    a rationale

9
Assessor scoring sheet - ratings
  • Three 4-point scales
  • Communications generic scale
  • Rapport
  • Organization and flexibility (adaptive to the
    client/situation)
  • Verbal and nonverbal skills (including language
    proficiency)
  • Outcome (problem solving) station specific
    scale
  • Based on critical checklist items
  • Overall Performance inclusive, global scale
  • Communications and outcome
  • Process quality and thoroughness (critical and
    noncritical items)
  • Accuracy (vs misinformation)
  • Risk (occurrence, degree)

10
Assessor scoring sheet - checklist
  • Critical items (?)
  • essential to solve the problem meet station
    objective/s
  • each linked to a competency assessed in the
    station
  • Noncritical items
  • represent good practice contribute to effective
    outcome(s)
  • each linked to a competency
  • Risk and misinformation
  • Unique response (UR) - for scoring QA purposes
  • Comment boxes - to record evidence to support
    scores (used for QA purposes)

11
Scoring the examination
  • Analytical scores
  • Each checklist item relates to one competency
  • Competency sub-scores percent of items related
    to each competency to which candidate responds
  • Frequency of risk and misinformation tabulated
  • Holistic scores
  • Each scale 1 to 4 points
  • 12 points per client station (Comm, Outc, Perf)
  • x 12 stations
  • 8 points per nonclient station (Outc, Perf)
  • x 3 stations
  • Raw score sum of all stations holistic scale
    scores
  • Holistic cut score set for each scale in each
    station
  • Cut score sum of all stations holistic cut
    scores

12
Mean scores alphas
  • Holistic Scales Coefficient ?
  • Communications .83
  • 12 stns - competency 4
  • Outcome .66
  • 15 stns all competencies
  • Performance .73
  • 15 stns all competencies
  • Analytical Scores n items Coefficient ?
  • Pharm care 107 .80
  • Ethics 7 .43
  • Drug information 8 .24
  • Communications 14 .33
  • Drug distribution 8 .57
  • Management 8 .55

13
Factors affecting competency sub-score
reliabilities
  • Candidate variability (or lack of)
  • Number and context of stations in which the
    competency was assessed
  • Number of non-critical items vs critical items
    (importance of their performance to the task at
    hand)

14
Reports to candidates
  • Results pass-fail status (all candidates)
  • Feedback (for failing candidates, on request)
  • Individual score breakdown
  • by major skill mean Communication, Outcome and
    Performance ratings aggregated across all
    stations
  • by competency mean percent scores aggregated
    across all stations in which the competency was
    assessed
  • by critical incident frequency of risk,
    misinformation
  • Comparative data
  • Reference group mean scores and frequencies for
    score comparison with a stable population
  • to show where performance needs to improve

15
Assessor scoring sheet
16
OSCE feedback report
17
Candidate findings
  • Most candidates understand the information
    provided but want more guidance (content
    information where they went wrong)
  • Some do not accept the exam results and feedback
    information may request hand-scoring
  • Failing candidates generally score low in
    Communications (rating scale and competency 4)
    and/or Pharmaceutical Care (competency 1
    clinical role)
  • Many failing candidates lack clinical training in
    Canada (or the US) - though many have some
    technical training / experience (as a pharmacy
    technician)

18
Are we really helping candidates?
  • Anecdotally, yes some do not know where to
    start, what to focus on
  • Skills scores and competency sub-scores are
    consistent enough to be meaningful in areas that
    are weighted more heavily
  • All candidates who fail show weaknesses in one or
    more of these areas (low scores relative to the
    reference group)

19
What questions do (can) we answer?
  • What area(s) do I need to improve?
  • What does PEBC expect in these areas?
  • What did I do wrong? What did I miss?
  • Why am I expected to perform at a higher level
    than what I see some pharmacists doing?
  • How many errors and omissions lead to a failing
    result?
  • How close was I to passing?

20
Should we try to provide more support?
  • Some Pros
  • Repeating candidates who try multiple times
    slowly discover and address their
    strengths/deficits
  • Meanwhile they waste their own and test agency
    resources and plug the system
  • Repeating candidates are a potential security
    threat
  • Some Cons
  • Perceived conflict of interest (however, test
    purpose is not to fail candidates but to measure
    candidates current capabilities)
  • Cost of gathering and providing such information
  • Potential challenges based on information provided

21
What other strategies are (may be) helpful?
  • Provide information about training and/or
    remedial resources, e.g.
  • Clear expressions, including visual exemplars, of
    good practice in each competency area
  • Recognized training programs and resources
  • Practice exams (e.g. mock OSCEs) for format
    familiarization
  • Provide general tips, e.g.
  • Typical performance errors/deficits in each
    competency
  • Competency-related descriptions of candidates who
    are clearly qualified, borderline qualified and
    unqualified

22
Contact information
  • Carol OByrne
  • Pharmacy Examining Board of Canada
  • 415 Yonge Street, Suite 601
  • Toronto, ON M5B
  • T 416-979-2431, ext 226
  • Email obyrnec_at_pebc.ca
  • Website www.pebc.ca
Write a Comment
User Comments (0)
About PowerShow.com