CompetencyBased Curriculum and Assessment

1
Competency-Based Curriculum and Assessment

• James M Shumway, PhD
• (jshumway_at_hsc.wvu.edu)
• Associate Dean for Medical Education
• West Virginia University
• School of Medicine
• 6-12-08

2
A Teachers View
Teacher
3
A Teachers View
Curriculum
Teacher
4
A Students View
5
A Students View
Student
6
A Students View
Examination
Student
7
A Teachers View vs. A Students View
Curriculum
Examination
Teacher
Student
_____________ Melnick D. Presentation at
Marshall U. February 12 2001.
8
A Teachers View vs. A Students View
Curriculum
Examination
Communicates Content
Teacher
Student
_____________ Melnick D. Presentation at
Marshall U. February 12 2001.
9
A Teachers View vs. A Students View
Curriculum
Examination
Communicates Content
Communicates Values
Teacher
Student
_____________ Melnick D. Presentation at
Marshall U. February 12 2001.
10
Part 1 Objectives Understanding Competencies

• The learner should be able to realize a different
  perception of the curriculum as seen through the
  eyes of a student,
• describe some of the shifts in the paradigms of
  health care and medical education,
• describe a simple model for explaining a
  hierarchy of competence, and
• describe at least three competency-based
  curricular models.

11
Why Our Role as Medical Educators Matters

• To improve the quality of health care!
• Creating and environment that fosters and rewards
  improvement.
• Teaching evidence-based practice.
• Facilitating the use of information technology.
• Preparing a workforce to better serve patients in
  a world of rapidly expanding knowledge and change.

______________ IOM. 2001. Crossing the Quality
Chasm A new health system for the 21st century.
Washington DC National Academy Press
12
Shifts in Paradigms in Health Care

• From individuals to community
• From cure of disease to preservation of health
• From episodic care to continuing care
• From the physician provider to teams of
  multidisciplinary providers
• From paternalism to partnerships
• From provider-centered to patient/family-
  centered
• From individual accountability to systems
  accountability

________________ Jordan Cohen MD, 2005
13
Shifts in Paradigms in Curriculum

• From passive/spoon-fed learning to active,
  student-directed learning
• From rote learning to curiosity-driven learning
  (e.g., PBL)
• From regurgitating facts to demonstrating
  competence
• From dept-based courses to interdisciplinary
  courses
• From biology of disease to determinants of
  illness (inc. social)
• From physical exam to comprehensive clinical
  skills (inc. communication)
• From structured to individualized curricula based
  on learning objectives

________________ Jordan Cohen MD, 2005
14
A Definition

• Competency is noted when a learner is observed
  performing a task or function that has been
  established as a standard by the profession. The
  achievement of professional competency requires
  the articulation of learning objectives as
  observable, measurable outcomes for a specific
  level of learner performance. Such specific
  detailing of performance expectations defines
  educational competencies. They are verified on
  the basis of evidence documenting learner
  achievement, and must be clearly communicated to
  learners, faculty, and institutional leaders
  prior to assessment.

____________ Identified by members of work group
on competency-based womens health education at
APGO Interdisciplinary Womens Health Education
Conference in September, 1996, Chantilly, VA.
15
Levels of Competence
_______________ Miller GE. 1990 The assessment
of clinical skills / competence / performance.
Academic Medicine,. 65(Supplement) p. S63-S67)
16
Levels of Competence
Knows
_______________ Miller GE. 1990 The assessment
of clinical skills / competence / performance.
Academic Medicine,. 65(Supplement) p. S63-S67)
17
Levels of Competence
Knows How
Knows
_______________ Miller GE. 1990 The assessment
of clinical skills / competence / performance.
Academic Medicine,. 65(Supplement) p. S63-S67)
18
Levels of Competence
Shows How
Knows How
Knows
_______________ Miller GE. 1990 The assessment
of clinical skills / competence / performance.
Academic Medicine,. 65(Supplement) p. S63-S67)
19
Levels of Competence
Does
Shows How
Knows How
Knows
_______________ Miller GE. 1990 The assessment
of clinical skills / competence / performance.
Academic Medicine,. 65(Supplement) p. S63-S67)
20
Levels of Competence
Does
Shows How
Knows How
Cognition
Knows
_______________ Miller GE. 1990 The assessment
of clinical skills / competence / performance.
Academic Medicine,. 65(Supplement) p.
S63-S67) Womens health Care Competencies for
Medical Students Taking Steps to Include Sex and
Gender Differences in the Curriculum, 2005.
21
Levels of Competence
Does
Behavior
Shows How
Knows How
Cognition
Knows
_______________ Miller GE. 1990 The assessment
of clinical skills / competence / performance.
Academic Medicine,. 65(Supplement) p.
S63-S67) Womens health Care Competencies for
Medical Students Taking Steps to Include Sex and
Gender Differences in the Curriculum, 2005.
22
Accreditation Requires Competencies

• The Liaison Committee on Medical Education
  (LCME) is certified by the US Department of
  Education as the accrediting agency for
  allopathic medical education in the US and Canada
  the LCME
• requires programs to provide assurances that
  graduates can exhibit general professional
  competencies to prepare them for their next stage
  of education and prepare them to be life-long
  learners.
• ED-1-A The objectives and their associated
  outcomes must address the extent to which
  students have progressed in developing the
  competencies that the profession and the public
  expect of a physician.

23
Characteristics ofCompetency-Based Education

• Where the learning outcomes (competencies) are
  identified, performance based, and communicated
  to all.
• Provides a compelling statement of significant
  exit outcomes to the public, families,
  colleagues, and all providers.
• May be adapted to suit local context and needs.
• Emphasizes the personal development of the health
  care provider as a professional.
• Where the accomplishment of competence is able to
  be assessed.
• Where a criterion-referenced system of student
  assessment is the norm.
• Is applicable to all phases of the continuum of
  professional education.

24
Models of Competency-Based Approaches in Medical
Education

• CanMEDS 2000 Project (1996)
• AAMC Medical Student Objectives Project (1998)
• Institute of Medicine Competencies (2001)
• University of Dundee, Scotland (1999)
• ACGME Core Competencies (2000)

25
CanMeds (1/2)

• Medical Expert
• demonstrate diagnostic and therapeutic skills for
  ethical and effective patient care apply
  relevant information demonstrate effective
  consultation services.
• Communicator
• establish therapeutic relationship with
  patients/families/health care team obtain and
  synthesize relevant history listen effectively.
• Collaborator
• consults effectively with other physicians and
  health care team.
• Manager
• utilizes resources effectively works within
  health care organization utilizes information
  technology to optimize patient care and lifelong
  learning.
• 
  (continued)

_______________ The Royal College of Physicians
and Surgeons of Canada. CanMEDS 2000 Project
Skills for the new millennium. Report of the
Societal Needs Working Group. September, 1996.
26
CanMeds (2/2)

• Health Advocate
• identifies important determinants of health
  contribute to improved health of patients and
  community recognizes where advocacy is
  appropriate.
• Scholar
• implement a personal continuing education
  strategy critically appraise sources of medical
  information facilitate the learning of patients
  and other health professionals contribute to new
  knowledge.
• Professional
• deliver highest quality care with integrity,
  honesty and compassion demonstrate appropriate
  professional behaviors practice medicine
  ethically consistent with the obligations of a
  physician.

_______________ The Royal College of Physicians
and Surgeons of Canada. CanMEDS 2000 Project
Skills for the new millennium. Report of the
Societal Needs Working Group. September, 1996.
27
AAMC Medical Student Objectives Project (MSOP)

• Physicians must be knowledgeable.
• Physicians must be skillful.
• Physicians must be altruistic.
• Compassionate treatment of patients
• Ethics, honesty, integrity, respect, advocacy
• Recognize and accept ones limitations
• Physicians must be dutiful.
• Knowledge of non-biological determinants of poor
  health
• Epidemiology, risk factors, IT, health care
  delivery
• Commitment to provide care for under-served
  populations

_______________ AAMC. 1998 Learning Objectives
for Medical Student Education guidelines for
medical schools. Association of
American Medical Colleges Washington, DC. p.
1-13.
28
IOM Core Competencies

• Broad recommendations
• Physicians and other health professionals must be
  educated to deliver patient care as
  multidisciplinary teams (presently -- typically
  isolated by discipline),
• must emphasize evidence-based practice and
  continuous quality improvement,
• must understand and know how to effectively use
  information technology (to learn, to care for
  patients, to access information),
• must be prepared with lifelong-learning skills
  for ongoing licensure and certification as well
  as the continuing assessment of competencies, and
  
• must align the education with the needs of the
  health care system.

______________ IOM. 2001. Crossing the Quality
Chasm A new health system for the 21st century.
Washington DC National Academy Press
29
U. of Dundee Three-Circle Model of Competence
What the doctor does. (7 outcomes)
How the doctor approaches practice. (3 outcomes)
The doctor as a professional. (2 outcomes)
_______________ Harden RM, et. al. 1999 From
competency to meta-competency a model for the
specification of learning outcomes.
Medical Teacher 21(6) 546-552.
30
ACGME General Competencies (1/3)

• Patient Care
• Caring and respectful behaviors
• Interviewing
• Informed decision making
• Patient management
• Educating patients and families
• Performing procedures
• Providing preventive health services
• Working with a team
• Medical Knowledge
• Investigatory and analytical thinking
• Knowledge and application of basic science
• 
  (continued)

_______________ ACGME. 2000. Outcomes Project.
31
ACGME General Competencies (2/3)

• Practice-based Learning and Improvement
• Analyze own practice for improvement
• Use of evidence from scientific studies
• Application of research and statistical methods
• Use of information technology
• Facilitate learning of others
• Interpersonal and Communication Skills
• Creation of therapeutic relationship with
  patients
• Listening skills
• 
  (continued)

_______________ ACGME. 2000. Outcomes Project.
32
ACGME General Competencies (3/3)

• Professionalism
• Respectful, altruistic
• Ethically sound practice
• Sensitive to cultural, age, gender, disability
  issues
• Systems-based Practice
• Understand interaction of practice in larger
  system
• Knowledge of practice and delivery systems
• Practice cost effective care
• Advocate for patients within the health care
  system

_______________ ACGME. 2000. Outcomes Project.
33
An Evolving Competency Model over the Continuum
of Medical Education
PRACTICE
GRADUATE MEDICAL EDUCATION
MEDICAL SCHOOL
COLLEGE
34
Articulation Through the Continuum
COMMUNITY SETTINGS
INSTITUTIONAL SETTINGS
PRACTITIONER
CanMEDS
MEDICAL EXPERT
COMM./ COLLAB.
HEALTH ADVOCATE
PROFES- SIONAL
MANAGER
SCHOLAR
RESIDENT
ACGME
PRACTICE- BASED LEARNING
COMMUNI- CATION SKILLS
SYSTEM- BASED PRACTICE
PATIENT CARE
MEDICAL KNOWLEDGE
PROFES- SIONALISM
ADVOCATE
MEDICAL STUDENT
COM- MUNICATOR
PROFES- SIONAL
WVU
COLLA- BORATOR
SCHOLAR
CLINICIAN
UNDERGRADUATE
LIBERAL ARTS REQS
SCIENCE MAJOR REQS
35
Assessing Competence
36
(No Transcript)
37
Part 2 Objectives Assessment of Competencies

• 5. The learner should be able to explain the
  difference between norm-referenced and
  criterion-referenced assessment,
• 6. understand assessment authenticity and the
  move to more performance-based assessment
  practices, and
• 7. summarize the current issues surrounding
  assessment practices and the need for further
  development and study.

38
Understanding A Basic Philosophy

• Norm-referenced Assessment
• vs.
• Criterion-referenced Assessment

39
Assessment Criteria

• Reliability
• Validity
• Cost
• Good assessment is costly.
• The cost of assessment requires compromise in
  practice.
• Investing in assessment is investing in teaching
  and learning.
• Impact on learning
• Assessment drives learning through its content,
  format, information given, and programming
  (frequency, timing)
• Exams are the hidden curriculum. Exams define
  academic success for the learner.
• Exams can reinforce (un)desirable learning
  behavior.
• The effects of assessment are often difficult to
  assess.
• Any assessment action will result in an
  educational reaction.

40
Reliability
41
Reliability
42
Reliability
43
Validity
44
Validity
45
Validity
46
Categories of Assessment Instruments

• Written Assessments
• MCQs, CRQs, EMIs, PMPs
• Essays, Short Answer
• Practicals
• Diagnostic Thinking Inventory
• Progress Tests
• Clinical/Practical Assessments
• OSCE
• Simulations
• Standardized (Simulated) Patients
• Simulations Models

• Observation
• Checklists
• Rating Scales
• Patient reports
• 360 Evaluations
• Portfolio Assessments
• Logbooks
• Portfolios
• Procedural logs
• Peer- Self-Assessment
• Peer report
• Self report
• (360 Evaluations)

_________________________________ Shumway J.M.
and Harden R.M. AMEE Guide No. 25. The
assessment of learning outcomes for the competent
and reflective physician. Medical Teacher, Vol
25, No. 6, 2003, pp. 569-584.
47
Levels of Competence
Does
Shows How
Knows How
Knows
_______________ Miller GE. 1990 The assessment
of clinical skills / competence / performance.
Academic Medicine,. 65(Supplement) p. S63-S67)
48
Levels of Competence
Does
Shows How
Knows How
Tests of Knowledge MCQ, CRQ, EMI, SAQ.
Knows
_______________ Miller GE. 1990 The assessment
of clinical skills / competence / performance.
Academic Medicine,. 65(Supplement) p.
S63-S67) Van der Vleuten C. 2000 The Assessment
of Professional Competence Developments,
Research and Practical Implications. Ottawa Conf.
Presentation. South Africa, 2000.
49
Levels of Competence
Does
Shows How
Clinical Based Tests PMPs, Practicals, Essay,
Orals.
Knows How
Tests of Knowledge MCQ, CRQ, EMI, SAQ.
Knows
_______________ Miller GE. 1990 The assessment
of clinical skills / competence / performance.
Academic Medicine,. 65(Supplement) p.
S63-S67) Van der Vleuten C. 2000 The Assessment
of Professional Competence Developments,
Research and Practical Implications. Ottawa Conf.
Presentation. South Africa, 2000.
50
Levels of Competence
Does
Performance Assessment Observation, OSCEs, SPs.
Shows How
Clinical Based Tests PMPs, Practicals, Essay,
Orals.
Knows How
Tests of Knowledge MCQ, CRQ, EMI, SAQ.
Knows
_______________ Miller GE. 1990 The assessment
of clinical skills / competence / performance.
Academic Medicine,. 65(Supplement) p.
S63-S67) Van der Vleuten C. 2000 The Assessment
of Professional Competence Developments,
Research and Practical Implications. Ottawa Conf.
Presentation. South Africa, 2000.
51
Levels of Competence
In Practice Real Patients, Videos, Logs,
Undercover SPs.
Does
Performance Assessment Observation, OSCEs, SPs.
Shows How
Clinical Based Tests PMPs, Practicals, Essay,
Orals.
Knows How
Tests of Knowledge MCQ, CRQ, EMI, SAQ.
Knows
_______________ Miller GE. 1990 The assessment
of clinical skills / competence / performance.
Academic Medicine,. 65(Supplement) p.
S63-S67) Van der Vleuten C. 2000 The Assessment
of Professional Competence Developments,
Research and Practical Implications. Ottawa Conf.
Presentation. South Africa, 2000.
52
Matching Assessment Approachesto Competency

• Patient Care
• OSCEs, SPs, Simulations, Logs, Practicals,
  Observation
• Medical Knowledge
• MCQ, Essay, Progress Tests
• Practice-Based Learning Improvement
• Peer- Self-Assessment, Portfolios
• Interpersonal Communication Skills
• Peer- Self-Assessment, Portfolios, Observation,
  OSCEs, 360 Evaluations
• Professionalism
• 360 Evaluations, OSCEs, Observation
• Systems-Based Practice
• Observation, Portfolios

53
Example of an Assessment Profile for a Student
ACHIEVEMENT
Min. Standard
Patient Care Medical Practiced-
Interpersonal Professionalism
Systems- Knowledge
Based Communication
Based
Learning Skills

Practice
E.g. ACGME Competencies
54
Implications for Further Development and Study

• Develop faculty and staff.
• Pay attention to standard setting
• Understand what marks mean.
• Have a quality control process in place.
• Listen to student input.
• Use evidence-based assessment practices.
• Use test blueprints to adequately sample the
  content domain.
• Set standards for assessing in the clinical
  setting.
• Include the use of qualitative measures.

55
Characteristics of the Future

• Accessible and affordable for diverse populations
• Learner-centered
• Focus on the continuum of education
• Patient- and system-centered
• Evidence- and competence-based (all domains)
• Culturally sensitive and effective
• Multidisciplinary teams and technologically based
• Reduction of medical errors
• Population oriented

56
Now, you should be able to

• Understand what we mean by competency.
• Distinguish among types/levels of competency.
• State common themes of competency models.
• Understand the relationship of competency and the
  assessment of such.

57
A Students View of Competency?

• There is something I dont know that I am
  supposed to know?
• I dont know what it is I dont know, and yet am
  supposed to know.
• And I feel I look stupid if I seem both not to
  know it and not know what it is I dont know.
• Therefore I pretend I know it.
• This is nerve-racking since I dont know what I
  must pretend to know.
• Therefore I pretend to know everything.
• I feel you know what I am supposed to know but
  you cant tell me what it is, because you dont
  know that I dont know what it is.
• You may know what I dont know.
• But not that I dont know it, and I cant tell
  you.
• So you will have to tell me everything.

_________ R.D. Laing. Knots