Educational Scholars Program

1
Academic Pediatric Association

• EducationalScholars Program
• PAS May 6, 2008

2
CME Disclosures

• The speakers at this session have no relevant
  financial interests to disclose or conflicts of
  interest to report.

3

• Mid-course monitoring of an educational program
• What are the crucial sources of information?
• How should one respond to these data?

4
How would you apply these guiding principles?

• Planned curriculum 14 learning goals
• Principles of adult learning in all interactions
• Curriculum reflects identified needs of learners
• Hands-on practice of authentic tasks
• Scholarly projects with mentoring and regular
  feedback
• Developmental evaluations of authentic educator
  documents
• Professional networking and peer learning
• Dynamic, evolving learning program, shaped by
  needs assessments, evaluation and feedback from
  participants
• Serious research and evaluation of program to
  help build sustainable support

5
How would you apply needs assessment results?
6
Needs Data See Handout

• Cohort I Years 1 vs 2
• Items with greatest and least reported increase
  in competence
• Cohort 2 Start of Year 1
• Items with greatest and least reported competence

7
Our Response to the Needs Assessment Results

• Modified curriculum each year to address highest
  perceived needs
• Yr 1 Evaluation, qualitative research (ISMs and
  PAS session)
• Yr 2 Education technology and problem learners
  (ISMs)
• Yr 3 Peer feedback (scholarly writing wasnt
  included in the needs assessment)

8
How would you apply program evaluation results?
9
Program Evaluation by Scholars
Scale 1low 5high
10
Program Evaluation by Scholars
Scale 1low 5high
11
Overall Ratings of ESP by Scholars
Scale 1low 9 very high

• Effort expended 5.9
• Value received 6.2

12
Our Response to the Program Evaluation Results

• Modified timeline of assignments
• Reduced number of Intersession Modules
• Continued strong emphasis on project reports and
  EPs
• Considering reducing workshop review requirement
• Hope to increase faculty interactions with
  scholars!

13
How can you use an educational program as a
springboard for educational scholarship?
14
Rationale for the EP Research Project

• ESP Scholars needed EP for career development
• We needed EPs for scholar and program evaluation
• We needed research opportunities to reward our
  volunteer faculty
• RESULT The EP Project

15
Quick Project Overview Since Jan 2006

• Developed EP template based on findings of AAMC
  national consensus conference
• MedEdPortal review and approval
• Construction of EP analysis tool by consensus
  development process, using Scholars EPs as raw
  material
• Revision of template to match tool
• Presentation to AAMC-GEA Steering Comm
• Article submitted to Academic Medicine
• Multiple workshops on EPs

16
Development of the Educator Portfolio Analysis
Tool
Inter-rater Reliability Testing
Tool Development
Template Development
EP Test 4 n2 8 raters
EP Test 3 n3 5 raters
EP Test 5 n25-35 8 raters
EP Test 1 n27 6 raters
List of gt100 Quantitative Items
Tool 1.1 Selected and Combined 43 Items
Tool 1.2 Refined and Reconciled Items
Tool 1.3 Final content and Format
EP Template
List of 52 Qualitative Items
EP Template Revision
EP Template Revision
MedEd Portal Approval
MedEd Portal Review
EP Test 2 n5 4 raters
See handout for item list
17
Future Plans

• Interrater reliability testing with new EPs (with
  article)
• National consensus development process through
  AAMC
• National dissemination via workshops
• EP website with electronic data entry/analysis
  ( to support ESP)
• National norming study

18
ESP Business
19
Review of Program Requirements Cohort 1

• 2 workshop reviews/yr (due at ESP session)
• Intersession Module 1 by Oct 15
• Final Educator Portfolio and updated Curriculum
  Vitae by Jan 15, 2009
• Peer review of another scholars EP by Mar 15
  (substitutes for Intersession Module 2)
• Evidence of peer reviewed acceptance of a ESP
  project publication or national presentation by
  Dec 2009 or thereafter
• Final project report at program completion
• We will ask for EPs and CVs periodically in the
  future

20
Cohort 2 Simplified ESP Assignment Timeline

• At PAS 2 workshop reviews
• July 15 Annual Project Reports
• Aug 30 Program evaluation survey
• Oct 15 Intersession Module 1 (technology in
  educ)
• Jan 15 Updated Educator Portfolios and CVs
• Mar 15 Intersession Module 2 (EP peer review)

21
Agenda for Today
Break

• 800-830   Introduction/orientation
• 830-1200 Turning Scholarly Projects Into
  Scholarly Products
• 830 - 1000 Writing educational scholarship
  articles
• 1000 - 1100 Planning and presenting workshops
  
• 1100 - 1200 Preparing and giving platform
  presentations
• 1200- 130  Lunch and discussion
• Cohort 1 How to take lessons learned from ESP
  to move
• forward in ones career?
• Cohort 2 Discussion of projects
• 130-415 Developing Peer Feedback Skills
• 130-145 Giving and receiving peer feedback
• 145-245  Platform session with feedback
• 245-345  Poster review with feedback
• 345-415  Feedback discussion and ESP session
  evals
• 415-500 Feedback from Scholars on the ESP
  program