Evaluating Readiness: Course Content and Program Evaluation Bioterrorism Training and Curriculum Dev

1
Evaluating ReadinessCourse Content and Program
EvaluationBioterrorism Training and Curriculum
Development ProgramSeptember 21, 2004

• Lynn Rothberg Wegman, M.P.A.
• Director, Division of State, Community, Public
  Health (DSCPH)
• Bureau of Health Professions (BHPr)
• Health Resources and Services Administration
  (HRSA)
• Department of Health and Human Services (DHHS)

2
Evaluation

• Varying levels of evaluation
• Individual health care providers level of
  preparedness
• Is each individual course/module an effective
  training tool?
• Is each grantee effective in overall training
  strategy for the geographic area?
• Is the national BTCDP an effective means of
  training the Nations health care providers?
• Ultimate question Are the Nations health care
  providers better prepared to respond in an
  emergency?

3
Preparedness Needs of Healthcare Workforce

• True need vs. perceived need
• Training
• Drills/Exercises
• Retention/re-training
• Level of competencies- who needs to know what?
• Everyone needs to have basic level of training-
  who needs intermediate or advanced (who
  determines the need), how should it be delivered?
• HCPs have different roles in different states

4
Issues to Consider in Training the Healthcare
Workforce

• Important to evaluate whats working and whats
  not working and why
• Take developed courses and modify them for the
  audience
• Different states, different institutions have
  different needs and procedures to provide
  training
• Coordinate drills with HRSA Hosp Prep, DHS, and
  CDC as an evaluation strategy
• JCAHO will allow tabletop drills as long as the
  community is brought into it
• Full scale vs. tabletop drills

5
Issues to Consider in Training the Healthcare
Workforce (contd)

• Need evaluators trained to observe drills
• Students (CD) should be involved in simulations
  rather than drills
• Can learn from large scale drill if you can
  evaluate it properly (have enough staff, etc.)
• Practicing HCPs dont necessarily want public
  health-focused training and competencies

6
How Do We Measure Preparedness?

• Possible options to consider
• Core Competencies
• 15 potential DHS scenarios of biological,
  chemical, nuclear, natural disasters, etc.
• Universal Task List of general needs to respond
  to the 15 scenarios (not details of how to
  respond)
• CDCs 19 preparedness topics
• Columbias Hospital Workforce competencies

7
Curriculum Development Perspective
8
CD Core Content

• What level of training is required for health
  professions students?
• Defined by the future role of each health care
  provider serving in an emergency
• Basic competencies for all health care providers
  vs. targeted, advanced competencies for specific
  groups of providers
• Must knows for all core concepts
• Add discipline-specific material (same topics but
  varying levels of knowledge)

9
CD Evaluation

• How are competencies being measured?
• Competencies defined as a measurable
  skill/knowledge
• Areas of student evaluation (pre and post tests)
• Cognitive knowledge of preparedness
• Attitudes toward preparedness
• Prior exposure to preparedness content
• National health professions licensing exams what
  questions relate to emergency preparedness?
• CE requirements- one state requires 2 hours of CE
  on emergency preparedness for licensure renewal
• Problem-based learning (core)
• Schools creating own evaluation

10
CD Program Evaluation

• Needs Assessment
• Course Content Validity
• Faculty Evaluation and Development
• Course Evaluation for faculty, students, and
  instructional technology methods

11
Continuing Education Perspective
12
CE Content

• There should be a basic core program that leads
  to more detailed or advanced training
• What are the real basics that everyone needs to
  know?
• Role
• Piggyback on training on requirements (like JCAHO
  requirements on training for all workers)
• Practicing professionals want to know what their
  responsibilities are and not all nuances about
  diseases

13
CE Content

• Need to provide training for health
  professionals not just first responder training
• Match local training needs to national goals
• Tiered educational programs are part of the
  answer
• Just in Time training to help combat burn-out on
  terrorism preparedness training
• Need to refer back to the four program goals in
  designing/developing content

14
CE Content

• ½ doing NIMS all doing HEICS
• ½ doing in depth courses
• All doing broad spectrum training
• More than ½ are part of a state strategy for
  training healthcare professionals all believe
  they should be part of a comprehensive state
  strategy

15
Evaluation Competencies

• Have to be based on training needs analysis for
  each group of trainees
• CE grantees develop their own competencies based
  on variety of available competency sets
• Competencies and objectives are derived from
  needs assessments
• ½ of CE awardees believe no single set of
  national competencies exist that are easily
  adaptable to state and local needs
• Curricula in the CE program use competencies
  drawn from a variety of sources ODP/ERG, ACEP,
  CDC/Columbia, ASTM, etc.

16
Evaluation Competencies

• Competencies should not be developed without
  including appropriate groups accrediting bodies,
  professional associations, federal agencies,
  provider groups
• Everyone believes there needs to be consensus on
  national preparedness competencies for basic
  level training
• Competencies must be broadly defined at national
  level, but with specifics developed at awardee
  level

17
Evaluation Courses

• Pre/Post testing alone is not sufficient
• Tiered approach to evaluation
• Future follow-up testing alone may not be
  sufficient to determine knowledge/competencies
• Basic competencies re-delivered may be better
  than evaluations of retention of specifics

18
Evaluation BTCDP Program

• How can we help HRSA determine whether we have
  met the four program goals?
• National impact
• Numbers trained by discipline
• Contact hours
• Geographic region
• Population served
• Special populations
• By describing how we are helping the state meet
  preparedness goals

19
BTCDPQuestionnaire
20
CE Awardees

• Are you implementing a basic emergency
  preparedness orientation course?
• Number of CE Awardees 19
• Number of CE Respondents 17
• Number of Yes Responses 16

21
Data from CE Awardees
22
Data from CE Awardees
23
CE Awardees

• Are you using core competencies in your
  bioterrorism and emergency preparedness courses?
• Number of CE Awardees 19
• Number of CE Respondents 17
• Number of Yes Responses 14

24
CE Awardees

• Have you conducted any community-based drills as
  part of your curriculum or course content?
• Number of CE Awardees 19
• Number of CE Respondents 17
• Number of Yes Responses 8

25
CD Awardees

• Are you implementing a basic
• emergency preparedness orientation
• course?
• Number of CD Awardees13
• Number of CD Respondents 13
• Number of Yes Responses 8

26
Data from CD Awardees
27
Data From CD Awardees
28
CD Awardees

• Are you using core competencies in your
  bioterrorism and emergency preparedness courses?
• Number of CD Awardees 13
• Number of CD Respondents 13
• Number of Yes Responses 12

29
CD Awardees

• Have you conducted any community-based drills as
  part of your curriculum or course content?
• Number of CD Awardees 13
• Number of CD Respondents 13
• Number of Yes Responses 3

30
Thank You

• Contact Information
• Lynn Rothberg Wegman, M.P.A.
• Director, Division of State, Community Public
  Health, HRSA, BHPr
• 301-443-1648
• lwegman_at_hrsa.gov