AETC National Evaluation Center

1
AETC National Evaluation Center

• Kevin Khamarko, MA
• Assistant Director, AETC NEC
• AIDS Policy Research Center
• Kevin.khamarko_at_ucsf.edu
• http//aetcnec.ucsf.edu/

2
Outline

• The AETC network and their training levels
• Services provided by the NEC
• The NECs new focus on how people use what they
  learn and cross-region evaluation
• Overview of current NEC evaluation projects,
  including
• Application Focus
• Barriers and Facilitators Evaluation Project
• Minority AIDS Initiative Chart Review Evaluation
  Project
• Cross-site Focus
• Cross-region Training Evaluation Project
• Behavior Change Evaluation Project
• Cross-region HIV Testing Evaluation Project

3
Overview of the AETCs

• Federally funded via HRSA/HAB
• Provide clinical HIV training and education to
  medical professionals
• National network of 11 Regional Centers and 4
  National Centers
• Primary audience
• Physicians
• Nurses, Nurse Practitioners
• Physician assistants
• Dentists
• Clinical pharmacists
• Other ancillary health care staff

4
AETC Program
5
AETC Levels of Training

• Level 1
• Didactic presentations, panel discussions,
  self-instructional materials, and teleconferences
• Level 2
• Interactive learning through discussion of cases
  supplied by trainer, role play, simulated
  patients, and other skill building activities
• Level 3
• Preceptorships, mini-residencies, and
  observations of clinical care
• Level 4
• Clinical consultation (distance-based and
  on-site)
• Level 5
• Technical assistance and capacity building

6
Why Evaluate Trainings?

• To determine whether the training program is
  achieving the identified objectives
• To provide information to trainers for improving
  course design, content and delivery
• To determine whether participants acquire the
  intended knowledge and skills and are able to
  apply them to the job

7
AETC National Evaluation Center

• Serves as resource to regional AETC sites
• Evaluation training web-based modular curriculum
• Evaluation community building
• Evaluation planning design
• IRB (human subjects) training and TA
• Implementation
• Linking process data with outcome-oriented data
• Analysis
• Dissemination

8
NEC Goals

• To provide leadership in the development, design,
  testing and dissemination of effective evaluation
  models with emphasis on outcomes
• Determine the outcomes of AETC clinical education
  and training programs with respect to changes in
  provider behavior and clinical practice

9
Kirkpatricks Model(Kirkpatrick, D.L.,
Evaluating Training Programs, 2nd edition, 1998)
10
Training Evaluation Framework
During Training
After Training
Before Training
Training Process Evaluation Number and Type of
Activities, Trainee Reaction
Training Outcome Evaluation Learning,
Provider Behavior Change
Pre-Training Conditions Individual
Characteristics, Training Motivation, Pre-traini
ng Environment
Training Impact Evaluation Results (Patient
Outcomes)
Columbia NEC
UCSF NEC 1
UCSF NEC 2
UCSF NEC 2
11
Current AETC NEC Evaluation Projects

• Application Focus
• Barriers and Facilitators Evaluation Project
• Minority AIDS Initiative Chart Review Evaluation
  Project
• Cross-site Focus
• Cross-region HIV Testing Evaluation Project
• Cross-region Training Evaluation Project
• Behavior Change Evaluation Project

12
Barriers and FacilitatorsEvaluation Project

• Purpose
• To identify whether and how trainers/faculty
  address issues in overcoming barriers to applying
  the information learned, so trainees can actually
  make behavior changes in their clinical practice
  settings
• To utilize the information gained to develop case
  studies and pre-training quantitative questions
  to evaluate the outcomes of training on provider
  practice and behavior change
• To link trainees responses to the pre-training
  quantitative questions with follow-up questions,
  which will aid the AETCs in better understanding
  issues in application

13
Barriers and FacilitatorsEvaluation Project

• Qualitative interviews with AETC regional
  directors and evaluators
• Participants are asked about the organizational
  structure of their AETC and what kinds of things
  affect whether their trainees can apply what
  theyve learned to their setting
• Qualitative interviews with AETC faculty and
  training coordinators
• Participants are asked broadly about whether
  their trainees had made changes following
  training and what barriers and facilitators they
  experienced during that process

14
Barriers and FacilitatorsEvaluation Project

• Project Status
• The NEC began interviewing regional directors and
  evaluators in April 2008
• To date, weve interviewed 4 regional directors
  and 5 regional evaluators
• Preliminary findings include
• Facilitator long-term relationships with
  trainees and trainee environments facilitate
  uptake of training
• Barriers thus far relate to conducting trainings
• Geographical challenges (e.g., rural areas)
• Recruiting trainees can be difficult
• In general, regional AETCs are interested in
  conducting evaluations of training application

15
Minority AIDS Initiative Chart Review Evaluation
Project

• Purpose
• Cross-region evaluation facilitated by the NEC to
  measure changes in clinical practice and
  patient-related clinical indicators associated
  with improved quality and reduced disparities in
  care resulting from MAI programs
• Project assesses the impact of MAI-funded
  clinical training on the quality of HIV clinical
  care using information abstracted from patients
  medical charts

16
Minority AIDS Initiative Chart Review Evaluation
Project
17
Minority AIDS Initiative Chart Review Evaluation
Project

• Methods
• Data from the baseline chart abstraction on 30
  patients in each clinic will be used to assess
  adherence to clinical practice guidelines.
• Chart abstraction team provides oral and/or
  written summary of findings to local clinical
  and/or AETC trainers.
• Local AETC trainers may choose to target training
  based on feedback.

18
Minority AIDS Initiative Chart Review Evaluation
Project

• Methods
• 1-year later, after implementation of MAI-funded
  clinical training, return to assess change in
  targeted aspects of clinical care at each clinic.
• We anticipate information from 30 patients in
  each clinic (N690) to be abstracted at both
  baseline and follow-up.
• We anticipate a 10 change in the degree to which
  aspects of clinical care targeted by MAI-funded
  clinical training in each clinic are consistent
  with clinical practice guidelines.

19
Minority AIDS Initiative Chart Review Evaluation
Project

• Project Status
• The NECs nurse team leader has been hired and
  trained by staff at the Florida/Caribbean AETC in
  the administration of their chart review protocol
• Baseline data collection has been completed in 14
  clinics we anticipate completion of baseline
  data collection by the end of June 2008
• Follow-up data collection has been completed in
  9 clinics
• Retrospective data collection for both baseline
  and follow-up will be employed in 3 clinics this
  abstraction will be complete by August 2008

20
Minority AIDS Initiative Chart Review Evaluation
Project

• Preliminary findings include
• Baseline chart abstraction has served as a
  valuable needs assessment tool
• Local AETC trainers have been receptive to
  feedback from chart abstraction and have used
  information to target future trainings
• We have identified many individual,
  organizational, structural and social barriers
  which impede the provision of primary HIV care

21
Linking the Two Evaluation Projects

• Because of the time spent within a clinic, NEC
  evaluators are uniquely situated to understand
  potential barriers (individual, organizational,
  structural and/or social) providers may face in
  attempting to apply what they learned during a
  training
• Based on our 3-day experience in the clinics, the
  NEC can provide important insight into previously
  unforeseen obstacles that may impede the uptake
  of training information and help to inform and
  tailor local trainings to best meet the needs of
  clinics

22
Cross-region Training Evaluation Project

• National project involving all regional AETCs and
  the National Minority AETC
• 3 quantitative questions selected for
  implementation in Level 1-3 training programs
• Questions will be asked immediately after the
  training program
• Data from the evaluation questions will be sent
  to the AETC NEC for cross-site analysis
• More questions will be added in the near future

23
Cross-region Training Evaluation Project

• Questions
• How would you rate your level of knowledge about
  this content?
• Likert-type scale with responses ranging from
  Novice (1) to Expert (5)
• How would you rate the overall quality of the
  program?
• Likert-type scale with responses ranging from
  Poor (1) to Excellent (5)
• I can apply the information learned in my
  practice/service setting.
• Likert-type scale with responses ranging from
  Disagree Strongly (1) to Agree Strongly (5)

24
Behavior Change Evaluation Project

• Purpose
• To measure whether trainees demonstrate and apply
  in their practice the information learned
  (knowledge and skills) following the level 2
  training attended
• A set of open-ended questions, based on response
  logic, were created to better understand the
  effects of training across AETC regions
• To utilize the information gained to translate
  the open-ended questions into quantitative
  questions for use across the AETC network

25
Behavior Change Evaluation Project

• Collaborating regions include
• Midwest AETC, Northwest AETC, Pacific AETC,
  Texas/Oklahoma AETC and the National Minority
  AETC
• Methods
• Online evaluation project, requiring the
  collection of email addresses
• Participants will be contacted 6-8 weeks
  following the training

26
Behavior Change Evaluation Project

• Trainees able to apply the information learned
  are asked the following questions
• Provide an example or examples of how you have
  been able to apply the information learned during
  the blank training.
• What helped facilitate your ability to apply the
  information learned?
• What obstacles did you face in applying the
  information and how did you address these
  obstacles?

27
Behavior Change Evaluation Project

• Trainees not able to apply the information
  learned are asked
• Why were you not able to apply what you had
  learned at blank training?
• Trainees not yet able to apply the information
  learned are asked
• If you plan to apply what you learned, describe
  how you will do so?

28
Cross-region HIV Testing Evaluation Project

• National project involving all regional AETCs and
  the National Minority AETC
• Purpose to assess changes in the ability to
  provide HIV screening in clinics receiving
  enhanced capacity building assistance as part of
  the AETC HIV Training Initiative
• The evaluation for this project is still in the
  planning phase

29
Benefits of Cross-region Evaluation

• Allows the AETC network to present their
  successes and/or challenges in a collective
  manner
• Provides larger sample size for data analysis
• Also benefits our funders, as they are able to
  report the accomplishments across the AETC
  network and advocate for us based on our findings

30
Lessons Learned

• Easier to design a cross-region evaluation for
  the HIV testing initiative since most regions do
  not have an evaluation program in place
• Developing quantitative questions for 12 sites
  can be difficult, as the context of each site is
  different, so requires different response
  categories, etc.
• Regions are doing an excellent job of evaluating
  their programs and many of their questions can be
  tailored by other regions for evaluation purposes

31

• Questions?
• Email kevin.khamarko_at_ucsf.edu