Development of a Rating Scale to Assess Geropsychology Practice Competencies

1
Development of a Rating Scale to Assess
Geropsychology Practice Competencies

• Michele J. Karel, PhD
• VA Boston Healthcare System
• Harvard Medical School
• For the CoPGTP Task Force on Geropsychology
  Competency Assessment
• Michele Karel, Jeannette Berman, Jeremy Doughan,
  Erin E. Emery, Victor Molinari, Sarah Stoner,
  Yvette N. Tazeau, Susan K. Whitbourne, Janet
  Yang, Richard Zweig
• APPIC Conference, April 18, 2009

2
Objectives

• Increase awareness of efforts to define and
  evaluate competencies for geropsychology practice
• Consider challenges for designing and
  implementing a competency rating scale
• Utility in internship and postdoc settings
• Address ongoing challenges and directions for
  work in this area

3
Importance of Professional Geropsychology
Training and Evaluation

• Aging population
• Workforce shortages for geriatric health care,
  across disciplines
• Relatively few psychologists have formal training
  in geropsychology
• Increasing numbers will be working with older
  adults
• How to evaluate competence, and define training
  needs, for psychological practice with older
  adults
• How to help training programs, and post-licensure
  psychologists, expand opportunities for
  geropsychology training

4
Growth of Professional Geropsychology

• Growing number of grad, intern, and fellowship
  programs offering geropsychology training
• Growth of research, journals, professional
  organizations
• Several recent milestones
• APA Office on Aging, established 1998
• APA Guidelines for Psychological Practice with
  Older Adults (2004)
• CRSPPP recognition of gero as proficiency area,
  1998 and 2005
• Pikes Peak Model for Geropsychology Training,
  2006 National Training Conference
• Pikes Peak Training Model defined Attitude,
  Knowledge, Skill competencies for geropsychology
  practice
• How to evaluate professional geropsychology
  competence?
• Knight, B.G., Karel, M.J, Hinrichsen, G.A.,
  Qualls, S.H., Duffy, M. (in press). Pikes Peak
  model for training in professional
  geropsychology. American Psychologist.

5
Project Initiation

• Council of Professional Geropsychology Training
  Programs (CoPGTP)
• Formed after 2006 Pikes Peak conference
• For training programs at all levels
• www.uccs.edu/cpgtp/
• CoPGTP undertook to explore methods to evaluate
  Pikes Peak competencies
• CoPGTP established a Task Force
• Agreed to start with goal of developing a
  competency rating scale

6
Project Goals

• Develop an evaluation tool that would
• Address the Pikes Peak geropsychology attitude,
  knowledge and skill competencies
• Be useful for supervisor evaluation of trainees
• Formative and summative evaluation
• Be useful for self-evaluation by learners
• Be useful across levels of training
• Graduate practicum, internship, fellowship,
  post-licensure
• Be useful to help define ongoing training goals

7
Pikes Peak Geropsychology Practice Competencies

• Competency definitions informed by
• APA Guidelines for Psychological Practice with
  Older Adults (2004)
• Cube Model for Competency Development (Rodolfa et
  al., 2005)
• Psychology foundational and functional
  competencies develop over course of training
  career
• 2006 Pikes Peak Conference process
• Working groups address 6 competency domains
• (separate working groups address 4 levels of
  training)
• Iterations of competencies lists during
  conference, and afterwards via listserv discussion

8
Pikes Peak Competencies

• Are aspirational
• Aim to facilitate as many psychologists as
  possible to work effectively with older adults,
  not to pose barriers to joining the field
• Competencies are aimed at level of practice of a
  newly licensed psychologist who has finished one
  year of geropsychology postdoctoral training
• Competencies not geared to level of an expert,
  but to someone who practices with wide range of
  older adults, families, systems

9
Pikes Peak Competencies Attitudes

• Work within ones scope of competence
• Recognize how ones beliefs about aging and older
  adults may affect practice
• Appreciate diversity among older adults, and
  interactions between age/cohort and other aspects
  of individual diversity
• Seek continuing education, supervision,
  consultation

10
Pikes Peak Competencies Knowledge Base

• General knowledge about adult development, aging,
  and the older adult population
• Foundations of professional geropsychology
  practice
• Assessment
• Intervention
• Consultation

11
Pikes Peak Competencies Skills

• Professional Geropsychology Functioning
• i.e., foundational skills applied to
  geropsychology practice
• Assessment
• Intervention
• Consultation
• Note Skills for functional domains of research,
  supervision, and management are important for
  growth of geropsychology, but were not included
  in Pikes Peak core competencies for practice with
  older adults

12
Pikes Peak CompetenciesSettings of Care

• Geropsychologists should be able to deliver
  services effectively in range of (at least 2)
  settings, such as
• Outpatient mental health services
• Outpatient primary care/medical settings
• Inpatient medical service
• Inpatient psychiatric service
• Long-term care settings including nursing homes,
  assisted living facilities, home care, day
  programs
• Rehabilitation settings
• Hospice
• Others

13
CoPGTP Competency Evaluation Task Force
Methodology

• Monthly conference calls, 9/07 to 7/08
• Reviews of relevant, informing materials
• Small working groups define behavioral anchors
  for each competency domain
• Iterative reviews of all scale items by larger
  group
• Group discussion, selection, editing of
• Rating scale/anchors and vignette
• Introduction to the evaluation tool
• Task Force and other CoPGTP members try out the
  scale to evaluate a student, oneself, or both

14
Informing Materials Psychology Competencies
Assessment

• Guiding principles and recommendations for the
  assessment of competence
• (Kaslow et al., 2007 and other publications of
  APA Task Force on Assessment of Competence)
• Assessment of Competency Benchmarks Work Group
• A developmental model for defining and measuring
  competence in professional psychology
• Joint effort of APA BEA and CCTC
• The Practicum Competencies Outline
• Hatcher Lassiter (2007), workgroups of ADPTC
  and CCTC

15
Informing MaterialsGeropsych Competency
Assessment

• Example of evaluation tools used in various
  geropsychology practicum, internship, and
  postdoctoral programs
• Also, examples of competencies and evaluation
  methods from
• Medicine ACGME Outcome Project, 2000
• Geriatrics, Palliative Care, Geriatric Social
  Work, Geriatric Psychiatry

16
Delineation of Geropsychology Knowledge and Skill
Competencies

• Each Pikes Peak competency domain (e.g.,
  foundational geropsychology skills) is defined by
  3-8 specific competencies (e.g., ethical
  practice, appreciation of diversity,
  self-reflection)
• We aimed to further specify each of these
  competencies with behavioral anchors
• How would you know it if you saw it (in oneself
  or another)?
• e.g., what does it really mean to Relate
  effectively and empathically with older adults
  clients, families, and other stakeholders in a
  range of professional roles and settings?
• This approach leads to a very long evaluation
  tool!

17
Example Delineating one of the foundational
skill competencies
18
Example Delineating one of the assessment skill
competencies
19
Evaluation of Attitudes

• Pikes Peak attitudes for competent geropsychology
  practice are embedded in the foundational skills
• E.g., self-reflection re biases/discomforts,
  awareness of limits of ones competence, seeking
  consultation, seeking continued education
• Attitudes are specified in tools introduction
• For now, attitudes not evaluated separately
• Perhaps they should be? How?

20
Rating Scale

• Developmental rating scale
• Novice, Intermediate, Advanced, Proficient,
  Expert
• Scale anchors conceptually complex
• Adapted definitions from Hatcher and Lassiter
  (2007)
• Included in anchor definitions how much
  supervision/consultation likely needed
• Provided vignette and approach to that case to
  illustrate each level of competence
• Instructions to rate each Pikes Peak competency,
  not each behavioral specifier (but can if
  desired)

21
Rating Scale

• Abbreviated scale anchors, printed on each page
• N Novice Possesses entry-level skills needs
  intensive supervision
• I Intermediate Has a background of some
  exposure and experience ongoing supervision is
  needed
• A Advanced Has solid experience, handles
  typical situations well requires supervision for
  unusual or complex situations
• P Proficient Functions autonomously, knows
  limits of ability seeks supervision or
  consultation as needed
• E Expert Serves as resource consultant to
  others, is recognized as having expertise

22
Rating Scale

• Expected ratings vary considerably through levels
  of training
• Practicum Novice through Advanced
• Internship Novice through Proficient
• Fellowship and post-licensure Intermediate
  through Expert
• Ideally, usable to evaluate a trainee, AND to
  evaluate oneself

23
Introduction to the Evaluation Tool

• Reviews
• Purpose of tool
• Pikes Peak competencies as aspirational
• Application across range of geropsychology
  practice and training settings
• Instructions for use
• Including that evaluation should include the
  learner's perspective (self-assessment),
  observation of the learner's work, and regular
  supervision involving case discussion
• Psychologists and trainees conducting
  self-assessments can evaluate their training and
  supervision needs in each area
• Definition of rating scale anchors, and
  illustration of its use through a case vignette

24
Piloting the Evaluation Tool

• Asked CoPGTP members, and ourselves, to complete
  the tool to evaluate a student or oneself
• 13 responses
• 6 CoPGTP members (other than us)
• 4 Task Force members
• 3 from Task Force members students/staff
• Used tool to
• 4 to evaluate a student
• 5 to evaluate oneself
• 2 evaluated oneself and a student
• 2 provided general feedback without specifying

25
Asked for Feedback

• Asked folks to answer these questions
• How did you use the tool (to evaluate a student,
  or yourself?)
• What was your general impression of using the
  tool?
• Was the introduction to the tool clear?
• Was the rating scale (Novice, Intermediate,
  Advanced, Proficient, Expert) understandable?
• Was the length of the tool a problem? Did it
  adequately address breadth/depth of
  geropsychology competencies?
• What do you think the best use of this tool will
  be?
• Do you expect to use this tool? In what way?
• Any other feedback?

26
Feedback

• Most felt tool is very long, but worth it given
  attention to breadth and depth
• A few felt it was too long for practical purposes
• (Challenge how to cut?)
• Most felt introduction was clear, but needed
  clarification not to rate each specifier
• Most liked the NIAPE rating scale

27
Feedback

• Concerns raised about the scale included
• Does it apply as well to knowledge as skills?
• Does it work as well for practicum level training
  where students may be novices in most areas?
• Is Expert rating too loaded (people not
  comfortable using that term?)
• Can scale capture development over time?
• If/how best to evaluate attitudes
• Suggestion to have summary page
• Strengths
• Areas for growth
• Ongoing training goals

28
Use of the Scale to Date

• Disseminated to CoPGTP training programs and APA
  12-2 and 20, PLTC listservs
• Programs (grad, intern, and postdoc) are starting
  to use the tool for
• Initial training needs assessment with students
• Mid-year/final supervisory evaluations
• Student self-evaluation (encouraging
  metacompetence)

29
Use of the Scale to Date

• Tool being used by some for program self-study
  and development
• Defining training program objectives
• Evaluating, in discussion with students and
  supervisors, how well program is addressing each
  competency domain
• Helping to evaluate competence of new
  geropsychology supervisors and define their
  training goals
• Being used in community/CE education programs on
  geropsychology

30
Uses for Intern/Postdoc Training

• Along with APA standards, can help define
  rotation/track training objectives
• Gero evaluation scale is rotation/track specific,
  and adjunctive to training programs more general
  evaluation tools
• Extremely helpful for initial training needs
  assessment because students come in with widely
  varying previous gero experience
• Clinical, academic, research
• Can tailor initial training plan and review over
  course of the training year/rotation
• It is additional work at evaluation time for
  supervisor and student not clear how strong a
  barrier this may pose

31
Study-in-the-Works

• Aims Examine utility and validity of the rating
  scale, initially for self-evaluation purposes
• Which competencies rated as most vs least
  well-developed?
• Do self-rated competencies relate to years of
  training or practice?
• To settings and major activities of practice?
• What feedback is provided after completing the
  scale?
• Planning on-line survey
• Of psychologists who work with older adults and
  geropsychology graduate, intern, posdoc students
• Eventually, plan longitudinal study of
  supervisory evaluations of developing
  geropsychologists

32
Concerns/Challenges

• Will clinicians see such a tool as helpful or
  somehow restrictive to practice?
• Will supervisors/learners use such a long
  evaluation tool (even if can pick and choose
  domains of relevance)?
• Can competencies be reduced/streamlined?
• Can we boil down further core competencies? Are
  some more essential than others? (??)
• May vary across settings/populations of older
  adults
• Will tool be useful as self-evaluation,
  self-study aid?
• Can competency ratings be linked to training
  resources?

33
Opportunities/Future Directions

• Helpful for initial training needs assessment
  with a student, to guide training plan
• May help to develop on-line version, then select
  only domains that are relevant to training
  program, rotation
• Useful as guide to developers of CE programs?
• Develop organized, sequential CE offerings?
• Ultimately, plan to link competencies to training
  resources and opportunities
• Working to keep such list updated at CoPGTP
  website
• Expand toolbox of gero competence evaluation
  tools
• Largest challenge - how to expand interest in,
  and opportunities for, training in professional
  geropsychology