Assessment in Psychiatric Rehabilitation for Planning Interventions and Tracking Outcomes

1
Assessment in Psychiatric Rehabilitation for
Planning Interventions and Tracking Outcomes

• Kim MacDonald-Wilson, Sc.D.
• Alexis Henry, Sc.D.
• Sandra Wilkniss, Ph.D.
• Joanne Nicholson, Ph.D.

2
Agenda

• Introduction to Assessment
• Brief Primer on Issues in Measurement
• Person Level Assessment for Intervention Planning
• Program Level Assessment for Outcome Tracking
• Integrating Assessments in PR Programs

3
Introduction to Assessment

• Kim MacDonald-Wilson

4
Review of Assessment Experiences

• Questions
• How do you use assessments?
• What assessments do you use?
• What questions do you have about assessment
  (problems, issues)?

5
What is it?

• Assessment is appraising or estimating the
  attributes or characteristics of a person, a
  group, or programs. The tools of assessment can
  include
• Checklists
• Inventories
• Observational Schedules
• Needs Assessments
• Rating Scales
• Tests
• In PR, we focus most often on
• Assessment of skills, resources, supports
• Measurement of outcomes of our interventions

6
Why Do It?

• Intervention planning
• Understanding clients strengths, resources,
  challenges, and needs
• Understanding the clients values and goals
• Setting intervention goals
• Outcome tracking
• Recording change in key areas
• Demonstrating impact of the intervention

7
Principles of Assessment in PR

• Focus on functioning
• Centered on and driven by person in recovery
• Environment is important
• Useful for practitioners and managers
• Easily administered

8
Issues in Measurement

• Alexis Henry

9
Assessment Methods

• Standardized or informal
• May assess skills, knowledge, attitudes
• Methods
• Observational simulated or situational
• Generally used to assess skills
• Paper-and-pencil self-ratings or checklists
• Generally used to assess knowledge and attitudes
• Interviews
• Can be used to assess skills, knowledge and
  attitudes
• Best for developing rapport and setting
  intervention goals

10
General Considerations in Conducting Assessment

• When
• Intervention planning
• At intake (of course), but also periodically
• Outcome tracking
• When it occurs (of course)
• Short, medium or longer term outcomes
• Where
• Program/clinic in real-world context
• Relevance
• Are assessments relevant to interventions/services
  
• Important for both intervention planning and
  outcome tracking
• Reliability, validity, cultural sensitivity

11
Reliability of Assessment

• Stability, consistency, dependability
• Internal consistency
• how well do items on an instrument measure the
  same concept/construct
• test-retest reliability
• stability over time
• interrater reliability
• stability/agreement across raters

12
Validity of Assessment

• Does assessment measure what it purports to
  measure?
• meaningfulness, usefulness
• Face validity
• Does the assessment look valid to the person
  being assessed
• Content validity
• does the assessment adequately cover the domain
  being assessed
• Construct validity
• does the assessment correspond to the theoretical
  concepts (constructs) underlying the measure
• e.g. self-esteem scale

13
Validity

• Construct Validity
• Convergent and divergent (discriminant) validity
• Criterion-related validity
• concurrent - uses an existing criterion
• predictive - uses a future criterion
• Responsiveness
• Is the assessment sensitive enough to detect
  small changes

14
Cultural Sensitivity

• Considerations
• Gender
• Age
• Race
• Ethnicity and language
• Cultural norms
• Socioeconomic status
• Context/environment

15
Person Level Assessment for Intervention Planning

• Kim MacDonald-Wilson

16
Why conduct assessments for intervention planning?

• Involve the person in rehabilitation and recovery
• Provide direction
• Efficiency
• Mobilize needed resources
• Organize the interventions

17
Domains for Person Level Assessment

• Work
• School
• Housing/Community Living
• Social/Recreational/Community Participation
• Parenting and other Social Roles
• Quality of Life
• Empowerment and Recovery

18
What to Measure in Person Level Assessment

• What is the goal environment or domain?
• Skills strengths and deficits
• What skills are required to be successful and
  satisfied?
• What can the person do or not do?
• Environmental Resources and Supports
• What supports are required to be successful and
  satisfied?
• What does the person have or not have?

19
Assessment of Functioning

• Assessment of Functioning does not Skills
  assessment
• Global Functioning Focus on Outcome of
  Functioning
• Performance in Roles (spouse, worker)
• Performance in Domains (social, psychological)
• Status (employment, educational, residential)
• Tools GAF, BASIS-32, Multnomah Community
  Ability Scale, Life Skills Profile

20
Skill Assessment Examples

• Skill Functioning Focus on Behaviors
• Living
• Test of Grocery Shopping
• Independent Living Skills Survey ILSS
• Client Assessment of Skills, Interests, and Goals
  CASIG
• Work
• Work Behavior Inventory WBI
• Vocational Cognitive Rating Scale - VCRS
• Parenting
• Parenting Well Strengths and Goals

21
Resource Assessment Examples

• Social Support
• MOS Social Support Scale
• Parenting
• Family Resources Scale

22
Examples of Assessments for Intervention Planning

• Observational assessments
• Test of Grocery Shopping Skills
• Vocational Cognitive Rating Scale
• Work Behavior Inventory
• Self-report measures
• ParentingWell Strengths and Goals
• Family Resource Scale
• MOS Social Support Scale
• Interviews
• Independent Living Skills Survey
• Client Assessment of Strengths, Interests and
  Goals (CASIG)

23
Test of Grocery Shopping Skills

• Developed to be used in conjunction with a
  grocery shopping skills training curriculum for
  people with serious mental illness
• Assesses persons ability to locate 10 specific
  common food items in a regular grocery store
• Can be used as a baseline assessment and a
  follow-up assessment after training 2 versions
• Developed by Hamera and Brown at University of
  Kansas Medical Center
• http//www2.kumc.edu/pdrp/interventionandoutcome.h
  tm

24
Test of Grocery Shopping Skills

• Examines the persons ability to find
• Correct item
• Correct size
• Lowest price
• Takes speed into account
• Sample items
• 15 oz. can red kidney beans
• 1 lb. bag salad mix
• 16 oz. frozen corn
• 8 oz. low fat/nonfat strawberry yogurt
• 16 oz. spaghetti

25
Vocational Cognitive Rating Scale

• A 16-item instrument designed to measure
  cognitive skills in the workplace
• Ratings are accomplished by a clinician who has
  observed the individual in the work setting and
  has interviewed the onsite supervisor
• Behavioral anchors are provided for each rating
  on each item
• Developed by Greig, Nicholls, Bryson, Bell
  (2004) at Yale University School of Medicine

26
VCRS Sample Item

• Stays focused when performing a simple task
• Consistently Inferior performance. Is
  consistently off task. Requires nearly continuous
  help from job coach/supervisor to stay focused.
• Occasionally inferior performance. Can remain on
  task for brief intervals, but requires frequent
  monitoring by job coach to stay focused.
• Adequate performance. Generally is able to stay
  on task and keep focused when doing simple tasks
  with minimal or infrequent assistance.
• Occasionally superior performance. Stays on task,
  and only rarely requires assistance with
  focusing.
• Consistently superior performance. Consistently
  completes simple tasks in a focused, organized
  manner.

27
Work Behavior Inventory

• A 36-item work performance assessment
• Social Skills
• Cooperativeness
• Work Habits
• Work Quality
• Personal Presentation
• Ratings on a 5-point scale are accomplished by a
  clinician who has observed the individual in the
  work setting and has interviewed the onsite
  supervisor
• Indicates both strengths and deficits, Behavioral
  anchors are provided for each rating on each item

28
WBI Sample Items

• Each item rated on a 5-point scale from
  1-Consistently inferior performance to
  5-Consistently superior performance
• Expresses positive feelings appropriately
• Accepts constructive criticism without becoming
  upset
• Begins work tasks promptly
• Corrects own mistakes
• Personal hygiene is satisfactory
• General Rating of Work Behavior

29
ParentingWell Strengths and Goals

• A parenting self-assessment and collaborative
  goal setting tool
• Developed for use with parents with mental
  illness and their families
• 26 items covering everyday demands of parenting
• Developed by Nicholson Henry
• Available at www.parentingwell.org

30
ParentingWell Strengths and Goals Sample Items
31
Family Resource Scale

• Assesses family self-sufficiency by determining
  the adequacy of different types of resources in
  households of young children.
• 30 item self-report tool, rated using a 5-point
  Likert-type scale anchored by "not at all
  adequate" to "almost always adequate.
• Covers access to food, shelter, financial
  resources, transportation, health care, time to
  be with family, child care, and time for self.
• Developed by Leet Dunst
• Published in Dunst, C. J., Trivette, C. M.,
  Deal, A. G. (Eds.) (1994). Supporting and
  strengthening families. Methods, strategies and
  practices. Cambridge, MA Brookline Books.
  http//brooklinebooks.com/

32
Family Resource Scale Sample items
33
MOS Social Support Survey

• 19 item self-report tool measuring multiple
  dimensions of social support
• Emotional/informational
• Tangible
• Affectionate
• Positive social interaction
• Designed for use with individuals with long-term
  conditions, including serious mental illness
• Developed by Sherbourne Stewart
• Available from http//www.rand.org/health/surveys
  _tools/mos/mos_socialsupport.html

34
MOS Social Support Survey Sample items
35
Independent Living Skills Survey

• Designed to assess ADLs and IADLs among people
  with schizophrenia and other serious mental
  illnesses
• Two versions
• Self-Report Interview 76 items
• Personal appearance care of possessions food
  prep and storage health maintenance money
  management transportation leisure and community
  participation job seeking and job maintenance
• Informant Interview 104 items
• Same as above also includes eating behaviors
  social relationships
• Both instruments are available in appendix of
  published article Wallace, C. J., Liberman, R.
  P., Tauber, R., Wallace, J. (2000). The
  Independent Living Skills Survey A comprehensive
  measure of the community functioning of severely
  and persistently mentally ill individuals.
  Schizophrenia Bulletin, 26(3), 631-658.

36
Independent Living Skills Survey-I Sample items
37
Clients Assessment of Strengths, Interests and
Goals (CASIG)

• A lengthy, structured interview designed for use
  with people with serious mental illness, covering
  multiple domains of functioning
• Assesses clients goals related to
• Housing/living financial/vocational
  relationships spiritual/religious health
• Assesses clients skills/strengths/needs related
  to
• Lifestyle supports money management health
  management nutrition vocational
  transportation friends leisure personal
  hygiene care of possession medications and side
  effects personal rights cognition symptoms
  risk behaviors
• Assesses satisfaction with
• Quality of life quality of treatment
• Developed by Wallace, Lecomte, Wilde, Liberman
  (2001) at UCLA
• Available at http//www.ct.gov/dmhas/LIB/dmhas/MR
  O/CASIG.pdf

38
CASIG Sample items

• Goals
• Housing/Living Goals
• One year from now, what would you like your
  living arrangements to be?
• Financial/Vocational Goals
• Would you like to improve your money/finances in
  the next year? How might you be able to improve
  them? Do you want to get a job or go to school
  in the coming year?

39
CASIG Sample items

• Skills/Strengths
• Health management
• at any time in the last 3 months did you
• Make most of your own appointments with your
  doctor, case manager, etc?
• Administer your own medication?
• Food preparation
• at any time in the last 3 months did you
• Plan your meals with a health balance of foods?
• Use a stove to prepare meals?

40
Individualized assessment and goal setting

• Goal Attainment Scaling
• Alexis Henry

41
Goal Attainment Scaling

• Framework for developing service goals (outcomes)
  that are measurable, relevant, attainable
• Individualized goals
• Quantitative index of clients progress
• Can compare client to self over time
• Can compare attainment across clients

42
Advantages of GAS

• Not bound by any theoretical orientation,
  treatment or outcome measure
• Allows for individualization of goals
• Allows for prioritizing goals
• Numeric index of performance over time
• Easily computed
• Useful in informal evaluations or in formal
  experimental designs

43
Disadvantages of GAS

• Reliability issue needs to be considered
• Outcomes best assessed by blind judge
• Possibility of floor effect
• Possibility of misuse if goals are too easy

44
Steps in GAS

• Identify overall objective or domain
• Identify specific problem area
• Identify behavior that indicates improvement
• Determine data collection method
• Specify expected outcome
• Specify most and least favorable outcome
• Determine current status and timeframe for
  evaluation

45
Goal Attainment Scale

• -2 most unfavorable outcome
• -1 less than expected outcome
• 0 expected outcome
• 1 greater than expected outcome
• 2 most favorable outcome
• Goals can be weighted to reflect priorities

46
GAS Example 25 y.o. male wants to increase work
hours and earnings

• 2 works 20 hrs/week at 10/hour
• 1 works 10 hrs/week at 10/hour
• 0 works 10 hrs/week at minimum wage
• -1 works less than 10 hours/week
• -2 no longer working

47
GAS Example35 y.o. mother wants to have more
funwith kids age 8 and 10

• 2 family participates in mutually enjoyable
  leisure/recreational activity 1x per week
• 1 family identifies mutually enjoyable
  leisure/recreational activity participate 2x per
  month
• 0 family engages in leisure together about
  once per month
• -1 family leisure/recreational activities
  happen less than once per month
• -2 no regular family leisure/recreational
  activities

48
Program Level Assessment for Outcome Tracking

• Sandra Wilkniss, PhD

49
Program Level Assessment for Outcome Tracking

• Why track outcomes on a program level?
• Domains of outcome tracking
• Clinical Status
• Recovery Status
• Quality of Life
• Functional Status
• Safety and Health
• Context and Methods for collecting and
  interpreting outcomes
• Types of Services and related outcome tracking
  issues
• Sample outcome tracking tools
• Exercise Selecting assessment tools for outcome
  tracking

50
Why track outcomes on a program level?

• Access to services
• Quality of services
• Cost/benefit

51
Domains of outcome tracking

• - Demographics Clinical Status
• Recovery Status/Quality of Life
• Functional Status
• Roles/Community Integration
• Skills
• Resources
• Safety and Health

52
Demographics Clinical Status

• Demographics
• Diagnostic and Historical information
• Service use (baseline and change)
• Physical health related outcomes
• Cognition
• Hospitalization
• Jail and prison contact

53
Recovery/Quality of Life Status

• Domains
• Recovery
• Empowerment
• Self Efficacy
• Hope
• Satisfaction
• Quality of Life
• Methods
• Consumer Self-report tools
• Provider Evaluation tools
• Family/significant other input
• Interviews
• Quantitative
• qualitative
• Proximal markers

54
Functional Status

• Living situation
• Housed or homeless
• Level of independence
• In community of choice
• Education
• Type of institution
• Number of credit hours attempted/completed
• Degree or voc/tech certificate
• Work
• In a job of consumers preference
• Competitive
• Salary and/or benefits
• How much
• For how long
• Advancement
• Follow along support variables

55
Functional Status (continued)

• Community Integration
• Instrumental Roles
• Skills and Personal Resources
• Social
• Vocational
• Financial
• Recreational
• Cognitive
• Others
• Environmental resources and supports available
• Community
• Family
• Social capital
• Others

56
Safety and Health

• Personal and other safety
• Physical Health and Wellness

57
Context and Methods for Collecting and
Interpreting Outcomes

• Context
• Cultural Sensitivity
• Philosophy and interpretation
• Specific measures
• Cognition
• Methods
• Baseline tracking
• Interval tracking
• sources

58
Services and Interventions Offered Type,
quantity, and quality

• Outreach
• Linkage
• PSR
• Individual and group
• Peer Support
• Integrated Health care (primary care mental
  health)
• Pharmacotherapy
• Adjunct Interventions (e.g., cognitive support)
• Evidence-Based or emerging best practices
• GOI/penetration
• Fidelity

59
Sample Tracking Tools

• Comprehensive outcomes packages
• Specific Measures
• Cultural diversity measures
• Recovery/Satisfaction Measures
• Empowerment
• Cognitive Functioning
• Skills
• Environment

60
Exercise Selecting Tools for Outcome Tracking

• Selection Strategies
• Determine program emphasis/focus
• Evaluate needs and preferences of individual
  consumers
• Evaluate resources and means for data collection,
  description/analysis, interpretation and
  dissemination
• Review sample tools in each domain of interest
• evaluate pros and cons
• Psychometric properties vs. uniqueness
• Thorough vs. brief and more accessible
• Language geared toward consumers
• Issues of cultural diversity
• Etc.

61
Integrating Assessments in Psychiatric
Rehabilitation ProgramsWhere the rubber meets
the road

• Joanne Nicholson
• So far weve covered
• Principles for assessment
• Issues in measurement
• Person-level assessment
• Program-level assessment

62
Integrating assessments in programs requires
consideration of

• Relevant domains, e.g., work, education, etc.
• Skills
• Resources and supports
• Individuals goals
• Programs goals
• Additional outcome tracking needs, e.g., data
  required for program funders?
• Organizational resources and supports

63
Providers Motivations for Using Evidence-Based
Practices

• Enhanced professional competence
• Perceived utility
• Fit with usual tasks and duties
• Fit with ones own experience
• Fit with mission of organization, structure of
  the intervention, organizing conceptual model

(Aarons Palinkas, 2007)
64
Potential Barriers to Use

• Lack of training, motivation, connection
• Measures, methods viewed as irrelevant
• Measures, methods viewed as burdensome
• Measures, methods not viewed as useful
• Lack of organizational support?resources
• Program measures and research measures dont
  mix well

65
Strategies for Overcoming Barriers to Assessment

• Educate re rationale
• Engage program staff participants in process
• Anticipate address logistical challenges
• Create opportunities for feedback refinement
• Use data in meaningful, productive ways

66
Learning Activity

• Break into groups ? Select a domain
• Discuss assessment issues
• Intervention planning? Outcomes?
• Existing measures? Adaptation? Refinement?
• Logistics Why? Who? How?
• Mechanisms for feedback to staff, participants?
• Identify implementation challenges suggest
  strategies for addressing
• Report back