IAPSRS Annual Conference - PowerPoint PPT Presentation

1 / 59
About This Presentation
Title:

IAPSRS Annual Conference

Description:

Ellen Tate, M.A., Centre for Addiction and Mental Health, Toronto, Ontario. Agenda ... Ellen Tate. Meghan Prosser. FUNDING. Ontario Mental Health Foundation (OMHF) ... – PowerPoint PPT presentation

Number of Views:42
Avg rating:3.0/5.0
Slides: 60
Provided by: camh61
Category:

less

Transcript and Presenter's Notes

Title: IAPSRS Annual Conference


1
IAPSRS Annual Conference
  • June 10, 2002
  • 130-500 p.m.
  • Toronto

2
CRITICAL CHARACTERISTICS OF COMMUNITY SUPPORT
PROGRAMS FOR PEOPLE WITH A SEVERE MENTAL ILLNESS
  • Progress on the Road to Measurement and
    Understanding

3
Presenters
  • Ross Norman, Ph.D. Clin.Psych., London Health
    Sciences Centre, University of Western Ontario,
    London, Ontario
  • Bonnie Kirsh, Ph.D. , University of Toronto,
    Toronto, Ontario
  • Ellen Tate, M.A., Centre for Addiction and Mental
    Health, Toronto, Ontario

4
Agenda
  • Introductions
  • Background and rationale
  • Identification of important domains and small
    group work
  • Break
  • Development of specific items and small group
    work
  • Where we go from here
  • Discussion and wrap-up

5
Learning Objectives
  • To understand the rationale for identifying
    critical program characteristics.
  • To understand the process used for identifying
    major domains of characteristics.
  • To examine agreement between results of a formal
    approach to identifying important domains and
    participants perceptions based on experience.
  • To identify and appraise methodological
    challenges in developing and refining items.
  • To understand potential applications of the
    instrument being developed.

6
Research Team Project Staff
  • Brian Rush
  • Ross Norman
  • Bonnie Kirsh
  • Cam Wild
  • Steve Lurie
  • Ellen Tate
  • Meghan Prosser

7
FUNDING
  • Ontario Mental Health Foundation (OMHF)
  • One of six projects under the larger provincial
    Community Mental Health Evaluation Initiative
    (CMHEI) concerning the evaluation of community
    mental health programs and supports
  • Five year funding (1999-2004)

8
COLLABORATIVE PROJECT
  • Research Committee of the Ontario Federation of
    Mental Health and Addictions Programs
  • Centre for Addiction and Mental Health
  • Other researchers
  • Advisory Committee

9
Advisory Committee Composition
  • Range of Stakeholders
  • Consumers, Family Members, Service Providers from
    Community
  • Researchers
  • Geographic diversity
  • Range of program backgrounds and experience

10
Advisory CommitteeRole
  • Advice including conceptual and methodological
    issues, user friendliness, etc.
  • Identification of limitations and potential
    problems.
  • Facilitation of pilot studies.
  • Review and consultation regarding each phase of
    the plan and its results.

11
Why Is This Important?
12
TRADITIONAL COMPARATIVE EVALUATION
Program Model B
Program Model A
Outcomes
Outcomes
13
TRADITIONAL PROGRAM EVALUATION
  • INTERPRETATIVE PROBLEMS
  • Adherence to models?
  • Overlap in content across models?
  • Differences in programs unrelated to models
  • Non-program content contributors to outcome?
  • Small sample sizes

14
Other Factors to Consider
  • Stage and nature of illness (acute, recovery,
    etc.)
  • Age of person
  • Length of experience with illness
  • Individual interests and abilities
  • Social support
  • Community context

15
GOALS OBJECTIVES
  • To develop an instrument package that will
    measure critical program characteristics
  • Longer-term goal is to help explain outcomes and
    offer practical, informed advice on how to
    improve services to achieve better outcomes

16
Multiple Perspectives
  • Consumers
  • Family
  • Service providers
  • Program managers

17
PROJECT OVERVIEW 5 PHASES
  • Phase I ? Determining Content Domains
  • Phase II ? Initial Development of Items
  • Phase III ? Pre-testing
  • Phase IV ? Pilot-testing (current)
  • Phase V ? Product Development

18
Whats Been Done
  • Phase I ? Determining Content Domains
  • qualitative analysis of literature review and
    interviews with consumers, family members, and
    service providers
  • Phase II ? Initial Development of Items
  • development of package of questionnaires
  • Phase III ? Pre-testing
  • 6 programs -

19
Qualitative Research
  • Assumes there is not one single objective reality
  • Attempts to understand phenomena in terms of the
    meanings people bring to them
  • Based on experiences, "moments and meanings" in
    people's lives

20
Phase I ? Determining Content Domains
  • Qualitative analysis of 42 semi-structured
    interviews with consumer/survivors, family
    members, and service providers
  • Purposive selection of interviewees to reflect
    the diverse experience of living in urban, rural
    or frontier areas, of different service models,
    and cultural affiliation in Ontario
  • Grounded theory used to analyze text and identify
    themes and domains

21
Grounded Theory
  • Theory is generated inductively, i.e., grounded
    in data
  • Data analysis constant comparative method
  • Unitizing
  • Categorizing
  • Forming themes

22
Content of Interviews
  • The type of community support program a person is
    involved with and what this provides for
    consumers
  • How consumers became involved in the program
  • How the program benefits consumers
  • Specific program components they feel are most
    helpful to consumers
  • How specified program components work in practice
  • How staff of the program are helpful to consumers
  • Other ways in which the program might be helpful

23
Overview of Literature Selection Review
  • Included multiple perspectives
  • Selection of key words
  • Limited by date country (Canada, United States,
    Britain, Australia)
  • Reviewed 7000 citations to select articles
    according to specific criteria
  • Categorized by type of article
  • Selected approximately 200 articles

24
Overview of Process
25
Overview of Process
26
Sample of Quotes Basic Needs
  • " Yeah and some people... the sense of
    achievement and sense of autonomy and the fact
    that they have their own apartment and they're
    able to manage it in terms of self esteem, it's
    just tremendous... in terms of for some people
    what that's like... to be able to do that as
    opposed to... living in a boarding home, and
    everything... you know, your meals are provided,
    and rules that you have to follow it's just a
    tremendous... effect on people to be able to
    achieve having their own apartment...."
    (Source service provider)

27
Sample of Quotes Productivity
  • Clinicians can contribute their insight to
    appropriate job matches that will support their
    clients illness management as well as vocational
    needs. Clinicians who know what strategies their
    clients use to manage their illnesses can assist
    in planning effective placements. For example,
    some clients with psychotic illnesses keep
    themselves isolated because they cannot be around
    people without feeling paranoid and persecuted.
    When this is true, clinicians can help vocational
    specialists to find isolated evening work for
    their clients. Other clients feel insecure when
    alone and need to be with others to feel safe
    (Source Torrey et al, 1998).

28
Sample of Quotes Personal and Social Supports
  • now my family comes to the self-help program
    space andone of my sisters had a placement
    here, and she has an understanding about it but
    its been . like ten years and nobody
    understoodId given pamphlets, um, tried to
    talk to my family of origin about things I
    collected from the mental health association
    and things that are laying around the hospital
    and its like they dont even want to read
    it.and now, after all this time, my family
    have become members of self-help group and they
    dont see it asthey call me normal nowthey see
    that there are levels of function that people
    have and they see that Im not ill all the
    time (Source consumer)

29
Sample of QuotesService Climate
  • Therapeutic goals are highly individualized and
    flexible. Because goals are set and specified
    collaboratively by the patient, family and
    clinical team, treatment objectives can vary
    widely from case to case. Similarly, because
    plans for achieving identified goals are based on
    the strengths and weaknesses of each patient and
    family, such plans vary from case to case.
    Flexibility in goal setting and intervention
    efforts is needed to respond to changing
    patient/family circumstances and outcomes
    (Source Santos et al, 1995,p.1119)

30
Sample of Quotes Services Provided
  • When my son was in hospital, um his counselor
    visited him, about two or three times a week, and
    had a pretty good idea of how he was getting on -
    if he was becoming better or not - and at one
    point they were going to release him and um, the
    counsellor let them know that he really wasnt
    well, quite well enough. So I think that they
    helped him a lot (Source family member)

31
Sample of Quotes Organizational Functioning
  • And we minute resolutions to problems so that
    if things occur in future we can refer back to
    them and say remember this is what we agreed so
    lets everyone stick to this unless we want to
    reopen the issue, and discuss it again. (Source
    service provider)

32
Sample of QuotesSystems Issues
  • In addition to sabotaging treatment
    effectiveness, Drake et al. (1996) state that the
    lack of cooperation between services forces
    clients with co- occurring addictive and mental
    disorders to assume responsibility for
    coordinating care in the parallel model, the
    burden of integrating different philosophies,
    recovery models, and interventions fell on the
    patient in other words, it rarely occurred
    (Drake et al., 1996, p. 45). As a result,
    patients are usually excluded from one or both
    types of programs, and are rarely engaged in
    concurrent mental health and substance abuse
    treatments (Drake et al., 1996 Inderlin
    Belisle, 1991)" p.74. (Cited from El mallakh
    (1998)).

33
Phase II ? Initial Development of Items
  • To synthesize information to question format...

34
Brainstorming for Items
35
Sample of Quotes from Interviews Housing
  • " They'll branch you out anywhere you want to go,
    anything you need, uh... all you got to do is
    ask, and like youve finally got housing, got a
    place, they could make you call a place. They'll
    get you furniture, they'll get you moved, you
    know they'll get you bedding. Here, they have
    bedding, dishes, things like that..... it's
    absolutely incredible, that resources, are just
    like, for the asking really. You know that old
    saying, ask and you shall receive, it happens....
    -laughter-, it makes a really big difference. "
    (Source Consumer)

36
From data to question
Basic Needs, Housing, Suitability, Living
conditions gt
Item on Consumer Questionnaire My
worker/program helped me make my house feel like
a home (e.g., getting furniture, decorating)
37
Phase II - (continued)
  • Development of separate instruments for each
    group
  • Review of existing instruments in the field
  • Organized around domains and sub-domains

38
Phase III ? Pre-testing
  • Tested draft questionnaires with six programs
  • Revisions based on feedback

39
Who was Involved in Pre-test?
  • Variety of programs
  • model type (ACT, Case Management, Housing
    Support, Employment Support, Self-Help,
    Clubhouse)
  • urban/rural
  • N39 (2 consumers, 2 family members, 2 service
    providers and 1 program administrator from each
    of 6 programs)

40
Feedback from Consumers
  • Liked opportunity to reflect on service
    experience and personal needs
  • Liked opportunity to speak well of program
  • Length not a problem
  • Some fatigue factor but overall seen as
    worthwhile

41
Family Members
  • Desire to help others through sharing experience
  • Didnt enjoy as much as consumers
  • More difficulty responding because of less
    familiarity

42
Service Providers/Managers
  • Too long for us
  • But it is all potentially important
  • Gee, we sure do a lot
  • Do we do this? Should we do this?
  • Some statistical information not available

43
Feedback by Program Type
  • Issue of how the questions fit with different
    program contexts
  • We always refer for that (e.g .housing)
  • Some domains dont work for all
  • Volunteering
  • We are a non-medical model
  • Extent of activity (what is the threshold)

44
Some of the issues which keep arising
  • What is the unit to be described i.e. the
    program?
  • How much can people remember assimilate in
    responding?
  • Should we focus on what the program provides or
    the individual receives?
  • How and when to aggregate?
  • How much agreement between perspectives?
  • Where to go for lunch?

45
Further Issues to be Addressed in Developing
Instruments
  • Reduce length
  • Establish response distributions, reliability and
    validity for individuals
  • Assess degree of agreement on program
    characteristics across sources
  • Establish reliability and validity for use at
    program (aggregate) level

46
Objectives of Phase IVPilot Testing Stage One
  • Elimination of redundant items or items unlikely
    to be useful and thereby reduce scale length 
  • Examination of extent of agreement of information
    provided from different perspectives
  • Preliminary investigation of the extent to which
    program characteristics may be linked to outcomes
    for individual consumers.

47
Phase IV - Pilot Testing Stage One
  • Testing of package of instruments with 6
    programs
  • Cross-sectional design
  • Includes preliminary linking of results to
    outcome data
  • hospitalization
  • satisfaction with services

48
Sample
  • 6 programs
  • Stratified random selection of 25 consumers per
    program (n150)
  • gender
  • length of time in program
  • diagnosis
  • All clinical staff (est. 15/program) program
    manager per program (n96)

49
Will Select Programs which
  • Offer long-term individualized support to people
    with a severe mental illness
  • Are sufficiently large that gaining a sample of
    25 consumers is realistic
  • Offer a range of services (i.e., not restricted
    only to housing or employment support)

50
Selection of Programs (contd)
  • Are using the PSR Toolkit or a similar outcome
    tracking system
  • Represent different parts of the province
  • Are able to provide a lead hand to assist

51
Objectives of Phase IVPilot Testing Stage One
  • Elimination of redundant items or items unlikely
    to be useful and thereby reduce scale length 
  • Examination of extent of agreement of information
    provided from different perspectives
  • Preliminary investigation of the extent to which
    program characteristics may be linked to outcomes
    for individual consumers.

52
Overview of AnalysisItem reduction
  • Will examine
  • location and percent of missing data
  • response distributions
  • variation between programs
  • inter-item correlations
  • feedback regarding clarity, feasibility, etc.

53
Overview of AnalysisAgreement in perspectives
  • Will examine
  • similarity of metric structure between consumer
    and service provider responses
  • agreement in responses appropriate indices

54
Overview of AnalysisTentative Links to Outcomes
  • Testing of association between program level and
    aggregated outcome is not feasible
  • Correlation and multiple regression to examine
    links to common outcome indices (hospitalization
    and satisfaction with service)
  • Exploratory analysis for unique outcome data
    within each program.

55
Summary of Overall Data
  • In which domains did consumers want/need
    support?
  • How much help do consumers report receiving?
  • How much variation is there between programs in
    responses to the instrument?

56
Next Stages of Analysis
  • Test-retest
  • Sub-study on validity (e.g., use of service log)
  • Larger samples of programs and subjects
  • aggregation issues
  • Sub-study on family perspective?

57
Phase V ? Product Development Dissemination
  • Research Products
  • Instrument Package
  • Manual
  • Final Reports Presentations
  • Program/System Checklist to
  • raise questions
  • identify needs for enhancement
  • suggest strategies
  • Peer evaluation instrument
  • Possibly a Provincial Tool for Program Description

58
Template for Program/System Assessment

Examples Housing Physical Needs Employment Educati
on
59
Project Information
  • www.ontario.cmha.ca (CMHEI link)
  • Ellen Tate, Project Coordinator
  • Centre for Addiction and Mental Health
  • 33 Russell St.
  • Toronto, ON M5S 2S1
  • Tel 416-535-8501 ext. 6575
  • email ellen_tate_at_camh.net
Write a Comment
User Comments (0)
About PowerShow.com