Outcome Measurement What have we learned about reallife approaches

1
Outcome Measurement What have we learned
about real-life approaches

• Presented by Virginia Wright (Bloorview Research
  Institute, Toronto) at the OACRS Conference,
  Waterloo, ON, October 23, 2007
• Co-investigators The OACRS Steering Committee
  (Vicky Earle, Gill Davis, Jenny Greensmith,
  Dorothy Harvey, Karen Horgan, Laurie Lessard,
  Donna Litwin-Mackey, Helene Mercier) and Virginia
  Wright.
•  

2
The Ontario Association of Childrens
Rehabilitation Services (OACRS) Measuring Change
Study

• Funded by Government of Ontario Ministry of
  Children and Youth Services (MCYS) October 2006
  to March 2007
• The opinions expressed are the consultants' and
  should not be considered reflective of those of
  the MCYS.
• Virginia Wright is supported by a Career
  Development Award (2007 to 2011) through the
  Canadian Child Health Clinician Scientist
  Program, a Canadian Institutes of Health Research
  Strategic Training Initiative.

3
We thank our Research Assistants (centres listed
in alphabetical order)

• Susan Cohen and Joan Walker (Bloorview Kids
  Rehab)
• Janet Coppold (Child Development Centre, Hotel
  Dieu Hospital)
• Joanne Leclair Peterson (Childrens Treatment
  Centre)
• Theresa Hudson (Erinoak Kids)
• Angela Harrison (Five Counties Childrens Centre)
  
• Helene Mercier (George Jeffrey Childrens Centre)
• Karen Koseck (Grandview Childrens Treatment
  Centre)
• Julie Bially (John McGivney Childrens Centre)
• Bridget OBrien (KidsAbility Centre for Child
  Development)

4
We thank our research assistants

• Heidi Nelson (Lansdowne Childrens Centre)
• Linda Wallman (Niagara Peninsula Childrens
  Centre)
• Andree Cornish (One Kids Place/La place des
  enfants)
• Bonnie Lowry Bagshaw (Ottawa Childrens Treatment
  Centre)
• Cathy Dufort Gibbs (Pathways Health Centre for
  Children)
• Janice Dekker (Prism Centre for Audiology and
  Childrens Rehabilitation), and
• Lisa Peacock (Childrens Treatment Network of
  Simcoe York).

5
Background

• One of five primary initiatives in the 2005 OACRS
  Key Directions Report focused on accountability
  in evaluation of child- and family-based outcomes
  that link with a best-practice approach to
  service and with family satisfaction
• Report refers to assisting Ontario MCYS in
  working towards more streamlined, focused
  requirements for data collection, processing and
  reporting, that are relevant to the Ministry and
  to the CTCs, and that meet MCYS needs for
  analysis, research and accountability.

6
Overall goal of our clinical outcomes project

• To propose a set of up to five common outcome
  indicators for the OACRS network based on our
  studys results and also in consideration of the
  recommendations from the CanChild Outcome
  Measures Project Report (Parts II and III)

7
Summary of Previous Related Work

• Measuring outcomes for children with special
• needs and their families Parts 2 and 3.
• CanChild Reports for the MCYS, March 2005
  and 2007.
• One key component of this work was to develop
  the foundation for a system wide set of outcome
  measures What impacts do our services have?
• Our work is intended to dovetail with the
  CanChild work, providing complementary and
  enhancing information!

8
We wanted to find out

• What clinicians are doing as far as outcome
  measurement. What works for them and where are
  the gaps?
• Not what they say they use (we know some of
  this from previous OACRS project), but what they
  actually do!
• What systems and supports need to be in place to
  enhance outcome measurement at the level of the
  individual child and family?
• What systems and supports do we need to put in
  place to make system-wide outcome measurement
  happen?

9
Measuring Change Project Objectives

• 1. Produce an inventory of current outcome
  measurement practice across CTCs based on
  clinical practice, based on what is being done.
• Summarize key goal domains that clinicians
  identified as the focus of their intervention
  work
• Identify clinicians methods of measuring change
• Summarize the value of the identified measures in
  the evaluation of change

10
Measuring Change Project Objectives

• 2. Identify gaps in measurement according to the
  ICF framework and goal areas identified
• 3. Determine the link between these findings and
  those of the CanChild Part 2 report
  complementary, confirming, different?
• 4. Make recommendations re plan for consistent
  use of outcome measures across OACRS CTCs and for
  the system requirements that will support this
  work

11
Methods

• Ethnographic study.
• It used reflective interviews with clinicians as
  a means of learning about measurement practice
  with current out-patients.
• Health records were selected at random within
  designated program areas so that the study
  results were representative of overall outcome
  practice.
• Each clinician reflected on a selected
  out-patient clients health record as the basis
  for their responses to the interview questions
  about measurement.

12
Methods

• The study did not involve direct contact with
  clients, or the sharing of clients actual data.
• The identity of clients and clinicians was not
  known by anyone other than the local centres RA.
  
• The only client data that were collected were
  age, gender, and primary diagnosis. Actual scores
  from clients assessments were not discussed

13
We needed clinicians to be candid in their
responses

• Clinicians were assured that the information that
  they provided would remain confidential. The
  interviewing RAs each signed a confidentiality
  agreement for the interview process.
• Data entry staff and investigators were not able
  to identify either the client discussed or the
  clinician interviewed
• We were careful to talk about ways of measuring
  change rather than talking about outcome measures
  used wanted to give permission to talk about
  any methods used!

14
Used an ICF Framework

• Recognition that family-centred pediatric
  rehabilitation practice must be grounded in the
  framework of the International Classification of
  Function and Disability (ICF)
• Interventions and outcome indicators need to take
  all components of this framework into account

15
ICF Focus for Measurement

• The ICF framework also embraces the notion that
  interventions can be directed toward making
  changes in the child and/or to changing the
  people, systems, and environments that are
  connected with the child.

16
Feedback from OACRS administrators after the
study

• We used member checking to be confirm the extent
  to which the results reflected practice. Each
  administrator reviewed a summary of the main
  results from their centre as well as summary of
  overall results
• ? or ? or X

17
Results
18
We achieved representation of programs/services

• Sixteen of 22 OACRS facilities participated
• Each facilitys administrative contact helped us
  determine which program/services to survey
• Representation from all areas in Ontario (from
  Windsor to Sudbury to Ottawa).
• The work represented various clinician groups,
  diagnostic groups, and program/service areas
• Between 10 and 30 charts in total were included
  from each centre. (mean 26 interviews/centre,
  varied from 6 to 45)
• Obtained our required sample size in just 4
  months!!!!! Enthusiasm was high!

19
Clinicians Interviewed (n 408)
Other music therapists, behaviour therapists,
infant development specialists, audiologists,
communication disorder assistants, resource
teachers, family counsellors, service
co-ordinators, kinesiologists, psychologists,
audiologists, and recreation therapist all
self-identified according to role with client
20

• 73 additional clinicians (i.e., 13 PTs, 20 OTs,
  12 SLP, 13 social workers, and 18 others) were
  approached for interview and declined
• gt 60 of the time, declined because the
  clinician was not working with the targeted
  client for intervention (i.e., one time
  assessment or monitoring), client was new and had
  not been fully assessed by the clinician yet, or
  there was minimal contact within the last year.

21
Type of client (n 408 interviews)
22
Type of client (n 408 interviews)
Direct intervention was defined as being seen at
least once per month. Consultation was a child
who had a full assessment but was then seen less
than once monthly.
23
Age of client (n 408 interviews)
24
Location of Service Provision (n 408
interviews)
25
Primary diagnosis of client (n 408 interviews)
26
Question 2

• What were the main goals for your work with this
  client based on the assessment findings?

27
Examples of goal areas

• Body structure/functions
• Develop lateral tongue movement
• Increase hand strength (to do buttons and snaps
  independently)
• Activity
• Improve ability to ascend stairs Increased
  ability to ride bike with training wheels
• Play functionally and appropriately with toys for
  a specified period of time

28
Examples of goal areas

• Participation
• Participation in girls group -e.g. increased
  self-esteem, leadership skills, increased social
  skills and network, increased life skills
• Be part of band class at school (on clarinet)
• Education
• Increase familys understanding of a sensory diet
• Teach parents how to work on above speech sounds
  at home

29
Examples of goal areas

• Equipment/environment
• Provide a seating and mobility system that is
  safe
• Adaptations in classroom to make it accessible
• Process
• Compile a financial request form
• Develop a behaviour management strategy

30
ICF Goal Areas (n 408 interviews)
Note no specific goals relating to quality of
life
31
Goal Categorization
according to Type of Service provided ( of time
used)
32
Goal Categorization
according to Type of Service provided ( of time
used)
33
Goal Categorization
according to Type of Service provided ( of time
used)
34
Goal category by clinical profession
35
Question 4

• What tools and approaches will you use to
  measure change?
• OR
• If you have recently evaluated change, tell me
  what you used.
• (Describe for each goal listed in question 2).

36
Examples of tool and approaches used (Q4)

• Behaviour observation (check box form)
• Age-appropriate language tests (SPAT-D)
• Vowels re-administer test/observe
• Discussion with client
• COPM
• Trial of 2 access methods
• Device tracking with software
• Dialogue/talking
• GMFM, GAS, 6 minute walk

37
Measurement approaches (n 1,610 listings)
38
What did clinical observation consist of?
(listed 37 of time)

• Clients ability to get down to floor and stand
  up
• Client sitting in desk in new seating set-up
  -observe arm and leg position
• Looked at ability to negotiate various terrain
  within daycare
• Number of stairs he could climb and how much
  assistance he required to accomplish
• During structured play activities-watch how
  marker was held during multiple opportunities
  graded assistance in order to achieve success)
• Observe client's printing sample and observe
  classroom notebooks

39
Use of parent/client report
(listed 27 of time)

• In the case of parent, client, teacher or other
  clinician report/interview, accomplishment scores
  were linked directly to the rating of the stated
  goals, and were typically done by a verbal report
  (e.g., yes, no accomplished, partially
  accomplished) in response to targeted questions
  from the clinician about the childs abilities.
• Within the report/interview category, the parents
  were the respondent group that was most
  frequently cited as sources for information. It
  is likely that the report occurred within the
  context of an intervention session.

40
Most commonly-used standardized clinical
measures (listed 26 of time)
41
Most commonly-used measures
42
Most commonly-used measures
43

• This list almost entirely reflects the measures
  that Hanna et al.(2007) identified in their
  survey of OACRS PTs, OTs and SLPs when they were
  asked to list the measures they used most.
• Hanna SE, Russell DJ, Bartlett DJ, Kertoy ML,
  Rosenbaum PL, Wynn K. Measurement practices in
  pediatric rehabilitation. Physical Occupational
  Therapy in Pediatrics. 20072726-42.

44
Q5 Ratings of usefulness of measurement
approaches

• Score each with a number of 1 through 5 on the
  five point response scale from not at all
  useful to very useful.
• Rated for
• Identification of issues _____
• Setting goals _____
• Evaluation of change _____

45
Examples of ratings for Q 5

• Goal Attainment Scaling
• Identification of issues somewhat useful (3)
• Setting goals very useful (5)
• Evaluation of change very useful (5)
• Check box observation
• Identification of issues very useful (5)
• Setting goals very useful (5)
• Evaluation of change somewhat useful (3)

46
Usefulness of measurement approaches overall (/
5)
There were differences between approaches for
their usefulness in evaluation of change when
compared for the 15 diagnostic groupings (lower
ratings for ABI, developmental delay, learning,
and seizure disorder).
47
Clinical observation usefulness for evaluation
of change
48
Limitations of clinical observation
(when not in context of a larger measure)

• No standard means of documentation or reporting
  was used
• Clinicians acknowledged that results cannot be
  rolled up to look at changes across children for
  a particular intervention or program not useful
  for a systems-level evaluation
• Unknown reliability as far as scoring of outcomes

49
Report/interview usefulness for evaluation of
change (/5)
50
Standardized clinical measures usefulness for
evaluation of change (/5)
51
Ratings of commonly-used standardized clinical
measures (/5)
52
Ratings of commonly-used standardized clinical
measures (/5)
53
Ratings of commonly-used standardized clinical
measures (/5)
54
Question 6

• Are there any challenges you think you might
  encounter in measuring change in this child
  (e.g., lack of good measures in the goal area, no
  time, not sure how to go about it, did not do
  measures at baseline)?

55
Measurement challenges
56
Barriers to measurement

• Time was not identified as the key barrier to
  measurement.
• Challenges related to child-based factors and
  parent-related factors predominated when it came
  to discussing standardized measures.
• Clinicians highlighted the lack fit of a
  particular tool with a childs or familys
  characteristics and issues.
• The perceived inadequacies of many of the
  measures from the perspective of content or
  availability or complexity also were noted

57
Clinicians made do with the tools that they had
at hand

• NOTE In many of the areas of pediatric
  rehabilitation (e.g., speech language, social
  work and psychology), there are few if any
  measures that have been designed for the purpose
  of evaluating change.

58
Measurement barriers

• Refer as well to work by Hanna et al. (2007) for
  more details on measurement barriers and to
  Swinton (2007) for details on adaptations
  commonly made to measures and measurement
  processes validity issues!

59
Question 7

• Are there other approaches to measuring change
  that you use with other clients?

60
Question 7

• The primary measurement approaches that were
  described for the clients in question were
  representative of the approaches taken for other
  clients on a clinicians caseload
• Only 9 additional standardized measures were
  added across the group when clinicians considered
  what they used for other clients.  

61
What did we learn?
62
Goal setting is a key part of practice within
OACRS centres

• Clinicians felt that they generally addressed
  measurement at the level of the individual well
  because of this goal-based focus
• Goals and associated measures focused on change
  in relation to impairment and activity goals
  rather than on participation
• The focus on body functions/structure and
  activity was expected since these tend to be
  issues that clinicians and parents can easily
  identify and work at
• Both of these ICF areas lend themselves well to
  evaluation by observation

63
Use of a diversity of measurement approaches

• Measurement of change usually involved a
  combination of use of clinical observation of
  targeted behaviours/skills, parent/client report
  and use of a standardized measure(s)
• Selection of measures from a wide array of tools
  allowed an individualized approach that dealt
  with the breadth and diversity of goals
  identified
• Also reflected the approaches/measures of
  greatest comfort/familiarity/availability for a
  particular clinician.

64
Clinical observation was the most frequent
approach to measurement

• Across all CTCs, clinical observation of
  specified skills/ behaviours allowed focused
  evaluation of goal areas
• Interestingly, use of standardized goal
  evaluation approaches such as the COPM and GAS
  was sporadic despite indication from several CTCs
  that they had received training in these
  approaches.

65
Use of client/parent report was the next most
popular method of evaluation of change

• Tied into the goals of intervention
• Used a variety of informal methods of recording
  change
• Parents were the respondent group that was most
  frequently cited as sources for information. It
  is likely that the report occurred within the
  context of an intervention session.

66
Standardized clinical measures were used as one
part of the assessment approach

• A wide variety of measures were seen as very
  useful to establish a baseline and set goals.
• Many of these tests are discriminative in nature
  (e.g., Bruininks Oseretsky Test of Motor
  Proficiency or the SPAT-D) and were not intended
  for this since they usually lack sensitivity to
  change.
• Clinicians were aware of this limitation, given
  the mean scores of less than 4/5 for evaluation
  of change for these discriminative tools
• Overall, standardized clinical measures did not
  rate as highly as clinical observation for
  measurement of change!

67
Comments

• It is very interesting to note that clinicians
  really value parent/caregiver report and clinical
  observations as sensitive measures of change.
  Standardized measures need to be interpreted
  cautiously when the family has noted changes but
  in fact the results (standard scores and
  percentiles) may actually have worsened. The
  child may have made many gains but may not have
  closed the gap (a child who makes 6 months change
  in 12 months time according to standardized
  measures).

68
What measures were missing that we expected to
see?

• A whole category of measures that are parent or
  child-report and pertain to functional
  abilities!!!!
• Functional measures such as PEDI, WeeFIM
  Instrument, ASK
• Why were they missing? Clinicians told us that
  they knew about them and had access to them but
  found that they did not efficiently address the
  clients goal areas too much detail!

69
Idea of diagnosis-free measures

• There was nothing to indicate that people are
  typically measuring with diagnosis-specific
  tools.
• This freedom will be a good thing given the
  diversity of clinical populations within and
  across the OACRS centres

70
Measurement gaps

• There were no common measures reported at the
  higher level of participation, family-functioning,
  or quality of life (child or family) among any
  of the clinician groups.

71
'Participation was rarely articulated as a goal
or measured directly

• Changes in participation may have been implied in
  goals set but not specified
• Changes in participation often seen as
  consequences of changes in activity and
  impairment rather than changes that are under
  the direct control of the treating clinician.
• Until very recently, there were no measures
  available to address participation in depth.

72
Participation

• In terms of the ICF classification definitions,
  we certainly see that goal areas under Impairment
  are easier to isolate and measure however beyond
  pre-school years our goal areas are more linked
  to Participation and Education which are more
  challenging to measure. (OACRS administrator
  comment)

73
Measurement gaps

• Measurement at the systems level is a challenge
  still to be conquered!
• We do not have effective tools for considering
  evaluation from a systems level. We are working
  with a consultant in the development of a matrix
  in the Multidisciplinary Program based upon a
  continuum of cognitive and physical limitations.
  We are currently defining the assessments
  measures to be used in each of the nine cells in
  the matrix. (OACRS administrator comment)

74
Comments

• I believe that an OACRS wide set of outcome
  indicators should be done in collaboration with
  both OACRS Standards and Benchmarks and Best
  Practice Committees as well as with collaboration
  with CanChild who also completes research along
  similar lines.  (OACRS administrator comment)

75
Overall goal of our clinical outcomes project

• To propose a set of up to five common outcome
  indicators for the OACRS network based on our
  studys results and also in consideration of the
  recommendations from the CanChild Outcome
  Measures Project Report (Parts II and III)

76
Measurement approaches used
77
Six main recommendations to MCYS

• 1. Use of a validated individualized goal-based
  approach such as Goal Attainment Scaling (GAS) or
  the Canadian Occupational Performance (COPM) for
  each clinician/client dyad.
• One aim of an individualized goal approach is to
  integrate impairment, activity,
  participation, education, and environment
  goals within the same goal evaluation template.
  This encourages a conscious linkage between the
  various levels of the ICF.
• Fits well with clinicians use of clinical
  observation and parent/child/teacher report as
  primary approaches to measuring change

78
Recommendation 2

• 2. Require that a minimum of one standardized,
  validated outcome measure from the list of
  measures in our OACRS toolbox will be used by
  each clinician working with a client/family. This
  (these) would be applied at the beginning and end
  of the designated intervention period.
• These standardized evaluations clearly are
  integral to the determination of
  clients/families strengths and challenges and
  for setting goals.

79
Recommendation 2

• Our suggestion is restrict this list to measures
  at the ICF levels of activity or
  participation to encourage clinicians to think
  beyond the level of impairment.
• In order to be granted membership in the
  validated measure toolbox, a measure had to
  have published evidence of psychometric strength
  in at least one pediatric rehabilitation
  population, demonstrated utility with respect to
  clinical practice, and show strong potential as a
  measure of outcome.
• We anticipate that by providing a toolbox of
  measures, the efficiency of outcome measurement
  will increase through the use of a reduced set of
  best measures.

80
Recommendation 3

• 3. Inclusion of a measure(s) from an emerging
  measures toolbox in situations when outcome
  areas are not sufficiently addressed by the
  recommended validated tools, or when one of these
  emerging measures is a strong fit with the
  service/program/intervention that is being
  provided.
• This includes outcome measures that are
  undergoing formal instrument development and/or
  validation within the CTC network facilities
  e.g., the FOCUS preschool speech scale (Nancy
  Thomas-Stonell et al.), The Family Impact of
  Assistive Technology Scale (FIATS) (Steve Ryan et
  al.).

81
Recommendation 4

• 4. Incorporation of a generic participation
  measure as proposed in the CanChild 2007 Outcome
  Measures Report as a way of capturing
  higher-level outcomes that having meaning in a
  childs life role.
• Coverage of participation could be addressed
  by use of the Life-Habits (LIFE-H) questionnaire
  (child and preschool versions)

82
Recommendation 4

• LIFE-H is a Canadian measure, thus from a content
  perspective should fit well with our childrens
• LIFE-H is diagnosis free can be used within
  any of our clinical populations
• Versions of both measures now available to
  address young children as well as school-aged
  children/teens
• There is evidence of reliability and validity of
  the LIFE-H with work underway on ability to
  detect change (AACPDM 2007)

83
Recommendation 5

• 5 Use the MPOC-20 as a means to evaluate the
  extent to which services are family-centred (and
  by extension, measure parents satisfaction with
  services provided).
• MPOC-20 (20 items) is a Canadian measure and
  from a content perspective should be consistent
  with care expectations within the Canadian health
  system
• Allows quantitative diagnosis-free measurement
  of essential aspects of service that relate to
  outcome and family well-being
• Benchmarks for target scores are available from
  previous CanChild/OACRS work  

84
Recommendation 6

• 6. There must be co-ordinated design,
  development, staff education, roll-out and
  implementation of systems across the OACRS CTCs
  to support the proposed outcome indicators
  initiative, as well as a clearly-defined
  strategic plan for roll-out and implementation of
  indicators.
• The identified challenges to measurement (i.e.,
  factors related to the child, clinician, tool,
  parent and environment) will help to guide the
  recommendations for design and implementation of
  the OACRS-wide system.

85

• The oneyear Preschool School Speech and Language
  Outcome Measurement Pilot Project that MCYS is
  currently conducting within 6 Ontario preschool
  speech language programs could be used by the
  MCYS as a prototype for the outcome indicators
  pilot work.

86
Recommendation 6

• Staff education needs to be put in place across
  CTCs for any new outcome system that is
  recommended given the current lack of a common
  approach to measurement
• Education should also include a component on the
  key features of outcome measurement and how
  outcome and process goals differ
• Knowledge broker concept?
• On-line training programs for key measures
  available through a provincial web-site -
  materials will need to be developed/acquired for
  this and the website would need to be developed.

87
Recommendation 6

• Resources and a clearly identified implementation
  process must be added to the system to permit the
  within-centre and network capture of outcome
  results for the selected measures
• At present, most of the outcomes information
  still resides in the client chart and is
  inaccessible even at the program evaluation level
  within a CTC.

88
Recommendation 6

• Changes likely needed for administration
  protocols, e.g., a shift from paper forms to a
  computer administered approach for a measure
  such as the PEDI, MPOC-20 or LIFE-H (software
  versions of these tools are being created at
  present) or use of new language translations.
• Once the measures are put into place, the idea of
  what counts as a meaningful difference is
  critical to consider especially when we are using
  the outcome results to drive policy and programs.

89
Acknowledgements

• We greatly appreciate
• The Ontario Ministry of Children and Youth
  Services (MCYS) for funding the project
  (contacts Jeff Wright, Shannon Fenton, Jane
  Cleve, Lisa Schultz)
• The extensive group of clinicians across the 16
  CTCs who participated in the interviews
• Susan Cohen for her work as research co-ordinator
• Christina Hay and Angela McDonald for their
  assistance with data entry and analysis
• Carolyn Hicks and Wendy Moyle for their strong
  support in the OACRS office..
• The Bloorview Research Institute for supporting
  Virginia Wrights time on the project

90
References

• Hanna SE, Russell DJ, Bartlett DJ, Kertoy ML,
  Rosenbaum PL, Wynn K. Measurement practices in
  pediatric rehabilitation. Physical Occupational
  Therapy in Pediatrics. 20072726-42.
• CanChild Centre for Childhood Disability
  Research. Measuring outcomes for children with
  special needs and their families Part 1. 2003.
  Report to the Ontario Ministry of Child and Youth
  Services.
•  CanChild Centre for Childhood Disability
  Research. Measuring outcomes for children with
  special needs and their families Part 2. 2005.
  Report for the Ministry of Children and Youth
  Services.
• Rosenbaum P, Jaffer, S, Russell D. Measuring
  outcomes for children and youth with special
  needs and their families part 3. 2007.
  CanChild Centre for Childhood Disability and
  Research. Report for the Ontario Ministry of
  Children and Youth Services.
• Swinton, M. Understanding measurement issues in
  pediatric rehabilitation. In press. 2007. MSc
  Thesis, McMaster University  

91
Ideas?
Virginia Wright vwright_at_bloorview.ca