Validation of Intermediate Measures VIM : RAND Panel, Study Organization and Design

1
Validation of Intermediate Measures (VIM) RAND
Panel, Study Organization and Design

• Nina R. Schooler, Ph.D.
• Department of Psychiatry Behavioral Sciences
• SUNY Downstate Medical Center
• Department of Veterans Affairs, VISN 6 Mental
  Illness, Research, Education and Clinical Center
  (MIRECC)

2
Steps in the VIM Study
3
Step 1Identification of Evaluation Criteria
for Measures

• Test-retest reliability
• Correlation with measures of real-life
  functioning
• Correlation with cognitive performance
• Practicality - the testers perspective
• Tolerability the test-takers perspective
• Utility as a repeated measure
• Sensitivity to change
• Applicability to international clinical trials

4
Step 2Solicit Nominations of Intermediate
Measures

• Definition of intermediate measures
• Suitable for use in short-term studies
• Candidate for cognition co-primary
• Measure functional capacity or assess cognition
• Identification
• Wide solicitation from field
• Review of literature using Pub-Med and other
  sources
• Distinction
• Performance
• Interview

5
Step 3 Selection of Candidate MeasuresFunctional
Capacity performance based

• Everyday Functioning Battery
• Micro-Module Learning Test
• Maryland Assessment of Social Competence
• Social Skills Performance Assessment
• Test of Adaptive Behavior in Schizophrenia
• UC-SD Performance Based Skills Assessment
• UC-SD Performance Based Skills Assessment Brief
• Everyday Problems Test
• Independent Living Skills

6
Step 3Selection of Candidate Measures Interview
Based Measures of Cognition

• Clinical Global Impression of Cognition in
  Schizophrenia (CGI-CogS
• Schizophrenia Cognition Rating Scale (SCoRS)
• Cognitive Assessment Interview (CAI)
• Cognitive Functioning Scale (CogFu)
• Global Assessment of Cognition Functioning (GACF)
• CGI for Cognitive Impairment clinician version

7
Step 4Creation of Data Base for RAND Panel
Evaluation

• The available literature for each measure is
  reviewed
• In addition to published literature, authors and
  users are contacted for unpublished data
• A data matrix of evidence for each criterion
  identified in Step 1 and each measure
  identified in Steps 2 and 3 is generated
• e.g. test re-test reliability for each measure
• Problem
• Inadequate data to fill the matrix for interview
  measures

8
Step 5 VIM RAND Panel

• Panel Expertise
• Cognition in schizophrenia
• Cultural adaptation
• Clinical trials
• Statistics / psychometrics
• Community adaptation
• Psychiatric rehabilitation

9
VIM Rand Panel Members Feb 2008

• Deanna Barch Washington U.
• John Brekke - USC
• Judith Cook U IL Chicago
• Patrick Corrigan IL Inst. Tech
• Michael Egan - Merck
• Helena Kraemer Stanford U.
• William Lawson Howard U.
• Andy Leon Cornell U
• Steve Romano - Pfizer
• Larry Seidman Harvard U
• Sophia Vinogradov UC SF

10
Evaluation Criteria for RAND PANEL

• Test-retest reliability
• Correlation with measures of real-life
  functioning
• Correlation with cognitive performance
• Practicality - the testers perspective
• Tolerability the test-takers perspective
• Utility as a repeated measure
• Sensitivity to change
• Applicability to international clinical trials

11
RAND Panel Evaluation CriteriaRating Scale

• 1 Poor
• 2
• 3 Fair
• 4
• 5
• 6 Good
• 7
• 8
• 9 Super b

12
RAND PanelPerformance based Measures Results
13
RAND PanelInterview-based MeasuresChallenges
for Review

• Concept of interview based assessment of
  cognition is new
• First publication in the area is Bilder et al
  (2003)
• Data regarding the formal evaluation criteria are
  limited
• Instead of formal ratings, panelists held a
  focused review and discussion evaluating the
  criteria
• No formal ranks were given
• Addressed questions about interviewing for
  cognition
• Need for informants
• Global judgment vs. domain-based assessment

14
RAND PanelInterview-based MeasuresRecommendation
s

• Cognitive Assessment Interview CAI - domain
  based
• Empirically derived from CGI-CogS and SCoRS
• Patient as only source of information
• Global Assessment of Cognitive Function - 100 pt
  scale
• Novel , experimental measure based on interview
• Attractive format related to GAF
• CGI for cognition 7-point clinician rated
  measure
• Can clinicians assess cognition

15
Step 6Selection of Measures for VIM
studyPerformance based Measures

• Test of Adaptive Behavior in Schizophrenia (TABS)
• UCSD Performance-based Skills Assessment (UPSA)
• Independent Living Scales (ILS)
• Shorter versions assessed
• UPSA Brief
• Finances
• Communication
• TABS Brief
• Medication management
• Work Productivity
• ILS factors extracted

16
Step 6Selection of Measures for VIM
studyInterview Measures

• Cognitive Assessment Interview (CAI )
• Clinical Global Impression (1-7 pt scale)
• CGI for Cognitive Impairment
• rated by psychopathology assessor

17
Step 7 VIM Study Specific Aims

• Examine psychometric properties of the measures
• Reliability - test-retest and inter rater
• Repeatability - utility as a repeated measure
  within the time frame of a clinical trial
• absence of practice effects that yield ceiling
  effects)
• Examine validity of the measures
• Correlation with measures of real-life
  functioning
• Correlation with cognitive performance
• Examine practicality and tolerability of the
  measures
• ease of set up, tester training, and scoring,
  missing data, assessment duration,
• ratings of subject satisfaction with measures

18
The VIM Study Sites / PIs

• Project PI Green Co-PI Schooler
• Sites and Principal Investigators
• UCLA / Los Angeles VA
• Robert Kern, Michael Green
• Collaborative Neuroscience Network
• David Walling, Ph.D.
• Harvard University, Deaconess Beth Israel
  Hospital
• Larry Seidman, Ph.D., William Stone, Ph.D.
• Uptown Research, Chicago
• John Sonnenberg, Ph.D.

19
VIM Study Design

• 160 subjects 40 at each of four sites
• Two assessments
• Baseline
• Four weeks
• Intermediate measures based on RAND panel review
• MATRICS Consensus Cognitive Battery (MCCB)
• Quality of Life Scale to assess functioning
• PANSS to assess psychopathology
• Three independent interviewer/raters
• MCCB and intermediate performance measures
• Interview based intermediate measures
• PANSS

20
VIM Study Timeline

• Spring/Summer 2008 Finalize study design
• Identify sites
• Prepare study protocol
• September 2008 Train research staff
• October 2008 May 2009 Recruit participants
• June 2009 End of data collection
• July September 2009 Data analysis
• October 2009 Presentation of results