The NIH Toolbox: An Instrument for Assessment of Neurological and Behavioral Function

1
The NIH Toolbox An Instrument for Assessment of
Neurological and Behavioral Function
2
David Tulsky - Kessler (KMRREC)
UMDNJ-NJMSSandra Weintraub - Northwestern
University Medical SchoolSureyya Dikmen -
University of WashingtonRichard Havlik -
WestatDavid Blitz - Evanston Northwestern
HealthcareRobert Heaton - University of
California, San DiegoBeth Borosh - Northwestern
University Medical SchoolNancy Chiaravalloti -
Kessler (KMRREC) UMDNJ-NJMS Amanda OBrien -
Kessler (KMRREC) UMDNJ-NJMS Joni Machamer
University of WashingtonRichard Gershon
Evanston Northwestern HealthcareMolly Wagster
NIH (NIA)
Contributors
3
NIH Toolbox for Neurological Behavioral Function

• There is little uniformity among the measures
  used to assess neurological function and
  behavioral health.
• The NIH Toolbox initiative seeks to assemble
  brief, comprehensive assessment tools that will
  be useful to clinicians and researchers in a
  variety of settings
• NIH Toolbox places particular emphasis on
  measuring outcomes in longitudinal epidemiologic
  studies and prevention or intervention trials
  across the lifespan.
• Currently, standardized measures of cognitive,
  sensory, motor, and emotional functioning are not
  included.

4
NIH Toolbox for Assessment of Neurological and
Behavioral Function

• If there were unified/integrated measures of
  multiple indicators (cognitive, emotional, motor,
  sensory) of neural and behavioral health
  functioning .
• Could be used as a form of common currency
  across diverse study designs and populations in
  large cohort studies
• Would maximize yield from large, expensive
  studies with minimal increment in subject burden
  and cost

5
The NIH Blueprint for Neuroscience Research

• Aims to develop new tools, resources, and
  training opportunities to accelerate the pace of
  discovery in neuroscience research.

• Initiated in 2004
• Designed to enhance collaboration among the NIH
  Office of the Director and 15 NIH institutes and
  centers (ICs) that support research on the
  nervous system.
• Pooling resources and expertise, the Blueprint
  confronts challenges that transcend any single
  institute or center and serves the entire
  neuroscience community.

http//www.neuroscienceblueprint.nih.gov/
6
Sponsorship

• Funding is provided by 15 NIH agencies
• The Toolbox effort is overseen by the National
  Institute of Aging (Molly Wagster)
• Funding is through a cost reimbursement contract
•  This project has been funded in whole or in part
  with Federal funds from the National Institute on
  Aging, National Institutes of Health, under
  Contract No.  HHS-N-260-2006-00007-C

7
NIH Project Team Members

• National Institute on Aging (NIA)
• National Institute on Mental Health (NIMH)
• National Institute on Drug Abuse (NIDA)
• National Institute on Neurological Disorders
  Stroke (NINDS)
• National Institute on Environmental Health
  Sciences (NIEHS)
• National Institute on Deafness and Other
  Communicative Disorders (NIDCD)
• National Eye Institute (NEI)
• National Institute on Child Health and Human
  Development (NICHD)
• National Institute of Nursing Research (NINR)
• National Institute on General Medical Sciences
  (NIGMS)
• Office of Behavioral and Social Sciences Research
  (OBSSR)
• National Center on Complementary and Alternative
  Medicine (NCCAM)
• National Institute on Alcohol Abuse and
  Alcoholism (NIAAA)
• National Center on Research Resources (NCRR)

8
Eight Domain-Sites
PI Richard C. Gershon, PhD

• Northwestern University - ENHRI
• Center for Outcomes, Research and Education and
  Northwestern University
• Rehabilitation Institute of Chicago
• Feinberg School of Medicine
• Northwestern School of Communications
• Primary Collaborators
• University of Washington
• University of Pittsburgh
• UCLA
• Kessler Medical Rehabilitation Research and
  Education Center

9
Four Domains

• Cognition
• Emotion
• Motor Functioning
• Sensation

10
Phase I 12 Months

• Expert Survey of criteria to select sub-domains
• Scientific Advisory Board (SAB) meetings to
  select final criteria
• Identification of existing psychometric tests and
  measurement tools
• Interviews with a variety of scientists
• Identification of domains of functioning
• Review of all potential tests
• Nomination of tests to use in toolbox.

11
Phase II 18 Months

• Official approval of tests (from NIH).
• Development of the new tests, items for adults
  and children.
• OMB approval of tests and items
• Pilot Testing of the tests and comparison against
  gold standards

12
Phase III (30 months)

• Spanish translation
• Program of CAT administration
• Calibration analysis of item bank based
  Measures
• Field testing/Normative study of toolbox
• Psychometric testing

13
Cognition Team ProgressOctober 19, 2007
NIH Toolbox Meeting 10.18.07
14
Cognition Team
SENIOR SCIENTISTS Sureyya Dikmen, University of
Washington Nathan Fox, Duke University Robert
Heaton, UCSD David Tulsky, KMRREC Sandy
Weintraub, Northwestern University JUNIOR
SCIENTISTS Beth Borosh, Northwestern University
Nancy Chiaravalloti, KMRREC Joni Machamer,
University of Washington Amanda OBrien,
KMRREC PROCESS/ORGANIZATION David Blitz, ENH
CORE Dick Havlik, Westat Jerry Slotkin, ENH CORE
NIH Toolbox Meeting 10.18.07
15
MANDATES FOR COGNITION TEAM

• Cover 4-6 Subdomains
• Take no more than 30 minutes
• Cover the full range of normal function
• Cover ages 3-85
• Instrument should be free/minimal cost
• Use state of the art methods

16
May Meeting 2007 and Beyond

• Submitted initial literature review
• Subdomains proposed
• Executive Function
• Episodic Memory
• Language
• Processing Speed

17
May Meeting 2007 and Beyond
FEEDBACK Executive Function is very
important. Working Memory should be included as
part of EF or separate, additional domain.
Questions about Attention. Get verification from
experts outside of the team.
18
RESPONSE

• Recruited Expert Consultants
• Split into focused subdomain working groups,
  weekly conference
• July 2007 Meeting

19
Recruitment of Expert Consultants

• Initial lists assembled by CORE
• NIH recommendations
• N. Fox recommendations
• Researchers and Clinicians
• Pediatric emphasis

20
SUBDOMAIN WORKING GROUPS
Processing Speed David Tulsky, KMRREC Noelle
Carlozzi, KMRREC Nancy Chiaravalloti,
KMRREC Timothy Salthouse, University of
Virginia Keith Yeates, Ohio State
University Amanda OBrien, KMRREC
Attention Sandy Weintraub, Northwestern
University Nathan Fox, University of Maryland
Koraly Perez-Edgar, George Mason
University Frank Zelko, Northwestern
University Dick Havlik, Westat
Working Memory David Tulsky, KMRREC Noelle
Carlozzi, KMRREC Nancy Chiaravalloti,
KMRREC Timothy Salthouse, University of
Virginia Keith Yeates, Ohio State University
21
SUBDOMAIN WORK GROUPS
Executive Function Phil Zelazo, University of
Minnesota Adele Diamond, University of
Vancouver Joel Kramer, University of California,
San Francisco Beth Borosh, Northwestern University
Episodic Memory Sureyya Dikmen, University of
Washington Patricia Bauer, Duke University Gordon
Chelune, University of Utah Dean Delis,
University of California, San Diego Joni
Machamer, University of Washington
Language Sandy Weintraub, Northwestern
University Jean Berko Gleason, Boston
University Kathy Hirsch-Pasek, Temple
University Roberta Golinkof, University of
Delaware Beth Borosh, Northwestern University
22
GENERAL COGNITIVE Bob Heaton, University of
California, San Diego Ida Sue Baron, University
of Virginia Dan Mungas, University of California,
David Esther Strauss, University of
Vancouver Jerry Sweet, Northwestern University
23
Instrument Review Methods

• CORE Librarian Search, supplemented with
  recommendations
• from initial RFI and Expert Interviews 400
• Grid of all instruments containing information
  about time to
• administer, psychometric properties, Spanish
  translation,
• acceptable age range, or if no information
• Grid divided into subdomains for working groups
  to cull
• Initial instrument cut

24
INITIAL CRITERIA FOR CUTTING INSTRUMENTS

• Self-Report, Proxy (for adults), Clinician-Rated
• Commercial instruments (IP Rights/purchase
  required) but if an exemplar or gold standard,
  or available in the public domain, consider as a
  Model
• Screening/Global measures (e.g., anything that
  only screens for impairment)
• Disease- or Population-Specific (e.g.,
  Alzheimers, Aged/Elderly, etc.)
• Administration Time over 20 minutes for
  Executive Function and
• Working Memory 10 minutes for other subdomains,
  in most instances
• unless instrument had very high standards value

25
July 2007 Meeting
ATTENDEES Beth Borosh Nancy Chiaravalloti
(P) Kevin Conway (NIH) Gordon Chelune Dean
Delis Sureyya Dikmen Nathan Fox Dick Havlik NIH
Phone Ellen Witt, Emmeline Edwards, Claudia
Moy CORE STAFF Blitz, Gershon, Nowinski, Lai,
Swantek, Wang
Bob Heaton Joel Kramer Jennifer Manly (P) Amanda
OBrien (P) David Tulsky Timothy Salthouse Phil
Zelazo Frank Zelko
26
July 2007 Meeting - Considerations

• Clinical and experimental measures each have
  their positives and negatives
• Experimental technologically advanced,
  measure more discrete behavioral components,
  validated against biological markers
• Clinical sensitive to a variety of
  disorders, more ecological validity
• Computer VS Paper-and-Pencil
• FINAL PRODUCT SHOULD BE ACCEPTABLE TO
• RESEARCHERS AND CLINICIANS

27
RECOMMENDED TESTS/PARADIGMS
28
Set-shifting task

• Objective is for participant to match figures by
  either color or shape depending on instruction
  that appears on screen.
• 3 blocks of trials
• All color
• All shape
• Random switching

29
Dimensional Change Card SortP. Zelazo, Nature
Protocols, 2006
Shape Trial Color Trial
30
Flanker Test (P. Zelazo)

• Target fish appear flanked by other fish, user
  indicates direction target fish is going
• Congruent, they are all going in the same
  direction Incongruent, target is different
  direction than the flanker fish
• Reverse condition, user must indicate the
  direction of the flanker fish (rather than the
  target fish).

Standard Incongruent Standard
Congruent
Reverse Congruent Reverse Incongruent
31
RECOMMENDED TESTS/PARADIGMS
PROCESSING SPEED SPEED OF PERFORMING SIMPLE
MENTAL ACTIVITIES NUMBER OF ITEMS THAT CAN BE
PROCESSED IN A SET AMOUNT OF TIME Opportunity
to combine across subdomains, using RT, time to
complete measures from other tests
32
RECOMMENDED TESTS/PARADIGMS
PROCESSING SPEED
33
Person is instructed to classify the pairs as
Same (S) or Different (D) as quickly as possible
(Salthouse, 1991)
34
WORKING MEMORY
Components

• to process information across a series of tasks
  and modalities,
• to hold the information in a short-term buffer,
• to manipulate the information, and
• 4) to hold the products in the same short-term
  buffer

35
RECOMMENDED TESTS/PARADIGMS
WORKING MEMORY
36
Instrument Selection and Development Plan
THREE-PRONGED APPROACH

• IDENTIFY CRITERION MEASURES FOR EACH SUBDOMAIN
• IDENTIFY KEY EXPERIMENTAL PARADIGMS FOR EACH
  SUBDOMAIN
• IDENTIFY EXISTING COMPUTER MODELS THAT COULD BE
  ADAPTED FOR TOOLBOX (E.G. ANAM)

37
DEVELOPMENT PLAN
TEST AGAINST VALIDATION MEASURES Work with
Epi-Biostats group to finalize sampling
plan Administer experimental tests we developed
and ANAM, clinical measures, demographic
information (age, education, employment status,
SES, ethnicity, reading level, acculturation,
etc.) Do our experimental measures
work? Acceptable psychometric properties Cover
age span Correlate with criterion measures
38
DEVELOPMENT PLAN - NO SUPRISES
RETAIN PERFORMERS MAKE NECESSARY REVISIONS MOVE
TO STANDARDIZATION WITH FINAL BATTERY
39
Cognition Emotion

• Processing Speed
• Memory
• Attention
• Working Memory
• Language/Verbal
• Executive Functioning

• Negative Affect
• Positive Affect
• Stress Coping
• Social Relationships

Rev. 4-07
40
Motor Functioning Sensory

• Locomotion (20 Ft Walk Test)
• Endurance (6 min walk test)
• Balance (Single Leg Stance)
• Dexterity (9-hole Peg test)
• Upper Strength (grip strength dynamometry)
• Lower/Trunk Strength (Electronic Muscle Strength)

Vision Hearing Somatosensation Olfaction Taste
41
Toolbox Open Meeting