RTI ppt template

1
Considerations When Using Family Surveys with
Spanish-Speaking Families Georgina
McAvinchey Murrey Olmsted, Ph.D. Don Bailey,
Ph.D. Measuring Child and Family Outcomes
National TA Meeting Baltimore, MD August 27,
2007
RTI International is a trade name of Research
Triangle Institute
2
Goals for this Session

• Provide background of issues and early findings
• Describe cognitive testing as a methodology for
  understanding how respondents understand and
  respond to survey directions and items.
• Present findings from a study using cognitive
  testing interviews with 40 Spanish speaking
  families in early intervention programs
• Initiate a forum for discussing ways to maximize
  valid participation in state surveys by families
  whose primary language is not English

3
Background Information

• Families participating in early intervention
  programs are ethnically diverse

Percent of Participants
4
Background Information

• The number and proportion of Hispanic/Latino
  families enrolled in early intervention is
  increasing

Number of Hispanic/Latino Families Enrolled
5
Background Information

• Hispanic/Latino families enrollment in early
  intervention is growing much faster than African
  American enrollment

6
Background Information

• The NEILS study found differences by
  race/ethnicity in initial experiences with early
  intervention (Bailey et al., 2004)

7
Background Information

• A more recent NEILS paper (Bailey et al., 2005)
  showed that race/ethnicity was a significant
  predictor of family outcomes at 36 months of age
• White families significantly higher than black
• Black families significantly higher than other
  minority groups
• A forthcoming NEILS paper (Bailey et al., in
  press) using structural equations modeling found
  that white families tended to report more
  positive perceptions of quality of family
  services and were more likely to report that EI
  had a positive effect on their family

8
Background Information

• A recent pilot study of the Family Outcomes
  Survey showed substantial differences in outcomes
  reported by English-speaking versus
  Spanish-speaking families

9
Background Information
10
Study Goals

• Fundamental Question
• Do differences in responses represent true
  differences in outcome attainment or might they
  reflect problems with translation, instrument
  format, different cultural interpretations of the
  meanings of items, or different expectations for
  what programs can and should do for families?
• Research Sponsor
• This project was sponsored by the Early Childhood
  Intervention Division (ECID) of the Texas
  Department of Assistive Rehabilitative Services
• Research Approach
• To answer this question, we conducted a set of
  cognitive testing interviews with 40
  Spanish-speaking families in Illinois and Texas

11
What is Cognitive Interviewing?

• Cognitive interviewing is a qualitative testing
  methodology used to explore common problems in
  the way information is displayed, questions are
  asked, and tasks are presented to respondents in
  questionnaires (Willis, Royston, Bercini, 1991,
  Willis, 1994, 2005).
• Based on a cognitive psychology model of the
  mental processes involved in responding to
  questionnaires and surveys by Tourangeau (1984)
  and Eisenhower et al. (1991)
• Encoding in memory (have some knowledge/memory of
  the question)
• Comprehension (understand the question and
  relevant concepts)
• Retrieval (retrieve the information from memory)
• Judgment (assess the completeness/relevance of
  their memories)
• Communication (decide whether their answer fits
  the answer categories provided and also decide
  whether they actually want to provide an answer
  or provide one that might be socially acceptable)

12
What Kinds of Questions Does it Address?

• Using structured interview techniques, a
  methodologist examines the following kinds of
  questions
• What do respondents think the question is asking?
  
• What do specific words and phrases in the
  questions mean to respondents?
• Are words used that respondents do not
  understand?
• Are the memory and/or decision requirements of
  the questionnaire reasonable?
• How do respondents choose their answers?
• How well do respondents understand instructions
  and visual cues?

13
How Does it Work in Practice?

• Identify issues in a questionnaire that may be
  problematic. This can often be done via a
  structured instrument review process.
• Develop a cognitive testing guide to address each
  issue
• Recruit participants that are representative of
  the target audience
• 8-12 participants are typical for an average
  questionnaire
• If federally funded, OMB restricts to 9
  interviews per questionnaire
• Conduct testing interviews in a one-to-one
  setting
• Select a quiet location for the interviews where
  you will not be disturbed
• Record all interviews and take good hand-written
  notes
• Provide an incentive as a Thank you gift to
  participants
• Review interviews and summarize the findings and
  the recommendations for possible changes to the
  questionnaire
• Discuss the findings and implications with the
  project team and determine which changes are
  appropriate

14
Probing Techniques Used in Interviewing

• Primary approaches for probing used in cognitive
  interview are concurrent and retrospective
  probing.
• Concurrent probing This involves asking the
  participant questions as they fill out the
  questionnaire to explore their understanding of
  concepts, questions, and other related factors.
• Retrospective probing Participants are first
  asked to complete a copy of the questionnaire and
  then are asked questions about their
  understanding of terms, questions, and concepts
  as the interviewer reviews the questionnaire with
  them.
• Some examples of types of probes used include
• Comprehension probe What do you think is meant
  in the question by your childs development?
• Paraphrase probe In your own words, what is this
  question asking?
• Confidence judgment probe Do you feel that you
  know enough about your childs development?
  Explain.
• Elaborative/Expansive probe Can you explain?
• Recall probe Think of the person or persons that
  provide services to your child. Do you feel that
  you can adequately communicate with them?
  Explain.

15
What Are the Results of Cognitive Interviewing?

• Develop a cognitive testing summary report
  providing an overview of the background, methods,
  findings, and recommendations
• Identification of problematic questions
• Recommendations for possible solutions
• Marked-up questionnaire showing possible changes
• Vet possible changes to ensure that solutions
  address findings and do not cause further
  problems
• Ensure that the meaning of the questions are not
  lost in recommended changes
• Evaluate the language and cultural implications
  of any changes
• Update and test the new version of the
  questionnaire

16
Background of the Spanish Testing

• ECID requested that RTI conduct cognitive testing
  of the FOS with Spanish speaking families in
  Texas and Illinois
• Earlier pilot data of the original Spanish
  version of the FOS showed that families had
• Lower average scores on many items
• Lower overall response rates
• Significant item-level missing data
• Testing sought to determine whether predominantly
  Spanish-speaking parents are able to understand
  and respond to the survey easily

17
Spanish Cognitive Interview Protocol

• Protocol was based on the materials that were
  used to test the English version of the FOS
  during the summer of 2006
• Also included new questions about access to
  services and experience with early intervention
• Previous research support the idea that cognitive
  testing interviews can provide valuable insights
  when testing cross-cultural survey instruments
• It can reveal ways in which cultural differences
  may affect how respondents perform cognitive
  tasks involved in survey responding

18
Methodology and Participants

• Cognitive testing were completed with
• 40 family members
• 6 sites across Texas and Illinois
• Qualified participants were Hispanic/Latino
  families in Texas and Illinois
• Interviews conducted in June and July 2007
• 90 to 120 minute interviews
• 70 cash incentive

19
Recruiting Participants

• Recruiting methods
• Posting flyers at early intervention centers
• Utilized contacts established by coordinators,
  family contact personnel and parent liaisons to
  reach out to families
• Focused recruiting effort on obtaining
• Parents or family members who had some caretaking
  role of a child who was currently enrolled in an
  early intervention program
• Spanish speakers who are monolingual or bilingual
  with Spanish language dominance

20
Methodology Inclusion Criteria

• Inclusion criteria
• Linguistic skills (monolingual and bilingual
  speakers with Spanish dominance)
• Education
• Region of origin
• Demographic diversity

21
Language Dominance

• Language dominance was assessed for each
  participant
• Evaluated the language that the participant spoke
  in the home and studied in early school
• Current usage of language
• Language the participant feels most comfortable
  using for communication
• Participants were classified as
• Monolingual English speaker (speaks only English
  not eligible)
• Monolingual Spanish speaker (speaks only Spanish,
  eligible)
• Bilingual, English dominant speaker (speaks both
  Spanish and English dominant in English not
  eligible)
• Bilingual, Spanish dominant speaker (speaks both
  Spanish and English dominant in Spanish
  eligible)

22
Respondent Profile

• Demographics - 40 family members in 6 sites
  across Texas and Illinois
• Language dominance characteristics
• 6 monolingual Spanish speakers
• 34 bilingual speakers with Spanish dominance
• 3 men and 37 women
• Median age 32 (Min 23, Max 53)
• Exposure to early childhood service ranged from 1
  to 4 years
• Typically referred to early intervention by
  hospitals and/or pediatricians, except for a few
  children whose parents requested a referral

23
Respondent Profile Spanish-speaking respondents

• Countries represented by participants included
• Mexico Peru
• El Salvador Dominican Republic
• Guatemala Ecuador
• Honduras Venezuela
• The majority of respondents (75) reported being
  of Mexican descent

24
Findings and Recommendations

• Presentation of the finding and recommendations
  will be presented as follows
• General Findings Related to Early Intervention
  Services
• Specific Instrument Design Issues
• Example of Findings and Recommendations
• Overall Recommendations

25
General Findings Related to Early Intervention
Services

• Participants Experience and Understanding of
  Early Intervention
• Most participants praised the early intervention
  services
• Tenía una venda y me la quitaron. Me hicieron
  ver las cosas y aprender. Fue una nueva
  experiencia.
• (I was blindfolded and they took it off. They
  made me see and learn things. It was a new
  experience.)

26
General Findings Related to Early Intervention
Services

• Desire to have early intervention services
  expanded past the 3 years of age
• Parents initial expectations originated from
  first Early Intervention meeting(s)
• Parents expectations have been met and exceeded
  in several cases
• Expected some type of improvement in their
  childs special situation
• Eager to learn how to better interact and help
  their children

27
Impact of Language on Early Intervention Services

• Good communication with service providers
• Service providers responded very well to the
  childs needs
• Some language barrier
• Si yo pudiera hablar inglés estaríamos mejor,
  nos entenderíamos mucho mejor y me entenderían
  más lo que yo les trato de decir
• (It would be better if I could speak English, we
  could understand each other much better, and they
  would understand what I try to tell them).
• Fairly high overall satisfaction with having
  bilingual personnel available

28
Specific Instrument Design Issues

• Participants generally liked the FOS
  questionnaire
• A few consistent issues emerged from the
  cognitive testing which should be addressed
  including
• Instructions and 7-point Scale
• Problems with Concepts and Terminology
• Issues Affecting Comprehension of Survey Items
  and Instruction

29
Instructions for answer option 2, 4, and 6

• If a statement almost describes your family, but
  not quite, circle the number just to the left or
  the right. For example if you feel that the
  statement 5 We know a good amount about
  dinosaurs almost describes your family, but not
  quitecircle the 4.
• How much does your family know about dinosaurs?

30
7-point Scale

• Many participants appeared uncomfortable or
  struggled using the scale during the first few
  questions
• The spaces between response options were
  described as confusing by some but not all
  participants.
• Estos huequitos de en medio confunden.
• (These little holes in the middle are
  confusing.)

31
Instructions and 4-point Scale

• Recommendations for possible change to the FOS
• Eliminate instructions for answer option 2, 4 and
  6
• Remove spaces between response options (see below)

32
Problems with Concepts and Terminology

• Most of the concepts found in the questionnaire
  were well understood by parents
• Some concepts and terms were confusing or needed
  clarification
• Incorrect assumptions about participants
  knowledge of the terminology
• Others were associated with vagueness
• Some additional issues are presented in the
  question by question review of the report

33
Key concepts that caused difficulty

• Child - Most participants understood this term as
  intended but there were a few parents with more
  than one child receiving early intervention (EI)
  services who were unsure which child to consider
  when completing the survey
• Early Intervention Services (EI services) - The
  inclusion of this term in the survey assumed
  parental knowledge of early intervention
  services. Most participants reported lacking this
  knowledge prior to receiving these services for
  their children
• Special needs - This term was found to be
  sensitive, and widely misunderstood by
  participants. Several of the participating family
  members associated this term with severe physical
  or mental impairments

34
Issues affecting Comprehension

• Reading level Many questions appeared to be
  over the participants current reading level and
  limited vocabulary.
• Si se me hace difícil a veces el español. El
  español es diferente. A veces hay palabras que
  uno no las usa y si no las usa, a veces no está
  uno seguro si uno sabe lo que quiere decir. Esta
  muy elevado la forma de preguntar. Esta muy
  formal. Lo pondría en palabras más sencillas
• (Sometimes I have a hard time with Spanish.
  Spanish is different. There are some words that
  if you do not use them for a while, you are not
  sure what they mean. The way of asking is too
  high. Its too formal. I would use simpler
  words)

35
Issues affecting Comprehension

• Unnecessary wording in some questions Parents
  had trouble understanding some questions due to
  unnecessary or confusing text appearing in the
  questions
• Las preguntas no van directo al punto. Dan
  primero una dan como dos preguntas en una sola
  pregunta. Simplificaría haciendo las preguntas
  más directas.
• (Questions should go straight to the point.
  Sometimes they ask like two questions in one. I
  would simplify them by making them more direct).

36
Issues affecting Comprehension

• Cultural issues There were some cultural issues
  and assumptions that might have affected
  comprehension
• Creo que es tabú, porque si dices que tu niño es
  autista, creen que es mongolito, y como si
  fuera malo. Como aceptar que tu hijo es un
  perdedor. No solo en la comunidad mexicana, sino
  en los latinos. No sabemos del tema.
• (I think its taboo, if you mention that your
  child is autistic, they think he is retarded, as
  if it was something bad. It is like admitting
  that your child is a loser. This not exclusive to
  Mexicans but the whole Latino community. (In our
  culture) we do not know about this.)

37
Example of Findings Question 2

• Example
• Question 2 (Original)
• Algunos niños tienen necesidades médicas
  especiales, alguna discapacidad o retraso en el
  desarrollo. Estas a menudo se conocen como
  necesidades especiales. Qué tan familiarizada
  está su familia con las necesidades especiales de
  su hijo?
• (Some children have special health needs, a
  disability, or are delayed in their development.
  These are often referred to as special needs.
  How familiar is your family with your child's
  special needs?)

38
Example of Changes Question 2

• Example
• Question 2 (Revised)
• Qué tan familiarizada está su familia con las
  necesidades especiales de su hijo? (Las
  necesidades especiales incluyen necesidades
  médicas, alguna discapacidad, problemas del
  habla, visión o audición, o demora en el
  desarrollo.)
• How familiar is your family with your child's
  special needs?) (Special needs include health
  needs, a disability such as a speech, vision or
  hearing problem, or delayed development).

39
Overall Recommendations

• Simplify the survey items by moving from a
  two-sentence format to a single statement
• Reduce the number of scale points from 7 to 4 by
  eliminating the current ratings of 2, 4, and 6
  for which there are no behavioral anchors
• Revise each question to clarify the meanings of
  key terms/concepts and simplify the provided
  response options
• Make selected changes in the Spanish translation
  using words that are more likely to be understood
  by the majority of Spanish speakers

40
Questions Answers

• Any questions?