Class 9 Analyzing Pretest Data, Modifying Measures, Keeping Track of Measures, Creating Scale Scores

1
Class 9 Analyzing Pretest Data, Modifying
Measures,Keeping Track of Measures, Creating
Scale Scores November 15, 2007

• Anita L. Stewart
• Institute for Health Aging
• University of California, San Francisco

2
Overview of Class 9

• Analyzing pretest data
• Modifying/adapting measures
• Keeping track of your study measures
• Creating and testing scales in your sample

3
Summarize Data on Pretest Interviews

• Summarize problems and nature of problems for
  each item
• Determine how important problems are
• Results become basis for possible
  revisions/adaptations

4
Methods of Analysis

• Optimal transcripts of all pretest interviews
• For each item - summarize all problems
• During standard administration
• Responses to specific probes
• Types of problems
• Interviewer problems
• Respondent problems

5
Methods of Analysis

• Analyze dialogue (narrative) for clues to solve
  problems
• During standard administration
• Responses to specific probes

6
Behavioral Coding

• Complements cognitive interviews
• Systematic approach to identifying problems with
  items
• interviewer and respondent problems

7
Examples of Interviewer Behaviors Indicating
Problem Items

• Question misread or altered
• Slight change meaning not affected
• Major change alters meaning
• Question skipped

8
Examples of Respondent Behaviors Indicating
Problem Items

• Asks for clarification or repeat of question
• Did not understand question
• Doesnt know the answer
• Qualified answer (e.g., it depends)
• Indicates answer falls between existing response
  choices
• Refusal

9
Summarize Behavioral Coding For Each Item

• Proportion of interviews with each problematic
  behavior
• For standard administration
• of occurrences of each problem divided by N,
  e.g., 7/48 respondents requested clarification

10
Behavioral Coding Summary Sheet Standard
Administration (N20)
11
Additional Behavioral Codes Based on Probes

• Respondents may appear to answer question
  appropriately
• Additional problems identified with probes

12
Examples of Behavioral Codes Based on Probes

• Probe on meaning
• Response indicates lack of understanding
• Probe on use of response options
• Response indicates options are problematic

13
Behavioral Coding of Probe Results

• I asked you how often doctors asked you about
  your health beliefs. What does the term health
  beliefs mean to you?
• Behavioral coding times response indicated
  lack of understanding as intended
• e.g., 2/15 respondents did not understand meaning
  based on response to probe

14
Behavioral Coding Summary Sheet Standard
Administration (N20) Probes
15
Interpret Results

• Determine if problem is common
• Items with only a few problems may be fine
• Items are questionable when
• several types of problems were found
• several subjects experienced the same problem
• Another approach
• Problem identified in gt15 of interviews as
  criterion for further exploration

16
Interpret Results (cont.)

• Determine if common problems with an item are
  serious
• Gross misunderstanding of the question
• Yields completely erroneous answer
• Couldnt answer the question at all
• Some less serious common problems can be
  addressed by improved instructions or a slight
  modification

17
Behavioral Coding Identifies Problem Items

• Solution not always obvious
• How to determine ways to modify the items

18
Content Analysis of Entire Interview

• Use qualitative analysis software (e.g., NVIVO)
• Review all dialogue that ensued during
  administration of structured items and open-ended
  probes
• can reveal source of problems
• can help in deciding whether to keep, modify or
  drop items

19
Results Probing Meaning and Cultural
Appropriateness

• I asked you how often doctors asked you about
  your health beliefs? What does the term health
  beliefs mean to you?
• .. I dont want medicine
• .. How I feel, if I was exercising
• .. Like religion? --not believing in
  going to doctors?

20
Results Probing Meaning and Cultural
Appropriateness

• I asked you how often doctors asked you about
  your health beliefs? What does the term health
  beliefs mean to you?
• .. I dont want medicine
• .. How I feel, if I was exercising
• .. Like religion? --not believing in
  going to doctors?
• We changed the question to personal beliefs
  about your health

21
Results Probing the Meaning of a Phrase

• What does the phrase office staff mean to
  you?
• the receptionist and the nurses
• nurses and appointment people
• the person who takes your blood
  pressure and the clerk in the front
  office

22
Modification Probing the Meaning of a Phrase

• What does the phrase office staff mean to you?
• the receptionist and the nurses
• nurses and appointment people
• the person who takes your blood
  pressure and the clerk in the front
  office
• We changed the question to receptionist and
  appointment staff

23
Other Examples

• On about how many of the past 7 days did you eat
  foods that are high in fiber, like whole grains,
  raw fruits, and raw vegetables?
• Probe what does the term high fiber mean to
  you?

24
Other Examples

• On about how many of the past 7 days did you eat
  foods that are high in fiber, like whole grains,
  raw fruits, and raw vegetables?
• Behavioral Coding of item
• Over half of respondents exhibited a problem
• Review answers to probe
• Over ¼ did not understand the term

Blixt S et al., Proceedings of section on survey
research methods,American Statistical
Association, 19931442.
25
Other Examples (2)

• I seem to get sick a little easier than other
  people (definitely true, mostly true, mostly
  false, definitely false)
• Behavioral coding of item
• Very few problems

Blixt S et al., Proceedings of section on survey
research methods,American Statistical
Association, 19931442.
26
Other Examples (2)

• I seem to get sick a little easier than other
  people (definitely true, mostly true, mostly
  false, definitely false)
• Review answers to probe
• Almost 3/4 had comprehension problems
• Most problems around term mostly (either its
  true or its not)

27
Exploring Differences by Diverse Groups

• Back to issues of equivalence of meaning across
  groups
• All cognitive interview analyses can be done
  separately by group

28
Results Use of Response Scale

• Do diverse groups use the response scale in
  similar ways?
• Re questions about cultural competence of
  providers
• Interviewers reported that Asian respondents who
  were completely satisfied did not like to use the
  highest score on the rating scale

CPEHN Report, 2001
29
Results Probe on DifficultyCES-D Item

• During the past week, how often have you felt
  that you could not shake off the blues, even with
  help from family and friends
• Probe Do you feel this is a question that people
  would or would not have difficulty understanding?
• Latinos more likely than other groups to report
  people would have difficulty

TP Johnson, Health Survey Research Methods, 1996
30
Use of Response Scale (Not in Pretest)

• In an exercise class of Samoans, instructor asked
  them to rate the difficulty of the exercise he
  just did on a 1-10 scale
• They did not understand what he meant by a 1-10
  scale
• Western metric?

31
Overview of Class 9

• Analyzing pretest data
• Modifying/adapting measures
• Keeping track of your study measures
• Creating and testing scales in your sample

32
Now What!

• Issues in adapting measures based on pretest
  results

33
Switzer et al. reading

• From class 3 section of class binder
• p 405-406 modifying measures

34
Criteria for Whether or Not to Modify Measure

• Contact author
• May be open to modifications, working with you
• Be sure your opinion is based on extensive
  pretests with consistent problems
• Dont rely on a few comments in a small pretest
• Work with a measurement specialist to assure that
  proposed modifications are likely to solve problem

35
Tradeoffs of Using Adapted Measures

• Advantages
• Improve internal validity
• Disadvantages
• Lose external validity
• Know less about modified measure
• Need to defend new measure

36
Strategies for Modifying

• Retain original intact items (if feasible)
• Add modified items
• New items
• Slight modifications
• If modifications are extensive
• Pretest your new items

37
Modifying response categories

• If response choices are too few and/or coarse,
  can improve without compromising too much
• Try adding levels within existing response scale

38
One Modification Too Many Response Choices

• SF36 version 1
• 1 - All of the time
• 2 - Most of the time
• 3 - A good bit of the time
• 4 - Some of the time
• 5 - A little of the time
• 6 - None of the time

• SF36 version 2
• 1 - All of the time
• 2 - Most of the time
• 3 - Some of the time
• 4 - A little of the time
• 5 - None of the time

39
Modification of Health Perceptions Response
Choices for Thai Translation

• Usual responses
• 1 - Definitely true
• 2 - Mostly true
• 3 - Dont know
• 4 - Mostly false
• 5 - Definitely false

• Modified
• 1 Not at all true
• 2 A little true
• 3 - Somewhat true
• 4 - Mostly true
• 5 Definitely true

e.g., My health is excellent, I expect my health
to get worse
40
Modifying Item Stems

• If item wording will not be clear to your
  population
• Can add parenthetical phrases
• Have you ever been told by a doctor that you have
  diabetes (high blood sugar)?

41
Writing New Items

• One approach if you find serious problems with a
  standard measure
• Write new items that you think will be better
• Same format
• Always include entire original measure (if
  feasible)
• New items are extra

42
Strategy for Modified Measures

• Test measure in original and adapted form
• Choose measure that performs the best

43
Analyzing New Measure (Scale)

• Factor analysis
• All original items
• Original plus new items replacing original
• Correlations with other variables
• Does the new measure detect stronger
  associations?
• Outcome measure
• Does the new measure detect more change over
  time?

44
Overview of Class 9

• Analyzing pretest data
• Modifying/adapting measures
• Keeping track of your study measures
• Creating and testing scales in your sample

45
Questionnaire Guides

• Organizing your survey data and measures
• Way to keep track of measurement decisions
• Documents sources of measures before you forget
• Any modifications

46
See Sample Guide to Measures Used in
Questionnaire/Survey Handout

• Type of variable
• Concept
• Measure
• Data source
• Number of items/survey question numbers
• Number of scores or scales for each measure
• References

47
Codebook See Sample Questionnaire Guide
Summary of Variables.. Handout

• Develop codebook of scoring rules
• Several purposes
• Variable list
• Meaning of scores
• Special coding
• How you want missing data handled

48
Item Naming Conventions

• Optimal coding is to assign raw items their
  questionnaire number
• Can always link back to questionnaire easily
• Some people assign a variable name to the
  questionnaire item
• This will drive you crazy

49
Variable Naming Conventions

• Assigning variable names is an important step
• make them as meaningful as possible
• plan them for all questionnaires at the beginning
• For study with more than one source of data, a
  suffix can indicate which point in time and which
  questionnaire
• B for baseline, 6 for 6-month, Y for one year
• M for medical history, L for lab tests

50
Variable Naming Conventions (cont)

• Medical History Questionnaire
• HYPERTMB HYPERTM6
• Baseline 6 months

51
Variable Naming Conventions (cont)

• A prefix can help sort variable groupings
  alphabetically
• e.g., S for symptoms
• SPAINB, SFATIGB, SSOBB

52
Overview of Class 9

• Analyzing pretest data
• Modifying/adapting measures
• Keeping track of your study measures
• Creating and testing scales in your sample

53
On to Your Field Test or Study

• What to do once you have your baseline data
• How to create summated scale scores

54
Preparing Surveys for Data Entry 4 Steps

• Review surveys for data quality
• Reclaim missing and ambiguous data
• Address ambiguities in the questionnaire prior to
  data entry
• Code open-ended items

55
Review Surveys for Data Quality

• Examine each survey in detail as soon as it is
  returned, and mark any..
• Missing data
• Inconsistent or ambiguous answers
• Skip patterns that were not followed

56
Reclaim Missing and Ambiguous Data

• Go over problems with respondent
• If survey returned in person, review then
• If mailed, call respondent ASAP, go over missing
  and ambiguous answers
• If you cannot reach by telephone, make a copy for
  your files and mail back the survey with request
  to clarify missing data

57
Address Ambiguities in the Questionnaire Prior to
Data Entry

• When two choices are circled for one question,
  randomly choose one (flip a coin)
• Clarify entries that might not be clear to data
  entry person

58
Code Open-Ended Items

• Open-ended responses have no numeric code
• e.g., name of physician, reason for visiting
  physician
• Goal of coding open-ended items
• create meaningful categories from variety of
  responses
• minimize number of categories for better
  interpretability
• Assign a numeric score for data entry

59
Example of Open-Ended Responses

• 1.What things do you think are important for
  doctors at this clinic to do to give you high
  quality care?
• Listen to your patients more often
• Pay more attention to the patient
• Not to wait so long
• Be more caring toward the patient
• Not to have so many people at one time
• Spend more time with the patients
• Be more understanding

60
Process of Coding Open-Ended Data

• Develop classification scheme
• Review responses from 25 or more questionnaires
• Begin a classification scheme
• Assign unique numeric codes to each category
• Maintain a list of codes and the verbatim answers
  for each
• Add new codes as new responses are identified
• If a response cannot be classified, assign a
  unique code and address it later

61
Example of Open-Ended Codes

• Communication 1
• Listen to your patients more often 1
• Pay more attention to the patient 1
• Access to care 2
• Not to wait so long 2
• Not to have so many people at one time 2
• Allow more time 3
• Spend more time with the patients 3
• Emotional Support 4
• Be more understanding 4
• Be more caring toward the patient

62
Verify Assigned Codes

• Ideally, have a second person independently
  classify each response according to final codes
• Investigator can review a small subset of
  questionnaires to assure that coding assignment
  criteria are clear and are being followed

63
Reliability of Open-Ended Codes

• Depends on quality of question, of codes
  assigned, and the training and supervision of
  coders
• Initial coder and second coder should be
  concordant in over 90 of cases

64
Data Entry

• Set up file
• Double entry of about 10 of surveys
• SAS or SPSS will compare two for accuracy
• Acceptable 0-5 error
• If 6 or greater consider re-entering data

65
Print Frequencies of Each Item and Review Range
Checks

• Verify that responses for each item are within
  acceptable range
• Out of range values can be checked on original
  questionnaire
• corrected or considered missing
• Sometimes out of range values mean that an item
  has been entered in the wrong column
• a check on data entry quality

66
Print Frequencies of Each Item and Review
Consistency Checking

• Determine that skip patterns were followed
• Number of responses within a skip pattern need to
  equal number who answered skip in question
  appropriately

67
Print Frequencies of Each Item and Review
Consistency Checking (cont.)

• 1. Did your doctor prescribe any medications?
  (yes, no)
• 1a. If yes, did your doctor explain the side
  effects of the medication?
• If 75 respondents (of 90) said yes to 1, expect
  75 responses to question 1a.
• Often will find that more people(e.g., 80)
  answered the second question than were supposed to

68
Print Frequencies of Each Item and Review
Consistency Checking (cont.)

• Go back to a questionnaires of those with
  problems
• check whether initial filter item was
  incorrectly answered or whether respondent
  inadvertently answered subset
• sometimes you wont know which was correct
• Hopefully this was caught during initial review
  of questionnaire and corrected by asking
  respondent

69
Deriving Scale Scores

• Create scores with computer algorithms in SAS,
  SPSS, or other program
• Review scores to detect programming errors
• Revise computer algorithms as needed
• Review final scores

70
Creating Likert Scale Scores

• Translate codebook scoring rules into program
  code (SAS, SPSS)
• Reverse all items as specified
• Apply scoring rules
• Apply missing data rules
• Sample for SAS (see handout)

71
Testing Scaling Properties and Reliability in
Your Sample for Multi-Item Scales

• Obtain item-scale correlations
• Part of internal consistency reliability program
• Calculate reliability in your sample (regardless
  of known reliability in other studies)
• internal-consistency for multi-item scales
• test-retest if you obtained it

72
SAS Chapter 3 Assessing Reliability with
Coefficient Alpha

• Review statements and output
• How to test your scales for internal consistency
  and appropriate item-scale correlations

73
SAS Chapter 3 Assessing Scale Reliability with
Coefficient Alpha

• PROC CORR
• DATAdata-set-name
• ALPHA
• NOMISS
• VAR (list of variables)
• Output
• Coefficient alpha
• Item correlations
• Item-scale correlations corrected for overlap

74
Testing Reliability in STATA

• www.stata.com/help.egi?alpha
• Alpha varlist if in , options
• SEE HANDOUT

75
What if Reliability is Too Low?

• Have to decide if you need to modify a scale
• New scales under development
• Modify using item-scale criteria
• Standard scales cannot change
• Simply report problems as caveats in your
  analyses
• If problem is substantial
• Can create a modified scale and report results
  using standard and modified scale

76
Homework for Class 10

• Summarize briefly your pretest results
• Indicate whether the measure appears to be
  appropriate for the people in your pretest
• No inferences to broader sample needed

.