Questionnaire Development

1
Questionnaire Development

• Bob Woldman
• Head, Survey Operations
• NC State Center for Health Statistics

2
Developing good questions for survey
questionnaires is one of THE HARDEST aspects of
survey research
3
Have you stopped beating your wife?

• See anything wrong here?
• Who says I started?
• Who says I have a wife?
• What does beating mean?
• What does stopped mean?
• Who wants to answer a question like that anyway?

4
Is your work made more difficult because you are
expecting a baby?

• What does a no answer mean?
• Could be that work is not affected, but could
  also mean they are not expecting a baby.

5
Do you agree that acquired immunodeficiency
syndrome (AIDS) can be transmitted by shaking
hands with a person or through other means of
physical contact?

• What does a no answer mean here?
• May mean not through other means of physical
  contact, not through shaking hands, or both.
• Known as a double-barreled question two
  questions in one.

6
How often do you Exercise?a) regularlyb)
occasionally

• What do regularly and occasionally mean?
• Need to be specific, otherwise you leave too much
  to personal interpretation
• Example
• Twice a week or more
• Once a week
• Less than once a week

7
Have you had a recent colonoscopy or
sigmoidoscopy?

• What is recent?
• Give a time frame like In the last 12 months
• Do not say in the last year as this will cause
  some to think calendar year instead of 12 months.
• Whats a colonoscopy or sigmoidoscopy?
• Better Sigmoidoscopy and colonoscopy are exams
  in which a tube is inserted in the rectum to view
  the colon for signs of cancer or other health
  problems. Have you had either of these exams in
  the last 12 months?

8
Do you have diabetes?

• Allows for personal opinion about their health
  condition.
• Also, what if they simply do not know?
• Better Have you ever been told by a doctor that
  you have diabetes?
• A) Yes
• B) No
• C) Dont know/Not sure

9
How many cigarettes do you smoke a day?A) 1-5,
B) 5-25, C) 25 or more

• Whats wrong with this?
• Overlapping intervals
• What if they dont smoke?
• Better
• A) None, B) 1-4, C) 5-24, D) 25 or more
• Should also specify 1 pack 20 cigarettes

10
How many beers do you drink per month?

• Too far out.
• Unless the answer is none, it will be hard to
  give an accurate answer.
• Which month?
• Better to use specifically the most recent month
• Better How many beers did you drink in the last
  7 days?
• (in the last week may cause respondent to
  think calendar week)

11
To assist us in quantifying the prevalences of
disease in our region, we require that you inform
us where you reside.

• Huh?
• Too complex
• For the general public, try to keep all questions
  and statements at a 6th grade or lower reading
  level.
• Better To help us understand and measure which
  parts of our county have the least and most
  public health problems, we will need to know
  where you live.

12
Do you do physical exercise such as playing
tennis?

• Leading question. Respondent will focus on
  tennis
• Also avoid phrases like, Dont you agree that
  or Wouldnt you agree that

13
Does your child ever feel lonely or depressed?

• Proxy reporting. Many problems with this and
  with how to interpret the resulting data.
• Better Has a doctor or other health care
  provider ever told you that your child has a
  depressive disorder (including depression, major
  depression, dysthymia, or minor depression)?

14
Question Order

• Order does matter
• Think about if the question before may have an
  affect of leading the person to a certain
  response in the next question
• Always put sensitive questions at end

15
Response Order

• It matters too
• Need to be careful not to bias a respondent with
  first choices offered if possible.

16
Questionnaire Length

• Shorter is better
• Going too long can lead to response fatigue
• Rule of thumb try to keep it under 20 minutes to
  complete if possible

17
Avoid Change When Possible

• Sometimes improvements to questions should be
  made when creating a new questionnaire, but be
  very careful about comparing resulting statistics
  to other data
• Even the slightest change in a question can
  change the responses dramatically

18
How to write a good questionnaire

• Resist the impulse to write specific questions
  until you have thought through your research
  questions first
• Write down your research questions and have a
  hard copy available when you are working on the
  questionnaire
• Every time you write a question, ask yourself,
  Why do I want to know this?
• Feel free to borrow questions from established,
  national surveys.

Note 1-3 come from Asking Questions by
Bradburn, et al.
19
Composing Responses

• When creating your questionnaire be as careful
  about the response options as you are about the
  questions
• Think ahead to how you will analyze the data
• Simpler is better
• Avoid write-in responses except in unique cases

20
Test Your Questions

• Test any questions you develop or questions you
  have borrowed but changed in some way.
• You can do this by randomly selecting phone
  numbers from the phone book. You will need to
  develop a script for interviewers to use (I can
  provide an example if needed).
• In the script, explain to the respondent that you
  are developing survey questions for your program
  and you need to test them. Explain that you will
  be asking them the questions and then asking then
  if they find the question hard to understand or
  hard to answer.

21
Resources

• A catalog of biases in questions (Very good)
  http//www.cdc.gov/pcd/issues/2005/jan/04_0050.htm
  
• Great Book Asking Questions by Bradburn, Sudman
  and Wansink
• Internet Catalog of State and National Surveys
  (for borrowing) http//www.schs.state.nc.us/SC
  HS/brfss/surveys/

22
Contact Information

• Bob Woldman
• State Center for Health Statistics
• 2422 Mail Service Center
• Raleigh, NC 27699-2422
• Phone [phone]
• Fax [phone]
• Bob.Woldman_at_ncmail.net
• http//www.schs.state.nc.us/SCHS