Surveys and Questionnaires

1
Surveys and Questionnaires

• Research Methods and Data
• College of Advancing Studies
• Brendan Rapple

2
Surveys Provide Important Knowledge

• Economists, psychologists, health professionals,
  political scientists, and sociologists conduct
  surveys to study such topics as
• Income and expenditure patterns among households
• Voting behavior
• Effects on family life of women working outside
  the home, etc.
• Auto manufacturers use surveys to find out how
  satisfied people are with their cars.
• And a multitude of other topics
• This presentation owes much to the American
  Statistical Association brochure series on survey
  research
• http//www.amstat.org/sections/srms/whatsurvey.htm
  l

3
Specific Purpose Essential

• Objectives of a survey should be as
• Specific
• Clear-cut
• Unambiguous as possible

4

• "Men's Health Practices" is a very nebulous
  topic.
• Better
• How often do African-American males aged 40-49
  visit the dentist?
• Or
• A survey of 50-60 year old male professors at BC
  about their weekly exercise habits

5
Steps in Conducting A Survey

• Define precise purpose
• Specify population
• Specify appropriate sample
• How to administer survey?
• Draft of survey instrument
• Pretest it
• Revise it
• Administer survey
• Analyze, write it up, and communicate the results
• Use results meaningfully

6
Decide on Mode of Data Collection

• Mailed Questionnaire
• Telephone
• In Person Interview
• Computer Questionnaire

7
Use Mailed/Computer Questionnaires or Not?

• Perhaps better responses by phone -- but phone
  interviewing is very time-consuming.
• On other hand, people tend to be more truthful
  with anonymous questionnaires.
• Generally cheaper than one-on-one interviews.
• Perhaps a mix of questionnaire and interviewing?

8
For All Surveys it is Essential to

• explain purpose of survey very clearly and
  precisely.
• explain any potential use the results will have
  for the respondents themselves.
• stress voluntary nature - respondents are doing a
  favor to the researcher.
• use letter-head official for a
  mailed/computer questionnaire.
• explicitly promise confidentiality.
• mention a clear expression of thanks.

9
Whatever Format Used . . . . . .

• Important to specify a deadline for response.
• Follow-up letter (e-mail) often advisable for a
  mailed/computer questionnaire
• Always essential for PILOT STUDY or PRE-TEST
• Main problems revolve about
• Question content, e.g. confusion with overall
  meaning of question as well as misinterpretation
  of individual terms or concepts
• Formatting, e.g. problems with how to skip or
  navigate from question to question may result in
  missing data and frustration for both
  interviewers and respondents.

10
Population ? Sample

• Usually the population to be surveyed is too
  large.
• Accordingly, one must select a smaller,
  representative sample.
• This sample is usually just a fraction of the
  population being studied.

11
Samples

• The quality of the sample whether it is
  up-to-date and complete is probably the
  dominant feature for ensuring adequate coverage
  of the desired population to be surveyed.
• Must be representative of population.
• Are the distributions of attributes, opinions,
  and beliefs in the sample the same as in the
  population?
• You want to be able to make inferences about the
  population as a whole based on what you find to
  be true of the sample.

12
Variability

• Variability is large, then sample should be large
• Converse also true

13
2 Barrels of Apples

• Barrel A (low variability) -- all apples about 5
  ins. in diameter (range 5.1 to 4.8 ins.)
• Barrel B (high variability) -- apples range from
  2 to 6 ins. in diameter
• Picking 3 apples from Barrel B might give result
  well below (above) average.

14
Still, Size of Sample Isn't Everything

• Large numbers do not, in and of themselves,
  increase the representativeness of a sample.
• Most professional survey conductors hold that a
  moderate sample size is enough statistically and
  operationally.

15
Representative Sample

• Survey Success of unwed teenage mothers in a
  specific community in raising children?
• To be representative, sample must contain same
  proportion of unwed teenage mothers at --each
  age level --each educational level --each
  socio-economic status in the community

16
Population -- Sample

• It is essential that you select sample in such a
  way that every name on the population list has an
  equal chance of being included in the sample.

17
Random Sample

• Example 500 part-time students in Advancing
  Studies
• Sample of 20 is required
• Assign each student a number from 1 to 500
• Randomly select 100 numbers

18
Systematic Random Sampling

• Example 1.
• 2,000 in population and you want a sample of 200,
  then you might select every 10th name
• Example 2.
• 500 part-time students in Advancing Studies
• Sample of 20 is required --Randomly Select a
  Number from 1 to 5 --Select Every 5th
  Person --002, 007, 012. 017, 022, and up to 497.

19
Possible Problem

• Staff in govt. agency may be listed unit by unit
• Each unit has 9 line-level workers and 1
  supervisor.
• The supervisor is the 10th person on the list.
• Its a survey of 20 -- every 5th person is
  selected.
• If first no. selected is 1, 2, 3, or 4 then no
  supervisor will be selected, though they comprise
  10 of population.
• If first number selected is 5, then supervisors
  will be greatly overrepresented.Thus,
  possibility of bias due to periodicity or
  patterns.

20
Stratified Sampling

• Population 2,000 (800 females 1,200
  males)Sample required 200 If gender is an
  important variable in your survey, then both
  females and males should be included in
  appropriate numbers, that is, in proportions that
  correspond to their presence in the population.

21
Strategy

• Treat both sexes as separate populations and
  take 10 sample from each. ORMake sure
  that all females are listed first and then take
  every tenth name.Either way you will end up
  with 80 females and 120 males

22
Convenience Sampling

• Could ruin an otherwise well-conceived survey.
• Its simple and cheap to select a sample of
  names from a phone-directory to find out which
  candidate people intend to vote for.
• However, this sampling procedure could give
  incorrect results since persons without
  telephones or with unlisted numbers would have no
  chance to be reflected in the sample,
• Their voting preferences might be quite
  different from persons who have listed
  telephones.

23
Confidentiality

• Confidentiality of data supplied by respondents
  is of prime concern to all reputable survey
  organizations.
• Important that individual respondents are not
  identified in reporting survey findings.
• All of the surveys results should be presented
  in totally anonymous summaries, such as
  statistical tables and charts.

24
Problems with Volunteers

• Example
• TV programs asking viewers to vote.
• people call who are most committed to issue.
• stuffing of ballots by multiple calls.
• Time of day is important whos available?

25
Margin of Error

• Error margin of 1,000 randomly chosen individuals
  is said to be 3.1.
• Thus, if a random sample of 1,000 indicates that
  59 will vote for Obama, the actual number could
  range from 55.9 to 62.1.

26
Questions in a Questionnaire Should Be

• Woven together
• Flow smoothly
• Avoid confusion
• A good questionnaire forms an integrated whole.

27
Different Understandings

• Everyone should see/understand the exact same
  question no ambiguity.
• But people from different backgrounds, with
  diverse frames of reference, may have different
  perceptions of the same question.

28
KISS PrincipleKeep It Simple, Statistician

• Questions should be, as far as possible
• Simple
• Clear,
• Easy to answer
• Personally relevant to them
• Often recommended that questionnaires be written
  at the 5th grade reading level

29
Avoid Ambiguity

• Do you favor governmental involvement in health
  care?
• What is your income? __________________
• do you mean weekly monthly annual pre
  tax after tax from salary or from all
  sources

30
Another Example

• (A blouse manufacturer wants to ascertain what
  type of sleeves teenage females prefer in their
  blouses) 1. Do you like short sleeve
  blouses? YES__ NO__2. Do you like long
  sleeve blouses? YES__ NO__3. Do you like
  sleeveless blouses? YES__ NO__

31
Another Example

• Do you jog regularly? Problem, of course,
  lies with REGULARLY

32
Avoid jargon, slang, abbreviations

• Plumbers talk about snakes
• Psychologists about oedipus complex
• Lawyers about mens rea

33
NATO usually means North Atlantic Treaty
Organization

• But some respondents might take it to
  mean --National Auto Tourist Organization
• --Native Alaskan Trade Orbit
• --North African Tea Office

34
Types of Questions

• Open-Ended Questions
• What is your age?
• What is the total turnover in your company?
• Which of the four seasons do you prefer?
• How would you spend a a 1,000,000 lottery win?

35
Disadvantages of Open-ended Questions

• Variation in answers make coding/scoring
  difficult

36
Advantages of Open-ended Questions

• Do not impose researcher's opinion on respondent
• Can lead to a very precise answer

37
Forced-Choice Questions

• How many books do you read each year? Please
  check as appropriate below ___ none
  ___ 1 to 5 ___ 6 to 10 ___ 11 to
  20 ___ more than 20

38

• What do you recall about the frequency of
  snowfall in Boston during the winter of
  1994? ___ It snowed almost every day ___ It
  snowed about once a week ___ It snowed about
  once every two weeks ___ It snowed about once a
  month

39
Importance of Wording

• Take a very simple question
• How many drinks do you have each day? (Check
  one of the following)
• ____ 5 or more ____ 4 ____ 3 ____
  2 ____ 1 ____ none

40
Better to Ask

• Are your daily drinking habits reasonably
  consistent --i.e. do you take about the same
  number of alcoholic drinks each day?
• _____ YES _____ NO (if you mark "NO,"
  skip the following question).

41
Checking Respondents' Consistency

• Q. 4. Check one of the following _x_ I believe
  that manufacturing should be increasingly
  computerized, even if layoffs ensue.
• ___ Preserving the jobs of workers is more
  important than computerizing
  manufacturing.Later on, the Questionnaire might
  ask
• Q. 30. Check one of the following ___ I support
  G.M.'s increasing use of robotics, despite
  the resulting massive layoffs.
• _x_ The government should force G.M. to curtail
  computerization to ensure a reduction in
  layoffs.

42
Avoid Emotional Language

• What do you think about a policy to pay
  murderous terrorists who threaten to steal the
  freedoms of peace-loving people?Problematic
  Words murderous freedoms steal peace

43
Avoid Loaded Questions

• "Should the mayor spend even more tax money
  trying to keep the streets in top shape?"
• "Should the mayor fix the pot-holed and dangerous
  streets in our city?"

44

Arrangement Is Important

• What is your present marital status?
• 1 never married 2 married 3 divorced 4 separated
  5 widowed

45
A Better Arrangement Would Be

• What is your present marital status? (circle
  number) 1. never married 2. married 3. di
  vorced 4. separated 5. widowed

46
Danger of Overlapping

• Check How Many Children are in Your Family
• a. 0-1 children
• b. 1-2 children
• c. 2 or more children

47
Avoid Prestige Bias

• Most doctors say that cigarette smoke causes
  lung disease for those near a smoker. Do you
  agree?
• Do you support the presidents policy regarding
  Zanozui?

48
Avoid Double-Barreled Questions

• "Does your company have pension and health
  insurance benefits?"

49
Ranking Answers (Likert Scale Format)

• Women should automatically receive three months
  maternity leave in your company
• strongly agree partly agree
  disagree strongly
• agree disagree
• ___ ___ ___
  ___ ___

50
How often do you attend church?

• Never Yearly Monthly
  Weekly Daily
• ____ ___ ___ ___
  ___

51
Visual Analog Scales BIPOLAR scales

• I see myself asExtremely Extremely
  interested in uninterested politics
  ______________________________ in politics