PPA 502 Program Evaluation

1
PPA 502 Program Evaluation

• Lecture 5a Survey research

2
Introduction

• Government administrators and elected officials
  love to claim that they possess a profound
  understanding of their publics needs, desires,
  and disaffection.

3
Introduction

• Unfortunately, the administrators and officials
  are learning that storms of controversy provide
  meager evidence of the workaday values of the
  everyday people they govern.
• Surveys of the public, conducted following the
  basic precepts of survey design and analysis, are
  fast becoming the vehicle for genuine connection
  to the public will.

4
Introduction

• Uses.
• Evaluations of government services.
• Changing demographics that may signal shifts in
  service demand.
• Patterns of service utilization.
• Problem identification.
• Customer service.

5
Introduction

• Surveys have several important qualities.
• Anonymity to respondents.
• Point of view, characteristics, or use patterns
  can be characterized with little confusion.
• Good surveys provide input from a representative
  cross-section.

6
Begin Before the Beginning

• The best surveys grow from well-conceived and
  well-articulated reasons for conducting them.
• Resist the temptation to hit the ground running.
• Be certain of the purposes of the survey.

7
Begin Before the Beginning

• Identify the appropriate audiences.
• Identify the political and personal will for
  doing the survey.
• Determine whether the questionnaire to be
  developed is better as a one-time or periodic
  survey.
• Think about the usefulness of comparative data.

8
Getting Started

• Convene a steering committee with key
  stakeholders.
• Enlist the help of top government officials or
  administrators.

9
Designing the Survey

• Sampling.
• Choose the appropriate sampling frame about
  what population do you wish to generalize?
• A sampling plan must give every respondent in the
  sampling universe an equal chance of ending up in
  the sample.
• Simple random sample.
• Stratified sampling.
• Stratified random cluster sampling.

10
Designing the Survey

• Targeting the individual in the household.
• If no list exists, you may only have addresses or
  phone numbers. If so, use household member with
  most recent birthday.

11
Mail, Phone, or In-person Interviews

• The best ways to conduct surveys vary by
  accuracy, speed, and cost.
• Most common are mail and phone surveys.

12
Mail, Phone, or In-person Interviews
13
Mail, Phone, or In-person Interviews

• Increasing response rates.
• Multiple mailings (up to three) with stamped,
  return address envelope.
• Press coverage.
• Combination of methods often best Mail survey
  with telephone and in-person followup.

14
Mail, Phone, or In-person Interviews

• Selecting sample size.
• The size of sample depends on desired precision
  of estimates.
• Generally speaking, if opinions are split as much
  as possible, than 100 residents will have a
  margin of error of /- 10 with 95 percent
  confidence. Four hundred residents the margin is
  /- 5.
• In general, 100 is a good minimum number,
  especially for subgroups.

15
Questionnaire Construction

• Each question should be judged against the
  purposes of the survey and the uses to which it
  will be put.
• Steal widely.
• National Citizen Survey from International City
  Management Association and National Research
  Center.

16
Questionnaire Construction

• Major principles.
• Consistency.
• Clarity.
• Vague wording.
• Double-barreled questions.
• Assumed knowledge.
• Overlapping response categories.
• Simplicity.
• Specificity.
• Brevity (30 min. Phone, 60 min. In-person, 10
  page mail).
• Context sensitivity.

17
Questionnaire Construction

• Major principles.
• Security.
• Demographic at end.
• General to specific.
• Fairness.
• Option symmetry (balanced responses).
• Option wording and order.
• Background info, pros and cons, opinion.
• Randomize pros and cons in a complicated survey.

18
Conducting the Survey

• The survey steering committee.
• Double check questionnaire with steering
  committee.
• Frequency of surveys.
• For most multipurpose surveys, no more than once
  per year.
• Pretest.
• Test on twenty people at random. Ask questions
  about format and clarity.

19
Conducting the Survey

• Training.
• Survey assistants must be trained. All must
  operate uniformly, asking the questions in the
  same way, coding in the same way.
• Consistent open-ended coding.
• 10 recontact of survey respondents.
• Trying hard and keeping track.
• Three contacts by telephone for each number.
• Warning and at least two mailings for mail
  surveys.

20
Reporting Results

• Data analysis.
• For most government surveys, percentages, average
  responses, simple cross-classifications.
• The most complicated analysis will be to get
  accurate population estimates weighting.
• Report writing and presentation.
• Executive summary.
• Bulleted lists.
• Document survey methods in appendix.
• Augment tables with bar and pie charts.
• Powerpoint for in person presentation.

21
Hiring a Consultant

• Previous experience.
• Ability to communicate findings.
• Share work with in-house staff.
• Intuition.