Research Methods and Statistics in Psychology Lecture 5: Survey Design

1
Research Methods and Statistics in
PsychologyLecture 5 Survey Design

• Overview of lecture
• 1. What is a survey?
• 2. Differences between surveys and experiments
• 3. Design issues
• a) Sample selection
• b) Sample size
• c) Types of survey
• Reading for this lecture
• Chapter 5 in HM.

2
Research Methods and Statistics in
PsychologyLecture 5 Survey Design

• 1. What is a survey?
• Most of us are very familiar with an opinion
  poll, and just about everyone here will have
  participated in such a survey at some time or
  another to find out who they are going to vote
  for or how well they thought their car had been
  serviced.
• Sometimes this occurs because the person asking
  the questions wants your particular opinions (Did
  we fix the engine rattle?), but more often they
  want to use your opinions to estimate those of
  others (e.g., to find out whos going to win the
  election).

3
Research Methods and Statistics in
PsychologyLecture 5 Survey Design

• 1. What is a survey?
• Its useful though, to use a more precise
  definition of surveys that distinguishes them
  from experiments. The definition rests on the
  idea (discussed in Lecture 3) that in any piece
  of research you have to measure variables but you
  dont necessarily have to manipulate them.
• If you measure dependent variables and manipulate
  independent variables you are doing an
  experiment, if you measure both then you are
  doing a survey.
• You may ask How can you have independent
  variables that you dont manipulate?
• This is a good question. The answer is that in
  experiments IVs are causal, in surveys they are
  variables believed to be causal.

4
Research Methods and Statistics in
PsychologyLecture 5 Survey Design

• 2. Differences between surveys and experiments
• One commonly-encountered, but mistaken, belief is
  that the difference between surveys and
  experiments is a question of location, with
  surveys being conducted in the community and
  experiments in the laboratory. This is often
  true, but it is not always.
• The main differences between experiments and
  surveys relate to the sorts of questions that
  surveys and experiments can answer.
• As we discussed in Lecture 3, experiments tend to
  be concerned with establishing causal
  relationships between variables and they achieve
  this by randomly assigning participants to
  different treatment conditions.
• In contrast, surveys tend to be concerned with
  measuring naturally-occurring and enduring
  relationships between variables.

5
Research Methods and Statistics in
PsychologyLecture 5 Survey Design

• 2. Differences between surveys and experiments
• For this reason surveys tend to be concerned more
  with description than with explanation.
• As part of this description process, researchers
  usually want to generalize from sample data to a
  population using the sample to estimate the
  characteristics of their population of interest.
  
• Given the emphasis that we have previously placed
  on explanation, you may well ask, Why dont
  researchers do experiments all the time?
• Well sometimes scientists are only interested in
  observing relationships and sometimes
  manipulations simply arent possible.

6
Research Methods and Statistics in
PsychologyLecture 5 Survey Design

• 2. Differences between surveys and experiments
• These issues apply in all sciences. Astronomers
  or geologists rarely do experiments, simply
  because it is often impossible to manipulate the
  IVs of interest (e.g., the position of certain
  stars).
• Instead they rely largely on the same logic of
  controlled observation that underpins
  psychological surveys. Clearly, this does not
  mean that astronomy or geology are unscientific
  or pointless.
• As well as this, surveys can allow researchers to
  eliminate some causal links. For example, if
  there is a relationship between age and
  intelligence, it is impossible for intelligence
  to cause age.
• Similarly, if there is no relationship between
  variables this allows us to conclude that one
  does not cause the other (at least in this survey
  environment noting that a relationship may be
  concealed by a third, or background, variable).

7
Research Methods and Statistics in
PsychologyLecture 5 Survey Design

• 3. Design issues
• a) Sample selection
• For purposes of generalization, survey
  researchers need to ensure that their sample is a
  representative random sample of the population.
  That is, the survey must obtain data from a
  random sample of the population that has the same
  characteristics as the population.
• If the sample isnt representative in this way,
  this can cause serious problems.

8
Research Methods and Statistics in
PsychologyLecture 5 Survey Design

• 3. Design issues
• a) Sample selection
• A classic case is the Literary Digest survey of
  voter preference for the 1936 US election. This
  magazine conducted a massive survey contacting
  every US voter who was listed in the telephone
  directory or who was registered as owning a car.
  
• They obtained over 2 million responses and
  concluded that the Republican candidate Landon
  would win by a landslide. In fact though,
  President Roosevelt was re-elected.
• Why did they fail? Possibly because they limited
  their sample to the wealthier segments of the
  population who owned cars and telephones.

9
Research Methods and Statistics in
PsychologyLecture 5 Survey Design

• 3. Design issues
• a) Sample selection
• However, we noted in Lecture 4 that for many
  psychological experiments the population in which
  researchers are interested is all people who
  display the psychological process they are
  investigating.
• For example, with research into visual perception
  the population of interest is usually people with
  normal vision. So, for research in this area, no
  matter how narrowly people are chosen, they
  should be representative of this population.
• Similar sampling considerations are true in many
  other areas of psychology (which is why many use
  first-year ? students). This is called
  convenience sampling.

10
Research Methods and Statistics in
PsychologyLecture 5 Survey Design

• 3. Design issues
• a) Sample selection
• The alternatives to convenience sampling are
  systematic sampling methods which can be divided
  into probability sampling and non-probability
  sampling techniques.
• Probability sampling involves drawing people from
  the population so that any member of the
  population has a specifiable probability of being
  sampled.
• This sounds complex but all it means is that when
  we select a probability sample we have to know
  what each population members chance of being
  included in the sample is.
• In simple random sampling (a special case of
  probability sampling) every individual has
  exactly the same probability of being sampled.

11
Research Methods and Statistics in
PsychologyLecture 5 Survey Design

• 3. Design issues
• a) Sample selection
• Simple random sampling involves getting a
  complete listing of the population of interest,
  whether that be a class list, the voter roll, a
  telephone directory or something else. This
  listing is called a sampling frame.
• For the Literary Digest survey the telephone
  directory and the list of car registrants was the
  sampling frame (here, though, the sampling frame
  ? population of interest).
• To draw a simple random sample from the
  population researchers attach a number to each
  person in the listing and if that number is
  chosen by a random process that person is
  included in the sample. This produces a
  representative random sample of the population.

12
Research Methods and Statistics in
PsychologyLecture 5 Survey Design

• 3. Design issues
• a) Sample selection
• Non-probability sampling includes all techniques
  where there is not an identifiable probability of
  each member of the population being included in
  the population.
• Convenience sampling is one form of
  non-probability sampling, another is purposive
  sampling. Purposive sampling involves obtaining
  a sample who all have a particular
  characteristic.
• For example, researchers who are interested in
  the behaviour of singletons (only children) might
  take a random sample of the whole population and
  then exclude households where there were no
  singletons. But it might be more sensible just
  to go out and try to find a group of only
  children.

13
Research Methods and Statistics in
PsychologyLecture 5 Survey Design

• 3. Design issues
• b) Sample size
• How big should a sample be?
• If the researchers eliminate systematic bias from
  the sample, then the bigger the sample is, the
  better it will reflect the population and so the
  better it will be for the research.
• This is because a larger sample size reduces
  uncertainty about the inferences drawn from
  sample data.
• However, if they dont eliminate systematic bias
  they will just make incorrect inferences more
  confidently (as in the Literary Digest case).

14
Research Methods and Statistics in
PsychologyLecture 5 Survey Design

• 3. Design issues
• b) Sample size
• The size of the chosen sample will be reduced by
  non-response. This occurs, for example, where
  people dont feel like participating in, or
  forget to participate in the research, or their
  responses are lost in the mail.
• Two particular problems here are mortality and
  reactivity (see Lecture 4). If particular people
  (e.g., lazy ones, or people who are offended by
  the survey) dont respond this can bias the
  sample.

15
Research Methods and Statistics in
PsychologyLecture 5 Survey Design

• 3. Design issues
• c) Types of surveys
• So far we have talked about survey design in very
  general terms. In fact there are many different
  survey methods and the wide choice reflects the
  many different interests, approaches and
  objectives of psychologists.
• We cannot, for example, expect that the same
  techniques would be applicable to studying adults
  and children, or for answering questions in
  cognitive and clinical psychology.
• It is difficult to say that one method is always
  better than another, though some techniques are
  certainly better than others for particular
  purposes.

16
Research Methods and Statistics in
PsychologyLecture 5 Survey Design

• 3. Design issues
• c) Types of surveys
• A number of options exist here (for more detail
  see HM Chapter 5). These include interviews,
  computer-aided interviewing and naturalistic
  observational studies.
• One problem that can arise in all these settings
  is that people respond in ways they think they
  should, rather than as they normally would (again
  this is the problem of reactivity).
• In this way, data can be biased by a concern to
  provide socially desirable responses. This
  happens, for example, in self-reports of TV
  viewing.

17
Research Methods and Statistics in
PsychologyLecture 5 Survey Design

• 3. Design issues
• c) Types of surveys
• One good way around this problem is to use
  non-obtrusive measures. These are research
  procedures which ensure that the participants are
  not aware that they are being involved in the
  research process (these methods are also called
  non-reactive techniques).
• The logic of such research is that people cannot
  change their behaviour in response to being
  observed if they do not actually know they are
  being observed.

18
Research Methods and Statistics in
PsychologyLecture 5 Survey Design

• 3. Design issues
• c) Types of surveys
• One way of doing this is to use archival records
  or behavioural trace measures. Studies that use
  these techniques check through records of
  behaviour (e.g., internet site use, sales of a
  product or the contents of peoples rubbish bins
  "garbology").
• These are non-reactive because the behaviour has
  occurred in the past so the participants cannot
  change it as a reaction to being observed.

19
Research Methods and Statistics in
PsychologyLecture 5 Survey Design

• 3. Design issues
• c) Types of surveys
• This lecture can only really scratch the surface
  of the strategies and techniques available to
  researchers who want to use surveys.
• Hopefully though, it serves to make it clear that
  there are a large number of options here that can
  be tailored to the circumstances the researcher
  faces. This makes survey design much more of an
  art than is commonly supposed, not least because
  there are so many pitfalls for the under-informed