Designing Web Surveys Dr Lisa Wise

1
Designing Web Surveys Dr Lisa Wise
2
Overview

• web surveys
• overview of survey design
• when to use surveys
• how to design survey questions
• data analysis and reporting
• data be collection, storage and analysis
• ethical issues to do with handling sensitive or
  personal data

3
Web-based Surveys

• can do from anywhere with web access
• but samples are biased against those with no
  internet access (gt 50 of Aust households are
  connected - AC Nielson Mar 2002)
• data can be validated before submission
• note that validation is in terms of correct data
  type rather than meaningfulness
• issue of multiple submissions vs anonymity
• quick data collection phase for researcher

4
Web Survey
survey form
processing scripts
database
web server
filesystem
Survey data might go to a database or might be
stored in a file in the file system or could be
emailed to the researcher
email
5
Web survey design

• use standard web design principles
• especially ensure that
• survey is not too long (or can be saved)
• information travels correctly between screens
• error messages, if they occur, are meaningful
• all fields are validated and can handle
  appropriate input and reject inappropriate input
• check field length, data type, illegal characters

6
Making a web survey

• put the questions into an HTML form
• get the data formatted for analysis using the
  scripting tools of your choice
• set up access controls if required
• test across a range of browsers with a range of
  good and bad input
• ensure security / integrity of your data, but
  also of the system collecting your data

7
Data collection and storage

• use cgi-mailer or FormMail to email formatted
  form variables to researcher
• http//www.its.monash.edu.au/web/resources/cgi-mai
  ler.html
• http//www.its.monash.edu.au/web/resources/formmai
  l.html
• use a script (eg Perl, PHP) to write to a file on
  the filesystem
• use a script to put data into a database
• dont put into a database if you are then going
  to take it out into SPSS / excel to analyse it !!

8
Privacy

• Privacy laws impose quite specific requirements
  regarding data storage and security, access and
  correction rights, ensuring data is accurate
  before use, used only for the purpose for which
  it was collected or for which consent has been
  given, and disclosure only in limited
  circumstances.
• Legal requirements may be more detailed than
  ethical requirements
• http//www.monash.edu.au/resgrant/human-ethics/pri
  vacy/index.html

9
Handling data

• shared responsibility of technical staff and
  researchers to ensure that privacy and
  confidentiality requirements are met
• information is only de-identified only if all
  identifying information has been irreversibly
  removed from the record
• retention of codes which allow recovery of
  identifiable information means record is not
  de-identified

10
Cost effectiveness

• even a simple web survey takes a couple of days
  to implement and test
• cost-effectiveness of solution needs to include
  real costs
• eg email vs database solution might involve 3
  days of coding for programmer versus 1 day of
  data entry (cutting and pasting from email to
  analysis program) for researcher
• cost saving or cost shifting?

11
Making a survey ...

• technical design requirements covered in
  previous slides but what about designing the
  actual survey questions ?
• there is a whole discipline area focused on
  designing surveys and questionnaires and
  analysing data collected using these tools
• it is not a trivial exercise and committees /
  managers are not ideal survey designers

12
Overview of survey design

• survey design requires that you have clear
  research questions
• survey questions need to be focused on answering
  your research questions
• survey design includes generating the questions
  and planning the data analysis
• planning the data analysis happens before the
  survey is released !!

13
When to use surveys

• surveys allow you collect lots of data relatively
  quickly and cheaply
• lots of poor quality data is never better than
  smaller amounts of high quality data
• lack of time and money are not excuses for
  collecting poor quality data - if you cant
  afford to collect reasonable quality data, dont
  do the research.

14
Personal data

• most surveys request demographic information
  about respondents
• usually ask for opinions about something
• collecting either type of information has ethical
  and privacy implications
• survey designers should be familiar with
• http//www.monash.edu.au/resgrant/human-ethics/
• above page has link to Use of Personal Information

15
from Privacy statement on web

• On-line Surveys
• All research surveys conducted on-line by
  University staff and /or students which involve
  the collection of personal information, will have
  received approval from the University's Committee
  for Human Ethics in Research. A survey might
  ask visitors for unique identifiers (such as
  login information).

16
Sensitive information

• Do the records or information you are collecting
  or using include any sensitive information (such
  as political opinion or memberships, religious
  beliefs or affiliation, philosophical beliefs
  )
• note that peoples opinions are considered to be
  personal information
• be aware of perceived power relationships and
  potential access to confidential information

17
Classroom and workplace surveys

• work / class surveys have ethical issues related
  to perceived power relationships between
  respondents and researchers
• even if survey does not require specific ethics
  committee approval, the ethical and privacy
  principles should be considered
• go through all the ethics forms whether or not
  they need to be submitted

18
SCERH ethics forms

• based on National Statement on Ethical Conduct in
  Research Involving Humans (NHMRC, 1999)
• apply to anyone who is gathering information
  about human beings and organisations through
  interviewing, surveying, administering
  questionnaires, observing human behaviour, taking
  human tissue / fluids
• there are no exceptions, exclusions or
  blanket permissions

19
Sampling

• Sampling design and survey design must be tightly
  coupled
• Who or what are you planning to draw conclusions
  about ?
• Are they a homogenous group or are there
  sub-groupings in the population ?
• do you need a stratified sample?
• do you want to compare between groups?

20
Sampling

• Surveys usually use convenience samples
• Demographic information is collected to allow
  comparisons between target groups
• If targeting specific groups is critical to your
  research, consider interviewing participants, or
  delivering and collecting surveys from selected
  participants
• convenience sample / random sample

21
Sampling

• for statistical techniques, calculate the sample
  size required for valid conclusions
• consider whether you want lots of general data or
  whether you are actually interested in very
  specific focussed data
• a large survey doesnt give more objective data
  than eg a focus group unless it follow rigorous
  methodology

22
Types of questions

• Open ended
• more difficult to answer and to code or to
  analyse objectively
• Closed questions
• forced choice (one of two mutually exclusive)
• multiple choice (one of several)
• checklist (one or more of several)
• partially closed (alternatives including Other)

23
Questions

• avoid double-barrelled questions
• avoid leading questions
• avoid motherhood statements
• avoid undefined terms
• ensure that your questions lead to responses that
  interest you and conversely that responses that
  interest you are elicited by your questions

24
Order effects

• funnel questions from general to specific
• can use general filter questions to determine
  whether respondent should be asked detailed
  questions
• can have some practice questions
• counterbalance order of presentation
• prevent response sets
• can use alternate forms of questions

25
Content analysis

• talk to expert about content analysis
• consider whether you are happier with the
  assumptions underlying content analysis or your
  research teams ability to interpret and code
  responses
• if you code responses, take measures of inter-
  and intra-rater reliability
• do raters make similar / consistent judgements

26
Response scales

• response scales should allow people to
  communicate what they want
• eg there are differences between neutral,
  undecided, dont know, dont care, n.a.
• anchor points should be bipolar
• need to test this on pilot sample

boring
interesting
fun
boring
27
Measurement scales

• Ratio scales (true zero - eg age, height)
• Equal Interval scales
• Thurstone scales, Likert scales, Guttman scales,
  semantic differentials
• Ordered scale (eg lt18, 18-30, 31-50, gt50)
• Categories (ITS, Med, Sci, Arts )
• Different types of data are amenable to different
  analyses consult a statistician!

Albrecht et al, Social Psychology, pp 190-198
28
Rating scales
Likert scales should have 5 marked values not 7
strongly disagree
strongly agree
undecided
disagree
agree
The following multiple choice format is still an
interval scale 1. Strongly disagree 2.
Disagree 3. Undecided 4. Agree 5. Strongly agree
29
Rating scales

• constructing rating scales to have specific
  psychometric properties is labour-intensive and
  requires expertise
• multiple forms or minimal number of questions?
• group similar questions or intermix?
• alter format to avoid response bias or keep
  consistent to avoid response errors?
• normalise data or trust participants responses?
• consult a statistician or social scientist

30
Measurement

• rating scales should be equal-interval scales to
  use parametric statistics
• the fact that you have numerical data does not
  mean it is accurate or reliable
• male (1) - female (0) obvious that 0 and 1 are
  codes not numbers
• rarely (1) - sometimes (2) - often (3) things
  that can be ranked are not necessarily equal
  interval scales - can you take the mean?

31
Statistics

• Descriptive
• describe a sample distribution in terms of shape,
  central tendency and variance
• Inferential
• draw inferences about population parameters based
  on sample statistics
• hypothesis testing - test whether a statement is
  true or false based on sample statistics

32
Reliability and validity

• Reliability
• test - retest
• surveys are only a snapshot at particular time
• Validity
• face validity
• criterion-based validity
• construct validity
• internal / external validity

33
Hypothesis testing

• null hypothesis H0 (no effect of experimental
  manipulation)
• alternative hypothesis H1 (experimental
  manipulation has an effect on this test
  statistic)
• type 1 error (accept H1 when H0 true)
• type 2 error (accept H0 when H1 true)

34
Inferential statistics

• use statistics to make inferences about
  populations from your samples
• need to be aware of assumptions underlying
  statistical tests
• eg t-tests and anovas assume continuous
  underlying variable, normal distribution and
  homogeneity of variance
• what happens when assumptions are violated?

35
Statistics

• most survey data is not ratio and may not be
  interval data (depending on your perspective on
  this)
• non-parametric statistical tests dont make
  assumptions about underlying distribution
• dont use statistics to show things you cant see
  by eye - statistics help decide if what you can
  see is significant

36
Correlational studies

• surveys usually correlate opinions with
  demographic information
• correlations show relationships between variables
  but dont address causality
• correlation coefficients indicate strength of
  relationship (between 0 and 1)
• r of .8 explains 64 of variance, or the degree
  to which knowing X can predict the value of Y

37
But Im not doing real research ...

• I just want to run a quick survey
• This stuff doesnt apply to me cos Im not doing
  real research
• . . . so what are you doing?
• anyone who is gathering information about human
  beings and organisations through interviewing,
  surveying, administering questionnaires,
  observing human behaviour, no exceptions ...

38
What constitutes research?

• If you are going to summarise responses on your
  survey and you are going to act on them in any
  way, you are doing research
• for that research to be meaningful, you need be
  aware of proper research methodology and the
  limitations of what you are doing
• if you cant do it properly, dont do it at all
  !!
• informed professional opinion can far more
  valuable than poor quality survey data

39
When to use web surveys

• small amount of non-confidential data from wide
  range of people
• not good for sensitive or confidential data
• not good where loss of data would be a major
  problem
• not good for long surveys (usability issues)
• can use web to download a survey which is printed
  out and submitted