Quality and purpose: quality in quantitative methods

1
Quality and purpose quality in quantitative
methods

• Angela Dale
• University of Manchester

2
Quality in survey methods

• A survey is of adequate quality if it can answer
  specified question with an adequate degree of
  accuracy
• it is fit for purpose
• If it can provide an answer at a 1 level of
  confidence when only 5 is required, then it may
  not be cost effective

3
Standards for surveys

• Random sample best bet of avoiding bias in
  sample but need adequate sampling fame
• Adequate numbers for required subgroups depends
  on level of accuracy needed
• Good response rate but extent of bias is
  crucial
• Questionnaire must be well developed
• well designed questions good flow salient

4
Can a scoring system work?

• A scoring system can indicate overall strength on
  all these dimensions, but
• Higher quality costs more
• Increased spend may have diminishing returns
• But there will be a level below which a survey
  would have very low value/low credibility
• Need to find a way of assessing quality against
  requirement

5
Analysis issues

• Results based on analysis need to reflect the
  quality of the survey
• Importance of making clear the level of accuracy
  of the results
• The quality of the analysis is also of great
  importance
• An excellent survey may be badly analysed ,
• methods may be used poorly interpretations
  incorrect
• inappropriate assumptions about causality may be
  made
• Claims may go beyond what the data can support

6
Can a poor survey have value?

• A very poor survey may still have value
• if analysed with care and weaknesses recognised
• If only very limited conclusions, that can be
  justified, are drawn
• Where there is no better alternative

7
Conclusions

• It is of value to have a reference framework for
  assessing quality
• But equally important to use it critically and
  with care
• The key to quality lies not just in good data but
  in ensuring that claims made can be supported by
  the data