How Research Methods Textbooks Mislead Psychologists About Scientific Method

1
How Research Methods Textbooks Mislead
Psychologists About Scientific Method

• Brian Haig
• University of Canterbury

5 April 2013
2
Introduction

• 1. Textbooks are the major source of formal
  learning in undergraduate education, and they are
  prominent in graduate education as well.
• 2. In psychology, students take more methods
  courses than any other type of course (e.g.,
  University of Canterbury Psyc206 Psyc344
  Psyc460/Psyc461 are all compulsory for graduate
  research students).
• 3. All undergraduate students take a generic
  second- or third-year course on research methods.
• 4. There are dozens of textbooks available for
  adoption in such courses.

3

• 4a A Sample of Textbooks
• Gravetter, F. J., Forzano, L-A. B. (2009).
  Research methods for the behavioral sciences (3rd
  ed.). Belmont Wadsworth.
• Leary, M. R. (2012). Introduction to behavioral
  research methods (6th ed.). Boston Allyn
  Bacon.
• Mitchell. M. L., Jolley, J. M. (2013). Research
  design explained (8th ed.). Belmont Wadsworth.
• White, T. L., McBurney. D. H (2013). Research
  methods (9th ed.). Belmont Wadsworth.

4

• 5. Thomas Kuhn (1970) argued that textbooks in
  the various sciences contribute in a major way to
  the dogmatic initiation of learners into
  established research traditions.
• 6. Psychologys research methods textbooks are
  largely uncritical in the presentation of their
  subject matter.
• 7. It follows that they cannot be a source for a
  genuine education in research methods.
• 8. I identify a number of deficiencies in the way
  psychologys research methods textbooks deal with
  scientific method.
• 9. Method is central to science, and scientific
  method is, therefore, important. Most of what we
  know in science is acquired through application
  of its methods.

5
The Typical Research Methods Textbook

• 1. The standard textbook possesses a common
  structure (psychology as a science, research
  questions, ethics, writing, validity,
  experimental and nonexperimental research, data
  exploration and inference, .).
• 2. The first chapter (approx. 35 pages) is often
  dedicated to scientific method, broadly
  understood.
• 3. Cameron Ellis and I did a content analysis of
  a representative sample of 16 current
  undergraduate research methods textbooks (Haig
  Ellis, 2013).
• 4. All textbooks have been through multiple
  editions (range2-11 median5.5 latest
  edition10/16).

6
A Bill of Indictment

• 1. Most textbook authors are not methodologists
  by speciality (only 6 of 28 in the sample, with
  methodologist construed generously).
• 2. The most serious limitation of textbooks on
  research methods is that the methods dealt with
  are not informed by their accompanying
  methodology (Methodology is the interdisciplinary
  field that studies methods. It includes
  statistics, philosophy of science, and cognitive
  science).
• This results in an impoverished understanding of
  the methods.

7

• 3. Rarely are methodological references on
  scientific method provided.
• i) few citations of professional
  methodologists and
• their work (including philosophers of
  science)
• ii) heavy reliance on secondary sources
  (when citations
• provided). Example Stanovich (1998),
  How to think
• straight about psychology (5th ed.)
• iii) methodological points about scientific
  method often
• made by appealing to substantive
  research
• (particularly in social psychology),
  which is
• referenced much more than methodological
  
• research.

8

• 4. The changes in successive editions of books
  are often
• minimal.
• a) revisions are driven more by attempts to
  capture,
• or retain, market share than present new
• methodological developments.
• b) publishers encourage authors to produce
  new
• editions regularly in order to encourage
  sales of new
• books (the problem of second-hand book
  sales).

9
Theories of Scientific Method Ignored

• An Argument
• P1 Credentialed theories are the major vehicles
  of
• scientific knowledge.
• P2 This holds for method as well as matter.
• P3 Major theories of scientific method are not
• presented in textbooks.
• Cl Texts books do not present genuine knowledge
  about
• scientific method.
• This state of affairs is both embarrassing and
  unacceptable.

10
Major Theories of Scientific Method

• 1. Inductive Method
• - 10/16 texts mention induction, but
• - Most characterize inductive inference without
  relating it to inductive method (e.g., radical
  behaviorist method, Bayesian confirmation
  theory).
• - Induction characterized as enumerative (or
  generalizing) induction. But there are different
  types of induction (e.g., enumerative,
  eliminative, probabilistic). These are not
  mentioned or distinguished in any of the 16
  texts.
• - Inductive nature of constructive replication
  not explicitly stated in any text.

11

• 2. Hypothetico-Deductive Method
• - The most popular account of scientific method
• - Claims a hypothesis or theory is evaluated in
  terms of its test predictions
• - 9/16 texts neither mention nor describe it.
• - 5/16 vaguely describe it without naming it.
• - 2/16 name it and describe it.
• - 1/16 says it is controversial.
• - No text talks about its limitations or its
  proper scope of application.

12

• 3. Inference to the Best Explanation
• - Concerned with explanatory reasoning. In brief,
  it captures the idea that theories are evaluated
  according to their explanatory worth.
• - Frequently used in the evaluation of scientific
  theories (e.g., Darwins theory of natural
  selection)
• - Not mentioned or referenced in any of the 16
  texts
• - Methods of inference to the best explanation
  are available (e.g., the theory of explanatory
  coherence).
• - Structural equation modeling that combines
  model fit with model parsimony can be regarded as
  a hybrid account of inference to the best
  explanation.

13

• 4. Bayesian Method
• - Widely regarded as the leading theory of
  scientific confirmation in contemporary
  philosophy of science
• - Not mentioned, referred to, or indexed in any
  of the 16 textbooks (as Thomas Bayes, Bayes
  theorem, Bayes factors, Bayesian statistics,
  or Bayesianism)
• - Note that Bayesian statistics is beginning to
  be promoted by a small group of behavioral
  science methodologists.
• Conclusion One cannot learn much of worth about
  scientific method from methods textbooks that
  claim to deal with scientific method!
  (Additionally, one can learn a good number of
  things that are not so).

14
Psychologists on Scientific Method

• A few psychologists have written about scientific
  method in a serious vein
• Cattell, R. B. (1966). Psychological theory and
  scientific method. In R. B. Cattell (Ed.),
  Handbook of multivariate experimental psychology
  (pp. 118). Chicago, IL Rand McNally.
• Haig, B. D. (2005a). An abductive theory of
  scientific method. Psychological Methods, 10,
  371-388.
• ODonohue, W., Buchanan, J. A. (2001). The
  weaknesses of strong inference. Behavior and
  Philosophy, 29, 1-20.
• Rozeboom, W. W. (1997). Good science is abuctive,
  not hypothetico-deductive. In L. L. Harlow, S. A.
  Mulaik, J. H. Steiger (Eds.), What if there
  were no significance tests? (pp. 335-391).
  Hillsdale, NJ Erlbaum.
• Skinner, B. F. (1956). A case history in
  scientific method. American Psychologist, 11,
  221-233.
• None of these references are cited in any of the
  16 texts.

15
What Do Methods Textbooks Say About Scientific
Method?

• 1. Scientific Method is the Scientific Method
• - 11/16 texts spoke of the scientific method
• (with no agreement on what it is!).
• - 3/16 spoke about a scientific method.
• - 2/16 didnt mention scientific method at
  all.
• There is no such thing as the scientific method.
• Instead, we have a plurality of scientific
  methods.

16

• 2. Scientific Method is a Series of Steps
• - 9/16 texts described scientific inquiry as
  a series of
• steps (as few as 4 steps, as many as
  11)
• - the steps listed vary greatly
• Example 1 Gravetter Forzano, Research
  methods for
• the behavioral sciences (2009).
• Step 1 Observe behavior or other
  phenomena
• Step 2 Formulate tentative explanatory
  hypothesis
• Step 3 Use the hypothesis to make a
  testable prediction
• Step 4 Evaluate the prediction via
  planned observations
• Step 5 Support, refute, or refine the
  original hypothesis
• and so on in a circle or spiral

17

• Example 2
• HOMER An acronym for the five steps of the
  scientific method
• Each letter of Homer stands for one of the five
  steps in the scientific method hypothesize,
  operationalize, measure, evaluate, and
  replicate/revise/report.
• (Lakin, et al., 2007)
• The propensity to characterize scientific method
  as a series of fixed steps is probably due to
  John Deweys (1910) five-step account of
  scientific method, which had an enormous
  influence on American education (Rudolph, 2005).
  Dewey later revised his formulation to do away
  with fixity of order and spoke of phases, not
  steps.

18

• 3. Scientific Method is a Process of
  Falsification
• a) Falsification is the process of showing a
  claim to be
• false.
• b) 13/16 texts mention falsification.
• c) All texts fail to distinguish between the
  process of
• falsification and Poppers theory of
  falsificationism
• with its attendant ideas of conjectures
  and
• refutations, corroboration, and
  demarcation.
• d) 3/13 texts mention that
  falsification(ism) is
• controversial, but none of them deal
  with the well-
• known criticisms of it in the philosophy
  of science.

19
Phenomena Detection and Theory Construction

• 1. The methodological distinction between
  phenomena detection (e.g., the discovery of
  empirical regularities) and theory construction
  (the generation, development, and appraisal of
  explanatory theories) is of fundamental
  importance to science.
• 2. These are two quite different sorts of
  undertaking and they employ different research
  methods (Haig, 2013).
• 3. 1/16 texts explicitly draws the distinction
  (Leary, 2012).
• 7/16 almost draw it (e.g., law/theory
  distinction).
• 8/16 fail to draw the distinction.
• 4. No text systematically discusses different
  research methods in relation to the distinction.

20

• 5. At best, textbook treatments of theory
  construction focus on hypothetico-deductive
  theory testing for empirical adequacy.
• 6. Methods specifically tailored to theory
  generation (e.g., exploratory factor analysis),
  theory development (e.g., analogical modeling),
  and theory appraisal (e.g., inference to the best
  explanation) all deserve a proper place in
  research methods textbooks (Haig, 2005).
• 7. Methods textbooks in psychology emphasize data
  analysis at the expense of theory construction.

21
The Neglect of Philosophy of Science

• The philosophy of science is an important part of
  scientific methodology.
• It is a major resource for leaning about the
  conduct of scientific research.
• Yet, the 16 texts make little or no use of it
  (3/16 mention it none use it).
• Popper (1959) and Kuhn (1962), are sometimes
  mentioned Lakatos (1970) and Laudan (1977) are
  occasionally mentioned. None of them are used.
  The last 25 years of developments in the
  philosophy of science have been completely
  ignored (see Blachowicz, 2009).

22

• As a consequence of this general neglect, these
  texts have failed to capture the profound changes
  in our understanding of science that philosophy
  of science has brought about in the last 50 years
  (e.g., Proctor Capaldi, 2001).
• Early editions of the McGuigan and Leary texts
  stressed the importance of philosophy of science
  in understanding research methods.
• An understanding of the philosophy of
  science is important to an understanding of what
  science is, how the scientific method is used,
  and particularly of where experimentation fits
  into the more general framework of scientific
  methodology. (McGuigan, 1960, p. 111)

23

• Interestingly, both McGuigan and Leary dropped
  reference to philosophy of science in later
  editions (publishers pressure to conform?).
• In his section on philosophy of science, Leary
  (2001) claimed
• 1. Many philosophers of science are behavioral
  scientists.
• 2. Many researchers take courses in the
  philosophy of science.
• I think both claims are false. It would be good
  for both disciplines if they were true.

24
Concluding Thoughts

• 1. Textbooks should present inductive,
  hypothetico-deductive, and Bayesian methods, as
  well as inference to the best explanation, as
  major local theories of scientific method with
  domain specific applications.
• 2. Knowledge of these scientific methods would
  help learners better understand the process of
  phenomena detection, the complexities of theory
  testing, and the nature of explanatory reasoning
  in science.
• 3. The dominant twofer model (Aiken, West,
  Millsap, 2008) (where researchers with expertise
  in a substantive area of psychology, teach
  statistics and research methods courses) extends
  to writing research methods textbooks.

25

• This two-tiered twofer model poses serious
  problems for psychology because it contributes to
  a decrease in the quality of instruction about
  scientific method generally (and research methods
  more specifically).
• 4. The content of these four major theories of
  method cannot be taught without making
  substantial use of the relevant philosophy of
  science literature (this use of philosophy of
  science should extend to instruction about
  methodological processes more generally).
• 5. Textbooks on research methods should be
  written by professional research methodologists,
  thus ensuring that they are appropriately
  methodologically informed.
• 6. These texts should deal with the full range of
  behavioral research methods and strategies.

26

• 7. Psychology needs just a few up-to-date general
  research methods textbook (say, 5, not 50)
  written by professional methodologists.
• 8. These texts would likely vary depending on the
  
• methodological predilections of the authors.
• Example Rosnow and Rosenthal (2013),
  Beginning Behavioral Research A Conceptual
  Primer (emphasis on meta-analysis, effect sizes,
  and the binomial effect-size display, reflecting
  Rosenthals own methodological work).
• 9. This variation in textbook content would be
  appropriate for an education in behavioral
  research methods, where it is genuinely
  controversial as to what we should learn.

27
References

• Aiken, L. S., West, S. G., Millsap, R. E.
  (2008). Doctoral training in statistics,
  measurement, and methodology in psychology.
  Replication and extension of Aiken, West,
  Sechrest, and Renos (1990) survey of PhD
  programs in North America. American Psychologist,
  63, 32-50.
• Blachowicz, J. (2009). How science textbooks
  treat scientific method A philosophers
  perspective. British Journal for the Philosophy
  of Science, 60, 303-344.
• Dewey, J. (1910). How we think. Boston Heath.
• Haig, B. D. (2005). An abductive theory of
  scientific method. Psychological Methods, 10,
  371-388.
• Haig, B. D. (2013). Detecting psychological
  phenomena Taking bottom-up research seriously.
  American Journal of Psychology, 126, 135-153.

28

• Haig, B. D. , Ellis, C. (2013). How research
  methods textbooks mislead psychologists about
  scientific method (in preparation).
• Kuhn, T. S. (1962/1970). The structure of
  scientific revolutions (2nd ed.) Chicago
  University of Chicago Press.
• Lakatos, I. (1970). Falsification and the
  methodology of scientific research programmes. In
  I. Lakatos A. Musgrave (Eds.), Criticism and
  the growth of knowledge (pp. 91-196). Cambridge
  Cambridge University Press.
• Lakin, J. L., Giesler, R. B., Moros, K. A.,
  Vosmik, J. R. (2007). HOMER as an acronym for the
  scientific method. Teaching of Psychology, 34,
  94-96.
• Laudan, L. (1977). Progress and its problems
  Towards a theory of scientific growth. Berkeley
  University of California Press.

29

• Nickles, T. (1987). Methodology, heuristics, and
  rationality. In J. C. Pitt M. Pera (Eds.),
  Rational changes in science (pp. 103-132).
  Dordrecht Reidel.
• Popper, K. R. (1959). The logic of scientific
  discovery. London Hutchinson.
• Proctor, R. W., Capaldi, E. J. (2001).
  Improving the science education of psychology
  students Better teaching of methodology.
  Teaching of Psychology, 28, 173-181.
• Rudolph, J. L. (2005). Epistemology for the
  masses The origins of the scientific method in
  American schools. History of Education Quarterly,
  45, 341-376.