VALID EMPIRICAL RESEARCH ON MANAGEMENT ACCOUNTING

1
VALID EMPIRICAL RESEARCH ONMANAGEMENT ACCOUNTING

• MICHAEL SHIELDS
• MICHIGAN STATE UNIVERSITY
• 2005

2
CRITERIA FOR VALID EMPIRICAL RESEARCH

• External
• Construct
• Internal
• Statistical Conclusion
• Campbell and Stanley (1963)
• Rosenthal and Rosnow (1969)

3
PREDICTIVE VALIDITY FRAMEWORK

• Libby, Bloomfield and Nelson (2002) and Runkel
  and McGrath (1972)
• Relations between constructs are not tested
  directly (External validity). Instead, they are
  tested by the assessed relation between variables
  (Internal validity)
• For a test to be valid, the links between
  constructs and variables must be valid (Construct
  validity) and other variables besides the IV that
  might affect the DV must be controlled
  (Statistical conclusion validity)

4
PREDICTIVE VALIDITY FRAMEWORK
5
EXTERNAL VALIDITY

• Concepts and causal relations (i.e., theory) can
  be generalized to other people, organizations,
  setting, and times
• Internal validity is a necessary but not
  sufficient condition for external validity
• Testing the external validity of theory includes
  testing the theorys boundary condition, which
  are defined by the theorys assumptions
  (rationality, equilibrium)
• Testing boundary conditions requires extreme
  values of the IVs and/or samples at or beyond
  the limit of an assumption
• Guidelines for valid concepts and causal
  relations are presented later

6
THREATS TO EXTERNAL VALIDITY

• Interaction of nonrandom assignment and IV
• Sampling units (people) not randomly assigned to
  a treatment respond to it differently than
  sampling units randomly assigned to it
• Interaction of research setting and IV
• Stimuli in a studys setting cause sampling units
  to respond differently to the IV than how they
  respond to the IV in the natural environment
• Interaction of time and IV
• Sampling units respond to the IV differently at
  the time of the study than at other times
• Interaction of pretest and IV
• Pretest affects sampling units response to the
  IV
• Multiple IV interference
• Repeated exposure to the IV affects sampling
  units response to it
• Carryover or demand effects
• Reactivity
• Sampling units are aware their behavior is being
  observed and recorded (obtrusive), which causes
  them to modify their behavior

7
CONSTRUCT VALIDITY

• Correspondence between a concept and a variable
• Concept is an abstraction formed by
  generalization from particulars
• Variable is the operationalization of the concept
• Measurement or manipulation
• Convergent validity is convergence across
  different measures of the same concept
• Divergent validity is the divergence across
  measures of different concepts

8
THREATS TO CONSTRUCT VALIDITY

• Inadequate operationalization of concept
• Variable excludes some defined meaning of concept
  or includes unintended meaning of concept
• Levels of manipulated IV
• Extreme vs modal value
• Single-item measures
• Researcher expectancies
• Researchers expectations about a study can
  influence their research design, conduct of the
  study, and interpretation of the results
• How researchers decide to manipulate or measure
  variables
• How researchers decide to analyze the data and
  interpret the data in relation to the hypotheses

9
INTERNAL VALIDITY

• Changes in the DV are caused by changes in the IV
  only
• A valid test of a causal relation must establish
  not only that if the IV changes then the DV
  changes, but also that if the IV does not change
  then the DV does not change
• The absence of the IV is a control to ensure that
  the DV changes only when the IV changes and the
  DV does not change when the IV does not change
• If research cannot establish that changes in the
  DV are caused only by changes in the IV, then
  there is little, if anything, to generalize

10
THREATS TO INTERNAL VALIDITY

• History Observed effect can be due to an event
  that occurred between the pretest and posttest
  other than the change in the IV
• Maturation Observed effect is due to sampling
  units getting older, wiser, or more experienced
• Pretest pretest changes sampling units posttest
  scores
• Measuring sampling units before their exposure to
  the IV influences how they respond to the IV and
  thus the DV
• Instrumentation A change in the calibration of
  the measuring instrument between pretest and
  posttest or changes in observers
• Nonrandom Assignment Sampling units are not
  randomly assigned to treatments
• Self-selection and survivorship bias

11
CONTINUED

• Statistical Regression Sampling units are
  assigned to treatments based on their extreme
  scores
• Their next scores are likely to be less extreme
  but not because of the treatment
• Mortality Different types of sampling units drop
  out of a study a different rates
• Lack of Control Lack of constancy of conditions
  to observe the effect of the IV on the DV
• Variables other than the IV are not held constant
• Demand Effects Sampling units behave in ways
  they believe a researcher desires, either to
  please the research or because they believe the
  demanded behavior is socially preferred

12
CONTINUED

• Evaluation Apprehension Sampling units want to
  be appealing to researchers based on their
  personality, intelligence, competence or
  emotional adjustment, which causes them to modify
  their behavior in a study
• People Surrogation Sampling units that
  participate in study are poor surrogates for the
  target population
• Mundane Realism Research context includes
  superficially features of the natural environment
  that are not germane to the study
• Distracts sampling units introduces competing
  explanations for effects
• Research Realism Research context does not
  engage, involve, or motivate sampling units in
  the study

13
STATISTICAL CONCLUSION VALIDITY

• All changes in the DV are due to changes in the
  IV only
• The effects of all variables except the IV that
  can influence the DV or IV-DV relation are
  controlled for
• Random assignment of sampling units to treatments
  (levels of the IV)
• Research design, by holding constant other
  variables that influence the DV
• Measurement of other variables that influence the
  DV and then statistically controlling for their
  effects

14
THREATS TO STATISTICALCONCLUSION VALIDITY

• Lack of random assignment of sampling units
• Research design that does not hold constant all
  variables that can influence the DV except for
  the IV
• Reliability of treatment manipulation
• Not covarying out the effects of all variables on
  the DV except the effects of the IV
• Low statistical power
• Cannot reject the null hypothesis when it should
  be rejected
• Violate assumptions of the statistical methods
  used
• Effective error rate with multiple tests
• Allocation of overall intended alpha level of
  significance across all tests