Research validity can be simply understood as the quality of a research study. There are two kinds of quality issues that are referred to as internal validity and external validity.

1

• Research validity can be simply understood as
  the quality of a research study. There are two
  kinds of quality issues that are referred to as
  internal validity and external validity.
• Internal validity is the extent to which the
  outcomes of a study result from the variables
  which were manipulated, measured, or selected in
  the study rather than from other variables not
  included in the study.
• Internal validity is about how confident we are
  about the stated causal relationship between
  the independent variables and the dependent
  variable. That is, the dependent variable is due
  to independent variable but not due to something
  else.

2

• Internal Validity Threat
• History Events take place during the study that
  might affect its outcome in the same way that the
  independent variable is hypothesized to affect
  the outcome.
• Maturation Especially for developmental studies
  where children grow with the passage of time to
  become more mature in certain developmentally
  related abilities.
• Testing When people are measured repeatedly,
  e.g., pretest-posttest, they become better not
  because of the independent variable but because
  they become test smarter.
• Instrumentation The effect on the dependent
  variable is not due to the independent variable
  but due to aspects of the instrument used in the
  study.
• Regression towards the mean Particularly
  problematic when subjects are chosen because of
  extreme scores where high scoring individuals are
  more likely to score lower and low scoring
  individuals are likely to score higher the next
  time they are tested merely due to random
  measurement error.

3

1. Selection Results due to assignment to
   different treatment or control groups but not due
   to the independent variable that makes the two
   groups.
2. Mortality Especially for longitudinal studies
   that last for an extended period of time,
   attrition or dropping off from the study in a
   non-random manner may affect the outcome of the
   study.
3. Diffusion or imitation of treatment The control
   group or one of the treatment groups somehow end
   up receiving some of the same treatment as the
   other groups resulting in few differences among
   the treatments.
4. Hawthorne effect refers to the fact that when
   participants received unusal treatment in a field
   experiment, they may temporarily change their
   behavior or performance not because of the
   manipulation of the independent variable but
   because of the special attention they received
   during the experimentation.
5. John Henry effect Whereas the Hawthorne effect
   is due to some unusal performance of the
   experiemental group, sometimes the control group
   may also put up some extraordinary performance to
   outperform the experimental group due to a sense
   of demoralization for not being included in the
   special treatment experimental group.

4

• External validity is the extent to which the
  findings of a particular study can be generalized
  to people or situations other than those observed
  in the study. Many threats to external validity
  can be understood in the form of an interaction
• Treatment-attribute interaction Certain
  personality and other characteristics may
  interact with the independent variable so that
  the effect of the independent variable may be
  different on people having different personality
  characteristics.
• Treatment-setting interaction The independent
  variable may interact with other external factors
  or contexts to result in different effects for
  different settings so that the effect cannot be
  generalized to all settings.

5

• Pretest sensitization The effect of the
  independent variable may be due to the pretest
  which serves to sensitize the subjects whereas in
  the population (the real world to which the
  findings are to be generalized), there is no
  pretest and thus the treatment (independent
  variable) may not work as it does in the study.
• Posttest sensitization The effect of the
  independent variable is part due to sensitization
  or exercising effect of the posttest which is not
  available in the population to which the results
  are to be generalized.

6

• Sampling Techniques
• A population is all the elements in a defined set
  about which we wish to make an inference.
• Sampling units are non-overlapping collections of
  elements from the population.
• Sampling frame is a list of sampling units.
• A sample is a collection of sampling units drawn
  from a frame.

7

• Simple random sample
• If a sample of size n is drawn from a population
  of size N in such a way that every possible
  sample of size n has the same chance of being
  selected.
• M Sxi / n is the sample estimate of population
  mean, µ.
• Stratified random sampling
• The population of N units is divided into
  subpopulations of N1, N2... Nh units which are
  non-overlapping so that N1 N2 .. Nh N. The
  subpopulations are called strata. A sample is
  drawn from each stratum. The sample is denoted
  as n1, n2, nh. If a simple random sample is taken
  from each stratum, the whole procedure is called
  stratified random sampling.
• Cluster sampling
• A cluster sample is a simple random sample in
  which each sampling unit is a collection, or
  cluster, of elements.
• Systematic sampling
• Randomly selecting one element from the first k
  elements in the frame and every kth element
  thereafter is called a one-in-k systematic
  sample.

8

• Sample size
• The method used to select the sample is of utmost
  importance in judging the validity of the
  inference made from the sample to the population.
  The representativeness of the sample is more
  important than the size of the sample.
• Sample size is considered in relation to power of
  the test to be used, effect size to be expected,
  and number of variables investigated.

9

• Research Design
• The various ways to control extraneous variables
  and to make sure that the independent variables
  indeed cause the dependent variable make up the
  research design.
• Experimental design
• There is experimental manipulation of the
  independent variable and random assignment of
  subjects into different treatment conditions.
• Quasi-Experimental Design
• There is experimental manipulation of the
  independent variable but the assignment of
  subjects into different treatment conditions is
  not random but is based on existing
  non-equivalent groups.
• For lack of randomization, pretest is an integral
  part of quasi-experiment that enables comparisons
  among the nonequivalent groups, whereas, in most
  experiments, a pretest is often unnecessary or
  undesirable.

10

• Non-Experimental or Ex Post Facto Research
• There is no manipulation of the independent
  variable and, thus, no random assignment of
  subjects into different treatment groups.
• In experimental and quasi-experimental research,
  inferences are made from the independent
  variables (the causes) to the dependent variable
  (the effect).
• In non-experimental research, also called "ex
  post facto research", inferences are generally
  made in the opposite direction. That is beginning
  with the observation of the dependent variable,
  attempts are made to uncover, detect, or find the
  reasons (independent variable) for the existing
  variations.
• Variations are not the result of the manipulation
  of the independent variables but are pre-existing
  and are explained post hoc of after the fact.

11

• Two general strategies to protect internal
  validity are using
• (1) large samples to compensate for the lack of
  random assignment and,
• (2) large numbers of "independent" variables to
  eliminate rival explanations.
• According to some authors, there are two kinds of
  non-experimental designs, causal comparative and
  correlational studies.
• The difference lies in the measurement of the
  "independent" variable which can be either
  categorical or continuous. The categorical or
  continuous "independent" variable is also called
  a grouping variable in causal comparative studies
  and an exogenous variable in correlational
  studies.

12
Summary of Research Designs
Manipulation of Independent V. Random Assignment Sample Size Variable Number
Experimental Yes Yes Small Small
Quasi-Exp. Yes No Medium Medium
Non-Exp. No No Large Large
13

• Statistical Techniques to Control for Extraneous
  Variances
• Partial correlation. The calculation is
  complicated but the idea of partial correlation
  is simple. It is an estimate of the correlation
  between two variables in a population that is
  homogeneous on the variable (or variables) that
  is being controlled.
• Spurious effect. When two variables are
  correlated solely because they are both affected
  by the same cause, the correlation between these
  two variables is spurious.
• Mediating variable. The correlation between two
  variables can be the result of a mediating
  variable.
• Suppressor variable. A special case when a
  partial correlation is larger than its zero-order
  correlation is called a suppressor variable
  effect.
• In general, correlational studies include many
  relevant variables in an attempt to statistically
  tease out the cause and effect.

14

• Quantitative versus Qualitative Research
• Quantitative focuses on the specific or most
  salient causal link whereas qualitative takes
  into consideration the chain of events as
  contributing to a specific social process.
• Alternative conditions sufficient but not
  necessary
• Presence of the conditions associates with the
  presence of the outcome but the absence of
  conditions does not associate with the absence of
  the outcome. It is sufficient by itself but not
  necessary. Flue virus is a sufficient but not
  necessary condition of headache.
• Contingent conditions necessary but not
  sufficient
• Absence of the conditions indicates the absence
  of the outcome but presence of the conditions
  does not indicate the presence of the outcome. It
  is necessary but not sufficient by itself.
  Ability to discriminate letters is necessary but
  not sufficient to reading.

15

• Conclusion drawn from the two philosophies
• Quantitative To be able to infer causality,
  conditions have to be both sufficient and
  necessary i.e., the presence of the conditions
  is accompanied by the presence of the outcome and
  the absence of the conditions is accompanied by
  the absence of the outcome.
• Qualitative Constellation of conditions that
  are individually insufficient but necessary and
  jointly unnecessary but sufficient (INUS) to
  bring about the outcome

16

• Quantitative
• Philosophy Isolated causal link
• Method Experiment. Control extraneous variable
  to isolate out the particular linkage e.g.,
  attitude of subject, history, instrumentation,
  testing, maturation. standardize data collection.
  Following physical science tradition, studying
  only that is observable, measurable, and
  testable. Latent constructs have to
  operationalized. Lot of topics are not attempted
  for research.
• Qualitative
• Philosophy INUS, causal chain
• Method Field study. Consider combinations of
  factors. Look at context. Use different data
  collection schemes to obtain all sorts of
  information. Social phenomenon and human
  behaviour are not directly observable which
  include intentions, feelings, aspirations
  influenced by norms, culture, values. Observable
  behaviour aren't any real than internal
  phenomenon.

17

• Quantitative
• P Following the physical science tradition,
  study the social phenomenon or human behaviour as
  an objective and impartial observer.
• M control internal validity threats, such as,
  observer bias and characteristics. Keep the
  subjects unaware of your research purpose. Hire
  data collector and standardize the data
  collecting condition by training them. Structured
  interview. Let the data speak.
• Terminology Subject, researcher
• Qualitative
• P Take the perspective of the people being
  studied. See the world from the way they see it.
• M Participation. Researcher is the only or
  major source of data collection. Subjects play a
  role in data interpretation. Having subjects read
  your report and modify afterwards.
• Terminology Informants, collaborators, teachers,
  vs. participant.

18

• Quantitative
• P Deductive reasoning, formulate theory from
  previous research and conduct specific empirical
  test.
• M Hypothesis testing. Ask questions before data
  are collected. Use standardized test.
  Confirmatory or explanatory studies.
• Qualitative
• P Inductive reasoning, theory grounded in
  observation. From pieces of specific events and
  observations, develop an explanation.
• M Start from scratch. Extensive and prolonged
  observation. Going back and forth between data
  and explanation until a theory is fully grounded
  by observation. No measurement. Measurement is
  not just asking questions but knowing what to
  ask. Exploratory or discovery oriented research.

19

• Quantitative
• P Generalization
• M Random sampling, hypothesis testing,
  inferential statistics.
• Qualitative
• P Context dependent, Generalization with caution
• M Purposive sample to gather data from most
  representative situation to draw generalization.
  Informants are selected for their willingness to
  talk, their sensitivity, knowledge, and insights
  into a situation, and their ability and influence
  to gain access to new situations. No intention to
  use statistical inference. Lengthy text report.
  Text analysis.