Multiple Regression

1
Multiple Regression
2
Multiple Regression

• Regression
• Attempts to predict one criterion variable using
  one predictor variable
• Addresses the question Does the predictor
  significantly predict the criterion?

3
Multiple Regression

• Multiple Regression
• Attempts to predict one criterion variable using
  2 predictor variables
• Addresses the questions Do the predictors
  significantly predict the criterion? If so, which
  predictor is best?
• Allows for variance to be removed from one
  predictor prior to evaluating the rest (like
  ANCOVA)

4
Multiple Regression

• How to compare the predictive value of 2
  predictors
• When comparing multiple predictors within an
  experiment
• Use standardized b (ß)
• ß bxs/sintercept
• z-score lets you compare performance between 2
  variables with different metrics, by addressing
  performance relative to a sample mean SD

5
Multiple Regression

• How to compare the predictive value of 2
  predictors
• When comparing multiple predictors between
  experiments
• Use b
• SE highly variable between experiments ? the SE
  from Exp. 1 ? the SE from Exp. 2 ? ßs from both
  experiments not comparable
• Cant compare z-score of your Stats grade from
  this semester with your Stats grade if you take
  the class again next semester
• If next semesters class is especially dumb, you
  appear to have gotten much smarter

6
Multiple Regression

• Magnitude of the relationship between one
  predictor and a criterion (b/ß) in a model
  dependent upon the other predictors in that model
• Relationship between IQ and SES (with College GPA
  and Parents SES in the model) will be different
  if more, less, or different predictors included
  in the model

7
Multiple Regression

• When comparing the results of 2 experiments using
  regression, coefficients (b/ß) will not be the
  same
• Will be similar to the extent that the regression
  models are similar
• Why not?

8
Multiple Regression

• Coefficients (b/ß) represent partial and
  semipartial (part) correlations, not traditional
  Pearsons r
• Partial Correlation the correlation between 2
  variables with the variance from one or more
  variables removed
• I.e. correlation between the residuals of both
  variables, once variance from one or more
  covariates has been removed

9
Multiple Regression

• Partial Correlation the amount of the variance
  in a criterion that is associated with a
  predictor that could not be explained by the
  other covariate(s)

10
Multiple Regression

• Semipartial/Part Correlation -the correlation
  between 2 variables with the variance from one or
  more variables removed from the predictor only
  (i.e. not the criterion)
• I.e. correlation between the residuals of the
  predictor, once variance from one or more
  covariates has been removed, and the criterion

11
Multiple Regression

• Part Correlation the amount of variance that a
  predictor explains in a criterion once variance
  from the covariates has been removed
• I.e. the percentage of the total variance left
  unexplained by the covariate that the predictor
  accounts for
• Since the variance that is removed from the
  criterion depends on the other predictors in the
  model, different models yield different
  regression coefficients

12

• Partial Correlation B
• Part Correlation B/A B

13
Multiple Regression

• How to compare the predictive value of 2
  predictors
• Remember Regression coefficients are very
  unstable from sample to sample, so interpret
  large differences in coefficients only (gt .2)

14
Multiple Regression

• Like regression, tests
• Ability of each predictor to predict the
  criterion variable (tests bs/ßs)
• Overall ability of the model (all predictors
  combined) to predict the criterion variable
  (Model R2)
• Model R2 total variance in criterion
  accounted for by predictors
• Model R correlation between predictors and
  criterion
• Also can test
• If one or more predictors can predict the
  criterion if variance from one or more other
  predictors is removed
• If each predictor significantly increases the
  Model R2

15
Multiple Regression

• Predictors are evaluated with variance from other
  predictors removed
• More than one way to remove this variance
• Examine all predictors en masse with variance
  from all other predictors removed
• Remove variance from one or more predictors
  first, then look at second set
• Like in factorial ANCOVA

16
Multiple Regression

• This is done by specifying different selection
  methods
• Selection method method of inputting predictors
  into a regression equation
• Four most commonly used methods
• Commonly-used Only 4 methods offered by SPSS

17
Multiple Regression

• Selection Methods
• Simultaneous Adds all predictors at once is
  therefore the lack of a selection method
• Good if there is no theory to guide which
  predictors should be entered first
• But when does this ever happen?

18
Multiple Regression

• Selection Methods
• All Subsets Computer finds method of entering
  predictors that maximizes overall Model R2
• But SPSS doesnt do it and it finds best subset
  in your particular dataset since data, not
  theory, guiding selection method not guarantee
  that the model will generalize to other datasets,
  particularly in smaller samples

19
Multiple Regression

• Selection Methods
• Backward Elimination Starts will all predictors
  in the model and eliminates the predictor with
  least unique variance related to criterion
  iteratively until all predictors are significant
• Iterative process involving several steps
• It begins with all predictors, so predictors with
  least variance not overlapping with other
  predictors (i.e. that would be partialled out)
  are removed
• But, also atheoretical/based on data only

20
Multiple Regression

• Selection Methods
• Forward Selection the opposite of backward
  elimination - starts will the predictor in the
  model most strongly related to the criterion and
  adds the predictor next most strongly-related to
  criterion iteratively until a nonsignificant
  predictor is found
• Step 1 predictor most correlated with the
  criterion (P1) ? Step 2 add strongest predictor
  when P1 partialled out
• But also atheoretical

21
Multiple Regression

• Selection Methods
• Stepwise
• Technically, any selection method that procedes
  iteratively (in steps) is stepwise (i.e. both
  backward elimination and forward selection)
• However, usually refers to method where order of
  predictors is determined in advance by the
  researcher based upon theory

22
Multiple Regression

• Selection Method
• Stepwise
• Why would you use it?
• Same reason as covariates in ANCOVA
• Want to know if Measure A of treatment adherence
  is better than Measure B? Run stepwise regression
  and enter Measure B first, then Measure A with
  treatment outcome as the criterion.
• Addresses the question Does Measure A predict
  treatment outcome even when variance from Measure
  B has already been removed (i.e. above and beyond
  Measure B)?

23
Multiple Regression

• Selection Method
• Stepwise
• Why would you use it?
• Running a repeated-measures design and want to
  make sure your groups are equal on pre-test
  scores? Enter the pre-test into the first step of
  your regression.

24
Multiple Regression

• Assumptions
• Linearity of Regression
• Variables linearly related to one another
• Normality in Arrays
• Actual values of DV normally distributed around
  predicted values (i.e. regression line) AKA
  regression line is good approximation of
  population parameter
• Homogeneity of Variance in Arrays
• Assumes that variance of criterion is equal for
  all levels of predictor(s)

25
Multiple Regression

• Issues to be aware of
• Range Restriction
• Heterogenous Subsamples
• Outliers
• With multiple predictors, must be aware of both
  univariate outliers (unusual values on one
  variable) as well as multivariate outliers
  (unusual values on two or more variables)

26
Multiple Regression

• Outliers
• Univariate outlier a man weighing 500 lbs.
• Multivariate outlier a man who is 6 tall and
  weights 120 lbs. Note neither value is a
  univariate outlier, but both together are quite
  odd
• Three variables define the presence of an outlier
  in multiple regression
• Distance distance from the regression line
• Leverage distance from predictor mean
• Influence average of distance and leverage

27

• Distance distance from the regression line
• See A
• Leverage distance from predictor mean
• See B
• Influence average of distance and leverage

28
Multiple Regression

• Degree of Overlap in Predictors
• Adding predictors is like adding covariates in
  ANCOVA In adding one that correlates too highly
  with others, model R2 remains unchanged but df
  decreases, making the regression less powerful
• Tolerance multiple R2 between all predictors
  want to be low
• Examine bivariate correlations between
  predictors, if correlation exceeds internal
  consistency (a), get rid of one of them

29
Multiple Regression

• Multiple regression can also test for more
  complex relationships, such as mediation and
  moderation
• Mediation when one variable (predictor)
  operates on another variable (criterion) via a
  third variable (mediator)

30

• Math self-efficacy mediates math ability and
  interest in a math major
• Must establish paths A B, and that path C is
  smaller when paths A B are included in the
  model (i.e. math self-efficacy accounts for
  variance in interest in a math major above and
  beyond math ability)

31

• Find significant correlations between the
  predictor and mediator (path A) and mediator and
  criterion (path B)
• Run a stepwise regression with the predictor
  entered first, then the predictor and mediator
  entered together in step 2

32
Multiple Regression

• The mediator should be a significant predictor of
  the criterion in step 2
• The predictor-criterion relationship (b/ß) should
  decrease from step 1 to step 2
• Full mediation If this relationship is
  significant in step 1, but nonsignificant in step
  2
• Partial mediation This relationship is
  significant in step 1, and smaller, but still
  significant, in step 2

33
Multiple Regression

• Partial mediation
• Sobels test (1982) tests the statistical
  significance of this mediation relationship
• Regress predictor on mediator (path A) and
  mediator on criterion (path B) in 2 separate
  regressions
• Calculate sß for path A B, where sß ß/t
• Calculate a t-statistic, where df n 3 and

34
Multiple Regression

• Multiple regression can also test for more
  complex relationships, such as mediation and
  moderation
• Moderation (in regression) when the strength of
  a predictor-criterion changes as a result of a
  third variable (moderator)
• Interaction (in ANOVA) when the strength of the
  relationship between an IV and DV changes as a
  function of levels of the IV

35
Multiple Regression

• Moderation
• Unlike in ANOVA, you have to create a moderator
  term for yourself by multiplying the predictor
  and moderator
• In SPSS, go to Transform ? Compute
• Typical to enter the predictor and mediator in
  the first step of a regression and the
  interaction term in the second step to determine
  the contribution of the mediator above and beyond
  the main effect terms
• Just like how variance is partitioned in a
  factorial ANOVA

36
Logistic Regression

• Logistic Regression used to predict a
  dichotomous criterion (only 2 levels) variable
  with 1 continuous or discrete predictors
• Cant use linear regression with a dichotomous
  criterion because
• Dichtotomous assuming the criterion isnt
  normally distributed (i.e. assumption of
  normality in arrays is violated)

37

• Cant use linear regression with a dichotomous
  criterion because
• Regression line fits data more poorly when
  predictor 0 (i.e. assumption of homogeneity of
  variance arrays is violated)

38
Logistic Regression

• Logistic Regression
• Interpreting coefficients
• In logistic regression, b represents change in
  log odds in criterion with one point increase in
  predictor
• Raise ex where x b, to find the odds b
  -.0812 ? e-.0812 .9220

39
Logistic Regression

• Logistic Regression
• Interpreting coefficients
• Continuous predictor One pt. increase in
  predictor corresponds to decreasing (because b is
  neg) odds of criterion by factor of .922 (almost
  100 or twice as likely)
• Dichotomous predictor Odds of change in one
  group vs. other group (sign indicates increase or
  decrease)