Chapter 3: Linear Regression

1
Chapter 3 Linear Regression
2

• 1. Meaning of Regression

3
Meaning of Regression

• Examine relationship between dependent and
  independent variables
• Ex how is quantity of a good related to price?
• Predict the population mean of the dependent
  variable on the basis of known independent
  variables
• Ex what is the consumption level , given a
  certain level of income

4
Meaning of Regression

• Also test hypotheses
• Ex About the precise relation between
  consumption and income
• How much does consumption go up when income goes
  up.

5

• 2. Regression Example

6
Regression Example

• Assume a country with a total population of 60
  families.
• Examine the relationship between consumption and
  income.
• Some families will have the same income
• Could split into groups of weekly income (100,
  120, 140, etc)

7
Regression Example

• Within each group, have a range of family
  consumption patterns.
• Among families with 100 income we may have six
  families, whose spending is 65, 70, 74, 80, 85,
  88.
• Define income X and spending Y.
• Then within each of these categories, we have a
  distribution of Y, conditional upon a certain X.

8
Regression Example

• For each distributions, compute a conditional
  mean
• E(Y(XX i).
• How do we get E(Y(XX i) ?
• Multiply the conditional probability (1/6) by Y
  value and sum them
• This is 77 for our example.
• We can plot these conditional distributions for
  each income level

9
Regression Example

• The population regression is the line connecting
  the conditional means of the dependent variable
  for fixed values of the explanatory variable(s).
• Formally E(YXi)
• This population regression function tells how
  the mean response of Y varies with X.

10
Regression Example

• What form does this function take?
• Many possibilities, but assume its a linear
  function E(YXi) 1 2Xi
• 1 and 2 are the regression coefficients
  (intercept and slope).
• Slope tells us how much Y changes for a given
  change in X.
• We estimate 1 and 2 on the basis of actual
  observations of Y and X.

11

• 3. Linearity

12
Linearity

• Linearity can be in the variables or in the
  parameters.
• Linearity in the variables
• Conditional expectation of Y is a linear function
  of X -
• The regression is a straight line
• Slope is constant
• Can't have a function with squares, square root,
  or interactive terms- these have a varying slope.

13
Linearity

• We are concerned with linearity in the parameters
• The parameters are raised to the first power
  only.
• It may or may not be linear in the variables.

14
Linearity

• Linearity in the parameters
• The conditional expectation of Y is a linear
  function of the parameters
• It may or may not be linear in Xs.
• E(YXi) 1 2Xi is linear
• E(YXi) 1 ?2Xi is not.
• Linear if the betas appear with a power of one
  and are not multiplied or divided by other
  parameters.

15

• 4. Stochastic Error

16
Stochastic Error

• Individual values could be higher or lower than
  the conditional mean
• Specify ui Yi - E(YXi), where ui is the
  deviation of individual values from conditional
  mean.
• Turn this around Yi E(YXi) ui
• ui is called a stochastic error term
• It is a random disturbance.
• Without it, model is deterministic.

17
Stochastic Error Example

• Assume family consumption is linearly related to
  income, plus disturbance term. Some examples
• Family whose spending is 65. This can be
  expressed as
• Yi 65 1 2(100) ui
• Family whose spending is 75
• Yi 75 1 2(100) ui

18
Stochastic Error Example

• Model has a deterministic part and a stochastic
  part.
• Systematic part determined by price, education,
  etc.
• An econometric model indicates a relationship
  between consumption and income
• Relationship is not exact, it is subject to
  individual variation and this variation is
  captured in u.

19
Expected Value of U

• Yi E(YXi) ui
• Take conditional expectation
• E(YiXi) E(EYXi) E(uiXi)
• E(YiXi) E(Y Xi) E(uiXi )
• Expected value of a constant is a constant and
  once the value of Xi is fixed, E(YXi) is a
  constant
• So E(uiXi) 0
• Conditional mean values of ui 0

20
What Error Term Captures

• Omitted variables
• Other variables that affect consumption not
  included in model
• If correctly specified our model should include
  these
• May not know economic relationship and so omit
  variable.
• May not have data
• Chance events that occur irregularly--bad
  weather, strikes.

21
What Error Term Captures

• Measurement error in the dependent variable
• Friedman model of consumption
• Permanent consumption a function of permanent
  income
• Data on these not observable and have to use
  proxies such as current consumption and income.
• Then the error term represents this measurement
  error and captures it.

22
What Error Term Captures

• Randomness of human behavior
• People don't act exactly the same way even in the
  same circumstances
• So error term captures this randomness.

23

• 5. Sample Regression Function

24
Sample Regression Function

• If have whole population, we can determine a
  regression line by taking conditional means
• In practice, usually have a sample.
• Suppose took a sample of population
• Cant accurately estimate the population
  regression line since we have sampling
  fluctuations.

25
Sample Regression Function

• Our sample regression line can be denoted

26
Sample Regression Function

• In stochastic form

We can have several independent variables - this
is multivariate regression e.g. consumption may
depend on interest rate as well as income.
27

• 6. Ordinary Least Squares

28
OLS Regression

• Estimate the PR by the method of ordinary least
  squares.
• We have a PRF Yi 1 2Xi ui
• The PRF is not directly observable, so we
  estimate it from the SRF
• Yi b1 b2Xi ei
• We can rewrite as
• ei actual Yi - predicted Yi
• ei Yi - b1 - b2Xi

29
OLS Regression

• We determine the SRF is such a manner that it is
  a good fit.
• We make the sum of squared residuals as small as
  possible.

By squaring, we give more weight to larger
residuals.
30
OLS Regression

• Residuals are a function of the betas
• Choosing different values for beta gives
  different values for squared residuals.
• We choose the beta values that minimize this sum.
• These are the least-squares estimators.

31
Normal Equations

• The least squares estimates are derived in the
  following manner

32
Normal Equations

33
Normal Equations

34

• 8. Assumptions of Classical Linear Regression
  Model

35
Assumptions

• Using model Y B1 B2X u
• Y depends on X and u
• X values are fixed and u values are random.
• Thus Y values are random too.
• Assumptions about u are very important.
• Assumptions are made that ensure that OLS
  estimates are BLUE.

36
Linearity Assumption

• The regression model is linear in the parameters
  and the error term.
• Y B1 B2X e.
• Not necessarily linear in the variables
• We can still apply OLS to models that are
  nonlinear in the variables.

37
Specification Assumption

• Assume the regression model is correctly
  specified
• All variables included (no specification bias).
• Otherwise, specification error results.

38
Expected Value of Error

• Expected value of the error term0
• E(ui) 0
• Its mean value is 0, conditional on the Xs.
• Add a stochastic error term to equations to
  explain individual variation.
• Assume the error term is from a distribution
  whose mean is zero

39
Expected Value of Error

• In practice the mean is forced to be zero by
  intercept term, which incorporates any difference
  from zero
• Intercept represents the fixed portion of Y that
  cannot be explained by the independent variables.
• The error term is the random portion

40
No Correlation with Error

• Explanatory variables are uncorrelated with the
  error term
• There is zero covariance between the disturbance
  ui and the explanatory variable Xi.
• Cov(Xiui) 0
• Alternatively, X and u have separate influences
  on Y

41
No Correlation with Error

• Suppose the error term and X are positively
  correlated.
• Estimated coefficient would be higher than it
  should because the variation in Y caused by e is
  attributed to X

42
No Correlation with Error

• Consumption function violates this assumption
• Increase in C leads to increase in income which
  leads to increase in C.
• So error term in consumption and income move
  together
• If we do not have this assumption - then
  simultaneous equation estimation

43
Constant Variance of Error

• The variance of each ui is the same given a value
  of Xi.
• var(ui) ?2 a constant (Homoscedasticity)
• Ex variance of consumption is the same at all
  levels of income
• Alternative variance of the error term changes
  (Heteroscedasticity)
• Ex variance of consumption increases as income
  increases

44
No Correlation Across Error Terms

• No correlation between two error terms
• The covariance between the u's zero
• Cov (ui, uj) 0 for i not equal to j

45
No Correlation Across Error Terms

• Often shows up in time series - serial
  correlation
• Random shock in one period which affects the
  error term may persist and affect subsequent
  error terms.
• Ex positive error in one period associated with
  positive error in another

46
No Perfect Linear Function Among Variables

• No explanatory variable is a perfect linear
  function of other explanatory variables
• Multicollinearity occurs when variables move
  together
• Ex explain home purchases and include both real
  and nominal interest rates for a time period in
  which inflation was constant.

47

• 9. Properties of OLS Estimators

48
OLS Properties

• 1)linear (linear functions of Y) Y b1 b2X
• 2)Unbiased
• E(b1) B1and E(b2) B2
• In repeated sampling, the expected values of b1
  and b2 will coincide with their true values B1
  and B2.

49
OLS Properties

• 3) They have minimum variance
• var b1 is less than the variance of any other
  unbiased linear estimator of B1
• var b2 is less than the variance of any other
  unbiased linear estimator of B2

50
BLUE Estimator

• Given the assumptions of the CLRM, OLS
  estimators, in the class of unbiased linear
  estimators, have minimum variance
• They are BLUE.

51

• 10. Variances and Standard Errors of OLS
  Estimators

52
Variances and Standard Errors
53
Variances and Standard Errors
54
Variances and Standard Errors

• ?2 is the variance of the error term, assumed
  constant for each u (homoscedasticity.)
• If know ?2 one can compute all these terms.
• If don't know it use its estimator.
• The estimator of ?2 is (ei)2/n-2

55
Degrees of Freedom

• n-2 is degrees of freedom for error
• Sum of independent observation
• To get e, we have to compute predicted Y
• To compute predicted Y, we must first obtain b1
  and b2, so we lose 2 df.

56
Standard Error of Estimate

This is called the standard error of the estimate
(the standard deviation of the Y values about
the regression line)
It is used as a measure of goodness of fit of the
estimated regression line.
57
Example

• Estimated regression line

Y 24.47 0.509 X se (6.41) (.036)
t 3.813 14.243
58
Example

59
Example

• The the estimated slope coefficient is 0.509 and
  its standard error (standard deviation) is 0.036.
  
• This is a measure of how much ?2 varies from
  sample to sample.
• We can say our computed ?2 lies within a certain
  number of standard deviations from the true ?2.

60

• 11. Hypothesis Testing

61
Hypothesis Testing

• Set up the null hypothesis that our parameter
  values are not significantly different from zero
• H0?2 0
• What does this mean?
• Income has no effect on spending.
• So set up this null hypothesis and see if it can
  be rejected.

62
Hypothesis Testing

• In problem 5.3, ?2 1.25
• This is different from zero, but this is just
  derived from one sample
• If we took another sample we might get 0.509 and
  a third sample we might get 0
• In other words, how do we know that this is
  significantly different from zero?

63
Hypothesis Testing

• ?2 N(?2, (??2)2)
• Can test either by confidence interval approach,
  or by test of significance approach.
• ?2 follows the normal distribution with mean and
  variance as above

64
Hypothesis Testing

• However, we do not know the true variance ?2
• We can estimate ?2
• Then we have

65
Hypothesis Testing

• However, we do not know the true variance ?2
• We can estimate ?2
• Then we have

More generally (?2 - B2)/ se ?2
66
Problem 5.3 Example

• ?/se(?)1.25/0.03931.793t(n-2)
• At 95 with 7 df, t2.365 so reject the null.
• Also could do a one-tail test
• Set up the alternative hypothesis that ?2gt0
• Also reject the null since t 1.895 for
  one-tailed test.

67
Problem 5.3 Example

• Most of the time, we assume a null that the
  parameter value 0.
• There are occasions where we may want to set up a
  different null hypothesis.
• In Fisher example, we set up hypothesis that b2
  1.
• So now 1.25-1 /se 0.25/.039 6.4 So it is
  significant.

68
Confidence Interval Approach

B2 0 and B2 1 do not lie in this interval
69

• 12. Coefficient of Determination--R2

70
Coefficient of Determination

• The coefficient of determination, R2, measures
  the goodness of fit of the regression line
  overall

variation in variation in Y Y from mean
explained by X unexplained value around its
mean variation
71
Coefficient of Determination

Total variation in observed Y values about their
mean is partitioned into 2 parts, one
attributable to the regression line and the other
to random forces.
72
Coefficient of Determination

• If the sample fits the data well, ESS should be
  much larger than RSS.
• The coefficient of determination (R2) ESS/TSS
• Measures the proportion or percentage of the
  total variation in Y explained by the regression
  model.

73
Correlation Coefficient

• The correlation coefficient is the square root of
  R2
• Correlation coefficient measures the strength of
  the relationship between two variables.
• However, in a multivariate context, R has little
  meaning.

74

• 13. Forecasting

75
Forecasting

• Suppose we want to predict out of sample and know
  relation between CPI and SP (Problem 5.2)
• Have data to 1989 and want to predict 1990 stock
  prices.
• Expect inflation in 1990 to be 10 so CPI is 124
  12.4 136.4
• Y -195.08 3.82CPI
• Estimated Y for 1990 is 325.97-195.08
  3.82(136.4)

76
Forecasting

• There will be some error to this forecast -
  prediction error.
• This has quite a complicated formula.
• This error increases as we get further away from
  the sample mean.
• Hence, we cannot forecast very far out of sample
  with a great deal of certainty.