Applied Regression Midterm

Front 1

Central Limit Theorem

Back 1

For populations that aren’t normally
distributed, if n ≥ 25, X ̄ approx. ∼ N(μ,σ2/n).

Front 2

significance level

Back 2

aka Type I error rate

if H0 true, when # tests -> infinity, % of tests rejected under alpha by chance -> 5 percent

Front 3

cor(X, Y )

Back 3

cov(X, Y )/σX σY

Front 4

cor(X, Y ) = 0 =⇒??

Back 4

ONLY IF X, Y ~N, X, Y are independent

Front 5

Four Kinds of Influence

Back 5

Front 6

Reversability of correlation

Back 6

Cor(X,Y) = Cor(Y,X)

Front 7

Error term in Simple Regression

Back 7

assumed mean/E(epsilon-i) = 0, variance constant, epsilon-i and epsilon-j uncorrelated so covariance is zero for all i, j i=/j

Front 8

Adjectives for regression models and what they mean

Back 8

simple = one predictor
'linear in the parameters' = no paramater appears as exponent or multiplied/divided by other parameter (nonlinear)

Front 9

Yi (response variable) -- random?

Back 9

Yes: random error term added makes Yi a random var

Front 10

E(Yi)

Back 10

Since E(error-i) = 0, E(Yi) = Beta0 + Beta1*(Xi)

so predicted model is E(Y|X) = Beta0 + Beta1*(X)

Front 11

Distribution of Yi

Back 11

Since errors are (are assumed) to be normally distributed N(0, sigma^2),

Yi ~ N(Beta0 + Beta1(Xi), sigma^2)

Front 12

Predicting changes in Y when X changes

Back 12

Beta0 and Beta1 describe the relationship between the *mean* of Y and X, while the variance of the residuals sigma^2 describes variability of Yi around the mean

Front 13

Beta0 and Beta1 in terms of E(Y)

Back 13

E(Y|X=0) = Beta0
E(Y|X=x+1) - E(Y|X=x) = Beta1

Front 14

Ordinary Least Squares regression

Back 14

a way to find Betas; minimizes the sum of squared residuals, where residual ei = Yi - Yhat (i.e. solves calculus minimization for sum of squared residuals Sigma[(Yi - Beta0 - Beta1*Xi)^2] )

Front 15

Three Assumptions of LS Regression

Back 15

All observations are uncorrelated:
- corr(Yi, Yj) = corr(error-i, error-j) = 0

All observations are equally informative:
- Var(Yi) = Var(error-i) = sigma^2

There is no systematic bias in the model
E(error-i) = 0

For testing & inference, fourth assumption is that errors are ~ N(0, sigma^2)

Front 16

Least Square regression estimate of B1

Back 16

E(Beta1-hat) = Beta1;
var(Beta1-hat) = sigma^2/Sigma[(Xi - Xbar-i)^2]

Front 17

Least Square regression estimate of B0

Back 17

E(Beta0-hat) = Beta0

Front 18

B0-hat Variance

Back 18

Front 19

B0hat and B1hat distributions

Back 19

Because sigma^2 (of Yi and error-i) isn't known, distributed t with n-2 degrees of freedom

Front 20

Consequences of LS assumptions

Back 20

Sum of residuals is 0, Sum of observations = sum of fitted values (Sigma(Yi) = Sigma(Yhat-i), regression line always goes through (Xbar, Ybar)

Front 21

Why least squares estimators are random variables

Back 21

different samples yield different values !

Front 22

Distribution of least square estimators & associated test statistic

Back 22

if errors are normally distributed as assumed, they are normally distributed;

For testing H0: Beta = 0eta-hat - 0 / S.e.(Beta-hat) ~ N or t(n-2) depending on sample size

Front 23

"Is there a relationship between...?"

Back 23

First look at correlation to determine if vars are linearly related and if so what direction relationship is in -- can also just look at a scatterplot to glean this. To quantify the relationship we employ SLR (simple linear regression) and look at R^2, which is the square of the correlation coefficient between X and Y (for simple linear regression).

Front 24

What R^2 is and means

Back 24

R^2 is the amount of variation in response that can be explained by predictor(s) in model ALONE, ignoring the effect of other things not in model; R^2 = SSR/SST

R^2 is NOT GOOD FOR TESTING, only for understanding.

Front 25

Bonferroni correction

Back 25

Testing both hypotheses at once requires adjustment to the significance level of each in order to preserve the overall significance level of the entire test

If k tests are performed, at least one of them needs to be significant at alpha/k for overall testing to be signficiant

Front 26

Standard error for mean response (prediction)

Back 26

note smaller than forecast because it's harder to estimate a single point; "standard error of the fitted value"

Front 27

Standard error for one observation (forecast)

Back 27

"standard error of the forecast value"

Forecast interval is technically NOT a confidence interval. It is not for testing! This is a 95% highest probability density interval for the new forecast variable.

Front 28

Graphical interpretation of Betas in MLR

Back 28

Beta1 = effect of X1 on Yhat with all other predictors held constant ('adjusted for other Xs')

Front 29

Collinearity

Back 29

inter-predictor relationship

Front 30

An example of model fitting procedure

Back 30

1) look at scatter matrix, check out which predictors have a clearly linear relation to outcome

2)

Front 31

Comparing R^2 of regress y x1 and regress y x2 to R^2 of regress y x1 x2

Back 31

Rcombined^2 <= R1^2 + R2^2; only equal if X1 and X2 are totally unrelated

Front 32

Deriving betas for regress y x1 x2 from regress y x1 and regress y x2

Back 32

To find the unique contribution of x2 that is not already explained by x1, we regress the unexplained part of y onto the unexplained part of x2. We use **eyx1∼ex2x1**. Beta1 from this regression is x2's regression coefficient.

Front 33

Marginal vs. partial coefficients

Back 33

marginal coefficient Beta1 in SLR describes effect of X1 on Y ignoring all other variables; partial coefficient Beta1 in MLR describes effect of X1 on Y 'adjusted for' other predictors (i.e. with other predictors fixed).

Front 34

Difference in distribution of predictors for MLR

Back 34

All holds as in SLR, but estimates are distributed on t- with n-p-1 degrees of freedom. Residual variance still estimated with s^2.

Front 35

How to test H0: all coefficients are zero

Back 35

1) compare p-values associated with each Beta to alpha/(# parameters); if at least one exceeds significance can reject H0

OR

2) Cf. Overall F-test in regression output w/ associated p-value; this p-value is defaulted to be a measure of extremity against your H0

Front 36

How to test H0: some coefficients are zero

Back 36

1) Bonferroni

2) Run FM (full model) and RM (reduced model) regressions and compare adjusted R^2 (adjusts for # parameters)

3) Partial F-test ('exact' way)

Front 37

Adjusted R^2

Back 37

unlike R2, adjusted R2 has expectation zero, and it can be negative

Front 38

Partial F-test

Back 38

df of distribution is difference in # of parameters in two models and n-p-1 (df of full model)

judiciously, can be used for the following scenarios:

Testing whether one single coefficient is 0, Testing whether all coefficients are simultaneously 0, Testing whether several coefficients are simultaneously 0

Front 39

Testing H0: Beta1=Beta2

Back 39

Two ways to do this; first way is to generate a new variable Star = X1+X2

If full model is, for example:
Y = Beta0 + Beta1X1 + Beta2X2
reduced model to compare is
Y = Beta0 + Beta1(X1+X2 = Star)

Alternatively, can derive SSE of reduced model from Root MSE which is on output of 'constrain', and can calculate partial F-statistic from there.

Front 40

Implications of Unmet Assumptions

Back 40

for example: If errors are not ~N, least square estimates don't have known distribution, can't test hypotheses b/c statistics are undefined

Front 41

Implications of Unmet Assumptions

Back 41

for example: If errors are not ~N, least square estimates don't have known distribution, can't test hypotheses b/c statistics are undefined

Front 42

Implications of Unmet Assumptions

Back 42

for example: If errors are not ~N, least square estimates don't have known distribution, can't test hypotheses b/c statistics are undefined

Front 43

Implications of Unmet Assumptions

Back 43

for example: If errors are not ~N, least square estimates don't have known distribution, can't test hypotheses b/c statistics are undefined

Front 44

Main assumption of regression inference

Back 44

Errors are iid ~ N(0,Sigma^2)

Front 45

Main assumption of regression inference

Back 45

Errors are iid ~ N(0,Sigma^2)

Front 46

Main assumption of regression inference

Back 46

Errors are iid ~ N(0,Sigma^2)

Front 47

Main assumption of regression inference

Back 47

Errors are iid ~ N(0,Sigma^2)

Front 48

Leverage

Back 48

hii, the amount of weight Yi is given in predicting Yhat-i, is the leverage of the i-th observation

Front 49

Leverage

Back 49

hii, the amount of weight Yi is given in predicting Yhat-i, is the leverage of the i-th observation

Front 50

Leverage

Back 50

hii, the amount of weight Yi is given in predicting Yhat-i, is the leverage of the i-th observation

Front 51

Leverage

Back 51

hii, the amount of weight Yi is given in predicting Yhat-i, is the leverage of the i-th observation

Front 52

About Variance of Residuals and Why its the case

Back 52

although residuals do have mean 0, but each has different variance. In addition, they are correlated

because looking at Yhat-i in terms of leverage, we see that all residuals are estimated from the same data, and they each interdepend on the rest of the observations

Front 53

About Variance of Residuals and Why its the case

Back 53

although residuals do have mean 0, but each has different variance. In addition, they are correlated

because looking at Yhat-i in terms of leverage, we see that all residuals are estimated from the same data, and they each interdepend on the rest of the observations

Front 54

About Variance of Residuals and Why its the case

Back 54

although residuals do have mean 0, but each has different variance. In addition, they are correlated

because looking at Yhat-i in terms of leverage, we see that all residuals are estimated from the same data, and they each interdepend on the rest of the observations

Front 55

About Variance of Residuals and Why its the case

Back 55

although residuals do have mean 0, but each has different variance. In addition, they are correlated

because looking at Yhat-i in terms of leverage, we see that all residuals are estimated from the same data, and they each interdepend on the rest of the observations

Front 56

Leverage point

Back 56

if hii (leverage) is close to 1; means that Yi plays a strong role in predicting Yhat-i; means further that Yi is an outlier

Front 57

Leverage point

Back 57

if hii (leverage) is close to 1; means that Yi plays a strong role in predicting Yhat-i; means further that Yi is an outlier

Front 58

Leverage point

Back 58

if hii (leverage) is close to 1; means that Yi plays a strong role in predicting Yhat-i; means further that Yi is an outlier

Front 59

Leverage point

Back 59

if hii (leverage) is close to 1; means that Yi plays a strong role in predicting Yhat-i; means further that Yi is an outlier

Front 60

Screen for leverage points

Back 60

Front 61

Screen for leverage points

Back 61

Front 62

Leverage in SLR

Back 62

Front 63

Leverage in SLR

Back 63

Front 64

standardized residual

Back 64

mean 0, variance 1; note that this relies on Sigma (variation of error) which is infrequently known (IE THESE ARE RARE!)

Front 65

standardized residual

Back 65

mean 0, variance 1; note that this relies on Sigma (variation of error) which is infrequently known (IE THESE ARE RARE!)

Front 66

studentized residual

Back 66

s is estimate of Sigma

in STATA: rstandard = studentized (our class) = 'internally studentized' (the book)

"too big" is > 2, and these residuals no longer add up to 0

Front 67

studentized residual

Back 67

s is estimate of Sigma

in STATA: rstandard = studentized (our class) = 'internally studentized' (the book)

"too big" is > 2, and these residuals no longer add up to 0

Front 68

Checking Homoscedacity with Studentized residuals

Back 68

rstandard vs fitted values 'box of judgement'

Front 69

Checking Homoscedacity with Studentized residuals

Back 69

rstandard vs fitted values 'box of judgement'

Front 70

Checking normality with studentized residuals

Back 70

Q-Q plot!

Front 71

Checking normality with studentized residuals

Back 71

Q-Q plot!

Front 72

Checking linearity with studentized residuals

Back 72

rstandard vs fitted values scatter

Front 73

Checking linearity with studentized residuals

Back 73

rstandard vs fitted values scatter

Front 74

Checking independence with studentized residuals

Back 74

Only pertinent if order matters in data; index plot of rstandard

Front 75

Checking independence with studentized residuals

Back 75

Only pertinent if order matters in data; index plot of rstandard

Front 76

I decided to omit a predictor from my model. How do I check if this omitted content is correlated with my new model's residuals?

Back 76

scatter plot of residuals vs omitted predictor; if you see a linear pattern you should see if adding the variable back in is a benefit!

Front 77

I decided to omit a predictor from my model. How do I check if this omitted content is correlated with my new model's residuals?

Back 77

scatter plot of residuals vs omitted predictor; if you see a linear pattern you should see if adding the variable back in is a benefit!

Front 78

Review Placard

Back 78

Front 79

Review Placard

Back 79

Front 80

Review placard contd

Back 80

Front 81

Insidious Outliers!

Back 81

outlier(s) --> big residual --> overstated s.e. --> internally studentized residuals are too SMALL --> ability to detect less glaring outliers and to check assumptions is impaired

EXTERNALLY STUDENTIZED residuals conceived in response -- these are from MSE, which don't need the whole data (can leave out outliers to make a more 'usual' data set)

Front 82

externally studentized or 'jackknife' procedure

Back 82

dont want to contaminate w obs that will be dropped later, so calculate s.e. w/o current obs

tends to make outlier residuals more prominent than internally studentized residuals

Front 83

Potential harm of keeping an outlier

Back 83

regression surface tilted to accommodate them, OR if MSE inflated too much. This extra noisiness could mask other
outliers and/or regression violations

Front 84

Potential harm of dropping an outlier

Back 84

may introduce bias!

Front 85

Influence

Back 85

a point is influential if removing the point causes substantial model change

In order to have much influence, a case must have large leverage AND residual

Front 86

Cook's Distance

Back 86

Residual^2 * potential

Cook’s distance combines the size of the residuals with the amount of leverage. Both are required to be large for Cook’s distance to be large. Large Cook’s distance means likely outlier and likely leverage point (i.e. likely influence point).