Applied Regression Final

Front 1

NB: When you've transformed a variable for regression ...

Back 1

Make sure you remember the units are transformed, too!

Front 2

First step when you're not sure if data set needs to be transformed into linear

Back 2

'center' the independent variables (predictors): i.e. subtract mean of x from each x, so transformed var is original's distance from the mean

just wanna bring observations closer to zero!

Front 3

Second step when you're not sure if data needs to be transformed

Back 3

look at X-Y scatter -- is there curvature? if so, take transform (maybe log or exp) on Y.... if linear, result: E(log(Y)) = Beta0 + Beta1(X)

can derive meaning of betas like above

Front 4

how confidence bands change when converting to and from transformation

Back 4

Front 5

'Least serious' regression violation vis a vis transformation consideration

Back 5

Non-normality; also happens to usually be remedied automatically after other violations are addressed

Front 6

Violations of linearity

Back 6

1) Analytically (Table 6.1 in book; try logit if response is a probability etc.)

2) Numerically (look at scatter & try what looks best!)

Front 7

Other transformations other than Logarithmic (mostly about boxcox)

Back 7

Ladder of power: exponentiating Y by -1, -0.5, 0.5, 1, 2

Box-Cox transform: attached;

NB: If after running boxcox, lambda = 0 can't be rejected, 'simply' use log transform (i.e. Y-prime = log(Y))

if lambda = -1 can't be rejected, use Y-prime = 1/Y (reciprocal)

if lambda = 1 can't be rejected, do nothing! Y-prime = Y!

after running boxcox, can either use suggested lambda in boxcox fit (last of printout) or can follow above guidlines if last bit didn't reject all hypotheses

Front 8

How do we spot non-linearities?

Back 8

Scatterplots, partial plots, augmented CPR plots ('acrplot' in STATA), LOOK FOR CURVED PATTERNS IN THESE PLOTS

acpr is augmented partial residual plot, need to do one-by-one if have multiple predictors

',lowes' fits curved line and fitted line

Front 9

Encountering heteroscedastic errors (POWER TRANSFORM)

Back 9

In applications, when increasing variance over predictor var encountered, st.dev of residuals increases as predictor increases. so we hypothesize that st.dev(error) = k*x

leading to the following transform: Y' = Y/X' where X' = 1/X

generate ty = y/x
generate tx = 1/x

and re-scatter

Can try log too to see if it helps but didn't help like this way did in example!

next what we did: regress y x (x^2)

it worked OK but better to use Y/X and 1/X because it has fewer predictors than log(Y) ~ X + X^2

Front 10

Transform options

Back 10

square root, reciprocal (^(-1)), reciprocal of square root!, log, log(sqrt(y))!

~ AVOID UNNECESSARY TRANSFORMATIONS ~

Front 11

Box-Cox before and after

Back 11

Front 12

Weighted Least Square method

Back 12

another way to correct for heteroscedacity, by assigning weight to each observation's SSE

. regress y x [w=1/x^2]
(analytic weights assumed)
(sum of wgt is 1.0470e-04)

useful when it looks like predictor is influencing magnitude of variance spread; we 'standardize' by dividing by their own variance

"each observation is weighted by the estimate of its
standard deviation"

Computing the weights from the data: how to estimate these within-region variances?
Answer: Find the MSE for each region!

regress y x1 x2 x3
. predict res, resid
. generate r2 = res^2
. egen r = mean(r2), by(region)

r represents MSE for each region

Weight Least Squares (WLS) divides each term in SSE by r

Front 13

what's good...

Back 13

Compared with LS, the R2
, F-statistic and the MSE of WLS are
much better! Much better t

Front 14

NB Regarding WLS

Back 14

The WLS returns the residual using y y^. But our fitted y^ is obtained by WLS. To check residuals, we need to
obtain the weighted residuals.�

Front 15

autocorrelation

Back 15

if observations follow a natural order ---> have patterns, the errors are correlated and correlation is called 'autocorrelation'

Front 16

Common autocorrelation patterns

Back 16

Front 17

Common causes of autocorrelation

Back 17

1. same observations measured in adjacent periods (stock market today and tomorrow)
2. spatially close observations (temp in NC and VA)
3. another important predictor omitted

Front 18

How autocorrelation and heteroscedacity really endangers Ordinary Least Squares regression

Back 18

1. OLS estimates are unbiased but not efficient so no longer minimize SSE
2. sigma^2 and s.e. of betas may be way over- or under-estimated. Positive correlation -> under-estimation of s.e. -> increased false positives

==> C.I., OTHER TESTS INVALID

Front 19

Time series & Runs

Back 19

Time series (ordered) data -> index plot (standardized residual vs time)

IF errors are correlated, plot will display clusters of + or - data called 'runs'; sequence plot shows runs

under H0: no autocorrelation, expected number of runs / variance of statistic is as attached

'runtest' in STATA tests if observed # runs is different from expected; " thresh(0) just means 0 is the threshold or fulcrum for +/- whereas median is default threshold "

Front 20

Durbin-Watson test

Back 20

for detecting serial correlation; based on assumption that successive errors are correlated in the following, where et is the t-th OLS residual. H0: rho = 0 => errors are uncorrelated

Front 21

Approximations and tips on Durbin-Watson statistic

Back 21

when d is near 2, there isn't much autocorrelation in the error, but if d is far from 2, it suggests autocorrelation is present

Front 22

How close is 'close to 2' for D-W statistic?

Back 22

for positive autocorrelations (d is below 2):

d < dL => reject null (that there is NO positive autocorrelation)

d > dU => we fail to reject the null

dL < d < dU => test is inconclusive

dL and dU are in tables, vary by n, p

*FOR NEGATIVE AUTOCORRELATION, "WORK WITH (4-d) and use the same procedure as above"

Front 23

You can remove autocorrelation by transformation!

Back 23

Cochrane-Orcutt transformation is one way to remove autocorrelation:

'prais' command in STATA;

'two' stops procedure after first estimate of p

OR you can let the iteration run until convergence

we generate residual from et and e(t-1), generate latter from lagged variable "L.r2"

Front 24

Remember that one cause of autocorrelation is...

Back 24

artificial: the omission of another predictor
variable

Front 25

Limitations of DW statistic

Back 25

When TWO LEVELS OF AUTOCORRELATION PRESENT, DW statistic only look at the overall autocorrelation, and it can't detect higher level of autocorrelation. Thus, high DW statistic does not necessarily indicate no autocorrelation.

i.e. omitted season in ski rental data; regressing with season will fix autocorrelation which is different in ski season and warm season

Front 26

Multicollinearity

Back 26

The symptom: p-values for betas in regression are insignificant but F-statistic is signaling signficance!


==> multicollinearity, which is when some of the 'independent' predictors are correlated

multicollinearity almost always exists but concern is about severity

Front 27

If all predictors in a model are independent...

Back 27

we call the predictors 'orthogonal'

Front 28

Multicollinearity is not...

Back 28

an error! is from lack of info in data set... what if X3 can be approx. expressed as combo of X1 and X2 ? etc.

Front 29

Collinearity

Back 29

when one variable can be expressed completely as function of other variables; "most serious case"; HAVE TO DROP VAR FROM MODEL!

stata autodetects perfect collinearity and makes decision about what var to drop

Front 30

What if we ignore multicollinearity?

Back 30

If mild, major consequence is that CIs are wider than usual ("more conservative tests"), but if multicollinearity is too severe CIs are TOO big and Beta estimates are uselessly haphazard

Front 31

If we add a predictor and C.I. blows up in size / changes other estimates drastically, then

Back 31

new predictor is highly correlated with old one(s)

need assumption of independence b/c we assume beta represents roc of xn on y WITH ALL OTHER VARS HELD CONSTANT

Front 32

When is multicollinearity "serious" and how do we detect it?

Back 32

If F-test is significant and ALL individual t-tests are INsignificant, multicollinearity is likely

if fewer t-tests are insignificant, probably just means that some variables are non-important

another sign is if a predictor has a coefficient w/ a sign that is opposite of sense/context/expectation

Front 33

BEWARE: multicollinearity and pairwise combos

Back 33

Front 34

How to combat fact that multicollinearity can arise out of any linear combo of predictor vars and so is impossible to screen with just pairwise scatter?

Back 34

Front 35

tolerance

Back 35

tolerance for predictor Xj measures fraction of unexplained variance (1-(R^2)j ) in Xj after adjustment for other variables

Front 36

VIF

Back 36

variance inflation factor; 1/tolerance

VIF > 10 suggests multicollinearity

Front 37

Once we detect multicollinearity...

Back 37

First: if Xj is collinear with other variables => Xj is basically redundant

So:
1) can OMIT predictor (~ fitting model with Beta-j = 0 )
2) can CENTER near mean before constructing powers and interaction terms... will reduce quadratic/interaction collinearity coming with bigger (uncentered) coefficients
3) get more data / study up on mechanism
4) CONSTRAINED REGRESSION

Front 38

Constrained regression

Back 38

realize that there's a conceptual reason that your residual plot looks like this (i.e. distinctive change in error correlation around point), DO SEPARATE ANALYSIS! here, about x = 12

Next, make sure R^2 is high and predicted Betas all make sense conceptually... if concept-check turns out with something unreasonable, it's v. likely it's highly correlated

check with pwcorr; VIF -- > if high, run separate regressions on each var that has high VIF

HERE IT IS: instead of omitting one regressor, we keep both of them in 'constrained' context

Front 39

How constrained regression works

Back 39

Constrained regression avoids collinearity by using a same
parameter for two predictors. Relationship of the two coefficients is predefined by the constraint function.

Front 40

Variable Selection

Back 40

How do we pick between a bunch of OK models?

Various methods: F-test (if nested), adj-R^2, Mallow's Cp, likelihood-based criteria like AIC and BIC. Gets annoying tho! n variables --> 2^n possible models

Front 41

How to be efficient in var selection?

Back 41

What is model being used for? Exploration (building simple -> complicated, considering variables' conceptual inclusion), predictive (already know what's up, not thinking about vars much), explanatory (parsimonious variable choosing here, worried about confounding, i.e. corr. vs causation )

ULTIMATELY we want some of all => minimize bias^2 + variance == Mean Square Error

model should be easy to interpret and watch out for confounders!

Front 42

Stepwise regression

Back 42

USE W/ CAUTION!

Front 43

Stepwise regression: forward selection

Back 43

4. With new model, return to step 2

The probability to enter option, pe, was set to .99 so that all of the variables would enter and their order of entry depends on their signifcance.

The procedure is repeated until adding any other remaining variables would have the added variable's p-value being > pe.

Front 44

AIC and BIC

Back 44

Front 45

Stepwise regression: Backward selection

Back 45

The probability to remove option, pr, was set to .01 so that all of the variables except the last one would be removed and their order of removal also depends on their signicance.

The procedure is repeated until the p-values of the variables in the model are all smaller than pr.

The probability to remove option, pr, was set to .33 to correspond to a t-statistic of 1.0

Front 46

Warning about stepwise regression:

Back 46

Completely meaningless results can happen. Unless you can use theory and common sense to justify the resulting models, don't rely on this method.

NB: pr should be > pe

Front 47

Robustness

Back 47

find a relationship that holds for most of the data and isn't excessively influenced by small # of deviant data

mean is NOT robust estimate of center, but median is

Front 48

Least Median of Squares regression

Back 48

Robust version of mean of squared residuals in Ordinary Least Regression (which minimizes " " )

Front 49

Robust regression

Back 49

another alternative to Least Squares regression, better dealing with data contamination like outliers or overly influential observations; 'rreg' in stata

messes with weights of data so that data with Cook's D > 1 are excluded from robust analysis

doesn't assume normality, allows for outliers

Front 50

robust regression vs OLS

Back 50

In OLS regression, all cases have a weight of 1. Hence, the more cases in the robust regression that have a weight close to one, the closer the results of the OLS and robust regressions.

Try robust, if it's close to OLS go back to OLS model and treat the robust regression as a check for outliers (negative!)

Front 51

Balancing weight in robust regression

Back 51

Front 52

Poisson regression

Back 52

used to model count variables as outcome (can't be negative), aka 'log-linear'

N.B. The 'i.' before a categorical var indicates that it is a factor variable (i.e., categorical variable), and that it should be included in the model as a series of indicator variables.

Front 53

Big assumption of Poisson

Back 53

recall mean and variance of Poisson are the same => BIG assumption: conditional on predictors, est. mean and est. var are the same

Front 54

estat gof

Back 54

goodness of fit chi square; is is not
a test of the model coefficients, but a test of the model form: does the Poisson model form fit our data? Large p-value indicates good t (i.e. H0 is that model fits)

Front 55

Chi-square goodness-of-fit test indicate bad fit?

Back 55

Try to figure if there are missing predictors, if linearity assumption holds, or/and if conditional mean and conditional variance are very different

Front 56

In count situation, what if conditional mean and variance aren't the same?

Back 56

Can try to fit negative binomial

Front 57

Poisson (for counts) regression summary

Back 57

Poisson model characterized by v. Strong assumption that conditional variance of outcome variable equals the conditional mean

if bad fit, first check that model is "appropriately specified" i.e. no omitted variables or bad functional forms

if modeled bona fide, assumption that conditional var = conditional mean should be checked

Poisson regression estimated with MLE, * requires large sample *

Front 58

Logistic regression

Back 58

for binary situations (binomial); Y ~ Bin(n, pi)

logit(pi) called 'log odds' and pi / (1 - pi) called 'odds'

Front 59

Generalized linear models and the 'link' function

Back 59

Front 60

Commonly used link functions

Back 60

identity link: f(t) = t; if Y ~ normal => OLS regression

log link: f(t) = log(t); if Y ~ Poisson => Poisson regression

logit link: see attached; if Y ~ Binomial => logistic regression

Cf. command 'glm'

Front 61

Interpreting logistic regression results

Back 61

Front 62

Relationship between 'odds' and Betas (logistic regression output)

Back 62

Front 63

Multiple logistic regression

Back 63

How do we interpret Beta-hats here? analog to the multiple linear regression: Beta-hat is interpreted as the partial log-odds ratio.

What Beta-hat_j is, in words: when comparing the odds of two groups that differ by only one unit of Xj, Beta-hat_j is the log of the odds ratio between those two groups.

Front 64

Multiple logistic regression if interaction present

Back 64

Front 65

Logistic regression for grouped data

Back 65

mind that in both cases you want fitted values between 0 and 1 (because est. prob)

Front 66

Logistic reg for grouped data vs ordinary logistic reg

Back 66