Assumptions of Linear Regression

Requirements

• Linear Regression needs at least two variables
• The amount of samples is preferred to be at minimum 20 observations per variable to have confidence in the model

1. Linear Relationship

• The relationship between the independent variable (response) and the dependent variable(s) (predictors) needs to be linear

• Limit the amount of outliers as Linear Regression is sensitive to outliers - test with scatter plots

• R methods: scatter plots

plot(cars)

2. Multivariate Normality

• The distribution of each of the variables is normal (or Gaussian) - the averages of each random, independent observations converge (tend towards) a central mean.

• R methods: Histograms, Shapiro-Wilk Test, Q-Q plot, Kolmogorov-Smirnof test

hist(cars$speed)

hist(cars$dist)

shapiro.test(cars$speed)

## 
##  Shapiro-Wilk normality test
## 
## data:  cars$speed
## W = 0.97765, p-value = 0.4576

shapiro.test(cars$dist)

## 
##  Shapiro-Wilk normality test
## 
## data:  cars$dist
## W = 0.95144, p-value = 0.0391

qqplot(cars$speed, cars$dist)

ks.test(cars$speed, cars$dist)

## 
##  Two-sample Kolmogorov-Smirnov test
## 
## data:  cars$speed and cars$dist
## D = 0.76, p-value = 5.735e-13
## alternative hypothesis: two-sided

3. No or little multicollinearity

• Multicollinearity occurs when the independent variables are not independent from each other - One variable can be a predictor of the other

• A second important independence assumption is that the error of the mean has to be independent from the independent variables.

• R methods: correlation, tolerance, variance inflation factor (VIF)

cor(cars)

##           speed      dist
## speed 1.0000000 0.8068949
## dist  0.8068949 1.0000000

# example of correlated variables
cars2 <- cars
cars2$speed2 <- cars2$speed
cor(cars2) # if there is a 1.0 in any other position than the diagonal there is perfect correlation

##            speed      dist    speed2
## speed  1.0000000 0.8068949 1.0000000
## dist   0.8068949 1.0000000 0.8068949
## speed2 1.0000000 0.8068949 1.0000000

4. No auto-correlation

• Autocorrelation occurs when the residuals are not independent from each other

• correlation of a signal with itself at different points in time - common in time series analysis

• R methods: Durbin-Watson Test

library(car)

fit <- lm(dist ~ speed, data=cars)
summary(fit)

## 
## Call:
## lm(formula = dist ~ speed, data = cars)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -29.069  -9.525  -2.272   9.215  43.201 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept) -17.5791     6.7584  -2.601   0.0123 *  
## speed         3.9324     0.4155   9.464 1.49e-12 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 15.38 on 48 degrees of freedom
## Multiple R-squared:  0.6511, Adjusted R-squared:  0.6438 
## F-statistic: 89.57 on 1 and 48 DF,  p-value: 1.49e-12

durbinWatsonTest(fit)

##  lag Autocorrelation D-W Statistic p-value
##    1       0.1604322      1.676225   0.166
##  Alternative hypothesis: rho != 0

5. Homoscedasticity

• The error terms along the regression are the same and not heteroscedastic.

• R methods: Residual plot, Goldfeld-Quandt Test

library(car)
residualPlot(fit)

# or in base
plot(fit$fitted.values, fit$residuals)

Resources

• Statistics Solutions

• Quick R: Diagnostics

• Quick R: Probability