Linear Regression¶
Simple and Multiple Linear Regression¶
1. Motivation¶
In many problems in data science, we want to understand how a response variable depends on one or more predictor variables.
Examples:
- Predicting house prices from area, location, and age.
- Predicting sales from advertising budget.
- Predicting risk from financial indicators.
- Predicting demand from price and macroeconomic variables.
Linear regression models the relationship
$$
Y = f(X_1,\dots,X_p) + \varepsilon
$$
using a linear function of the predictors.
2. The Linear Regression Model¶
The general multiple linear regression model is
$$
Y_i = \beta_0 + \beta_1 X_{i1} + \cdots + \beta_p X_{ip} + \varepsilon_i,
\quad i=1,\dots,n
$$
where
- $Y_i$ — response variable
- $X_{ij}$ — predictors (features)
- $\beta_j$ — unknown parameters
- $\varepsilon_i$ — random noise
Assumptions:
$$
\varepsilon_i \sim N(0,\sigma^2), \qquad \text{i.i.d.}
$$
Thus
$$
\mathbb{E}[Y_i | X_i] = \beta_0 + \beta_1 X_{i1} + \cdots + \beta_p X_{ip}
$$
and
$$
\operatorname{Var}(Y_i | X_i) = \sigma^2
$$
3. Simple Linear Regression¶
The simplest case has one predictor.
$$
Y_i = \beta_0 + \beta_1 X_i + \varepsilon_i
$$
Interpretation:
- $\beta_0$ — intercept
- $\beta_1$ — slope
The expected value of $Y$ is
$$
\mathbb{E}[Y|X] = \beta_0 + \beta_1 X
$$
4. Least Squares Estimation¶
We observe data
$$
(X_1,Y_1),\dots,(X_n,Y_n)
$$
Define the residuals
$$
e_i = Y_i - (\beta_0 + \beta_1 X_i)
$$
The least squares method minimizes
$$
S(\beta_0,\beta_1) = \sum_{i=1}^n e_i^2
$$
that is
$$
S(\beta_0,\beta_1) =
\sum_{i=1}^n
\left(Y_i-\beta_0-\beta_1 X_i\right)^2
$$
The least squares estimators are
$$
\hat{\beta}_1 =
\frac{\sum (X_i-\bar X)(Y_i-\bar Y)}
{\sum (X_i-\bar X)^2}
$$
and
$$
\hat{\beta}_0 = \bar Y - \hat{\beta}_1 \bar X
$$
The fitted regression line is
$$
\hat Y = \hat{\beta}_0 + \hat{\beta}_1 X
$$
6. Interpretation of the slope¶
The slope coefficient measures the expected change in $Y$ when $X$ increases by one unit.
$$
\beta_1 =
\frac{\operatorname{Cov}(X,Y)}{\operatorname{Var}(X)}
$$
If $\beta_1 > 0$:
If $\beta_1 < 0$:
7. Residuals¶
The residuals are
$$
e_i = Y_i - \hat Y_i
$$
Important properties:
$$
\sum e_i = 0
$$$$
\sum X_i e_i = 0
$$
These orthogonality conditions are central to least squares.
8. Variance of the estimator¶
The noise variance is estimated by
$$
\hat{\sigma}^2 =
\frac{1}{n-2}
\sum e_i^2
$$
The variance of the slope estimator is
$$
\operatorname{Var}(\hat{\beta}_1)
=
\frac{\sigma^2}{\sum (X_i-\bar X)^2}
$$
Estimated standard error:
$$
SE(\hat{\beta}_1)
=
\sqrt{
\frac{\hat{\sigma}^2}{\sum (X_i-\bar X)^2}
}
$$
9. Hypothesis Testing¶
We often test
$$
H_0:\beta_1 = 0
$$
Test statistic:
$$
t =
\frac{\hat{\beta}_1}{SE(\hat{\beta}_1)}
$$
Under the model assumptions
$$
t \sim t_{n-2}
$$
10. Confidence Interval¶
A confidence interval for $\beta_1$ is
$$
\hat{\beta}_1
\pm
t_{1-\alpha/2,n-2}
SE(\hat{\beta}_1)
$$
11. Coefficient of Determination ($R^2$)¶
Define the sums of squares
Total variability:
$$
SST =
\sum (Y_i-\bar Y)^2
$$
Explained variability:
$$
SSR =
\sum (\hat Y_i-\bar Y)^2
$$
Residual variability:
$$
SSE =
\sum (Y_i-\hat Y_i)^2
$$
Relationship:
$$
SST = SSR + SSE
$$
The coefficient of determination is
$$
R^2 =
\frac{SSR}{SST}
=
1-\frac{SSE}{SST}
$$
Interpretation:
- proportion of variance explained by the model.
12. Multiple Linear Regression¶
In many data science problems we have multiple predictors.
The model becomes
$$
Y_i =
\beta_0
+
\beta_1 X_{i1}
+
\beta_2 X_{i2}
+
\cdots
+
\beta_p X_{ip}
+
\varepsilon_i
$$
Define
$$
Y =
\begin{pmatrix}
Y_1 \\
\vdots \\
Y_n
\end{pmatrix}
$$$$
X =
\begin{pmatrix}
1 & X_{11} & \dots & X_{1p} \\
1 & X_{21} & \dots & X_{2p} \\
\vdots & \vdots & & \vdots \\
1 & X_{n1} & \dots & X_{np}
\end{pmatrix}
$$$$
\beta =
\begin{pmatrix}
\beta_0 \\
\beta_1 \\
\vdots \\
\beta_p
\end{pmatrix}
$$
The model is
$$
Y = X\beta + \varepsilon
$$
We minimize
$$
S(\beta) =
(Y-X\beta)^T (Y-X\beta)
$$
Taking derivatives gives the normal equations
$$
X^T X \hat{\beta} = X^T Y
$$
If $X^T X$ is invertible:
$$
\hat{\beta} =
(X^T X)^{-1} X^T Y
$$
15. Fitted Values and Residuals¶
Predicted values:
$$
\hat Y = X \hat{\beta}
$$
Residuals:
$$
e = Y - \hat Y
$$
Projection interpretation:
$$
\hat Y = P_X Y
$$
where
$$
P_X = X (X^T X)^{-1} X^T
$$
is the projection matrix onto the column space of $X$.
16. Variance of the estimator¶
Under the model assumptions
$$
\operatorname{Var}(\hat{\beta})
=
\sigma^2 (X^T X)^{-1}
$$
Estimate:
$$
\hat{\sigma}^2 =
\frac{SSE}{n-p-1}
$$
17. Testing regression coefficients¶
For each coefficient we test
$$
H_0:\beta_j = 0
$$
Test statistic:
$$
t =
\frac{\hat{\beta}_j}{SE(\hat{\beta}_j)}
$$
with
$$
t \sim t_{n-p-1}
$$
18. Global F-test¶
We test
$$
H_0:
\beta_1 = \cdots = \beta_p = 0
$$
Statistic:
$$
F =
\frac{SSR/p}{SSE/(n-p-1)}
$$
Under $H_0$
$$
F \sim F_{p,n-p-1}
$$
19. Important assumptions¶
The classical linear regression model assumes:
- Linearity
- Independent observations
- Homoscedasticity
$$
\operatorname{Var}(\varepsilon_i) = \sigma^2
$$
- Normal errors (for inference)
20. Regression in Data Science¶
In practice, regression is used for:
- prediction
- feature importance
- causal modeling
- economic forecasting
- machine learning baselines
Extensions include:
- ridge regression
- lasso
- generalized linear models
- nonlinear regression
Summary¶
Linear regression provides:
- interpretable models
- statistical inference
- efficient estimation via least squares
- a geometric projection interpretation
It is one of the fundamental tools in statistics and data science.
