Parametric Estimation for Simple Linear Regression

$r=f(X) + \epsilon$;$\epsilon\sim N(\mu,\sigma^2)$

is a line that minimizes squared error.

$g(x^t|w_0,w_1) = w_1x^t+w_0$

is the line defined by parameters $w_0, w_1$. We need to find the line that minimizes squared error, and to do so, we need to compute the values for $w_0, w_1$ that minimize the squared error.

We have $X=\{x^t,r^t\}_{t=1}^N$

We need to compute $argmin_{w_0, w_1} \,\, \sum_t \,(r^t - (w_1x^t+w_0))^2$

To solve for $w_0, w_1$, take partial derivatives w.r.t $w_0$ and $w_1$ and equate them to 0. We will get 2 equations:

$\sum_t \, r^t = Nw_0 + w_1 \sum_t x^t$

$\sum_t \, r^tx^t = w_0\sum_t\, x^t + w_1\sum_t\, (x^t)^2$

To solve for $w_0, w_1$, we use the closed form solution:

$W=A^{-1}y$

where $W=\begin{bmatrix}w_0\\w_1\end{bmatrix}, \, A=\begin{bmatrix}N & \sum_t x^t\\ \sum_t x^t & \sum_t (x^t)^2\end{bmatrix}, \, Y=\begin{bmatrix} \sum_t r^t \\ \sum_t r^tx^t\end{bmatrix}$