> For the complete documentation index, see [llms.txt](https://vikram-bajaj.gitbook.io/cs-gy-6923-machine-learning/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://vikram-bajaj.gitbook.io/cs-gy-6923-machine-learning/types-of-machine-learning/supervised-learning/bias-and-variance-of-an-estimator.md).

# Bias and Variance of an Estimator

Consider the following estimators of the mean of a distribution. X is an i.i.d. sample from the distribution.

1. $$m\_1 = \frac{\sum\_t x^t}{N}$$ (this is the MLE)
2. $$m\_2 = \frac{x^1+x^N}{2}$$
3. $$m\_3 = 5$$

Now, draw a sample of size N (say N=3): X={6,1,5}

$$m\_1 = 4, , m\_2= 11/5, , m\_3=5$$

If we consider the means to be random variables, each of them will have a variance.

Say we want to estimate $$\Theta$$ (here, $$\Theta=\mu$$ of the distribution from which we are drawing X)

The desirable property of the estimator d of $$\Theta$$ is that the expected value of d must be equal to the quantity we want to estimate i.e. $$E\[d] = \Theta$$. d is then called the **unbiased estimator**.

The **bias of an estimator** 'd' is given by:

$$b\_\Theta(d) = E\[d]-\Theta$$

If $$b\_\Theta(d) = 0$$, d is an **unbiased estimator**.

**Is** $$m\_2$$ **an unbiased estimator of** $$\mu$$**?**

$$m\_2 = \frac{x^1+x^N}{2}$$\
$$E\[m\_2] = E\[\frac{x^1+x^N}{2}] = \frac{1}{2}E\[x^1+x^N] = \frac{1}{2} E\[x^1] + \frac{1}{2} E\[x^N]$$

Since $$E\[x^t] = \mu$$ (by definition), $$E\[m\_2] = \frac{1}{2}\mu + \frac{1}{2}\mu = \mu$$

Therefore, $$m\_2$$ is an unbiased estimator of the mean $$\mu$$.

**Is** $$m\_3$$ **an unbiased estimator of** $$\mu$$**?**

$$E\[m\_3] = E\[5] = 5$$

Clearly, $$E\[m\_3]-\mu = 0$$ iff $$\mu=5$$. Therefore, $$m\_3$$ is **not** an unbiased estimator of the mean $$\mu.$$

The **variance of an estimator** 'd' is given by

$$E\[(d-E\[d])^2]$$

More data leads to lower variance.

$$m\_3$$ has the least variance (it is always 5!). $$m\_1$$ has a lower variance than $$m\_2$$.

The **square error** of an estimator is given by: $$E\[(d-\Theta)^2] = (E\[d]-\Theta)^2 + E\[(d-E\[d])^2] =$$ $$Bias^2 + Variance$$