# Bias and Variance of an Estimator

Consider the following estimators of the mean of a distribution. X is an i.i.d. sample from the distribution.

1. $$m_1 = \frac{\sum_t x^t}{N}$$ (this is the MLE)
2. $$m_2 = \frac{x^1+x^N}{2}$$
3. $$m_3 = 5$$

Now, draw a sample of size N (say N=3): X={6,1,5}

$$m_1 = 4, \, m_2= 11/5, \, m_3=5$$

If we consider the means to be random variables, each of them will have a variance.

Say we want to estimate $$\Theta$$ (here, $$\Theta=\mu$$ of the distribution from which we are drawing X)

The desirable property of the estimator d of $$\Theta$$ is that the expected value of d must be equal to the quantity we want to estimate i.e. $$E[d] = \Theta$$. d is then called the **unbiased estimator**.

The **bias of an estimator** 'd' is given by:

$$b_\Theta(d) = E[d]-\Theta$$

If $$b_\Theta(d) = 0$$, d is an **unbiased estimator**.

**Is** $$m_2$$ **an unbiased estimator of** $$\mu$$**?**

$$m_2 = \frac{x^1+x^N}{2}$$\
$$E[m_2] = E[\frac{x^1+x^N}{2}] = \frac{1}{2}E[x^1+x^N] = \frac{1}{2} E[x^1] + \frac{1}{2} E[x^N]$$

Since $$E[x^t] = \mu$$ (by definition), $$E[m_2] = \frac{1}{2}\mu + \frac{1}{2}\mu = \mu$$

Therefore, $$m_2$$ is an unbiased estimator of the mean $$\mu$$.

**Is** $$m_3$$ **an unbiased estimator of** $$\mu$$**?**

$$E[m_3] = E[5] = 5$$

Clearly, $$E[m_3]-\mu = 0$$ iff $$\mu=5$$. Therefore, $$m_3$$ is **not** an unbiased estimator of the mean $$\mu.$$

The **variance of an estimator** 'd' is given by

$$E[(d-E[d])^2]$$

More data leads to lower variance.

$$m_3$$ has the least variance (it is always 5!). $$m_1$$ has a lower variance than $$m_2$$.

The **square error** of an estimator is given by: $$E[(d-\Theta)^2] = (E[d]-\Theta)^2 + E[(d-E[d])^2] =$$ $$Bias^2 + Variance$$
