Bias and Variance of an Estimator

Consider the following estimators of the mean of a distribution. X is an i.i.d. sample from the distribution.

1. $m_1 = \frac{\sum_t x^t}{N}$ (this is the MLE)

2. $m_2 = \frac{x^1+x^N}{2}$

3. $m_3 = 5$

Now, draw a sample of size N (say N=3): X={6,1,5}

$m_1 = 4, \, m_2= 11/5, \, m_3=5$

If we consider the means to be random variables, each of them will have a variance.

Say we want to estimate $\Theta$ (here, $\Theta=\mu$ of the distribution from which we are drawing X)

The desirable property of the estimator d of $\Theta$ is that the expected value of d must be equal to the quantity we want to estimate i.e. $E[d] = \Theta$. d is then called the unbiased estimator.

The bias of an estimator 'd' is given by:

$b_\Theta(d) = E[d]-\Theta$

If $b_\Theta(d) = 0$, d is an unbiased estimator.

Is $m_2$ an unbiased estimator of $\mu$?

$m_2 = \frac{x^1+x^N}{2}$ $E[m_2] = E[\frac{x^1+x^N}{2}] = \frac{1}{2}E[x^1+x^N] = \frac{1}{2} E[x^1] + \frac{1}{2} E[x^N]$

Since $E[x^t] = \mu$ (by definition), $E[m_2] = \frac{1}{2}\mu + \frac{1}{2}\mu = \mu$

Therefore, $m_2$ is an unbiased estimator of the mean $\mu$.

Is $m_3$ an unbiased estimator of $\mu$?

$E[m_3] = E[5] = 5$

Clearly, $E[m_3]-\mu = 0$ iff $\mu=5$. Therefore, $m_3$ is not an unbiased estimator of the mean $\mu.$

The variance of an estimator 'd' is given by

$E[(d-E[d])^2]$

More data leads to lower variance.

$m_3$ has the least variance (it is always 5!). $m_1$ has a lower variance than $m_2$.

The square error of an estimator is given by: $E[(d-\Theta)^2] = (E[d]-\Theta)^2 + E[(d-E[d])^2] =$ $Bias^2 + Variance$