Bayesian Approach to Parameter Estimation

Treat $\Theta$ as a random variable with prior $p(\Theta)$.

According to Bayes' Rule, $p(\Theta|X) = \frac{p(\Theta)p(X|\Theta)}{p(X)}$

• The ML estimate is given by:

  $\Theta_{ML} = argmax_\Theta \, p(X|\Theta)$

• The MAP estimate is given by:

  $\Theta_{MAP} = argmax_\Theta \,p(X|\Theta)p(\Theta)$

• The Bayes Estimate is given by:

  $\Theta_{BAYES'} = E[\Theta|X] = \int_\Theta \Theta p(\Theta|X) d\Theta$ (the integral becomes a summation for discrete values)

Example with a Discrete Prior on $\Theta$

Consider a parameterized distribution uniform on $[0,\Theta]$.

Say the discrete prior on $\Theta$ is given by:

$P(\Theta=1) = 2/3$

$P(\Theta=2) = 1/3$

Suppose X={0.5,1.3,0.7} Given X, we know that $P(\Theta|X) = 0$ and therefore, $P(\Theta=2|X)=1$. So, the ML, MAP and BAYES' hypotheses are all 2.

Now, suppose X={0.5,0.7,0.1} $p(X|\Theta=1) = 1^3 = 1$

$p(X|\Theta=2) = (1/2)^3 = 1/8$

So, $p(X) = P(\Theta=1)p(X|\Theta=1) + P(\Theta=2)p(X|\Theta=2) = 51/72$

Therefore, $P(\Theta=1|X) = \frac{p(X|\Theta=1)P(\Theta=1)}{p(X)} = 48/51$ and $P(\Theta=2|X) = 3/51$

In this case, the MAP hypothesis is 1 and the ML hypothesis is 1. The Bayes' hypothesis can be computed as $E[\Theta|X] = 1*(48/51) + 2*(3/51) = 54/51 = 1.06$

The posterior density of x given X is given by:

$p(x=0.82|X) =p(\Theta=1|X)p(x=0.82|\Theta=1) + p(\Theta=2|X)p(x=0.82|\Theta=2)$

$=(48/51)*1 + (3/51)*(1/2) = 99/102$

Example with a Continuous Prior on $\Theta$

Assume the data X is drawn from a Gaussian with a known variance $\sigma^2$ and an unknown mean $\mu$ (this is now the $\Theta$).

Assume a Gaussian prior on $\Theta$ i.e. $\Theta \sim N(\mu_0, \sigma_0^2)$ and $\mu_0, \, \sigma_0^2$ are known.

Then, generate X from $N(\Theta, \sigma^2)$ (this $\Theta$ is the mean of the Gaussian from which X was chosen. It is what we need to estimate.)

Given X, we have:

$\Theta_{ML} = m\, (i.e.\, the\,\, sample\,\, mean) = \frac{\sum_t x^t}{N}$

$\Theta_{MAP} = \frac{N/\sigma^2}{N/\sigma^2 + 1/\sigma_0^2}m + \frac{1/\sigma_0^2}{N/\sigma^2 + 1/\sigma_0^2}\mu_0$

$\Theta_{BAYES'} = \Theta_{MAP}!$

As $N\rightarrow\infty$, m dominates the weighted sum of m and $\mu_0$.

Estimates of Mean and Variance of a Distribution (not just Gaussian)

The ML estimate for the mean is m i.e. the sample mean.

The ML estimate of variance is $\frac{\sum_t (x^t-m)^2}{N}$ (this is biased since $E[\sigma^2] < \sigma^2$).

Note that the estimate for variance is lower than the actual value because we use the sample mean m to compute it instead of using the actual mean.

However, $\frac{\sum_t (x^t-m)^2}{N-1}$ is an unbiased estimate.