> For the complete documentation index, see [llms.txt](https://vikram-bajaj.gitbook.io/cs-gy-6923-machine-learning/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://vikram-bajaj.gitbook.io/cs-gy-6923-machine-learning/main-4/types-of-machine-learning/supervised-learning/parametric-estimation/bayesian-approach-to-parameter-estimation.md).

# Bayesian Approach to Parameter Estimation

Treat $$\Theta$$ as a random variable with prior $$p(\Theta)$$.

According to Bayes' Rule, $$p(\Theta|X) = \frac{p(\Theta)p(X|\Theta)}{p(X)}$$

* The ML estimate is given by:

  $$\Theta\_{ML} = argmax\_\Theta , p(X|\Theta)$$
* The MAP estimate is given by:

  $$\Theta\_{MAP} = argmax\_\Theta ,p(X|\Theta)p(\Theta)$$
* The **Bayes Estimate** is given by:

  $$\Theta\_{BAYES'} = E\[\Theta|X] = \int\_\Theta \Theta p(\Theta|X) d\Theta$$ (the integral becomes a summation for discrete values)

## Example with a Discrete Prior on $$\Theta$$

Consider a parameterized distribution uniform on $$\[0,\Theta]$$.

Say the discrete prior on $$\Theta$$ is given by:

$$P(\Theta=1) = 2/3$$

$$P(\Theta=2) = 1/3$$

Suppose X={0.5,1.3,0.7}\
Given X, we know that $$P(\Theta|X) = 0$$ and therefore, $$P(\Theta=2|X)=1$$. So, the ML, MAP and BAYES' hypotheses are all 2.

Now, suppose X={0.5,0.7,0.1}\
$$p(X|\Theta=1) = 1^3 = 1$$

$$p(X|\Theta=2) = (1/2)^3 = 1/8$$

So, $$p(X) = P(\Theta=1)p(X|\Theta=1) + P(\Theta=2)p(X|\Theta=2) = 51/72$$

Therefore, $$P(\Theta=1|X) = \frac{p(X|\Theta=1)P(\Theta=1)}{p(X)} = 48/51$$ and $$P(\Theta=2|X) = 3/51$$

In this case, the MAP hypothesis is 1 and the ML hypothesis is 1.\
The Bayes' hypothesis can be computed as $$E\[\Theta|X] = 1\*(48/51) + 2\*(3/51) = 54/51 = 1.06$$

The **posterior density of x given X** is given by:

$$p(x=0.82|X) =p(\Theta=1|X)p(x=0.82|\Theta=1) + p(\Theta=2|X)p(x=0.82|\Theta=2)$$

$$=(48/51)*1 + (3/51)*(1/2) = 99/102$$

## Example with a Continuous Prior on $$\Theta$$

Assume the data X is drawn from a Gaussian with a known variance $$\sigma^2$$ and an *unknown* mean $$\mu$$ (this is now the $$\Theta$$).

Assume a Gaussian prior on $$\Theta$$ i.e. $$\Theta \sim N(\mu\_0, \sigma\_0^2)$$ and $$\mu\_0, , \sigma\_0^2$$ are known.

Then, generate X from $$N(\Theta, \sigma^2)$$ (this $$\Theta$$ is the mean of the Gaussian from which X was chosen. It is what we need to estimate.)

Given X, we have:

$$\Theta\_{ML} = m, (i.e., the,, sample,, mean) = \frac{\sum\_t x^t}{N}$$

$$\Theta\_{MAP} = \frac{N/\sigma^2}{N/\sigma^2 + 1/\sigma\_0^2}m + \frac{1/\sigma\_0^2}{N/\sigma^2 + 1/\sigma\_0^2}\mu\_0$$

$$\Theta\_{BAYES'} = \Theta\_{MAP}!$$

As $$N\rightarrow\infty$$, m dominates the weighted sum of m and $$\mu\_0$$.

## Estimates of Mean and Variance of a Distribution (not just Gaussian)

The ML estimate for the mean is m i.e. the sample mean.

The ML estimate of variance is $$\frac{\sum\_t (x^t-m)^2}{N}$$ (this is biased since $$E\[\sigma^2] < \sigma^2$$).

Note that the estimate for variance is **lower** than the actual value because we use the sample mean m to compute it instead of using the actual mean.

However, $$\frac{\sum\_t (x^t-m)^2}{N-1}$$ is an unbiased estimate.