# Bayesian Approach to Parameter Estimation

Treat $$\Theta$$ as a random variable with prior $$p(\Theta)$$.

According to Bayes' Rule, $$p(\Theta|X) = \frac{p(\Theta)p(X|\Theta)}{p(X)}$$

* The ML estimate is given by:

  $$\Theta\_{ML} = argmax\_\Theta , p(X|\Theta)$$
* The MAP estimate is given by:

  $$\Theta\_{MAP} = argmax\_\Theta ,p(X|\Theta)p(\Theta)$$
* The **Bayes Estimate** is given by:

  $$\Theta\_{BAYES'} = E\[\Theta|X] = \int\_\Theta \Theta p(\Theta|X) d\Theta$$ (the integral becomes a summation for discrete values)

## Example with a Discrete Prior on $$\Theta$$

Consider a parameterized distribution uniform on $$\[0,\Theta]$$.

Say the discrete prior on $$\Theta$$ is given by:

$$P(\Theta=1) = 2/3$$

$$P(\Theta=2) = 1/3$$

Suppose X={0.5,1.3,0.7}\
Given X, we know that $$P(\Theta|X) = 0$$ and therefore, $$P(\Theta=2|X)=1$$. So, the ML, MAP and BAYES' hypotheses are all 2.

Now, suppose X={0.5,0.7,0.1}\
$$p(X|\Theta=1) = 1^3 = 1$$

$$p(X|\Theta=2) = (1/2)^3 = 1/8$$

So, $$p(X) = P(\Theta=1)p(X|\Theta=1) + P(\Theta=2)p(X|\Theta=2) = 51/72$$

Therefore, $$P(\Theta=1|X) = \frac{p(X|\Theta=1)P(\Theta=1)}{p(X)} = 48/51$$ and $$P(\Theta=2|X) = 3/51$$

In this case, the MAP hypothesis is 1 and the ML hypothesis is 1.\
The Bayes' hypothesis can be computed as $$E\[\Theta|X] = 1\*(48/51) + 2\*(3/51) = 54/51 = 1.06$$

The **posterior density of x given X** is given by:

$$p(x=0.82|X) =p(\Theta=1|X)p(x=0.82|\Theta=1) + p(\Theta=2|X)p(x=0.82|\Theta=2)$$

$$=(48/51)*1 + (3/51)*(1/2) = 99/102$$

## Example with a Continuous Prior on $$\Theta$$

Assume the data X is drawn from a Gaussian with a known variance $$\sigma^2$$ and an *unknown* mean $$\mu$$ (this is now the $$\Theta$$).

Assume a Gaussian prior on $$\Theta$$ i.e. $$\Theta \sim N(\mu\_0, \sigma\_0^2)$$ and $$\mu\_0, , \sigma\_0^2$$ are known.

Then, generate X from $$N(\Theta, \sigma^2)$$ (this $$\Theta$$ is the mean of the Gaussian from which X was chosen. It is what we need to estimate.)

Given X, we have:

$$\Theta\_{ML} = m, (i.e., the,, sample,, mean) = \frac{\sum\_t x^t}{N}$$

$$\Theta\_{MAP} = \frac{N/\sigma^2}{N/\sigma^2 + 1/\sigma\_0^2}m + \frac{1/\sigma\_0^2}{N/\sigma^2 + 1/\sigma\_0^2}\mu\_0$$

$$\Theta\_{BAYES'} = \Theta\_{MAP}!$$

As $$N\rightarrow\infty$$, m dominates the weighted sum of m and $$\mu\_0$$.

## Estimates of Mean and Variance of a Distribution (not just Gaussian)

The ML estimate for the mean is m i.e. the sample mean.

The ML estimate of variance is $$\frac{\sum\_t (x^t-m)^2}{N}$$ (this is biased since $$E\[\sigma^2] < \sigma^2$$).

Note that the estimate for variance is **lower** than the actual value because we use the sample mean m to compute it instead of using the actual mean.

However, $$\frac{\sum\_t (x^t-m)^2}{N-1}$$ is an unbiased estimate.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://vikram-bajaj.gitbook.io/cs-gy-6923-machine-learning/main-4/types-of-machine-learning/supervised-learning/parametric-estimation/bayesian-approach-to-parameter-estimation.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.