> For the complete documentation index, see [llms.txt](https://vikram-bajaj.gitbook.io/cs-gy-6923-machine-learning/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://vikram-bajaj.gitbook.io/cs-gy-6923-machine-learning/types-of-machine-learning/supervised-learning/parametric-estimation.md).

# Parametric Estimation

This section discusses how to estimate the parameters of a distribution i.e. $$\mu, \sigma^2$$ of the line $$f(x) = w\_0 + w\_1x$$.

We denote the parameters by $$\Theta = (\mu, \sigma^2)$$

The **likelihood** of $$\Theta$$ given a sample X is given by:

$$l(\Theta|X) = \sum\_t ,p(x^t|\Theta)$$

Therefore, the **Log Likelihood** of $$\Theta$$ given a sample X is denoted by:

$$L(\Theta|X) := log ,l(\Theta|X) = \sum\_t log p(x^t|\Theta)$$

This assumes that the observations in X are independent.

The **Maximum Likelihood Estimator** (MLE) is given by:

$$\Theta^\* := argmax\_\Theta L(\Theta|X)$$

## Estimating the Parameter $$P\_h$$ of a Bernoulli Distribution

X is a Bernoulli Random Variable.

$$P\_h = P\[X=1]$$

For example, consider the following:

Let 1 denote Heads, and 0 denote Tails.\
Say X = {1,1,0}\
We need to determine $$\Theta$$ i.e. $$P\_h$$.

We have $$l(P\_h|X) = P(X|P\_h) = P\_h*P\_h*(1-P\_h)$$

More generally, for $$X = {x^t}\_{t=1}^N$$, we have:

$$p(X|p\_h) = \Pi\_{t=1}^N p\_h^{x^t}(1-p\_h)^{(1-x^t)}$$

It can be proved that the MLE is given by $$p\_h = \frac{\sum x^t}{N}$$.

## Estimating the Parameters of a Multinomial Distribution

Consider a die with 6 faces numbered from 1 to 6.

If X is a Multinomial Random Variable, there are k>2 possible values of X (here, 6).

Say X={5, 4, 6}. We can imagine indicator vectors for each observation as \[0 0 0 0 1 0], \[0 0 0 1 0 0] and \[0 0 0 0 0 1].

Say X={4,6,4,2,3,3}. The MLE of $$x\_i$$ i.e. side **i** shows up, can be given by:

$$p\_i = \frac{\sum\_{t=1}^N x\_i^t}{N}$$.

## Estimating the Parameters of a Gaussian Distribution

The MLE for the mean m is $$\frac{\sum\_t x^t}{N}$$ and the MLE for the variance $$\sigma^2$$ is $$\frac{\sum\_t,(x^t-m)^2}{N}$$.

However, if we divide by N-1 instead of N (for variance), it is called the **unbiased estimate**.