Parametric Estimation

This section discusses how to estimate the parameters of a distribution i.e. $\mu, \sigma^2$ of the line $f(x) = w_0 + w_1x$.

We denote the parameters by $\Theta = (\mu, \sigma^2)$

The likelihood of $\Theta$ given a sample X is given by:

$l(\Theta|X) = \sum_t \,p(x^t|\Theta)$

Therefore, the Log Likelihood of $\Theta$ given a sample X is denoted by:

$L(\Theta|X) := log \,l(\Theta|X) = \sum_t log p(x^t|\Theta)$

This assumes that the observations in X are independent.

The Maximum Likelihood Estimator (MLE) is given by:

$\Theta^* := argmax_\Theta L(\Theta|X)$

Estimating the Parameter $P_h$ of a Bernoulli Distribution

X is a Bernoulli Random Variable.

$P_h = P[X=1]$

For example, consider the following:

Let 1 denote Heads, and 0 denote Tails. Say X = {1,1,0} We need to determine $\Theta$ i.e. $P_h$.

We have $l(P_h|X) = P(X|P_h) = P_h*P_h*(1-P_h)$

More generally, for $X = \{x^t\}_{t=1}^N$, we have:

$p(X|p_h) = \Pi_{t=1}^N p_h^{x^t}(1-p_h)^{(1-x^t)}$

It can be proved that the MLE is given by $p_h = \frac{\sum x^t}{N}$.

Estimating the Parameters of a Multinomial Distribution

Consider a die with 6 faces numbered from 1 to 6.

If X is a Multinomial Random Variable, there are k>2 possible values of X (here, 6).

Say X={5, 4, 6}. We can imagine indicator vectors for each observation as [0 0 0 0 1 0], [0 0 0 1 0 0] and [0 0 0 0 0 1].

Say X={4,6,4,2,3,3}. The MLE of $x_i$ i.e. side i shows up, can be given by:

$p_i = \frac{\sum_{t=1}^N x_i^t}{N}$.

Estimating the Parameters of a Gaussian Distribution

The MLE for the mean m is $\frac{\sum_t x^t}{N}$ and the MLE for the variance $\sigma^2$ is $\frac{\sum_t\,(x^t-m)^2}{N}$.

However, if we divide by N-1 instead of N (for variance), it is called the unbiased estimate.