# Hyperparameter Tuning, Batch Normalization and Programming Frameworks

## Hyperparameter Tuning

As discussed earlier, hyperparameters control the parameters of a deep network.

It is, therefore, important to set the right values for these hyperparameters. Doing so can be a time-consuming process.

Some hyperparameters include $$\alpha$$ (the learning rate), $$\beta$$ (for the momentum algorithm), the number of layers, the number of hidden units, the mini-batch size, learning rate decay,$$(\beta_1, \beta_2, \epsilon)$$ for Adam Optimization etc.

In traditional ML, we had much fewer hyperparameters, allowing us to use grid search. However, in Deep Learning, we have a large number of hyperparameters and must instead perform a search over a random set of values for the hyperparameters. A _coarse-to-fine_ approach may be employed, where we first search over random values and then narrow the search to a region where more suitable values exist.

Note that we must use an appropriate scale while choosing hyperparameters randomly.

For example, to get a set of random values between 0.0001 and 1 i.e. between $$[10^{-4}, 10^0]$$, do:

```
r = -4 * np.random()  # random power between -4 and 0
x = 10**r  # random value between 10^-4 and 10^0
```

If we have limited computational resources, we must restrict ourselves to hyperparameter tuning on a single model over several hours/days. However, if we have sufficient computational resources, we can afford to try out different hyperparameter settings on models in parallel, and choose the one that works best.

## Batch Normalization

It was earlier discussed that normalizing the inputs could speed up training.

Batch normalization aims at normalizing the z values of each layer which then get passed through an activation function and become the input for the next layer of a neural network. This speeds up training.

Given some intermediate values $$z^{(1)}, z^{(2)}, z^{(3)}, ..., z^{(m)}$$:

$$\mu = \frac{1}{m}\sum_{i=1}^{m}z^{(i)}$$

$$\sigma^2 = \frac{1}{m}\sum_{i=1}^{m}(z^{(i)}-\mu)^2$$

$$z^{(i)}_{norm} = \frac{z^{(i)}-\mu}{\sqrt{\sigma^2 + \epsilon}}$$

$$\tilde{z}^{(i)} = \gamma z^{(i)}_{norm} + \beta$$

where $$\gamma, \beta$$ are learnable parameters that are used to ensure that z doesn't have zero mean and unit variance, which is caused by normalization.

(We use $$\epsilon$$ to avoid division by 0).

Note that batch normalization is usually applied on mini-batches and when we use batch normalization, we can eliminate the b values for each layer. We must, however, also calculate $$d\gamma, d\beta$$ during backpropagation and update $$\gamma, \beta$$ as well, while updating W values for each layer.

While testing, since we have only one test example at a time, we can't calculate mean and standard deviation. Instead, we must use $$\mu$$ and $$\sigma^2$$ estimated using an exponentially weighted average across mini-batches seen during training.

## Softmax Regression

Linear Regression is used for Binary Classification, whereas Softmax Regression can be used for multi-class classification.

Say we have C class labels. Softmax Regression must output C probabilities, one for each class.

So, in the last layer, we use the _softmax activation function_, which is as follows:

First calculate $$t = e^{z^{[L]}}$$

Then, $$a^{[L]}=\frac{t}{\sum t_i}$$

The class with the highest a value i.e. highest probability is the predicted class.

For Softmax Regression, we have the following loss and cost functions:

$$L(\hat{y}, y) = -\sum_{i=1}^{C}y_i log \hat{y}_i$$

$$J(w^{[1]}, b^{[1]},...) = \frac{1}{m}\sum_{i=1}^{m}L(\hat{y}^{(i)}, y^{(i)})$$

## Programming Frameworks

There are several deep learning frameworks that make it easier to apply deep learning, without having to implement everything from scratch. Some of them include:

* Caffe/Caffe2
* TensorFlow
* Torch
* Keras
* Theano
* CNTK
* DL4J
* Lasagne
* mxnet
* PaddlePaddle
