# Advice/Tricks and Issues to Train a Neural Network

## Advice/Tricks and Issues to Train a Neural Network

## Adaptive Learning Rate

The idea is to start with a higher learning rate and decrease it as time progresses.

## Momentum

$$\Delta w\_i^t = -\eta \frac{\partial E^t}{\partial w\_i} + \alpha \Delta w\_i^{t-1}$$

(t is time). $$\alpha$$ can be default or computed.

## Early Stopping

Too much training could cause overfitting. Stop training when the validation error starts to increase.