Advice/Tricks and Issues to Train a Neural Network

Advice/Tricks and Issues to Train a Neural Network

Adaptive Learning Rate

The idea is to start with a higher learning rate and decrease it as time progresses.

Momentum

$\Delta w_i^t = -\eta \frac{\partial E^t}{\partial w_i} + \alpha \Delta w_i^{t-1}$

(t is time). $\alpha$ can be default or computed.

Early Stopping

Too much training could cause overfitting. Stop training when the validation error starts to increase.