Training a Neuron

Batch Training

This is possible when all the data is already available.

Training is done using batches of the data.

Each time the weights are updated, compute gradient for the entire dataset.

Repeat until convergence:
    for all i in {1...d}
        w_i = w_i + eta*sum((r^t-y^t)x_i^t)

One pass through the dataset constitutes an epoch.

On-Line Training

Train using one example at a time. Update the weights only based on that example.

$w_i \leftarrow w_i + \eta (r^t-y^t)x_i^t$

This is more efficient than batch training.

Stochastic Training

The idea is to train in an on-line fashion.

Repeat until convergence of weights (or some other stopping condition):
    for each x^t (in some order):
        for all i in {1...d}
            w_i = w_i + eta*(r^t-y^t)x_i^t