which reduces the error function), approaching a minimum. Due to the fact that 6x is a
vector it is described by two quantities: a magnitude and a direction. In order to choose
the direction of 6x the dot product is applied.
Definition of the dot product:
a (a, a2, ...., a,) b (bl, b2,....,bn) (3.4)
a -b =a I b Icos 6 (3.5)
Where I a I and I b I is the magnitude of a and b, respectively, and 0 is the angle between
the two vectors. The dot product may be applied to the following relation:
E(x+ x) ~E(x)+ VETx (3.6)
The last term in equation 3.6 may be written as:
=E BE QE
VE Ox -cl x+ 3- +....+- (3.7)
al 1 x2 x
This is the dot product of the gradient VE and the parameter change 6x. Therefore, a unit
vector 6x will produce a maximum change if it points in the direction of the gradient and
a minimum (negative change) if it points in the negative direction of the gradient. The
steepest descent method uses this information by choosing the 6x to point in the negative
gradient direction. The steepest descent technique proceeds as follows:
1. An initial set of parameters x is chosen, which results in an error E(x).
2. The gradient VE is calculated at point x. The parameter direction changes are chosen
to be in the direction of the negative gradient:
x = a VE (a is a positive constant) (3.8)
3. This process is iterated until the desired minimum is reached.
In an optimization problem it is usually desired to find a global minimum and avoid
getting trapped in a local minimum. Failing to find the global minimum is not just a