Global error minimization

As we said in the previous article discussing the perceptron structure, the success of training a single layer perceptron is strictly correlated to the kind of input we are considering and the update algorithm used to modify weights. In the first case we can have problems due to the presence of complex patterns of input, reason why it is always recommendable to perform multiple training with different splits of input data. Regarding the second case, the update algorithm is based on global error and how to minimize it by modifying the weighted connections.

To understand it, we have to introduce some concepts:

• Input pattern

Figure 1: input pattern

where x are the input values and y are the expected outputs related to the k-th pattern

• Relative error

Figure 2: relative error

where e(n) = (d−y). For neuron we compute the difference between the ground truth d and the value returned from the network y related to the elaboration of the k-th pattern.

• Global error

Figure 3: global error

where w is the weight that we want to modify, and E is the error value to minimize.

Get Paola Benedetti’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Once introduced these concepts, we can define the weights update factor ∆w, as the description of the positive/negative contribution of weights variation to the global error:

Figure 4: weight update factor

∆w is the value we have to add to modify a single weight i of a certain neuron j in order to evaluate w(k + 1). δE and δw can be both directly or inversely proportional, defining the way we operate to modify weights in order to minimize the global error:

• δE / δw > 0 (DP), weight value has to be reduces
• δE / δw < 0 (IP), weight value has to be raised

The update formula is given by:

Figure 5: weight update formula I

Since the update of the j-th neuron’s weights does not effect the output of the other n-1 nodes, in order to modify all weights of a generic j-th neuron for a pattern k, we proceed by defining the weight variation as:

Figure 6: weight variation

This means that we can then write the update formula , Figure 5 as:

Figure 7: weight update formula II

where w(k) values of the weights vector corresponding to the neuron j, x is the pattern value at the k-th iteration and η(d − y) is the modification coefficient that remains the same for all weights and inputs of the neuron.

This formula is known as Delta Rule, it is a gradient descent learning rule used in the a single layer Perceptron architecture, in order to perform weights update according to the neuron input.