Perceptron — Delta Rule Python Implementation

In our previous post, we discussed about training a perceptron using The Perceptron Training Rule. In this blog, we will learn about The Gradient Descent and The Delta Rule for training a perceptron and its implementation using python.

Why Gradient Descent ?

As we have discussed earlier, the perceptron training rule works for the training samples of data that are linearly separable. Another limitation is that if there are multiple local minima, then the previous rule might converge to a local minima instead of global minima. To overcome these limitations, we gonna use gradient descent for training our perceptron.

How it works ?

The idea behind the gradient descent or the delta rule is that we search the hypothesis space of all possible weight vectors to find the best fit for our training samples. Gradient descent acts like a base for BackPropogation algorithms, which we will discuss in upcoming posts.

Delta Rule can be understood by looking it as a unthresholded perceptron, which is unlike the perceptron training rule where we use a weight treshold based on the other weights. The linear combination of weights and the inputs associated with them acts as an input to activation function same as in the previous one.

O(x)= w.x

Before going into the activation function, we need to know how do we calculate the training error in the weights in case of misclassification. In Gradient Descent, we commonly use half the squared difference between the ouput and obtained value as the training error for our hypotheses.

• D is set of training samples.
• t is the target ouput for training example ‘d’.
• o is the output of linear unit for training example ‘d’.

We would use the same unit step function as activation function for this example too.

Visualizing the Weight vectors and their Error values:

• w represents all possible weight vectors.
• J(w) represents the error values with respect to weight vectors.
• Parabolic path of error surface, which has a global minima at which the weight vectors are most suited for the training samples.

Derivation of Gradient Descent Rule

In Gradient Descent, we calculate the steepest descent along the error surface for finding local minima. For this we need to calculate the derivative of E with respect to the weight vector. This vector derivative is called Gradient of E with respect to weight vector w. The training rule can now be represented as :

where E(w) is given by differentiating the training errors of data samples with their corresponding weight vectors. Substituting the whole thing in the above equation gives us:

Differentiating the above equation (Fig 1.2)and substituting o = w.x in it would give us the result as follows:

Therefore, by substituting Fig 1.3 in Fig 1.1 we get the final weight updation rule of Gradient Descent:

We have arrived at our final equation on how to update our weights using delta rule. One more key difference is that, in perceptron rule we modify the weights after training all the samples, but in delta rule we update after every misclassification, making the chance of reaching global minima high.

Algorithm:

• Initialize the weight vector.
• For each training sample in dataset, apply the activation function and if any error occurs, update the weight according to the rule.
• Repeat it for finite number of epochs, to make it more accurate.

Implementation of Perceptron using Delta Rule in python

Wow, we entered our most interesting part. As perceptron is a binary classification neural network we would use our two-class iris data to train our perceptron. Our dataset contains 100 records with 5 features namely petal length, petal width, sepal length, sepal width and the class (species).

Get Tarun Nanduri’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Create a python file namely Perceptron.py and class called Perceptron with a constructor which initializes weight vector to an empty list.

class Perceptron: def __init__(self): self.weights=[]

Define an activation function which takes each row in our dataset and does a dot product of it with weight vector and return 1 or 0 depending on the value of dot product.

#activation function def activation(self,data): #initializing with threshold value activation_val=self.weights[0] activation_val+=np.dot(self.weights[1:],data) return 1 if activation_val>=0 else 0

Now we have our activation function and ready to check for an misclassification. Now lets implements the fit method which incase of any misclassification updates the weight vector according to the delta rule. The fit method takes an input of training data and the associated class values along with the learning rate ( η ) and no.of epochs.

def fit(self,X,y,lrate,epochs): #initializing weight vector self.weights=[0.0 for i in range(len(X.columns)+1)] #no.of iterations to train the neural network for epoch in range(epochs): print(str(epoch+1),"epoch has started...") for index in range(len(X)): x=X.iloc[index] predicted=self.activation(x) #check for misclassification if(y.iloc[index]==predicted): pass else: #calculate the error value error=y.iloc[index]-predicted #updation of threshold self.weights[0]=self.weights[0]+lrate*error #updation of associated self.weights acccording to Delta rule for j in range(len(x)): self.weights[j+1]=self.weights[j+1]+lrate*error*x[j]

Lets write a method predict which takes test values and predicts the outcome based on the weights of the neural network.

def predict(self,x_test): predicted=[] for i in range(len(x_test)): #prediction for test set using obtained weights predicted.append(self.activation(x_test.iloc[i])) return predicted

We are all set to drive our code for our iris dataset. Let’s take a second and implement accuracy method in our Perceptron class which returns accuracy depending on predicted values and original values.

def accuracy(self,predicted,original): correct=0 lent=len(predicted) for i in range(lent): if(predicted[i]==original.iloc[i]): correct+=1 return (correct/lent)*100

We always love to have a look on the updated weights in our neural network. Lets write a simple method get_weights which returns the weights of trained perceptron.

def getweights(self): return self.weights

Create another python file namely driver.py and import our iris.csv using pandas and split it into train and test data using train_test_split method from model_selection module of sklearn.

from Perceptron import Perceptron import pandas as pd from sklearn.model_selection import train_test_split #read data from .csv file data=pd.read_csv("iris.csv") data.columns=["petal_length","petal_width","sepal_length","sepal_width","class"] classes=data["class"] data=data.drop(columns="class") #splitting test and train data for iris x_train,x_test,y_train,y_test=train_test_split(data,classes) #training the percpetron model=Perceptron() model.fit(x_train,y_train,0.5,10) pred=model.predict(x_test) print("accuracy: ",model.accuracy(pred,y_test)) print("weights: ",model.getweights())

Yeah, we are all done. Lets run this driver.py code and have a look at our results😎😎.

Oh, we have got an accuracy of 100 percent on our test set. Feel free to train this perceptron on any other two-class data and check out the results. You can clone or download the whole source code and dataset in my github at https://github.com/TarunNanduri/Perceptron. Lets discuss about multilayer neural networks in our next blog. Until then cheers..🤞

Originally published at http://learnai1.home.blog on November 16, 2019.