How to Build a Neural Network from Scratch: A Step-by-Step Guide
Building Neural Networks from the Grounds Up: A Hands-on Exploration of the Math Behind the Magic
If you have studied linear algebra, calculus and statistics in high school and/or college, chances are that you have more of a head-start to machine learning and deep learning than you realize. Let us explore the mathematics and intuition behind Logistic Regression and a Multi-Layered Neural Network.
For the purpose of this article, we will use the telecom churn dataset on Kaggle to learn how to predict customer churn. We will observe if we can get the performance of our bottoms-up model comparative to the implementation using TensorFlow/Kera. If you want to learn how to implement Logistic Regression, Random Forest, XGBoost and TensorFlow to predict and prevent customer churn, then read my article here. I would suggest to read the first part of that article either way to understand the dataset as I will not spend much time on data exploration in this article.
What is Logistic Regression?
Before we dive into a Neural Network, let’s first look at how a simple Logistic Regression model works as the intuition from Logistic Regression will help you understand how a Neural Network works. Logistic Regression is a classification algorithm. In our example we will implement a binary classification model, where we aim to classify data points into one of the two classes (1 = churn and 0 = not churned).
You can think of a simple logistic regression model as below:
Let’s look at the various components of the model and the math behind it:
Linear transformation:
This is where the model taken input from all the input features and performs a linear transformation. For a single example (that is what the subscript i represents in the example model above), this would correspond to adding a weight to each example. We then also add a bias term, that is independent of the input feature. The weights represent how much an input feature contribute to the eventual prediction, while the bias represents an additional learnable parameter that is independent of the input. It allows the network to shift the activation function’s output.
For a single example in the training set, the equation would be:
z⁽ⁱ⁾ = w₁⁽ⁱ⁾x₁⁽ⁱ⁾ + w₂⁽ⁱ⁾x₂⁽ⁱ⁾ + w₃⁽ⁱ⁾x₃⁽ⁱ⁾ + b
This equation should be familiar to you from linear algebra. Now to perform this function across the entire training set, you can either run a loop or borrowing another concept from your linear algebra class, we can represent all our input features and weights in vector format.
If you were wondering, when will you ever utilize your matrix multiplication skills learned during your linear algebra class — well, this would be a great time. We perform vectorization as that is much faster for the computer to process then to run a loop.
The vectorized equation for the above is listed below.
Z = WᵀX + b
The superscript T represents that we would transpose the matrix. If you recall the dot product from your linear algebra class, it incorporates multiplying each element of your row of first matrix to each element of column of your second matrix. Therefore the columns of the first matrix should align to the rows of the second matrix, which is why we transpose W. See example below:
Activation function:
Once, we have performed the linear transformation, we then pass our result Z through an activation function to introduce non-linearity into the output. For Logistic Regression, we will use a Sigmoid activation function. The sigmoid function takes any real-valued input and maps it to a value between 0 and 1. It does not always map it to zero and we will use a threshold to make the predicted values 0 or 1 — this is also referred to as a decision boundary in Logistic Regression.
Cost function:
The cost function is essentially a summation over all the losses for the training example, where the loss is simply the difference between the prediction value ŷ to the actual value of y.
This is where if you recall your statistics classes, you would have calculated something called a Mean Squared Error for Linear Regression, which is calculated as below:
J = 1/m * ∑(y — ŷ )²
In Logistic Regression, as we are dealing with binary classifications we cannot simply use the formula above and therefore we will use a slightly modified formula called a binary cross-entropy cost function:
J(w,b) = -1/m * ∑((y*log ŷ) + (1-y)*log(1- ŷ))
We use a negative cost function as we need to maximize the probability of minimizing the loss function. This is where the magic of machine learning comes in. We will use gradient descent to optimize our cost function by adjusting the weights and bias.
Gradient descent:
At the base of any machine learning/deep learning method, is the optimization function. While there are many optimization algorithms, with the most commonly used optimization function being Adam, it is important that you understand the simple gradient descent as the other optimization algorithms are variants to the simple gradient descent. All right, let’s revise your high school/college calculus.
To calculate gradient, we need to calculate the slope or derivative of the curve. Consider the example below. A derivative of a function at a specific point represent the the instantaneous rate of change of the function at that point. Calculating the derivative requires you to use differentiation, i.e. it involves calculating the rate of change of a function with respect to its independent variables.
Gradient is simply the generalization of the derivative functions with multiple variables. It represents the vector of partial derivatives of a multivariate function with respect to each of its variables. See example using a computation chart below.
The goal of any machine learning algorithm is to minimize this gradient. We adjust the weights and bias iteratively to minimize the cost function. This process is called back propagation, while the process of linear transformation and activation function is called forward propagation.
Get Arsalan Pardesi’s stories in your inbox
Join Medium for free to get updates from this writer.
Toward the end of the article, we have the implementation of Logistic Regression in Python. Let’s move to Neural Networks now.
What are Neural Networks?
Neural Networks are composed of interconnected neurons. Each neuron receives input, performs a computation, and produces an output. The inputs are multiplied by weights, summed up, and passed through an activation function. In addition to the weights the Neural Networks also incorporate bias terms, which represents an additional learnable parameter that is independent of the input. Sounds familiar? — because it is. Let’s use our intuition from the Logistic Regression and see how a Neural Network works.
Similar to our Logistic Regression, there is an input layer and an output layer. In the middle, we have multiple hidden layers. We perform similar calculations in these hidden layers, i.e. each node in the hidden layer performs a linear transformation (using a similar equation) and then applies an activation function to calculate an output. That output is then fed into the nodes of the following hidden layer which also performs similar calculations.
Given the number of hidden layers and the depth of hidden layers, it allows the model to learn better and create connections that a simple Logistic Regression model was not capable for. We also have the flexibility to define different activation functions for different layers.
Similar to Logistic Regression, we will perform the following steps:
- Perform linear transformation
- Apply activation functions
- Apply forward propagation (through all the hidden layers all the way to the output)
- Calculate cost
- Apply backpropagation and gradient descent
- Update the weights and bias
- Continue iterating to minimize the cost function
Let’s dive into Python implementation
Recap of the data — the data is labeled customer churn data for a telecom business.
Logistic Regression from scratch in Python
Data cleansing and preparation. Refer to the article referenced at the start of this article for data exploration and understanding the data cleansing steps.
#----main libraries----
import numpy as np
import pandas as pd
import copy
import scipy
#----plotting libraries----
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline
# Load data - we will load the data and perform the same cleaning steps as before
tel_cus_df = pd.read_csv("./data/WA_Fn-UseC_-Telco-Customer-Churn.csv")
# Data cleansing and preparation - refer to the old notebook for explanation
#Total charges is type object. Check for empty values and covert to a numeric data type
tel_cus_df['TotalCharges'] = tel_cus_df['TotalCharges'].replace(' ', np.nan, regex=True)
tel_cus_df['TotalCharges'] = pd.to_numeric(tel_cus_df['TotalCharges'])
tel_cus_df.isnull().sum()
# create number of months for contract type to fill null totalcharges
tel_cus_df['ContractMonths'] = np.where(tel_cus_df['Contract']=="Month-to-month", 1, 0) + np.where(tel_cus_df['Contract']=="One year", 12, 0) + np.where(tel_cus_df['Contract']=="Two year", 24, 0)
tel_cus_df['ContractMonths'] = pd.to_numeric(tel_cus_df['ContractMonths'])
tel_cus_df.loc[tel_cus_df['TotalCharges'].isnull(), ['Contract','ContractMonths','MonthlyCharges','TotalCharges']]
tel_cus_df['cal_TotalCharges'] = tel_cus_df['ContractMonths'] * tel_cus_df['MonthlyCharges']
tel_cus_df['TotalCharges'] = np.where(tel_cus_df['TotalCharges'].isnull(), tel_cus_df['cal_TotalCharges'], tel_cus_df['TotalCharges'])
tel_cus_df.isnull().sum()
# Drop CustomerID and other columns not required
tel_cus_df.drop(['customerID','ContractMonths','cal_TotalCharges'], axis=1, inplace=True)
tel_cus_df.columns.values
# importing sklearn for processing data
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder
from sklearn.preprocessing import MinMaxScaler
le = LabelEncoder()
tel_cus_df['Churn'] = le.fit_transform(tel_cus_df['Churn'])
# converting categorical items into dummy variables
df_dummy = pd.get_dummies(tel_cus_df)
df_dummy.head()
# creating x and y variables
X = df_dummy.drop("Churn", axis = 1)
y = df_dummy['Churn']
# creating train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state = 100, test_size = 0.2)
#lets check the shape of our training and test set
m_train = X_train.shape[0]
m_test = X_test.shape[0]
num_x = X_train.shape[1]
print ("Number of training examples: m_train = " + str(m_train))
print ("Number of test examples: m_test = " + str(m_test))
print ("Number of features: num_x = " + str(num_x))
print ("X_train shape: " + str(X_train.shape))
print ("y_train shape: " + str(y_train.shape))
print ("X_test shape: " + str(X_test.shape))
print ("y_test shape: " + str(y_test.shape))
#Change shapes to have X to (nx, m) and y to (1,m) and dataframes to numpy arrays
train_set_x = X_train.to_numpy().T
test_set_x = X_test.to_numpy().T
train_set_y = y_train.values.reshape(y_train.shape[0], 1).T
test_set_y = y_test.values.reshape(y_test.shape[0],1).T
print ("train_set_x shape: " + str(type(train_set_x)) + " " + str(train_set_x.shape))
print ("train_set_y shape: " + str(type(train_set_y)) + " " + str(train_set_y.shape))
print ("test_set_x shape: " + str(type(test_set_x)) + " " + str(test_set_x.shape))
print ("test_set_y shape: " + str(type(test_set_y)) + " " + str(test_set_y.shape))Logistic regression code starts now:
# Defining a function to calculate sigmoid
def sigmoid(z):
s = 1. / (1+np.exp(-z))
return s
# Initializing parameters w and b
def initialize_with_zeros(dim):
w = np.zeros((dim,1))
b = 0.0
return w, b
# Define forward propagation and backward propagation
def propagate(w, b, X, Y):
m = X.shape[1]
#forward propagation
A = sigmoid(np.dot(w.T, X) + b)
cost = (-1./m) * np.sum((Y * np.log(A)) + ((1-Y)* (np.log(1-A))))
#backward propagation
dz = A - Y
dw = (1./m) * np.dot(X, dz.T)
db = (1./m) * np.sum(dz)
cost = np.squeeze(np.array(cost))
grads = {"dw": dw,
"db": db}
return grads, cost
# optimization function - Learn w and b by minimizing the cost function J
def optimize(w, b, X, Y, num_iterations=100, learning_rate=0.009, print_cost=False):
w = copy.deepcopy(w)
b = copy.deepcopy(b)
costs = []
for i in range(num_iterations):
grads, cost = propagate(w, b, X, Y)
dw = grads["dw"]
db = grads["db"]
w = w - (learning_rate * dw)
b = b - (learning_rate * db)
# print cost for every 100 iteration
if i % 100 == 0:
costs.append(cost)
if print_cost:
print("Cost after iteration %i: %f" %(i, cost))
params = {"w": w,
"b": b}
grads = {"dw": dw,
"db": db}
return params, grads, costs
# predicting the values
def predict(w, b, X):
m = X.shape[1]
Y_prediction = np.zeros((1,m))
w = w.reshape(X.shape[0],1)
A = sigmoid(np.dot(w.T, X) + b)
for i in range(A.shape[1]):
if A[0,i] > 0.5:
Y_prediction[0,i] = 1
else:
Y_prediction[0,i] = 0
return Y_prediction
# Defining the model
def model(X_train, Y_train, X_test, Y_test, num_iterations=2000, learning_rate=0.005, print_cost=False):
mX_train = X_train.shape[0]
w, b = initialize_with_zeros(mX_train)
params, grads, costs = optimize(w, b, X_train, Y_train, num_iterations=num_iterations, learning_rate=learning_rate, print_cost=print_cost)
w = params["w"]
b = params["b"]
Y_prediction_test = predict(w, b, X_test)
Y_prediction_train = predict(w, b, X_train)
#print costs
if print_cost:
print("training accuracy: {} %".format(100 - np.mean(np.abs(Y_prediction_train - Y_train)) * 100))
print("test accuracy: {} %".format(100 - np.mean(np.abs(Y_prediction_test - Y_test)) * 100))
accuracy = 100 - np.mean(np.abs(Y_prediction_test - Y_test)) * 100
true_pos = np.multiply(Y_prediction_test, Y_test)
recall = np.squeeze(np.sum(true_pos, axis = 1) / np.sum(Y_test, axis = 1) * 100)[()]
precision = np.squeeze(np.sum(true_pos, axis = 1) / np.sum(Y_prediction_test) *100)[()]
results_dic = {"costs": costs,
"Y_prediction_test": Y_prediction_test,
"Y_prediction_train": Y_prediction_train,
"w": w,
"b": b,
"learning_rate": learning_rate,
"num_iterations": num_iterations,
"accuracy": accuracy,
"recall": recall,
"precision": precision}
return results_dic Lets train our model now:
LR = model(train_set_x, train_set_y, test_set_x, test_set_y, num_iterations=2000, learning_rate=0.005, print_cost=True)You will see an output similar to below:
Cost after iteration 0: 0.693147
Cost after iteration 100: 0.576174
Cost after iteration 200: 0.538158
Cost after iteration 300: 0.516368
Cost after iteration 400: 0.500736
Cost after iteration 500: 0.488704
Cost after iteration 600: 0.479173
Cost after iteration 700: 0.471486
Cost after iteration 800: 0.465195
Cost after iteration 900: 0.459982
Cost after iteration 1000: 0.455614
Cost after iteration 1100: 0.451917
Cost after iteration 1200: 0.448758
Cost after iteration 1300: 0.446039
Cost after iteration 1400: 0.443679
Cost after iteration 1500: 0.441618
Cost after iteration 1600: 0.439806
Cost after iteration 1700: 0.438204
Cost after iteration 1800: 0.436779
Cost after iteration 1900: 0.435506
training accuracy: 79.69471068512603 %
test accuracy: 78.14052519517388 %
As you can see our Logistic Regression model from scratch was able to replicate the performance of logistic regression implemented using Scikit-learn. Let us now look at the implementation of Neural Network in Python.
Neural Network from scratch in Python
#setting up random seed to get the same answer while working
np.random.seed(100)
"""
Similar to logistic regression, we will follow the steps below:
1) Defining the sigmoid function and ReLU
2) Initializing our parameters
3) Defining Forward propagation
4) Calculate cost
5) Defining Backward propagation
6) Defining Optimization function
7) Bringing it all together and training the model
"""
# sigmoid function
def sigmoid2(Z):
A = 1. / (1+np.exp(-Z))
cache = Z
return A, cache
#ReLU function - it is essentially a linear function that returns Z, if positive, otherwise zero
#We will use ReLU to activate hidden layers
def relu(Z):
A = np.maximum(0, Z)
assert(A.shape == Z.shape)
cache = Z
return A, cache
#backward propagation for sigmoid
def sigmoid2_backward(dA, cache):
Z = cache
s = 1. / (1+np.exp(-Z))
dZ = dA * s * (1-s)
assert (dZ.shape == Z.shape)
return dZ
#backward propagation for ReLU
def relu_backward(dA, cache):
Z = cache
dZ = np.array(dA, copy=True)
dZ[Z <= 0] = 0
assert (dZ.shape == Z.shape)
return dZ
# Initialize parameters
def initialize_parameters_deep(layers_dims):
np.random.seed(100)
parameters = {}
L = len(layers_dims)
for l in range(1, L):
parameters["W" + str(l)] = np.random.randn(layers_dims[l], layers_dims[l-1]) * np.sqrt(2/layers_dims[l-1])
parameters["b" + str(l)] = np.zeros((layers_dims[l],1))
assert(parameters["W" + str(l)].shape == (layers_dims[l], layers_dims[l-1]))
assert(parameters["b" + str(l)].shape == (layers_dims[l], 1))
return parameters
# linear forward propagation
def linear_forward(A, W, b):
Z = np.dot(W, A) + b
cache = (A, W, b)
return Z, cache
# linear forward activation
def linear_activation_forward(A_prev, W, b, activation):
if activation == "sigmoid":
Z, linear_cache = linear_forward(A_prev, W, b)
A, activation_cache = sigmoid2(Z)
elif activation == "relu":
Z, linear_cache = linear_forward(A_prev, W, b)
A, activation_cache = relu(Z)
cache = (linear_cache, activation_cache)
return A, cache
# Applying forward propagation over L layers
def L_model_forward(X, parameters):
caches = []
A = X
L = len(parameters) // 2
for l in range(1, L):
A_prev = A
A, cache = linear_activation_forward(A_prev, parameters["W" + str(l)], parameters["b" + str(l)], activation = "relu")
caches.append(cache)
AL, cache = linear_activation_forward(A, parameters["W" + str(L)], parameters["b" + str(L)], activation = "sigmoid")
caches.append(cache)
return AL, caches
# defining cost - cross-entropy cost
def compute_cost(AL, Y):
m = Y.shape[1]
cost = (-1/ m) * np.sum((Y * np.log(AL)) + ((1-Y)*(np.log(1-AL))))
cost = np.squeeze(cost)
return cost
# linear backward propagation
def linear_backward(dZ, cache):
A_prev, W, b = cache
m = A_prev.shape[1]
dW = (1./m) * np.dot(dZ, A_prev.T)
db = (1./m) * np.sum(dZ, axis=1, keepdims = True)
dA_prev = np.dot(W.T, dZ)
return dA_prev, dW, db
# linear backward activation
def linear_activation_backward(dA, cache, activation):
linear_cache, activation_cache = cache
if activation == "relu":
dZ = relu_backward(dA, activation_cache)
dA_prev, dW, db = linear_backward(dZ, linear_cache)
elif activation == "sigmoid":
dZ = sigmoid2_backward(dA, activation_cache)
dA_prev, dW, db = linear_backward(dZ, linear_cache)
return dA_prev, dW, db
# applying backward propagation over L layers
def L_model_backward(AL, Y, caches):
grads = {}
L = len(caches)
m = AL.shape[1]
Y = Y.reshape(AL.shape)
#for L-th layer
dAL = - (np.divide(Y, AL) - np.divide(1-Y, 1-AL))
current_cache = caches[L-1]
dA_prev_temp, dW_temp, db_temp = linear_activation_backward(dAL, current_cache, activation="sigmoid")
grads["dA" + str(L-1)] = dA_prev_temp
grads["dW" + str(L)] = dW_temp
grads["db" + str(L)] = db_temp
#for all prior layers
for l in reversed(range(L-1)):
current_cache = caches[l]
dA_prev_temp, dW_temp, db_temp = linear_activation_backward(grads["dA" + str(l+1)], current_cache, activation="relu")
grads["dA" + str(l)] = dA_prev_temp
grads["dW" + str(l+1)] = dW_temp
grads["db" + str(l+1)] = db_temp
return grads
def update_parameters(params, grads, learning_rate):
parameters = params.copy()
L = len(parameters) // 2
for l in range(L):
parameters["W" + str(l+1)] = parameters["W" + str(l+1)] - (np.multiply(learning_rate, grads["dW" + str(l+1)]))
parameters["b" + str(l+1)] = parameters["b" + str(l+1)] - (np.multiply(learning_rate, grads["db" + str(l+1)]))
return parameters
#setting layers for the NN
layers_dims = [45, 16, 8, 4, 1]
# defining the model
def L_layer_model(X, Y, layers_dim, learning_rate = 0.0075, num_iterations = 3000, print_cost = False):
np.random.seed(100)
costs = []
parameters = initialize_parameters_deep(layers_dims)
for i in range(0, num_iterations):
AL, caches = L_model_forward(X, parameters)
cost = compute_cost(AL, Y)
grads = L_model_backward(AL, Y, caches)
parameters = update_parameters(parameters, grads, learning_rate)
if print_cost and i % 100 == 0 or i == num_iterations -1:
print("Cost after iteration {} : {}".format(i, np.squeeze(cost)))
if i % 100 == 0 or i == num_iterations:
costs.append(cost)
return parameters, costs, learning_rate
# predicting results
def predict(X, y, parameters):
m = X.shape[1]
n = len(parameters) //2
p = np.zeros((1, m))
accuracy = []
recall = []
precision = []
probas, caches = L_model_forward(X, parameters)
for i in range(0, probas.shape[1]):
if probas[0,i] > 0.5:
p[0,i] = 1
else:
p[0,i] = 0
accuracy = np.sum((p == y)/m)*100
true_pos = np.multiply(p, y)
recall = np.squeeze(np.sum(true_pos, axis = 1) / np.sum(y, axis = 1) * 100)
precision = np.squeeze(np.sum(true_pos, axis = 1) / np.sum(p) *100)
print("Accuracy: " + str(np.sum((p == y)/m)*100))
print("Precision: " + str(precision))
print("Recall: " + str(recall))
return p, accuracy, recall, precision
# using Smote to balance data
from imblearn.over_sampling import SMOTE
sm_train_set_x = train_set_x.T.copy()
sm_train_set_y = train_set_y.T.copy()
print("Before OverSampling, counts of label '1': {}".format(np.sum(train_set_y == 1)))
print("Before OverSampling, counts of label '0': {} \n".format(np.sum(train_set_y == 0)))
sm = SMOTE(random_state = 2)
sm_train_set_x, sm_train_set_y = sm.fit_resample(sm_train_set_x, sm_train_set_y.ravel())
print('After OverSampling, the shape of X_train_sm: {}'.format(sm_train_set_x.shape))
print('After OverSampling, the shape of y_train_sm: {} \n'.format(sm_train_set_y.shape))
print("After OverSampling, counts of label '1': {}".format(np.sum(sm_train_set_y == 1)))
print("After OverSampling, counts of label '0': {}".format(np.sum(sm_train_set_y == 0)))
#correcting data shape
sm_train_set_x = sm_train_set_x.T
sm_train_set_y = sm_train_set_y.reshape(sm_train_set_y.shape[0], 1).T
print ("sm_train_set_x shape: " + str(type(sm_train_set_x)) + " " + str(sm_train_set_x.shape))
print ("sm_train_set_y shape: " + str(type(sm_train_set_y)) + " " + str(sm_train_set_y.shape))
print ("test_set_x shape: " + str(type(test_set_x)) + " " + str(test_set_x.shape))
print ("test_set_y shape: " + str(type(test_set_y)) + " " + str(test_set_y.shape))Let’s now train our model:
# Training the model
parameters, costs, learning_rate = L_layer_model(sm_train_set_x, sm_train_set_y, layers_dims, num_iterations = 5000, learning_rate = 0.5, print_cost = True)You will see an output similar to below:
Cost after iteration 0 : 0.8572924818863291
Cost after iteration 100 : 0.47659100157570944
Cost after iteration 200 : 0.46609928353897256
Cost after iteration 300 : 0.4557107007692631
Cost after iteration 400 : 0.44987292179345945
Cost after iteration 500 : 0.44537055880293525
Cost after iteration 600 : 0.43835540121111033
Cost after iteration 700 : 0.4326959244164279
Cost after iteration 800 : 0.4338103808173082
Cost after iteration 900 : 0.42459768784053226
Cost after iteration 1000 : 0.4176331166282963
Cost after iteration 1100 : 0.4202514168798555
Cost after iteration 1200 : 0.4097172241175017
Cost after iteration 1300 : 0.40989156950154865
Cost after iteration 1400 : 0.4018158635670034
Cost after iteration 1500 : 0.3983160806523625
Cost after iteration 1600 : 0.39724161715279527
Cost after iteration 1700 : 0.3969395515985822
Cost after iteration 1800 : 0.38643397233568455
Cost after iteration 1900 : 0.39242895999507477
Cost after iteration 2000 : 0.3838637729547984
Cost after iteration 2100 : 0.3855860120515013
Cost after iteration 2200 : 0.38160263015760415
Cost after iteration 2300 : 0.38400085573415066
Cost after iteration 2400 : 0.37975935992092713
Cost after iteration 2500 : 0.38112222342668806
Cost after iteration 2600 : 0.3703163963590343
Cost after iteration 2700 : 0.3730233967547954
Cost after iteration 2800 : 0.3718907878223719
Cost after iteration 2900 : 0.3684439621152349
Cost after iteration 3000 : 0.3634103161797431
Cost after iteration 3100 : 0.3683369828889806
Cost after iteration 3200 : 0.361010973699081
Cost after iteration 3300 : 0.36847923425013596
Cost after iteration 3400 : 0.35697326877984636
Cost after iteration 3500 : 0.3626213087793491
Cost after iteration 3600 : 0.3574022316456651
Cost after iteration 3700 : 0.36001614497753653
Cost after iteration 3800 : 0.3639604823968065
Cost after iteration 3900 : 0.3575649734979984
Cost after iteration 4000 : 0.37053989344963106
Cost after iteration 4100 : 0.3534590866033876
Cost after iteration 4200 : 0.3615740406236957
Cost after iteration 4300 : 0.34662628690870306
Cost after iteration 4400 : 0.3583193044125404
Cost after iteration 4500 : 0.35453190697760345
Cost after iteration 4600 : 0.3541091728383913
Cost after iteration 4700 : 0.3635171968719434
Cost after iteration 4800 : 0.345996088688778
Cost after iteration 4900 : 0.34127086325617784
Cost after iteration 4999 : 0.350196665945362
The performance of the simple Neural Network is very close to the model implemented using TensorFlow. However, given the gap between train and dev/test results. It seems that the model has high variance and is over-fitting the training set. Adding regularization and dropout to the model might make it better. It is a very small dataset so it will likely over-fit the training set.
Conclusion
The performance of our Logistic Regression and the simple Neural Network was very close to ones we implemented using Scikit-learn and TensorFlow. Now that you have the intuition of how a Neural Network works, you will feel more comfortable in using TensorFlow and Pytorch to implement your Neural Networks in no time.
Access the full notebook here.
Credit to Prof. Andrew Ng and Deeplearning.ai for teaching the concepts in detail in the Deep Learning specialization.
