<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jaleel Adejumo on Medium]]></title>
        <description><![CDATA[Stories by Jaleel Adejumo on Medium]]></description>
        <link>https://medium.com/@jaleeladejumo?source=rss-180bf3e26104------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*ZOwCyiShHAHzSgFD.</url>
            <title>Stories by Jaleel Adejumo on Medium</title>
            <link>https://medium.com/@jaleeladejumo?source=rss-180bf3e26104------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 19:30:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jaleeladejumo/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Regression Analysis from Scratch: Incorporating p-value Computation]]></title>
            <link>https://medium.com/@jaleeladejumo/regression-analysis-from-scratch-incorporating-p-value-computation-c8a16823e04f?source=rss-180bf3e26104------2</link>
            <guid isPermaLink="false">https://medium.com/p/c8a16823e04f</guid>
            <category><![CDATA[p-value]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[regression-analysis]]></category>
            <dc:creator><![CDATA[Jaleel Adejumo]]></dc:creator>
            <pubDate>Sat, 12 Aug 2023 22:57:10 GMT</pubDate>
            <atom:updated>2023-08-12T23:07:31.599Z</atom:updated>
            <content:encoded><![CDATA[<p>In a previous article, I explained the concept of gradient descent and its application in a linear model. If you’d like to delve deeper, you can access more information through this link. In this present article, I will guide you through the process of crafting your own version of a linear regression model similar to sklearn’s, complete with the incorporation of a p-value option.</p><p>In today’s machine learning landscape, many experts tend to overlook linear regression when constructing models. However, it’s important to recognize that sometimes, especially with limited datasets, linear regression outperforms more complex models. As a machine learning engineer, it’s crucial to understand that when faced with relatively modest data, starting with a linear regression model serves as an ideal baseline.</p><blockquote>“A Machine Learning Engineer that doesn’t know regression is just a farmer.Give him a field to cultivate” Jaleel Adejumo, 2023</blockquote><p>This article is structured as follows:</p><ul><li>Definition of Linear Regression</li><li>The Anatomy of linear regression</li><li>The draw back of the model</li><li>What is p-value and how it can be interpreted.</li><li>Implementation from scratch in python</li></ul><h4>So, What Exactly is a Linear Regression Model?</h4><p>Linear regression represents a fundamental type of supervised machine learning algorithm, renowned for its simplicity and predictive capabilities. However, for optimal effectiveness, certain fundamental assumptions must be satisfied. Chief among them is the presence of a linear relationship between the independent and target variables. Failure to meet this prerequisite could lead to inconsistent results. The image below showcases various relationships between the target (dependent) and feature (independent) variables. The first two relationships are well-suited for linear models, while the last necessitates some form of transformation or even the application of a polynomial model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/1*wUy64pYQq5AB7nvnw_Rhpw.png" /><figcaption><a href="https://medium.datadriveninvestor.com/linear-regression-and-its-mathematical-implementation-29d520a75ede">Image source</a></figcaption></figure><h4>Now you know what linear regression is. What is the goal of linear regression?</h4><p>The goal of simple linear regression is to estimate the parameters of the line that best describes the relationship between the variables and to use this line to make predictions about the dependent variable based on the independent variable. It is a common method in data analysis and is often used in fields such as economics, social sciences, and engineering.</p><p><strong>The two types of linear regression models</strong></p><p>Simple linear regression: Simple linear regression is a statistical method used to establish a linear relationship between a dependent variable (target) and a single independent variable (Feature) by fitting a straight line to the observed data points.</p><p>Multiple linear regression: Is a statistical method used to analyse the relationship between a dependent variable and multiple independent variables. It builds upon simple linear regression by allowing for more than one predictor variable. The goal of multiple regression is to estimate the effects of each independent variable on the dependent variable while controlling for the effects of the other variables.</p><p><strong>The Anatomy of linear regression</strong></p><p>The mathematical notation of linear regression is given in the image below:</p><p>Simple linear regression</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mc3RbslhEzryDuUR" /></figure><p>Multiple regression</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/0*cpmmEtYxtFkRjVB-" /></figure><p>Image source: Dataquest</p><h4><strong>The draw back of the Linear regression</strong></h4><p><strong>Susceptible to underfitting: </strong>Underfitting refers to a situation in machine learning where a model fails to adequately capture the underlying data patterns. This commonly occurs when the chosen hypothesis function is unable to effectively fit the data. Linear regression, for example, assumes a linear relationship between input and output variables, which may not accurately capture complex datasets. In real-life scenarios, where the relationships between variables are often nonlinear, a simple straight line may not adequately fit the data. Therefore, using a more complex function can lead to a more effective representation of the data. Consequently, linear regression models often exhibit low accuracy due to the limitations of their linear assumptions.</p><p><strong>Vulnerable to outliers: </strong>Outliers are exceptional or extreme values in a dataset that deviate significantly from the majority of data points. These outliers can have a substantial impact on the performance of machine learning models and frequently result in decreased accuracy. Linear regression is particularly sensitive to outliers, as they can heavily influence the model’s estimation and prediction process. It is crucial to appropriately address outliers before applying linear regression to a dataset in order to mitigate their adverse effects and enhance the model’s accuracy.</p><p><strong>Now you are familiar with linear regression right?</strong></p><p>In machine learning problems, the focus typically lies more on the contribution of a feature to a model in terms of its weight, rather than being deeply concerned with the feature’s level of significance. On the other hand, in the realm of statistical modeling, the significance level of a feature holds substantial importance, largely contingent on the associated p-value.</p><p><strong>You might find yourself questioning the concept of the p-value. Well, I’ve got you covered in the next subsection…</strong></p><h4><strong>P-value and how it can be interpreted.</strong></h4><p>In statistics, the p-value is a measure of the evidence against a null hypothesis. It quantifies the probability of obtaining results as extreme as or more extreme than the observed data, assuming that the null hypothesis is true. In simpler terms, the p-value assesses the strength of the evidence against a specific claim or hypothesis.</p><p>The interpretation of the p-value depends on the chosen significance level (often denoted as α). The significance level represents the threshold below which the p-value is considered statistically significant. Commonly used values for α include 0.05 (5%) and 0.01 (1%).</p><p><strong>Interpreting the p-value involves comparing it to the significance level:</strong></p><p>1. If the p-value is less than the significance level (p &lt; α), it is typically interpreted as strong evidence against the null hypothesis. In other words, the data suggests that the observed results are unlikely to have occurred by chance alone, assuming the null hypothesis is true. This may lead to rejecting the null hypothesis in favour of an alternative hypothesis.</p><p>2. If the p-value is greater than or equal to the significance level (p ≥ α), it is generally interpreted as weak evidence against the null hypothesis. In this case, the data does not provide sufficient evidence to reject the null hypothesis. However, it is important to note that failing to reject the null hypothesis does not necessarily mean the null hypothesis is true. It simply means that the evidence is not strong enough to support an alternative hypothesis.</p><p><strong>You’re familiar with linear regression and p-values, right?</strong></p><p>Let’s build a linear regression library that includes the ability to calculate p-values from scratch.</p><h4>Implementation from scratch in python</h4><pre>#Importing numpy and scipy<br>import numpy as np<br>from scipy.stats import t<br><br>#Regression class<br><br>class LinearRegression:<br>   def __init__(self):<br>       self.coefficients = None<br>       self.p_values = None<br><br><br>   def fit(self, X, y):<br>       # Add a column of ones to X for the intercept term<br>       one = np.ones((len(X))).reshape(len(X),1)<br>       X = np.hstack((one,X))<br><br><br>       # Calculate the coefficients using the normal equation<br>       self.coefficients = np.linalg.inv(X.T @ X) @ X.T @ y<br><br><br>   def p_values_cal(self, X, y):<br>       one = np.ones((len(X))).reshape(len(X),1)<br>       X = np.hstack((one,X))<br>       # Calculate the residuals<br>       y_pred = X @ self.coefficients<br>       residuals = y - y_pred<br><br><br>       # Calculate the residual sum of squares (RSS)<br>       RSS = np.sum(residuals ** 2)<br><br><br>       # Calculate the degrees of freedom<br>       n = X.shape[0]<br>       p = X.shape[1] - 1<br>       df = n - p - 1<br><br><br>       # Calculate the standard error of the coefficients<br>       XTX_inv = np.linalg.inv(X.T @ X)<br>       coef_se = np.sqrt(np.diagonal(XTX_inv) * RSS / df)<br>       # Calculate the t-statistic and p-value for each coefficient<br>       t_stat = self.coefficients / coef_se<br>       p_values = (1 - t.cdf(np.abs(t_stat), df)) * 2<br><br><br>       self.p_values = p_values<br>       return p_values<br><br><br>   def predict(self, X):<br>       # Add a column of ones to X for the intercept term<br>       one = np.ones((len(X))).reshape(len(X),1)<br><br><br>       X = np.hstack((one,X))<br><br><br>       # Calculate the predicted values<br>       y_pred = X @ self.coefficients<br><br><br>       return y_pred</pre><p><strong>Implementation</strong></p><pre>#Import data from the web<br>def get_data():<br> data_url = &quot;http://lib.stat.cmu.edu/datasets/boston&quot;<br> raw_df = pd.read_csv(data_url, sep=&quot;\s+&quot;, skiprows=22, header=None)<br> X = np.hstack([raw_df.values[::2, :], raw_df.values[1::2, :2]])<br> y = raw_df.values[1::2, 2]<br> return X,y<br><br>#Features and target data<br>X,y  = get_data()<br><br><br>X.shape</pre><p><strong>Data splitting</strong></p><pre>#Spliting data into train and test set<br><br>def split_data(X,Y, train_size):<br> # shuffle the data before splitting it<br> x_train_size = round(len(X)* train_size)<br> x_test_size = len(X) - x_train_size<br>  np.random.shuffle(X)<br> np.random.shuffle(Y)<br> X_train = X[:x_train_size]<br> X_test = X[x_train_size:]<br> y_train = y[:x_train_size]<br> y_test = y[x_train_size:]<br><br> return X_train, X_test, y_train, y_test<br><br><br><br><br># Split (X,y) into X_train, X_test, y_train, y_test<br>X_train, X_test, y_train, y_test= split_data(X,y,train_size= 0.8)<br><br></pre><p><strong>Training and Testing model</strong></p><pre>model = LinearRegression()<br><br>model.fit(X_train,y_train)<br><br>y_predict = model.predict(X_test)<br><br>p_value = model.p_valuess(X_train, y_train)<br><br>p_value</pre><p>Here you have it.</p><h4>Conclusion</h4><p>Linear regression is the foundation of machine learning and statistical modelling. This article discussed simple and multiple regression. Also, p-value was introduced and why using p-value. Furthermore, the algorithm was implemented from scratch with p-value integration.</p><h4>References</h4><p><a href="https://www.investopedia.com/ask/answers/060315/what-difference-between-linear-regression-and-multiple-regression.asp#:~:text=Multiple%20linear%20regression%20is%20a,linear%20regression%20is%20often%20better.">https://www.investopedia.com/ask/answers/060315/what-difference-between-linear-regression-and-multiple-regression</a></p><p><a href="https://www.geeksforgeeks.org/linear-regression-implementation-from-scratch-using-python/">https://www.geeksforgeeks.org/linear-regression-implementation-from-scratch-using-python/</a></p><p><a href="https://iq.opengenus.org/advantages-and-disadvantages-of-linear-regression/#:~:text=Since%20linear%20regression%20assumes%20a,t%20fit%20the%20data%20properly.">https://iq.opengenus.org/advantages-and-disadvantages-of-linear-regressio.</a></p><p><a href="https://www.analyticsvidhya.com/blog/2021/06/getting-started-with-machine-learning%E2%80%8A-%E2%80%8Aimplementing-linear-regression-from-scratch/">https://www.analyticsvidhya.com/blog/2021/06/getting-started-with-machine-learning%E2%80%8A-%E2%80%8Aimplementing-linear-regression-from-scratch/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c8a16823e04f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Gradient Descent From Scratch- Batch Gradient Descent, Stochastic Gradient Descent, and Mini-Batch…]]></title>
            <link>https://medium.com/@jaleeladejumo/gradient-descent-from-scratch-batch-gradient-descent-stochastic-gradient-descent-and-mini-batch-def681187473?source=rss-180bf3e26104------2</link>
            <guid isPermaLink="false">https://medium.com/p/def681187473</guid>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[gradient-descent-scratch]]></category>
            <category><![CDATA[gradient-descent-python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[momentum]]></category>
            <dc:creator><![CDATA[Jaleel Adejumo]]></dc:creator>
            <pubDate>Wed, 12 Apr 2023 00:51:44 GMT</pubDate>
            <atom:updated>2023-04-12T00:55:19.732Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Gradient Descent From Scratch- Batch Gradient Descent, Stochastic Gradient Descent, and Mini-Batch Gradient Descent.</strong></h3><p>In this article, I will take you through the implementation of Batch Gradient Descent, Stochastic Gradient Descent, and Mini-Batch Gradient Descent coding from scratch in python. This will be beginners friendly. Understanding gradient descent method will help you in optimising your loss during ML model training. Also, I will explain the impact of momentum in the learning process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3BIDGdKjh88SLF0YTBz5oA.png" /><figcaption>Gradient Descent</figcaption></figure><p>What you will learn in this article:</p><ul><li>What is loss and cost Function.</li><li>What is gradient descent.</li><li>How gradient descent works.</li><li>Types of gradient descent.</li><li>Impact of momentum in the learning process.</li><li>Implementation from scratch in python.</li></ul><h3><strong>Loss and cost Function</strong></h3><p>Loss and cost functions are used in machine learning to quantify the discrepancy between the actual values and the predicted value during the model training.Both are important components in machine learning.</p><p><strong>A loss function</strong> is used to evaluate the performance of a model on a single training observation. It takes the predicted output from the model and compares it to the true target output, quantifying the discrepancy between the two.</p><p><strong>Cost Function:</strong> It is commonly computed as the average or sum of the individual loss function values across the full training dataset. The cost function is a global measure of how well the model performs and is used by optimization methods such as gradient descent to iteratively adjust the model’s parameters during training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/349/1*cgAMuGVeioJPABk1-2Hlbg.png" /><figcaption>Cost Function</figcaption></figure><h3><strong>Gradient descent</strong></h3><p>Gradient is a linear approximation of a function. Gradient Descent is an iterative process of finding the local maximum and minimum of a function. As defined by IBM “Gradient descent is an optimization algorithm which is commonly-used to train machine learning models and neural networks.”</p><p>The gradient of the cost function represents the direction and magnitude of the steepest increase in the cost. By moving in the opposite direction of the gradient, which is the negative gradient, during optimization, the algorithm aims to converge towards the optimal set of parameters that provide the best fit to the training data. Gradient descent is a versatile optimization technique that can be utilised in various machine learning algorithms, such as linear regression, logistic regression, neural networks, and support vector machines.</p><p><strong>How gradient descent works.</strong></p><p>As said earlier, Gradient descent is a prominent optimization approach for minimising a model’s cost function. The cost function quantifies the difference between the expected and actual values based on the training data to assess the model’s performance.</p><p>The gradient descent formula consist is given below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/378/1*Htv890e-RRHududhNNP80A.png" /><figcaption>Gradient Descent</figcaption></figure><p>Firstly, you initialise a weight randomly and use it to estimate the model to obtain the value of the cost function. Find the derivative of the cost function with respect to the parameter of the model, this will give the gradient of the cost function. The gradient will point towards the steepest ascent direction. The movement of the gradient descent will continue until it reaches the point with the smallest possible loss function value.</p><h4><strong>Types of gradient descent.</strong></h4><p><strong>Batch Gradient Descent</strong></p><p>In batch gradient descent, the loss for all the points in the training set are averaged, and the model (weight) is updated only after evaluating all the training examples in a single training iteration.</p><p>The advantage of this method is that it gives a more accurate value for the gradient. One drawback of batch gradient descent is that the stable error gradient can sometimes lead to convergence at a suboptimal solution, failing to reach the best possible model performance.</p><p><strong>Stochastic Gradient Descent</strong></p><p>Stochastic Gradient Descent (SGD) is a simplified version of Gradient Descent (GD) that addresses some of its challenges. In SGD, the gradient is computed for only one randomly selected partition of the shuffled dataset during each iteration, instead of using the entire dataset. This modification significantly reduces computational time. However, because SGD iterates one observation at a time, When compared to GD, which evaluates the complete dataset in each iteration, it may produce noisier results.</p><p><strong>Mini-Batch Gradient Descent</strong></p><p>This is balanced between batch gradient descent and stochastic gradient descent. Here the data is splitted into small batches and then compute loss for each. The weights are updated after each batch.</p><p><strong>Impact of momentum in the learning process</strong></p><p>In order to speed up a model’s learning process, optimization methods like gradient descent frequently utilise momentum. It adds a new element that affects the direction in which the model parameters should be updated depending on the collected gradient data from earlier iterations.</p><p>The impact of momentum in the learning process can result in faster convergence, smoother optimization paths, enhanced escape from local optima, and improved generalisation ability, making it a useful technique for optimising the training process of machine learning models.</p><h3><strong>Implementation from scratch with python</strong></h3><p>A full training loop using gradient descent algorithm will follow these steps;</p><p>- initialise parameters- Run some number of epochs</p><p>- use parameters to make predictions</p><p>- Compute and store losses</p><p>- Compute gradients of the loss with respect to parameters</p><p>- Use gradients to update the parameters</p><p>- Do anything else or End!</p><p>Lets write a few lines of code that does a part of each of these steps before the main training loop</p><pre>def linear_function(X, theta):#Linear function<br>  assert X.ndim &gt; 1<br>  assert theta.ndim &gt; 1<br>  return np.dot(X, theta)<br><br>#Loss function (MSE)<br>def mean_squared_error(ytrue, ypred):<br>  return np.mean((ytrue - ypred)**2)<br><br><br>#Here we initialize the weight(parameter)<br>def initialize_theta(D): <br>  return np.zeros([D, 1])<br><br>#Here we compute the gradient of lost. This is gotten by finding the derivative of the loss with respect to the weight<br>def batch_gradient(X, y, theta):<br>  return -2.0 * np.dot(X.T, (y - linear_function(X, theta)))<br><br>#Updating function: gradient descent<br>def update_function(theta, grads, step_size):<br>  return theta - step_size * grads<br><br></pre><p><strong>Batch Gradient Descent</strong></p><p>The main loop for batch gradient descent. The function we created above were used accordingly.</p><pre>def train_batch_gradient_descent(X, y, num_epochs, step_size=0.1, plot_every=1):<br><br>  N, D = X.shape<br>  theta = initialize_theta(D)<br>  losses = []<br>  for epoch in range(num_epochs): # Do some iterations<br>    ypred = linear_function(X, theta) # make predictions with current parameters<br>    loss = mean_squared_error(y, ypred) # Compute mean squared error<br>    grads = batch_gradient(X, y, theta) # compute gradients of loss wrt parameters<br>    theta = update_function(theta, grads, step_size) # Update your parameters with the gradients<br>    <br>    losses.append(loss)<br>    print(f&quot;\nEpoch {epoch}, loss {loss}&quot;)<br>  return losses</pre><p>You can use the matplotlib library to plot the loss.</p><pre>plt.plot(losses)<br>plt.xlabel(&quot;epoch&quot;)<br>plt.ylabel(&quot;loss&quot;)<br>plt.title(&quot;training curve&quot;)<br></pre><p><strong>Stochastic Gradient Descent</strong></p><p>Stochastic gradient Descent unlike batch gradient descent, pick random sample(s) or subset of samples and updates parameters with their gradients.</p><p>Below, we will write code that gets a single sample and computes gradients for that sample.Its also important we shuffle our data.</p><pre>def per_sample_gradient(xi, yi, theta):<br>  return -2.0 * xi * (yi - linear_function(xi, theta))<br><br>def shuffle_data(X, y):<br>  N, _ = X.shape<br>  shuffled_idx = np.random.permutation(N)<br>  return X[shuffled_idx], y[shuffled_idx]</pre><pre>def train_with_sgd(X, y, num_epochs, step_size, plot_every=1):<br>  &quot;&quot;&quot;Train with stochastic gradient descent&quot;&quot;&quot;<br>  N, D = X.shape<br>  theta = initialize_theta(D)<br>  losses = []<br>  epoch = 0<br>  loss_tolerance = 0.001<br>  avg_loss = float(&quot;inf&quot;)<br><br>  while epoch &lt; num_epochs and avg_loss &gt; loss_tolerance:<br>    running_loss = 0.0<br>    shuffled_x, shuffled_y = shuffle_data(X, y)<br><br>    for idx in range(shuffled_x.shape[0]):<br>      sample_x = shuffled_x[idx].reshape(-1, D)<br>      sample_y = shuffled_y[idx].reshape(-1, 1)<br>      ypred = linear_function(sample_x, theta)<br>      loss = mean_squared_error(sample_y, ypred)<br>      running_loss += loss<br>      grads = per_sample_gradient(sample_x, sample_y, theta)<br>      theta = update_function(theta, grads, step_size)<br><br>    avg_loss = running_loss/ X.shape[0]<br>    losses.append(avg_loss)<br>    print(f&quot;Epoch {epoch}, loss {avg_loss}&quot;)<br><br>    epoch += 1<br><br>  return losses</pre><p>Also, I explained momentum in the introduction, this code will show how you can integrate momentum into SDG.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sIeYG6q7qtr2v2SnAMujgA.png" /><figcaption>Momentum formula</figcaption></figure><pre>def get_momentum(momentum, grad, beta):<br>  return beta * momentum + (1. - beta) * grad</pre><p>SDG with momentum</p><pre>def train_sgd_with_momentum(X, y, num_epochs, step_size, beta, plot_every=1):<br>  &quot;&quot;&quot;Train with stochastic gradient descent&quot;&quot;&quot;<br>  N, D = X.shape<br>  theta = initialize_theta(D)<br>  losses = []<br>  epoch = 0<br>  loss_tolerance = 0.001<br>  avg_loss = float(&quot;inf&quot;)<br><br>  while epoch &lt; num_epochs and avg_loss &gt; loss_tolerance:<br>    momentum = 0.0<br>    running_loss = 0.0<br>    shuffled_x, shuffled_y = shuffle_data(X, y)<br><br>    for idx in range(shuffled_x.shape[0]):<br>      sample_x = shuffled_x[idx].reshape(-1, D)<br>      sample_y = shuffled_y[idx].reshape(-1, 1)<br>      ypred = linear_function(sample_x, theta)<br>      loss = mean_squared_error(sample_y, ypred)<br>      running_loss += loss<br>      grad = per_sample_gradient(sample_x, sample_y, theta)<br>      momentum = get_momentum(momentum, grad, beta)<br>      theta = update_function(theta, momentum, step_size)#in this function the gradient will be replaced by the momentum<br><br>    avg_loss = running_loss/ X.shape[0]<br>    losses.append(avg_loss)<br>    print(f&quot;Epoch {epoch}, loss {avg_loss}&quot;)<br><br>    epoch += 1<br><br>  return losses</pre><p><strong>The Mini-batch gradient Descent</strong></p><p>Let’s now conclude with a mini-batch gradient descent.</p><p>Instead of computing gradients for the full dataset in this case, we split the data into batches, with the exception of the last batch, each batch having the size “batch_size” that we defined. Do you understand the reason?</p><p>Several factors should be noted in this case;</p><p>Since we only take a batch at a time, we must compute loss for the batch before averaging over all sample points.</p><pre>def minibatch_gradient_descent(X, y, num_epochs, step_size=0.1, batch_size=3, plot_every=1):<br>  N, D = X.shape<br>  theta = initialize_theta(D)<br>  losses = []<br>  num_batches = N//batch_size<br>  X, y = shuffle_data(X, y) # shuffle the data<br><br>  for epoch in range(num_epochs): # Do some iterations<br>    running_loss = 0.0<br>    <br>    for batch_idx in range(0, N, batch_size):<br>      x_batch = X[batch_idx: batch_idx + batch_size] # select a batch of features<br>      y_batch = y[batch_idx: batch_idx + batch_size] # and a batch of labels<br><br>      ypred = linear_function(x_batch, theta) # make predictions with current parameters<br>      loss = mean_squared_error(y_batch, ypred) # Compute mean squared error<br>      grads = batch_gradient(x_batch, y_batch, theta) # compute gradients of loss wrt parameters<br>      theta = update_function(theta, grads, step_size) # Update your parameters with the gradients<br>      running_loss += (loss * x_batch.shape[0]) # loss is mean for a batch, dividing by N_batch gives <br>                                                # us a sum for the batch so we can average later by diving <br>                                                # by the full data size<br>    <br>    avg_loss = running_loss/ N<br>    losses.append(avg_loss)<br>    print(f&quot;\nEpoch {epoch}, loss {avg_loss}&quot;)<br>  return losses<br></pre><h3><strong>Conclusion</strong></h3><p>Gradient Descent method can be used in other ML algorithms. It is used in backward propagation during training parameter to find the parameters of the models. Gradient Descent can be used to increase the accuracy of a machine learning model, making it a more potent tool for pattern recognition and/or prediction.</p><p>In this article, we discussed gradient descent in terms of its type, what is momentum, and it applications. We use Python to implement the algorithms from scratch at the conclusion of the explanation.</p><p>Follow me on:</p><p>Twitter: <a href="https://twitter.com/jel__ade">@jel__ade</a></p><p>LinkedIn: <a href="https://www.linkedin.com/in/abduljaleel-adejumo/">Adejumo Abduljaleel</a></p><p>GitHub: <a href="https://github.com/jelade">jelade</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=def681187473" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>