<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Joshua Anang on Medium]]></title>
        <description><![CDATA[Stories by Joshua Anang on Medium]]></description>
        <link>https://medium.com/@joshuaanang783?source=rss-963808c552ed------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*8bkyDyWHO3pYsBFx</url>
            <title>Stories by Joshua Anang on Medium</title>
            <link>https://medium.com/@joshuaanang783?source=rss-963808c552ed------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 11:19:25 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@joshuaanang783/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How Machine Learning models ACTUALLY learn.]]></title>
            <link>https://medium.com/@joshuaanang783/how-machine-learning-models-actually-learn-146d26f969ab?source=rss-963808c552ed------2</link>
            <guid isPermaLink="false">https://medium.com/p/146d26f969ab</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[mathematics]]></category>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[gradient-descent]]></category>
            <dc:creator><![CDATA[Joshua Anang]]></dc:creator>
            <pubDate>Mon, 21 Jul 2025 01:18:15 GMT</pubDate>
            <atom:updated>2025-07-21T01:18:15.685Z</atom:updated>
            <content:encoded><![CDATA[<h3>GRADIENT DECSCENT</h3><p>Have you ever wondered how machine learning models adjust their parameters (weights and biases) after making a wrong prediction during training? A method called Gradient Descent is used. That term must bring back horrors from math class, but this paper will be made easy.</p><p>We’ll use an example involving linear regression to make explanations much simpler. You don’t need to understand what linear regression is to read this paper. With this example involving linear regression, we will try to predict the heights of a group of people given their weights.</p><h3>Linear Regression</h3><p>In your reality, you can become accustomed to estimating the weight or height of that object without needing to make a measurement because of your continuous interactions with these objects provided both weight and height values. That’s how linear regression works, only it doesn’t involve images. With Linear regression, a linear graph is used to predict output values given input values on an x-axis using a ‘<strong>Line of best fit</strong>’.</p><figure><img alt="Linear regression plotted on a linear graph." src="https://cdn-images-1.medium.com/max/1024/1*K6L_N-Nm7nXOiMv5jt1vEg.png" /><figcaption>Image made by author. Demonstration of linear regression.</figcaption></figure><p>But this will all be explained in a bit.</p><p>STEPS INVOVLED WITH LINEAR REGRESSION.</p><ol><li>Plot points and estimate a <strong>line of best fit</strong> to help in value prediction.</li><li>Calculate residuals which are the difference between the predicted values and the expected values. (Observed — Predicted values).</li><li>Find the sum of the square of all the residuals previously calculated.</li><li>Use those values to find a new intercept and slope to adjust the estimated line to get the prefect line of best fit.</li></ol><p><strong>STEP 1 — Plotting points on graph and drawing a line of best fit.</strong></p><p>Imagine you had a linear graph containing weight points on the x axis, and height points on the y axis. This becomes your dataset. The provided weight and height of a specific individual can be plotted on the graph with the weight has the x value and the height as the y value. E.g. A weight and height value of (1, 1.5) will be plotted on the graph at that specific point. We do that for each value pair we have in our dataset.</p><p>(1, 1.5,)</p><p>(1.4, 2.3,)</p><p>(2.9, 3.7)</p><p>weights, heights</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K5MW_fvH4jJErHeizRqwVA.png" /></figure><p>This graph represents the plane for our linear regression model. But it’s incomplete. In order for the linear regression model to predict height values given ONLY the weights, we need to draw our<strong> line of best fit</strong>. This is a line that is drawn through the graph at the very center of all the points.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qbLzyFPEXuSl86mMWD1_2w.png" /></figure><p>How does it work? Given the weights (x value), we find the point on this ‘line of best fit’ which corresponds with the x value and matches a specific point on the y-axis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2Rz_vDIMxth3z72g8JOF5Q.png" /></figure><p>However, before training, the model wouldn’t know where the ideal point is to draw the line of best fit, so it makes an estimate. Every line requires a formula, correct? The equation y = mx + b represents the formula for all lines drawn on a graph.</p><p>The y in this formula represents the height values which the model is expected to predict, the ‘m’ represents the slope of the line, the x represents the input value which is the person’s weight on the graph, and the b represents the intercept.</p><p>The model initially needs to estimate an intercept and slope to begin drawing the line. In this scenario, we estimate the intercept to be 0, meaning the line will begin from the 0 point on the graph, and the slope to be 0.57, causing it to look slanted. During training, the model will use various techniques to get new intercept and slope values in order to draw the perfect ‘<strong>LINE OF BEST FIT</strong>’. We’ll discuss that later.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*46jEfTQou4eaG3L9Cqz-Rg.png" /><figcaption>Image representing line of best fit with initial estimates of both intercepts and slopes.</figcaption></figure><p><strong>STEP 2 — Predicting the y values (heights) and calculating residuals</strong></p><p>We plotted the data points on the graph earlier, but the model still doesn’t know which weight (x value) corresponds to which height (y value). Hence, it relies on the estimated line of best fit to make predictions.</p><p>With the new line of best fit we have, we can estimate what the model will think the height is given the weight using the formula for the line. We’re already aware that y=mx+b, hence we fill in each part of the formula with the values we already have.</p><p>E.g. Using y = (0.57*1) + 0, this gives us 0.57 as the model’s first predicted output for the height with a weight of 1, with the current ‘line of best fit’. The same method is repeated for all the x values or weight input values the model currently has.</p><p>CALCULATION</p><p>y = (0.57*1) + 0 = 0.57</p><p>y = (0.57*1.4) + 0 = 0.798</p><p>y = (0.57*2.9) + 0 = 1.653</p><p>We arrived at 0.57, 0.798 and 1.653 as our predicted values given the current line of best fit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YdJMHBxu0jMmcMXxggZRyA.png" /><figcaption>Linear regression model representing initial predictions with current line of best fit.</figcaption></figure><p>To move forward we need to follow another step which involves calculating the difference between the observed values and predicted values. These are known as residuals.</p><figure><img alt="Linear graph representing the difference between the observed points and predicted points on the regression model." src="https://cdn-images-1.medium.com/max/1024/1*SDldJN7GzuLsBGqoPGl4og.png" /><figcaption>Linear graph representing the difference between the observed points and predicted points on the regression model.</figcaption></figure><p>CALCULATION</p><p>Formula = Observed Height — Predicted Height</p><p>1.5–0.57 = 0.93</p><p>2.3–0.798 = 1.5</p><p>3.7–1.653 = 2.0</p><p>To find the new intercept which will be used to adjust the estimated line of best fit, we need to square each residual and add them all up. These are called the sum of squared residuals.</p><p>= 0.93² + 1.5² + 2.0² = 7.11 or 7.3 if computed as a single equation on a calculator.</p><p>To make things simple all the steps above can be expressed as one equation. This equation will represent the sum of squared residuals.</p><p>FORMULA</p><p>Sum of squared residuals = (Observed height — Predicted height (The formular for the line given the x value, slope value and intercept))</p><p>= (1.5 — (0.57*1 + 0)) ² + (2.3 — (0.57*1.4 + 0)) ² + (3.7 — (0.57*2.9 + 0)) ²</p><p>= 7.3.</p><p>Now let’s draw a new graph to plot the sum of squared residuals. This graph isn’t directly related to the linear regression graph we drew earlier. We’ll explain how these are used to adjust the line of best fit in a few.</p><p>After getting the first sum of squared residuals when the intercept was 0, and the slope 0.57, the resulting value will be plotted on this new graph representing the sum of squared residuals given a specific intercept value.</p><p>— — — — —</p><p><strong>NEW GRAPH REPRESENTING SUM OF SQUARED RESIDUALS</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ka2z725huzog-Upi7R9d8g.png" /><figcaption>Point 7.3 on graph representing sum of squared residuals.</figcaption></figure><p>The graph tells us when the intercept is 0, the sum of squared residuals is 7.3. To find the optimal value, we need to continuously plug in new estimates for the intercept to calculate new sum of squared residuals.</p><p>With this formula we can plug in any intercept value and calculate the sum of squared residuals given different intercept values continuously and that requires a lot of time. Later, we’ll discuss a more optimized technique.</p><p>You’re probably asking what the point of this graph is. The point of this graph is to get the optimal intercept value on the graph. The goal is to find the intercept that minimizes the sum of squared residuals. That intercept will be used to draw the line of best fit on the linear regression graph.</p><p>The optimal value is the value which produces the lowest point on the graph representing the sum of squared residuals. The final result of this graph would be numerous points in the shape of a curve, and it’s the model’s job to calculate the slope of each point on the curve of points to get the optimal slope value for this graph. (Not the regression graph). You’re probably wondering again how we calculate the intercept value with the most optimal or lowest slope. That will be explained later.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rM9HfETiPx0TfSqMDknvmQ.png" /><figcaption>Slope of each point on the graph representing the sum of squared residuals.</figcaption></figure><p>The next step involves calculating the slope of each point on the curve until we arrive at the intercept value which corresponds with the optimal slope. Then the intercept value corresponding with the point on the graph with the most optimal slope will be used to draw the final line of best fit.</p><p>Now you’re catching up. You now understand that new estimated intercept values will be plugged into the formular for the sum of squared residuals continuously, the resulting values will be plotted on this graph, and the intercept value corresponding with the most optimal slope will be used to draw the line of best fit.</p><p>To clarify, the whole point of this step is to continuously find the optimal intercept value, to draw the line of best fit.</p><p><strong>NOTE:</strong> The intercept determines how far up the ‘line of best fit’ goes on the y-axis.</p><h4>STEP 3 — Gradient descent</h4><p>Manually estimating and plugging new intercept values to find the intercept value with the optimal slope on the sum of squared residuals graph is inefficient. Hence, gradient descent was invented to solve this problem by significantly reducing training time.</p><p>Gradient descent involves a few new rules we must apply to our formulas and a few new steps. With gradient descent, continuously estimating random intercept values wouldn’t be necessary.</p><p>— — — — —</p><p><strong>STEPS APPLIED IN GRADIENT DESCENT.</strong></p><p>Everything will be explained.</p><ol><li>Find the derivative of the sum of squared residuals by applying the chain rule to that formula. This will directly give us the ability to calculate the slope of each point of the curve created on the graph representing the sum of squared residuals.</li><li>Multiply the found slope by a value called learning rate to get the STEP SIZE.</li><li>To estimate the next intercept, we subtract the new step size from the previous intercept.</li><li>Place that new intercept back into the formula for the sum of squared residuals and calculate the slope of that point.</li><li>The process is repeated until the found slope is noted to be the optimal value or the model reaches the max number of steps.</li></ol><p>— — — — —</p><p><strong>Sum of squared residuals.</strong></p><p>Now let’s head back to square 1. So far, everything we’ve calculated has been intended to help us acquire new intercept values to draw a new line of best fit for the regression model.</p><ul><li>We calculated the sum of squared residuals for random intercept values we estimated. Our first estimated intercept value was 0, and we arrived at 7.3 for the sum of squared residuals with that value. However, we haven’t found the slope at that point on the curve (7.3).</li><li>We mentioned that we’d have to plug in new estimated intercept values until the intercept value with the most optimal slope was found.</li><li>We mentioned that the target intercept value would be the point on the graph or the curve with the slope closest to 0. Even if the slopes are negative, we’ll still need to find a value closer to 0.</li></ul><p>The method involving continuously estimating new intercept values is inefficient. Thereafter, we would need to calculate the slope of each point of the curve on the graph continuously until we arrived at the lowest slope point.</p><p>With gradient descent, we can find the derivative of the sum of squared residuals and use that to compute the slopes right away to get the target intercept value much faster without needing to first calculate the sum of squared residuals to create points on the graph and then calculate the slope for each of those points.</p><p>Let’s see how it works.</p><p><strong>FINDING THE RIGHT INTERCEPT VALUE WITH GRADIENT DESCENT.</strong></p><p>We’ve already discussed that we need to find the intercept value corresponding with the optimal point on the graph representing the sum of squared residuals. Meaning, the point with the slope closest to 0.</p><p>Rather than just finding the sum of squared residuals, we will find the derivative of the sum of squared residuals to immediately get the slope of the point on the curve. (This helps bypass the need to find the sum of squared residuals, plotting it on the graph, and calculating the slope of that point, saving us time.)</p><p>— — — — —</p><p><strong>Formula for finding derivative of the intercept of the sum of squared residuals.</strong></p><p><strong>d/d intercept (Sum of squared residuals)</strong> = d/d intercept (Observed height — Predicted height (The formular for the line given the x value, slope value and intercept)) ² +</p><p>d/d intercept (Observed height — Predicted height (The formular for the line given the x value, slope value and intercept)) ² +</p><p>d/d intercept (Observed height — Predicted height (The formular for the line given the x value, slope value and intercept)) ²</p><p>The final value for this calculation will give us the slope of the point on the graph where the sum of squared residuals was 7.3, and the intercept was 0. With this step, we didn’t need to find 7.3 before finding the slope of that point. We only needed to get the derivative of the sum of squared residuals with respect to the intercept and immediately move on ahead to find the slope. Simple right?</p><p>Now let’s see it in action.</p><p>— — — — —</p><p><strong>SEEING IT IN ACTION</strong></p><p>You’re probably wondering what the derivative is. Take this formula for example.</p><p>Sum of squared residuals = (1.5 — (0.57*1 + 0)) ²</p><p>Before we find the derivative, we need to apply a mathematical method called ‘<strong>THE CHAIN RULE</strong>’. The chain rule allows us to differentiate composite functions by bringing the exponent down and then multiplying by the derivative of the inner function. But in this case, it involves moving the square to the front to be multiplied by the derivative of the sum of square residuals. Why you ask? Because the outer function is squared hence, its derivative is 2 times the inner.</p><p>d/d intercept Sum of squared residuals = 2(1.5 — (0.57*1 + 0))</p><p>Recap: 2(Observed height — Predicted Height (mx + b))</p><p>x is the weight value.</p><p>Since we’re finding the derivative with respect to the intercept, our focus will be on this part of the equation: — (0.57*1 + 0). We need to find the value which multiplies the intercept in the brackets. That value will the invisible -1 at the beginning of this point of the equation (-1) at -1(0.57*1 + 0). And with the intercept as 0, we found the derivative of the intercept to be -1.</p><p>d/d intercept Sum of squared residuals = 2(1.5 - (1) (0.57*1-0)).</p><p>The next step would be to multiply that -1, by the 2 in front of the equation.</p><p>d/d intercept Sum of squared residuals = -2(1 -(0.57*1 + 0)), and this becomes our final formular but only for the first point. Remember that we had 3 points on the graph and the sum of squared residuals involves the sum of all those points, but we used just 1 point for this example. Our final formular will look like this for all 3 points.</p><p>d/d intercept Sum of squared residuals = -2(1.5- (0.57*1 + 0)) + -2(2.3- (0.57*1.4 + 0)) + -2(3.7- (0.57*2.9 + 0))</p><p>Found slope = -8.95</p><p>Therefore, the gradient of the sum of squared residuals when the intercept is 0 is -8.95.</p><figure><img alt="Graph representing sum of squared residuals with -8.75 as the slope of the point on the imaginary curve." src="https://cdn-images-1.medium.com/max/1024/1*DPuiq4LEpizJJxgmT8Ee0Q.png" /><figcaption>Graph representing sum of squared residuals with -8.75 as the slope of the point on the imaginary curve.</figcaption></figure><p>In the previous example, our intercept was 0. But we need a new intercept to plug into the formula to keep moving forward. Initially, we mentioned that we needed to constantly estimate random intercept values until we arrived at the optimal point on this graph (the point with the optimal slope). But continuous estimation takes too long, so we’ll apply another method which gradient descent adopts and that involves calculating the <strong>NEW INTERCEPT</strong> by subtracting another value called the <strong>STEP SIZE</strong>, from the <strong>OLD INTERCEPT (0)</strong>. The old and new intercept in this case represent the line of best fit in the regression model.</p><p>The <strong>STEP SIZE</strong> helps us get closer to the point on the graph with the most optimal slope much faster without needing to constantly estimate random new intercept values for a very long time.</p><p>— — — — —</p><p><strong>STEP SIZE</strong> — What is the step size? The previous method of estimating random intercept values in order to find the point on the curve with the most optimal slope takes a long time. However, the step size helps us get to the point on the curve with the most optimal value much faster by taking large steps when the model is far from the most optimal slope, and much smaller steps when it gets closer.</p><figure><img alt="Graph representing big steps a regression model takes to get to the point on the graph with the most optimal slope much faster." src="https://cdn-images-1.medium.com/max/1024/1*lvW50VoQldNT5U--7hIeEQ.png" /><figcaption>Graph representing big steps a regression model takes to get to the point on the graph with the most optimal slope much faster.</figcaption></figure><p>The step size is acquired by multiplying the calculated slope (-8.95) by the learning rate. That’s a value that’s defined during the buildup of your model. It is mostly set to 0.001, but we’ll use 0.1 in this example.</p><p>STEP SIZE = Found slope x learning rate.</p><p>STEP SIZE = -8.95 * 0.1 = -0.895</p><p>New intercept = Old intercept — step size</p><p>New intercept = 0 — (-0.895) = 0.895</p><p>— — — — —</p><p>After finding the new intercept, we can try placing it back into our old regression model, and you’ll notice that the line will shift upwards significantly.</p><figure><img alt="Graph demonstrating the adjustment of a line of best fit in a regression model with a new intercept value." src="https://cdn-images-1.medium.com/max/1024/1*eUFPBvaQB4S3Eu_CT9Q0mg.png" /><figcaption>Graph demonstrating the adjustment of a line of best fit in a regression model with a new intercept value.</figcaption></figure><p>— — — — —</p><p>The next step would be to place our new intercept back into the formula to determine whether or not the found slope will be optimal.</p><p>d/d intercept Sum of squared residuals = -2(1.5 — (0.57*1 + 0.895)) + -2(2.3 — (0.57*1.4 + 0.895)) + -2(3.7 — (0.57*2.9 + 0.895))</p><p>Found slope = -3.588. Not exactly optimal.</p><p>STEP SIZE = Found slope x learning rate.</p><p>STEP SIZE = -3.588 * 0.1 = -0.3588</p><p>New intercept = Old intercept — Step size</p><p>New intercept = 0.895 — (-0.3588) = 1.2</p><p>— — — — —</p><p>Now let’s plug this <strong>new intercept</strong> into our regression model and observe the difference in our line of best fit.</p><figure><img alt="Graph demonstrating the adjustment of a line of best fit in a regression model with a new intercept value." src="https://cdn-images-1.medium.com/max/1024/1*qPINnSvXG2LIl6WkvERUZQ.png" /><figcaption>Graph demonstrating the adjustment of a line of best fit in a regression model with a new intercept value.</figcaption></figure><p>We continue this process until we find the most optimal slope value on the graph representing the sum of squared residuals. Meaning the slope value closest to 0 and use the intercept used to calculate that slope value as the intercept value to draw our final line of best fit.</p><p>So far, we’ve discussed how to adjust and find the right intercept value to draw our line of best fit. But we also need to adjust the slope or slant of our line of best fit in order to best find the points on our regression model.</p><p><strong>FINDING THE RIGHT SLOPE VALUE WITH GRADIENT DESCENT.</strong></p><p>Finding the right slope value involves the exact same steps used to find new intercepts. But this time, the resulting values will be interchanged. We’ll be finding the derivative of the sum of squared residuals with respect to the slope this time. Meaning, our final output will be a <strong>new slope</strong> rather than a <strong>new intercept</strong>.</p><p>Previous formula</p><p>d/d intercept Sum of squared residuals = -2(1.5 — (0.57*1 + 0)) + -2(2.3 — (0.57*1.4 + 0)) + -2(3.7 — (0.57*2.9 + 0))</p><p>slope = -8.95</p><p>With this formular we found the derivative with respect to the intercept. That derivative was -1, because it multiplies the expression in the brackets. We took that -1 and multiplied that by the 2 with which we applied the CHAIN RULE to, and we got -2.</p><p>— — — — —</p><p><strong>New formula to find the intercept of each point on the graph representing the sum of squared residuals.</strong></p><p>d/d slope Sum of squared residuals = 2*-1(1.5 — (0.57*1 + 0)) + 2*-1.4(2.3 — (0.57*1.4 + 0)) + 2*-2.9(3.7 — (0.57*2.9 + 0))</p><p>Found intercept = -17.93</p><p>Note: This isn’t the intercept we plug into the formula to adjust the line of best fit, the intercept we’ll plug into the formular to adjust the line of best fit would be the <strong>new intercept</strong> we calculated before.</p><p>This formula is a bit different. Finding the derivative with respect to the slope involves locating the values which multiply the slope in the brackets. Remember y = mx+b.</p><p>mx+b = -(0.57*1 + 0).</p><p>Hence our derivative for the first point which has an observed height of 1.5, and a weight of 1, will be -1. Because the -1 which multiplies the expression in the brackets will in turn be multiplied by the x value which multiplies the slope. We then take that value to the front of the equation and multiply it by the 2. Then we arrived at “2*-1(1.5 — (0.57*1 + 0))” for the first point.</p><p>The same method was applied to arrive at this expression for the second point: 2*-1.4(2.3 — (0.57*1.4 + 0)), and the third: 2*-2.9(3.7 — (0.57*2.9 + 0)).</p><p>After this step, we also calculate the step size for the derivative with respect to the slope just like we did for the intercept. Both formulas have their own step sizes.</p><p>STEP SIZE = Found intercept x learning rate</p><p>STEP SIZE = -17.93 * 0.1 = -1.793</p><p>New slope = Old slope — step size</p><p>New slope = 0.57 — (-1.793) = 2.3</p><p>We can plug this new slope back into our regression model and observe the difference.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pVGPd9hN5gsN4jjEpFdAmA.png" /></figure><p>We repeat the formular again but this time, with the new slope.</p><p>d/d slope Sum of squared residuals = 2*-1(1.5 — (2.3*1 + 0)) + 2*-1.4(2.3 — (2.3*1.4 + 0)) + 2*-2.9(3.7 — (2.3*2.9 + 0))</p><p>Found intercept = 21.40</p><p>STEP SIZE = Found intercept x learning rate</p><p>STEP SIZE = 21.40 * 0.1 = 2.140</p><p>New slope = old slope — step size</p><p>new slope = 2.3–2.14 = 0.24</p><p>This new slope can be plugged into our regression model to get a new line of best fit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KahK-uZd2FQiDwGcwqmY-A.png" /></figure><p>But the numbers we’ve found so far are a bit too large or too small. Reason being our learning rate was 0.1. In most cases, the learning rate is set to 0.001, but in our next example, our new learning rate will be 0.01.</p><p><strong>COMBINING BOTH SLOPE AND INTERCEPT FORMULARS INTO ONE.</strong></p><p>We’ve observed how to find new intercepts and new slopes independently. But gradient descent involves calculating for both at once. That’ll help us adjust both the slope and intercept of the line at the exact same time.</p><p>It involves the exact same formulars and nothing changes. But with this example we’ll change our parameters. We’ll begin with a learning rate of 0.01.</p><p>Intercept = 0</p><p>Slope = 0.57</p><p>Learning rate = 0.01</p><h4><strong>STEP 1. APPLY CHAIN RULE AND MOVE THE SQUARE TO THE FRONT.</strong></h4><p><strong>FINDING SLOPE WITH RESPECT TO INTERCEPT</strong></p><p>d/d intercept sum of squared residuals = (1.5 — (0.57*1 + 0))² + (2.3 — (0.57*1.4 + 0))² + (3.7 — (0.57*2.9 + 0))²</p><p>d/d intercept sum of squared residuals = 2(1.5 — (0.57*1 + 0)) + 2(2.3 — (0.57*1.4 + 0)) + 2(3.7 — (0.57*2.9 + 0))</p><p><strong>FINDING INTERCEPT WITH RESPECT TO SLOPE</strong></p><p>d/d slope Sum of squared residuals = (1.5 — (0.57*1 + 0)) + (2.3 — (0.57*1.4 + 0)) + (3.7 — (0.57*2.9 + 0))</p><p>d/d slope Sum of squared residuals = 2(1.5 — (0.57*1 + 0)) + 2(2.3 — (0.57*1.4 + 0)) + 2(3.7 — (0.57*2.9 + 0))</p><h4>STEP 2. FIND THE DERIVATIVE OF EACH PART.</h4><p><strong>FINDING SLOPE WITH RESPECT TO INTERCEPT</strong></p><p>d/d intercept sum of squared residuals = 2(1.5 — (0.57*1 + 0)) + 2(2.3 — (0.57*1.4 + 0)) + 2(3.7 — (0.57*2.9 + 0))</p><p>d/d intercept sum of squared residuals = 2(1.5–1(0.57*1–0)) + 2(2.3–1(0.57*1.4 + 0)) + 2(3.7–1(0.57*2.9–0))</p><p>d/d intercept sum of squared residuals = -2(1.5 — (0.57*1 + 0)) + -2(2.3 — (0.57*1.4 + 0)) + -2(3.7–1(0.57*2.9–0))</p><p>Derivative is the value multiplying the intercept in the brackets. -1 in this case and multiply that by the 2 we applied the chain rule to.</p><p>— — — — —</p><p><strong>FINDING INTERCEPT WITH RESPECT TO SLOPE</strong></p><p>d/d slope Sum of squared residuals = 2(1.5 — (0.57*1 + 0)) + 2(2.3 — (0.57*1.4 + 0)) + 2(3.7 — (0.57*2.9 + 0))</p><p>d/d slope Sum of squared residuals = 2x-1(1.5 — (0.57*1 + 0)) + 2x-1.4(2.3 — (0.57*1.4 + 0)) + 2x-2.9(3.7 — (0.57*2.9 + 0))</p><p>Derivative is the value multiplying the slope in the brackets. -1 in this case. But the derivative involving the slope includes the x value multiplying the slope as well. Hence, that -1, will multiply the slope.</p><h4>STEP 3. CALCULATE BOTH THE SLOPE AND INTERCEPT FOR THE GRAPH REPRESENTING SUM OF SQUARED RESIDUALS.</h4><p><strong>FINDING SLOPE WITH RESPECT TO INTERCEPT</strong></p><p>d/d intercept sum of squared residuals = -2(1.5 — (0.57*1 + 0)) + -2(2.3 — (0.57*1.4 + 0)) + -2(3.7–1(0.57*2.9–0))</p><p>Found slope = -8.95</p><p>Remember, this slope represents the slope on the graph representing the Sum of squared residuals not our regression model.</p><p>— — — — —</p><p><strong>FINDING INTERCEPT WITH RESPECT TO SLOPE</strong></p><p>d/d slope Sum of squared residuals = 2x-1(1.5 — (0.57*1 + 0)) + 2x-1.4(2.3 — (0.57*1.4 + 0)) + 2x-2.9(3.7 — (0.57*2.9 + 0))</p><p>Found intercept = -17.93</p><p>Remember, this intercept represents the intercept on the graph representing the Sum of squared residuals not our regression model.</p><h4><strong>STEP 4. FIND STEP SIZE AND CALCULATE THE NEW SLOPE AND INTERCEPT FOR OUR REGRESSION MODEL.</strong></h4><p><strong>FINDING SLOPE WITH RESPECT TO THE INTERCEPT</strong></p><p>d/d intercept Found Slope = -8.95</p><p>STEP SIZE = Found Slope x learning rate</p><p>STEP SIZE = -8.95 x 0.01</p><p>STEP SIZE = -0.0895</p><p>New intercept = Old intercept — STEP SIZE</p><p>New intercept = 0 — (-0.0895)</p><p>New intercept = 0.0895</p><p>This can be plugged into our regression model to adjust our line of best fit.</p><p>— — — — —</p><p><strong>FINDING INTERCEPT WITH RESPECT TO THE SLOPE</strong></p><p>d/d slope Found Intercept = -17.93</p><p>STEP SIZE = Found Slope x learning rate</p><p>STEP SIZE = -17.93 x 0.01</p><p>STEP SIZE = -0.1793</p><p>New slope = Old slope — STEP SIZE</p><p>New slope = 0.57 — (-0.1793)</p><p>New slope = 0.749</p><p>This can be plugged into our regression model to adjust our line of best fit.</p><h4>STEP 5. REPEAT FROM STEP 2 BUT PLUG IN THE NEW SLOPE AND NEW INTERCEPT TO REPLACE OLD VALUES.</h4><p>Replace 0.57, and 0 with 0.749 and 0.0895.</p><p>Formular will look like this: -2(1.5 — (0.749*1 + 0.0895)) + -2(2.3 — (0.749*1.4 + 0.0895)) + -2(3.7–1(0.749*2.9–0.0895))</p><p>d/d intercept sum of squared residuals = -2(1.5 — (0.749*1 + 0.0895)) + -2(2.3 — (0.749*1.4 + 0.0895)) + -2(3.7 — (0.749*2.9 + 0.0895))</p><p>Found slope = -6.5236</p><p>d/d slope sum of square residuals = 2x-1(1.5 — (0.749*1 + 0.0895)) + 2x-1.4(2.3 — (0.749*1.4 + 0.0895)) + 2x-2.9(3.7 — (0.749*2.9 + 0.0895))</p><p>Found intercept = -12.91</p><p>— — — — —</p><p><strong>FINDING SLOPE WITH RESPECT TO THE INTERCEPT</strong></p><p>d/d intercept Found Slope = -6.5236</p><p>STEP SIZE = Found Slope x learning rate</p><p>STEP SIZE = -6.5236 x 0.01</p><p>STEP SIZE = -0.065236</p><p>New intercept = Old intercept — STEP SIZE</p><p>New intercept = 0.0895 — (-0.065236)</p><p>New intercept = 0.15</p><p>This can be plugged into our regression model to adjust our line of best fit.</p><p>— — — — —</p><p><strong>FINDING INTERCEPT WITH RESPECT TO THE SLOPE</strong></p><p>d/d slope Found Intercept = -12.91</p><p>STEP SIZE = Found Slope x learning rate</p><p>STEP SIZE = -12.91 x 0.01</p><p>STEP SIZE = -0.1291</p><p>New slope = Old slope — STEP SIZE</p><p>New slope = 0.749 — (-0.1291)</p><p>New slope = 0.878</p><p>This can be plugged into our regression model to adjust our line of best fit.</p><p>You’re probably wondering when the steps come to an end. They come to an end when the step size reaches 0, or the number of steps taken reaches a maximum number. e.g. 1000 steps.</p><p>Congratulations. You made it to the end of this paper without being horrified. Now you understand how Machine Learning models, specifically linear regression models, learn to make correct predictions.</p><p>LinkedIn: <a href="https://www.linkedin.com/in/joshua-anang-611462285/">Joshua Anang | LinkedIn</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=146d26f969ab" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What makes CNNs so special? Exploring the true nature of images and how they work with CNNs.]]></title>
            <link>https://medium.com/@joshuaanang783/what-makes-cnns-so-special-exploring-the-true-nature-of-images-and-how-they-work-with-cnns-36adc103c4be?source=rss-963808c552ed------2</link>
            <guid isPermaLink="false">https://medium.com/p/36adc103c4be</guid>
            <category><![CDATA[image-processing]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[cnn]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Joshua Anang]]></dc:creator>
            <pubDate>Sat, 10 May 2025 09:00:32 GMT</pubDate>
            <atom:updated>2025-05-15T15:12:32.888Z</atom:updated>
            <content:encoded><![CDATA[<h3>The dark truth behind images and convolutional neural networks.</h3><h3>CONVOLUTIONAL NEURAL NETWORKS</h3><p>What are they?</p><p>You’ve probably watched sci-fi detective movies and seen characters use futuristic technology to detect human faces in an image and sometimes draw a square around their heads or bodies to help in their investigation.<br>Or you’ve probably seen how self-driving cars operate. They have a camera in front of them used to see the road ahead and avoid obstacles.</p><p>All of these technologies are powered by convolutional neural networks.</p><p>How do they work? They extract features from an image using a filter or kernel and pass that through a neural network to come up with a prediction or an output. But before we dive into convolutional neural networks, let’s explore how images truly work in detail.</p><h3>IMAGES</h3><p>Images are made up of tiny particles called pixels, just like how atoms make up the universe. These tiny particles are cells containing numbers. Yes, like cells in a matrix.<br>Images are actually matrices with a defined number of rows and columns, and each cell in the matrix of an image is a pixel. That pixel contains a number between 0 and 255. When you see the size of an image being 1920 x 1080, it essentially means the image is made up of a matrix of 1920 columns and 1080 rows, and each and every cell or pixel in this matrix contains a number between 0 and 255.</p><figure><img alt="Matrix diagram demonstrating what an image truly looks like to a computer." src="https://cdn-images-1.medium.com/max/1024/1*upScqO6ewIIVxIDQJHw-WA.png" /><figcaption>Matrix diagram demonstrating what an image truly looks like to a computer. [Image by Author]</figcaption></figure><h4>COLOUR IN IMAGES.</h4><p>Would you believe me if I told you that all images on your screen are made up of only the colours red, green, and blue? Blending them together creates the effect of white light or different colours in an image.</p><p>You understand that images are made up of matrices with pixels or cells containing values between 0 and 255. 0 represents the lowest colour intensity (black), and 255 is the highest colour intensity inside of each pixel.</p><p>Every image with colour actually contains three matrices of the same size. The first matrix represents the colour red, the second matrix represents the colour green, and the third matrix represents the colour ,blue each containing cells with values between 0 and 255 representing their colour intensity. If each cell in all three matrices has a value of 255, the image will appear white, and black if they all have a value of 0. These three matrices which give colour are known as channels.</p><figure><img alt="Diagram showing representation of channels in an image." src="https://cdn-images-1.medium.com/max/1024/1*SjqmlFrAxNQjA6JF6AFUvg.png" /><figcaption>Diagram showing representation of channels in an image. [Image by Author]</figcaption></figure><p>These three matrices are stacked on top of each other like transparent papers with lightbulbs inside of each pixel each producing their respective light colour.</p><figure><img alt="Diagram showing how all 3 colour channels of an image are stacked on top of each other." src="https://cdn-images-1.medium.com/max/1024/1*BkzlcZhU9RzSDI0GY7Csug.png" /><figcaption>Diagram showing how all 3 colour channels of an image are stacked on top of each other. [Image by Author]</figcaption></figure><p>Changing the numbers on each matrix from 0 to 255 creates different colours. For example, If I were to change all of the pixels in the blue and green matrix to 0, and all pixels in the red matrix to 255, the image would appear completely red. If I were to do the same for green and blue, the image would appear in their respective colours only.</p><p>But guess what, if I were to change a few pixel values in each coloured matrix to a random number, the image would produce a different colour like brown, yellow, orange, or cyan in the portion of the image where the pixel values for each matrix were changed.</p><h4>BACK TO CONVOLUTIONS!</h4><p>Imagine you were deciphering a 3000-year-old scroll, which contained secret information about the future which could save the world someday, or a 2000-year-old manuscript containing the blueprints for an ancient weapon capable of mass destruction. In order to gather information from both examples, you would use a magnifying glass to peek into the manuscripts and slowly write down all of the information you’ve been able to acquire in your own journal.</p><p>Convolutional Neural Networks work the same way. We have a matrix which functions like a magnifying glass also containing numbers. That matrix is slid across the image pixel by pixel, and information is extracted from the image by multiplying all numbers inside that matrix with the corresponding pixels in the image being analyzed and adding the results. That small matrix is called a filter or kernel.</p><figure><img alt="Image showing demonstration of an image as a matrix and a kernel or filter next to it." src="https://cdn-images-1.medium.com/max/1024/1*_9t8PC2ds-73s9N-iscQmg.png" /><figcaption>Image showing demonstration of an image as a matrix and a kernel or filter next to it. [Image by Author]</figcaption></figure><p>NOTE:</p><p>Before we begin the convolution process, there’s something else we need to do. Inside the three matrices or channels representing each colour, each pixel contains values between 0 and 255. These values are too big for a machine learning model to compute and could lead to slower training and issues like <strong>exploding gradients</strong>.</p><p>To avoid that, we need to scale every pixel value from a range of 0 and 255 to 0 and 1. How do we do that?<br>We divide each pixel value by 255 to scale it to a value between 0 and 1. This step is known as image preprocessing, and it’s a crucial step to follow when training a convolutional neural network.</p><figure><img alt="Image preprocessing step. Dividing all pixels by 255 to place their values between 1 and 0." src="https://cdn-images-1.medium.com/max/1024/1*RVtdNUUku3VRp0te0JT2kA.png" /><figcaption>Image preprocessing step. Dividing all pixels by 255 to place their values between 1 and 0. [Image by Author]</figcaption></figure><p>This preprocessing step is done for all 3 channels in the image.</p><figure><img alt="Image showing preprocessed channels of an image." src="https://cdn-images-1.medium.com/max/1024/1*UzzayuVmm_6gt_rwRZFhbg.png" /><figcaption>Image showing preprocessed channels of an image. [Image by Author]</figcaption></figure><p>In this paper, we’ll use a convolutional neural network to classify images. Imagine you had two images: a hot dog and a pizza. A convolutional neural network can determine which class each image belongs to. For example, an image of a pizza would be classified under the pizza class and an image of a hot dog would be classified under the hot dog glass.</p><p>In this paper, we’re going to train a convolutional neural network to classify two images, one containing the digit ‘1’ and the other containing the digit ‘2’.</p><p><strong>STEPS:</strong></p><p>Detailed explanations will follow.</p><ol><li>The image is first preprocessed by dividing all pixel values by 255 to scale them to a range between 0 and 1.</li><li>The image is slid across (convolved) with filters or kernels which multiply each overlapping pixel and sum the results of their multiplication to get a single number.</li><li>Each colour channel has its own kernel or filter. After the results have been acquired for each pixel region, the final values from each channel are added together to produce a single value which will be placed into a single output matrix called a feature map.</li><li>That feature map is max pooled. Max pooling involves extracting the max value in a specific region of pixels. E.g. If you had a 6x6 matrix and a max pooling layer of 2x2, inside of the 6x6 matrix, the max pooling layer will extract the max value in every 2x2 section of the 6x6 matrix. This will be explained in more detail.</li><li>The max pooling layer is flattened into a 1-dimensional array, which is then passed through an ordinary feed forward neural network to finally make the prediction.</li></ol><h4>DETAILED STEPS EXPLANATION</h4><p><strong>A. PREPROCESSING</strong></p><p>The image is first preprocessed by dividing all pixel values in all three channels by 255 to place them in a range between 0 and 1. Just to make things easier, let’s imagine that the image is a black and white image after preprocessing. Since 255 represents white or highest colour intensity and 0 represents black, dividing 255 by 255 gives us 1, representing white, and 0 by 255 gives us 0, representing black.</p><figure><img alt="Black and white 6x6 pixel image containing the number 1." src="https://cdn-images-1.medium.com/max/1024/1*a4cq-PXZjFYkNJidWJPidQ.png" /><figcaption>Black and white 6x6 pixel image containing the number 1. [Image by Author]</figcaption></figure><p><strong>B. CONVOLUTION PROCESS</strong></p><p>i) Extracting features into a <strong>FEATURE MAP.</strong></p><p>A feature map is a single matrix that contains features extracted from the preprocessed input image. It’s like the journal I spoke of.</p><p>In order to extract the features from the input image, a random filter or kernel which is essentially a small matrix with random numbers, is slid across the input image. Each channel in the input image has its own filter or kernel. The calculation process involves multiplying overlapping pixel values in the kernel with the input image.</p><p>E.g. If we had a 3x3 kernel, being run across a 6x6 image, the 3x3 kernel will first be placed at the top left corner of the 6x6 image overlapping a 3x3 portion of the 6x6 image. Since the kernel is of size 3x3, it will only cover a 3x3 portion of the actual input image and the first two overlapping pixels will multiply each other. The process repeats itself until it gets to the ninth pixel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k0DOqTRkN-YYMw1hOScweg.png" /><figcaption>Diagram showing input image and filter/kernel. [Image by Author]</figcaption></figure><p>When that’s done, the results of all of their corresponding multiplications will be added to get a single value. That single value is added to another value connected to the kernel known as a bias. That bias is initialized as a random number meant to diversify the CNN model and give it more flexibility.</p><p>We will repeat the same process for each channel which will each produce their own output values for the upper left 3x3 section of their matrices. E.g. The red channel could produce an output of 3, the green channel could produce an output of -1, and the blue channel could produce an output of 2.</p><p>In the image below, the red channel ended up with 2 as an output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oM3D6YHkNfSXJ1eV21GSIg.png" /><figcaption>Diagram showing matrix calculation between a kernel and an input image channel. [Image by Author]</figcaption></figure><p>The output value from each kernel multiplication is added to a bias which is a single value associated with each kernel or filter. This bias is just a random number which is added to the output to add more flexibility to the model. Since the bias for the kernel for the red channel is -2, that value will be added to the red channel’s first output value of 2 (from the top left corner).</p><p>NOTE: Biases are adjustable trainable parameters which allow the model to better fit the training data by shifting the activation value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RK0sEmCtoOVofrtEXzU6BA.png" /><figcaption>[Image by Author]</figcaption></figure><p>VERY IMPORTANT:</p><p>The image below shows the kernels and biases for each of the three channels we will be computing for.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gg9_VmLkP-eMRLr_oTbx3g.png" /></figure><p>Demonstration of the convolutional calculation below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9rdgj0PzFkqaj6bFIUKc3g.png" /><figcaption>[Image by Author]</figcaption></figure><p>The kernel for the green channel has a bias of -1. That -1 will be added to the first output of the green channel which represents the top 3x3 matrix of the image which is (0 + (-1)).</p><p>The kernel for the blue channel has a bias of 0.9. That 0.9 will be added to the output for the blue channel which is (1+0.9). This is how features are extracted from an image.</p><p>CALCULATION FOR ALL 3 CHANNELS AS SEEN IN IMAGE ABOVE</p><p>RED CHANNEL<br>(0x0)+(0x1)+(0x1)+<br>(0x0)+(0x0)+(1x1)+ <br>(0x0)+(1x1)+(0x0)<br>= 2 + (-2) = 0<br>(Output for top left corner of red channel)</p><p>GREEN CHANNEL<br>(0x1)+(0x0)+(0x0)+<br>(0x1)+(0x1)+(1x0)+<br>(0x1)+(0x1)+(0x0) = 0<br>0 + (-1) = -1 <br>(Output for top left corner of green channel)</p><p>BLUE CHANNEL <br>(0x0)+(0x0)+(0x1)+<br>(0x1)+(0x0)+(1x0)+<br>(0x0)+(1x1)+(0x1) = 1<br>1 + 0.9 = 1.9 <br>(Output for top left corner of blue channel)</p><p>But wait, the feature map is a single matrix, and we have 3 output values. What do we do? We need to add all of the three output values from each channel together to get the first value for the feature map. Since the values for the top left corner of each channel were 0, -1, and 1.9, all values will be summed up to produce an output of 0.9.</p><p>The 0.9 will be placed inside of the FEATURE MAP being the single output matrix containing the extracted features and that 0.9 will become a single pixel in the top left corner of the feature map representing the top left corner of all channels of the image.</p><p>The size of the feature map will match the number of times the kernel was shifted to reach the edge of the columns and rows.</p><figure><img alt="Image showing feature map mapping the number of times the kernel is shifted across the image." src="https://cdn-images-1.medium.com/max/1024/1*ZA4Yv0LkRjmssGQyQVmGKg.png" /><figcaption>Image showing feature map mapping the number of times the kernel is shifted across the image. [Image by Author]</figcaption></figure><p>FURTHER EXPLANATION</p><p>After the first pixel for the feature map has been calculated, each kernel for all 3 channels is shifted one pixel or one column to the right. The number of columns a kernel is shifted by is known as a stride. Using a stride of 1 means each kernel for all 3 channels will be shifted one pixel at a time to the right.</p><p>The entire calculation process will repeat itself creating a new number for the feature map. Having 2 strides will cause the kernel to shift itself two columns to the right after the first calculation.</p><p>After the kernel reaches the edge, it shifts itself downwards, based on the number of strides provided. If the stride is set to 1, the kernel for each channel will shift one row downwards after getting to the edge and begin the calculation process to create another value for the feature map.</p><p>If the kernel shifts itself downwards, the next calculated value for the feature map will also appear downwards. The kernel shifted itself 4 times across the 6x6 matrix until it got to the edge. Hence, it will produce a row of four values in the feature map. But after getting to the edge the kernel will shift downwards to begin a new calculation.</p><p>When that happens, the new calculated value for the feature map will also appear beneath the previously calculated values. In the end, the feature map will have 4 rows with 4 columns, matching the number of times the kernel was shifted as mentioned previously.</p><p>SEE IMAGE BELOW:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*--nVIcidebDcq3z0Pn93QA.png" /><figcaption>[Image by Author]</figcaption></figure><p>The process is repeated until the kernel has been overlapped with all parts of the image. After all parts of the image are convolved and calculated, the feature map below is produced.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2d33hxbDuj1QCVFcECeWzQ.png" /><figcaption>Final feature map after convolution. [Image by Author]</figcaption></figure><p>After acquiring the final feature map, each pixel value is passed through a ReLU activation function. The ReLU activation function turns all negative pixel values in the feature map into 0s and maintains all positive pixel values.</p><p>Let’s convert all positive pixel values into 1s, for simplicity. Note: This is not part of the convolution process.</p><p>Finally, you would have a matrix comprising of only 1s and 0s. This new matrix becomes the new Feature Map. See image below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JILNefu-78pMwm9E3sUjsg.png" /><figcaption>[Image by Author]</figcaption></figure><p><strong>C. MAX POOLING STEP</strong></p><p>Max pooling typically follows the convolution layer every time. It extracts the highest value in a particular region of the feature map and creates a new matrix containing the extracted values. Max pooling layers are similar to kernels but don’t contain any values. They reduce the spatial dimensions of the feature map by extracting max values in specific regions of the feature map.</p><p>Just like the convolution kernel, the max pooling layer moves along the feature map like a magnifying glass based on its size. Let’s imagine we have a max pooling layer of size 2x2. The max pooling layer will initially be placed at the top left corner of the feature map to extract the max value in that 2x2 region.</p><p>The feature map has dimensions of 4x4 pixels, meaning the max pooling layer will extract the max value in the first 2x2 region in the feature map.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tp__hCdfhFrQzoCNF82NSA.png" /><figcaption>Image showing extraction of max values from areas in feature map. [Image by Author]</figcaption></figure><p>The max pooling layer moves in strides equal to its size, though other stride values can also be used. Since the max pooling layer is of size 2x2, it will slide across the feature map in 2x2 regions. This means, it will move 2 columns to the side until it gets to the edge, and 2 rows down after it extracts information from the edge. In the end, the final result will be a new 2x2 matrix.</p><p>The top left corner of the max pooled 2x2 matrix represents the top left corner of the feature map.</p><p><strong>D. FLATENNING</strong></p><p>After acquiring the max pooled image, the final values in the max pooled layer need to be passed through an ordinary feed-forward neural network to finally make the classification or prediction.</p><p>But wait a minute, we have a 2x2 matrix (2D matrix) from the max pooling layer, but ordinary feed-forward neural networks only accept 1-dimensional arrays. Due to this, we need to flatten the 2D matrix into a 1-dimensional array. The output values from the max pooling layer will be converted into a 1-dimensional array with a single row, even though the image below appears to be a column.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l_WriRCYd8iQGXLGh-PTbQ.png" /><figcaption>Image showing max pooling layer being flattened to become a 1D array. [Image by Author]</figcaption></figure><p><strong>E. FEED-FORWARD LAYER. PREDICTION TIME!</strong></p><p>An ordinary feed-forward neural network contains weights and biases. From what you see in the image below, each value is connected to a line. That line is known as a weight and contains a value. Each weight will be multiplied by the corresponding value in the 1-dimensional array.</p><p>The result of their multiplication will be summed up and added to a bias, which is also a random number, and the output will be passed through an activation function. The activation function could either be ReLU, SoftMax or Sigmoid.</p><p>The sigmoid activation function converts values to be within the range 0 to 1. It is typically used for binary classification, meaning classifying between two classes only. SoftMax is best used for multi-class classification which returns a probability distribution across multiple classes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1ytu0FecoEweFupqyPWTwQ.png" /><figcaption>[Image by Author]</figcaption></figure><p>After that single output is passed through the activation function, it is then multiplied by two different weights as seen in the image above and added to corresponding biases in pink to produce two values in two output neurons respectively.</p><p>One output neuron represents the first image class (the digit 1), and the other represents the second image class (the digit 2).</p><p>But since computers understand numbers and not letters, the output values in each of the output neurons will either be a 1 or a 0. If the neuron containing the 1 represents the 1st class (black and white image with the digit 1), the first class becomes the predicted class and vice versa for the second.</p><p><strong>VERY IMPORTANT DETAIL:</strong></p><p>The same procedure is repeated for the image containing the digit 2 maintaining the weights, biases and kernels. During training, if the model makes a wrong prediction, all the weights, biases and kernels will be changed or adjusted until the right prediction is made. That’s how the model learns.</p><h4><strong>SUMMARY:</strong></h4><ol><li>The image is first preprocessed by dividing all of the pixels by 255 to place their values between 0 and 1.</li><li>A convolution kernel or filter is applied to the image to extract features which will then be placed inside of another matrix called a feature map.</li><li>The feature map is run through a ReLU activation function which converts all negative values to 0 and maintains all positive values.</li><li>A max pooling layer is applied to extract the max value within a specific region of pixels based on the size of the max pooling layer or matrix.</li><li>The final result of the max pooling layer is converted into a 1-dimensional array to be fed into a feed-forward neural network.</li><li>The feed-forward neural network contains weights and biases along with another activation function which makes the final prediction to determine the class the image belongs to.</li></ol><p>Well done, Sailor! You made it to the end of this paper!</p><p>Make sure to follow for more insightful AI content and read my previous paper on LLMs exploring the math and logic behind them.</p><p>You can find me on LinkedIn here: <a href="https://www.linkedin.com/in/joshua-anang-611462285/">Joshua Anang | LinkedIn</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=36adc103c4be" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The math and logic behind ChatGPT. This paper is all you need.]]></title>
            <link>https://medium.com/@joshuaanang783/the-math-and-logic-behind-chatgpt-this-paper-is-all-you-need-57b82a0527f9?source=rss-963808c552ed------2</link>
            <guid isPermaLink="false">https://medium.com/p/57b82a0527f9</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Joshua Anang]]></dc:creator>
            <pubDate>Wed, 07 May 2025 12:01:52 GMT</pubDate>
            <atom:updated>2026-08-25T22:16:35.693Z</atom:updated>
            <content:encoded><![CDATA[<p>This paper explains everything there is to know about Large Language Models in simple and understandable terms.</p><p>We’ve all heard of ChatGPT and DeepSeek, which are Large Language Models (LLMs). These Large Language Models are powered by a technology called transformers or transformer neural networks.</p><p>What makes them so special? They’re able to understand context between words in a sentence and predict the next or expected word in an output sentence. That’s the reason why ChatGPT and other LLMs generate words sequentially; because this complex neural network generates or predicts the next word step by step based on the input sentence.</p><p>For example, if I were to input a sentence like ‘Thank you’, obviously the LLM should respond by saying ‘You are welcome’. So, it uses algorithms to predict the first word which is ‘You’, and then the next ‘are’, then finally ‘welcome’. I’m going to show you how they work in detail, so weigh anchor and prepare to set sail!</p><h3>Logic used by a transformer to predict the next word in a sentence.</h3><p>A transformer model is made up of two main parts, which are the encoder and the decoder. The encoder processes and encodes the input sentence while the decoder processes and decodes the output sentence.<br>The results are used to predict the expected or next word which would be the output sentence provided to the decoder.</p><p>During training, we provide both the input sentence and the output sentence to teach the transformer model how to respond in the output sentence when the input sentence is provided.</p><p>For example, if I wanted to teach you how to speak Japanese, I would give you a list of sentences in English, and their corresponding translation in Japanese, such that next time, you’d be able to respond without the translations. In this scenario, the English sentences will be provided to the encoder, and the output or expected Japanese sentences will be provided to the decoder.</p><p>We’re going to translate the sentence ‘Thank you’ into ‘Arigato’ — Japanese. Imagine you were in a conversation with a person speaking a language you don’t fully understand, your brain will take some time to automatically convert each word the person said in the foreign language into your language, before you can finally have the understanding. That’s how transformers work. But in this case, <em>you’re</em> the transformer, and the <em>person</em> is the user.</p><h4>STEPS</h4><p>These are the steps the transformer goes through in order to predict each word in a sentence.<br>This operation is performed on both the ENCODER and DECODER. We’ll explore all of this in detail so brace yourselves.</p><ol><li>Word embeddings. — The transformer converts each word in a sentence into a vector of real numbers to represent each word.</li><li>Positional encoding. — This is used to keep track of word order in a sentence because you can have two sentences with the exact same words but with different meanings. E.g. ‘The bat ate the cat’, ‘The cat ate the bat’.</li><li>Self-Attention — In order to provide more context to the transformer model, the relationship between each word in a sentence is calculated by generating queries, keys and values for each word and multiplying and adding them to get the final results which are self-attention values or scores. That’s essentially what self-attention means.</li><li>Residual connections — The previous positional encoding values are then added to the final self-attention values for each word to get another set of outputs. After the addition, the outputs are then normalized.</li><li>Feed-forward layer — The normalized outputs are passed through a standard feed-forward neural network which creates feed-forward outputs, as I like to call them.</li><li>Residual connections — New residual connections are added to these feed forward outputs but this time, the operation includes the addition between the previously normalized outputs and the feed forward outputs. After the addition, the outputs are then normalized again, just like before.</li></ol><p>Note: All of the mentioned operations apply both to the encoder and the decoder however, the decoder has more layers and includes more steps compared to the encoder. Keep reading to find out what they are.</p><h4>Next Steps for the decoder:</h4><ol><li>Encoder-decoder attention — The next step is to calculate the relationship between each word in the input sentence, and each word in the output sentence. This is called encoder-decoder attention, not self-attention this time, and it helps provide more context and understanding to the transformer.</li><li>Residual connections are added again. But this time, the decoder’s previous final output after normalization, will be added to the encoder-decoder attention scores and normalized again to get new outputs (not the positional encoding values).</li><li>Feed forward layer — The final outputs are passed through a standard feed forward neural network which creates new feed forward outputs.</li><li>Residual connections — Another set of residual connections are added which involves addition between the normalized encoder-decoder attention values and the new feed forward outputs.</li><li>The SoftMax activation function is used at the very final step to convert the finally normalized outputs into probabilities over the output vocabulary. The word with the highest probability becomes the predicted word and is used in conjunction with the previous words and input sentence to predict the next word in the output sentence.</li></ol><p>VERY IMPORTANT DETAIL:</p><p>Residual connections are added after each major step (self-attention, encoder-decoder-attention, feed-forward). This architecture is based on the paper ‘Attention is all you need’, but this example, will contain just one residual connection after each attention step (self-attention, encoder-decoder attention) and one feed forward layer after encoder-decoder attention.</p><p>ChatGPT uses a decoder only transformer, but this paper describes how transformers work based on the paper ‘Attention is all you need’ without completely adopting its architecture.</p><h3>THE ENCODER</h3><p>The encoder is where your input sentence is fed. E.g. If you were to say ‘Hello there’ to an LLM, your ‘Hello there’ would be fed into the encoder of the LLM. But since we are training the model, we need to provide the model with an expected response, which would be ‘Hi there, how can I help you?’. This is a response we would expect from the LLM hence, it would be fed into the decoder during training to teach the LLM to respond to ‘Hello there’ with ‘Hi there, how can I help you?’.</p><p>The input sentence would be surrounded by words known as start and end tokens. &lt;SOS&gt; (Start of Sequence) and &lt;EOS&gt; (End of Sequence). This tells the transformer model where the sentence begins, and where it ends.</p><p>Anytime you type in a sentence into ChatGPT, your sentence is wrapped inside of these words which we will call ‘tokens’.</p><p>During training, the exact same procedure is done with the output sentence inside of the decoder. They are surrounded by &lt;SOS&gt; and &lt;EOS&gt; tokens, but in this encoder, we will only be dealing with the &lt;EOS&gt; token.</p><h4><strong>STEPS FOR THE ENCODER.</strong></h4><h4>1. Word Embeddings.</h4><p>The encoder first calculates word embeddings. Word embedding is a way of converting words into numbers, because computers operate on numbers not letters. These numbers are used to represent each word in a sentence.</p><p>You can have 4-word embeddings, 2-word embeddings for each word or even 512-word embeddings for each word, depending on the complexity of your transformer model. These numbers represent the identity of a word in a sentence. E.g. ‘Hi there’. ‘Hi’ could have 4 word embedding values which could be [0.1, 3.2, -1.7, 2.1] and ‘there’ could also have its own set of word embedding values which could be [0.9, -3.2, 1.8, 0.4].</p><p>These numbers represent each word in a sentence and could be thousands. The lists are vectors, and their sizes are based on the number of items (numbers) they contain. Using 1 number to represent each word in a sentence is not very ideal as the same number could represent a different word.</p><p>We’re going to find 2 word embedding values for ‘Thank’, ‘you’ and the ‘&lt;EOS&gt;’ token. Even though ‘to’ is part of the sentence, we won’t be encoding that word because it doesn’t make sense to add it into the sentence.</p><p>The &lt;EOS&gt; token represents an end of sequence. It tells the model where the sentence ends. In this case, we can’t use a full stop, because you could have multiple sentences as an input in one paragraph containing multiple full stops.</p><p>In order to find the word embedding values for each word, each neuron representing each word is placed with a value. Since we’re trying to find the word embedding values for ‘Thank’ we will place a 1 inside of the neuron representing ‘Thank’ (of the colour red), and place zeros inside all of the other words. Then we perform the calculation. The calculation involves multiplying each number value by the weight or line they’re connected to.</p><figure><img alt="Diagram showing neural network calculation to find the word embedding values for a single word." src="https://cdn-images-1.medium.com/max/1024/1*A-GWY7LimgyuU99CwrXcag.png" /><figcaption>Diagram showing neural network calculation to find the word embedding values for a single word.</figcaption></figure><p>What you see above is a typical representation of an artificial neural network. In this case, this artificial neural network is being used to calculate the word embedding values for ‘Thank’.</p><p>We can see in the diagram that the number [1] inside of the first neuron representing the word ‘Thank’ is connected to a weight (line) with the value [1.2] which is then connected to the first word embedding output neuron or value. That same [1] also has another connection to the weight with the value [2.9] which is connected to the second word embedding value or neuron. It is of the colour red.</p><p>Note: the squares are all neurons, and the lines are called weights. The number [1] will multiply the weight it’s connected to [1.2], and all of the other [0]s which are connected to weights whose weights are connected to the first neuron will multiply each other correspondingly as explained.</p><p>The final results of their multiplications are added and passed through an identity function which essentially gives out the same output as their multiplication. The result becomes the first word embedding value in the output neuron which is [1.2].</p><p>The second word embedding value repeats the same process. Each neuron containing a [1] or a [0] representing each word, is multiplied by their corresponding weights, and all of the results are added up to get the second word embedding value. Now we have the word embedding values for the word ‘Thank’. If you’re wondering about the weights, they are random numbers which will be altered during training as the model learns.</p><p>In order to get the word embedding values for the word ‘you’, the same process is repeated, but this time, a [0] is placed inside of the neuron for ‘Thank’ and all the other words, and a [1] is placed inside of the neuron for ‘you’.</p><p>The same weights are used as before and the two word embedding values for ‘you’ are finally calculated. The steps are repeated for the final &lt;EOS&gt; token which represents the end of sequence or sentence. See images below.</p><figure><img alt="Diagram showing neural network calculation to find the word embedding values for a single word." src="https://cdn-images-1.medium.com/max/1024/1*33LmspkxeBQhcfFmc6QlnQ.png" /><figcaption>Diagram showing neural network calculation to find the word embedding values for a single word.</figcaption></figure><figure><img alt="Diagram showing neural network calculation to find the word embedding values for a single word." src="https://cdn-images-1.medium.com/max/1024/1*Zl79mVcXid99-ErFYl7GuQ.png" /><figcaption>Diagram showing neural network calculation to find the word embedding values for a single word.</figcaption></figure><p>VERY IMPORTANT DETAIL:</p><p>The real calculation process between weights and neurons is essentially matrix multiplications, however, the diagrams above give you a more simplified and understandable view of the entire process.</p><p>To explain further, the neurons representing each word used to calculate their word embeddings, are converted into rows. E.g. Neurons for calculating word embeddings for ‘Thank’ will look like this in a matrix [1, 0, 0, 0]. The same for the other words being ‘you’ and ‘&lt;EOS&gt;’ [0,0,1,0], [0,0,0,1]. These are what the neurons look like based on the diagram, but they are actually referred to as one-hot vectors.</p><p>One-hot vectors:</p><p>[1, 0, 0, 0]<br>[0, 0, 1, 0]<br>[0, 0, 0, 1]</p><p>In order to calculate their word embeddings, the weights or lines connected to each output word embedding neuron are converted into a column this time. Or the weights connected to each input neuron are converted into rows. Each one-hot vector (row) in this case is multiplied by the weights respectively, with their results finally added to produce a single word embedding value.</p><p>Weights:</p><p>[1.2, 2.9]<br>[0.7, -0.3]<br>[1.3, 0.8]<br>[0.3, 0.9]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*My2WVarTXKhAmbybQt8xgw.png" /></figure><h4><strong>2. POSITIONAL ENCODING</strong></h4><p>You can have two sentences with the exact same words but different meanings. E.g. ‘The bat ate the cat.’ and ‘The cat ate the bat’. Both sentences use the exact same words but have different meanings.</p><p>Positional encoding is used to keep track of word order to help the transformer understand distinctive meanings in sentences with the exact same words. Positional encoding is calculated by adding a set of points on sine and cosine waves to each word embedding value representing each word.</p><p>See diagram below. The yellow squares contain sine and cosine points from the waves which are added to the word embedding values to produce the final positional encoding values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gQFfjG2D9i_L2VWI7Mkeng.png" /></figure><h4>3. SELF ATTENTION</h4><p>Transformers are special. In order to understand context, they try to find the similarities between each word in a sentence using a technique known as self-attention.</p><p>This is where true meaning is derived from a sentence. Self-attention is a technique used to find the relationships between words in a sentence to provide context to the transformer. For example, in a sentence like ‘The Pirate bombarded the ship, and it sank to the bottom of the ocean’, ‘Ship’ and ‘it’ are closely related in the sentence as we can see. Hence, the self-attention scores for ship and it must be calculated for the transformer to understand in future sentences or predictions that ‘ship’ and ‘it’ are closely related.</p><p>The same is done for each word in the sentence. The self-attention scores between each word including itself are calculated using various mathematical operations. But before finding self-attention scores, we need to find precursor values known as dot-products or similarity scores. These are the true relationship scores between each word in a sentence, and they reflect how strongly each word relates to every other in a sentence.</p><p><strong>Procedure:</strong><br><strong>i.</strong> In order to find the dot product or similarity scores between each word including itself, each word must be given a QUERY, KEY and VALUE. These are new numbers which represent each word, just like the word embeddings and are calculated by multiplying the previous positional encoding values with weights.</p><p>E.g. If we wanted to find the similarity scores between ‘Thank’ and all other words in the sentence including itself, we need to generate QUERY values for ‘Thank’, and KEY values for each word including ‘Thank’, then multiply the QUERY values for ‘Thank’ by its KEY values and by the KEY values of every other word correspondingly, and finally add the results each time to get the similarity scores.</p><p>NOTE: In the upcoming procedure, we will calculate the self-attention scores for ‘Thank’ only by first finding the similarity scores between ‘Thank’ and ‘you’ excluding the &lt;EOS&gt; token.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JRHfUXQPhu5BbV8W1BBRDw.png" /></figure><p>- Inside of the diagram, (4.47 and 2.28) are the QUERY values for ‘Thank’, and (9.54 and 8.22) are the KEY values for ‘Thank’ as well. The VALUES for ‘Thank’ are (2.1 and 6.3).<br>- The KEY values for ‘you’ are (5.64 and 5.72) and the VALUES for ‘you’ are (2.36 and 4.8).</p><p>To find the similarity score between ‘Thank’ and itself and ‘Thank’ and ‘you’. We must multiply the QUERIES of ‘Thank’ by the ‘KEYS’ of ‘Thank’ and add the results to get the similarity score between ‘Thank’ and itself being 61.3. See diagram below.</p><p>After, we multiply the QUERIES of ‘Thank’ by the KEYS of ‘you’ to get the similarity score between ‘Thank’ and ‘you’ being 38.2. See diagram below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9716igPq6jBcf05qUrRAvA.png" /></figure><p>CALCULATION:<br>- (4.47 x 9.54) + (2.28 x 8.22) = 61.3 (similarity score between ‘Thank’ and itself).<br>- (4.47 x 5.64) + (2.28 x 5.72) = 38.2 (similarity score between ‘Thank’ and ‘you’).</p><p><strong>ii.</strong> The final values are placed through a SoftMax function. The SoftMax function is a mathematical operation which essentially compresses both values (61.3 and 38.2) such that their sum adds up to 1. Where 0 corresponds to 0% attention, and 1 corresponds to 100% attention.</p><p>If the similarity score between ‘Thank’ and itself is 61.3, and the similarity score between ‘Thank’ and ‘you’ is 38.2, after both values are passed through the SoftMax function, 61.3 will be converted to a higher value such as 0.62, and 38.2 will be converted to a lower value such as 0.38. The 1 and 0 become SoftMax percentages. This tells us that ‘Thank’ relates to itself, more than it relates to ‘you’.</p><p>Note: SoftMax percentages are proportional probabilities. In practice they typically provide values which are close to 1 or 0 such as 0.56 and 0.44.</p><figure><img alt="A diagram showing a model of a neural network used to calculate the similarity scores between two words being ‘Thank’ and ‘you’." src="https://cdn-images-1.medium.com/max/1024/1*9716igPq6jBcf05qUrRAvA.png" /><figcaption>A diagram showing a model of a neural network used to calculate the similarity scores between two words being ‘Thank’ and ‘you’.</figcaption></figure><p>Finally, the SoftMax output for the similarity between ‘Thank ’and itself is 1, and 0 for the SoftMax output for the similarity between ‘Thank’ and ‘you’.</p><p><strong>iii.</strong> The 1 and 0 are then multiplied by VALUES which are calculated by multiplying new weights by the previously acquired positional encoding values just like we did with the word embeddings. The first generated VALUES belonging to ‘Thank’ must be used to multiply the similarity score or SoftMax percentage for ‘Thank’ only. These VALUES multiply the 1 only. Both generated VALUES multiply the 1 to produce new values which will be used to get the final self-attention scores.</p><p>2.1 and 6.3 are the previously generated VALUES for ‘Thank’ which were acquired by multiplying the previous positional encoding values by weights and adding their results respectively.</p><p>(2.1 x 1) = 2.1 — Let’s call this x0<br>(6.3 x 1) = 6.3 — Let’s call this x1</p><p>Note: Combining these two values creates a single vector with multiple dimensions.</p><p>See image above.</p><p><strong>iv.</strong> The same operation is performed for the similarity score between ‘Thank’ and ‘you’. ‘You’ generates new VALUES by multiplying its positional encoding values with weights to get two new VALUES (2.36 and 4.8) which are light green in colour. Both values multiply the 0 which is the SoftMax percentage of the similarity score between ‘Thank’ and ‘you’. <br>(2.36 x 0) = 0 — Let’s call this y0<br>(4.8 x 0) = 0 — Let’s call this y1</p><p>Finally, to get the self-attention scores for ‘Thank’, we add the corresponding multiplied values being (x0 + y0), and finally (x1 + y1). The results become the final self-attention scores for ‘Thank’ only being (2.1 and 6.3).</p><p>See image below for <strong>MATRIX </strong>example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o36WfaBqYXSGwia4Ugv9oQ.png" /><figcaption>Matrix multiplication demonstrating the final calculation to get the attention scores for ‘Thank’.</figcaption></figure><p><strong>v. </strong>The same operation is performed between ‘you’ and ‘&lt;EOS&gt;’. QUERIES are created for both ‘you’ and ‘&lt;EOS&gt;’, which are then multiplied by the KEYS for every word in the sentence including themselves (‘Thank’, ‘you’, and ‘&lt;EOS&gt;’).</p><p>VERY IMPORTANT DETAIL:</p><p>In this example, we had just 1 self-attention layer, meaning just 1 set of QUERIES, KEYS and VALUES for each word. However, in most LLMs like ChatGPT, you could have about 8 self-attention layers each with their own set of QUERIES, KEYS and VALUES.</p><h4>4. RESIDUAL CONNECTIONS.</h4><p>After the self-attention scores have been calculated, they are then added to the previously calculated positional encoding values to improve performance and help the transformer understand and remember word positions.</p><p>NOTE: Residual connections are added after steps like self-attention and feed-forward layers.</p><h4>MATRIX DEMONSTRATION FOR SELF-ATTENTION.</h4><p>The matrix demonstration is what the true calculations inside of a transformer actually look like. The diagrams above without matrices only show you a more simplified and understandable version of the entire procedure.</p><p><strong>a.</strong> The QUERIES, KEYS and VALUES for each word in each row are calculated by multiplying each Row by a set of column weights and adding the results just like before. (Row by Column multiplication) is applied.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Bnvl4HeGUNGoj_nVeycV1Q.png" /></figure><p>CALCULATION EXAMPLE:</p><p>(1.2 x 3.4) + (3.9 x 0.1) = 4.47 (first QUERY value for ‘Thank’)</p><p>(1.2 x 0.6) + (3.9 x 0.4) = 2.28 (second QUERY value for ‘Thank’)</p><p>The process continues for each word. etc.</p><p><strong>b.</strong> The QUERIES should multiply the Transposed version of the KEYS to get the <strong>dot product similarities</strong>. What does <strong>Transpose </strong>mean? <br>Row by Column multiplication wouldn’t work when you have just 2 columns in the QUERIES, and 3 rows in the KEYS. The final row would be left out.</p><p>Matrix multiplication requires the number of columns in the first matrix to match the number of rows in the second. Hence, KEYS would have to be transposed or turned horizontally, such that the KEYS for each word are no longer in rows but reshaped into three columns with two rows with their rows matching the number of columns in the QUERY matrix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TI4EGbE9GIK2zCi1CQCfGw.png" /></figure><p><strong>c.</strong> According to the formula from the original paper ‘Attention is all you need’, each number in the output dot product similarity matrix needs to be divided by the square root of the dimensions of the KEYS matrix (Dk) or (dk). Meaning, the number of KEYS representing each word which are 2 in this case.<br>This is to get the ‘SCALED’ dot product similarity scores between each word. The scaled dot product similarities are passed through a SoftMax function row by row to get their SoftMax percentages. It compresses the values inside of a row, such that their sum produces 1.<br>E.g. SoftMax (13.5, 8.6, 5.6) = [0.99, 0.0073, 0.00037] — Final SoftMax percentage for last row. See image below for reference to this example.</p><p>FORMULAR BELOW.</p><figure><img alt="Formular for calculating self-attention." src="https://cdn-images-1.medium.com/max/474/1*UCaFD1-ukYVtFjtgax4X6w.jpeg" /><figcaption>Formular for calculating self-attention.</figcaption></figure><ul><li>QK — This demonstrates the multiplication between the QUERIES and the Transpose of the KEYS.</li><li>Square root (dk) — This represents the division between the results of the multiplication between the QUERIES AND KEYS by the square root of the dimensions of the KEYS, being the number of KEYS representing each word in this matrix to get the <strong>scaled dot product similarities.</strong></li><li>SoftMax — This represents finding the SoftMax percentages between the results of the <strong>scaled dot product similarities.</strong></li><li>V — This represents the final multiplication of SoftMax percentages by the previously generated VALUES to get the <strong>self-attention scores</strong>.</li><li>This formular follows the exact same process as described before in calculation self-attention values.</li></ul><figure><img alt="Matrix calculation to get the scaled dot product similarities for each word in a sentence by dividing the entire matrix by the square root of the dimensions of the KEY matrix (dk). The image also shows how the SoftMax percentages between each number in each row are calculated." src="https://cdn-images-1.medium.com/max/1024/1*YwvM2QPTJD4IUizpD1KR5w.png" /></figure><p><strong>e. </strong>The SoftMax percentages for each word are then multiplied by the previously generated VALUES to get the <strong>final self-attention scores</strong> for each word as seen in the image below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6V_oUHHO7VNX1kRC8mDkGg.png" /></figure><p><strong>f.</strong> The final self-attention scores for each word are added to the previous positional encoding values, to get the final encoder outputs for each word. These are called ‘<strong>Residual Connections</strong>’. See image below.</p><figure><img alt="Residual connections are acquired by adding the positional encoding values to the final self-attention scores for each word." src="https://cdn-images-1.medium.com/max/1024/1*9PHwhfWJ824fHoI6VNVTJA.png" /></figure><p>Simplified diagram explaining the entire encoder process up to this point.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GLNwbm-bqTF3Dcx0acyBWw.png" /></figure><ul><li>Red — represents the input token for the word ‘Thank’.</li><li>Cyan — represents the input token for the word ‘you’.</li><li>Green — represents the input token for the &lt;EOS&gt;.</li><li>Yellow — are the sine and cosine waves which are added to the word <strong>embedding values</strong> beside them to produce the <strong>positional encoding</strong> values above the ‘=’ signs.</li><li>Pink — is where the <strong>scaled dot product similarities scores</strong> are calculated along with the <strong>SoftMax percentages</strong> to get the <strong>final self-attention scores</strong> for each word.</li><li>Black — represents the final self-attention values or scores for each word which are then added to the positional encoding values using a method known as <strong>residual connections</strong> to get the values on top of the black-filled squares.</li><li>The final values on top of the black-filled squares are the results of the <strong>residual connections </strong>which are the final encoder outputs.</li></ul><p>VERY IMPORTANT DETAIL:</p><p>The weights used to calculate the QUERIES, KEYS and VALUES for each word are the same as seen in the matrix diagrams. That explains why there are three sections for each word in the diagram above us right now.</p><h3>THE DECODER</h3><p>The decoder holds the output or expected sentence which contains the words to be predicted in order to generate a meaningful response or sentence.<br>The decoder goes through the exact same steps as the encoder to decode each word in the output sentence, but the decoder has a few key differences we’ll discuss shortly.</p><p>During training, the decoder’s inputs are the expected output values or words which the transformer needs to predict in order.</p><p>Even though we’re feeding ‘Arigato’ and &lt;SOS&gt; into the decoder during training, the transformer still needs to learn to predict these words on its own during actual usage (inference), because these words wouldn’t be fed to the decoder during inference. The only word which will be fed to the decoder during inference is the &lt;SOS&gt; token or Start of Sequence token which determines the beginning of the sentence. Anytime you use an LLM, just like in the encoder, the &lt;SOS&gt; token is secretly fed to the decoder to initiate the prediction sequence.</p><p><strong>STEPS</strong></p><ol><li><strong>Word Embeddings.</strong> Converting each word into two numbers. See diagram below.</li></ol><figure><img alt="Decoder layer using matrix math to calculate the word embedding values for each word in the decoder." src="https://cdn-images-1.medium.com/max/1024/1*HQ2APdc8DCwUEh_bs7xf2A.png" /><figcaption>Calculating word embedding values for words inside of a decoder using matrix mathematics.</figcaption></figure><p><strong>2. Positional Encoding </strong>— Keeping track of word order in a sentence, because you could have sentences with the exact same words but with different meanings.</p><figure><img alt="Diagram showing how positional encoding values are found for words inside of a transformer using sine and cosine waves which are added to word embedding values to produce the final positional encoding values." src="https://cdn-images-1.medium.com/max/1024/1*6KjEmC8kSQbhvcC6nOmaVg.png" /><figcaption>Diagram showing how positional encoding values are found for words inside of a transformer using sine and cosine waves which are added to word embedding values to produce the final positional encoding values.</figcaption></figure><p><strong>3.</strong> Generating <strong>QUERIES</strong>, <strong>KEYS </strong>and <strong>VALUES </strong>to find the relationships or dot product similarities amongst words in the output sentence which is fed as an input to the decoder.</p><figure><img alt="Matrix math to get QUERIES, KEYS and VALUES for each word inside of a transformer’s decoder by multiplying word embedding values by weights." src="https://cdn-images-1.medium.com/max/1024/1*GRQGgm-aVZwj_cB1OnNk-A.png" /><figcaption>Matrix math to get QUERIES, KEYS and VALUES for each word inside of a transformer’s decoder by multiplying word embedding values by weights.</figcaption></figure><p><strong>4.</strong> The QUERIES are multiplied by the Transpose of the KEYS to get the unscaled dot product similarities. Even though we have the same 2x2 matrix for both QUERIES and KEYS this time, the KEYS still need to be Transposed.</p><figure><img alt="Matrix math to get the unscaled dot product similarity scores inside of a transformer by multiplying the QUERIES of each word, by the Transpose of the KEYS for each word." src="https://cdn-images-1.medium.com/max/1024/1*gpX6d5xj-GZAYVEV2gYMqA.png" /><figcaption>Matrix math to get the unscaled dot product similarity scores inside of a transformer by multiplying the QUERIES of each word, by the Transpose of the KEYS for each word.</figcaption></figure><p>VERY IMPORTANT DETAIL:</p><p>The decoder is slightly different from the encoder. First, when the model is predicting the next words, it needs to know where the sentence begins. Because of that, just like how the &lt;EOS&gt; token was used to mark the end of the sentence in the encoder, an &lt;SOS&gt; token will be used this time to mark the beginning of the sentence in the decoder. &lt;SOS&gt; stands for start of sequence.</p><p>The &lt;SOS&gt; token will always be fed into the model’s decoder, whether during training, or prediction. During your use of various LLMs, even though you don’t type in the &lt;SOS&gt; token into the text box, secretly that &lt;SOS&gt; token is placed into decoder to initiate the prediction sequence, while your input sentence is placed inside of the encoder. The &lt;SOS&gt; token tells the model where the sentence begins.</p><p><strong>5.</strong> The <strong>unscaled dot product similarities</strong> are then scaled by dividing each and every single number by the dimensions of the KEYS matrix which is essentially the number of KEYS each word has. Two for each word in this case.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*feeW3IpBuMS3PLYL_I4Y0A.png" /><figcaption>Matrix calculation to scale the unscaled dot product similarities inside of a decoder by dividing them by the square root of the dimensions of K.</figcaption></figure><p>VERY IMPORTANT DETAIL:</p><p>In the image above, we can see that the similarity scores between &lt;SOS&gt; itself, and ‘Arigato’ have been calculated. These are -29.9 and 25.1 respectively. One detail about the decoder is, we don’t want to use the similarity score between the &lt;SOS&gt; token and ‘Arigato’, so during training, we use something known as an <strong>Attention Mask</strong> to ignore the similarity score between &lt;SOS&gt; and ‘Arigato’ which is outlined in black.</p><p>Why do we do this?</p><p>Since we’re training the model, we don’t want the model to see what is to come next after the &lt;SOS&gt; token or after any word before the next word in the output sentence. We want it to predict the next word which would be ‘Arigato’ without knowing that the next word is ‘Arigato’.</p><p>It’s like seeing the path to the end of the maze before you begin moving. The idea would be to get to the end of the maze without knowing how or where to find the end, based on your own discretion. But if you were offered a peek of the route to the end, you’d essentially be cheating without using your discretion, hence hindering your thinking ability, just like in the maze runner.</p><p>So, we keep all of the other similarity scores in the matrix and add -infinity to the similarity score between &lt;SOS&gt; and ‘Arigato’, turning 25.1 into -infinity. To elaborate, during training, we add -infinity to all similarity scores representing future words and maintain similarity scores representing past words. This prevents the decoder from seeing future tokens during training. That would be cheating.</p><p><strong>6.</strong> And just like before we calculate the<strong> SoftMax percentages</strong> between each row and multiply them by the VALUES to get the decoder self-attention scores for &lt;SOS&gt; and ‘Arigato’. Which are shown in black.</p><figure><img alt="Matrix calculation to find the Decoder Self-attention scores by first finding the SoftMax percentages between each row in the scaled dot product similarities and multiplying those percentages by the previously generated values." src="https://cdn-images-1.medium.com/max/1024/1*UHPY1C9W-w7in1BLPAEyuA.png" /><figcaption>Matrix calculation to find the Decoder Self-attention scores by first finding the SoftMax percentages between each row in the scaled dot product similarities and multiplying those percentages by the previously generated values.</figcaption></figure><p><strong>7.</strong> To get the <strong>final decoder outputs</strong>, we add the positional encoding values row by row this time to get the final decoder outputs for each word. These are called <strong>residual connections</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FUUscCKNQ0FDmqWoxJqpZg.png" /><figcaption>This diagram demonstrates the matrix calculation to get the final decoder outputs by adding the decoder self-attention scores with the previous positional encoding values of the decoder.</figcaption></figure><h4>DIAGRAM DESCRIBING THE ENTIRE PROCESS.</h4><figure><img alt="Diagram demonstrating how the decoder outputs inside of the transformer are calculated from top to bottom." src="https://cdn-images-1.medium.com/max/1024/1*bQDujkfTMMyS-IEX7j_TJA.png" /><figcaption>Diagram demonstrating how the decoder outputs inside of the transformer are calculated from top to bottom.</figcaption></figure><ul><li>Purple represents the input token for &lt;SOS&gt;</li><li>Red represents the input token for ‘Arigato’.</li><li>Yellow represents the sine and cosine points which will be added to the word embedding values beside them to produce the positional encoding values outlined in black.</li><li>The pink represents the QUERIES, KEYS and VALUES used to calculate the dot product similarities and the self-attention scores.</li><li>The black represents the final self-attention scores which are then added to the positional encoding values to produce the final outputs in red and purple for each word.</li></ul><h4><strong>ENCODER-DECODER ATTENTION</strong></h4><p>Self-attention isn’t the only special thing about a transformer. After self-attention is calculated between each word in the input or output sentences both in the encoder and decoder, another mechanism called ENCODER-DECODER attention needs to be performed.</p><p>Why? This helps determine the relationship between words in the input sentence and words in the output sentence. Earlier, we only calculated self-attention scores between the input words and output words individually. Now, we need to find the relationships between input sentence and the output sentence conjunctionally.</p><p>The only difference is, the encoder-decoder attention scores apply only to the output sentence fed into the decoder; in this case &lt;SOS&gt; and ‘Arigato’. So even if we were to find the encoder-decoder attention scores, they would only be used to amplify the decoder’s understanding, because the actual next word prediction happens in the decoder. We’ll explain that later.</p><p>Now, we’re going to find the encoder-decoder attention scores for &lt;SOS&gt; and ‘Arigato’, meaning, how both words relate to each word in the input sentence fed to the decoder. If we had 10 words in the output decoder sentence besides &lt;SOS&gt; and ‘Arigato’, the same operation will be performed on each of those words, but in this case we have just 2.</p><h4><strong>STEPS:</strong></h4><p><strong>1.</strong> Just like before, we generate QUERIES, KEYS and VALUES, but this time, the QUERIES are only generated by the decoder’s output and the KEYS and VALUES are generated by the encoder’s output. This is done by multiplying each output for each word, by a bunch of random numbers called weights, to get the QUERIES, KEYS, and VALUES for encoder-decoder attention.</p><figure><img alt="Matrix diagram demonstrating how QUERIES, KEYS and VALUES for the encoder-decoder attention are calculated." src="https://cdn-images-1.medium.com/max/1024/1*2ljuLln3dx5Kpx7X406DMQ.png" /><figcaption>Matrix diagram demonstrating how QUERIES, KEYS and VALUES for the encoder-decoder attention are calculated.</figcaption></figure><p><strong>2.</strong> The QUERIES from the decoder are multiplied by the Transpose of the KEYS from the encoder just like before, to get the unscaled dot product similarities, which are then divided by the square root of the dimensions of the KEY matrix (Dk) or (dk) which are the number of KEYS representing each word to finally get the scaled dot product similarities.</p><figure><img alt="Diagram demonstrating the multiplication between QUERIES and the transpose of KEYS to get the Unscaled dot product similarities, which are then divided by the square root of the dimensions of the KEY matrix to get the Scaled dot product similarities." src="https://cdn-images-1.medium.com/max/1024/1*IzcBRTKLlnGXGjVZs6Hy_g.png" /><figcaption>Diagram demonstrating the multiplication between QUERIES and the transpose of KEYS to get the Unscaled dot product similarities, which are then divided by the square root of the dimensions of the KEY matrix to get the Scaled dot product similarities.</figcaption></figure><p><strong>3.</strong> The SoftMax percentages between the scaled dot product similarities are calculated row by row. The new SoftMax percentages are finally multiplied by the VALUES from the encoder’s outputs to get the final encoder-decoder attention scores for both words in the decoder, being &lt;SOS&gt; and ‘Arigato’. &lt;SOS&gt; = [13.87, 0.72] ‘Arigato’ = [16.2, 0.78].</p><figure><img alt="Matrix calculation to get the Final encoder-decoder attention scores by first finding the SoftMax percentages between the Scaled dot product similarities and multiplying them by the VALUES generated from the encoder’s outputs." src="https://cdn-images-1.medium.com/max/1024/1*eVvNrCFv_0IKnDKTDTt-Hg.png" /><figcaption>Matrix calculation to get the Final encoder-decoder attention scores by first finding the SoftMax percentages between the Scaled dot product similarities and multiplying them by the VALUES generated from the encoder’s outputs.</figcaption></figure><p><strong>4.</strong> The final encoder-decoder attention scores after multiplying the SoftMax percentages by the VALUES will be added to the old decoder outputs using residual connections. Row by Row addition is used this time to arrive at [2.57, 5.49] for the &lt;SOS&gt; token and [9.5 and 0.38] for ‘Arigato’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mIqPNQy2FtSUocvd8mYr5w.png" /></figure><h4>DIAGRAM DESCRIBING THE ENTIRE PROCESS FROM THE ENCODER TO THE DECODER TO THE ENCODER-DECODER ATTENTION.</h4><figure><img alt="Final diagram explaining the entire transformer process ending at the final outputs after encoder-decoder attention." src="https://cdn-images-1.medium.com/max/1024/1*TmLWazgtn96vbVzdzl0iOQ.png" /><figcaption>Final diagram explaining the entire transformer process ending at the final outputs after encoder-decoder attention.</figcaption></figure><p>In the diagram above, 2.57 and 5.49 are the final outputs after residual connections between encoder-decoder attention scores for &lt;SOS&gt;, and the old decoder outputs for &lt;SOS&gt; which are in black.</p><p>9.5 and 0.38 are the final outputs after residual connections between encoder-decoder attention scores for ‘Arigato’, and the old decoder outputs for ‘Arigato’ which are also in black.</p><h4>FINAL DESTINATION WHERE THE TRUE PREDICTION HAPPENS. THIS MARKS THE TRUE END OF THE DECODER.</h4><p>After the encoder-decoder attention scores have been calculated and the residual connections have been added, we now need to predict the next output word, which would be ‘Arigato’. The procedure would be to pass the new final outputs for each word in the decoder through a standard feed-forward neural network with the number of output neurons matching the number of words in the entire vocabulary. In this example we will only be showing 4 output neurons matching 4 words.</p><p>The number of output neurons in the feed-forward neural network must match the vocabulary size of the entire dataset, however, we have just 4 in this case.</p><p>It adopts a similar procedure used to find the word embedding values for each word, only this time, biases are added to each output neuron just like how we added the positional encoding values to each output neuron of the word embedding values for each word, and one-hot encoded values aren’t utilized. Biases are random numbers which will be altered as the model continues to learn.</p><p><strong>Procedure:</strong></p><ol><li>The final outputs for each word after the residual connections, are passed through a standard feed-forward layer or neural network.</li><li>Biases are added to each result in the output neurons. Biases are just random numbers just like the weights which will be altered as the model learns.</li><li>The results are passed through a SoftMax function to get the highest SoftMax probabilities of the output word which would be ‘Arigato’.</li></ol><p>See diagram below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CzA0mfToKeMzAbcGKezsXA.png" /></figure><p>After ‘Arigato’ has been predicted, the transformer model needs to know when to stop making predictions. As I mentioned earlier, we can’t use full stops because input and output paragraphs to the LLM might contain full stops within them so, an &lt;EOS&gt; token is used instead.</p><p>For the model to stop making predictions, it needs to predict &lt;EOS&gt; as its final word after ‘Arigato’, so the model knows when to stop generating future words or tokens.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ark5B4RShP6E_rUGIDBhXQ.png" /></figure><h4>FINAL DIAGRAM EXPLAINING HOW THE DECODER FINALLY PREDICTS EACH WORD IN THE OUTPUT SENTENCE.</h4><figure><img alt="Diagram explaining how the decoder predicts each word in the output sentence of a transformer." src="https://cdn-images-1.medium.com/max/1024/1*yiis8Gt3h2ECMEZwh6xXLA.png" /><figcaption>Diagram explaining how the decoder predicts each word in the output sentence of a transformer.</figcaption></figure><ul><li>First layer with the 1s and 0s represents the <strong>input tokens</strong> for each word. These are called one-hot encoded vectors.</li><li>Second layer represents the <strong>word embedding values</strong> for each word which are added to s<strong>ine and cosine wave points </strong>in yellow to get the <strong>positional encoding</strong> values.</li><li>Third layer represents the <strong>positional encoding values</strong>.</li><li>Fourth layer represents the <strong>self-attention layer</strong>, where the <strong>similarity scores and self-attention values</strong> are calculated.</li><li>Fifth layer represents the <strong>final self-attention values</strong> after calculation.</li><li>Sixth layer represents the <strong>final decoder outputs</strong> which are the results of the addition between the <strong>self-attention values</strong> and the <strong>positional encoding values </strong>called <strong>residual connections</strong>.</li><li>Seventh layer represents the <strong>encoder-decoder attention layer</strong>, where the similarities or relationships between each word in the encoder is calculated relative to that single word in the decoder.</li><li>Eighth layer represents the <strong>final encoder-decoder attention scores </strong>which are then added to the final decoder outputs as <strong>residual connections</strong> again to get the values above it.</li><li>Ninth layer represents the final outputs after residual connections between the <strong>final encoder-decoder attention scores </strong>and the previously calculated decoder outputs.</li><li>Tenth layer represents the beginning of the <strong>final feed-forward neural network</strong> used to predict the next token.</li><li>Eleventh layer contains the <strong>SoftMax percentages</strong> of the tenth layer.</li><li>Twelfth layer contains the <strong>final predictions or SoftMax probabilities </strong>for each word in the vocabulary.</li></ul><p>FINAL NOTE:</p><p>During training, if the final predicted output word from the decoder after the &lt;SOS&gt; token was predicted incorrectly, all of the weights in the entire transformer including the encoder and decoder and self-attention layers will be changed for the right word to be predicted (‘Arigato’).</p><p>RECAP:</p><p>The input sentence is fed into the encoder, and the expected output sentence is fed into the decoder during training. E.g. If I wanted to teach you how to respond to a sentence like ‘Hi there’, I would feed ‘Hi there’ into your encoder, and feed the expected response ‘Hi, how can I help you?’ into your decoder. That’s the reason for an encoder and decoder. Ultimately you will learn how to mention each word inside of the expected response (‘Hi, how can I help you?’) anytime I mention ‘Hi there’, only this time, I wouldn’t provide you with the expected response.</p><p>FINAL SUMMARY:</p><p>Encoder:</p><ol><li>The input sentence is fed into the encoder and word embedding values are calculated for each word in the sentence.</li><li>Positional encoding values are calculated by adding points on sine and cosine waves to the word embeddings to keep track of word order in a sentence because you could have two sentences with the exact same words but with different meanings.</li><li>Self-attention is calculated to find the relationship among each word in the input sentence to better understand the context. This is done by generating QUERIES, KEYS and VALUES, computing the dot products of the QUERIES and the transposed KEYS, dividing the dot products by square root of the dimensions of the KEY matrix, applying the SoftMax function row-wise on the scaled results and finally multiplying them by the VALUES to get the final self-attention scores.</li><li>Residual connections are added which involves adding the previously calculated positional encoding values to the new self-attention values to prevent values from being forgotten. These final values become the encoder’s outputs. Note: Residual connections are added after feed-forward and attention layers.</li></ol><p>Decoder:</p><ol><li>The input sentence is fed into the encoder and word embedding values are calculated for each word in the sentence.</li><li>Positional encoding values are calculated by adding points on sine and cosine waves to the word embeddings to keep track of word order in a sentence because you could have two sentences with the exact same words but with different meanings.</li><li>Self-attention is calculated to find the relationship among each word in the input sentence to better understand the context. This is done by generating QUERIES, KEYS and VALUES, computing the dot products of the QUERIES and the transposed KEYS, dividing the dot products by square root of the dimensions of the KEY matrix, applying an attention mask or<strong> look ahead mask </strong>to prevent the decoder from peeking ahead to see next words during training, applying the SoftMax function to get their SoftMax percentages row by row, and finally multiplying them by the VALUES to get the final self-attention scores.</li><li>Residual connections are added which involves adding the previously calculated positional encoding values to the new self-attention scores. These final values become the decoder’s outputs.</li></ol><p>Encoder-Decoder:</p><ol><li>Encoder-decoder attention is performed by first generating KEYS and VALUES from the encoder’s output, and QUERIES from the decoder’s current outputs, and performing dot product operations or calculations on all values. This is to find the relationship between all words in the encoder and each word in the decoder.</li><li>The encoder-decoder attention values are first divided by the square root of the dimensions of the KEY matrix (the number of KEYS representing each word) and then scaled by passing all values row by row into a SoftMax function to get their SoftMax percentages.</li><li>These SoftMax percentages are multiplied by the previously generated VALUES from the encoder’s output with their results added correspondingly to get the final encoder-decoder attention scores.</li><li>Residual connections are added which involves adding the previous decoder’s final outputs to the new final encoder-decoder attention scores to get the new outputs for each word.</li><li>These new outputs for each word are passed through a standard feed-forward neural network, with their results passed through a SoftMax function, to get the SoftMax percentages for the next output word to be predicted.</li><li>The output word with the highest SoftMax percentage becomes the predicted word output.</li></ol><p>Well done sailor! You made it to the end of this paper. Now you’re an expert on Large Language Models and transformers.</p><p>Source code to my LLM on GitHub:</p><p><a href="https://github.com/Piraytekinz/DeepLearning-projects/blob/main/Large%20Language%20Model.ipynb">DeepLearning-projects/Large Language Model.ipynb at main · Piraytekinz/DeepLearning-projects</a></p><p>[1] J. Anang, <em>LinkedIn Profile</em> (2025), <a href="https://www.linkedin.com/in/joshua-anang-611462285">Joshua Anang | LinkedIn</a></p><p>[2] J. Anang,<em> Twitter Profile</em>, <a href="https://x.com/ChatDoc783">Joshua Anang (@ChatDoc783) / X</a></p><p>[3] J. Anang, Current Project — An app which turns your images into awesome particle effects, https://array-psi.vercel.app</p><p><em>Credits to </em><a href="https://statquest.org/"><em>StatQuest</em></a><em> for similar diagram designs.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=57b82a0527f9" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>