<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Akshat Khatri on Medium]]></title>
        <description><![CDATA[Stories by Akshat Khatri on Medium]]></description>
        <link>https://medium.com/@akshat.dev?source=rss-cb3d13125e1c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*InCS8l9suYcUyRKNL6u_9w.png</url>
            <title>Stories by Akshat Khatri on Medium</title>
            <link>https://medium.com/@akshat.dev?source=rss-cb3d13125e1c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 04:37:37 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@akshat.dev/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Turn Detection From Scratch: Combining Audio and Language Models for Hindi-English Speech]]></title>
            <link>https://medium.com/@akshat.dev/turn-detection-from-scratch-combining-audio-and-language-models-for-hindi-english-speech-b85cb5d4da1d?source=rss-cb3d13125e1c------2</link>
            <guid isPermaLink="false">https://medium.com/p/b85cb5d4da1d</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[turn-detection]]></category>
            <category><![CDATA[audio-engineering]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Akshat Khatri]]></dc:creator>
            <pubDate>Tue, 15 Sep 2026 08:28:17 GMT</pubDate>
            <atom:updated>2026-09-15T08:28:17.369Z</atom:updated>
            <content:encoded><![CDATA[<p><strong><em>Greetings of the day, <br>The below given report presents my solution &amp; thinking process to the assignment problem statement , I have approached the problem with clear ML baselines , and a research-oriented approach (focusing on going from the most basic model to more advanced deep learning models) and measuring evaluation metrics at each step..</em></strong></p><blockquote>Please read till end of the article , to capture the full picture as i have used multiple approaches each with their own reasoning.</blockquote><p><em>Github pages and Google drive link is at end of article (Recommended!)</em></p><p>So Let’s START !</p><blockquote>The problem we are given is that of turn detection , i first researched into what turn detection means and this is what i arrived at</blockquote><blockquote>Turn — detection is when we want to know whether a user has stopped speaking or not … i connected this concept to how google voice chat knows when to stop and send my speech to the google search engine and when to hold .. so what i understood was we had to build such a system</blockquote><h3>Data Exploration</h3><p>The first thing i did was see what type of data we are dealing with .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MhcONcOt1FnaeHQ3W9y16Q.png" /><figcaption>HF dataset</figcaption></figure><p>The original Dataset is 271k rows (approx 41 GB) in size .. Also the assignment asked us to operate on only two languages</p><ul><li>Hindi (4.5 %)</li><li>English (24%)</li></ul><p>So i filtered the dataset to only include these 2 languages hence bringing down total rows from 271k -&gt; 62k datapoints</p><ul><li>Hindi (3%)</li><li>English (97%)</li></ul><p>Now our data is ready .. and after inspection i found that our two columns</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rZPxmn9TJ9EXfSB2IXR8og.png" /><figcaption>Both audio &amp; endpoint bool columns don’t have any null values present in them</figcaption></figure><h3>ML problem formulation</h3><p>I formulated the problem as a Binary classification Problem.<br>where ,</p><ul><li>X = Audio waveform</li><li>Y = endpoint_bool (label)..</li><li>T = Threshold (tunable)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WqgeFzYs1pZW-cyyJ8GJVg.jpeg" /></figure><h3>Data Splitting &amp; Preparation</h3><p>First, I will split the data into train, eval, and test sets, and lock away the test set to prevent exposure bias.</p><p>Now as we say earlier (english language dominates) , in such case if we randomly split the dataset into 80–10–10 rule .. we would risk not having hindi in eval or test set .. hence we can get overtly optimistic results..</p><blockquote>To make sure Hindi wasn’t left out , i did stratified sampling on language labels..<br>Ensuring that Hindi language has proportional representation in both validation and test set..</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8NdTo0AW8cMPi-DxhsQH0Q.png" /><figcaption>Startatified sampling</figcaption></figure><p>After stratified 80–10–10 split , we got the following counts of data points</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/973/1*QEV3K6dNqnoN-6hZMobBxA.png" /></figure><h3>Model Evaluation</h3><p>Now once we have the data , we will need a way to evalute our ML model on our task to measure its performance.</p><p>Since the Task is Binary classification , we select the following 4 metrics</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fdatawrapper.dwcdn.net%2FmxGH6%2F1%2F&amp;display_name=Datawrapper&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FmxGH6%2F1%2F&amp;image=https%3A%2F%2Fdatawrapper.dwcdn.net%2FmxGH6%2Fplain-s.png%3Fv%3D1&amp;type=text%2Fhtml&amp;schema=dwcdn" width="600" height="330" frameborder="0" scrolling="no"><a href="https://medium.com/media/eb89c1de91a4dd4a6184095ef8dd054d/href">https://medium.com/media/eb89c1de91a4dd4a6184095ef8dd054d/href</a></iframe><h3>Data Modelling</h3><p>Now , we will come to the most important part of trying out different methods to model the data using techniques such as</p><ul><li>Baseline Heuristic methods</li><li>Classic ML models (Logistic Regression, Random Forest, XGBoost)</li><li>Deep neural models (CNN, GRU, wav2vec2 e.t.c.)</li><li>Semantic Text models (Like google/muril-base-cased)</li><li>Hybrid models (Usually semantic models + Acoustic models).</li></ul><p><strong><em>I have divided the below section(s) in three parts</em></strong></p><ul><li><strong><em>1 to 3 [Acoustic Models]</em></strong></li><li><strong><em>4 [Semantic Text models]</em></strong></li><li><strong><em>5 [Hybrid Models]</em></strong></li></ul><h4>1. Baseline Model (RMS energy threshold)</h4><p>Our baseline strategy relies on an RMS energy threshold to detect conversational turns. We split the incoming audio into micro-frames, flag any frame below a predefined volume limit as silence, and count backwards from the end of the input. If the clip ends with at least 0.7 seconds of continuous silence, the system detects a turn boundary</p><p>In plain terms , it simply listens to the last ’n’ seconds of an audio and sees if voice is above a threshold or not..</p><p>After running this simple heuristic and evaluating it on our validation_set , the results were disappointing as expected by such naive algorithm.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TCjoUB5NNTt3YtenapGeKw.png" /><figcaption>Baseline model results</figcaption></figure><p>So now , we know that a heuristic model gives us a F1 score of only 0.13 , which is very low.. so now we will move on to better algorithms, and hope that we beat this F1-score.</p><h4>2. Classical ML Models on Acoustic Features</h4><p>Now , we would move into the realm of proper machine learning algorithms .. but before that we need some form of feature extraction from the data .. since the data we are dealing with is in audio waveform , we will take the help from <strong>librosa</strong> library to generate some features for ourselves.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fdatawrapper.dwcdn.net%2FhSnxO%2F3%2F&amp;display_name=Datawrapper&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FhSnxO%2F3%2F&amp;image=https%3A%2F%2Fdatawrapper.dwcdn.net%2FhSnxO%2Fplain-s.png%3Fv%3D3&amp;type=text%2Fhtml&amp;schema=dwcdn" width="600" height="525" frameborder="0" scrolling="no"><a href="https://medium.com/media/ac13e49beba14b7a55292c0e54ffd23e/href">https://medium.com/media/ac13e49beba14b7a55292c0e54ffd23e/href</a></iframe><p>More info can be found in the librosa library <a href="https://librosa.org/doc/">docs</a>.</p><p>For context, this is how a log-mel spectogram looks like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*4MMXPvHSWFDGBYlM.png" /></figure><p>Before i go forward , i want you to see the shape of what librosa feature extractor returns</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ltDFJ_AG8r3gy6cU2RE06g.png" /><figcaption>Shape of log-mel-spectogram</figcaption></figure><p>Nearly all classic ML algorithms need a fixed feature vector .. but acoustic features like log-mel-spectogram, or ZCR or MFCC all returns np.array of shape [n_mels, T] .. where T is the number of time frames..</p><p>Now , audios do not have fixed time frames an audio can have 5600 timeframes , other can have 7000 timeframes..</p><p>So we need a way to collapse those timeframes into one single objective number that we can then feed to our classic ML models, enter our 5 summary stats that we will now use..</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fHkF5dN7IIMCWTP5Dy46Bw.jpeg" /><figcaption>Summary stats Transform</figcaption></figure><p><em>The summary stats we used were</em></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fdatawrapper.dwcdn.net%2Frwc2M%2F1%2F&amp;display_name=Datawrapper&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2Frwc2M%2F1%2F&amp;image=https%3A%2F%2Fdatawrapper.dwcdn.net%2Frwc2M%2Fplain-s.png%3Fv%3D1&amp;type=text%2Fhtml&amp;schema=dwcdn" width="600" height="318" frameborder="0" scrolling="no"><a href="https://medium.com/media/aad180cfe34dd47278aa09c14d3f122f/href">https://medium.com/media/aad180cfe34dd47278aa09c14d3f122f/href</a></iframe><p>Now , due to computational constraint , as i do not have a laptop with GPU and time boundation i decided to move forward with 3 features only , instead of yin or pyin as they took much time to calculate with little benefit</p><p>So after , extracting acoustic Features this is our shape of X &amp; Y</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bZ7m2cRgm6KhiSKUHyw3Qg.png" /><figcaption>A total of 710 features…(142 * 5 summary stats)</figcaption></figure><p>Now , here i experimented with 3 different ML models namely</p><ul><li>RandomForest Classifier</li><li>Xgboost</li><li>SVM model for on-the-fly online learning.</li></ul><p>I won’t go deep into how each of the model works , as that is beyond the scope of the article but i can share the results i got from each.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nrEtffHroxS1_4MhWCznxA.png" /><figcaption>Evaluation result of each model</figcaption></figure><p>Clearly XGboost performed the best with an F1 score of 0.66.. compared to our earlier heuristic this is signifcant improvement .. but still we can do better ..</p><p>Also you may wonder , why did i use Online SVM .. the reason for that was earlier during development i took a lot of features .. and those took time to pre-compute so , i instead did partial-fit algorithms .. and online SVM was one of only non-linear online binary classifiers available .. so that was the reasoning behind my approach .</p><p>But after cutting down on features i pre-computed the features .. and XGBooost performed the best with an F1-score of 0.66.</p><h4>3. Time Series Deep Learning Models</h4><p>The main problem that we were facing with using a classic ML model was that we compressed time-frames into summary stats.</p><p>Doing such a compression is a lossy process and leads to loss of information that can’t be recovered by a model.. so we need such models that operate directly on time-frame data.</p><p>The models that came to my mind were the RNN family of models, specifically LSTM and GRUs.</p><p>The Architecture can be defined as below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PhHVPnP8vxpkegzbqf6t8g.jpeg" /><figcaption>Architecture.</figcaption></figure><p>You may or may not include the CNN layers included , the reason i included them was that after some experiments i found out that CNN helped me converge faster as it helped with spatial dependencies</p><p>We take the last hidden state of the GRU of size (B, T0) .. as the last layer of GRU encodes information of all the frames before it , topped off by a linear classifier on top with BCE logits loss and an AdamW optimizer.</p><p>For this model i only used a single feature of log-mel-spectogram.</p><p>The results are..</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2iWCDuheSEYrAnzQ8T-d2Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vINvXGNog-XmC7cDK0B3dA.png" /><figcaption>Results of CNN + GRU model</figcaption></figure><p>The CNN + GRU model performed even better than our heuristic and classical ML model with an F-1 score of 0.84.</p><p>Due to crunch of time , i couldnt experiment much with hyper-parameters that could have pushed F1 score to +- 0.3.</p><h3>4. Semantic Text Models</h3><p>Now we move from the realm of acoustic to semantic meaning.</p><p>The best way to Understand how semantic text models can help with turn detection is by an example.</p><blockquote><strong><em>Suppose person A is standing in a busy market with a lot of noise around him , and the person says “ find me a good place to dine with my ..” and then he stops (likely interrupted by someone) , now if i give you two sentences..</em></strong></blockquote><blockquote><strong><em>Sentence A = “find me a good place to dine with my”<br>Sentence B = “find me a good place to dine with my friends”</em></strong></blockquote><blockquote><strong><em>The more likely complete word is Sentence B. This is the crux of semantic text models being used for turn detection.</em></strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7cPCIVPW7eGt7BjazzWzjA.png" /><figcaption>Semantic Text model Architecture</figcaption></figure><p>For this we would need a Transcription model , which converts audio to text both in english and hindi , for our purposes we will be using</p><p><em>“whisper large v3 turbo model , with FP16 quantization”</em></p><p>The reason for choosing this is it supports both hindi, english and hinglish transcriptions, also turbo model is faster than regular large one , with a very minor drop in accuracy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hKqBbTeJb5CAQfvwQG59jQ.png" /><figcaption>Different whisper models</figcaption></figure><p>Here it took me a lot of time .. due to compute and time shortage i had to spend a lot of time tuning whisper models and experimenting with different ones .. so this added up to time.</p><p>After around 8 Hours all transcripts were done , and i got two files</p><ul><li>Train_transcription.jsonl</li><li>val_transcription.jsonl</li></ul><p>For my purposes i prepared two parquet files</p><ul><li>one with all “id, raw waveform” of our dataset.</li><li>another with “id , label” from the dataset.</li></ul><p>THEN i perform a SQL join on both the tables to get our dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l_YnYVEgwrfSOAqBNVpBkA.png" /><figcaption>Final table ready for BERT training.</figcaption></figure><p>Now our dataset is prepared .. and we can begin our BERT pre-training on a Sequence Classification objective.</p><p><strong><em>For our purpose we will be choosing google/muril-base-cased BERT model. MuRIL is a BERT model pre-trained on 17 Indian languages and english<br>making it perfect for us.</em></strong></p><p>Here are the results of the semantic text model on validation set for the first 4000 steps (for context one epoch has 65000 steps).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*eBWlkSKne2YrBhKgDx3RsQ.png" /><figcaption>BERT-pre training logs.</figcaption></figure><p>Even i was surprised by how good the semantic text model was ..</p><p>Apparently one of the reasons behind it was when whisper large transcribes something which has a pause or stutter at end it does so like this</p><p>Sentence A (raw) : i want to go to.<br>Sentence B (whisper-large) : i want to go to…</p><p>Apparently , these extra-dots give a useful signal to the semantic model indicating that its a pause.. , among other things ..</p><blockquote>So our semantic Text model is the clear winner upto now with a whooping F1-score of 0.94 on the validation set.</blockquote><h3>5. Hybrid Models</h3><p>The idea behind hybrid models is rather simple , we combine the confidence of both our Acoustic model , and semantic model or more formally we can say,</p><p>We perform soft-voting ensemble.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UHxXxMWc6A_DLPbhiXpYBA.jpeg" /><figcaption>Weighted Sum of both probabilties (where α + β = 1.0)</figcaption></figure><p>The idea behind it was simple suppose for a sentence acoustic is very confident say , but for some semantic is not much sure , then acoustic would offset that loss. for e.g.</p><p>Suppose,</p><p>P1 (audio = 1) = 0.90<br>P2 (audio = 1) = 0.40<br>C = 0.5 * P1 + 0.5 * P2 = 0.65<br>So the model predicts the class is positive.</p><p>Upon evaluating the results on the validation set , following came</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ytEHkZu-VStwCPqFs_hfiQ.png" /><figcaption>F1-score = 0.958</figcaption></figure><blockquote>Our best model yet is an ensemble of both (Text + Audio) achieving an impressive F1-score of ~0.96.</blockquote><h4>Deployment &amp; Code</h4><p>To make the project reproducible and easy to explore, I have made the complete implementation available on GitHub:</p><ul><li><strong>GitHub Repository:</strong> <a href="https://github.com/akshatkhatri/Turn_Detection_model">https://github.com/akshatkhatri/Turn_Detection_model</a></li><li><strong>Demo / Project URL:</strong> <a href="https://akshatkhatri.github.io/Turn_Detection_model/">https://akshatkhatri.github.io/Turn_Detection_model/</a></li><li><strong>Handwritten Notes for my thought process (G-drive Link)</strong><br><a href="https://drive.google.com/drive/folders/13RcCAnjqbES5rsWtFrkFtkZsi6PRbwha">https://drive.google.com/drive/folders/13RcCAnjqbES5rsWtFrkFtkZsi6PRbwha</a></li></ul><p>Deploying the complete pipeline presented an additional challenge because the final system combines multiple components, including audio processing, transcription, acoustic modelling, and semantic classification. The size of the models and dataset also made hosting the complete application on a lightweight static platform impractical.</p><p>For the demo, I therefore hosted the inference application separately and exposed it through a <strong>Cloudflare Tunnel</strong>, while using GitHub Pages as a persistent public entry point.</p><h4>Key Learnings &amp; Future Work</h4><p>This project gave me the opportunity to explore turn detection from several different perspectives rather than treating it as a single binary-classification problem.</p><p>Starting from a simple RMS-based heuristic and gradually moving toward classical ML, time-series deep learning, semantic modelling, and finally a hybrid ensemble helped me understand how different sources of information contribute to the task. The experiments improved the validation F1-score from <strong>0.13 for the baseline heuristic to approximately 0.96 for the final hybrid model</strong>.</p><p>Given more time and compute, I would also explore:</p><ul><li>More extensive hyperparameter tuning for the CNN-GRU acoustic model.</li><li>Optimizing the ensemble weights and classification threshold on the validation set.</li><li>Comparing models not only on F1-score, but also on <strong>precision, recall, inference latency, and computational cost</strong>.</li><li>Testing the system on more Hindi and Hinglish samples to better understand performance across languages.</li><li>Exploring a lighter production architecture that retains most of the hybrid model’s accuracy while reducing inference latency.</li></ul><p>Overall, this assignment gave me practical exposure to an area of audio ML that I had not worked deeply with before. More importantly, it showed me that turn detection is not purely an acoustic problem: <strong>silence, temporal patterns, linguistic completeness, and contextual information can all contribute useful signals</strong>.</p><p>Regards</p><p>Akshat Khatri (Learner &amp; ML enthusiast)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b85cb5d4da1d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Shiprocket Assignment: Human Report — Turn Detection Model from Scratch]]></title>
            <link>https://medium.com/@akshat.dev/shiprocket-assignment-human-report-turn-detection-model-from-scratch-8f781fb2dffa?source=rss-cb3d13125e1c------2</link>
            <guid isPermaLink="false">https://medium.com/p/8f781fb2dffa</guid>
            <category><![CDATA[xgboost]]></category>
            <category><![CDATA[time-series-data]]></category>
            <category><![CDATA[audio-engineering]]></category>
            <category><![CDATA[bert]]></category>
            <category><![CDATA[feature-engineering]]></category>
            <dc:creator><![CDATA[Akshat Khatri]]></dc:creator>
            <pubDate>Tue, 25 Aug 2026 05:26:41 GMT</pubDate>
            <atom:updated>2026-09-15T08:06:34.445Z</atom:updated>
            <content:encoded><![CDATA[<h3>Turn Detection From Scratch: Combining Audio and Language Models for Hindi-English Speech</h3><p><strong><em>Greetings of the day, <br>The below given report presents my solution &amp; thinking process to the assignment problem statement , I have approached the problem with clear ML baselines , and a research-oriented approach (focusing on going from the most basic model to more advanced deep learning models) and measuring evaluation metrics at each step..</em></strong></p><blockquote>Please read till end of the article , to capture the full picture as i have used multiple approaches each with their own reasoning.</blockquote><p><em>Github pages and Google drive link is at end of article (Recommended!)</em></p><p>So Let’s START !</p><blockquote>The problem we are given is that of turn detection , i first researched into what turn detection means and this is what i arrived at</blockquote><blockquote>Turn — detection is when we want to know whether a user has stopped speaking or not … i connected this concept to how google voice chat knows when to stop and send my speech to the google search engine and when to hold .. so what i understood was we had to build such a system</blockquote><h3>Data Exploration</h3><p>The first thing i did was see what type of data we are dealing with .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MhcONcOt1FnaeHQ3W9y16Q.png" /><figcaption>HF dataset</figcaption></figure><p>The original Dataset is 271k rows (approx 41 GB) in size .. Also the assignment asked us to operate on only two languages</p><ul><li>Hindi (4.5 %)</li><li>English (24%)</li></ul><p>So i filtered the dataset to only include these 2 languages hence bringing down total rows from 271k -&gt; 62k datapoints</p><ul><li>Hindi (3%)</li><li>English (97%)</li></ul><p>Now our data is ready .. and after inspection i found that our two columns</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rZPxmn9TJ9EXfSB2IXR8og.png" /><figcaption>Both audio &amp; endpoint bool columns don’t have any null values present in them</figcaption></figure><h3>ML problem formulation</h3><p>I formulated the problem as a Binary classification Problem.<br>where ,</p><ul><li>X = Audio waveform</li><li>Y = endpoint_bool (label)..</li><li>T = Threshold (tunable)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WqgeFzYs1pZW-cyyJ8GJVg.jpeg" /></figure><h3>Data Splitting &amp; Preparation</h3><p>First, I will split the data into train, eval, and test sets, and lock away the test set to prevent exposure bias.</p><p>Now as we say earlier (english language dominates) , in such case if we randomly split the dataset into 80–10–10 rule .. we would risk not having hindi in eval or test set .. hence we can get overtly optimistic results..</p><blockquote>To make sure Hindi wasn’t left out , i did stratified sampling on language labels..<br>Ensuring that Hindi language has proportional representation in both validation and test set..</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8NdTo0AW8cMPi-DxhsQH0Q.png" /><figcaption>Startatified sampling</figcaption></figure><p>After stratified 80–10–10 split , we got the following counts of data points</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/973/1*QEV3K6dNqnoN-6hZMobBxA.png" /></figure><h3>Model Evaluation</h3><p>Now once we have the data , we will need a way to evalute our ML model on our task to measure its performance.</p><p>Since the Task is Binary classification , we select the following 4 metrics</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fdatawrapper.dwcdn.net%2FmxGH6%2F1%2F&amp;display_name=Datawrapper&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FmxGH6%2F1%2F&amp;image=https%3A%2F%2Fdatawrapper.dwcdn.net%2FmxGH6%2Fplain-s.png%3Fv%3D1&amp;type=text%2Fhtml&amp;schema=dwcdn" width="600" height="330" frameborder="0" scrolling="no"><a href="https://medium.com/media/eb89c1de91a4dd4a6184095ef8dd054d/href">https://medium.com/media/eb89c1de91a4dd4a6184095ef8dd054d/href</a></iframe><h3>Data Modelling</h3><p>Now , we will come to the most important part of trying out different methods to model the data using techniques such as</p><ul><li>Baseline Heuristic methods</li><li>Classic ML models (Logistic Regression, Random Forest, XGBoost)</li><li>Deep neural models (CNN, GRU, wav2vec2 e.t.c.)</li><li>Semantic Text models (Like google/muril-base-cased)</li><li>Hybrid models (Usually semantic models + Acoustic models).</li></ul><p><strong><em>I have divided the below section(s) in three parts</em></strong></p><ul><li><strong><em>1 to 3 [Acoustic Models]</em></strong></li><li><strong><em>4 [Semantic Text models]</em></strong></li><li><strong><em>5 [Hybrid Models]</em></strong></li></ul><h4>1. Baseline Model (RMS energy threshold)</h4><p>Our baseline strategy relies on an RMS energy threshold to detect conversational turns. We split the incoming audio into micro-frames, flag any frame below a predefined volume limit as silence, and count backwards from the end of the input. If the clip ends with at least 0.7 seconds of continuous silence, the system detects a turn boundary</p><p>In plain terms , it simply listens to the last ’n’ seconds of an audio and sees if voice is above a threshold or not..</p><p>After running this simple heuristic and evaluating it on our validation_set , the results were disappointing as expected by such naive algorithm.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TCjoUB5NNTt3YtenapGeKw.png" /><figcaption>Baseline model results</figcaption></figure><p>So now , we know that a heuristic model gives us a F1 score of only 0.13 , which is very low.. so now we will move on to better algorithms, and hope that we beat this F1-score.</p><h4>2. Classical ML Models on Acoustic Features</h4><p>Now , we would move into the realm of proper machine learning algorithms .. but before that we need some form of feature extraction from the data .. since the data we are dealing with is in audio waveform , we will take the help from <strong>librosa</strong> library to generate some features for ourselves.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fdatawrapper.dwcdn.net%2FhSnxO%2F3%2F&amp;display_name=Datawrapper&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FhSnxO%2F3%2F&amp;image=https%3A%2F%2Fdatawrapper.dwcdn.net%2FhSnxO%2Fplain-s.png%3Fv%3D3&amp;type=text%2Fhtml&amp;schema=dwcdn" width="600" height="525" frameborder="0" scrolling="no"><a href="https://medium.com/media/ac13e49beba14b7a55292c0e54ffd23e/href">https://medium.com/media/ac13e49beba14b7a55292c0e54ffd23e/href</a></iframe><p>More info can be found in the librosa library <a href="https://librosa.org/doc/">docs</a>.</p><p>For context, this is how a log-mel spectogram looks like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*4MMXPvHSWFDGBYlM.png" /></figure><p>Before i go forward , i want you to see the shape of what librosa feature extractor returns</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ltDFJ_AG8r3gy6cU2RE06g.png" /><figcaption>Shape of log-mel-spectogram</figcaption></figure><p>Nearly all classic ML algorithms need a fixed feature vector .. but acoustic features like log-mel-spectogram, or ZCR or MFCC all returns np.array of shape [n_mels, T] .. where T is the number of time frames..</p><p>Now , audios do not have fixed time frames an audio can have 5600 timeframes , other can have 7000 timeframes..</p><p>So we need a way to collapse those timeframes into one single objective number that we can then feed to our classic ML models, enter our 5 summary stats that we will now use..</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fHkF5dN7IIMCWTP5Dy46Bw.jpeg" /><figcaption>Summary stats Transform</figcaption></figure><p><em>The summary stats we used were</em></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fdatawrapper.dwcdn.net%2Frwc2M%2F1%2F&amp;display_name=Datawrapper&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2Frwc2M%2F1%2F&amp;image=https%3A%2F%2Fdatawrapper.dwcdn.net%2Frwc2M%2Fplain-s.png%3Fv%3D1&amp;type=text%2Fhtml&amp;schema=dwcdn" width="600" height="318" frameborder="0" scrolling="no"><a href="https://medium.com/media/aad180cfe34dd47278aa09c14d3f122f/href">https://medium.com/media/aad180cfe34dd47278aa09c14d3f122f/href</a></iframe><p>Now , due to computational constraint , as i do not have a laptop with GPU and time boundation i decided to move forward with 3 features only , instead of yin or pyin as they took much time to calculate with little benefit</p><p>So after , extracting acoustic Features this is our shape of X &amp; Y</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bZ7m2cRgm6KhiSKUHyw3Qg.png" /><figcaption>A total of 710 features…(142 * 5 summary stats)</figcaption></figure><p>Now , here i experimented with 3 different ML models namely</p><ul><li>RandomForest Classifier</li><li>Xgboost</li><li>SVM model for on-the-fly online learning.</li></ul><p>I won’t go deep into how each of the model works , as that is beyond the scope of the article but i can share the results i got from each.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nrEtffHroxS1_4MhWCznxA.png" /><figcaption>Evaluation result of each model</figcaption></figure><p>Clearly XGboost performed the best with an F1 score of 0.66.. compared to our earlier heuristic this is signifcant improvement .. but still we can do better ..</p><p>Also you may wonder , why did i use Online SVM .. the reason for that was earlier during development i took a lot of features .. and those took time to pre-compute so , i instead did partial-fit algorithms .. and online SVM was one of only non-linear online binary classifiers available .. so that was the reasoning behind my approach .</p><p>But after cutting down on features i pre-computed the features .. and XGBooost performed the best with an F1-score of 0.66.</p><h4>3. Time Series Deep Learning Models</h4><p>The main problem that we were facing with using a classic ML model was that we compressed time-frames into summary stats.</p><p>Doing such a compression is a lossy process and leads to loss of information that can’t be recovered by a model.. so we need such models that operate directly on time-frame data.</p><p>The models that came to my mind were the RNN family of models, specifically LSTM and GRUs.</p><p>The Architecture can be defined as below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PhHVPnP8vxpkegzbqf6t8g.jpeg" /><figcaption>Architecture.</figcaption></figure><p>You may or may not include the CNN layers included , the reason i included them was that after some experiments i found out that CNN helped me converge faster as it helped with spatial dependencies</p><p>We take the last hidden state of the GRU of size (B, T0) .. as the last layer of GRU encodes information of all the frames before it , topped off by a linear classifier on top with BCE logits loss and an AdamW optimizer.</p><p>For this model i only used a single feature of log-mel-spectogram.</p><p>The results are..</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2iWCDuheSEYrAnzQ8T-d2Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vINvXGNog-XmC7cDK0B3dA.png" /><figcaption>Results of CNN + GRU model</figcaption></figure><p>The CNN + GRU model performed even better than our heuristic and classical ML model with an F-1 score of 0.84.</p><p>Due to crunch of time , i couldnt experiment much with hyper-parameters that could have pushed F1 score to +- 0.3.</p><h3>4. Semantic Text Models</h3><p>Now we move from the realm of acoustic to semantic meaning.</p><p>The best way to Understand how semantic text models can help with turn detection is by an example.</p><blockquote><strong><em>Suppose person A is standing in a busy market with a lot of noise around him , and the person says “ find me a good place to dine with my ..” and then he stops (likely interrupted by someone) , now if i give you two sentences..</em></strong></blockquote><blockquote><strong><em>Sentence A = “find me a good place to dine with my”<br>Sentence B = “find me a good place to dine with my friends”</em></strong></blockquote><blockquote><strong><em>The more likely complete word is Sentence B. This is the crux of semantic text models being used for turn detection.</em></strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7cPCIVPW7eGt7BjazzWzjA.png" /><figcaption>Semantic Text model Architecture</figcaption></figure><p>For this we would need a Transcription model , which converts audio to text both in english and hindi , for our purposes we will be using</p><p><em>“whisper large v3 turbo model , with FP16 quantization”</em></p><p>The reason for choosing this is it supports both hindi, english and hinglish transcriptions, also turbo model is faster than regular large one , with a very minor drop in accuracy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hKqBbTeJb5CAQfvwQG59jQ.png" /><figcaption>Different whisper models</figcaption></figure><p>Here it took me a lot of time .. due to compute and time shortage i had to spend a lot of time tuning whisper models and experimenting with different ones .. so this added up to time.</p><p>After around 8 Hours all transcripts were done , and i got two files</p><ul><li>Train_transcription.jsonl</li><li>val_transcription.jsonl</li></ul><p>For my purposes i prepared two parquet files</p><ul><li>one with all “id, raw waveform” of our dataset.</li><li>another with “id , label” from the dataset.</li></ul><p>THEN i perform a SQL join on both the tables to get our dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l_YnYVEgwrfSOAqBNVpBkA.png" /><figcaption>Final table ready for BERT training.</figcaption></figure><p>Now our dataset is prepared .. and we can begin our BERT pre-training on a Sequence Classification objective.</p><p><strong><em>For our purpose we will be choosing google/muril-base-cased BERT model. MuRIL is a BERT model pre-trained on 17 Indian languages and english<br>making it perfect for us.</em></strong></p><p>Here are the results of the semantic text model on validation set for the first 4000 steps (for context one epoch has 65000 steps).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*eBWlkSKne2YrBhKgDx3RsQ.png" /><figcaption>BERT-pre training logs.</figcaption></figure><p>Even i was surprised by how good the semantic text model was ..</p><p>Apparently one of the reasons behind it was when whisper large transcribes something which has a pause or stutter at end it does so like this</p><p>Sentence A (raw) : i want to go to.<br>Sentence B (whisper-large) : i want to go to…</p><p>Apparently , these extra-dots give a useful signal to the semantic model indicating that its a pause.. , among other things ..</p><blockquote><strong><em>So our semantic Text model is the clear winner upto now with a whooping F1-score of 0.94 on the validation set.</em></strong></blockquote><h3>5. Hybrid Models</h3><p>The idea behind hybrid models is rather simple , we combine the confidence of both our Acoustic model , and semantic model or more formally we can say,</p><p>We perform soft-voting ensemble.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UHxXxMWc6A_DLPbhiXpYBA.jpeg" /><figcaption>Weighted Sum of both probabilties (where α + β = 1.0)</figcaption></figure><p>The idea behind it was simple suppose for a sentence acoustic is very confident say , but for some semantic is not much sure , then acoustic would offset that loss. for e.g.</p><p>Suppose,</p><p>P1 (audio = 1) = 0.90<br>P2 (audio = 1) = 0.40<br>C = 0.5 * P1 + 0.5 * P2 = 0.65<br>So the model predicts the class is positive.</p><p>Upon evaluating the results on the validation set , following came</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ytEHkZu-VStwCPqFs_hfiQ.png" /><figcaption>F1-score = 0.958</figcaption></figure><blockquote>Our best model yet is an ensemble of both (Text + Audio) achieving an impressive F1-score of ~0.96.</blockquote><h4>Deployment &amp; Code</h4><p>To make the project reproducible and easy to explore, I have made the complete implementation available on GitHub:</p><ul><li><strong>GitHub Repository:</strong> <a href="https://github.com/akshatkhatri/Turn_Detection_model">https://github.com/akshatkhatri/Turn_Detection_model</a></li><li><strong>Demo / Project URL:</strong> <a href="https://akshatkhatri.github.io/Turn_Detection_model/">https://akshatkhatri.github.io/Turn_Detection_model/</a></li><li><strong>Handwritten Notes for my thought process (G-drive Link)</strong><br><a href="https://drive.google.com/drive/folders/13RcCAnjqbES5rsWtFrkFtkZsi6PRbwha">https://drive.google.com/drive/folders/13RcCAnjqbES5rsWtFrkFtkZsi6PRbwha</a></li></ul><p>Deploying the complete pipeline presented an additional challenge because the final system combines multiple components, including audio processing, transcription, acoustic modelling, and semantic classification. The size of the models and dataset also made hosting the complete application on a lightweight static platform impractical.</p><p>For the demo, I therefore hosted the inference application separately and exposed it through a <strong>Cloudflare Tunnel</strong>, while using GitHub Pages as a persistent public entry point.</p><h4>Key Learnings &amp; Future Work</h4><p>This project gave me the opportunity to explore turn detection from several different perspectives rather than treating it as a single binary-classification problem.</p><p>Starting from a simple RMS-based heuristic and gradually moving toward classical ML, time-series deep learning, semantic modelling, and finally a hybrid ensemble helped me understand how different sources of information contribute to the task. The experiments improved the validation F1-score from <strong>0.13 for the baseline heuristic to approximately 0.96 for the final hybrid model</strong>.</p><p>Given more time and compute, I would also explore:</p><ul><li>More extensive hyperparameter tuning for the CNN-GRU acoustic model.</li><li>Optimizing the ensemble weights and classification threshold on the validation set.</li><li>Comparing models not only on F1-score, but also on <strong>precision, recall, inference latency, and computational cost</strong>.</li><li>Testing the system on more Hindi and Hinglish samples to better understand performance across languages.</li><li>Exploring a lighter production architecture that retains most of the hybrid model’s accuracy while reducing inference latency.</li></ul><p>Overall, this assignment gave me practical exposure to an area of audio ML that I had not worked deeply with before. More importantly, it showed me that turn detection is not purely an acoustic problem: <strong>silence, temporal patterns, linguistic completeness, and contextual information can all contribute useful signals</strong>.</p><p>Really looking forward to learn with the Shiprocket Team :)</p><p>Regards</p><p>Akshat Khatri (Data scientist)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f781fb2dffa" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Two-Tower Recommendation Engine: Combining BERT, ALS, and Popularity-Weighted Negative…]]></title>
            <link>https://medium.com/@akshat.dev/a-walkthrough-of-building-a-personalised-discovery-engine-with-bert-als-and-smart-negative-0e114bae2322?source=rss-cb3d13125e1c------2</link>
            <guid isPermaLink="false">https://medium.com/p/0e114bae2322</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[bert]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[recommendation-system]]></category>
            <dc:creator><![CDATA[Akshat Khatri]]></dc:creator>
            <pubDate>Thu, 05 Feb 2026 17:29:29 GMT</pubDate>
            <atom:updated>2026-02-05T17:44:11.430Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building a Two-Tower Recommendation Engine: Combining BERT, ALS, and Popularity-Weighted Negative Sampling</h3><blockquote>Hello, my name is Akshat and i am an Engineering student of computer science, this recommendation system was given to me as part of an take-home assignment , but i took the assignment seriously and tried a hybrid approach to building such system, i took my inspiration by reading papers from Youtube 2016 , alibaba COLD to understand how these systems are built at scale and tried my own approach of building one such system.</blockquote><p>I tried building a recommendation system myself, drawing inspiration from different architectures, reasoning about assumptions, forming mental maps, and an old habit of mine: scribbling in my notebook for the smallest things I found interesting (SPOILER ALERT!! I found a lot of them interesting).</p><p>I’ve also shared my handwritten notes where I reasoned through the architecture and trade-offs while building this system. You can view them here: <a href="https://drive.google.com/drive/folders/1EDKrPt8dCWgsq8JRsipZtVrQUorqqnwU?usp=drive_link"><strong><em>[Handwritten design notes (Google Drive)]</em></strong><em>.</em></a></p><p><strong>A glimpse of what we are going to build</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/788/1*2cRwDhMMKzqejaGeRjnpCw.png" /><figcaption>Deep ranking two tower architecture.</figcaption></figure><h3>What was the problem?</h3><p>We are given a dataset which contains data about how users interacted with different products and what their reviews about those products are, along with ratings about those products, and other metadata.</p><p>And based on this data, we have to form a Recommendation System for the users, both new and existing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/523/1*A5ixWFq8pGPLr5acHEmsaw.png" /><figcaption>dataframe info</figcaption></figure><p>I will explain the ambiguous columns and what they mean</p><ul><li>Id: This is our product id , which is unique per product (alpha-numeric), for e.g. AV13O1A8GV-KLJ3akUyj.</li><li>brand : Brand of the product the user purchased.</li><li>categories : comma-separated categories the product belongs to such as books,music e.t.c.</li><li>ean,keys and name : these are product-specific identifiers.</li></ul><p>The rest of the columns are pretty self-explanatory,so i will move forward.</p><h3>Dealing with missing values in the dataset</h3><p>I filled the missing data using heuristics based on my feel and assumption of the dataset, here’s what i did</p><pre>df[&#39;reviews.title&#39;] = df[&#39;reviews.title&#39;].fillna(&#39;No Title&#39;) # fills empty titles<br>df[&#39;reviews.text&#39;] = df[&#39;reviews.text&#39;].fillna(&#39;No Review&#39;)<br>df[&#39;reviews.username&#39;] = df[&#39;reviews.username&#39;].fillna(&#39;Akshat&#39;) # Can also assign it as anonymous user but for sake of info i&#39;ll pretend i bought these products<br>df[&#39;manufacturer&#39;] = df[&#39;manufacturer&#39;].fillna(df[&#39;manufacturer&#39;].mode()) # most occuring manufacturer</pre><ul><li>Review title and text were filled as No title &amp; No review respectively (this is to make BERT understand them better)</li><li>Username was filled to be Akshat (this can be changed to anonymous users but they were already dominating so i switched it to me)</li><li>manufacturer was replaced with the mode of the column as it is a categorical variable.</li></ul><p>Note that this is not the best way to fill the missing values, but this is just an assumption and time was a crunch so i had to go with simple and effective methods.</p><h3>Label Encoding Products and users</h3><p>Next step is we create a mapping between unique products and users and give them unique id’s in the form of 0 to n-1 (where n is unique products or users present in the dataframe), this will make indexing easier for us in the future.</p><p>Also we convert our ratings column into a float (you will understand why we did it).</p><pre>user_map = {u: i for i, u in enumerate(df[&#39;reviews.username&#39;].unique())}<br>item_map = {i: j for j, i in enumerate(df[&#39;id&#39;].unique())}<br>df[&#39;user_id&#39;] = df[&#39;reviews.username&#39;].map(user_map)<br>df[&#39;prod_id&#39;] = df[&#39;id&#39;].map(item_map)<br>df[&#39;reviews.rating&#39;] = df[&#39;reviews.rating&#39;].astype(np.float32)<br><br></pre><h3>Popularity-Biased Negative Sampling</h3><blockquote><strong>Before i tell the next step i did , i want to add something, the step i am going to demonstrate next is something i added after my whole system was built and it failed by recomending only the most popular items in the dataset, so in this sub-section i will tell you how i dealt with popularity bias using weighted negative sampling</strong></blockquote><p>You must be wondering what went wrong , i think you can dissect it yourself after seeing this screenshot.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/706/1*vq0sUvo8nRpSJihj1__8Xg.png" /><figcaption>Notice how each product is the same, and has near same ratings.</figcaption></figure><p>Well to deal with this , i thought of an approach which got inspired from the idea of contrastive training when we train embeddings models from scratch , famous example is word2vec 2016.</p><p>First let us understand, why we need negative sampling at all , with a short story, suppose there is a small child and he needs to be taught what’s right with him , to make him remember what’s right for him , the child also needs to be implictly thought what’s wrong for him too , and that is the idea behind negative sampling for contrastive learning.</p><p>IN fact many modern embeddings models are trained on NLI datasets , which consists of a hypotheses , entailment and a rejection sample for this purpose only , i hope you get it.</p><p>See these screenshots and you will better understand the problem with the dataframe itself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/569/1*egiI40e3aB_2m46C1Me4kA.png" /><figcaption>Most products are 5 stars</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/552/1*pnVLKpZc3-O2CAgNpzvlwg.png" /><figcaption>Most users are power users</figcaption></figure><p>Even the describe() statistics of the user distribution tells the same thing</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/1*RWvEf3Y_dLOigvHAE2BHHw.png" /><figcaption>Statistical interpretation</figcaption></figure><ul><li>A user has on average given only 1.22 reviews, varying between -1 to 3 (Standard deviation)</li><li>even the 75th quantile or the third moment tells us that atleast 75 % of all users have only given 1 reviews.</li></ul><p>So all of this tells us that we have very weak signals for the user, but when i first tried building the model i didnt care about it and thought BERT will handle it but i was proved wrong , so getting to the crux of the matter now, here’s what i did in formal terms</p><blockquote>For each record or row in the dataframe, i generated synthetic samples by picking a product apart from the one the user purchased and giving it a weak rating indicating that the user didn’t like it or purchase it , hence giving the model a signal.</blockquote><p>Now i reasoned about how to do this task of selecting a datapoint from the data and these are the trade-offs i thought about.</p><ol><li><strong>A naive way</strong> would be to randomly take any point from the dataset and assigning it a rating of 0 or 1, while this approach could work it doesn’t fully cure the problem as a random sampling doesn’t guarentee that the popular products are chosen, and also assigning a hard rating of 0 or 1 can also cause posterior collapse.</li></ol><ul><li><strong>a slightly better way</strong> is to use labels sampled from a uniform distribution of U(0,2) and assign it hence regularizing the model.</li><li><strong>an even better way </strong>which i found was to normalize the scores and then use these normalized scores as the probability distribution for choosing the products, hence making popular products to be chosen more likely than less popular ones</li></ul><p>Formally, my approach could be stated as</p><blockquote>Instead of a flat random choice, you define the probability P of picking a product i as its frequency f_i raised to a power (usually 0.75), normalized across all items.</blockquote><blockquote>Why alpha = 0.75 This is the Word2Vec standard. It slightly “flattens” the distribution so that while popular items are still picked often, the very rare items still get a fighting chance to be seen as negatives.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/207/1*ggHPNa7b41LsLSkoFgh-5g.png" /><figcaption>Normalized probabiliy distribution for negative sampling</figcaption></figure><p>This can be implemented in code the following way</p><pre>&quot;&quot;&quot;  Negative Upsampling through normalized scores (alpha = 0.75) &quot;&quot;&quot;<br>alpha_weight = 0.75<br>weighted_sum = (df[&#39;prod_id&#39;].value_counts())**(alpha_weight)<br>normalized_prod_id = weighted_sum / weighted_sum.sum()<br>normalized_prod_id = normalized_prod_id.sort_index()<br><br>def generate_negative_prod_id_and_rating(normalized_prod_id):<br>    &quot;&quot;&quot;Generates one prod id using label smoothning and its negative rating&quot;&quot;&quot;<br>    negative_prod_id = np.random.choice(np.arange(0,600), p = normalized_prod_id) # samples a negative value based on popularity label smoothed<br>    negative_rating_of_prod_id = np.random.uniform(0,2)<br>    return (negative_prod_id,negative_rating_of_prod_id)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/1*t9rGO0yAhYnP4DtRDOZUog.png" /><figcaption>SO we smoothen the probabilties a bit</figcaption></figure><p>Same can be demonstrated by the CDF of the probability distribution which is not a straight line demonstrating my point.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/1*KkCxCTE8dBAE0_rck8Q-Qw.png" /><figcaption>CDF (Cumulative distributive frequency)</figcaption></figure><p>And below is the code for achieving the negative sampling we are talking about this whole section.</p><pre>&quot;&quot;&quot; Inserts k negative samples per record in the dataframe&quot;&quot;&quot;<br>rows = []<br>k = 3<br><br>for record in df.itertuples(index=False):<br>    curr_prod_id = record.prod_id<br><br>    for _ in range(k):<br>        while True:<br>            neg_prod_id, neg_rating = generate_negative_prod_id_and_rating(normalized_prod_id)<br>            if neg_prod_id != curr_prod_id: # Continue regenrating until we get a negative sample which is not the same as the positive one<br>                break<br><br>        prod_row = product_lookup.loc[neg_prod_id].to_dict()<br><br>        prod_row.update({<br>            &quot;user_id&quot;: record.user_id,<br>            &quot;reviews.rating&quot;: neg_rating,<br>            &quot;is_negative&quot;: True<br>        })<br><br>        rows.append(prod_row)<br><br>negative_samples_df = pd.DataFrame(rows)<br>negative_samples_df.to_csv(&quot;negative_samples.csv&quot;, index=False)</pre><ul><li>We decide the value of k = 3, which is the number of negative sampling we are doing , we chose 3 because it conforms to the golden ratio of 1 : 4 (TP rate).</li><li>Next we iterate on our pandas dataframe row by row using itertuples, and generate negative prod_ids and their ratings from the function we defined earlier.</li><li>Next we lookup all the information about that product from the product_lookup table and then update the dictionary with the information we want, and create a new dataframe which contains all of these negative samples, and save them in a seperate csv file.</li></ul><p>After this we concatenate our negative dataframe with our orignal dataframe to form a final dataframe which has total 213132 rows , which is a 3 fold increase from the previous 71000 rows, we achieved this using following code.</p><pre>df = df.copy()<br>df[&#39;is_negative&#39;] = False<br><br>final_df = pd.concat(<br>    [df, negative_samples_df],<br>    ignore_index=True<br>)<br>final_df = final_df.sample(frac=1).reset_index(drop=True) # shuffle the dataframe<br>final_df[&#39;prod_id&#39;] = final_df[&#39;id&#39;].map(item_map)<br><br>final_df.to_csv(&quot;final_df.csv&quot;, index=False)<br>df = final_df.copy()<br>print(df.shape)</pre><p>Output : (284176, 28)</p><h3>Feature Engineering and Signal Gathering</h3><p>The first signal we are going to use is a very popular but effective one which is known as matrix-factorization (also called as collaborative filtering).</p><p>FOr those who don’t know what collaborative filtering does is , it makes a 2-d Matrix consisting of (user-id),(product-id) as rows and columns, and value being interaction signals either implicit such as (CTR) or explicit such as ratings ( in our case ).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/673/0*39Z-OV9vZJJ1C2gQ.png" /><figcaption>Matrix factorization</figcaption></figure><p>Matrix factorization has a very simple intuition, how can we break a matrix of (m * n) dimensions such that m. n^t = v.</p><p>more formally,</p><p>We have a sparse matrix R (Users * Items). We want to find two smaller matrices, P and Q, such that their product approximates R.</p><ul><li><strong>P (shape = u * k) </strong>: The <strong>User Matrix</strong>, where each row is a k-dimensional vector representing a user’s latent preferences.</li><li><strong>Q (shape = i * k)</strong>: The <strong>Item Matrix</strong>, where each row is a k-dimensional vector representing an item’s latent features.</li><li><strong>k</strong>: The number of Latent Factors traits like price, brand loyalty, or review sentiment).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/159/1*bd652FWIdRad7MFBmXUP2g.png" /></figure><p>For our case we will be using a method for matrix factorization called ALS(alternating least squares) which works very well for sparse datasets, and converges well, the idea behind ALS is very simple but effective.</p><p>Freeze one , optimize for the other , then freeze the other and optimize for one , until iterations end.</p><p>Ok , but first we need to prepare our dataframe and make it of the form (users * products) = ratings.</p><pre>als_df = (<br>    df<br>    .groupby([&#39;user_id&#39;, &#39;prod_id&#39;], as_index=False)<br>    [&#39;reviews.rating&#39;]<br>    .mean()<br>)<br>als_df # Used for aggregating ratings of each user as mean , suppose a user 0 bought 5 items out of which 2 are repeat so we aggregate the rating to not confuse ALS<br>from scipy.sparse import coo_matrix<br>import numpy as np<br><br>rows = als_df[&#39;user_id&#39;].to_numpy()<br>cols = als_df[&#39;prod_id&#39;].to_numpy()<br>values = als_df[&#39;reviews.rating&#39;].to_numpy()<br><br>c_mat = coo_matrix((values, (rows, cols)))<br>c_mat = c_mat.tocsr()<br><br>print(c_mat.shape)# Output (58088, 600) sparse matrix dims<br>print(c_mat.nnz)# Output 268223 values filled in sparse<br></pre><p>We use a scipy sparse matrix as storing 58088 * 600 values in RAM is a lot, and since the matrix is sparse in nature scipy is the perfect choice, now we will train our ALS model , importing it from the implicit library.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/1*Qd4-GBZy6gsqV431S19bjw.png" /><figcaption>ALS training bar</figcaption></figure><p>Even so , when this model is trained it can generate predictions on what the user will like and presents a strong baseline to compare with other models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*wE_JhBdWKX0eIzJvYFO5RA.png" /><figcaption>Ignore the first prediction, as a product is similar to itself(ofc!).</figcaption></figure><blockquote>On an inside note, this solution till here is a valid recommendation system and can perform generally well , but it suffers from two problems first it does not rely on other features present in the dataset, and second it suffers from what we know as cold start problems in the recommendation systems (when a new user comes and we have no interaction data what should we present).</blockquote><h3>Deep ranking Models (Two Tower Architecture)</h3><p><strong>So from here, we move into the world of DEEP RANKING MODELS, more specifically Two Tower models, my version of two towers is hybrid in nature as i made my own tweaks to it.</strong></p><p>So what exactly is this two tower models, a two tower model, makes use of two towers , one for product data, and one for users, the two towers don’t meet until the very end where they concatenate.</p><p>I’ll just paste my architecture of the two Tower systems, and go on to explain each part first.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/788/1*2cRwDhMMKzqejaGeRjnpCw.png" /><figcaption>Deep ranking two tower architecture.</figcaption></figure><h3>User Tower</h3><p>First we generate BERT embeddings for the review title and review text, a BERT is a pre-trained encoder model which is capable of capturing user sentiment and semantic meaning of the text and output an embedding depicting the same.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qTLF5uobGH2IteC7Vgohqg.png" /><figcaption>User embeddings</figcaption></figure><p>We will pass review title and review concatenated through BERT.</p><pre>from tqdm import tqdm<br><br>def generate_embeddings_for_review_title(review_titles,review_texts,model = &#39;all-MiniLM-L6-v2&#39;):<br>    model = SentenceTransformer(model)<br>    concat_text = [f&#39;Review Title : {review_title} \n Review : {review_text}&#39; for review_title,review_text in tqdm(zip(review_titles,review_texts),total = len(review_titles))]<br>    return model.encode(concat_text)<br><br>all_review_embeddings = generate_embeddings_for_review_title(df[&#39;reviews.title&#39;],df[&#39;reviews.text&#39;])<br>df[&#39;review_embeddings&#39;] = list(all_review_embeddings) # stores all embeddings inside the dataframe</pre><p>Note that we will not use these embeddings directly for 1 main reason</p><p>The embeddings capture independent meanings , not temporal sentiment of the user across interactions, so we will average embeddings over interactions at each step, more formally, we will take a running mean of the independent user_review embeddings and store them in our dataframe.</p><pre>def running_mean_previous_embeddings(embeddings):<br>    result = []<br>    count = 0<br><br>    running_sum = np.zeros(384, dtype=np.float32)<br><br>    for emb in embeddings:<br>        if count == 0:<br>            result.append(np.zeros_like(emb))<br>        else:<br>            result.append(running_sum / count)<br><br>        running_sum += emb<br>        count += 1<br><br>    return result<br>res = df.groupby(by = &#39;reviews.username&#39;)[&#39;review_embeddings&#39;].transform(running_mean_previous_embeddings)<br>print(res)<br>df[&#39;average_mean_review_embeddings&#39;] = res</pre><p><strong>Output:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*R2l809NKg_s8A5bkTckiMA.png" /><figcaption>Running mean of BERT embeddings.</figcaption></figure><h4>Dealing with anonymous Users</h4><p>As we saw in the earlier part of the article, most of our dataset is dominated by one or two class of users called anonymous users, we cannot process them normally because these are not a single user but rather an aggregate of the masses, so we can deal with them in the following ways.</p><ul><li>Dropping them : this is the easiest choice to do , but doing this loses a lot of information about the products, and since anonymous users dominate the class frequency, we cant do this.</li><li>Zero-out the embeddings : We will keep anonymous users but zero out their embeddings, this approach is valid but doesnt capture the overall sentiment of the user.</li><li>Global — Average of the embeddings : we calculate the global average embedding of all the user embeddings in the dataframe(excluding anonymous users) , and assign this to anonymous users, in an essence, we will be treating anonymous user as an “Average Joe”.</li></ul><p>Also we zero the ALS vector of anonymous users , since they have no preference of their own.</p><pre>GLOBAL_BERT_EMBEDDING_AVERAGE = df[&#39;average_mean_review_embeddings&#39;].mean() # Average of all BERT reviews globally<br>df[&#39;average_mean_review_embeddings&#39;] = df.apply(lambda x: GLOBAL_BERT_EMBEDDING_AVERAGE if x[&#39;reviews.username&#39;] == &#39;Anonymous&#39; or x[&#39;reviews.username&#39;] == &#39;An anonymous customer&#39; or x[&#39;reviews.username&#39;] == &#39;ByAmazon Customer&#39; else x[&#39;average_mean_review_embeddings&#39;], axis=1)<br>df[&#39;is_anonymous_user&#39;] = df[&#39;reviews.username&#39;].apply(lambda x: 1 if x == &#39;Anonymous&#39; or x == &#39;An anonymous customer&#39; or x == &#39;ByAmazon Customer&#39; else 0) # We also create a binary feature indicating whether user is anonymous or not, to so the model learns to distinguis between the two.<br>df[&#39;user_ALS_embedding&#39;] = df[&#39;reviews.username&#39;].apply(lambda x: np.zeros(64) if x == &#39;Anonymous&#39; or x == &#39;An anonymous customer&#39; or x == &#39;ByAmazon Customer&#39; else user_als[user_map[x]]) # type: ignore<br>df[&#39;product_ALS_embedding&#39;] = df[&#39;prod_id&#39;].apply(lambda prod_id : product_als[prod_id]) # type: ignore<br>&quot;&quot;&quot; Creates ALS embeddings for users and products  with anonymous users having zero embeddings &quot;&quot;&quot;</pre><blockquote>Now, these 3 features will go inside our User Tower in our DRM, namely (BERT Embeddings for reviews, ALS user vector, is_anonymous binary feature)</blockquote><h3>Product Tower</h3><p>The product tower does something similar and calculates BERT embedding of the name of product, brand, categories the product belongs to and concatenates them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VmNBUh7ID943bNCz1C2wpA.png" /><figcaption>BERT embedding of product capturing semantics,</figcaption></figure><pre>def generate_embeddings_for_product_description(names,brands,categories,manufacturers,model = &#39;all-MiniLM-L6-v2&#39;):<br>    model = SentenceTransformer(model)<br>    concat_text = [f&#39;Name of product : {name} \n Brand of product : {brand} \n category of product : {category} \n manufacturer of product : {manufacturer}&#39; for name,brand,category,manufacturer in tqdm(zip(names,brands,categories,manufacturers),total = len(names))]<br>    return model.encode(concat_text)<br>embeddings_for_prod =  generate_embeddings_for_product_description(df[&#39;name&#39;],df[&#39;brand&#39;],df[&#39;categories&#39;],df[&#39;manufacturer&#39;])<br>df[&#39;product_embeddings&#39;] = list(embeddings_for_prod)</pre><p>So our product tower will take in 2 inputs namely (ALS product vector and BERT product vector) as inputs.</p><h3>Building and Training our Deep neural network</h3><p>we will be using keras/tensorflow for this implementation. and tf.data for input pipeline since we have very large vectors that we need to manipulate.</p><blockquote>Also for processing data efficiently we will be utilizing tf.data API ,The tf.data API is basically TensorFlow’s way of making data handling not the bottleneck in your training loop. Instead of manually loading data, preprocessing it, and worrying about memory or speed, we describe how data should flow into the model—shuffle it, batch it, preprocess it—and TensorFlow takes care of doing this efficiently.</blockquote><p>We begin by stacking the vectors on top of each other using numpy.stack() functionality.</p><pre>user_tower_als_matrix = np.stack(df[&#39;user_ALS_embedding&#39;].values)<br>user_tower_bert_matrix = np.stack(df[&#39;average_mean_review_embeddings&#39;].values)<br>product_tower_als_matrix = np.stack(df[&#39;product_ALS_embedding&#39;].values)<br>product_tower_bert_matrix = np.stack(df[&#39;product_embeddings&#39;].values)<br>is_anonymous_user_array = df[&#39;is_anonymous_user&#39;].values</pre><p>Then we define our labels as ratings, and initialize our tensorflow dataset, using tensors_from_slices, after intializing our dataset, we batch it in 32 values, shuffle it using buffer size = 1024, and then prefetch(what prefetch means is CPU always tries to stay one step ahead of the GPU by loading the next data point already.)</p><pre>labels = df[&#39;reviews.rating&#39;] # labels such as ratings (0 - 5 continous)<br>features = {<br>    &quot;user_ALS_Embedding&quot;: user_tower_als_matrix,<br>    &quot;user_BERT_Embedding&quot;: user_tower_bert_matrix,<br>    &quot;is_anonymous_user&quot;: is_anonymous_user_array,<br>    &quot;product_ALS_Embedding&quot;: product_tower_als_matrix,<br>    &quot;product_BERT_Embedding&quot;: product_tower_bert_matrix<br>}<br><br>full_dataset = tf.data.Dataset.from_tensor_slices((features, labels)) # Tensorflow dataset<br>train_dataset = (<br>    full_dataset <br>    .batch(32)<br>    .shuffle(buffer_size=1024)<br>    .prefetch(tf.data.AUTOTUNE)<br>)</pre><h4>Model Architecture definition</h4><p>We will define our model using Keras’s functional API, which allows for flexible models while still allowing model summary, and shape stats.</p><h4>Architecture for user tower</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8PGKswuVX5BCC3ikq1hykw.png" /></figure><ul><li>We will be starting with 3 inputs namely , 1. ALS vector of 64 dims , 2. BERT vector of 384 dims, 3. is_anonymous_feature of 1 dims.</li><li>Our next layer will concatenate all of these vectors into one vector of size (batch, 449)dims.</li><li>then we pass it through our first dense layer consisting of 256 neurons using ‘elu’ (exponential linear unit) activation along with he initalization of neurons, we progressively lower the ammount of neurons from 256-&gt;128 -&gt; 64</li><li>We introduce a dropout of 30 % between dense layers.</li></ul><pre>import keras<br><br>user_input_1 = keras.Input(shape=(64,), name=&#39;user_ALS_Embedding&#39;)<br>user_input_2 = keras.Input(shape=(384,), name=&#39;user_BERT_Embedding&#39;)<br>user_input_3 = keras.Input(shape=(1,), name=&#39;is_anonymous_user&#39;)<br>concatenated = keras.layers.Concatenate()([user_input_1, user_input_2, user_input_3]) #449 dims<br>user_dense_1 = keras.layers.Dense(256, activation=&#39;elu&#39;,kernel_initializer=&#39;he_normal&#39;)(concatenated)<br>user_dense_2 = keras.layers.Dense(128, activation=&#39;elu&#39;,kernel_initializer=&#39;he_normal&#39;)(user_dense_1)<br>user_dropout_output = keras.layers.Dropout(0.3,name = &#39;user_dropout&#39;)(user_dense_2)<br>user_tower_output = keras.layers.Dense(64, activation=&#39;elu&#39;, name=&#39;User_Tower_Output&#39;,kernel_initializer=&#39;he_normal&#39;)(user_dropout_output)</pre><h4>Product Tower</h4><p>We follow a similar architecture for product tower</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bmpCjlwloY2h8x1sXyPSAA.png" /><figcaption>Product tower</figcaption></figure><p>Here is the tensorflow code for the same</p><pre>product_input_1 = keras.Input(shape=(64,), name=&#39;product_ALS_Embedding&#39;)<br>product_input_2 = keras.Input(shape=(384,), name=&#39;product_BERT_Embedding&#39;)<br>product_concatenated = keras.layers.Concatenate()([product_input_1, product_input_2]) #448 dims<br>product_dense_1 = keras.layers.Dense(256, activation=&#39;elu&#39;,kernel_initializer=&#39;he_normal&#39;)(product_concatenated)<br>product_dense_2 = keras.layers.Dense(128, activation=&#39;elu&#39;,kernel_initializer=&#39;he_normal&#39;)(product_dense_1)<br>product_dropout_output = keras.layers.Dropout(0.3,name = &#39;product_dropout&#39;)(product_dense_2)<br>product_tower_output = keras.layers.Dense(64, activation=&#39;elu&#39;, name=&#39;Product_Tower_Output&#39;,kernel_initializer=&#39;he_normal&#39;)(product_dropout_output)</pre><h4>Final Ranking Model</h4><p>Now we will take the 64 -d (User Output) and 64-d(Product Output) , concatenate them and pass them through more dense layers to finally get 1 prediction which is the rating of the product</p><pre>&quot;&quot;&quot;Now  we will combine the outputs and then pass them through a final set of dense layers to get the final prediction&quot;&quot;&quot;<br><br>combined = keras.layers.Concatenate()([user_tower_output, product_tower_output]) # 128 dims<br>final_dense_1 = keras.layers.Dense(64, activation=&#39;elu&#39;,kernel_initializer=&#39;he_normal&#39;)(combined)<br>final_dropout_1 = keras.layers.Dropout(0.2)(final_dense_1)<br>final_dense_2 = keras.layers.Dense(32, activation=&#39;elu&#39;,name = &#39;final_dropout&#39;,kernel_initializer=&#39;he_normal&#39;)(final_dropout_1)<br>final_output = keras.layers.Dense(1, activation=&#39;linear&#39;, name=&#39;Final_Output&#39;)(final_dense_2)<br><br>model = keras.Model(inputs=[user_input_1, user_input_2,user_input_3, product_input_1, product_input_2], outputs=final_output)<br>model.compile(optimizer=&#39;adam&#39;, loss=&#39;mean_squared_error&#39;, metrics=[&#39;mae&#39;])</pre><ul><li>We will be using adam optimizer (standard)</li><li>MSE(mean squared error as the loss function).</li><li>And we will monitor the mean absolute error.</li></ul><h3>Final Visualization with shapes of our model</h3><p>After all is done our model will look something like this, along with shapes at each step of the way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Tvqsge8roBIsXx_9fXeP1Q.png" /><figcaption>Visualization of the whole model</figcaption></figure><p>Now we train our model for fixed 10 epochs, using model.fit()</p><pre>history = model.fit(train_dataset, epochs=15)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*67tEFD91R7LWLbEOlo1xkQ.png" /><figcaption>Training progress</figcaption></figure><p>Then we will save our models weights and architecture so it can be reused.</p><pre># Save weights specifically (safer than saving the whole model for subclassed models)<br>model.save_weights(&#39;my_two_tower_model_v5_power.weights.h5&#39;)<br>print(&quot;SAFE! Weights saved successfully.&quot;)<br>model.save(&#39;two_tower_recsys_v5_power.keras&#39;)</pre><h3>Inference of our model</h3><p>We will follow a method for ease of inference through our model, we will pre-compute all product tower forward passes for each product and store them in a dictionary , what this does is when new user comes in we dont have to concatenate their embeddings with each product and then compute scores, we can simply concatenate by a lookup.</p><pre>&quot;&quot;&quot; A dictionary mapping each product id to its corresponding forward pass through the product tower &quot;&quot;&quot;<br>product_tower_forward_pass_embedding_dict = {}<br>for i in df[&#39;prod_id&#39;].unique():<br>    df_temp = df[df[&#39;prod_id&#39;] == i]<br>    product_als_embedding = df_temp[&#39;product_ALS_embedding&#39;].values[0]<br>    product_bert_embedding = df_temp[&#39;product_embeddings&#39;].values[0]<br>    <br>    # Create input tensors<br>    product_als_input = tf.convert_to_tensor([product_als_embedding], dtype=tf.float32)<br>    product_bert_input = tf.convert_to_tensor([product_bert_embedding], dtype=tf.float32)<br>    <br>    # Get the output from the product tower<br>    product_tower_model = keras.Model(<br>        inputs=[product_input_1, product_input_2],<br>        outputs=product_tower_output<br>    )<br>    <br>    product_tower_output_embedding = product_tower_model.predict([product_als_input, product_bert_input])<br>    product_tower_forward_pass_embedding_dict[i] = product_tower_output_embedding[0]</pre><p>Now we will take a list of user_ids and product_ids and then recommend top k products for each user based on concatenating precomputed product tower embeddings and user tower embeddings and then passing through final dense layers to get the predicted rating and then recommending top k products with highest predicted ratings for each user</p><p>Also since ALS embeddings are same for each user BERT embeddings differ , so we will use the last BERT embedding of the user as user embedding and then concatenate with product tower embedding and then pass through final dense layers to get predicted rating for that product and then recommend top k products with highest predicted ratings for each user</p><pre>def predict_for_users_batched(user_ids, top_k=5, product_ids=None):<br>    if product_ids is None:<br>        product_ids = df[&#39;prod_id&#39;].unique()<br>    <br>    recommendations = {}<br>    <br>    model_to_use = keras.Model(<br>        inputs=[user_input_1, user_input_2, user_input_3, product_input_1, product_input_2],<br>        outputs=final_output<br>    )<br><br>    # Pre-bundle Product Embeddings<br>    p_als_matrix = np.array([als_product_embedding_dict[p] for p in product_ids])<br>    p_bert_matrix = np.array([bert_product_embedding_dict[p] for p in product_ids])<br>    num_products = len(product_ids)<br><br>    for user_id in user_ids:<br>        # Get the last review row for this user<br>        user_rows = df[df[&#39;user_id&#39;] == user_id]<br>        user_data = user_rows.iloc[-1]<br>        <br>        # Tile User Data<br>        u_als_batch = np.tile(user_data[&#39;user_ALS_embedding&#39;], (num_products, 1))<br>        u_bert_batch = np.tile(user_data[&#39;average_mean_review_embeddings&#39;], (num_products, 1))<br>        u_anon_batch = np.tile([user_data[&#39;is_anonymous_user&#39;]], (num_products, 1))<br><br>        # Single Batch Prediction<br>        predicted_ratings = model_to_use.predict(<br>            [u_als_batch, u_bert_batch, u_anon_batch, p_als_matrix, p_bert_matrix],<br>            batch_size=1024, # Higher batch size is usually faster for inference<br>        )<br><br>        scores = predicted_ratings.flatten()<br>        top_indices = np.argsort(scores)[-top_k:][::-1]<br>        <br>        # Map indices back to product IDs<br>        recommendations[user_id] = [(product_ids[i], scores[i]) for i in top_indices]<br><br>    return recommendations</pre><p>And finally, we predict for sample users what their recommendations are</p><pre>predict_for_users_batched([0,243,599], top_k=3)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/466/1*kUgfYWhTz60NuERp2qDZJg.png" /></figure><p>With this we can wrap up this , walkthrough cum tutorial of our recommendation system.</p><p>I hope you enjoyed reading this, and it took me a lot of effort juggling this project along with my internship and writing this article , i did not even use CHATGPT while writing this article or most of my code.</p><p>THe architecture i used was inspired by the two tower architectures, and since i am learning more and more about the fields , there is much to be improved in the project like all things in the world , i am happy to chat and connect for the same</p><p><em>I have committed all the relevant code and the dataset on my github go to the notebook link below to open the repo .</em></p><p><a href="https://github.com/akshatkhatri/Product-Recommendation-system-Two-tower-model">Project Repository Link</a></p><p><a href="https://github.com/akshatkhatri">Github Profile Link</a></p><p><em>Feel Free to connect with me on Linkedin</em></p><p><a href="https://www.linkedin.com/in/i-am-akshat-khatri/">Linkedin Profile Link</a></p><p><strong><em>Thanks for reading — means a lot :)</em></strong><em><br>Best,<br>Akshat Khatri</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0e114bae2322" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building GPT from Scratch in TensorFlow: A Deep Dive into Transformer Architecture]]></title>
            <link>https://medium.com/@akshat.dev/building-gpt-from-scratch-in-tensorflow-a-deep-dive-into-transformer-architecture-03fbbabe2693?source=rss-cb3d13125e1c------2</link>
            <guid isPermaLink="false">https://medium.com/p/03fbbabe2693</guid>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[gpt]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[tensorflow]]></category>
            <dc:creator><![CDATA[Akshat Khatri]]></dc:creator>
            <pubDate>Sun, 21 Sep 2025 19:32:25 GMT</pubDate>
            <atom:updated>2025-10-09T11:52:06.683Z</atom:updated>
            <content:encoded><![CDATA[<p>This article reflects my exploration into Language models and Transformers after studying the famous paper “<strong>Attention Is All You Need</strong>”. Using Jane Austen’s Book corpus from the Gutenberg Project.</p><h3>Live Demo &amp; Code</h3><p>The full codebase with detailed technical documentation and Functional API discussions is available on<strong> </strong><a href="https://github.com/akshatkhatri/GPT-from-Scratch-Tensorflow"><strong>GitHub here</strong></a><strong>.</strong></p><blockquote><em>The website is live at </em><a href="https://akshatkhatri.github.io/custom_gpt/"><em>Website Live Link</em></a><em>. (may be temporarily unavailable due to limited hosting resources). For a full walkthrough, see the </em><a href="https://youtu.be/l2RcEIe2LGE"><em>YouTube video demo </em></a><em>.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*fwACyb3kG_07Gekl.png" /><figcaption>Generative Transformers from scratch</figcaption></figure><h3>Motivation for building.</h3><p>I built this GPT from scratch because I wanted to truly understand transformers, not just know what they do, but how they work. Working hands-on with TensorFlow/Keras taught me how these frameworks handle complex architectures knowledge I need to eventually build my own novel models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9l05EKuyZAqY2u4aJWg0dw.jpeg" /><figcaption>Glimpse of final results.</figcaption></figure><h3>What we’ll Learn</h3><p>This blog post is an educational guide on how transformers (GPT-family) and language modelling work in general , so whether you know about how language models work or not , you will definitely learn something new after reading this article.</p><p>We will Learn —</p><ul><li>Internals of Transformer Architecture.</li><li>Concepts like masking, auto-regression in language models.</li><li>How data is processed and cleaned for Language models.</li><li>How to select appropriate model parameters using scaling laws</li><li>How to deploy models using Gradio interface.</li></ul><h4>What This Is:</h4><ul><li>A detailed examination of the GPT Transformer architecture, emphasizing the mechanisms behind sequence prediction and text generation.</li><li>A step-by-step guide to constructing and training a custom GPT language model from the ground up using TensorFlow and Keras.</li><li>A foundational resource aimed at enhancing understanding of the principles and internals of language models for researchers and practitioners.</li></ul><h4>What This Is Not:</h4><ul><li>Not a fully operational chatbot or conversational AI system capable of handling multi-turn dialogues or user interactions.</li><li>Not an API service like OpenAI’s GPT or a plug-and-play backend model ready to be called by applications.</li><li>Not a deployment-ready commercial API with built-in safeguards, moderation, or scalability for production use.</li></ul><h4>What are Language Models?</h4><p>In plain words, a language model is a special kind of ML model trained on large amounts of text data for generating the next best token, <strong>language models are models that are used to determine the probability of a given sequence of words occurring in a sentence.</strong></p><p>It can use what it learns from that text data to understand and generate human language, making it incredibly useful for tasks like translation, writing, and even coding.</p><h3>Deep-dive into the Transformer Architecture</h3><p>Transformers were developed by a <a href="https://arxiv.org/abs/1706.03762">group of researchers</a> back in 2017, in the paper named ‘Attention is All You Need’.</p><p>The original transformer paper follows an encoder-decoder stack specifically designed for seq2seq tasks like language translation , summarization e.t.c., The encoder processes the input sequence and creates meaningful contextual representations, while the decoder generates the output sequence based on both the encoder’s output and previously generated tokens.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*Om2hII7ZUbfs3DBV.png" /><figcaption>Transformer Architecture , credits : Dev shah</figcaption></figure><p>Let’s discuss each component in the architecture one by one</p><h4>Tokenizer</h4><p>Language models rely on tokens to model , tokenization is broadly divided into 3 schemes , char-level-tokenization , subword tokenization , and word level tokenization</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/921/1*1mo4YLobv-UxpHuee5VsWg.png" /><figcaption>Different tokenization schemes</figcaption></figure><h4>Input embeddings</h4><p>Embeddings are a way to represent tokens as an list or array of numbers , in most cases we deal with a fixed size of embedding vectors most common size is 128 .</p><p>In transformer architectures the input embeddings are initialized randomly and are learned during training via backpropogation .</p><p>Input embeddings are stored in a lookup table of size EMBEDDING_SIZE * VOCAB_SIZE , where VOCAB_size depends on our choice of tokenizer used and the size of the dataset.</p><p>For char level -&gt; vocab size ~ 30<br>for subword level -&gt; vocab size ~ 5k<br>for word level -&gt; vocab size ~ 2k</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/879/1*lF8Koll3xvipeqgz6MzbqA.png" /><figcaption>Embedding learned during training</figcaption></figure><h4>Positional Encoding</h4><p>Unlike RNN’s or CNN’s transformers have no inherent knowlegde of the token’s positions , so we need a way to inject knowledge about the position of each token into the embeddings we are learning.</p><p>Positional encoding is a technique used in transformer models to inject information about the relative or absolute position of tokens in the input sequence.<em> Since transformers process tokens in parallel without inherent sequence information, positional encoding allows the model to understand token order.</em></p><p>There are many positional encoding schemes , the choice of what scheme we want to use depend on our project’s complexity and use case —</p><ol><li>Sinusoidal Positional Encoding -&gt; adds a sin function of higher frequency to even position and a cosine function odd positions, the orignal transformer paper used sinusoidal positional embeddings and so will we.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/811/1*9C2lOUeYi4Twj4UAwsqFJg.png" /></figure><p>2. Learned positional encoding -&gt; Like embeddings we initialize positional embeddings randomly , allowing it to be learned during training , the problem with such an approach is that this may not generalize well.</p><p>3. Rotary positional Encoding (ROPE) -&gt; A newer and better alternative to traditional positional encoding methods</p><ul><li>Applies rotations to query and key vectors based on position.</li><li>Enables efficient extrapolation to longer sequences.</li><li>Used in recent models like GPT-NeoX.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OXegTGtNNAbU3Qaj.png" /><figcaption>Positional Encoding Credits : The code compass substack</figcaption></figure><h4>Multi-head Self Attention</h4><p>Attention Mechanism is often called the Heart of the transformer architecture</p><p>In the decoder stack ,the target sequence is fed to the Output Embedding and Position Encoding, which produces an encoded representation for each word in the target sequence that captures the meaning and position of each word. This is fed to all three parameters, Query, Key, and Value in the Self-Attention in the first Decoder which then also produces an encoded representation for each word in the target sequence, which now incorporates the attention scores for each word as well.</p><blockquote>In the Transformer, the Attention module repeats its computations multiple times in parallel. Each of these is called an Attention Head. The Attention module splits its Query, Key, and Value parameters N-ways and passes each split independently through a separate Head. All of these similar Attention calculations are then combined together to produce a final Attention score. This is called Multi-head attention and gives the Transformer greater power to encode multiple relationships and nuances for each word.</blockquote><p><em>Consider the sentence: “The animal didn’t cross the street because it was too tired.” When processing the word “it,” self-attention allows the model to associate “it” with “animal” rather than “street,” demonstrating its ability to resolve ambiguous references</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/915/1*8DQLScCOMv-X9iu_AMpjGg.png" /></figure><p><strong><em>Let’s understand the shapes we get along with an example.</em></strong></p><h4>Problem Setup :</h4><p>let’s suppose we have , Batch_size = 1, CONTEXT_LEN = 3 , D_MODEL = 4, Attention_HEADS = 2 ,</p><p>Suppose we get a sequence of text like =&gt; [[ “I love dogs” ]] , Suppose we are using word-level tokenization and we match these tokens to their token_ids and we get =&gt; [1,3,4] [1 = ‘I’ , 2 = ‘love’ , 3 = ‘dogs’] , after looking up the embeddings for each of the tokens we learnt we get their embeddings(learned + positional) as</p><h4>Embedding Lookup</h4><p>X [1,3,4] = <br>[<br> [1,0,1,0], # Embedding for token 1<br> [0,2,0,2], # Embedding for token 3<br> [1,1,1,1] # Embedding for token 4<br> ]</p><p>Shape =&gt; {BATCH,CONTEXT_LEN,EMBEDDING_SIZE}</p><h4>Learned Projection Matrices (Q, K, V)</h4><p>Next step is creating learned projection matrix Q,K,V also called as Query,key and value vectors of <br>shape =&gt; {EMBEDDING_DIM,EMBEDDING_DIM}<br>so W_Qproj = <br>[<br> [1,0,1,0]. # Random values learnt during training<br> [0,1,0,1],<br> [1,0,1,0],<br> [0,1,0,1]<br>]<br>Similarly W_Vproj and W_Kproj is learnt.</p><p><strong>Compute Q, K, V Vectors</strong></p><p>Next step, is projecting each of these learnt Q_proj,V_proj and K_proj matrices upon the embeddings we receive as inputs</p><blockquote>Q = X [dot] W_Q,<br>K = X [dot] W_K,<br>V = X [dot] W_V<br>[dot] =&gt; Dot product.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/189/1*GpWyoN9ynX5mhItWPBNULg.png" /><figcaption>Value of Q = X [dot] W_Q</figcaption></figure><p>Similarly we calculate K and V vector each of <br>shape =&gt; {CONTEXT_LEN , EMBEDDING_DIM}</p><p><strong>Split into Multi-Head Representations</strong></p><p><em>Next step , Depending on the number of heads split into [Q,K,V] vectors , in our case we took heads = 2 , Also NOTE that the number of heads should always be fully divided by the Embedding_dimension or else multi-head attention will not work properly , </em><strong>Head dimension:</strong> EMBEDDING_DIM // num_heads = 4 // 2 = 2</p><p>Head1(Q) = first 2 dims <br>[<br> [2,0],<br> [0,4],<br> [2,2]<br>]<br>Head2(Q) = last 2 dims<br>[<br> [2,0],<br> [0,4],<br> [2,2],<br>]<br>Similarly for K,V vectors.</p><p><strong>Compute Attention Scores</strong></p><p>Now the next step is to compute Attention score</p><blockquote>Head1 score =&gt; Q_1 * (K_1)^T <br>Head2_score = Q_2 * (K_2)^T <br>^T means transpose</blockquote><blockquote>Then we multiply those Head scores by the corresponding Value matrix to get attention scores</blockquote><p><strong>Apply Attention Weights</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*1GG58P9Fdka8ekwIK_VEew.png" /></figure><p><em>We scaled the Scores down by the square root of head_dimension to protect attention scores from exploding .We Then </em><strong><em>concatenate the outputs of all heads</em></strong><em> → final multi-head output.</em></p><p>The final step , is to pass the attention scores through a learned Linear Layer The concatenated output is passed through a <strong>final linear layer</strong> (learned weights) to mix the information from all heads and match the model’s embedding size.</p><p><strong>Advantages of Multi-Head Attention</strong></p><ul><li>Captures diverse relationships and features from different representation subspaces.</li><li>Improves model expressivity and ability to focus on different parts of input simultaneously.</li><li>Helps model capture long-range dependencies more effectively.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*21U6if9o0Y4sW-Vn" /><figcaption>Shapes during each pass</figcaption></figure><h4>MLP / Feed-Forward Neural Net Layer</h4><p>In a transformer architecture, the feedforward network is a crucial component responsible for processing and transforming the information obtained from the self-attention mechanism. Its primary role is to apply a nonlinear transformation to the representations learned through self-attention, introducing complexity and enabling the model to capture intricate patterns in the data.</p><p>The feedforward network typically consists of two linear transformations separated by a non-linear activation function. The input to the feedforward network is the output of the self-attention layer. Each position in the input sequence is treated independently during this operation.</p><p>Here’s how the feedforward network works:</p><ol><li>First Linear Transformation: The input sequence from the self-attention layer is passed through a linear layer (fully connected layer), projecting the input into a higher-dimensional space. This introduces non-linearity and allows the model to capture complex relationships within the data.<br>generally ,<em> the hidden_dim of this layer is generally 4 times the embedding_dimension</em></li><li>Activation Function: After the first linear transformation, a non-linear activation function, commonly a Gelu (Gaussian Error Linear Unit) , is applied element-wise to introduce non-linearity. This step enables the model to learn complex patterns and relationships in the data that may not be captured by linear transformations alone.</li><li>Second Linear Transformation: The output of the activation function is then passed through another linear layer, projecting it back into a lower-dimensional space. This transformation allows the model to compress and refine the information, focusing on the most relevant features.</li><li>Final Output: The output of the second linear transformation is the final representation of the sequence after passing through the feedforward network. This representation is then used as input for subsequent layers in the transformer architecture.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qHarF4PBUzYIu3-M.png" /><figcaption><em>The FFN comprises two linear (fully connected) layers that transform the input data. The first layer </em>expands the input dimension <em>from d</em>model<em>​=512 to a larger dimension </em>dff<em>​=2048, and the second layer projects it back to </em>dmodel<em>​.</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/897/1*2RPh1KV6tjRu2_3saSXl3A.png" /></figure><h4>Layer Normalization &amp; Residual Connection</h4><p>Layer Normalization is a normalization technique applied to the activations within a single layer of a neural network. Unlike batch normalization, which normalizes across the batch dimension, layer normalization normalizes across the feature (hidden) dimension for each individual sample.</p><p>Layer Normalization is particularly used for gradient stability and faster convergence , given an input embedding vector , Layer normalization first computes its mean and variance . <br>Then, much like standard scaler it applies</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/207/1*jUPVFFaM_DMsXwk9q9TdEA.png" /><figcaption>e is a small constant for numerical stability</figcaption></figure><p>This pushes its mean to 0 and standard deviation to 1 ,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/173/1*oHrV7DiblCp2D05QZiatSA.png" /><figcaption>both parameters decides how much to scale or shift the data</figcaption></figure><p>after that it learns two parameters where γ and β are learnable parameters that restore the network’s capacity to represent the identity transformation if needed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*W5tcQrYZw77sh4pS.png" /><figcaption>Layer Normalization</figcaption></figure><h4>Residual Connection</h4><p>Residual connections in transformers are skip connections that add the input of a sub-layer directly to its output before applying further processing like normalization or activation. This allows the network to preserve the original input signal and helps with gradient flow during training, addressing vanishing gradient issues in deep networks.</p><p>In a transformer, these connections are applied around both the self-attention and feed-forward sub-layers. Mathematically, if a sub-layer computes a function F(x) on an input <em>x</em>, the output with residual connection is:</p><p>Output = X + F(X).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hlgtKwogKO0jUsP1.png" /></figure><h4>Model Head / Final Linear Layer</h4><p>This is the more significant step that occurs at the very end of the Transformer model, after the data has passed through all the decoder layers. This final layer is responsible for generating the actual word predictions.</p><blockquote>The Model Head is a simple Linear layer whose only job is to lift up embeddings from size d_model to the size of Vocabulary . This resulting vector of raw, unnormalized scores is known as the logits vector.</blockquote><h4>Softmax Function</h4><p>The logits vector is then passed through a softmax function. This function converts the raw scores into a probability distribution, where each value is between 0 and 1, and the sum of all values is 1.</p><blockquote>Each probability corresponds to the likelihood of a specific word in the vocabulary being the next token in the sequence.</blockquote><p>For viewers interested in a more visual explanation , check <a href="https://poloclub.github.io/transformer-explainer/">this</a> out ,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/0*PxY9RYbTghnf1a9G.png" /><figcaption>Softmax Function , credits : MRI Question</figcaption></figure><blockquote>So till now , we have covered transformers theory in depth , what theory remains i will cover it as i explain the code , it’s time to now get started with how to actually build a transformer from scratch — Let’s start :)</blockquote><h3>Building From Scratch (Project Overview) :</h3><h4>Dependencies :</h4><ul><li>Tensorflow 2.x and Keras (Used for custom layers)</li><li>Python 3.8 +</li><li>Jupyter Notebook &amp; Gradio</li><li>libraries present inside <a href="https://github.com/akshatkhatri/GPT-from-Scratch-Tensorflow/blob/main/requirements.txt">requirements.txt</a></li></ul><h4>Dataset and preprocessing Approach :</h4><ul><li>We will be using <a href="https://www.kaggle.com/datasets/akshatkhatridata/jane-austen-books-corpus-cleaned">Jane austen book corpus dataset</a></li><li>The Book data is scraped from the gutenberg project</li><li>We will be using <a href="https://www.tensorflow.org/tutorials/load_data/tfrecord">TFRecord </a>for efficient processing of data , more on this later in the article</li><li>all characters are converted to lowercase , punctuation and special characters are removed.</li><li>We are choosing Jane austen novels due to abundant data availability and correct grammar and consistent writing patterns.</li></ul><h4>Model Architecture Decisions :</h4><ul><li>TOKENIZER_TYPE = char</li><li>VOCAB_SIZE ~ 38 (calculated automatically)</li><li>D_MODEL= 128</li><li>CONTEXT_LEN=256</li><li>ATTENTION_HEADS=8</li><li>DECODER_BLOCKS = 4</li><li>DROPOUT_RATE = 0.1</li><li>LEARNING_RATE = 0.0001</li><li>EPOCHS = 100</li><li>STEPS_PER_EPOCH = 500</li><li>WARMUP_RATIO = 0.1 ( We are using cosine decay scheduling more on that later).</li><li>PEAK_LEARNING_RATE = 3e-4</li><li>MIN_LEARNING_RATE = 1e-6</li><li>BATCH_SIZE = 8</li><li>RECORDS_PER_FILE = 100</li><li>Optimizer = AdamW</li><li>Size of dataset =&gt; 4.3 M tokens</li></ul><blockquote><strong>You can tweak any of them, but each adjustment carries consequences for model performance, training speed, and resource requirements. Finding the right balance is key to building an effective model.</strong></blockquote><h4>Implementation Deep Dive</h4><p>I will divide my implementation in 4 parts namely —</p><ol><li>Data handling and processing using <a href="https://www.tensorflow.org/tutorials/load_data/tfrecord">TFRecords</a>.</li><li>Custom Layers and core transformer concepts</li><li>Sampling and inference.</li><li>Deployment</li></ol><p>Let’s get started —</p><h4>1. Data Handling and preprocessing Using TFRecords</h4><p>The TFRecord format is a simple format for storing a sequence of binary records.</p><p><a href="https://developers.google.com/protocol-buffers/">Protocol buffers</a> are a cross-platform, cross-language library for efficient serialization of structured data. Protocol messages are defined by .proto files, these are often the easiest way to understand a message type.</p><p>The <a href="https://www.tensorflow.org/api_docs/python/tf/train/Example">tf.train.Example</a> message (or protobuf) is a flexible message type that represents a {&quot;string&quot;: value} mapping. It is designed for use with TensorFlow</p><h4>Step 1 : Read Text and Split into Tokens</h4><p>Splits the text file into character level tokens and assigns each token an token_id and vice_versa . Also do note that below code snippet is for character level tokenization , in my github repo i have added support for subword and word level tokenization too.</p><pre>@keras.saving.register_keras_serializable()<br>def tokenize_and_build_vocabulary_tf(file_path_list: List[str], existing_vocab: Dict[str, int] | None = None) -&gt; Dict[str, int]:<br>    sorted_tokens = sorted(vocab_set)<br>    token_to_id = {char: idx for idx, char in enumerate(sorted_tokens)}<br>    return token_to_id<br><br>@keras.saving.register_keras_serializable()<br>def build_id_to_token(token_to_id: dict[str, int]) -&gt; dict[int, str]:<br>    id_to_token = {idx: token for token, idx in token_to_id.items()}<br>    return id_to_token</pre><p>Example output:<br>{‘a’: 0, ‘b’: 1, ‘c’: 2, …, ‘ ‘: 26, ‘!’: 27, …}</p><h4>Step 2 : Creating Training Examples with Sliding Windows</h4><p>Instead of treating each sentence separately, GPT learns from continuous text by using a sliding window approach:</p><p>Text: “The quick brown fox jumps over the lazy dog” Context Length: 4</p><p>Window 1: [The, quick, brown, fox] → Target=[quick, brown, fox, jumps] <br>Window 2: [quick, brown, fox, jumps] → Target=[brown, fox, jumps, over] <br>Window 3: [brown, fox, jumps, over] → Target=[fox, jumps, over, the]</p><p>This ensures the model learns from overlapping contexts, improving its understanding of language flow.</p><h4>Step 3: Efficient Storage with TFRecord Format (Creating Example protobufs)</h4><ul><li><strong>Memory Efficiency</strong>: Processes large datasets without loading everything into RAM</li><li><strong>Fast I/O</strong>: Binary format optimized for TensorFlow</li><li><strong>Reproducibility</strong>: Same data order across training runs</li><li><strong>Distributed Training</strong>: Easy to shard across multiple workers</li></ul><pre>def create_tf_example(input_ids: List[int], target_ids: List[int], attention_mask: List[int]) -&gt; bytes:<br>    &quot;&quot;&quot;Convert training example to TFRecord format for efficient storage&quot;&quot;&quot;<br>    feature = {<br>        &#39;input_ids&#39;: tf.train.Feature(int64_list=tf.train.Int64List(value=input_ids)),<br>        &#39;target_ids&#39;: tf.train.Feature(int64_list=tf.train.Int64List(value=target_ids)),<br>        &#39;attention_mask&#39;: tf.train.Feature(int64_list=tf.train.Int64List(value=attention_mask))<br>    }<br>    <br>    example = tf.train.Example(features=tf.train.Features(feature=feature))<br>    return example.SerializeToString()</pre><h4>Step 4 : The Core Conversion Function</h4><pre>def convert_text_to_tfrecord(text_file_path, token_to_id_dict, tokenize_func, <br>                           output_dir, context_length=512):<br>    # Read and tokenize entire text<br>    with open(text_file_path, &#39;r&#39;) as f:<br>        text = f.read()<br>    <br>    token_ids = tokenize_func(text)<br>    <br>    # Create sliding windows<br>    for i in range(0, len(token_ids) - context_length, context_length):<br>        window = token_ids[i:i + context_length + 1]<br>        <br>        input_ids = window[:-1]    # [0, 1, 2, ..., n-1]<br>        target_ids = window[1:]    # [1, 2, 3, ..., n]<br>        <br>        # Save to TFRecord<br>        tf_example = create_tf_example(input_ids, target_ids, attention_mask)<br>        writer.write(tf_example)</pre><h4>Step 5: Train / Validation Split Stratergy</h4><ul><li>File-Level Splitting for Large Datasets</li><li>Instead of shuffling all examples (memory intensive), we split at the file level:</li><li>- First 10% of TFRecord files → Validation set<br>- Remaining 90% → Training set</li><li>This maintains efficiency while ensuring no data leakage between sets.</li></ul><h4>Step 6: Memory and Performance Optimizations</h4><pre># 1. Streaming processing - never load full dataset<br>dataset = dataset.repeat()  # Infinite iteration<br><br># 2. Parallel processing<br>num_parallel_calls=tf.data.AUTOTUNE<br><br># 3. Prefetching - prepare next batch while training<br>dataset = dataset.prefetch(tf.data.AUTOTUNE)<br><br># 4. Chunked file writing<br>records_per_file: int = 1000  # Manageable file sizes</pre><h4>Why choose TFRecords over classic csv or .txt files ?</h4><ul><li>Processing Speed: ~4M tokens/minute on modern hardware.</li><li>Training Throughput**: Optimized for GPU utilization.</li><li>Memory Usage : Constant regardless of dataset size.</li><li>Multiple TFRecord files enable distributed training.</li><li>Adjustable batch sizes for different GPU memory.</li><li>Context length experimentation without reprocessing.</li></ul><h3>2. Custom Layers and core transformer concepts</h3><h4>Sinusoidal Embeddings : Adds a unique fingerprint positionally to each token in the sequence.</h4><pre>@tf.function<br>@keras.saving.register_keras_serializable()<br>def prepare_sinusoidal_lookup_table(EMBEDDING_SIZE: int = 128, max_seq_len: int = 512):<br>    # Initialize the table<br>    lookup_table = np.zeros((max_seq_len, EMBEDDING_SIZE), dtype=np.float32)<br>    <br>    # Compute the angle rates for each dimension<br>    # angle_rates[k] = 1 / (10000^(2*(k//2) / EMBEDDING_SIZE))<br>    dims = np.arange(EMBEDDING_SIZE)[np.newaxis, :]   # shape (1, EMBEDDING_SIZE)<br>    positions = np.arange(max_seq_len)[:, np.newaxis] # shape (max_seq_len, 1)<br>    angle_rates = 1 / np.power(10000, (2 * (dims // 2)) / EMBEDDING_SIZE)<br>    <br>    # Compute the angle for each position and dimension: position * angle_rate<br>    angle_rads = positions * angle_rates  # shape (max_seq_len, EMBEDDING_SIZE)<br>    <br>    # Apply sin to even indices (0,2,4,...) and cos to odd indices (1,3,5,...)<br>    lookup_table[:, 0::2] = np.sin(angle_rads[:, 0::2])<br>    lookup_table[:, 1::2] = np.cos(angle_rads[:, 1::2])<br>    <br>    return tf.constant(lookup_table)</pre><blockquote><strong>I feel giving code + explanation for all custom layers that i implemented myself in this article here , would be too verbose , hence i have attached the link to the github file containing a link to all the implementation of all the custom layers in tensorflow ,</strong><a href="https://github.com/akshatkhatri/GPT-from-Scratch-Tensorflow/blob/main/models/layers.py"><strong> feel free to check it out here</strong></a><strong>.</strong></blockquote><h4>Cosine Decay with Warmup Learning Rate Schedule:</h4><p>The Cosine Decay with Warmup schedule is a two-phase learning rate strategy designed to stabilize training in the early iterations and then gradually reduce the learning rate using a cosine curve. It is commonly used in training large Transformer models and other deep networks.</p><ol><li>Warmup Phase<br>For the first warmup_steps training steps, the learning rate increases linearly from 0 up to the specified peak learning rate (peak_learning_rate).</li><li>Cosine Decay Phase<br>After the warmup, over the remaining steps (total_steps – warmup_steps), the learning rate follows a cosine decay curve from peak_learning_rate down to a minimum value (min_learning_rate).</li></ol><pre>@keras.saving.register_keras_serializable()<br>class CosineDecayWithWarmup(keras.optimizers.schedules.LearningRateSchedule):<br>    def __init__(self, <br>                 warmup_steps: int,<br>                 total_steps: int,<br>                 peak_learning_rate: float = 1e-4,<br>                 min_learning_rate: float = 1e-6,<br>                 name: str = &quot;cosine_decay_with_warmup&quot;):<br>        super().__init__()<br>        self.warmup_steps = warmup_steps<br>        self.total_steps = total_steps<br>        self.peak_learning_rate = peak_learning_rate<br>        self.min_learning_rate = min_learning_rate<br>        self.name = name<br>        <br>    def __call__(self, step):<br>        step = tf.cast(step, tf.float32)<br>        warmup_steps = tf.cast(self.warmup_steps, tf.float32)<br>        total_steps = tf.cast(self.total_steps, tf.float32)<br>        <br>        warmup_lr = self.peak_learning_rate * tf.math.divide(step, warmup_steps)<br>        <br>        decay_steps = tf.math.subtract(total_steps, warmup_steps)<br>        progress = tf.math.divide(tf.math.subtract(step, warmup_steps), decay_steps)<br>        cosine_decay_lr = self.min_learning_rate + 0.5 * (<br>            self.peak_learning_rate - self.min_learning_rate<br>        ) * (1 + tf.cos(tf.constant(np.pi, dtype=tf.float32) * progress))<br>        <br>        return tf.where(tf.less(step, warmup_steps), warmup_lr, cosine_decay_lr)</pre><h4>3. Sampling and inference</h4><p>Once a language model has been trained, the process of using it to generate new text is called inference. During inference, the model doesn’t just output a complete block of text at once. Instead, it builds the text sequentially, one token at a time. The strategies used to decide which token to choose at each step are known as sampling techniques. These methods allow us to control the trade-off between coherence and creativity in the generated output.</p><p><strong>The Auto-Regressive Framework</strong></p><p>At its core, a generative language model like GPT operates on an auto-regressive basis. This simply means that the prediction for each new token is dependent on the sequence of tokens that came before it.</p><p>Think of it like building a sentence word by word:</p><ol><li>You start with an initial prompt (e.g., “The weather today is”).</li><li>The model takes this input and calculates a probability distribution over its entire vocabulary for the very next word.</li><li>A word is chosen from this distribution using a sampling technique (e.g., “sunny”).</li><li>This new word is appended to the sequence, forming a new input (“The weather today is sunny”).</li><li>The process repeats: the model takes the updated sequence and predicts the next word, and so on, until a stop condition is met.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/453/1*rHSBRATfIWRk2UlroAzYHQ.png" /><figcaption>Credits : pierre marie</figcaption></figure><p><strong>Controlling Creativity with Temperature</strong></p><p>Temperature is a parameter that controls the level of randomness in the model’s predictions. It works by rescaling the raw output scores (logits) from the model before they are converted into probabilities via the softmax function.</p><ul><li>Low Temperature (e.g., 0.2): A low temperature makes the probability distribution “peakier,” exaggerating the difference between the most likely word and all others. The model becomes more confident and deterministic, almost always choosing the highest-probability word. This leads to more focused, coherent, but potentially repetitive and less creative text. It is ideal for factual tasks like summarization or question-answering.</li><li>High Temperature (e.g., 1.0 or higher): A high temperature flattens the probability distribution, making less likely words more probable. This increases randomness and encourages the model to take more creative risks, leading to more diverse and surprising outputs. However, if the temperature is too high, the output can become nonsensical and incoherent. This is useful for creative writing or brainstorming.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*ybMgktDhoG4S1huiKIkaMg.png" /></figure><pre>logits = model_predictions  # Raw model outputs<br>scaled_logits = logits / temperature  # Apply temperature scaling</pre><p><strong>Top-K Sampling</strong></p><p>While temperature adjusts randomness, it doesn’t prevent the model from considering truly bizarre or irrelevant words, especially at higher temperatures. Top-K sampling acts as a quality filter by restricting the model’s choices to a fixed number (k) of the most likely next words.</p><p>For example, if k=50, the model is forced to choose its next word from only the 50 most likely candidates, effectively cutting off the long tail of low-probability words that could derail the generation.</p><pre># Conceptual implementation in your generation:<br>def sample_with_top_k(logits, k=20):<br>    # 1. Get top-k most probable tokens<br>    top_k_logits, top_k_indices = tf.nn.top_k(logits, k=k)<br>    <br>    # 2. Zero out all other probabilities<br>    filtered_logits = tf.fill(logits.shape, float(&#39;-inf&#39;))<br>    filtered_logits = tf.tensor_scatter_nd_update(<br>        filtered_logits, top_k_indices, top_k_logits<br>    )<br>    # 3. Sample from filtered distribution<br>    probabilities = tf.nn.softmax(filtered_logits / temperature)<br>    next_token = tf.random.categorical(probabilities, 1)<br>    return next_token</pre><h4>Deployment</h4><p>After training my GPT model from scratch, deploying it enables easy interaction and sharing. This project uses Gradio, a lightweight Python library that provides an intuitive web interface for AI models with minimal code. The interface allows users to input prompts, adjust generation parameters (like text length, temperature, and top-k sampling), and receive real-time generated outputs styled in Jane Austen’s literary tone.</p><p>Deployment steps include:</p><ul><li>Loading the trained model weights and tokenizer.</li><li>Building an inference function to tokenize input, run model predictions autoregressively, and decode outputs.</li><li>Creating a Gradio app with text input, sliders for creativity controls, and display areas for generated text and analysis.</li></ul><h3>Training Metrics and Optimization Analysis</h3><p>The displayed figures provide a detailed quantitative overview of the model’s training dynamics over 100 epochs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gDEHafJp2OG74PckCOFzdg.jpeg" /><figcaption>Training metrics</figcaption></figure><ul><li>Loss Curves (Top-Left):<br>The training and validation loss curves show a monotonic decrease from an initial value above 3.0 to a final region near 1.2. The convergence pattern indicates successful optimization with the AdamW optimizer and the chosen learning rate schedule. The minimal divergence between training and validation loss suggests a well-regularized model with negligible overfitting. The smoothness of the curves confirms stable gradient updates during backpropagation.</li><li>Accuracy Curves (Top-Right):<br>The accuracy for both training and validation datasets increases rapidly in the early epochs, plateauing around 61–62%. This plateau reflects the model’s capacity to capture meaningful sequential token dependencies in the character-level language modeling task. The slight but consistent gap between training and validation accuracy is typical, reflecting natural variance in generalization.</li><li>Learning Rate Schedule (Bottom-Left):<br>This plot illustrates the cosine decay learning rate schedule enhanced with a warmup phase. The learning rate starts near zero, ramps linearly during the warmup to the peak learning rate (~3e-4), and then progressively decays following a cosine function toward a minimum bound (~1e-6). This schedule mitigates early training instability and encourages fine-tuning in later epochs, facilitating convergence to a minima with better generalization.</li><li>Final Metrics Bar Chart (Bottom-Right):<br>Quantitative assessments of final model performance present a training loss of approximately 1.19 and validation loss of 1.28, alongside training and validation accuracies of ~61.9% and ~60.4%, respectively. The small margin between these values empirically confirms the model’s ability to generalize to unseen data without overfitting, likely enabled by proper regularization techniques like dropout and careful hyperparameter tuning.</li></ul><h3>Conclusion</h3><blockquote>Building a GPT model from scratch is a challenging but immensely rewarding journey that i am proud to take on and complete to the best of my abilities , this project reflects my efforts and consistency over the last few months , and gives me the confidence and foundations to try out more novel and sophisticated architectures in the future . Most importantly, this journey reinforced that there’s no substitute for building something yourself. While pre-trained models are powerful tools, implementing from scratch forces you to truly understand the underlying principles.</blockquote><blockquote>For anyone considering a similar project — the deep understanding you’ll gain is worth every debugging session and training hiccup along the way.</blockquote><p><em>For those curious about implementation specifics and hands-on experimentation, the full codebase is available on</em><strong> </strong><a href="https://github.com/akshatkhatri/GPT-from-Scratch-Tensorflow"><strong>GitHub here</strong></a><strong> </strong><em>with much more detailed technical documentation and Functional API discussions.</em></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fl2RcEIe2LGE%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dl2RcEIe2LGE&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fl2RcEIe2LGE%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/1bb9c4a799d1c1e4c7376a8c888f65c0/href">https://medium.com/media/1bb9c4a799d1c1e4c7376a8c888f65c0/href</a></iframe><h4><strong>All Useful Links &amp; Connect</strong></h4><p><a href="https://akshatkhatri.github.io/custom_gpt/">Website Live Link</a></p><p><a href="https://github.com/akshatkhatri/GPT-from-Scratch-Tensorflow">Project Github Repository Link</a></p><p><a href="https://youtu.be/l2RcEIe2LGE">Youtube Video Link</a></p><p><em>Feel Free to connect with me on Linkedin</em></p><p><a href="https://www.linkedin.com/in/i-am-akshat-khatri/">Linkedin Profile Link</a></p><p><a href="https://github.com/akshatkhatri">Github Profile Link</a></p><p><strong><em>Thanks for reading — means a lot :)</em></strong><em><br>Best,<br>Akshat Khatri</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=03fbbabe2693" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I Built a Neural Network from Scratch using only Python and Numpy: A Step-by-Step Journey]]></title>
            <link>https://medium.com/@akshat.dev/how-i-built-a-neural-network-from-scratch-using-only-python-and-numpy-a-step-by-step-journey-3345d67c6d2b?source=rss-cb3d13125e1c------2</link>
            <guid isPermaLink="false">https://medium.com/p/3345d67c6d2b</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[numpy]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Akshat Khatri]]></dc:creator>
            <pubDate>Sun, 03 Aug 2025 19:35:40 GMT</pubDate>
            <atom:updated>2025-08-04T05:05:00.321Z</atom:updated>
            <content:encoded><![CDATA[<p>Over the past week, I started reading about deep learning from Aurélien Géron’s book <em>Hands-On Machine Learning</em>. Everywhere I looked deep learning was at the core, and feedforward neural networks (the backbone of models like transformers) always seemed to appear.</p><p>This made me realize that before moving on to powerful libraries like Keras and TensorFlow, I needed to build a strong foundational intuition about neural networks.</p><blockquote>To truly grasp what happens “under the hood,” I set out to implement a neural network specifically, a Multilayer Perceptron (MLP) from scratch, using nothing but Python and NumPy.</blockquote><p>My goal while making this was to understand deeply how they work, both in theory and in code. In this article, I’ll first break down some relevant theory (I’ll keep it short), followed by code snippets and the supporting math.</p><h4>A Glimpse of what we are about to build in this article :</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fy7fvfdrgQ40%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dy7fvfdrgQ40&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fy7fvfdrgQ40%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/3a6ea7904de8d3fb6c3da0d57c779008/href">https://medium.com/media/3a6ea7904de8d3fb6c3da0d57c779008/href</a></iframe><h3>What is a Neural network in Simple terms</h3><p>A Neural network is inspired by the brain’s mechanism of processing information , each node in a network is connected to all other nodes in the next layer , the contributions of these nodels is dictated by their respective weights and biases.</p><p>Neurons and nodes can be used interchangably in our context , neural networks are also called Artificial Neural networks or ANNs.</p><h3>The perceptron</h3><p>Before understanding neural network , let’s understand how a simple neural network ( the perceptron ) behaves.</p><p>A perceptron consists of input neurons , with their respective weights and biases (initialized randomly), it makes use of TLU (threshold logical unit).<br>The TLU is used to compute a linear combination of weights and biases of each input neuron , then it applies a step function to check if the class is 1 or 0 , much like the case of logistic regression.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/233/1*1gt2gY8s6RadytDvlw-MyA.png" /><figcaption>The perceptron ( the green node is the TLU)</figcaption></figure><p>It commonly used a heaviside as a step function , which outputs 0 if z &lt;0 , and 1 if z &gt; 0.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lHsRLo8xuL_7820CvtaRlg.png" /><figcaption>Heaviside function</figcaption></figure><p>The bigger question , is after initializing weights randomly , we have to train the perceptron network , by treaining i mean readjustings its weights and biases to reduce to overall loss , this is typically done using a process similar to gradient descent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/339/1*dXtHFl9SPLPKofXfGtJTpA.png" /><figcaption>Notice the ’n’ here is the learning rate .</figcaption></figure><h3>Multi-layer-perceptron (MLP)</h3><p>AN MLP(feed-forward neural network) is composed of one or more layers of Perceptrons called hidden layers , with the final layer called as output layer . The layers close to input are lower layers , and the ones close to output are called upper layers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/905/1*SoM_9WP6mUNFM0v9fxQtxQ.png" /></figure><blockquote><a href="https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi">https://www.youtube.com/playlistlist=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi</a> , This Playlist by 3 blue 1 brown , covers up all the deep learning theory in a visually intuitive way , i would highly recomend to check it out.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/395/1*NjrQwhTsA-Qhc3f-Ns9M5g.png" /><figcaption>Weights matrix (W) . input (x) and bias (b)</figcaption></figure><h4>Important terms</h4><ol><li>Neurons or nodes -&gt; At a higher level these can be visualized as simple numbers formed by a combination of weights , biases and an activation function.</li><li>Weights -&gt; Dictate how strongly a node impacts other neurons , value can range from anything negative or positive , initialized purely randomly.</li><li>Biases -&gt; Used to add some variability in the neurons outputs , you can understand it via linear regression , where if no bias is introduced the slope always starts from origin , bias is typically initialized to 0 or 0.1.</li><li>Activation Functions -&gt; Used to add non-linearity in the layer’s outputs , without the activation function , you can linearly combine 3 layers but it is just equivalent to 1 layer with a different combination , Popular activation functions are Relu , Leaky Relu , Tanh , sigmoid .</li><li>Forward Pass -&gt; One pass or walkthrough from the input layer to the output layer is called forward pass , usually followed by backpropogation in training . Used for predictions when testing.</li><li>Backpropogation -&gt; Used to Update weights and biases by starting from the output layer and going backwards to the input layer , (hence the name ‘back’).</li><li>Gradient Descent -&gt; Used to minimize the cost function by moving in the direction opposite to the steepest increase i.e. partial derivative , works in hand with the learning rate , which determines how much effect should the gradient have on the current weights or bias.</li><li>Loss function -&gt; Used to calculate how bad / off our predictions were , serves as a metric for backpropogation to penalize neurons , popular loss functions are cross-entropy , Binary cross-entropy for classification , MSE or MAE or huber-loss for Regression tasks.</li><li>Input Layer -&gt; Usually equivalent to the number of features in your dataset , such as x1,x2 …, xn .</li><li>Output Layer -&gt; Usually carries a different activation like sigmoid or softmax compared to hidden layers , equivalent to the number of classes to predict.</li><li>Hidden layers -&gt; All Layers between input layer and output layer are called hidden layers</li></ol><p>This is all the neccesary theory , we need to start coding now , i’ll explain each concept with the code seperately too.</p><h3>MNIST and Fasion-MNIST dataset</h3><h4>MNIST</h4><p>We will be working on these two datasets , Don’t worry much about these datasets , the core neural network architecture i will discuss will remain the same , but it’s good to have a dataset to test our model on .</p><ul><li>Classic dataset of <strong>handwritten digits</strong> (0–9)</li><li><strong>60,000 training</strong> images + <strong>10,000 test</strong> images</li><li>Each image is:</li><li>Size: <strong>28x28 pixels</strong></li><li>Grayscale (1 channel)</li><li>Label: digit between 0–9</li><li>Input: 28×28 pixel grayscale image (flattened to 784-dimensional vector)</li><li>Output: One of 10 classes (digits 0 to 9)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/557/0*d2UDIGXilWl154J_.png" /><figcaption>MNIST Dataset</figcaption></figure><h4>Fashion-MNIST</h4><ul><li>A <strong>more challenging</strong> alternative to MNIST</li><li><strong>60,000 training</strong> + <strong>10,000 test</strong> images</li><li>Each image: <strong>28x28 grayscale</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/1*LiuihYA6GtCDLnMe9joT_A.png" /><figcaption>Fasion MNIST dataset</figcaption></figure><p>You can load both of these datasets using keras , altough in my notebook i have used a different method to load the data and the variable names are also different , the concept remains the same</p><pre>from tensorflow.keras.datasets import mnist, fashion_mnist<br><br># Load MNIST<br>(x_train, y_train), (x_test, y_test) = mnist.load_data()<br><br># Load Fashion MNIST<br>(x_train_f, y_train_f), (x_test_f, y_test_f) = fashion_mnist.load_data()</pre><h3>Deciding Architecture for neural network</h3><p>When designing a neural network’s architecture, you are setting its <em>capacity</em>, <em>trainability</em>, <em>efficiency</em>, and ultimately its ability to <em>generalize</em> to new data.</p><p>We need to make several architectural decision before we begin training , namely -</p><ul><li>Depth(Number of Layers) : Increasing the number of layers allow our models to capture more complexities . But these are prone to vanishing and exploding gradients.</li><li>Width (Neurons Per Layer) : Considered slightly better than increasing depth , but increasing width too much can lead to poor generalization and over-fitting.</li><li>Activation Functions : Generally Relu (return max(0,x)) works best and the default in most cases , but if the network is suspectible to vanishing gradients (Very common with Relu) , it can lead to a problem called as the Dying neurons problem , where a neurons weight is always zero , so it’s just dead . To solve this we use leakyRelu , where instead of setting negative values to 0 , we set them to small value like -0.1 , to prevent neurons from dying , Other popular examples are tanh and sigmoid.</li><li>Initialization Scheme : Sets starting weights. Poor initialization (e.g., all zeros or too large) stalls or destabilizes training. He/Xavier initializations help maintain activation variance across layers, speeding convergence.</li><li>Output Layer &amp; Loss Function : Must match the task: e.g., softmax + cross-entropy for multiclass classification, sigmoid for binary, linear for regression. Mismatched choices lead to poor convergence or nonsensical outputs.</li></ul><p>In our case we will be going with the architecture below. Each image in the MNIST is a 28 * 28 pixel image , i flattened it into a 784 pixel array</p><pre>input_dimension = 784<br>input_layer_dimension = input_dimension<br>output_layer_dimension = 10<br>num_classes = 10<br>layers = 3 # includes hidden + output layer<br>layer_dimension = (128, 64, output_layer_dimension)  # How many neurons in each layer excluding input layer which has 784 neurons<br>activ_1 = &#39;Relu&#39;<br>activ_2 = &#39;Leaky Relu&#39;<br>initialization_scheme = &#39;He&#39; # Xavier and Normal Values b/w 0 and 1 can also be used<br>Output_layer_activ = &#39;softmax&#39;<br>loss_func = &#39;cross_entropy&#39;<br><br>assert layers &gt;= 1<br>assert layers == len(layer_dimension)</pre><h4>Initializing Weights and Biases Randomly</h4><p>Let’s look at the code for initializing weights and biases randomly , note that in our case we will use ‘He’ initialization , which is — <br>weight = np.random.randn(in_dim, out_dim) * np.sqrt(2.0 / in_dim)</p><pre>def initialize_weights_and_biases_randomly(input_dimension: int, layer_dimension: tuple):<br>    weights_matrix = [] <br>    bias_matrix = []<br><br>    in_dim = input_dimension<br>    for out_dim in layer_dimension:<br>        weight = np.random.randn(in_dim, out_dim) * np.sqrt(2.0 / in_dim)  # He initialization<br>        bias = np.full(shape = (1, out_dim),fill_value=0.1,dtype=np.float64)  # Standard: initialize biases to zeros<br>        # print(weight.shape)<br>        weights_matrix.append(weight)<br>        bias_matrix.append(bias)<br>        in_dim = out_dim  # Update for next layer<br><br>    return weights_matrix, bias_matrix</pre><p>As a result we have weights and biases for each layer in our architecture, initialized Randomly , You can initialize bias to 0 too , but i made it 0.1 for early learning .</p><h4>Activation Function , Loss function &amp; Their derivatives</h4><p>We will be using Relu or Leaky Relu for our architecture ,</p><ul><li>Relu (Rectified Linear Unit) is just simply 0 if -ve and same number if positive.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/686/1*ENDQTjq1t7f1weQd628lpQ.png" /></figure><ul><li>Leaky Relu is 0.01 if negative and same number if positive (Handles Vanishing Gradient better than Relu , but may take longer to converge).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/675/1*Wk28nkqQ32QMDTshQqQz-g.png" /></figure><pre><br># Returns the input directly if it is positive; otherwise, returns 0.<br>def Relu(x):<br>    return np.maximum(0, x)<br># np.heaviside(x, 1) returns 0 if x &lt; 0, and 1 if x &gt;= 0.<br>def relu_diff(x):<br>    return np.heaviside(x, 1)<br><br># This helps prevent the dying ReLU problem <br>def leaky_relu(z, alpha=0.01):<br>    return np.where(z &gt; 0, z, alpha * z)<br><br># Gradient is 1 for z &gt; 0 and alpha (small slope) for z &lt;= 0.<br>def leaky_relu_derivative(z, alpha=0.01):<br>    return np.where(z &gt; 0, 1, alpha)</pre><p>We will be using softmax in the output layer , Softmax can convert a list of numbers into outputs of probabilities , General formula for softmax is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*H2ZhfpJJJLMwZV_l" /><figcaption>Softmax Can be used for multi-class probabilties</figcaption></figure><pre>def softmax(z): # This is just a softmax with glorified numerical stability , simple softmax raised warnings <br>    assert isinstance(z,np.ndarray)<br>    z_shift = z - np.max(z, axis=1, keepdims=True)<br>    exp_z = np.exp(z_shift)<br>    return exp_z / np.sum(exp_z, axis=1, keepdims=True)</pre><p>Finally as our Loss function we will be using cross-entropy loss function , it penalized confident wrong predictions more than unsure wrong ones , so its is a perfect choice for classification problems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LlMoO0IrbE0CLsy6.png" /><figcaption>Cross-entropy loss for classification yi is the true label and y_hat is the predicated label (Please note that yi should be one-hot encoded first (will discuss it below)</figcaption></figure><pre>def cross_entropy(y_hat, y): # Stack overflowed this numerically stable cross-entropy , simple one can work too<br>    eps = 1e-15<br>    y_hat_clipped = np.clip(y_hat, eps, 1-eps)<br>    sample_losses = -np.sum(y * np.log(y_hat_clipped), axis=1)<br>    return sample_losses<br><br>def cross_entropy_diff(predictions,true_label):<br>    return (predictions - true_label) # y_should be one_hot encoded</pre><h3>Forward Pass through the network</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PuB2Tn-cNqHQ9XU6GAmjyQ.png" /></figure><p>When we initialized the weights randomly , we have to get the final output ,so to that we take our inputs and then pass them through the network . In simple words , if you look at the neural network as a blackbox , you give input and then a forward pass happens and you get the output .</p><p>At each step previous layer’s input X is passed to the next layer weights a matmul happens between the matrix , and the bias is added to the resultant matrix , After that an activation is generally applied to capture non-linearity.</p><pre>def forward_pass(X, weights_matrix, bias_matrix):<br>    activations = [X]  # store outputs of each layer<br>    A = X<br><br>    for i in range(len(weights_matrix)):<br>        Z = np.matmul(A, weights_matrix[i]) + bias_matrix[i]  # pre-activation<br>        if i == len(weights_matrix) - 1:<br>            A = softmax(Z)  # output layer<br>        else:<br>            A = leaky_relu(Z)     # hidden layers<br>        activations.append(A)<br><br>    return activations  # list of all activations layer by layer</pre><p>In the code above , we store all the activations(Values of neurons inside layers) inside the activations list , inside the for loop , we calculate value of Z = Relu(Xi @ Wi + bi) , if it’s the final layer the output layer , we apply softmax(Z) to capture probabilties , at the end we return a list containing activation values of each node in each layer.</p><h4>One-Hot encoding</h4><p>As discussed above , the Loss function doesn’t work if the target labels are not one-hot encoded , for people who don’t know what one-hot encoding is , it is the process of filling up an array with all the classes , and then marking the true class as 1 , and all the other classes as 0 , i.e. the name “one-hot”</p><pre>def one_hot_encode(y, num_classes):<br>    y = np.array(y).reshape(-1).astype(int)  # Flatten in case it&#39;s shape (n,1)<br>    new_y = np.zeros((y.shape[0], num_classes), dtype=np.float64)<br>    new_y[np.arange(y.shape[0]), y] = 1.0<br>    return new_y</pre><h3>Backpropogation</h3><p>Backpropagation is the core algorithm that enables neural networks to learn by iteratively adjusting weights and biases in the direction that most reduces the loss.</p><p>The maths behind backprop is heavy and the intuition is required to understand , i won’t be able to explain that in such a constrained enviornment , still there are some good resources i can point to</p><ul><li>3 Blue 1 brown video on gradient Descent [<a href="https://youtu.be/IHZwWFHWa-w?si=kLwqM4rvOjGQhAZy">https://youtu.be/IHZwWFHWa-w?si=kLwqM4rvOjGQhAZy</a>]</li><li>My own Handwritten short — notes on Gradient Descent and Maths behind backprop [<a href="https://drive.google.com/file/d/1l56RVCbF7RITNdtuCvD50jqfZs0LDXKI/view?usp=sharing">https://drive.google.com/file/d/1l56RVCbF7RITNdtuCvD50jqfZs0LDXKI/view?usp=sharing</a>]</li><li>3 Blue 1 brown video on calculus behind backprop [<a href="https://youtu.be/tIeHLnjs5U8?si=AXMj1zgRzjqDoyPn">https://youtu.be/tIeHLnjs5U8?si=AXMj1zgRzjqDoyPn</a>]</li></ul><p>I myself followed from these videos and understood better , hope you too.</p><blockquote>Still i took the liberty to make Claude AI comment my original backprop code , so that you can better understand , Raw code is still available in the notebook link at the end of article.</blockquote><h4>Output Layer Gradient Descent</h4><p>This function updates the weights and biases of the output layer using gradient descent. It computes the prediction error by comparing the network’s output to the true labels, calculates the gradients needed to adjust weights and biases, applies gradient clipping to keep updates stable, then performs the update step. Finally, it returns the “error signal” (gradient) needed to propagate learning to the previous (hidden) layer.</p><pre>def output_layer_gradient_descent(activations, weights_matrix, bias_matrix,<br>                                  Y_train_encoded, learning_rate):<br>    layer = -1<br>    # Get output activations a[L] and previous layer activations a[L-1]<br>    out_activ = activations[layer]<br>    prev_activ = activations[layer - 1]<br><br>    # Compute error signal δ[L] = a[L] – Y (softmax + cross-entropy derivative)<br>    delta = cross_entropy_diff(out_activ, Y_train_encoded)<br><br>    # Compute gradients: ∂L/∂W[L] = (a[L-1])ᵀ · δ[L], ∂L/∂b[L] = sum(δ[L])<br>    weight_grad = prev_activ.T @ delta<br>    bias_grad   = np.sum(delta, axis=0, keepdims=True)<br><br>    # Clip gradients to [-1,1] to prevent exploding updates<br>    weight_grad = np.clip(weight_grad, -1.0, 1.0)<br>    bias_grad   = np.clip(bias_grad,   -1.0, 1.0)<br><br>    # Compute gradient for next layer: δ[L-1] = δ[L] · W[L]ᵀ<br>    prev_layer_grad = delta @ weights_matrix[layer].T<br><br>    # Update parameters: W ← W - η·∂L/∂W, b ← b - η·∂L/∂b<br>    weights_matrix[layer] -= learning_rate * weight_grad<br>    bias_matrix[layer]    -= learning_rate * bias_grad<br><br>    return prev_layer_grad</pre><h4>Hidden layer Gradient Descent and Backpropogation</h4><ul><li><strong>Hidden_layer_gradient_descent :</strong><br>This function performs the backpropagation update for all hidden layers. For each hidden layer, it calculates how much that layer contributed to the output error by combining the error signal from the next layer with the derivative of the activation function (Leaky ReLU). It then computes, optionally clips, and applies weight and bias gradients, updating the parameters and passing the new error signal to the previous layer.</li><li><strong>Backpropagate</strong>:<br>This function runs backpropagation for the whole network: it first updates the output layer and gets the gradient to propagate, then steps backward through all hidden layers so each layer’s weights and biases adjust according to their contribution to the prediction error.</li></ul><pre>def hidden_layer_gradient_descent(activations, weights_matrix, bias_matrix,<br>                                  post_layer_gradient, learning_rate):<br>    num_layers = len(weights_matrix)<br><br>    # Loop backwards over hidden layers: L-1, L-2, …, 1<br>    for i in range(2, num_layers + 1):<br>        layer = -i<br>        curr_activ = activations[layer]        # a[k]<br>        prev_activ = activations[layer - 1]    # a[k-1]<br><br>        # δ[k] = (propagated δ[k+1]) ∘ f&#39;(z[k]) for LeakyReLU<br>        delta = post_layer_gradient * leaky_relu_derivative(curr_activ)<br><br>        # ∂L/∂W[k] = (a[k-1])ᵀ · δ[k], ∂L/∂b[k] = sum(δ[k])<br>        weight_grad = prev_activ.T @ delta<br>        bias_grad   = np.sum(delta, axis=0, keepdims=True)<br><br>        # Clip gradients<br>        weight_grad = np.clip(weight_grad, -1.0, 1.0)<br>        bias_grad   = np.clip(bias_grad,   -1.0, 1.0)<br><br>        # Prepare δ[k-1] = δ[k] · W[k]ᵀ for next iteration<br>        post_layer_gradient = delta @ weights_matrix[layer].T<br><br>        # Update parameters<br>        weights_matrix[layer] -= learning_rate * weight_grad<br>        bias_matrix[layer]    -= learning_rate * bias_grad<br><br>def backpropagate(activations, weights_matrix, bias_matrix,<br>                  Y_train_encoded, learning_rate):<br>    # 1) Handle output layer, get δ[L-1]<br>    hidden_grad = output_layer_gradient_descent(<br>        activations, weights_matrix, bias_matrix,<br>        Y_train_encoded, learning_rate)<br>    # 2) Backpropagate through all hidden layers<br>    hidden_layer_gradient_descent(<br>        activations, weights_matrix, bias_matrix,<br>        hidden_grad, learning_rate)</pre><h3>Training the neural network</h3><p>Now , we have everything we need forward_pass , activation functions , random initializations , backpropagation logic , Gradient Descent based weight updates everything.</p><p>For training , we will use a technique known by some as stochastic Gradient Descent or mini-batch Gradient Descent .</p><p>We will only supply a batch of 32 or 64 samples at once , for gradient calculations , the result of which is that gradient calculations happen a little faster than if we would have used batch_size as full input.</p><p>Also , at each epoch we will be measuring the accuracy of the model on the held-out validation set , To know when the model is over-fitting the data, and when to stop.</p><p>Function signature is given below + You can check out the full function in the notebook link below</p><pre>def train_neural_network(X, Y, X_t, <br>Y_t, input_dimension: int,<br>layer_dimension: tuple, num_classes: int = 10, <br>epochs: int = 1000, learning_rate: float = 0.001, <br>batch_size: int = 64,decision_boundary : bool = False,<br>decision_boundary_save_dir : str = None) -&gt; tuple:</pre><ul><li>X, Y, <em># Training features and labels</em></li><li>X_t, Y_t, <em># Test features and labels</em></li><li>input_dimension, <em># Number of input features</em></li><li>layer_dimension, <em># Tuple of hidden+output layer sizes</em></li><li>num_classes=10, <em># Number of output classes</em></li><li>epochs=1000, <em># Number of full passes over the data</em></li><li>learning_rate=0.001, <em># Step size for gradient descent</em></li><li>batch_size=64, <em># Number of samples per gradient update</em></li><li>decision_boundary=False, <em># Whether to save decision-boundary plots</em></li><li>decision_boundary_save_dir=None #<em> Where to save Decision boundary plot</em></li></ul><pre># In our case the function call looks like this.<br>epochs = 25<br>train_neural_network(X_train,y_train,X_test,y_test,input_layer_dimension,<br>layer_dimension,10,epochs,0.01,batch_size=32)</pre><h3>Result of Training Neural Network</h3><blockquote><strong>This is where it gets interesting , Read the below sections carefully even if you skimmed the above sections.</strong></blockquote><p>for MNIST , we receive the output logs like this below , which indicate that out model is doing pretty well minimizing the average loss and scores pretty well on the final test-set give it a few more iterations and a bigger batch size it may even touch 99 % accuracy on test set.</p><pre>Epoch 0: average_loss = 7.5146,entropy_loss = 14089.9313, test accuracy = 96.03%<br>Epoch 1: average_loss = 3.5791,entropy_loss = 6710.8479, test accuracy = 96.74%<br>Epoch 2: average_loss = 2.7530,entropy_loss = 5161.7971, test accuracy = 96.62%<br>Epoch 3: average_loss = 2.1904,entropy_loss = 4106.9433, test accuracy = 97.57%<br>Epoch 4: average_loss = 1.8242,entropy_loss = 3420.3643, test accuracy = 97.58%<br>Epoch 5: average_loss = 1.5239,entropy_loss = 2857.2397, test accuracy = 97.55%<br>Epoch 6: average_loss = 1.2772,entropy_loss = 2394.8182, test accuracy = 97.68%<br>Epoch 7: average_loss = 0.9973,entropy_loss = 1869.8481, test accuracy = 97.73%<br>Epoch 8: average_loss = 0.8967,entropy_loss = 1681.2330, test accuracy = 97.74%<br>Epoch 9: average_loss = 0.7300,entropy_loss = 1368.6913, test accuracy = 97.93%<br>Epoch 10: average_loss = 0.6425,entropy_loss = 1204.7196, test accuracy = 97.53%<br>Epoch 11: average_loss = 0.5529,entropy_loss = 1036.6888, test accuracy = 97.74%<br>Epoch 12: average_loss = 0.4468,entropy_loss = 837.6917, test accuracy = 97.98%<br>Epoch 13: average_loss = 0.2582,entropy_loss = 484.0781, test accuracy = 98.03%<br>Epoch 14: average_loss = 0.2055,entropy_loss = 385.2663, test accuracy = 97.89%<br>Epoch 15: average_loss = 0.1422,entropy_loss = 266.6131, test accuracy = 98.01%<br>Epoch 16: average_loss = 0.1409,entropy_loss = 264.1000, test accuracy = 97.97%<br>Epoch 17: average_loss = 0.0771,entropy_loss = 144.5212, test accuracy = 97.98%<br>Epoch 18: average_loss = 0.0397,entropy_loss = 74.5092, test accuracy = 98.18%<br>Epoch 19: average_loss = 0.0093,entropy_loss = 17.4965, test accuracy = 98.15%<br>Epoch 20: average_loss = 0.0046,entropy_loss = 8.5332, test accuracy = 98.12%<br>Epoch 21: average_loss = 0.0029,entropy_loss = 5.4469, test accuracy = 98.16%<br>Epoch 22: average_loss = 0.0022,entropy_loss = 4.1695, test accuracy = 98.17%<br>Epoch 23: average_loss = 0.0018,entropy_loss = 3.3318, test accuracy = 98.17%<br>Epoch 24: average_loss = 0.0016,entropy_loss = 2.9209, test accuracy = 98.17%</pre><p>Even so , when we plot the Training Loss and Test accuracy over epochs</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/1*xyQNUd_C4B7EBHg8u9F8Fw.png" /><figcaption>Notice how Test accuracy begin to climb around the 5 th epoch , and how average loss gradually plateud around 15th epoch , the model is generalizing well.</figcaption></figure><h4>Performance on Fashion-MNIST</h4><p>On the Fashin-MNIST dataset , our model performance looked like this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/1*AWEthvYghHlOJjkmfkzrmQ.png" /></figure><p>The model dropped the average loss atleast until 20 epochs , after that gradually the average loss started increasing , either the model diverged poorly due to high training rate or learning rate , or too severe over-fitting</p><p>The Test accuracy was on a high till around 17th epoch , after which it fell overall</p><p>A good model is the model we had around 20 epochs , that model will generalize well, and be less prone to over-fitting.</p><h3>Additional Visualization of Decision Boundary plots</h3><p>To better understand the model’s performance , i tried in my own capacity to mimic tensorflow playground a popular site used to experiment with neural networks and their decision boundaries .</p><blockquote>In machine learning, a decision boundary is a hypersurface that separates different classes or categories within a feature space, allowing a model to classify new data points. It’s essentially the “line” (in 2D) or “surface” (in higher dimensions) that distinguishes between different classes in a classification problem.</blockquote><ul><li>The Red Dots indicate Class 1</li><li>The blue dots indicate Class 2</li><li>Orange Background means the model is classifying that region as class 1 (Red Dots)</li><li>Blue background means the model is classifying that region as class 2 (Blue dots).</li></ul><p>Let’s take a few examples starting up with the moons dataset , downloadable via scikit-learn</p><pre>X_dummy, Y_dummy = make_moons(n_samples=1000, noise=0.13, random_state=42)<br>plt.scatter(X_dummy[:, 0], X_dummy[:, 1], c=Y_dummy, cmap=&#39;coolwarm&#39;, edgecolors=&#39;k&#39;)<br>plt.xlabel(&#39;Feature 1&#39;)<br>plt.ylabel(&#39;Feature 2&#39;)<br>plt.title(&#39;Make Moons Dataset (No Noise)&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*C6wKdrvLyIHycOt7JkF05Q.png" /><figcaption>It is the moons dataset with noise = 0.15</figcaption></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FHbdiWA0Pb2k%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DHbdiWA0Pb2k&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FHbdiWA0Pb2k%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/2d77fc2797d8e5944eaa556c8ef45699/href">https://medium.com/media/2d77fc2797d8e5944eaa556c8ef45699/href</a></iframe><p>Another Interesting Dataset , we have is the spiral Dataset , It is in the form of co-centric circles with varying radius difficult to seperate with any other machine learning algorithm.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/1*UcMy_fnTNgsOvmyK6IOZ9w.png" /><figcaption>The spiral Dataset</figcaption></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fy7fvfdrgQ40%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dy7fvfdrgQ40&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fy7fvfdrgQ40%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/f4baa6f8bd91cc5c4fe360811fe87573/href">https://medium.com/media/f4baa6f8bd91cc5c4fe360811fe87573/href</a></iframe><h3>What’s Next</h3><p>Now this chapter is complete , i will move forward with chapter 10 of hands on ML , this time implementing all this in keras and tensorflow , Ready for the Libraries :) /.</p><p>Also ,understanding neural networks and implementing one from scratch took me this whole week , so yeah it’s week 4 of my series . Signing off !</p><h3>Quick note before I go:</h3><p><em>I’ve recently started sharing my learnings in public, and i dont use ChatGPT or other AI tools to write (not that i hate them) but because I want these reflections to stay personal and true to what I’m actually picking up each day</em></p><p><em>I have committed all the relevant code and the dataset on my github go to the notebook link below to open the repo ,</em></p><p>The repo has two files</p><ul><li>neural network from scratch (consists of experimented code + working code)</li><li>neural network from scratch cleaned (consists of only working code)</li></ul><p><a href="https://github.com/akshatkhatri/Neural_network_from_scratch">Project Repository Link</a></p><p><a href="https://github.com/akshatkhatri">Github Profile Link</a></p><p><em>Feel Free to connect with me on Linkedin</em></p><p><a href="https://www.linkedin.com/in/i-am-akshat-khatri/">Linkedin Profile Link</a></p><p><strong><em>Thanks for reading — means a lot :)</em></strong><em><br>Best,<br>Akshat Khatri</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3345d67c6d2b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My Learnings About Scaling Laws, Model Size, and AI’s Real Bottlenecks]]></title>
            <link>https://medium.com/@akshat.dev/my-learnings-about-scaling-laws-model-size-and-ais-real-bottlenecks-42764c252c8e?source=rss-cb3d13125e1c------2</link>
            <guid isPermaLink="false">https://medium.com/p/42764c252c8e</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[training-data]]></category>
            <category><![CDATA[scaling]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Akshat Khatri]]></dc:creator>
            <pubDate>Tue, 29 Jul 2025 05:43:29 GMT</pubDate>
            <atom:updated>2025-07-29T05:43:29.182Z</atom:updated>
            <content:encoded><![CDATA[<p><em>I will be using the book AI engineering by Chip Huyen and I’m continuing with Chapter 2 (Understanding Foundational Models) ,Today marks Day 21.</em></p><blockquote>In this article, we will mainly be focusing on how model size, training data, and computational constraints interact, the limitations of simply increasing parameters, the impact of newer approaches like Mixture of Experts and Chinchilla scaling law, and the real-world bottlenecks shaping AI’s future.</blockquote><h3>Model Size</h3><p>Whenever we see a model , or listed , we often hear the word parameter , but what does it mean , parameter is usually appended to the end of the model’s name like Llama-15B referes to Llama model with 15 Billion Parameters.</p><p>In general , whenever we increase a model’s parameters its performance tends to increase , but this is not always the case as we will see , sometimes smaller models trained on newer technologies tend to outperform bigger and older ones.</p><p>These numbers of parameters helps us in having a good estimate of the GPU memory we will require to host these models.</p><h3>MOE( Mixture of experts Model — Sparse):</h3><p>MOE is different from traditional language models where the input has to do a full forward pass through the parameters regardless of the number of parameters it affects .</p><p>A mixture of experts model has the concept of conditional compute , where only a specific number of parameters associated with each expert are fired , whenever an input is received, This makes model size of MOE a misleading figure , an MOE or sparse model with 80 Billion parameters is not equal to an 80-Billion parameter transformer model , as MOE only fires up a limited number of parameters each time.</p><p>The MOE uses a gating mechanism to route the tokens to the appropriate expert , it uses a sigmoid or softmax function to achieve the same.</p><p>Example-&gt; Mistral 8x7B , i.e. 8 experts with 7 billion parameters each , but some experts share parameters , so its total parameters are 47 billion parameters.</p><p>It uses top-k Sampling , at each token only the most relevant 2 experts are selected , this means that while the model has 47 billion parameters , at any given time only around 13–14 billion parameters are in use.</p><h3>Data matters more than parameters</h3><p>A Large model say with 20 billion parameters , if trained on only 1 type of data , or a biased data or incorrect data is bound to learn these patterns , and replicate the behavior.</p><p>A smaller model say 3 Billion parameters ,if trained on diverse data , with unbiased opinions and factually correct , will outperform the bigger one.</p><p>An analogy i like to think of is a model’s parameter is like its IQ , how much can it learn , and how smart it is , but a model’s training data is the up-bringing what type of things it has seen , turns out even smarter models if not given good resources to learn from , will perform poorly.</p><p>So we measured parameters , but how to measure data , There are several ways-</p><ol><li>We can use number of training samples as a metric to compare , but training samples vary as for some a training sample might be a sentence or a word , for others it may be a book.</li><li>A word count or sentence count isn’t a accuracte metric either.</li><li>A token while not being perfect due to models having different tokenization scheme , a sub-word tokens are counted to compare how much training data a model is trained on. <br>for Example -&gt; Llama 1 was trained on 1.4 Trillion Tokens.</li><li>Number of Training tokens , this isnt the same as number of tokens , for example a model trained on 1 million tokens for 2 epochs , has training tokens has 2 million . (an epoch is a pass through the dataset)</li></ol><blockquote>Data Quantity, Quality and Diversity Matter more</blockquote><h3>How to measure compute ?</h3><p>Pre-training Large models require compute i.e. how much compute power is required to train one language model , we can use how much GPUs it requires , or how much VRAM , but that is not accurate , as different vendoe\rs have different criterias and no one bill fits all.</p><p>A more standardized way is to measure FLOP (Floating point operations) , It measures how many floating point operations are performed for a given task.</p><p>FLOPS can often be confused with FLOP/s , FLOPS is a metric for computing the requirement for a task such as this model requires 1⁰²³ FLOPS to complete , while FLOP/s is used to measure hardware efficiency ,<br>for example a GPU can deliver 1⁰¹³ FLOP/s.</p><p>THere is also utilization , utilization is a measure of how much efficiently are you using the hardware , generally if we get half of the advertized efficiency say 50 % we are good , anything around 70 % is better , there are other ways too of improving hardware efficiency that we will look at up in the future.</p><h3>Scaling Law : Compute-Optimal Models</h3><p>As the task of training (pre) a language model is very time-consuming and expensive , we need to adhere to the budget we are provided , say that for a given number of FLOPs , what model size and dataset size would give the best performance , These are questions that play a key role in model design.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/771/1*TOWr374lHAk8s1DuK5Wq9Q.png" /><figcaption>Source : <a href="https://arxiv.org/pdf/2203.15556">https://arxiv.org/pdf/2203.15556</a></figcaption></figure><h4>Chinchila Scaling Law</h4><p>At the core of this law is the fact that solely increasing the model’s parameters and making a bigger-better model without focusing on data used for model training will not make a better model , On the contrary more training data with less parameters will probably work better.</p><p>Earlier Open AI in 2020 developed their thinking that focusing on increasing parameters of a model leads to an increase in model’s perfomance getting better , that’s why GPT-3 model was trained on 300 B training tokens , with 175 B parameters , for each paramters the ammount token was around 1.7 tokens / parameters, which is not good at all.</p><blockquote>Chinchila in his hypothesis said that the Training tokens should be around 20x more than the number of parameters for compute-optimal training</blockquote><p>This means a 3B parameter model should have 60 B training tokens, The model size and training tokens should be scaled equally , for example for doubling of training tokens parameters should be doubled as well and vice-versa.</p><h3>Scaling Extrapolation</h3><p>Like any traditional ML models , Language models too depend on the optimal value of hyper-parameters ( parameters passed before training) , in case of light-weight models we can experiment with different hyper-parameters and see which one gives the most optimal result .</p><p>But we can’t do this in the case of LLMs as training one model takes so much time , doing it for other hyper-parameters is a big and compute extensive task .</p><p>So we try to predict the behavior of larger models on some hyper-parameters by first testing them on a smaller model and infer its results as the result of the larger model.</p><ul><li>Researchers train models at multiple smaller scales and fit their performance (e.g., loss, accuracy) using mathematical functions, commonly power laws.</li><li>These fits are then used to predict the expected performance of much larger models that would be impractical or too expensive to train without such extrapolation.</li></ul><h3>Bottlenecks or Problems with scaling</h3><p>Let’s see a bit of History when GPT-1 was release it had 117 million parameters , GPT-2 Increased it by orders of 2 magnitude converting to 1.5 Billion . GPT-3 Had two orders of magnitude more parameters than GPT-2 (175 Billion) . So going by the current rate if parameters keep increasing the model will grow to be more smarted , given the training data is scaling as well.</p><blockquote>would increasing model’s size continously result in better performance always or will it plateaus , after a stage , the question is hard to answer</blockquote><p>However , there are still 2 major bottlenecks — Training Data and Electricity.</p><p>Remeber chinchila scaling law , as parameters increase so should training data , and most of the foundational models are trained on open data of the internet , the rate at which these models are getting trained on new data , is faster than the ammount of data generated.</p><blockquote>If you’ve ever put something on the internet , you can assume that is already or will be included in the training set of some large language model.</blockquote><p>According to a projection , models would consume all human-generated internet data by 2028 or 2029.</p><p>Another worry , is that the internet is getting filled by AI-generated data or content , as this continues to rise , foundational models often train themselves on this data , and tend to forget earlier original patterns.</p><p>THe other bottlenecks is electricity , currently data-centers consume 2 % of global electricity . This number is estimated to reach between 15–20 % by 2030 ( A staggering ammount !) , Until we can figure out more energy sources like nuclear power plants , This can cause electricity demand to increase and prices to sky-rocket.</p><blockquote>Access to massive training data and cheap electricity will be the ultimate competitive edge in the AI race.</blockquote><h3>What’s Next</h3><p>Next step will be to discuss about Post-Training and how supervised Fine-tuning and preference fine-tuning impace models , RLFH .</p><h3>Quick note before I go:</h3><p><em>I’ve recently started sharing my learnings in public, and i dont use ChatGPT or other AI tools to write (not that i hate them) but because I want these reflections to stay personal and true to what I’m actually picking up each day</em></p><p><em>Also , i maintain a notebook where i practise all the codes if you are interested in that you can follow my github notebook link</em></p><p><a href="https://github.com/akshatkhatri">Notebook Link</a></p><p><a href="https://github.com/akshatkhatri">Github Profile Link</a></p><p><em>Feel Free to connect with me on Linkedin</em></p><p><a href="https://www.linkedin.com/in/i-am-akshat-khatri/">Linkedin Profile Link</a></p><p><strong><em>Thanks for reading — means a lot :)</em></strong><em><br>Best,<br>Akshat Khatri</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=42764c252c8e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My understanding of RWKV architecture (an alternative to transformers)]]></title>
            <link>https://medium.com/@akshat.dev/my-understanding-of-rwkv-architecture-an-alternative-to-transformers-506a1136e9e7?source=rss-cb3d13125e1c------2</link>
            <guid isPermaLink="false">https://medium.com/p/506a1136e9e7</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[rwkv]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[transformers]]></category>
            <dc:creator><![CDATA[Akshat Khatri]]></dc:creator>
            <pubDate>Mon, 28 Jul 2025 09:33:46 GMT</pubDate>
            <atom:updated>2025-07-28T09:33:46.158Z</atom:updated>
            <content:encoded><![CDATA[<p><em>I will be using the book AI engineering by Chip Huyen and will be continuing today with Chapter — 2(Understanding Foundational Models),Today marks Day 20.</em></p><blockquote><em>In this article, we will mainly discuss RWKV architecture on a high-level , contrasting it with RNN only and transformer only architecture , its shortcomings and where it can be used .</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/0*Vz9tS9BVK9oRAKdB.png" /><figcaption>RWKV achitecture , source : wiki.rwkv.com</figcaption></figure><h3>RWKV (Receptance-weighted-Key-Value):</h3><p>To understand RWKV , we first need to understand that both seq2seq RNN and transformers , have their own downsides and advantages.</p><h4>Short-comings of RNN —</h4><ol><li>a seq2seq architecture tended to forget earlier information easily , due to its concept of hidden states.</li><li>Also inference was slower as training is sequential and not parallelized.</li></ol><h4>Advantages of RNN —</h4><ol><li>RNN architecture had linear time complexity compared to quadratic in the case of transformers.</li><li>The same weights apply at every time step, reducing the number of parameters compared to deep feedforward networks</li></ol><h4>Short-comings of Transformers —</h4><ol><li>Transformer Architecture has a sequential output bottleneck , and a limit on the context length.</li><li>Training takes time due to quadratic time complexity.</li></ol><blockquote>RWKV aims to take the best of both worlds , reduce training time and make it paralleizable using transformers and then generate inferences using RNNs.</blockquote><ul><li>During training RWKV works like a transformer it can take whole sequence of texts or documents as input and train on them parallely , This make training fast and efficient as GPU are optimized for this.</li><li>During inference , RWKV acts as a RNN , it only remebers its previous hidden state and the current input , and doesn’t store information about the other sequences using KV cache or other mechanisms.</li></ul><p>RWKV is extremely good for low-end devices especially the ones that want to remeber long contexts , or if a line on sentence 1 , carries weight on meaning of sentence 99,.</p><p>This means constant memory usage, no matter how long the text. It’s a huge advantage for long documents or running models on smaller devices.</p><h3>RWKV block design</h3><p>Remember how we used Q,K,V vectors in transformers self-attention , similarly the RWKV uses R,W,K,V vectors named as Receptance(R) , Weight-decay(W) , Key vector (K) and Value vector (V) .</p><p>we discussed in detail about the Key and Value vector in our previous articles around transformers and attention .</p><p>We will now discuss the R and W vectors.</p><ul><li>The Receptance Vector (R) controls how much information to let-in and how much information is to be retained , it is similar to the LSTM gate in RNNs</li><li>It ensures that the model doesn’t forget too quickly or get overloaded by new inputs.</li><li>The weight vector (W) Implements a smooth, learnable decay on past contributions, giving more weight to recent tokens.</li><li>The weight vector is a copy or you can say mimics attention layer but is O(1) in complexity , more time-saving.</li></ul><h3>When should RWKV be used compared to Transformers</h3><p>On paper , RWKV is a gold , and it is for some tasks such as where the compute is very low , but RWKV struggles with long sequence of texts .</p><p>Also RWKV is still growing and doesn’t have a developed ecosystem like architectures like transformers do , also transformers are more adopted.</p><p>RWKV has its own merits for being able to work with lower-cost hardwares , and how not everyone who wants to run a transformer must have big GPUs , RWKV works for you.</p><h3>What’s Next</h3><p>Next step will be to go further into alternate architecture that are gaining pace like SSMs , and Mamba and Jamba , or alternatively move ahead with the chapter , we will see</p><h3>Quick note before I go:</h3><p><em>I’ve recently started sharing my learnings in public, and i dont use ChatGPT or other AI tools to write (not that i hate them) but because I want these reflections to stay personal and true to what I’m actually picking up each day</em></p><p><em>Also , i maintain a notebook where i practise all the codes if you are interested in that you can follow my github notebook link</em></p><p><a href="https://github.com/akshatkhatri/Machine_learning_playground/blob/main/Chapter_3_geron/classification.ipynb">Notebook Link</a></p><p><a href="https://github.com/akshatkhatri">Github Profile Link</a></p><p><em>Feel Free to connect with me on Linkedin</em></p><p><a href="https://www.linkedin.com/in/i-am-akshat-khatri/">Linkedin Profile Link</a></p><p><strong><em>Thanks for reading — means a lot :)</em></strong><em><br>Best,<br>Akshat Khatri</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=506a1136e9e7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Inside the Black Box: How Transformers Process Language (Beyond Tokens and Embeddings)]]></title>
            <link>https://medium.com/@akshat.dev/inside-the-black-box-how-transformers-process-language-beyond-tokens-and-embeddings-ceeb41a82893?source=rss-cb3d13125e1c------2</link>
            <guid isPermaLink="false">https://medium.com/p/ceeb41a82893</guid>
            <category><![CDATA[attention]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Akshat Khatri]]></dc:creator>
            <pubDate>Sun, 27 Jul 2025 11:23:32 GMT</pubDate>
            <atom:updated>2025-07-27T11:23:32.154Z</atom:updated>
            <content:encoded><![CDATA[<p><em>I will be using the book Hands on LLM by jay alammar ,I will be continuing today with Chapter — 3 (Looking Inside Large Language Models), you can check the previous article of chapter 2 written by me </em><a href="https://medium.com/@akshat.dev/how-language-models-capture-meaning-with-embeddings-1f15a14113b0"><em>here</em></a><em> , Today marks Day 18 &amp; 19.</em></p><p>In the previous article(s) , we understood about tokens and embeddings , now we will look at the core concepts inside the Language Model (Transformers).</p><blockquote>In this article, we go beyond tokens and embeddings to unpack how transformer language models work. We explore parallel processing, attention heads, feedforward networks, LM heads, sampling methods, positional embeddings (like RoPE), and efficiency techniques like KV-caching and local attention that power today’s LLMs.</blockquote><h3>Overview of Transformer Models</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yUJFr8Og8hefLJOP.png" /><figcaption>Transformer Architecture</figcaption></figure><ul><li>At an Abstract level , one can see a Transformer as taking in some text as input and then doing some black-box computations as it is trained on large datasets and returning a response text .</li><li>One of the Characterstics and perhaps a bottleneck of these transformer models is they give sequential output , one token after the another.</li><li>Each token generated is a forward pass through the model , we’ll talk more about the auto-regressive nature of the transformer models below.</li><li>After a token is generated , we generate another token by adding the generated token at the end of input and doing the forward pass again.</li><li>So in Theory , The transformer model is only predicting the next token based on previous tokens and it does it well . Most text-generation models are auto-regressive models .</li></ul><p>There are text-representation models like BERT too , which are non-auto-regressive.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LQIji8wpRps75hPSxu1tyw.png" /><figcaption>Text-generation using microsoft phi</figcaption></figure><p>The model generated all this text using auto-regressive technique and simply predicted the next token , as an end user we often don’t see , that’s why most companies stream tokens as they come as displaying the whole text might take some time.</p><h3>Forward Pass in Transformers</h3><p>Apart from the transformer blocks inside the transformer and the auto-regressive loop , there are 2 more components namely — Tokenizer and LM head, We discussed tokenizer in the previous articles , but i’ll explain it again briefly here .</p><p>THe main job of the tokenizer is to convert tokens into token_ids for the model to process , these token ids are then matched inside the token_embeddings table, Each token_id has an embedding for it.</p><p>First the input goes into the tokenizer , which i explained , then into the transformer blocks which is a feed — forward neural network and then into an LM head (Its job is to perform sampling and to output the probability of each token to occur).</p><p>We can look at the layers we have by printing out the model variable</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*1BZ9SB_xms_tHqOMX2Q43g.png" /><figcaption>Different layers inside the LLM</figcaption></figure><p>Let’s Draw some insights from the structure —</p><ol><li>Inside the model , we can see the embedding matrix of size 32064 * 3072 , it means that the model can understand a total of 32064 tokens and each token has a 3072 dimensional embedding associated with it.</li><li>The next major component is the phi3 Decoder Layer , there are 32 such layers .</li><li>Each of the decoder blocks / layer contain a self-attention module and a feedforward neural network (mlp or multi-level perceptron). We will cover this in detail in this article.</li><li>Then comes the normalization , after that we see at the bottom is the LM head , it takes a vector of size 3072 (Latest token) and outputs the probability score of the tokens all 32064 tokens .</li></ol><h3>Choosing a single token from probability Distribution ( Sampling / Decoding)</h3><p>Looking at step 4 , we can say that we have received the probabilty of each token occuring , now the question remains in which one of these tokens will we choose / sample (we will randomly select tokens , a token with say 40 % probabilty will have a 40 % chance of being chosen).</p><p>The easiest and most intuitive way of sampling is to select a token with the highest probabilty of occuring , while this sounds good on paper it doesn’t always lead to good results , Because if we choose to greedy sample everytime it can lead to the model being dull , less open-minded and always give predictable and safest answers , not coming up with creative ideas.</p><p>We can also add some randomness and then select the 2nd or 3rd token , altough practically we either use top-k sampling or nucleus sampling , but that’s a topic of dicussing for a later time.</p><p>Setting the Temperature to zero automatically results in greedy sampling / Decoding.</p><h3>Parallel Token Processing and context size</h3><p>The best Feature in transformers also one of the main reasons seq2seq RNN was replaced was transformers are capable of processing input parallely.</p><p>Current Transformer models have a limit on how much tokens they can process in one go , this is often called the context length of model.</p><p>First all the embeddings are processed inside the stack of transformer blocks parallely , then an output vector is generated for each token passed.</p><p>We only use the output embedding vector for the last token to generate the next token , a question that can come up to your mind is if we use only the last embedding vector , why generate the ones before it.</p><p>The answer is because attention mechanism requires us to do , to arrive at the embedding of the last token , we need the embeddings of the previous ones.</p><h4>Speeding up Generation by Caching Embedding vectors</h4><p>We can speed up the generation of tokens by caching the output vectors for the tokens, instead of recalculating them at each forward pass ( as it is a time-consuming process) , This is an optimization technique called the keys-value cache.</p><p>In Hugging-Face the cache is enabled by default , and often provides signinficant speed up , one downside is that it requires more memory to store the key-value pairs.</p><h3>Transformer Block (Most important)</h3><p>We have talked about how tokenization happens , how tokenizer comes into play , text embeddings ,forward pass , LM heads , sampling , but what happens inside the transformer block . Let’s Understand.</p><p>Transformer LLM’s consists of many transformer blocks arranged sequentially.</p><p>As we saw inside the model variable the transformer block is made up of two important components</p><ol><li>The attention Layer (whose main concern is to incorporate relevant information inside the embeddings)</li><li>feed forward neural layer which is the model’s processing capability.</li></ol><h4>Feed-forward neural network at a glance</h4><p>Feed-forward neural network is the learned brain of the transformers, say you give a input “ the Taj” , and the model gives output either hotel or mahal , It is most probably because it was trained on similar data , and likely memorized the behaviors and patterns.</p><p>But these models are not at all , all about memorization , the model’s likely relate how data is related to each other and interpolate so it generalizes better to the future inputs .</p><h4>The attention layer</h4><p>Attention mechanism is perhaps the most pivotal part of the transformer architecture , in a nut-shell the attention layer finds the words the current word should pay attention to and incorporates information from its embeddings to the current embedding.</p><p>There are two important parts of the attention layer —</p><ol><li>Relevance scoring → How relevant a particular token is that to another token</li><li>Combining information -&gt; Incorporating information of the token’s embeding into another token , based on how relevant they are.</li></ol><h4>Concept of Attention Heads</h4><p>Attention heads are present inside the attention layer , and each attention heads has its own set of query , key and value (QKV) vectors . Each token is passed through each attention head , with each attention head focusing on the relevant part of the sentence.</p><p>This increases the mode’s complexity to model complex inputs and deeper relationships between tokens.</p><h4>How attention is calculated</h4><p>we will take a single attention head for easier explanation, Before we start let’s first explore what is the input to the attention layer and the output.</p><ul><li>The inputs to the layers are vector represenations of the current position or token.</li><li>the vector representations of the previous tokens.</li></ul><p>And our goal is to process the last sentence , such as —<br> “The cat sat on the mat” , Attention mechanism should know that it should focus on mat in the context of cat and sat.</p><p>When you train a transformer model (or any neural network), you’re not explicitly telling it what the Query, Key, and Value vectors should be for each input. Instead:</p><ul><li>The model starts with random numbers (weights) in the Query, Key, and Value matrices .</li><li>As the model passes batches of data through these matrices during training, it makes predictions (like the next word in a sentence).</li><li>It measures how far off its guess was from the actual answer using a loss function.</li><li>Then, using a neural network process called backpropagation, it tweaks the numbers in all these matrices (including the ones for Query, Key, and Value) so that next time, its predictions get closer to the right answer.</li></ul><p>This cycle repeats over millions of examples, gradually “teaching” the model better patterns for forming the Q, K, and V vectors from its data. You don’t program these matrices — they are learned automatically, just like the weights in any regular neural network layer.</p><p>Once we have the pre-trained Q,K,V vectors we can multiply our input embedding to these Q,K,V vectors to get our own Qi, Ki,and Vi vectors,</p><p>Note that the size of Q,K,V vector is d_model * d_model , so multiplying it with input which is size 1 * d_model will result in an output of 1 * d_model.</p><p>This step is called projecting input to the Q,K,V matrices , for more stripped down detail , check out my introductort article on attention.</p><p>Now each token has its own Query , Keys and Value vectors , we can begin extracting information from them on how relevant they are to each other and how to enrich information of other tokens.</p><h4>Self-Attention : Relevance Scoring</h4><p>To compute the relevance of the latest token we have againist all the other tokens , The relevance scoring of the token with other tokens is calculated by multiplying the query vector of the current position with the keys vector of all the others.</p><p>This produces a score stating how relevant each previous token is , then we pass that into a softmax function which maps the number into a probability .</p><h4>Self-Attention : Combining Information</h4><p>Now that we have the relevant scores , next step is to combine the token’s embedding with another tokens based on the relevance scores.</p><p>For that we multiply each value vector with their respective relevance score and then sum them up to produce a resulting vector , which represents context + that token’s embedding.</p><h4>Local Attention : Efficient Attention Mechanism</h4><p>As transformer models grow and are applied to ever longer sequences, the original full self-attention mechanism — where every token can attend to every other token — becomes increasingly inefficient. This is due to its quadratic computational and memory complexity with respect to sequence length. To address this challenge, local and sparse attention mechanisms have emerged as efficient alternatives.</p><h3>What Is Local Attention?</h3><p>Local attention is an approach where each token only attends to a limited window of neighboring tokens, rather than the full sequence. For example, instead of each token considering all others, it might only look at the five tokens before and after itself.</p><ul><li>Why this works: In many NLP and time-series tasks, much of the relevant information is in a token’s immediate neighborhood such as a word’s context in a sentence or recent data points in a time series. Local attention leverages this intuition.</li><li>Computational advantage: Reduces time and memory complexity from O(N²) to about O(N⋅k), where N<em>N</em> is the sequence length and k<em>k</em> is the window size — much more efficient for long texts.</li></ul><h3>Multi-query and grouped — query attention:?</h3><p>Grouped Query attention mechanism and multi-query attention mechanisms are recent advancements in attention mechanism , with the main aim of reducing inference time.</p><p>These mechanisms apply on individual attention heads.</p><p>In standard, multi-head attention each head gets its own set of query,key and value projections</p><p>Multi-query attention (MQA) changes that , each attention head gets its own , queries vector but the keys and values vector remains the same for each head.</p><p>Grouped — query attention (GQA) gives a trade-off between standard and MQA , it groups attention heads into different groups like attention heads 1–4 in group A , 5–8 in group B , e.t.c. , and each group gets its own shared key and value vector , and each attention head gets its own query vector like MQA.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0PzcfvE9FbEa3PuPHeOviw.png" /></figure><h3>Positional Embeddings (RoPE) :</h3><p>Positional embeddings are a core technique in transformer models used to encode the order of tokens in a sequence. Since the self-attention mechanism in transformers processes all tokens in parallel (treating the input as a set rather than a sequence), it has no inherent knowledge of token order. Positional embeddings solve this problem by providing each token with additional information indicating its position in the sequence.</p><h4>What is Rotary Positional Embedding (RoPE)?</h4><ul><li>RoPE encodes position by rotating each token’s embedding vector in its feature space by an angle that depends on the token’s position in the sequence.</li><li>This rotation is applied mathematically to Query and Key vectors during self-attention, allowing the model to naturally capture both absolute and, very importantly, relative positional information.</li><li>For every token position, a unique rotation angle (based on position and embedding dimension) is computed.</li><li>Token embeddings are split into pairs and each pair is rotated in a 2D plane by this angle.</li><li>When Queries and Keys (which received the same rotation) interact in self-attention, their dot product reflects not just who they are, but how far apart they are in the sequence.</li><li>Classic positional encodings (learned/static or sinusoidal) only encode absolute position and have more trouble generalizing to longer contexts.</li><li>RoPE directly and efficiently encodes <em>relative positions</em> and generalizes better to unseen sequence lengths.</li></ul><h3>What’s Next</h3><p>Today wraps up Chapter — 3 of the Hands On LLM book , i will continue with AI engineering by chip huyen book chapter — 2 , i have some parts of it which are left such as mamba and jamba models and SSMs.</p><h3>Quick note before I go:</h3><p><em>I’ve recently started sharing my learnings in public, and i dont use ChatGPT or other AI tools to write (not that i hate them) but because I want these reflections to stay personal and true to what I’m actually picking up each day</em></p><p><em>Also , i maintain a notebook where i practise all the codes if you are interested in that you can follow my github notebook link</em></p><p><a href="https://github.com/akshatkhatri/Hands_on_LLM">Notebook Link</a></p><p><a href="https://github.com/akshatkhatri">Github Profile Link</a></p><p><em>Feel Free to connect with me on Linkedin</em></p><p><a href="https://www.linkedin.com/in/i-am-akshat-khatri/">Linkedin Profile Link</a></p><p><strong><em>Thanks for reading — means a lot :)</em></strong><em><br>Best,<br>Akshat Khatri</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ceeb41a82893" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ensemble Boosting and Stacking : Adaboost,Gradient Boosting and HGB]]></title>
            <link>https://medium.com/@akshat.dev/ensemble-boosting-and-stacking-adaboost-gradient-boosting-and-hgb-eb25bde5e516?source=rss-cb3d13125e1c------2</link>
            <guid isPermaLink="false">https://medium.com/p/eb25bde5e516</guid>
            <category><![CDATA[ensemble-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[adaboost]]></category>
            <category><![CDATA[matplotlib]]></category>
            <dc:creator><![CDATA[Akshat Khatri]]></dc:creator>
            <pubDate>Sat, 26 Jul 2025 10:49:21 GMT</pubDate>
            <atom:updated>2025-07-26T10:49:21.518Z</atom:updated>
            <content:encoded><![CDATA[<h3>Ensemble Boosting and Stacking : Adaboost,Gradient Boosting and HGB</h3><p><em>Today marks Day 17 of my public ML learning journey. I’m using Aurélien Géron’s Hands-On ML as my guidebook, I will be starting and wrapping up Chapter 7 part-2 (Ensemble Learning &amp; Boosting Techniques) Today..</em></p><p>In the previous article we discussed what is ensemble , how is Voting classified , Soft Voting and Hard voting , bagging and pasting , Out of Bag evaluation (OOB), Random patches and random subspaces , Random Forest, Extra Trees and Feature importance using random forests in the <a href="https://medium.com/@akshat.dev/understanding-ensemble-learning-and-random-forests-3beefc93d203"><em>previous article here</em></a><em>.</em></p><blockquote>In Today’s Article we are going to discuss about Boosting Techniques , Algorithms like AdaBoost , maths behind it , Gradient Boosting , How regularization is done in it , Histogram-based gradient boosting and then finally Stacking / Blender .</blockquote><h3>What is Boosting?</h3><p>Boosting is an ensemble method , i.e. it believes in crowd’s wisdom , specifically Boosting means to correct the mistakes of its predecessors , by training them one after the another . There are many boosting methods and we will discuss the most important ones.</p><h4>AdaBoost (Adaptive-Boosting) Algorithm</h4><p>Best way to learn from someone’s mistake , is to focus more on what they got wrong , and it is the motto of ada-boost .</p><p>Ada boost Focuses more on what its predecessor model got wrong , so that the next time the model is trained it learns to correct its mistakes , and fit well.</p><p>For example , Adaboost works on first the base estimator , say a decision tree for classification , wherever that decision tree made wrong decisions , adaboost raises the weight of that data point , effectively saying to the tree to focus more on this data instance next time , this continues until the difference is ≤ tol , or n_estimators threhold has been reached.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/851/1*ftEYe3z7dW-VM42TuBRptQ.png" /><figcaption>Code in Notebook</figcaption></figure><p>As we can see similar to Gradient Descent , giving too big Learning Rate will make it overshoot. Note in gradient Descent we tweak parameters until we get it right , here we tweak incorrect prediction weights, and train ensemble models until we get it right.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/180/0*IuQbOgkoLKv0jDBE" /></figure><p>This is the weighted error rate of the jth predictor [Classification Task]</p><ul><li>where wi is the weight of the i th data point</li><li>y hat (j i ) is the j th model’s prediction for the i th data point</li><li>y ^ i is the original label of the data point.</li></ul><p>In summary for each predictor for each wrong prediction multiply its value by the weight levied upon the predictor , and then sum up all such errors to get finally the weighted error rate of the predictor.</p><p>Then we compute the weight to give each predictor using the hyperparameter (n) , which defaults to 1 .If the predictor is accurate higher will its weight be , if its just guessing randomly its weight will be zero , and if it is performing worse than random its weight will be negative.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/1*qaBZL1e8pOWmB-4nRkLb7A.png" /><figcaption>eta (’n’) is also called as the learning rate hyper-paramter</figcaption></figure><p>For wrong classification weight is updates as —</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/183/1*14BNa_5d2lT5ObmGAaNA3Q.png" /><figcaption>alpha j is still the learning rate</figcaption></figure><p>Note that for correct classification wi = wi , no changes in weights are observed , but mathematically as we increase the wrong classification weights , and then normalize all weights automitcally increasing one decreases the other , so the model starts focusing more on the wrong classifications.</p><p>Normalization is necceasry or the weigths may explode or shrink too hard.</p><p>Finally a new predictor is trained using the updated weights and this process goes on and on , we discussed stopping conditions earlier.</p><p>Then finally Ada Boost makes up the predictions by generating predictions from each predictor , and then calculating their weighted sum and siding with the majority.</p><p>model_prediction = w1.m1 + w2.m3 + w3.m3 ..... wn . mn</p><h4>Scikit-Learn Implementation of AdaBoost</h4><pre>from sklearn.ensemble import AdaBoostClassifier<br><br>abc_clf = AdaBoostClassifier(DecisionTreeClassifier(max_depth=1),n_estimators=50,<br>learning_rate= 0.5,random_state=42)<br>abc_clf.fit(X_train,Y_train)</pre><p>Note that we 50 Decision Stumps as the Base estimator , you might ask why ? why not something stronger like random forest , Because AdaBoost is designed to work best with weak learners. Random Forest is already a strong learner.</p><p>Decision Stumps are Decision Trees with only 1 split at the root level.</p><blockquote>In case of Adaboost over-fitting the data , you can try using a weak learner like Decision stumps , reduce the n_estimators hyper-parameter or if it allows regularize the base-estimator.</blockquote><h3>Gradient-Boosting</h3><p>Another very popular Boosting algorithm is called Gradient Boosting , It works in a way similar to Ada-Boost that it corrects the mistakes of its predecessors , but with a unique difference.</p><blockquote>Ada-Boost was fiddling with weights of data points , Gradient-Boosting trains a new model on the residual errors only , and then ensembles the outcomes of all.</blockquote><p>Let’s Understand with a simple example , i will start by generating a dummy quadratic dataset</p><pre>import numpy as np<br>from sklearn.tree import DecisionTreeRegressor<br><br>np.random.seed(42)<br>X = np.random.rand(100,1) - 0.5<br>Y = 3 * X[:,0] ** 2 + 0.05 * np.random.randn(100)</pre><p>Then to demonstrate , i will fit a decision tree with depth 2 on it .</p><pre>tree_reg_1 = DecisionTreeRegressor(max_depth=2,random_state=42)<br>tree_reg_1.fit(X,Y)</pre><p>Then i will train another decision tree on the data , in which the first decision tree made errors , and then repeat this process across 3 iterations.</p><pre>y2 = Y - tree_reg_1.predict(X)<br>tree_reg_2 = DecisionTreeRegressor(max_depth=2,random_state=43)<br>tree_reg_2.fit(X,y2)<br><br>y3 = y2 - tree_reg_2.predict(X)<br>tree_reg_3 = DecisionTreeRegressor(max_depth=2,random_state=44)<br>tree_reg_3.fit(X,y3)</pre><pre>fig, axes = plt.subplots(3, 2, figsize=(14, 12))<br>fig.suptitle(&quot;Boosting with Decision Trees&quot;, fontsize=16, weight=&#39;bold&#39;)<br><br>X_sorted = np.sort(X, axis=0)<br><br>axes[0][0].scatter(X, Y, color=&#39;skyblue&#39;, marker=&#39;.&#39;, label=&quot;Actual&quot;)<br>Y_pred_sorted_1 = tree_reg_1.predict(X_sorted)<br>axes[0][0].plot(X_sorted, Y_pred_sorted_1, color=&#39;darkgreen&#39;, label=&quot;Tree 1 Prediction&quot;)<br>axes[0][0].set_title(&quot;Tree 1 Fit on Original Data&quot;)<br>axes[0][0].legend()<br><br>axes[1][0].scatter(X, y2, color=&#39;black&#39;, marker=&#39;+&#39;, label=&quot;Residual 1&quot;)<br>Y_pred_sorted_2 = tree_reg_2.predict(X_sorted)<br>axes[1][0].plot(X_sorted, Y_pred_sorted_2, color=&#39;orange&#39;, label=&quot;Tree 2 Prediction&quot;)<br>axes[1][0].set_title(&quot;Tree 2 Fit on Residuals from Tree 1&quot;)<br>axes[1][0].legend()<br><br>axes[2][0].scatter(X, y3, color=&#39;black&#39;, marker=&#39;+&#39;, label=&quot;Residual 2&quot;)<br>Y_pred_sorted_3 = tree_reg_3.predict(X_sorted)<br>axes[2][0].plot(X_sorted, Y_pred_sorted_3, color=&#39;purple&#39;, label=&quot;Tree 3 Prediction&quot;)<br>axes[2][0].set_title(&quot;Tree 3 Fit on Residuals from Tree 2&quot;)<br>axes[2][0].legend()<br><br>axes[0][1].scatter(X, Y, color=&#39;skyblue&#39;, marker=&#39;.&#39;, label=&quot;Actual&quot;)<br>axes[0][1].plot(X_sorted, Y_pred_sorted_1, color=&#39;darkgreen&#39;, label=&quot;Tree 1 Only&quot;)<br>axes[0][1].set_title(&quot;Prediction after Tree 1(Ensemble Learning Technique)&quot;)<br>axes[0][1].legend()<br><br>axes[1][1].scatter(X, Y, color=&#39;skyblue&#39;, marker=&#39;.&#39;, label=&quot;Actual&quot;)<br>combined_pred_2 = Y_pred_sorted_1 + Y_pred_sorted_2<br>axes[1][1].plot(X_sorted, combined_pred_2, color=&#39;crimson&#39;, label=&quot;Tree 1 + Tree 2&quot;)<br>axes[1][1].set_title(&quot;Prediction after Tree 1 + Tree 2&quot;)<br>axes[1][1].legend()<br><br>axes[2][1].scatter(X, Y, color=&#39;skyblue&#39;, marker=&#39;.&#39;, label=&quot;Actual&quot;,alpha = 1)<br>combined_pred_3 = combined_pred_2 + Y_pred_sorted_3<br>axes[2][1].plot(X_sorted, combined_pred_3, color=&#39;black&#39;, label=&quot;Tree 1 + Tree 2 + Tree 3&quot;)<br>axes[2][1].set_title(&quot;Final Boosted Prediction (3 Trees)&quot;)<br>axes[2][1].legend()<br><br>plt.tight_layout(rect=[0, 0, 1, 0.96])<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7h7oMKCYrwpLHkh8aou-sg.png" /><figcaption>On the Left is Each Trees individual Data Line , on the right is how well that line fits on original data</figcaption></figure><p>This Plot will make Things easier for us , Let’s Understand step by step —</p><ul><li>The first-tree is fitted on the original data , so it’s the same on left and right.</li><li>Tree 2 is trained on residual errors of Tree 1(see left) , Tree 2 performs a little better on the original data on the right.</li><li>Tree 3 is trained on residual errors of Tree 2 , which is indeed trained on residual errors of Tree 1 (On the left) , Look at the top right side of the right-bottom graph compare it with the graph above it , you will find the line fits better than tree 2.</li></ul><p><strong>As More trees are ensemble predictions will most likely improve , but WatchOut ! This can risk over-fitting if not done properly , for now we have limited max_depth to 2 , a Simple decision tree with default hyper-parameters would have definitely over-fitted</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jlyUNd_BAF9jHR8IPhwVtg.png" /><figcaption>If i set max_depth to 7 , The Decision Tree Starts Over-fitting the dataset , adding more trees in a case like this is making things difficult for you in the future than now.</figcaption></figure><p>Another Way of dealing with Over-fitting is by tweaking the learning rate , a learning rate hyper-parameter in the case of boosting algorithms is called <em>shrinkage (regularization Technique).</em></p><p><strong>Shrinkage</strong> refers to multiplying the contribution of each new weak learner by a small value called the <strong>learning rate (α)</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/802/1*8HIbxYYkKm2B2DND52znZg.png" /></figure><p>by Default Learning rate is 1 .</p><p>Below Visualization would help us understand better , effect of n_estimators and learning rate on data fitting.</p><pre>fig, axes = plt.subplots(2, 2, figsize=(12, 8))<br>fig.suptitle(&quot;Gradient Boosting: Effect of number of estimators and Learning Rate&quot;, fontsize=16, weight=&#39;bold&#39;)<br><br>X_sorted = np.sort(X, axis=0)<br><br>axes[0][0].scatter(X, Y, color=&#39;#e74c3c&#39;, marker=&#39;+&#39;, label=&quot;Training Data&quot;)<br>axes[0][0].plot(X_sorted, gbrt_1.predict(X_sorted), color=&#39;#2ecc71&#39;, label=&quot;Prediction&quot;)<br>axes[0][0].set_title(&quot;n_estimators=3, learning_rate=1.0&quot;)<br>axes[0][0].set_xlabel(&quot;X&quot;)<br>axes[0][0].set_ylabel(&quot;Y&quot;)<br>axes[0][0].legend()<br><br>axes[0][1].scatter(X, Y, color=&#39;#e74c3c&#39;, marker=&#39;+&#39;, label=&quot;Training Data&quot;)<br>axes[0][1].plot(X_sorted, gbrt_2.predict(X_sorted), color=&#39;#2ecc71&#39;, label=&quot;Prediction&quot;)<br>axes[0][1].set_title(&quot;n_estimators=92, learning_rate=1.0&quot;)<br>axes[0][1].set_xlabel(&quot;X&quot;)<br>axes[0][1].set_ylabel(&quot;Y&quot;)<br>axes[0][1].legend()<br><br>axes[1][0].scatter(X, Y, color=&#39;#e74c3c&#39;, marker=&#39;+&#39;, label=&quot;Training Data&quot;)<br>axes[1][0].plot(X_sorted, gbrt_3.predict(X_sorted), color=&#39;#2ecc71&#39;, label=&quot;Prediction&quot;)<br>axes[1][0].set_title(&quot;n_estimators=92, learning_rate=0.05&quot;)<br>axes[1][0].set_xlabel(&quot;X&quot;)<br>axes[1][0].set_ylabel(&quot;Y&quot;)<br>axes[1][0].legend()<br><br>axes[1][1].scatter(X, Y, color=&#39;#e74c3c&#39;, marker=&#39;+&#39;, label=&quot;Training Data&quot;)<br>axes[1][1].plot(X_sorted, gbrt_4.predict(X_sorted), color=&#39;#2ecc71&#39;, label=&quot;Prediction&quot;)<br>axes[1][1].set_title(&quot;n_estimators=3, learning_rate=0.05&quot;)<br>axes[1][1].set_xlabel(&quot;X&quot;)<br>axes[1][1].set_ylabel(&quot;Y&quot;)<br>axes[1][1].legend()<br><br>plt.tight_layout(rect=[0, 0, 1, 0.95])<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l2nqmnBD1AVDfqix6WC0Lw.png" /></figure><p>Let’s Pull some insights —</p><ul><li>The 1st graph at [0,0] , has less number of estimators (3) , but each estimator contributes fully as learning rate is 1 , outcome is mildly under-fitting.</li><li>The 2nd Graph at [0,1] has high learning rate (1) , and high number of estimators , outcome is Greatly over-fitting.</li><li>The 3rd graph at [1,0] has high number of estimators (92) , but less learning rate (0.05) , it models the data well , and is the most balanced out of the 4.</li><li>The 4th Graph at [1,1] has both less estimators and less learning rate , result is it is wildly under-fitting the data.</li></ul><p>The question is how to find the optimal number of trees then , we can perform cross_validation or Grid Search CV , but there is another way , it involves around the concept we saw earlier about early stopping</p><p>The answer to it is the hyper-paramter N_iter_no_change it takes an integer value say 5 , and then stops when the last 5 trees haven’t made any substantial difference , usually that difference is controlled by tol hyper-parameter and is by default 10^(-5) = 0.0001 .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*_rz_eiZY9otY1qJp5zEEGg.png" /></figure><h4>Histogram Based Gradient-Boosting</h4><p>Histogram-Based Gradient Boosting (HGB) is a faster and more memory-efficient version of traditional gradient boosting for decision trees. It’s used in high-performance libraries like <strong>LightGBM</strong> and now also in <strong>scikit-learn</strong></p><h4>Key Idea:</h4><p>Instead of splitting on every possible feature value like traditional decision trees, HGB:</p><ol><li>Bins continuous features into discrete intervals (like histograms)</li><li>Uses the bin borders (not exact feature values) to find splits</li><li>This reduces computation from O(n_features × n_samples) to O(n_features × n_bins)</li></ol><pre>from sklearn.ensemble import HistGradientBoostingClassifier<br><br>model = HistGradientBoostingClassifier(max_iter=100, learning_rate=0.1)<br>model.fit(X_train, y_train)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HkSreUm9p6Spj4gMpJ-zHg.png" /></figure><p>Rest all its working is same as <em>Gradient-Boosting</em> we discussed.</p><h3>Stacking</h3><p>The most effective but a little complex , competition winning ensemble method is stacking or some might call it a blender.</p><p>It replaces voting mechanisms we used earlier like soft-voting &amp; Hard-Voting but replaces it with a model that predicts labels with input as predictions from the base estimators .</p><p>To understand it better , let’s say that when our n base estimators says something both hard voting and soft-voting follow a metric and upon that metric it gives the final result , but in stacking we build a model on top of those predictions , it helps us to capture complex relationships in the data and generalize well.</p><p>The model which makes the final predictions is called a meta-Learner.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Nc1ZxS5z6ZnBo2tVwpnuow.png" /></figure><p>For blender , we first need to build a training set , this can be done by using cross_val_predict() to generate out of sample predictions from each base_model and then aggregating that information and passing to the meta model . Or you can seperate test, train and validation set using the classic approach . Choice is yours .</p><p>Once you have the predictions the meta-learner will use them to model , and the final label will be the original Y_train.</p><p>You can stack as much as you want for example you can have multiple blenders and a blender on top of another blender , this will increase model complexity and will take time , but can help with squeezing model’s performance to its realistic limit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*emtOFL2YeDdRBWLTdhTrIQ.png" /><figcaption>Predictions in a multi-layered Stacking Ensemble using outof fold predictions.</figcaption></figure><p>Scikit-Learn uses a Stacking_classifier class for this implementation</p><pre>from sklearn.ensemble import StackingClassifier<br><br>stacking_clf = StackingClassifier(<br>    estimators=[<br>        (&#39;lr&#39;,LogisticRegression(random_state=42)),<br>        (&#39;rf&#39;,RandomForestClassifier(random_state=42)),<br>        (&#39;svc&#39;,SVC(random_state=42))<br>    ],<br>    final_estimator=RandomForestClassifier(random_state=43),<br>    cv = 5<br>)<br>stacking_clf.fit(X_train,Y_train)</pre><blockquote>You can check out the code in the notebook , its too big to paste here , on the make_moons dataset each individual classifier was averaging a score of 94–95 % , but with stacking and training a blender , the score shot up to 97.7 % , a brilliant 2.7 % increase in score.</blockquote><p>Also , The exercise solutions are given in my notebook , in a seperate md file , you can check that out too , if you want , link is at the end of article.</p><h3>What’s Next</h3><p>Now this chapter is complete , i will start with Chapter — 8 (PCA) next time</p><h3>Quick note before I go:</h3><p><em>I’ve recently started sharing my learnings in public, and i dont use ChatGPT or other AI tools to write (not that i hate them) but because I want these reflections to stay personal and true to what I’m actually picking up each day</em></p><p><em>Also , i maintain a notebook where i practise all the codes if you are interested in that you can follow my github notebook link</em></p><p><a href="https://github.com/akshatkhatri/Machine_learning_playground/tree/main">Notebook Link</a></p><p><a href="https://github.com/akshatkhatri">Github Profile Link</a></p><p><em>Feel Free to connect with me on Linkedin</em></p><p><a href="https://www.linkedin.com/in/i-am-akshat-khatri/">Linkedin Profile Link</a></p><p><strong><em>Thanks for reading — means a lot :)</em></strong><em><br>Best,<br>Akshat Khatri</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eb25bde5e516" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ensemble Learning and Random Forests: Concepts and Code]]></title>
            <link>https://medium.com/@akshat.dev/understanding-ensemble-learning-and-random-forests-3beefc93d203?source=rss-cb3d13125e1c------2</link>
            <guid isPermaLink="false">https://medium.com/p/3beefc93d203</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[supervised-learning]]></category>
            <category><![CDATA[feature-engineering]]></category>
            <category><![CDATA[random-forest]]></category>
            <category><![CDATA[ensemble-learning]]></category>
            <dc:creator><![CDATA[Akshat Khatri]]></dc:creator>
            <pubDate>Thu, 24 Jul 2025 11:28:14 GMT</pubDate>
            <atom:updated>2025-07-24T11:36:28.819Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Today marks Day 15 &amp; 16 of my public ML learning journey. I’m using Aurélien Géron’s Hands-On ML as my guidebook, I will be starting and wrapping up Chapter 7 (Ensemble Learning And Random Forests) Today..</em></p><p>In the previous article we discussed about <em>how to train,visualize and predict using Decision trees , loss functions used , CART training algorithm , and how to regularize trees to prevent over-fitting to a task </em><a href="https://medium.com/@akshat.dev/understanding-decision-trees-in-ml-c12fe50d96ee"><em>here</em></a></p><h3>What is ensemble ? why is it needed?</h3><p>Ensemble in simple terms is a voting mechanism, employed by predictors to get better predictions , as one model is likely to fail or wrongly predict in some cases , a number of models might not , this is also called as the wisdom of the crowd . A group of predictors is called an ensemble , technique is called ensemble technique , and an ensemble learning algorithm is called ensemble method.</p><p>To give a taste of ensemble , visualize many decision trees , with each decision tree trained on subsets of data , and then combine their predictions , and boom you have a pseudo-random-forest.</p><p>Also most of the machine Learning competiton winning ideas use ensemble methods , as they generally result in better preds.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*1-pxjhRcI5HDeELmU81Q5g.png" /><figcaption>Ensemble Learning Technique</figcaption></figure><h3>Voting Classifier Techniques</h3><h4>Hard-Voting</h4><p>Suppose we train different models , lets say a logistic Regression , an SVC , an K-means classifier and a few more .</p><p>The simplest way of making ensemble learning technique from these , is to use hard-voting , it means that we will use all the classifier to generate a prediction and select the one which has the majority , for example if out of 5 classifiers 3 predicted 1 , and 2 predicted 0 , The final classification will be True (1).</p><p>In case of a tie , the first prediction is chosen to be the final prediction.</p><p>There are various Drawbacks to Hard Voting —</p><ol><li>Majority isn’t always right , sometimes a smarter model (98 % No) can be overshadowed by Less-confident Yes-models (51 % Yes 3 each) .</li><li>Tie-breaking mechanism isn’t pretty , this can result in bad predictions.</li></ol><blockquote>Ensemble methods Perform well when they behave independently , this can be achieved by two ways , either select random subsets of data to train on and/or Use different machine learning algorithms , as two different algorithms are unlikely to make the same mistake</blockquote><h4>Soft-Voting</h4><p>Soft — Voting on the other hand relies less on the majority vote , and instead relies on how more or less confident the estimator is about the class , generally you get this idea by using predict_proba() method.</p><p>After getting each Prediction , Soft-voting calculates the sum of probabilties for each class , normalizes it , and then predicts the class with highest probability .</p><p>Falling back to our previous example if one classifier is 98 % sure it is No , and other two are 51 % sure each that it is Yes , Hard Voting would give final prediction as Yes , But soft-voting would not.</p><p>Predictor 1 — Yes Probability = 0.98 , No Probability = 0.2</p><p>Predictor 2 — Yes Probability =0.51 , No probability = 0.49</p><p>Predictor3 — Yes Probability = 0.52 , No probability = 0.48</p><p>Sum of Yes = 2.00 , Sum of No = 1.49</p><p>So clearly Yes has a higher probability , so YES is the final outcome.</p><p>Note that for soft-voting to work , the estimators should expose a predict_proba() method , which is not enabled by default in the SVM , so for soft-voting to work when SVM is involved we have to set probability hyper-parameter to true in the SVM.</p><pre>from sklearn.ensemble import VotingClassifier<br>voting_clf = VotingClassifier(<br>    estimators=[<br>        (&#39;lr&#39;,LogisticRegression(random_state=42)),<br>        (&#39;rf&#39;,RandomForestClassifier(random_state=42)),<br>        (&#39;svc&#39;,SVC(random_state=42,probability=True))<br>    ]<br>    ,voting = &#39;soft&#39;<br>)<br><br>voting_clf.fit(X_train,Y_train)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/1*RV0gcYqWsdSJfRitDjIzrA.png" /><figcaption>Training a Voting Classifier Using Scikit-Learn</figcaption></figure><h3>Bagging and Pasting : Sampling Techniques</h3><p>The way we discussed till now , involved using different training algorithms on the same data , but what if we use different subsets of data too , this is called sampling of the data . i.e. we obtain a small subset of the data which ≤ m (Total Data points).</p><blockquote>Bagging also called as bootstrap aggregating when sampling is done with replacement.<br>Pasting is sampling without replacement.</blockquote><p>To understand bootstrap aggregating (bagging) let’s take an example that you are asked to draw 10 cards from a deck of 52 cards , when you draw one card , your chosen card is noted down and put in the deck again , the deck is then shuffled and you are asked to choose again .</p><p>On the contrary in pasting , when you choose a card it is removed from the deck , and noted down , so no repetition takes place.</p><p>Perhaps the most Important feature of bagging and pasting is that , they can be done in parallel in an ensemble architecture . Sampling is independent of each other , thus different models can be trained on different machines decreasing compute time.</p><p><strong>Both bagging and pasting scale really well</strong></p><h4>Bagging and Pasting in scikit-Learn</h4><p>Scikit-learn uses a simple interface to do bagging and pasting , these can be accessed from the module sklearn.ensemble import BaggingClassifier , if you want scikit-learn to use bagging set bootstrap = True, and if you want to use pasting set bootstrap = False. the n_jobs hyper-parameter is used to indicate how many cores in local machine to be used for training (As bagging and pasting can be parallelized).</p><pre>from sklearn.ensemble import BaggingClassifier<br>from sklearn.tree import DecisionTreeClassifier<br><br>bag_clf = BaggingClassifier(DecisionTreeClassifier(),n_estimators=100,n_jobs=-1,random_state=42,oob_score=True)<br>bag_clf.fit(X_train,Y_train)<br>dec_tree = DecisionTreeClassifier(random_state=42)<br>dec_tree.fit(X_train,Y_train)</pre><p>As you can see in the above code , first we trained a bagging classifier with DecisionTreeClasifier as the base estimator and number of trees = 100 , we will discuss about oob more in this article.</p><h4>Plotting Decision Boundary for both Bagging classifier and Decision Tree</h4><pre>import numpy as np<br>import matplotlib.pyplot as plt<br><br>def plot_decision_boundary(model, X_train, Y_train, feature_names=(&quot;Feature 1&quot;, &quot;Feature 2&quot;), title=&quot;Decision Boundary&quot;,self_plot = True):<br>    x_min, x_max = X_train[:, 0].min(), X_train[:, 0].max()<br>    y_min, y_max = X_train[:, 1].min(), X_train[:, 1].max()<br>    X1 = np.linspace(x_min, x_max, num=100)<br>    X2 = np.linspace(y_min, y_max, num=100)<br>    X_grid, Y_grid = np.meshgrid(X1, X2)<br>    grid = np.c_[X_grid.ravel(), Y_grid.ravel()]<br>    <br>    Z = model.predict(grid).reshape(X_grid.shape)<br>    plt.contourf(X_grid, Y_grid, Z, alpha=0.3, cmap=&#39;cool&#39;)<br>    <br>    class_0 = Y_train == 0<br>    class_1 = Y_train == 1<br>    <br>    plt.scatter(X_train[class_0, 0], X_train[class_0, 1],<br>                color=&#39;blue&#39;, edgecolor=&#39;k&#39;, marker=&#39;o&#39;, label=&#39;Class 0&#39;, alpha=0.7)<br>    plt.scatter(X_train[class_1, 0], X_train[class_1, 1],<br>                color=&#39;red&#39;, edgecolor=&#39;k&#39;, marker=&#39;^&#39;, label=&#39;Class 1&#39;, alpha=0.7)<br><br>    plt.xlabel(feature_names[0])<br>    plt.ylabel(feature_names[1])<br>    plt.title(title)<br>    plt.legend()<br>    if self_plot:<br>        plt.show()<br><br>fig, axes = plt.subplots(1, 2, figsize=(12, 5))<br>plt.sca(axes[0]) <br>plot_decision_boundary(bag_clf, X_train, Y_train, title=&quot;Decision Tree with Bagging&quot;,self_plot=False)<br>plt.sca(axes[1]) <br>plot_decision_boundary(dec_tree, X_train, Y_train, title=&quot;Simple Decision Tree&quot;,self_plot=False)<br>plt.tight_layout()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KhXsJU0GAFOEhKCrEpR1DQ.png" /><figcaption>Decision Boundary Visualization</figcaption></figure><h4>Insights from The Decision Boundaries</h4><ul><li>Bagging’s Decision Tree increases the bias (As it does not see all the training data) , for a decrease in variance , and will likely generalize better.</li><li>Decision Tree has overfitted the data , and has very low bias , but will most likely perform worse on the test data (high variance).</li></ul><h3>Out-Of-Bag Evaluation</h3><p>With Bagging , Some training instances may be sampled and some training instances be not sampled at all , By default the BaggingClassifier samples ‘m’ training instances . With Matematics it can be shown , that when sampled randomly with replacement , only about 67 % of the data will be sampled , the remaining 33 % will not be sampled at all.</p><p>Hence this 33 % of the data was never seen by the predictor , Rings a bell? , This is called cross-validation if we would have intentionally left out that data , but due to stochastic nature of sampling this 33 % of data for a predictor works as a validation set , and can be used for measuring the accuracy.</p><p>Once we have a accuracy for each ensemble base predictor using the OOB validation set , average out all the scores you received , thats your bagging classifier’s score.</p><p>In scikit-Learn we can achieve oob_score by setting `oob_score_` Hyperparameter to True.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/473/1*EuKW-Gz9DFaO4nEXbUXzDQ.png" /></figure><p>hmmm. the oob_score_ came to be pessimistic , around 1 % less than test_data accuracy , still oob provides a good metric without needing seperate test_data.</p><h3>Random Patches And Random Subspaces</h3><p>As we have seen till now Bagging Classifier supports sampling of data points , but what about features? . Yes , Bagging Classifier also supports sampling of features .</p><p>Sampling of features is controlled by two hyper-parameters : max_features and bootstrap_features , same as max_samples and bootstrap hyper-parameters in data sampling.</p><p>Feature sampling is extremely useful when we have high number of features around (50+) , for example image.</p><blockquote>Random Patches is when we sample both training instances and features.</blockquote><blockquote>Random Subspaces is when we keep all training instances but only sample features.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xV_dFEcJCGs2avgfYMQLSA.png" /><figcaption>Comparison Table of Bagging,Pasting , Random Subspaces and random pattches</figcaption></figure><h3>Random Forests</h3><p>We have seen BaggingClassifier pasting , bagging e.t.c , One can argue that Random FOrest is just Bagging classifier with estimator as Decision Tree and perform random patches , on paper Yes.</p><p>But in reality , RandomForestClassifier Class is much more optimized to handle it , with a few exceptions Random Forest Classifier has all the hyper-parameters of the BaggingClassifier.</p><p>The random-forest as the name suggests is randomized behavior in decision Trees , instead of searching for the best feature as it is in the case of Decision Trees , Random Forest samples features and look in it the best threshold and feature value it should cut.</p><p>By default random forest samples root(n) features. The algorithm generalizes well again trading high bias for low variance.</p><pre>from sklearn.ensemble import RandomForestClassifier<br>rnd_clf = RandomForestClassifier(n_estimators=500,max_leaf_nodes = 16,n_jobs= -1, random_state=42)<br>rnd_clf.fit(X_train,Y_train)<br>y_pred_rf = rnd_clf.predict(X_test)</pre><p>This is the same as writing</p><pre>bag_clf = BaggingClassifier(<br>  DecisionTreeClassifier(max_features = &#39;sqrt&#39;,max_leaf_nodes = 16),<br>  n_estimators = 100,n_jobs = -1,frrandom_state = 42)</pre><p>But Random-Forest Classifier is more optimized and preferred</p><h4>Extra-Trees OR extremely Randomized Trees Forest(ERTF)</h4><p>Extra-Trees stands for Extremely Randomized Trees , an ensemble method similar to Random Forests but with more randomness injected to reduce variance.</p><p>Instead of Random-Forests which still do the splitting according to the greedy method , searching for the best way to split.</p><p>Extra-Trees select only a random subset of the splitter , and finds the best split from them, A forest of such Random Trees is called Extemely randomized Trees Forest (ERTF) .</p><p>Once again , this technique trades higher bias for lower variance, also it is faster to train than regular Random Forest Classifier due to its stochastic nature , because the most time-consuming task is Splitting in random-forests or decision Trees , extra-trees help reduce this time.</p><pre>from sklearn.ensemble import ExtraTreesClassifier<br><br>clf = ExtraTreesClassifier(<br>    n_estimators=100,<br>    max_features=&#39;sqrt&#39;,<br>    bootstrap=False,          <br>    random_state=42<br>)<br><br>clf.fit(X_train, y_train)<br>y_pred = clf.predict(X_test)</pre><h3>Feature Importance Using Random Forests</h3><p>Feature importance in Random Forests helps identify which input features matter most for making predictions. As Random Forests build multiple decision trees, they track how much each feature reduces uncertainty (or impurity) when used for splitting. The more a feature helps in making better splits across the forest, the more important it’s considered.</p><p>These importance scores are averaged and normalized, giving a ranked list of features. This is useful for understanding your model, simplifying it by removing less important inputs, and improving performance.</p><p>That said, the method isn’t perfect — it can favor features with more levels or numeric values, and it reflects only what trees can learn. Still, it’s a simple and effective way to get insights into how your model thinks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/421/1*itiWqSxCSFuH2qYTctGI8g.png" /></figure><pre>from sklearn.datasets import fetch_openml<br><br>X_mnist, y_mnist = fetch_openml(&#39;mnist_784&#39;, return_X_y=True, as_frame=False,<br>                                parser=&#39;auto&#39;)<br><br>rnd_clf = RandomForestClassifier(n_estimators=100, random_state=42)<br>rnd_clf.fit(X_mnist, y_mnist)<br><br>heatmap_image = rnd_clf.feature_importances_.reshape(28, 28)<br>plt.imshow(heatmap_image, cmap=&quot;hot&quot;)<br>cbar = plt.colorbar(ticks=[rnd_clf.feature_importances_.min(),<br>                           rnd_clf.feature_importances_.max()])<br>cbar.ax.set_yticklabels([&#39;Not important&#39;, &#39;Very important&#39;], fontsize=14)<br>plt.axis(&quot;off&quot;)<br>save_fig(&quot;mnist_feature_importance_plot&quot;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/525/1*5szrJ6t2T4i-x-3Ircw4Fw.png" /><figcaption>Heatmap of MNIST pixel Importance</figcaption></figure><p>Random Forests are extremely Useful when it comes to Feature-selections i.e. Deciding which features actually have an impact on the final label .</p><h3>What’s Next</h3><p><em>Tomorrow i will Complete this chapter’s exercise and Part2 , and then start with Chapter — 8 PCA (Principal Components analysis).</em></p><h3>Quick note before I go:</h3><p><em>I’ve recently started sharing my learnings in public, and i dont use ChatGPT or other AI tools to write (not that i hate them) but because I want these reflections to stay personal and true to what I’m actually picking up each day</em></p><p><em>Also , i maintain a notebook where i practise all the codes if you are interested in that you can follow my github notebook link</em></p><p><a href="https://github.com/akshatkhatri/Machine_learning_playground/tree/main">Notebook Link</a></p><p><a href="https://github.com/akshatkhatri">Github Profile Link</a></p><p><em>Feel Free to connect with me on Linkedin</em></p><p><a href="https://www.linkedin.com/in/i-am-akshat-khatri/">Linkedin Profile Link</a></p><p><strong><em>Thanks for reading — means a lot :)</em></strong><em><br>Best,<br>Akshat Khatri</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3beefc93d203" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>