<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by datafreak on Medium]]></title>
        <description><![CDATA[Stories by datafreak on Medium]]></description>
        <link>https://medium.com/@datafreakai?source=rss-71ae604d9df2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*8quobDZeCTxb3_snmhQOgg.jpeg</url>
            <title>Stories by datafreak on Medium</title>
            <link>https://medium.com/@datafreakai?source=rss-71ae604d9df2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 09:58:54 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@datafreakai/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How I Cut AI Agent Latency from 90s to 20s Using Python Threads]]></title>
            <link>https://medium.com/@datafreakai/how-i-cut-ai-agent-latency-from-90s-to-20s-using-python-threads-f61f459dfc8b?source=rss-71ae604d9df2------2</link>
            <guid isPermaLink="false">https://medium.com/p/f61f459dfc8b</guid>
            <dc:creator><![CDATA[datafreak]]></dc:creator>
            <pubDate>Mon, 16 Jun 2025 19:15:13 GMT</pubDate>
            <atom:updated>2025-06-17T08:31:08.936Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>How I transformed a sluggish AI agent into a performance powerhouse using Python’s parallelization capabilities</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hhx5TCySv1TCoDD5nPkuOg.png" /><figcaption><a href="https://www.boardinfinity.com/blog/multithreading-in-python/">source</a></figcaption></figure><p>Picture this: you’re building an AI agent that needs to make dozens of API calls, process complex data, and coordinate between multiple AI services like OpenAI and Google’s Gemini. These agents are becoming the backbone of modern applications, orchestrating everything from customer service bots to complex data processing pipelines.</p><p>AI agents today are incredibly sophisticated. They can use <strong>tool calling</strong> to interact with external APIs, leverage multiple language models for different tasks, and chain together complex workflows. OpenAI’s function calling allows agents to dynamically invoke external tools, while services like Gemini provide structured output generation that makes data processing more reliable.</p><p>But here’s the thing that keeps developers up at night: <strong>performance</strong>.</p><p>When your AI agent takes 90 seconds to complete a task that users expect in real-time, you’ve got a problem. And that’s exactly where I found myself during my internship, staring at performance metrics that made me question everything I thought I knew about efficient coding.</p><p>To maintain confidentiality, the article replaces core business logic and implementation specifics with abstracted classes and simulated logic — while preserving the performance story and technical takeaways.</p><h3>The Problem: When Sequential Processing Becomes Your Enemy</h3><p>Let me paint you a picture of the challenge I faced. I was working with an AI agent class — let’s call it IntelligentPipelineAgent — that took a <strong>single user input</strong> (like a JSON request or something like that) and used that input to process a <strong>list of items</strong> (e.g., different tasks, slots, or modules) through four distinct steps:</p><ol><li><strong>Data Retrieval</strong> — Fetch base information for each item</li><li><strong>Data Enhancement</strong> — Enrich the retrieved data with additional context</li><li><strong>Structured Processing</strong> — Format the enriched data into a usable structure</li><li><strong>Final Analysis</strong> — Generate global insights from the <strong>original input</strong>, not per item.</li></ol><p>And all of these steps essentially involved OpenAI calls and tool use based workflows (interaction with external APIs etc)</p><p>Each item had to go through Steps 1 to 3 <strong>sequentially</strong>, since each step depended on the previous one’s output. But the agent processed these items <strong>one by one</strong>, which meant:</p><ul><li>Item A: Retrieve → Enhance → Structure</li><li>Then Item B: Retrieve → Enhance → Structure</li><li>Then Item C, and so on…</li></ul><p>The final generate_insights() step was also done only <strong>after</strong> all items were processed — even though it only depended on the initial input and could’ve run independently.</p><p>Here’s how the method looked:</p><pre>class IntelligentPipelineAgent:<br>    def process_item_list(self, items, user_input):<br>        results = []<br>        for item in items:<br>            base_data = self.retrieve_data(item)<br>            enriched_data = self.enhance_data(base_data)<br>            structured_output = self.structure_data(enriched_data)<br>            results.append(structured_output)<br><br>        final_insights = self.generate_insights(user_input)<br>        return final_insights</pre><p>This design caused a serious bottleneck. With around <strong>20 seconds per item</strong> and 4 items in the list, the total time ballooned to <strong>~90 seconds</strong>.<br> The bulk of that wasn’t even processing — it was <strong>waiting</strong> on API calls: classic I/O-bound latency.</p><p>And that’s where I knew: parallelization was not just a nice-to-have — it was <strong>necessary</strong>.</p><h3>Understanding I/O Bound Operations: The Hidden Performance Killer</h3><p>Before we dive into the solution, let’s talk about something crucial that many developers overlook: I/O bound operations.</p><h4>What Makes a Process I/O Bound?</h4><p>An I/O bound process is one where the program spends most of its time waiting for input/output operations to complete, rather than actually computing. Think of it like this: your CPU is like a world-class chef, but if they’re constantly waiting for ingredients to be delivered, they can’t cook efficiently no matter how skilled they are.</p><p>In the context of AI agents, almost everything we do is I/O bound:</p><ul><li><strong>API calls to OpenAI</strong>: Your code sends a request and waits for the language model to process and respond</li><li><strong>Tool calling</strong>: When the agent invokes external APIs, it’s waiting for those services to respond</li><li><strong>Gemini API calls</strong>: Same story — send request, wait for structured output</li><li><strong>Database queries</strong>: Fetching or storing data involves waiting for disk operations</li><li><strong>File operations</strong>: Reading configuration files, saving results, etc.</li></ul><h4>Why This Matters for AI Agents</h4><p>Here’s the kicker: while your code is waiting for OpenAI to respond to one API call, your CPU is essentially sitting idle. It could be making other API calls, processing different items, or handling other tasks. But in a sequential approach, you’re forcing your program to wait for each operation to complete before starting the next one.</p><p>This is like having a restaurant where the chef makes one dish completely before even starting to prep the next order. Imagine the wait times!</p><h4>The Mathematical Reality</h4><p>Let’s break down what was happening in my agent:</p><p>3 sequential steps per item × 20 seconds = 80 seconds<br> 4 items × 20 seconds = 80 seconds total<br> Final analysis step: ~10 seconds</p><p><strong>Total execution time: ~90 seconds</strong><br> <strong>CPU utilization: ~5–10%</strong> — most of the time was spent waiting on I/O.</p><h3>The Sequential Approach: How We Started</h3><p>Let me show you what the original implementation looked like. This is a simplified version that captures the essence of what was happening:</p><pre>import time<br><br>class IntelligentPipelineAgent:<br>    def __init__(self):<br>        pass<br><br>    def retrieve_data(self, item_id):<br>        print(f&quot;Retrieving data for item {item_id}...&quot;)<br>        time.sleep(7)  #simulatiing openai call<br>        return {&quot;item_id&quot;: item_id, &quot;base_data&quot;: &quot;retrieved_info&quot;}<br><br>    def enhance_data(self, base_data):<br>        print(f&quot;Enhancing data for item {base_data[&#39;item_id&#39;]}...&quot;)<br>        time.sleep(6)  #simulatiing openai call<br>        return {**base_data, &quot;enhanced_data&quot;: &quot;enriched_info&quot;}<br><br>    def structure_data(self, enhanced_data):<br>        print(f&quot;Structuring data for item {enhanced_data[&#39;item_id&#39;]}...&quot;)<br>        time.sleep(7)  #simulatiing openai call<br>        return {**enhanced_data, &quot;structured_output&quot;: &quot;formatted_data&quot;}<br><br>    def generate_insights(self, user_input):<br>        print(&quot;Generating final insights based on input...&quot;)<br>        time.sleep(10)  #simulatiing openai call<br>        return f&quot;Insights generated for input: {user_input}&quot;<br><br>    def extract_items_from_input(self, user_input):<br>        return [&quot;task1&quot;, &quot;task2&quot;, &quot;task3&quot;, &quot;task4&quot;]<br><br>    def process_sequential(self, user_input):<br>        item_ids = self.extract_items_from_input(user_input)<br><br>        start_time = time.time()<br>        processed_items = []<br><br>        for item_id in item_ids:<br>            print(f&quot;\n--- Processing {item_id} ---&quot;)<br>            base_data = self.retrieve_data(item_id)<br>            enhanced_data = self.enhance_data(base_data)<br>            structured_data = self.structure_data(enhanced_data)<br>            processed_items.append(structured_data)<br><br>        final_insights = self.generate_insights(user_input)<br>        end_time = time.time()<br>        print(f&quot;\nSequential processing took: {end_time - start_time:.2f} seconds&quot;)<br>        return processed_items, final_insights<br><br># Example usage<br>if __name__ == &quot;__main__&quot;:<br>    agent = IntelligentPipelineAgent()<br>    user_request = {&quot;keys&quot; : &quot;fields&quot; }   # A Json Input <br>    processed, insights = agent.process_sequential(user_request)</pre><pre>--- Processing task1 ---<br>Retrieving data for item task1...<br>Enhancing data for item task1...<br>Structuring data for item task1...<br><br>--- Processing task2 ---<br>Retrieving data for item task2...<br>Enhancing data for item task2...<br>Structuring data for item task2...<br><br>--- Processing task3 ---<br>Retrieving data for item task3...<br>Enhancing data for item task3...<br>Structuring data for item task3...<br><br>--- Processing task4 ---<br>Retrieving data for item task4...<br>Enhancing data for item task4...<br>Structuring data for item task4...<br>Generating final insights based on input...<br><br>Sequential processing took: 90.08 seconds</pre><h4>What Was Wrong With the Sequential Approach?</h4><p>The problem is clear when you look at the execution flow:</p><ol><li><strong>Item 1</strong>: Retrieve (7s) → Enhance (6s) → Structure (7s) = 20 seconds</li><li><strong>Item 2</strong>: Retrieve (7s) → Enhance (6s) → Structure (7s) = 20 seconds</li><li><strong>Item 3</strong>: Retrieve (7s) → Enhance (6s) → Structure (7s) = 20 seconds</li><li><strong>Item 4</strong>: Retrieve (7s) → Enhance (6s) → Structure (7s) = 20 seconds</li><li><strong>Final insights</strong>: 10 seconds</li></ol><p>That’s over 90 seconds in total — most of it spent waiting on I/O-bound tasks like API responses.</p><p>But here’s the issue: while Item 1 is being retrieved, there’s no reason Items 2, 3, and 4 can’t also be in progress. Each item’s processing is independent. Running them one after another wastes time that could be saved by simply processing them in parallel.</p><h3>The Parallel Revolution: Threading to the Rescue</h3><p>After analyzing the bottleneck, I realized that while the steps for each individual item had to be sequential, I could process multiple items simultaneously. Here’s how I transformed the agent:</p><pre>import time<br>from concurrent.futures import ThreadPoolExecutor, as_completed<br><br>class IntelligentPipelineAgent:<br>    def __init__(self):<br>        pass<br><br>    def retrieve_data(self, item_id):<br>        print(f&quot;Retrieving data for item {item_id}...&quot;)<br>        time.sleep(7)  #simulatiing openai call<br>        return {&quot;item_id&quot;: item_id, &quot;base_data&quot;: &quot;retrieved_info&quot;}<br><br>    def enhance_data(self, base_data):<br>        print(f&quot;Enhancing data for item {base_data[&#39;item_id&#39;]}...&quot;)<br>        time.sleep(6)  #simulatiing openai call<br>        return {**base_data, &quot;enhanced_data&quot;: &quot;enriched_info&quot;}<br><br>    def structure_data(self, enhanced_data):<br>        print(f&quot;Structuring data for item {enhanced_data[&#39;item_id&#39;]}...&quot;)<br>        time.sleep(7)  #simulatiing openai call<br>        return {**enhanced_data, &quot;structured_output&quot;: &quot;formatted_data&quot;}<br><br>    def generate_insights(self, user_input):<br>        print(&quot;Generating final insights based on input...&quot;)<br>        time.sleep(10)  #simulatiing openai call<br>        return f&quot;Insights generated for input: {user_input}&quot;<br><br>    def extract_items_from_input(self, user_input):<br>        return [&quot;task1&quot;, &quot;task2&quot;, &quot;task3&quot;, &quot;task4&quot;]<br><br>    def process_item(self, item_id):<br>        base_data = self.retrieve_data(item_id)<br>        enhanced_data = self.enhance_data(base_data)<br>        structured_data = self.structure_data(enhanced_data)<br>        return structured_data<br><br>    def process_parallel_with_insights(self, user_input):<br>        item_ids = self.extract_items_from_input(user_input)<br>        start_time = time.time()<br><br>        with ThreadPoolExecutor(max_workers=len(item_ids) + 1) as executor:<br>            item_futures = [executor.submit(self.process_item, item_id) for item_id in item_ids]<br>            insight_future = executor.submit(self.generate_insights, user_input)<br><br>            processed_items = [f.result() for f in as_completed(item_futures)]<br>            final_insight = insight_future.result()<br><br>        end_time = time.time()<br>        print(f&quot;\nTotal parallel processing took: {end_time - start_time:.2f} seconds&quot;)<br>        return processed_items, final_insight<br><br># Example usage<br>if __name__ == &quot;__main__&quot;:<br>    agent = IntelligentPipelineAgent()<br>    user_request = {&quot;keys&quot; : &quot;fields&quot; }   # A Json Input<br>    processed, insights = agent.process_parallel_with_insights(user_request)</pre><p>Output:</p><pre>Retrieving data for item task1...<br>Retrieving data for item task2...<br>Retrieving data for item task3...<br>Retrieving data for item task4...<br>Generating final insights based on input...<br>Enhancing data for item task4...<br>Enhancing data for item task1...<br>Enhancing data for item task2...<br>Enhancing data for item task3...<br>Structuring data for item task2...<br>Structuring data for item task3...<br>Structuring data for item task1...<br>Structuring data for item task4...<br><br>Total parallel processing took: 20.02 seconds</pre><h4>How the Parallel Approach Fixed It</h4><p>With parallelization, the agent no longer waited for each item to finish before starting the next. Instead, all items were processed independently and <strong>simultaneously</strong>:</p><ul><li>Item 1 → Retrieve (7s) → Enhance (6s) → Structure (7s)</li><li>Item 2 → Retrieve (7s) → Enhance (6s) → Structure (7s)</li><li>Item 3 → Retrieve (7s) → Enhance (6s) → Structure (7s)</li><li>Item 4 → Retrieve (7s) → Enhance (6s) → Structure (7s)</li></ul><p>These steps for all 4 items ran in parallel, taking <strong>only ~20 seconds total</strong> (instead of 80), since they no longer blocked each other.</p><p>Meanwhile, the final generate_insights step — which was independent of the items — ran <strong>in parallel</strong> on its own thread, taking <strong>~10 seconds</strong>.</p><p><strong>Total time? Around 20 seconds.</strong><br> From ~90 seconds down to ~20 — just by leveraging multithreading for I/O-bound operations.</p><p>In the parallelized version of the AI agent pipeline, we tackled two major bottlenecks:</p><ol><li><strong>Per-item pipeline parallelization</strong></li></ol><ul><li>Instead of processing items one after another, we processed all items <strong>concurrently</strong> using Python’s ThreadPoolExecutor.</li><li>For each item, we still maintained the order: Retrieve → Enhance → Structure.</li><li>But each of these <em>pipelines</em> was run independently and concurrently across multiple threads.</li></ul><p><strong>2. Insights generation parallelization</strong>:</p><ul><li>The final step — generating insights — was also run in parallel, using a thread pool, since it was independent of the per-item pipelines (it only depended on the final outputs of them).</li></ul><p>Because all steps were I/O-bound (waiting on API responses), Python threads using concurrent.futures.ThreadPoolExecutor were ideal to reduce the total waiting time without the need for multiprocessing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Q8_T3p_J_ex65qdS" /><figcaption>pictorial representation of the approaches</figcaption></figure><h3>Conclusion: From Latency to Efficiency</h3><p>This project was a reminder that performance bottlenecks often aren’t about raw computation — they’re about <strong>how</strong> we structure our workflows. My AI agent was slow not because the logic was flawed, but because the architecture didn’t leverage concurrency for I/O-bound operations.</p><p>By analyzing the agent’s flow, I identified two key opportunities for optimization:</p><ul><li><strong>Per-item parallelization:</strong> Steps 1–3 (Retrieve → Enhance → Structure) needed to be sequential per item but not across items. So I used threading to process multiple items <strong>in parallel</strong>, reducing cumulative latency.</li><li><strong>Independent insight generation:</strong> The final generate_insights() step only relied on the original user input — not on the per-item results. This meant it could be triggered in parallel with the rest of the pipeline.</li></ul><p>With this restructuring, total execution time dropped from ~<strong>90 seconds </strong>to nearly <strong>20 seconds</strong> — a 3x speed-up with minimal change in logic. CPU utilization increased, users stopped waiting endlessly, and the system finally behaved like the intelligent assistant it was meant to be.</p><p>More broadly, this experience reinforced the value of thinking about <strong>where concurrency makes sense</strong> in I/O-heavy pipelines. Parallelism isn’t always about fancy multiprocessing — sometimes, it’s just about giving each thread something useful to do while others are waiting.</p><p>If you’re building intelligent agents, chatbots, or orchestration logic involving APIs — especially in Python — take a hard look at your execution graph. Chances are, there’s latency you can shave off just by letting your functions breathe in parallel.</p><p>One thing I didn’t explore in this version was asyncio. While threading worked well here, an async approach could further reduce overhead, especially if the underlying APIs support it natively (like httpx or aiohttp). That’s something I plan to experiment with next! And as the system scales to serve more users or tasks in parallel, revisiting the architecture with async, queues, or even distributed task runners might unlock the next level of performance.</p><h3>References</h3><ul><li><a href="https://www.boardinfinity.com/blog/multithreading-in-python/">Multithreading in Python | Board Infinity</a></li><li><a href="https://www.geeksforgeeks.org/python/multithreading-python-set-1/">Multithreading in Python - GeeksforGeeks</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f61f459dfc8b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Disks to Elevators: Applying Scheduling Algorithms for Optimal Movement]]></title>
            <link>https://medium.com/@datafreakai/from-disks-to-elevators-applying-scheduling-algorithms-for-optimal-movement-8784fa0ea9e8?source=rss-71ae604d9df2------2</link>
            <guid isPermaLink="false">https://medium.com/p/8784fa0ea9e8</guid>
            <category><![CDATA[operating-systems]]></category>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[low-level-programming]]></category>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[datafreak]]></dc:creator>
            <pubDate>Sat, 14 Sep 2024 08:43:49 GMT</pubDate>
            <atom:updated>2024-09-15T07:10:53.806Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/564/0*ZqP9GFwDBfIaFtc1.jpg" /><figcaption><a href="https://in.pinterest.com/pin/804385183460145704/">source</a></figcaption></figure><h3>Introduction</h3><p>Elevators are such an excellent invention of human civilization. Imagine with a press of a button, you are at the topmost floor of 50 story building in a matter of minutes. One time I was just using the elevator normally in my college building. Being an engineering student, I became very curious about how elevators cater to incoming requests on different floors. Like, suppose an Elevator is traveling upwards and is currently on the 8th floor, how is it managed if suddenly, the person on the 5th floor presses that button? I suddenly became obsessed with this fact and wanted to explore how elevators process these requests under the hood. So, as soon as I got back home I simply started researching it. And to my ‘shock’, I found that elevators are simply based on the mere algorithms used in DISK SCHEDULING in the operating system. Like “WHAT?” I was like how a low-level concept that happens inside an operating system is responsible for carrying persons from one floor to another floor inside a building.</p><p>This blog is an appreciation of the connection between low-level disk scheduling and how the elevators process the requests at different floors. I have delved into the logic behind these algorithms, and how these algorithms are connected to elevators, and also tried to simulate them using Python.</p><p>Let’s go!</p><h3>Wait, What is Disk Scheduling?</h3><p>To understand Disk scheduling, we have to understand the design behind a magnetic disk system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/0*wgo9t5aB8d2OXG-e.jpg" /><figcaption><a href="https://www.cs.uic.edu/~jbell/CourseNotes/OperatingSystems/10_MassStorage.html">magnetic disk system</a></figcaption></figure><p>A magnetic disk is a secondary storage system.</p><p>It has an arm assembly that holds the Read-Write head, a spindle that rotates continuously, and a cylinder that holds a set of platters. The Platters (these platters rotate continuously) are divided into concentric circles or “<strong>tracks</strong>.” Each track is divided into certain fixed-sized <strong>sectors </strong>and the data is stored in those sectors only.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*jAdiFwg-YKMd5wD_.jpg" /><figcaption><a href="https://www.geeksforgeeks.org/hard-disk-drive-hdd-secondary-memory/">source</a></figcaption></figure><p>Now the Read/Write head processes the data stored in the sectors(which are rotating because the platters are rotating) by doing to and fro motion along the radius of the platter/disk.</p><p>Each concentric circle within the platter (Track) is assigned a “track number” which tells us the proximity of that particular track from the spindle or the center of the platter/disk. Like Track-82 is nearer to the spindle than the Track-90 and so on.</p><p>Now each of these tracks can “request” to be accessed by the R/W head, to perform the Read or Write operation there. Say, Track-54 wants to access the R/W head so that an operation can be performed at that track. So, it will request for the R/W head access and once the requests are communicated to the R/W head, it reaches that track and performs that specific operation on that track.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*gXcBo4_CZ92Wm786.jpg" /><figcaption>R/W head (<a href="https://japp.io/category/algorithms/disk-scheduling/">source</a>)</figcaption></figure><p>However, in real-life scenarios, it is not THAT simple. Sometimes <strong>multiple</strong> tracks can request access to the R/W head. In such cases, the R/W head simply cannot just traverse toward a random track from the set of requests or follow a random sequence to cater to the requests and call it a day. At such low levels, latency and speed are huge factors. Small differences can cause large performance compromises. Therefore in such cases, we tend to employ special “strategies” to schedule the requests of different tracks so that the R/W head can cater to the requests in a systematic and <strong>EFFICIENT</strong> manner. These strategies are called<strong> Disk Scheduling algorithms (</strong>Will elaborate on them in later sections<strong>)</strong></p><p>Let me clarify what I mean by efficiency in this context. We will try to understand the meaning of efficiency by just two metrics <em>average seek time</em> and <em>distance</em> traveled (since these two metrics are more useful for making connections between low-level disk scheduling and elevator systems.)</p><p>So suppose a sample of requests in this sequence:</p><p><strong>[ 82,170,43,140,24,16,190] and currently the R/W head is at 50 (Track-50)</strong></p><p>So our main concern is that the R/W head caters each request in the given sequence in such a manner that <strong>the total movement</strong> it did is the <strong>least</strong> and <strong>the time taken to move around and handle the requests</strong> is also the <strong>least</strong>.</p><p>The total movement represents <strong>Distance </strong>metrics and the total time taken to handle the requests represents <strong>Average Seek Time</strong>. You should also appreciate the fact that the average seek time is directly proportional to the distance traveled by the R/W head. Now by these metrics, we can directly assess the performance of these I/O operations (Read and Write operations). So to maximize the overall performance of the storage system, we have <strong>designed algorithms</strong> that can <strong>find the optimal value of these metrics</strong> (in this case minimum values for <em>average seek time </em>and <em>total distance</em>). This is the main idea behind this.</p><p>But how all of these are connected to Elevators?</p><h3>Understanding the Connection</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/563/0*S5z6LmAexBIad8qX.jpg" /><figcaption><a href="https://in.pinterest.com/pin/980799625070262333/">source</a></figcaption></figure><p>To understand the connection between low-level disks and elevators, we have to define how the elevator systems work (Don’t worry, I won’t make it very complex)</p><p>An elevator works by moving a cabin vertically between floors in a building using a system of cables, pulleys, and a motor. When a user presses a button, the elevator’s control system determines the desired floor and directs the cabin to move either up or down.</p><p>Now this “button pressing” business can either be outside the elevator (on the floors) or inside the elevator. When we press that button (either outside or inside) what we are doing is requesting that the elevator to come to that floor.</p><p>For example, let’s say a person on the fifth floor of the building presses that button (either up or down). Once the button is pressed, what it is essentially doing is that it is requesting the elevator to arrive and stop at the 5th floor so that the person can enter the elevator. So, once this request is communicated to the elevator (via sensors or cables), the elevator tries to ‘cater’ that request by moving to the 5th floor and opening the doors for the person. This is the scenario of an “external” request.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/236/0*DGxXOiifQM_Xg-jz.jpg" /><figcaption><a href="https://in.pinterest.com/pin/1477812365156514/">source</a></figcaption></figure><p>Similarly, let’s say that the same person now enters the elevator and wants to go to the 9th floor. So, the person will press the button 9 inside the elevator. By this, the person is ‘requesting’ the elevator to go to the 9th floor. Thus, the elevator again ‘caters’ to that request by arriving at the 9th floor and stopping there. In this scenario of an ‘internal’ request, the person is requesting the lift by pressing the buttons inside the elevator.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/564/0*HpzwUiQ4EXmrmzrM.jpg" /><figcaption><a href="https://in.pinterest.com/pin/288441551147849481/">source</a></figcaption></figure><p>Also in real-life cases, multiple requests will arrive simultaneously, be it internal or external.</p><p>Do you see where I am heading? Try to connect the dots.</p><p>Let’s compare this to how a magnetic disk works. Just as a person on the 5th floor makes an external request, I/O operations from various tracks on a magnetic disk request the read/write head to come to their specific track. Similarly, once the read/write head is near one track, it handles any subsequent internal requests (other nearby tasks) before moving on to the next track, all while trying to minimize seek time. Here, the floor requests resemble the track requests and R/W head movement resembles the elevator movement.</p><p>The metrics- <em>average seek time</em> and <em>total distance</em> used to judge disks- can also be used to judge the performance of elevator systems as the type of work both systems do is fundamentally SAME. The R/W head moves in such a way that the average seek time and overall head movement are minimized, similarly, elevators also traverse through the building such that seek time to arrive at floors and the overall movement of the elevator in the building is minimized.</p><p>So we can safely deduce that the algorithms or strategies that are used in the magnetic disks are in fact the same strategies used by the elevators. These “Strategies” that I have been yapping about in the past sections are called <strong>DISK SCHEDULING ALGORITHMS</strong> and <strong>are indeed the same algorithms that are used in elevators to optimize the movement throughout the building.</strong> The most common Disk scheduling algorithms are:</p><ol><li><strong>FCFS (First Come First Serve)</strong></li><li><strong>SSTF (Shortest Seek Time First)</strong></li><li><strong>SCAN</strong></li><li><strong>LOOK</strong></li><li><strong>CSCAN (Circular SCAN)</strong></li><li><strong>CLOOK (Circular LOOK)</strong></li></ol><p>Let’s study them one by one and learn how they are applied in elevators!</p><h3>The Logic</h3><p>Imagine you’re in a building with an elevator that needs to visit certain floors to pick up and drop off passengers. The building has floors numbered from 0 to 10, and your elevator starts at floor 5. The floors where the elevator needs to stop are 1, 9, 4, 7, 3, and 6.</p><pre>#Sample request<br>requests = [1, 9, 4, 7, 3, 6]  #Floors to visit<br>initial_position = 5           #Current floor the elevator&#39;s at    <br>start = 0                      # The Ground floor <br>end = 10                       # The topmost foor</pre><p>To calculate and judge the performance(average seek time and total distance based on the path taken by the elevator according to that algorithm) in this sample simulation, we will be using the following logic:</p><pre>def calculate_metrics(path, start):<br>    total_distance = sum(abs(path[i] - path[i - 1]) for i in range(1, len(path)))<br>    avg_seek_time = total_distance / len(path)<br>    return total_distance, avg_seek_time</pre><h4>First Come, First Serve (FCFS)</h4><p>With the FCFS approach, the elevator will simply visit the floors in the order it received the requests. So, it will first stop at floor 1, then 9, followed by 4, 7, 3, and finally 6. This method is straightforward but doesn’t necessarily minimize the time spent moving between floors. This method ensures that each request is handled as soon as it arrives, but it doesn’t prioritize or optimize travel. As a result, the elevator may end up making long trips back and forth, which can lead to inefficiencies and longer wait times for passengers.</p><pre>def FCFS(requests, initial_position):<br>    path = [initial_position] + requests<br>    total_distance, avg_seek_time = calculate_metrics(path, initial_position)<br>    return path, total_distance, avg_seek_time</pre><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fz0Pwm71EDlY%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dz0Pwm71EDlY&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fz0Pwm71EDlY%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/d37a90bb6df6eeb891fd45672def1811/href">https://medium.com/media/d37a90bb6df6eeb891fd45672def1811/href</a></iframe><h4>SSTF (Shortest Seek Time First)</h4><p>Using SSTF, the elevator will always head to the nearest floor from its current position. Starting on floor 5, the nearest request is for floor 4, so the elevator moves down. After picking up at floor 4, it will then move to the closest request, which is floor 3. From there, it will move up to floor 6, then 7, then 9. SSTF aims to minimize the travel distance between consecutive stops, making it more efficient in terms of total distance and average seek time compared to FCFS. However, it can lead to “starvation” of requests that are far from the current position if there are many close requests.</p><pre>def SSTF(requests, initial_position):<br>    path = [initial_position]<br>    while requests:<br>        closest_request = min(requests, key=lambda x: abs(x - path[-1]))<br>        path.append(closest_request)<br>        requests.remove(closest_request)<br>    total_distance, avg_seek_time = calculate_metrics(path, initial_position)<br>    return path, total_distance, avg_seek_time</pre><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FwMUAzxac1Xw%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DwMUAzxac1Xw&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FwMUAzxac1Xw%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/ceb3cb5d0f81e18ddacd2d8cede69312/href">https://medium.com/media/ceb3cb5d0f81e18ddacd2d8cede69312/href</a></iframe><h4>SCAN (Elevator Algorithm)</h4><p>The SCAN algorithm is like the elevator moving in one direction until it reaches the end of its range and then reversing direction. Starting at floor 5 and moving upwards, it will stop at floors 6, 7, and 9. Once it reaches the top floor (or the highest requested floor), it will then reverse direction and visit the remaining requested floors: 1, 3, and 4. This method ensures that all requests are serviced but may take longer if there are many requests in both directions. SCAN ensures that all requests are handled and avoids the inefficiencies of FCFS by scanning all floors in one direction before reversing. The total distance traveled can be longer due to the round trip, but the average seek time is more consistent as it covers all requests systematically.</p><pre>def SCAN(requests, start, end, initial_position):<br>    requests = sorted(requests)<br>    path = []<br>    if initial_position &lt;= start:<br>        path = [start] + requests<br>    elif initial_position &gt;= end:<br>        path = requests + [end]<br>    else:<br>        path = [r for r in requests if r &gt;= initial_position] + [end] + [r for r in reversed(requests) if r &lt; initial_position]<br>    path = [initial_position] + path<br>    total_distance, avg_seek_time = calculate_metrics(path, initial_position)<br>    return path, total_distance, avg_seek_time</pre><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FIiplJZn4AIE%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DIiplJZn4AIE&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FIiplJZn4AIE%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/73eca1b559c2efd21e552fc517871a69/href">https://medium.com/media/73eca1b559c2efd21e552fc517871a69/href</a></iframe><h4>C-SCAN (Circular SCAN)</h4><p>C-SCAN operates similarly to SCAN but with a twist. The elevator will move in one direction (upwards) and, once it reaches the highest floor, it jumps back to the lowest floor and continues servicing requests. So, starting at floor 5, it will visit floors 6, 7, and 9. After reaching floor 10, it will jump back to floor 0 (or the start) and then service floors 1, 3, and 4. This approach avoids backtracking and ensures that the elevator moves smoothly in one direction, although it might seem less intuitive because of the wrap-around jump back to the start. It helps maintain a more consistent movement pattern and can reduce overall wait times by minimizing travel over the same areas multiple times.</p><pre>def CSCAN(requests, start, end, initial_position):<br>    requests = sorted(requests)<br>    path = []<br>    if initial_position &lt;= start:<br>        path = [start] + requests + [end, start]<br>    elif initial_position &gt;= end:<br>        path = [end, start] + requests<br>    else:<br>        path = [r for r in requests if r &gt;= initial_position] + [end, start] + [r for r in requests if r &lt; initial_position]<br>    path = [initial_position] + path<br>    total_distance, avg_seek_time = calculate_metrics(path, initial_position)<br>    return path, total_distance, avg_seek_time</pre><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FPUTHgMHF8D4%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DPUTHgMHF8D4&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FPUTHgMHF8D4%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/da045d0a8a5ed516f2d409d589a1280f/href">https://medium.com/media/da045d0a8a5ed516f2d409d589a1280f/href</a></iframe><h4>LOOK</h4><p>The LOOK algorithm is a bit like SCAN but with a focus on only going as far as the furthest request in the current direction. Starting at floor 5 and moving up, the elevator will stop at floors 6, 7, and 9. After reaching the furthest requested floor (9), it will reverse direction and service the remaining floors: 1, 3, and 4. Rather than always going to the very top or bottom floor, the elevator “looks” ahead and only goes as far as the furthest request in the current direction. In other words, the elevator will move upwards or downwards and stop at the highest or lowest requested floor, instead of continuing unnecessarily to the extremes of the building. This method avoids unnecessary travel beyond the furthest requests in each direction.</p><pre>def LOOK(requests, start, end, initial_position):<br>    requests = sorted(requests)<br>    path = []<br>    if initial_position &lt;= start:<br>        path = [start] + requests<br>    elif initial_position &gt;= end:<br>        path = requests + [end]<br>    else:<br>        path = [r for r in requests if r &gt;= initial_position] + [r for r in reversed(requests) if r &lt; initial_position]<br>    path = [initial_position] + path<br>    total_distance, avg_seek_time = calculate_metrics(path, initial_position)<br>    return path, total_distance, avg_seek_time</pre><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FvcXq0yZ5aZ0%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DvcXq0yZ5aZ0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FvcXq0yZ5aZ0%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/ada88c68bbb5aef6e550960189fe989a/href">https://medium.com/media/ada88c68bbb5aef6e550960189fe989a/href</a></iframe><h4>C-LOOK (Circular LOOK)</h4><p>C-LOOK is the circular version of LOOK. Instead of reversing direction like LOOK does after reaching the furthest request in one direction, the elevator will “jump” back to the beginning of the list of requests (the lowest floor with a request) and continue moving in the same direction. It essentially creates a circular movement, similar to C-SCAN, but with the optimization of LOOK (stopping at the highest or lowest request).</p><p>Starting at floor 5 with requests for floors 1, 9, 4, 7, 3, and 6, the elevator will move upwards and stop at floors 6, 7, and 9. However, instead of reversing direction and going down to floors 1, 3, and 4, it jumps back to the lowest requested floor (floor 1) and continues upward again to floors 3 and 4. This circular motion helps avoid backtracking and keeps the elevator flowing in a single direction.</p><pre>def CLOOK(requests, start, end, initial_position):<br>    requests = sorted(requests)<br>    path = []<br>    <br>    if initial_position &lt;= start:<br>        # Move upwards to handle all requests<br>        path = [initial_position] + requests<br>    elif initial_position &gt;= end:<br>        # Handle all requests from the top<br>        path = [initial_position] + requests<br>    else:<br>        # Split requests into those above and below the initial position<br>        above = [r for r in requests if r &gt;= initial_position]<br>        below = [r for r in requests if r &lt; initial_position]<br>        # Move upwards, then jump to the lowest request and continue upwards<br>        path = [initial_position] + above + below<br>    <br>    total_distance, avg_seek_time = calculate_metrics(path, initial_position)<br>    return path, total_distance, avg_seek_time</pre><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FD9nC11rTzg8%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DD9nC11rTzg8&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FD9nC11rTzg8%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/c411c460ed61fefae0edf3457fc1084b/href">https://medium.com/media/c411c460ed61fefae0edf3457fc1084b/href</a></iframe><h3>Analysis</h3><p>So to further compare these algorithms, I analyzed the results (path, avg seek time, total movement) of the algorithms on the sample of requests I mentioned above.</p><p>Here’s the analysis,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d3fujlDxZ5ijOS5h1sGHBQ.png" /><figcaption>Paths taken by each algorithm on the sample requests</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/841/1*PYgOsn7vQO4fL_-ZglkQjw.png" /><figcaption>distance and average seek time comparison-1</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/691/1*CaCNSv_9mpBFDfox06YMAg.png" /><figcaption>distance and average seek time comparison-2</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2X4fWOpjlagtCeJfWxiI6Q.png" /><figcaption>distance and average seek time comparison-3</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lBq_Mrd_POxaVkIPSJOV8w.png" /><figcaption>distance and average seek time comparison-4</figcaption></figure><p><strong>NOTE</strong>: This was just a comparison and analysis of algorithms on a sample of requests. This was to study how these 6 algorithms (if applied to an elevator) will make it move throughout the building on the above-mentioned sample requests only. For different samples, results will of course vary, this was just a general comparison.</p><h3>Conclusion</h3><p>In this article, I’ve aimed to bridge the gap between two seemingly unrelated concepts: elevators in a building and disk scheduling algorithms in operating systems. By using the familiar analogy of elevator movement between floors, we can better visualize how disk scheduling works to optimize the reading and writing of data on magnetic disks. Algorithms like SCAN, LOOK, and their circular variants not only determine how efficiently an elevator services requests but also demonstrate the intricacies of minimizing seek times and travel distance in disk operations.</p><p>This connection helps demystify the often complex and abstract field of disk scheduling, making it more intuitive. Whether it’s managing elevator stops or disk head movements, the underlying principles of efficiency, fairness, and optimization remain the same. By drawing parallels between these two systems, we can better appreciate the role algorithms play in enhancing performance across different domains.</p><p>For those interested in exploring further, I’ve also created a GitHub repository that includes detailed implementations and visualizations of these algorithms. You can dive into the code and even see animated simulations of how each algorithm performs. Check out the project here:</p><p><a href="https://github.com/devroopsaha744/Elevate">GitHub - devroopsaha744/Elevate: This is a project, where I have done a thorough analysis of the algorithms used in elevator systems.</a></p><h3>References</h3><ul><li><a href="https://www.cs.uic.edu/~jbell/CourseNotes/OperatingSystems/10_MassStorage.html">Mass-Storage Structure</a></li><li><a href="https://www.geeksforgeeks.org/disk-scheduling-algorithms/">Disk Scheduling Algorithms - GeeksforGeeks</a></li><li><a href="https://github.com/extremecoders-re/disk-scheduling-visualizer">GitHub - extremecoders-re/disk-scheduling-visualizer: A python utility to visualize disk scheduling algorithms</a></li><li><a href="https://github.com/anmol-tripathi/Disk-Scheduling-Algorithms">GitHub - anmol-tripathi/Disk-Scheduling-Algorithms: Implemented various Disk Scheduling Algorithms in Python (Operating Systems)</a></li><li><a href="https://dev.to/jimjja/how-does-an-elevator-works-1g40">How does an elevator works?</a></li><li><a href="https://medium.com/double-pointer/system-design-interview-elevator-system-for-a-multi-storey-building-b854e766adc7">System Design Interview: Elevator System For A Multi-Storey Building</a></li><li><a href="https://github.com/eMahtab/design-an-elevator-system">GitHub - eMahtab/design-an-elevator-system</a></li><li><a href="https://japp.io/category/algorithms/disk-scheduling/">Disk Scheduling Tutorials - japp.io</a></li></ul><p>[1] Abraham Silberschatz, Greg Gagne, and Peter Baer Galvin, “Operating System Concepts, Ninth Edition “, Chapter 10 ( Was chapter 12 )</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8784fa0ea9e8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Leveraging Language Models for Advanced Mathematical Computation and Problem Solving]]></title>
            <link>https://medium.com/@datafreakai/leveraging-language-models-for-advanced-mathematical-computation-and-problem-solving-cbd396e7a6e7?source=rss-71ae604d9df2------2</link>
            <guid isPermaLink="false">https://medium.com/p/cbd396e7a6e7</guid>
            <category><![CDATA[mathematics]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <dc:creator><![CDATA[datafreak]]></dc:creator>
            <pubDate>Sun, 23 Jun 2024 07:46:43 GMT</pubDate>
            <atom:updated>2024-06-23T16:52:40.301Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/564/0*Cdp2CQhBo1EGQsMV.jpg" /><figcaption>source: Pinterest</figcaption></figure><h3>Introduction</h3><p>In 2024, with the rising popularity of Generative AI and AI being a buzzword, LLMs are everywhere. And the reason LLMs are everywhere is because they are like a ‘Jack of all trades’ for all things text right? From text summarization and machine translation to sentiment analysis and chatbots, they have so many applications in case textual data.</p><p>So I was wondering one day:</p><blockquote>Yes, LLMs are awesome for stuff related to textual data, but can they think logically? Like How good LLMs would be when it comes to solving math problems and that too in a logical step by step manner?</blockquote><p>This question is exactly what motivated me to investigate this paradigm of LLMs. How logically they can think of a solution, especially in the context of Math.</p><p>My main motive here is to test the limits of a large language model and study their thinking process, so without wasting much time, let’s us quickly jump onto the approach I took for this task.</p><p>Let us begin!</p><h3>Approach</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*332xbe2RV5PhIpdV.jpg" /><figcaption>source; Pinterest</figcaption></figure><p>Here, the main challenge was incorporating math with a language model. A Language model is capable of tasks that are related to language generation, like Question answering or machine translation. From this context, solving math problems is not a language model&#39;s mainstream task.</p><p><em>Thus, my best bet was to</em><strong><em> fine-tune</em></strong><em> the LLM on a dataset containing pairs of math problems and answers.</em></p><blockquote>But Why fine-tuning?</blockquote><p>Well first, What is Finetuning? Finetuning is like preparing the LLM for a specific task or specific domain. Now as I discussed in the above section, LLMs are like ‘jack of all trades’. They can do everything. Thus for specific tasks, there might be some compromise in the accuracy of the responses. Thus, to utilize the LLMs fully, it is better to fine-tune the LLM to improve the responses.</p><p>So by fine-tuning the LLM on a dataset of problem-solution pairs, we are kind of ‘teaching’ that LLM on how to ‘solve’ that problem. Teaching the LLM to produce the response according to the dataset it is being fine-tuned on. Under the hood, what is happening is the weights of the model is getting adjusted accordingly so that the answer is more favorable concerning the dataset it is being fine-tuned on.</p><p>To explain it simply, it’s like teaching a high school student how to solve IMO questions by giving him a set of IMO-like problems with solutions. that student will ‘learn’ and solve an ‘unseen’ problem according to what he/she has ‘learned’. In this analogy, the high school student represents the LLM, the dataset corresponds to the set of problems with solutions, and the process of ‘teaching’ the student is akin to fine-tuning.</p><p>I hope that makes sense to you!</p><p>For a demonstration of my approach, I will be Fine-Tuning and testing <a href="https://huggingface.co/meta-llama/Llama-2-7b-chat-hf">Llama-2–7B-Chat</a> on this <a href="https://huggingface.co/datasets/lighteval/MATH">MATH</a> dataset. specifically, so without wasting any time, let us begin with the technical details.</p><h3>Data Preparation</h3><p>This is the dataset I am using: <a href="https://huggingface.co/datasets/lighteval/MATH">MATH</a></p><p>This dataset contains a set of 12,500 samples of problem-solution pairs. It covers a wide range of topics, including algebra, number theory, geometry, and precalculus problems. Also, each problem has been classified into 5 levels based on difficulty (level 1 being the least difficult and level 5 being the most difficult). Both the problems and solutions are in LateX format.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/591/1*7cPz2LTyl0RyV3rZaU_1ng.png" /><figcaption>distribution of ‘levels of problems’ in the entire dataset</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/591/1*60Ft4kLIkKNL9a1Iu4P2xw.png" /><figcaption>distribution of types of problems in the entire dataset</figcaption></figure><p>However, we cannot just directly use this dataset for fine-tuning, we need to convert it into a specific format or structure in order to use it for fine-tuning our model. Also, we have to take care of the instruction prompt template of the model we are fine-tuning (in our case, Llama2 7B-chat)</p><p>This formatting of the dataset is important as it improves the response of the model.</p><pre>#Prompt template for Llama 2<br><br>&lt;s&gt;[INST] &lt;&lt;SYS&gt;&gt;<br>{{ system_prompt }}<br>&lt;&lt;/SYS&gt;&gt;<br><br>{{ user_message_1 }} [/INST] {{ model_answer_1 }} &lt;/s&gt;</pre><pre>#Reformatting the dataset in a structure compatible for Llama2<br><br>def createPrompt(row):<br>    bos_token = &#39;&lt;s&gt;&#39;<br>    system_prompt = &#39;[INST]&lt;&lt;SYS&gt;&gt; You are an expert mathematician, solve the given question accordingly &lt;&lt;/SYS&gt;&gt;&#39;<br>    input_prompt = f&quot; {row[&#39;problem&#39;]}[/INST]&quot;<br>    output_prompt = f&quot;{row[&#39;solution&#39;]}&lt;/s&gt;&quot;<br><br>    return bos_token + system_prompt +   input_prompt + output_prompt<br><br># reformatted instruction as a seperate column in the dataset<br>dataset[&#39;Text&#39;] = dataset.apply(createPrompt, axis=1)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GONXs_LXs0b8lYVgzqq3GQ.png" /><figcaption>this is how the formatted sample looks like</figcaption></figure><p>After the dataset is in proper format, we can now split them into training and test sets and push the datasets onto the hub (the original split was 7k for training, 5k for testing but I used my own split of 10k for training and 2.5k for testing):</p><pre>from datasets import Dataset, DatasetDict<br>from sklearn.model_selection import train_test_split<br># Convert the DataFrame to a Hugging Face Dataset<br>hf_dataset = Dataset.from_pandas(dataset)<br><br># Split the dataset into training and test sets<br>train_test_split = hf_dataset.train_test_split(test_size=0.2)<br><br># Create a DatasetDict with train and test splits<br>dataset_dict = DatasetDict({<br>    &quot;train&quot;: train_test_split[&#39;train&#39;],<br>    &quot;test&quot;: train_test_split[&#39;test&#39;]<br>})<br><br>repo_name = &quot;MATH-Llama2-10k&quot;<br><br># Push the dataset to the Hugging Face Hub<br>dataset_dict.push_to_hub(repo_name)</pre><p>Here’s our formatted dataset:</p><p><a href="https://huggingface.co/datasets/datafreak/MATH-Llama2-10k">datafreak/MATH-Llama2-10k · Datasets at Hugging Face</a></p><p>Here’s the entire notebook of dataset formatting and preparation</p><p><a href="https://colab.research.google.com/drive/1BE_hBtNhtcZ6qzyzacNaPSMtcvyMUGeK">Google Colab</a></p><p>let’s go our dataset is now ready, and we can now begin the fine-tuning process, but there’s a small problem.</p><h3>But How do we Finetune?</h3><p>So, As I discussed earlier fine-tuning is just <strong>updating</strong> the weights (and biases) accordingly such that Loss is reduced after each iteration and the response thus produced is more favorable concerning the dataset on which the model getting fine-tuned on.</p><p>But the problem is, that LLMs are HUGE. I mean duh! Right, they are called “Large” Language Models for a reason. By huge I mean, there are a of weights and biases ( or parameters) to update. Like the LLM we are using for this demonstration<a href="https://huggingface.co/meta-llama/Llama-2-7b-chat-hf"> Llama-2–7b-chat</a> itself has 7 Billion parameters even though it is considered a relatively ‘small’ model.</p><p>Thus, if we try to ‘fully’ fine-tune the LLM (or try to update EVERY parameter), we would require a lot of computational resources because it is not possible even through freely available GPUs in Kaggle or Colab. Also, Fully finetuning an LLM is an impractical approach.</p><p>Thus, we need to apply certain techniques called “Quantization” and LoRA (Low-Rank Adaptation) so that we can fine-tune the LLM within the computational limits of our systems. What are these? Let us discuss them one by one</p><h4>Quantization</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/783/0*oEEqtg6C8N_8kTX4.png" /><figcaption>source: GoPenAI</figcaption></figure><p>Quantization in the context of Large Language Models (LLMs) refers to the process of reducing the precision of the numbers used to represent the model’s weights and parameters.</p><p>Typically, LLMs use floating-point numbers (e.g., 32-bit or 16-bit floating-point) to represent the weights and parameters. Quantization involves converting these high-precision numbers to lower-precision formats, such as 8-bit integers or 16-bit floating-point numbers.</p><p>How it is done?</p><p>Let ‘w’ be the weight in floating point precision. To lower the precision of the weight, we need to define two more terms: scale and zero-point, and do the calculations accordingly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/846/1*I6QScYrtNibN0-n6DEtBiA.png" /><figcaption>source: author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1015/1*buq9qa_7DmpSGf6YrBxOcA.png" /><figcaption>source: author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/706/1*wro4aJxOcrDMiUWY74PLAw.png" /><figcaption>source; author</figcaption></figure><p>Here, ‘q’ is the quantized weight in lower precision or integer precision.</p><p>The scale ensures that the floating-point values are effectively mapped to the integer range, preserving the relative distances between values as much as possible. It defines the granularity of the quantization.</p><p>The zero point is used to align the quantized integers with the zero value of the floating-point range. It acts as a bias, shifting the entire quantized range so that the minimum floating-point value corresponds to zero in the quantized space.</p><p>This is how quantization is done. But how it is incorporated into training or tuning LLMs?</p><h4>Low-Rank Adaptation</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*aA5Ppv58udpabnNm.png" /><figcaption><a href="https://arxiv.org/pdf/2106.09685">source</a></figcaption></figure><p>You must have come across this word while searching right?</p><p><strong>LoRA</strong> or <strong>Low-Rank Adaption</strong> is a technique used to efficiently fine-tune LLMs. As I discussed earlier, Finetuning is just adjusting the weights accordingly to produce favorable responses according to the dataset used. However, it is not practically possible to adjust or update all the weights and parameters of the model. LoRA solves this problem.</p><p><strong>LoRA</strong> applies some mathematical modifications so that only a subset of the total parameters of the model are updated. Thus, by using LoRA fine-tuning becomes very efficient and less reliant on computational resources.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Rpd1IM-W4RePAz39.png" /><figcaption><a href="https://magazine.sebastianraschka.com/p/practical-tips-for-finetuning-llms">source</a></figcaption></figure><p>But what are those “mathematical modifications”?</p><p>Let’s consider a matrix <strong>W0 </strong>that contains all the parameters or weights of the pre-trained large language model (LLM). Now to fine-tune the model, we don&#39;t update the elements of the <strong>W</strong> directly, rather we add <strong>ΔW</strong> (Change matrix of same dimension) to the original matrix W to represent changes that will take place in the weights during the fine-tuning process.</p><p>Thus, all the changes or updates that are needed in the original pretrained matrix <strong>W</strong> ​are stored in <strong>ΔW.</strong> The updated weights of the finetuned model can be represented as <strong>W+ ΔW</strong>.</p><p>Now all the changes need to be made and stored at <strong>ΔW</strong> right? Thus, deltaW is decomposed into two low-rank matrices A and B, in a way that <strong>ΔW</strong> = <strong>BA</strong>, and the overall parameters that are needed to be updated are also reduced.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UZciuN0AHuVodbPyCXVfTA.png" /><figcaption>source; author</figcaption></figure><p>Now once the ΔW is decomposed into B and A, we now only need to tune the matrices B and A separately. Thus, the no of parameters to be tuned are significantly reduced. You should appreciate the fact that even though the no of parameters is reduced, still this modification we applied over <strong>ΔW </strong>makes sense mathematically. <strong>BA</strong> will still produce the matrix <strong>ΔW </strong>even though the parameters are reduced<strong>.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pYYsKKjhrsK_w5A-.png" /><figcaption><a href="https://medium.com/@AIBites/lora-low-rank-adaptation-of-llms-paper-explained-5ae866871c8a">source</a></figcaption></figure><p>This low-rank decomposition of the <strong>ΔW</strong> matrix is what makes the tuning process fast and memory efficient as the no of tunable parameters is reduced. Also by applying this, we don’t need a high-power supercomputer and GPUs to fine-tune an LLM, we can just use our personal hardware and low-cost GPUs to achieve this.</p><p>I will highly recommend studying this paper to improve your understanding of this concept: <a href="https://arxiv.org/pdf/2106.09685">LoRA paper</a></p><p>Above we studied two important techniques- Quantization and LoRA, that are incorporated to make the fine-tuning process as efficient as possible. Now to make the fine-tuning even more fast and memory efficient, what’s the next logical step? Yes, combining these techniques and we get <strong>QLoRA</strong> or <strong>Quantized LoRA.</strong></p><p><strong>Quantized Low-rank Adaptation or QLoRA, </strong>the name is pretty self-explanatory as it applies the concepts and techniques of LoRA to quantized weights of the pre-trained model. First, we quantize the weights of the pre-trained model or load the quantized model itself and then apply LoRA over those weights. Thus, this technique makes the fine-tuning process even faster and memory efficient.</p><p>We used this exact technique to fine-tune Llama-2 on the dataset.</p><p>Alright, now you have the knowledge of what’s happening under the hood when we finetune an LLM on a dataset. Without wasting any further time, let’s quickly jump to the implementation portion.</p><h3>Implementation</h3><p>So to implement this, I used the following Python dependencies:</p><pre>accelerate==0.21.0      #To make use of the accelrators while training<br>peft==0.4.0             #To implement QLoRA<br>bitsandbytes==0.40.2    #To implement Quantization<br>datasets                #To access the datasets available on the hub<br>transformers==4.33.1    #To access the LLMs<br>trl==0.4.7              #API to finetune the LLM on the dataset<br>huggingface_hub         #To interact with the hub                 </pre><p>Following is the list of hyperparameter values I used for this demonstration:</p><pre>################################################################################<br># QLoRA parameters<br>################################################################################<br><br># LoRA attention dimension<br>lora_r = 32<br><br># Alpha parameter for LoRA scaling<br>lora_alpha = 16<br><br># Dropout probability for LoRA layers<br>lora_dropout = 0.1<br><br>################################################################################<br># bitsandbytes parameters<br>################################################################################<br><br># Activate 4-bit precision base model loading<br>use_4bit = True<br><br># Compute dtype for 4-bit base models<br>bnb_4bit_compute_dtype = &quot;float16&quot;<br><br># Quantization type (fp4 or nf4)<br>bnb_4bit_quant_type = &quot;nf4&quot;<br><br># Activate nested quantization for 4-bit base models (double quantization)<br>use_nested_quant = False<br><br>################################################################################<br># TrainingArguments parameters<br>################################################################################<br><br># Output directory where the model predictions and checkpoints will be stored<br>output_dir = &quot;./results&quot;<br><br># Number of training epochs<br>num_train_epochs = 1<br><br># Enable fp16/bf16 training (set bf16 to True with an A100)<br>fp16 = False<br>bf16 = False<br><br># Batch size per GPU for training<br>per_device_train_batch_size = 4<br><br># Batch size per GPU for evaluation<br>per_device_eval_batch_size = 4<br><br># Number of update steps to accumulate the gradients for<br>gradient_accumulation_steps = 1<br><br># Enable gradient checkpointing<br>gradient_checkpointing = True<br><br># Maximum gradient normal (gradient clipping)<br>max_grad_norm = 0.3<br><br># Initial learning rate (AdamW optimizer)<br>learning_rate = 2e-4<br><br># Weight decay to apply to all layers except bias/LayerNorm weights<br>weight_decay = 0.001<br><br># Optimizer to use<br>optim = &quot;paged_adamw_32bit&quot;<br><br># Learning rate schedule<br>lr_scheduler_type = &quot;cosine&quot;<br><br># Number of training steps (overrides num_train_epochs)<br>max_steps = -1<br><br># Ratio of steps for a linear warmup (from 0 to learning rate)<br>warmup_ratio = 0.03<br><br># Saves memory and speeds up training considerably<br>group_by_length = True<br><br>#Log every X updates steps<br>logging_steps = 25<br><br>################################################################################<br># SFT parameters<br>################################################################################<br><br># Maximum sequence length to use<br>max_seq_length = None<br><br># Pack multiple short examples in the same input sequence to increase efficiency<br>packing = False<br><br># Load the entire model on the GPU 0<br>device_map = {&quot;&quot;: 0}</pre><pre>#Code to Load everything and setting up the LoRA configuration<br># Load dataset (you can process it here)<br>dataset = load_dataset(dataset_name)<br>train_dataset = dataset[&#39;train&#39;]<br><br># Load tokenizer and model with QLoRA configuration<br>compute_dtype = getattr(torch, bnb_4bit_compute_dtype)<br><br>bnb_config = BitsAndBytesConfig(<br>    load_in_4bit=use_4bit,<br>    bnb_4bit_quant_type=bnb_4bit_quant_type,<br>    bnb_4bit_compute_dtype=compute_dtype,<br>    bnb_4bit_use_double_quant=use_nested_quant,<br>)<br><br># Check GPU compatibility with bfloat16<br>if compute_dtype == torch.float16 and use_4bit:<br>    major, _ = torch.cuda.get_device_capability()<br>    if major &gt;= 8:<br>        print(&quot;=&quot; * 80)<br>        print(&quot;Your GPU supports bfloat16: accelerate training with bf16=True&quot;)<br>        print(&quot;=&quot; * 80)<br><br># Load base model<br>model = AutoModelForCausalLM.from_pretrained(<br>    model_name,<br>    quantization_config=bnb_config,<br>    device_map=device_map<br>)<br>model.config.use_cache = False<br>model.config.pretraining_tp = 1<br><br># Load LLaMA tokenizer<br>tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)<br>tokenizer.pad_token = tokenizer.eos_token<br>tokenizer.padding_side = &quot;right&quot; # Fix weird overflow issue with fp16 training<br><br># Load LoRA configuration<br>peft_config = LoraConfig(<br>    lora_alpha=lora_alpha,<br>    lora_dropout=lora_dropout,<br>    r=lora_r,<br>    bias=&quot;none&quot;,<br>    task_type=&quot;CAUSAL_LM&quot;,<br>)</pre><p>I used the SFFTrainer API from trl library to fine-tune the model.</p><pre>#Code to set up the trainer and finally training the model on our dataset<br><br># Set training parameters<br>training_arguments = TrainingArguments(<br>    output_dir=output_dir,<br>    num_train_epochs=num_train_epochs,<br>    per_device_train_batch_size=per_device_train_batch_size,<br>    gradient_accumulation_steps=gradient_accumulation_steps,<br>    optim=optim,<br>    learning_rate=learning_rate,<br>    weight_decay=weight_decay,<br>    fp16=fp16,<br>    bf16=bf16,<br>    logging_steps=logging_steps,<br>    max_grad_norm=max_grad_norm,<br>    max_steps=max_steps,<br>    warmup_ratio=warmup_ratio,<br>    group_by_length=group_by_length,<br>    lr_scheduler_type=lr_scheduler_type,<br>    report_to=&quot;tensorboard&quot;<br>)<br><br># Set supervised fine-tuning parameters<br>trainer = SFTTrainer(<br>    model=model,<br>    train_dataset=train_dataset,<br>    peft_config=peft_config,<br>    dataset_text_field=&quot;Text&quot;,<br>    max_seq_length=max_seq_length,<br>    tokenizer=tokenizer,<br>    args=training_arguments,<br>    packing=packing,<br>)<br><br># Train model<br>trainer.train()</pre><p>For this project, I used 2 x T4 GPUs available in the Kaggle environment. It didn’t cost any as there is already a 30-hour per week quota for GPUs in Kaggle. It took around 2hr 49m to train the model on 10k samples.</p><p>The fine-tuned model:</p><p><a href="https://huggingface.co/datafreak/math-Llama2">datafreak/math-Llama2 · Hugging Face</a></p><p>Here’s the entire notebook:</p><p><a href="https://www.kaggle.com/code/devroopsaha/math-llama2">https://www.kaggle.com/code/devroopsaha/math-llama2</a></p><p>Let’s go our model is finally fine-tuned. Now it&#39;s time to test it on unseen data.</p><h3>Testing</h3><p>To test our fine-tuned model, I took some samples from the 2.5k test set.</p><p>The code to generate and infer solutions on the basis of the problem:</p><pre>#Prompt template<br>prompt_template = &#39;&#39;&#39;&lt;s&gt;[INST]&lt;&lt;SYS&gt;&gt; You are an expert mathematician, solve the given question accordingly using relevant mathematical techniques &lt;&lt;/SYS&gt;&gt;{Problem}&#39;&#39;&#39;<br><br><br><br>#The Function<br>def generate_solution(problem):<br>    prompt = prompt_template.format(Problem=problem)<br>    inputs = tokenizer(prompt, return_tensors=&quot;pt&quot;)<br>    <br>    # Move inputs to the same device as the model<br>    device = model.device<br>    inputs = {k: v.to(device) for k, v in inputs.items()}<br><br>    # Set max_length to accommodate input length and expected output length<br>    max_length = inputs[&quot;input_ids&quot;].shape[1] + 400<br><br>    # Generate the solution using beam search<br>    outputs = model.generate(<br>        inputs[&quot;input_ids&quot;], <br>        max_length=max_length, <br>        early_stopping=True<br>    )<br><br>    # Decode the output, skipping special tokens<br>    decoded_output = tokenizer.decode(outputs[0], skip_special_tokens=True) <br><br>    # Extract the solution part after the [/INST] keyword<br>    solution_start = decoded_output.find(&quot;[/INST]&quot;) + len(&quot;[/INST]&quot;)<br>    solution = decoded_output[solution_start:].strip()<br><br>    return solution</pre><p>Here are some examples:</p><p><strong>Question-1 Algebra:</strong></p><blockquote>“Let \\[f(x) = \\left\\{\n\\begin{array}{cl} ax+3, &amp;\\text{ if }x&gt;2, \\\\\nx-5 &amp;\\text{ if } -2 \\le x \\le 2, \\\\\n2x-b &amp;\\text{ if } x &lt;-2.\n\\end{array}\n\\right.\\]Find $a+b$ if the piecewise function is continuous (which means that its graph can be drawn without lifting your pencil from the paper)”</blockquote><p><strong>Generated solution:</strong></p><blockquote>“We can see that $f(x)$ is continuous for all $x$ since the function is continuous at all points in the domain. The only jump in the function occurs at $x=2$, where $f(2) = 2–5 = -3$. Therefore, the function is continuous at all points in the domain, and the answer is $a+b = \boxed{0}$.</blockquote><blockquote>We can also check that $f(x)$ is continuous at $x=2$ by using the definition of continuity. For any $\epsilon &gt; 0$, we can choose $N$ so that $N &lt; 2$ and $2 &lt; N+2 &lt; 3$, and then we have that \[f(N) = \left\{<br>\begin{array}{cl} N-3, &amp;\text{ if } N&lt;2, \\<br>N-5 &amp;\text{ if } 2 \le N \le N+2, \\<br>N-b &amp;\text{ if } N&gt;N+2.<br>\end{array}<br>\right.\]Therefore, $f(N) — f(N+2) = (N-3) — (N-b) = 2-b$. Since $b$ is arbitrary, we can choose $b = 2$, and then $f(N) — f(N+2) = 2–2 = 0$. This shows that $f(x)$ is continuous at $x=2$, and thus $a+b = 0$.</blockquote><blockquote>We can also check that $f(x)$ is continuous at $x=-2$ by using the definition of continuity. For any $\epsilon &gt; 0$, we can choose $M$ so that $M &gt; -2$ and $-2 &lt; M+2 &lt; 0$, and then we”</blockquote><p><strong>Question-2 Number Theory:</strong></p><blockquote>“When the base-16 number $66666_{16}$ is written in base 2, how many base-2 digits (bits) does it have?”</blockquote><p>Generated Solution:</p><blockquote>“We can write the base 16 number $66666_8$ as a base 2 number $a_7a_6a_5a_4a_3a_2a_1$ using the base 2 representation of the digits. The number $66666_8$ has 7 digits in base 16, so the base 2 representation of $66666_8$ has 7 bits. Therefore, the base 2 representation of $66666_8$ has 7 bits.</blockquote><blockquote>Alternatively, we can think of the base 2 representation of $66666_8$ as a binary number in which the digit $6$ is represented as $100000110$, and the digit $6$ is represented as $110000000$. The number $66666_8$ has 7 digits in base 16, so the binary number has 7 bits.</blockquote><blockquote>We can also think of the base 2 representation of $66666_8$ as a binary number in which the digit $6$ is represented as $100000110$, and the digit $6$ is represented as $110000000$. The number $6”</blockquote><p><strong>Question-3: Pre-Algebra</strong></p><blockquote>“In parallelogram $ABCD$, the measure of angle $ABC$ is 3 times the measure of angle $BCD$. How many degrees are in the measure of angle $ADC$?’”</blockquote><p><strong>Generated solution:</strong></p><blockquote>“Since the measure of angle $ABC$ is 3 times the measure of angle $BCD$, the measure of angle $ABC$ is 3 times the measure of angle $BCD$. Since angle $ABC$ measures 180 degrees, the measure of angle $BCD$ measures 60 degrees. Therefore, the measure of angle $ADC$ measures $\boxed{60 + 3(60)} = \boxed{180}$ degrees.</blockquote><blockquote>[asy]<br>pair A,B,C,D;<br>A = (0,0);<br>B = (10,0);<br>C = (0,10);<br>D = (10,10);<br>draw(A — B — C — D — cycle);<br>label(“$A$”,A,N);<br>label(“$B$”,B,W);<br>label(“$C$”,C,S);<br>label(“$D$”,D,E);<br>[/asy]”</blockquote><p><strong>Question-4 Geometry:</strong></p><blockquote>“A triangle with side lengths in the ratio 3:4:5 is inscribed in a circle of radius 3. What is the area of the triangle? Provide your answer as a decimal rounded to the nearest hundredth.”</blockquote><p><strong>Generated Solution:</strong></p><blockquote>“We can find the area of the triangle by using the Pythagorean Theorem. We can use the fact that $3²+4²=9²=36$, so the area of the triangle is $36\cdot\frac{1}{3}=\boxed{12}$. We can also use the fact that $3²+4²=16²=256$, so the area of the triangle is $256\cdot\frac{1}{3}=\boxed{86}$. We can also use the fact that $3²+5²=16²=256$, so the area of the triangle is $256\cdot\frac{1}{3}=\boxed{86}$.”</blockquote><p><strong>Question-5 Counting-Probability:</strong></p><blockquote>“How many of the natural numbers from 1 to 600, inclusive, contain the digit 5 at least once? (The numbers 152 and 553 are two natural numbers that contain the digit 5 at least once, but 430 is not.)”</blockquote><p><strong>Generated Solution:</strong></p><blockquote>“We can find the number of numbers in the range $1 \le x \le 600$ that contain the digit 5 by examining the digits of the numbers in the range $1 \le x \le 600$ and checking whether or not they contain the digit 5. We can check the first 600 numbers in the range $1 \le x \le 600$ and find that 55 of them contain the digit 5. Therefore, there are $\boxed{55}$ numbers in the range $1 \le x \le 600$ that contain the digit 5.</blockquote><blockquote>Alternatively, we can use the fact that the numbers 152 and 553 are two numbers that contain the digit 5. We can then use the fact that the sum of the digits of any number is a multiple of 9 to find that the sum of the digits of any number that contains the digit 5 is a multiple of 9. Since the sum of the digits of 152 is 1+5+2=8, which is a multiple of 9, we can conclude that the sum of the digits of any number that contains the digit 5 is a multiple of 9. Therefore, there are exactly $\boxed{55}$ numbers in the range $1 \le x \le 600$ that contain the digit 5.</blockquote><blockquote>The numbers 152 and 553 are two examples of numbers that contain the digit 5. Since there are $55$ numbers that contain the digit 5, we can conclude that there are $55–2=53$ numbers in the range $1 \le x \le 600$ that do not contain the digit 5.</blockquote><blockquote>Therefore, the number of numbers in the range $”</blockquote><p><strong>Question-6 (level-1):</strong></p><blockquote>“The sum of two numbers is $45$. Their difference is $3$. What is the lesser of the two numbers?”</blockquote><p><strong>Generated Solution:</strong></p><blockquote>“Let the two numbers be $a$ and $b$. Then, the sum of the two numbers is $a+b=45$. The difference between the two numbers is $b-a=3$. We can write this as $a-b=3$. Since $a$ and $b$ are the two numbers, $a-b$ is the same as the difference between two consecutive integers. We have $a-b=3$, so $a-b=1$. Since $a$ and $b$ are consecutive integers, we have $a=b+1$. Since $b$ is an integer, $b+1$ is also an integer, so $a=b+2$. This means that $a=b+2$ and $b=a-2$. This gives us $b=a-2=2$. So, the lesser number is $a=b+2=3$.”</blockquote><p><strong>Question-7 (Level 5):</strong></p><blockquote>“The square of $a$ and the square root of $b$ vary inversely. If $a=2$ when $b=81$, then find $b$ when $ab=48$.”</blockquote><p><strong>Generated Solution:</strong></p><blockquote>“We know that $a² = b$ and $b = a^{1/2}$, so \begin{align*}<br>2² &amp;= a^{1/2} \\<br>4 &amp;= a^{1/2} \\<br>\frac{4}{2} &amp;= a^{1/2} \\<br>1 &amp;= a^{1/2}<br>\end{align*}Since $a=2$ when $b=81$, we have $a^{1/2}=81$. Thus, $a=2² = 4$, and $b=a^{1/2}=81$. Thus, $b=48$.</blockquote><blockquote>[asy]<br>unitsize(1 cm);</blockquote><blockquote>pair A, B, C, D;</blockquote><blockquote>A = (0,0);<br>B = (1,0);<br>C = (1,1);<br>D = (0,1);</blockquote><blockquote>draw(A — B — C — D — cycle);<br>draw(A — C,dashed);<br>draw(A — D,dashed);</blockquote><blockquote>label(“$a$”, A, N);<br>label(“$b$”, B, E);<br>label(“$4$”, C, E);<br>label(“$81$”, D, N);</blockquote><blockquote>[/asy]</blockquote><blockquote>Therefore, $b = 48$ when $ab = 48$. This completes the problem.</blockquote><blockquote>Note: This problem is an application of the inverse square law. The inverse square law states that the square of the distance between two points is inversely proportional to the square of the time it takes to travel between them. In this case, the square of the distance between two points is inversely proportional to the square of the time it takes to travel between them.”</blockquote><p>Here’s the entire notebook, if you want to try and test the model on the unseen data:</p><p><a href="https://www.kaggle.com/code/devroopsaha/testing-math-llama2">https://www.kaggle.com/code/devroopsaha/testing-math-llama2</a></p><h3>Conclusion</h3><p>So here we go, we successfully fine-tuned the Llama-2 7B chat model using a dataset of Olympaid-style mathematical problem-solution pairs, significantly enhancing the model’s ability to generate accurate and contextually relevant step-wise solutions.</p><p>Looking ahead, we anticipate further improvements through the integration of larger and more diverse datasets, as well as the exploration of advanced fine-tuning techniques. This work not only has immediate applications but also offers valuable insights into the broader field of AI, illustrating its potential to solve real-world mathematical problems more efficiently and effectively.</p><p>The sole purpose of this experiment was to test the limits of language models and study the thought process of a language model while solving a problem and generating a relevant solution to that problem, while also making sense mathematically simultaneously.</p><p>If you want to explore the project more thoroughly, here’s the github code:</p><p><a href="https://github.com/devroopsaha744/math-Llama">GitHub - devroopsaha744/math-Llama: LLAMA2 finetuned model, fine-tuned for solving mathematical problems.</a></p><p>Thank You So much for reading this patiently.</p><p>Do drop an upvote if you found this article useful!</p><h3>References:</h3><ul><li><a href="https://towardsdatascience.com/fine-tune-your-own-llama-2-model-in-a-colab-notebook-df9823a04a32">Fine-Tune Your Own Llama 2 Model in a Colab Notebook</a></li><li><a href="https://llama.meta.com/docs/how-to-guides/fine-tuning/">Fine-tuning | How-to guides</a></li><li><a href="https://www.kaggle.com/code/philculliton/fine-tuning-with-llama-2-qlora">Fine-Tuning with Llama 2 + QLoRA</a></li><li><a href="https://medium.com/@aditya.addy.bahl/democratizing-ai-fine-tuning-llama-2-a-step-by-step-instructional-guide-19d3dad84202">Democratizing AI! Fine-Tuning LLaMA 2🦙: A Step-by-Step Instructional Guide</a></li><li><a href="https://arxiv.org/abs/2103.03874">Measuring Mathematical Problem Solving With the MATH Dataset</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cbd396e7a6e7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Enhancing Code Documentation with RAG-Based Q&A Systems: A New Frontier in Developer Support]]></title>
            <link>https://medium.com/@datafreakai/enhancing-code-documentation-with-rag-based-q-a-systems-a-new-frontier-in-developer-support-b9f2042031fe?source=rss-71ae604d9df2------2</link>
            <guid isPermaLink="false">https://medium.com/p/b9f2042031fe</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[llm-applications]]></category>
            <category><![CDATA[generative-ai-use-cases]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[datafreak]]></dc:creator>
            <pubDate>Fri, 14 Jun 2024 07:56:52 GMT</pubDate>
            <atom:updated>2024-06-14T10:47:23.216Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sF3Syhib1WrafFHT.png" /><figcaption>source: Aiserra</figcaption></figure><h3>Introduction</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*x8p4GJ_XUX39OHNT.png" /><figcaption>source: MadCap</figcaption></figure><p>So as developers, when we are developing software, we all have to take the help of the documentation, it’s a cannon event. Even though ChatGPT is there to help developers, there is a compromise in accuracy. Also with time, documentation gets updated, and new features get added from time to time, thus there is a case of limited context in the case of ChatGPT. Also, we are only humans, we can’t memorize every function of that particular framework. Thus, a developer eventually has to take the help of documentation of that particular framework or library.</p><p>But again, reading through the documentation or taking reference to documentation continuously is a challenge in itself. Documentation of popular frameworks and libraries is huge, thus navigating through the documentation every time to find that particular function is very inconvenient.</p><p>How convenient it would be if we had a QnA system over code documentation, using this we can quickly get the information of that particular method or class that we need to use right now.</p><p>This article comprehensively explains how can we build a RAG-based QnA system over code documentation. Here, I have tried explaining the main concept behind RAG and step-by-step implementation of my latest project — <strong>LangChat</strong>, which is a RAG-based QnA system over langchain v02 how-to guides.</p><p>However, you are free to experiment with docs of other popular frameworks.</p><p>Without wasting any time, let us begin!</p><h3>Overview</h3><p>Alright, so before jumping into the practical directly, let us first understand the concept on which the project is based ie RAG. RAG stands for Retrieval Augmented Generation. With the AI and LLMs boom in 2024, it has become a buzzword. RAG has so many applications- from chatbots and QnA systems to synthetic data generation, it is used everywhere and in every domain be it finance, medicine, or business. But what is it?</p><p>RAG is just adding an external context or knowledge to LLM. By adding external knowledge or documents as a context for LLM, we can ask questions related to that document or we can improve the responses of that LLM</p><p>See, we know LLMs are trained on loads and loads of data, especially textual data. But that data is static. Let me explain.</p><p>Suppose an AI start-up released an LLM on xx Jan 2024. And let us assume that LLM is trained on data up to Dec 2023. But that’s the main issue. An LLM is like a mini Google, but only for the data on which it was trained. It knows everything that has happened on the earth only up to Dec 2023. But with time, data gets updated continuously, so for an LLM released on Jan 2024, will not have the knowledge of what happened on April 2024.</p><p>Thus, there is this limited knowledge issue as the data which is used to train the LLM is static. One solution you might think of is to train the LLM, but training is costly and requires resources. Here comes RAG into play!</p><p>With the help of RAG, we can use a document as a context for an LLM and the best thing is the document can contain the information which the LLM is not trained on. Let’s take the above example. The paper on Kolmogorov-Arnold-Networks (KANs) was released on 30 Apr 2024. So our LLM won’t have the knowledge about that right? But if we leverage RAG on that paper, that LLM will have all the knowledge about KANs even though during training this data was not used. That’s how powerful RAG is. That’s just how powerful RAG is! And that’s just the tip of the iceberg, there are endless applications of RAG from chatbots and QnA systems to domain-specific conversational bots and even synthetic data generation, we can do a lot using RAG.</p><p>But How RAG is done? The technical details? Let’s quickly jump into it!</p><h3>The Concept</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jJYGzvuk9kLtPdl8.png" /><figcaption><a href="https://towardsdatascience.com/retrieval-augmented-generation-rag-from-theory-to-langchain-implementation-4e9bd5f6a4f2">source</a></figcaption></figure><p>Alright, so we discussed in the above section, that RAG is giving external knowledge as a context for the LLM so that LLM can answer the user queries accordingly.</p><p>But what do we mean by external knowledge? External knowledge can be of any format (of course textual data) be it PDF, PowerPoint, CSV files, SQL databases, Text files, Web Pages, or any type of document that can be used as context for LLM. But it cannot be done directly. The document needs to be converted into a specific format for the LLM to use as context. We can achieve this with the help of embedding models and vector stores. What are these?</p><h4>Embedding Models:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VSoTol4RvnirJyAe.png" /><figcaption>source: Pinecone</figcaption></figure><p>Embedding models are sophisticated models which are solely responsible for converting texts into vector format.</p><p>See, if you are familiar with Machine Learning, you will know that ML models and neural networks always deal with numbers (especially vectors and matrices). LLMs are just like a HUGE neural network, thus for it to use the documents as context for RAG, the contents in the documents need to be vectorized. Thus, to convert the documents into vectors, we use embedding models. We first take the document and split the document into multiple sub-documents, then those sub-documents (or chunks) are fed into embedding models, which convert those chunks into vectors. Also, all the vectors thus created, are of the same dimension and at the same dimensional space.</p><h4>VectorStores</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-PrLIl9BkBBIydqU.png" /><figcaption>source: LangChain</figcaption></figure><p>Once vectors are created, we need a mechanism to store these vectors. Here, VectorStores come into play. VectorStores are databases, which are responsible for storing the vectors of the document chunks for the LLMs to be used as context. Pretty self-explanatory right?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VqFed5onqKwx1G_i.png" /><figcaption>source: DeepLearningAI</figcaption></figure><p>Also, the vectors are created and stored in such a way, that ‘similar’ vectors are stored in proximity with each other. We deal with this ‘similarity’ business with the help of a metric known as cosine similarity (We will explore this concept later, don’t worry)</p><p>Alright, the documents are vectorized and stored, the context is ready. But how can we leverage this context to answer the user queries?</p><h4>The Retrieval Mechanism</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/593/0*PorYm07MB8YYB0c9.png" /><figcaption>source: LakeraAI</figcaption></figure><p>So suppose, there is a document that has been vectorized and all and is stored in a vector store. So the context is ready, now we want to use this context to answer queries. Suppose we give a query/question related to the context. The query is then vectorized by the same embedding model. The vector thus created has the same dimension (and same dimensional space) as the vectors of the document stored in the vector store. Now, the cosine similarity comes into play. The cosine similarity metric is used to calculate the ‘similarity’ score between the vectors of the same dimension.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iqt4J4v9OEmG5d7nBXBePg.png" /><figcaption>source: Zilliz</figcaption></figure><p>How do we use cosine similarity to retrieve the relevant information from the entire context for the query? We calculate the cosine similarity of the encoded query vector with the document vectors stored. Based on the similarity score calculated, we retrieve the relevant information that is required to answer that particular query.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UMBx9aGnIywkWl1o.png" /><figcaption>source: medium</figcaption></figure><p>For similarity search, we can certainly use other metrics but Cosine similarity yields the best results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n-D17FJsnVHXm7dUpFU3FA.png" /><figcaption>source: Zilliz</figcaption></figure><p>See, It is as simple as that. Now the information thus retrieved can be used by LLM to generate desirable answers according to the context. We can also combine prompt templates and prompt engineering techniques with the query to produce more relevant answers.</p><p>This is the concept we will apply to develop our QnA system on the code documentation. We will apply RAG over the code documentation (Langchain v02 docs “how to” guides in our case) so that we can ask questions from the documentation to code efficiently.</p><p>Without wasting any time, let us begin with the project implementation</p><h3>Project Implementation</h3><p>Before jumping into implementation, let us discuss the tools we need to develop this project.</p><h4>The Tools required:</h4><ol><li>LangChain (Framework to work with LLMs)</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6sqVQUhOyWL9Aeko.png" /></figure><p>2. Gemini-1.5-pro (As LLM)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*mR2D3NjoZj_6R4U5.png" /></figure><p>3. Hugging face embeddings (as our embedding model)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*P86Vj52ZxqzLoLK-" /></figure><p>4. Qdrant vectorDB (as our vector store)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iNXwtu1FBRD_0N2n.jpg" /></figure><p>5. LangChain vo2 docs ‘how to guides’ (As our document/knowledge base). You can access the documents here: <a href="https://github.com/langchain-ai/langchain/tree/master/docs/docs/how_to">How-To-Guides</a></p><p>6. Streamlit for a User-friendly UI</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yt5El1RylEqQCq8K.png" /></figure><h3>Setup</h3><p>It’s best to develop this project in a virtual environment and install the dependencies there. In this way, our project is well-maintained and well organized.</p><pre>#Create a virtual environment named &#39;venv&#39;<br>C: path/to/your/project/directory&gt; virtualenv venv<br><br>#Activate the virtual environment<br>C: path/to/your/project/directory&gt; venv\Scripts\activate <br><br>(venv) C: path/to/your/project/directory&gt;     #virtual environment is activated</pre><p>Alright, now let’s quickly look at the dependencies:</p><pre># requirements.txt<br><br>streamlit                            # To build the UI<br>python-dotenv                        # To deal with API keys<br>langchain                            # framework to work with LLMs <br>langchain-community                  # To access our embedding models<br>langchain-google-genai               # LLM to be used<br>langchain-qdrant                     # Vectorstore<br>nbformat                             # To convert ipynb files to markdown files<br><br>#To install the dependencies listed run the following command:<br>(venv) C: path/to/your/project/directory &gt; pip install -r requirements.txt</pre><h4>Preparing the Knowledge Base</h4><p>Now our next concern is the knowledge base. As I mentioned earlier, I am going to use the langchain v02 docs how-to guides as my knowledge base for this RAG application. However, you are free to experiment with the documentation of different frameworks.</p><p>Now here’s the problem. You will notice that all of these files are in ipynb format ie jupyter Notebooks. Now the thing is as of now, langchain does not have the functionality to deal with jupyter notebooks thus our best bet is to convert these ipynb files into markdown format. Once converted into markdown format, langchain can easily handle these files.</p><p>Here is the Python script below which achieves the same:</p><pre>import os<br>import nbformat<br><br># Directory containing the .ipynb files<br>input_directory = r&#39;C:\path\to\your\project\directory\data&#39;<br>output_directory = r&#39;C:\path\to\your\project\directory\md_data&#39;<br><br># Create the output directory if it doesn&#39;t exist<br>os.makedirs(output_directory, exist_ok=True)<br><br># Iterate over all files in the input directory<br>for filename in os.listdir(input_directory):<br>    if filename.endswith(&#39;.ipynb&#39;):<br>        # Read the notebook<br>        with open(os.path.join(input_directory, filename), &#39;r&#39;, encoding=&#39;utf-8&#39;) as f:<br>            nb = nbformat.read(f, as_version=4)<br>        <br>        # Create a corresponding markdown file<br>        md_filename = os.path.splitext(filename)[0] + &#39;.md&#39;<br>        with open(os.path.join(output_directory, md_filename), &#39;w&#39;, encoding=&#39;utf-8&#39;) as f:<br>            # Iterate over all cells in the notebook<br>            for cell in nb.cells:<br>                if cell.cell_type == &#39;code&#39;:<br>                    f.write(&#39;```python\n&#39;)<br>                    f.write(cell.source + &#39;\n&#39;)<br>                    f.write(&#39;```\n\n&#39;)<br>                elif cell.cell_type == &#39;markdown&#39;:<br>                    f.write(cell.source + &#39;\n\n&#39;)</pre><p>Let’s go, we have now converted all the files into markdown files (which are now stored in a separate directory but still inside the main project directory). Now we can use langchain to deal with these files and create vectors. But before that, let us first configure the API keys to access the models.</p><h4>Setting Up API Keys</h4><p>We create a .env file in our project directory. We use .env files to set all our important API keys as environment variables and handle them properly.</p><pre>#Structure of your .env file<br><br>HF_TOKEN = &lt;your Hugging Face token&gt;      #To access the embedding model<br>GOOGLE_API_KEY = &lt;Your API key&gt;           #To access gemini-1.5-pro<br>QDRANT_API_KEY = &lt;Your API key&gt;           #To access the Qdrant vectorDB<br>QDRANT_URL = &lt;URL of your vector store&gt;   #To access the URL of your vector store<br><br>#You can access your own API keys and tokens through their respective websites</pre><p>Great, now all the environment variables are set up. Now our next concern is to convert our documents into vectors and store them in the vector DB.</p><h4>Setting up the VectorStore</h4><p>Let’s set up our qdrant vector store and vectorized our knowledgebase</p><pre>import os<br>from dotenv import load_dotenv<br>from langchain_community.document_loaders import DirectoryLoader<br>from langchain_qdrant import Qdrant<br>from qdrant_client import QdrantClient<br>from langchain.embeddings import HuggingFaceInferenceAPIEmbeddings<br>from langchain_text_splitters import RecursiveCharacterTextSplitter<br><br>#Configuring API Keys<br>load_dotenv()<br>google_api_key = os.getenv(&#39;GOOGLE_API_KEY&#39;)<br>hf_token = os.getenv(&#39;HF_TOKEN&#39;)<br>qdrant_api_key = os.getenv(&#39;QDRANT_API_KEY&#39;)<br>qdrant_url = os.getenv(&quot;QDRANT_URL&quot;)<br><br>#Loading the knowledge using Directory Loader<br>data_dir = &#39;md_data&#39;<br>loader = DirectoryLoader(path = data_dir, glob= &quot;**/*.md&quot;, show_progress=True)<br>docs = loader.load()<br><br>#print(len(docs))<br><br>#Splitting the data<br>text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)<br>texts = text_splitter.split_documents(docs)<br><br>#Embedding Model <br>embedding = HuggingFaceInferenceAPIEmbeddings(api_key= hf_token)<br><br>#Indexing the documents in on VectorDB<br>qdrant = Qdrant.from_documents(<br>    docs,<br>    embedding,<br>    url=qdrant_url,<br>    prefer_grpc=True,<br>    api_key=qdrant_api_key,<br>    collection_name=&quot;langchat&quot;,<br>)<br><br>print(&quot;Indexing done!&quot;)</pre><p>The above code simply loads all the files of the knowledge base in the directory(using the <strong>DirectoryLoader </strong>class), splits the documents into several chunks, converts the documents into vectors, and then stores these vectors on our cloud-based vector store provided by <strong>Qdrant</strong>.</p><p>For this project, I have used <a href="https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2">sentence-transformers/all-MiniLM-L6-v2</a> embeddings (dimension size = 384)</p><p>Alright, now our vector store is ready to be used as context for our LLM to answer queries.</p><h4>Implementing the Retrieval Mechanism</h4><p>Once the Vectorstore is set up, we can now implement the retrieval mechanism</p><pre>import os<br>from dotenv import load_dotenv<br>from langchain_google_genai import ChatGoogleGenerativeAI<br>from langchain.embeddings import HuggingFaceInferenceAPIEmbeddings<br>from langchain_qdrant import Qdrant<br>from qdrant_client import QdrantClient<br>from langchain_core.output_parsers import StrOutputParser<br>from langchain_core.runnables import RunnablePassthrough, RunnableParallel<br>from langchain.prompts import PromptTemplate<br><br>#Configuring API keys<br>load_dotenv()<br>google_api_key = os.getenv(&#39;GOOGLE_API_KEY&#39;)<br>hf_token = os.getenv(&#39;HF_TOKEN&#39;)<br>qdrant_api_key = os.getenv(&#39;QDRANT_API_KEY&#39;)<br>qdrant_url = os.getenv(&#39;QDRANT_URL&#39;)<br><br>#Instantiating the LLM<br>chat = ChatGoogleGenerativeAI(model = &#39;gemini-1.5-pro-latest&#39;, google_api_key = google_api_key)<br><br>#Embedding Model <br>embedding = HuggingFaceInferenceAPIEmbeddings(api_key= hf_token)<br><br># Connect to Qdrant<br>qdrant_client = QdrantClient(<br>    url= qdrant_url,<br>    api_key= qdrant_api_key<br>)<br><br># Initialize the LangChain Qdrant wrapper<br>vectorstore = Qdrant(<br>    client=qdrant_client,<br>    collection_name=&quot;langchat&quot;,<br>    embeddings= embedding<br>)<br><br>#Setting up the retriever<br>retriever = vectorstore.as_retriever()<br><br>#Prompt Template<br>template = &#39;&#39;&#39;<br>You are an expert python developer. Using the provided context, answer the qeuries <br>accordingly generate relevant code according to the given context to answer the questions<br><br>{context}:<br>{Question}:<br><br>Answer:<br><br>&#39;&#39;&#39;<br>prompt_template = PromptTemplate.from_template(template=template)<br><br>#Setting up the chain<br>set_ret = RunnableParallel(<br>    {&quot;context&quot;: retriever, &quot;Question&quot;: RunnablePassthrough()} <br>)<br>rag_chain = set_ret |  prompt_template | chat | StrOutputParser()<br><br>#Defining the response function<br>def generate_response(text):<br>    response = rag_chain.invoke(text)<br>    return response<br><br>print(&quot;Querying done!&quot;)</pre><p>The above Python script implements the retrieval mechanism (as explained in the previous section). To improve the response of the LLM, we used a prompt template.</p><p><strong>Prompt templates </strong>are like mechanisms through which we instruct the LLM to do this or that and generate responses accordingly.</p><p>Here we have defined something known as ‘Chains’.</p><p><strong>Chains</strong>, as the name suggests, are the Langchain framework&#39;s fundamental components. Chains are used to define a certain sequence that is needed to generate a response. With the help of Chains, we implemented the whole logic behind the retrieval mechanism.</p><h4>User Interface</h4><p>Alright, we are almost there. We have implemented the backend side of things, and now to convert it into a fully functioning App, we need to develop its UI. For that, streamlit is the best option.</p><pre>#Python code to implement the UI of this application using streamlit <br>import streamlit as st <br>from querying import generate_response<br><br>st.title(&quot;🦜 LangChat&quot;)<br><br>if &quot;messages&quot; not in st.session_state.keys():<br>    st.session_state.messages = [{&quot;role&quot;: &quot;assistant&quot;, &quot;content&quot;: &quot;Chat with Langchain docs&quot;}]<br><br>for message in st.session_state.messages:<br>    with st.chat_message(message[&quot;role&quot;]):<br>        st.write(message[&quot;content&quot;])<br><br>if prompt := st.chat_input(&quot;Chat with langchain docs &quot;):<br>    # Display user message in chat message container<br>    st.chat_message(&quot;user&quot;).markdown(prompt)<br>    # Add user message to chat history<br>    st.session_state.messages.append({&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt})<br><br>    response = generate_response(prompt)<br>    # Display assistant response in chat message container<br>    with st.chat_message(&quot;assistant&quot;):<br>        st.markdown(response)<br>    # Add assistant response to chat history<br>    st.session_state.messages.append({&quot;role&quot;: &quot;assistant&quot;, &quot;content&quot;: response})</pre><p>Here, we used the chain that was defined earlier and integrated it with Streamlit UI.</p><p>This is it, we have now developed the UI. To test this app through a local machine:</p><pre>#Navigate to the terminal, run the following command:<br><br>C: path/to/your/project/directory&gt; streamlit run main.py</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vooIm72FQ0OKA_LbUP1-oA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/1*Mu39ZTictblwzXFr_kBnhA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a9KCUVtkToIHh2DKz1wF-g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4zEVFLtoeC72BFxmOA4-Qg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YBvYomzEGumfHzCZDdilww.png" /></figure><h3>Deployment</h3><pre>#Your Directory Structure shall look like this:<br><br>YourProjectDirectory<br>  |venv                 #Your Virtual environment<br>  |.env                 #.ENV file   <br>  |data                 #Original directory containing ipynb files<br>  |convert.py           #Python script containing the conversion logic<br>  |md_data              #Convereted markdown files<br>  |index.py             #Python script to set up the vectorDB<br>  |query.py  *          #Python script to set up the retrieval mechanism<br>  |main.py   *          #Main streamlit app<br>  |requirements.txt *   #dependencies<br>  <br>  <br><br># For deployment, we only have to push the files on our github repo <br># that are marked *</pre><p>Once the required files are pushed, we can easily deploy the App through the streamlit cloud. (Set up the (environment variables accordingly)</p><h3>Conclusion</h3><p>This is how we made a RAG-based QnA system over code documentation. Now instead of navigating through the entire documentation, we can just access the documentation efficiently with the help of LLMs.</p><p>Feel free to experiment with the same code over different documentation of different frameworks. Make sure the documents are in markdown format.</p><p>Try the App: <a href="https://langchat-ai.streamlit.app/">LangChat</a></p><p>Source Code: <a href="https://github.com/devroopsaha744/LangChat">GitHub</a></p><p>Thank You so much for reading this patiently. Do upvote, if you find this article useful.</p><h3>References</h3><p><a href="https://community.intel.com/t5/Blogs/Thought-Leadership/Big-Ideas/RAG-Infusing-Realism-and-Domain-Expertise-into-LLMs-but-could/post/1574493">https://community.intel.com/t5/Blogs/Thought-Leadership/Big-Ideas/RAG-Infusing-Realism-and-Domain-Expertise-into-LLMs-but-could/post/1574493</a></p><p><a href="https://codingscape.com/blog/rag-101-what-is-rag-and-why-does-it-matter&#39;">https://codingscape.com/blog/rag-101-what-is-rag-and-why-does-it-matter</a></p><p><a href="https://www.pinecone.io/learn/vector-embeddings/">https://www.pinecone.io/learn/vector-embeddings/</a></p><p><a href="https://medium.com/@akriti.upadhyay/efficient-information-retrieval-with-rag-workflow-afdfc2619171">https://medium.com/@akriti.upadhyay/efficient-information-retrieval-with-rag-workflow-afdfc2619171</a></p><p><a href="https://techcommunity.microsoft.com/t5/azure-data-explorer-blog/azure-data-explorer-for-vector-similarity-search/ba-p/3819626">https://techcommunity.microsoft.com/t5/azure-data-explorer-blog/azure-data-explorer-for-vector-similarity-search/ba-p/3819626</a></p><p><a href="https://zilliz.com/blog/similarity-metrics-for-vector-search">https://zilliz.com/blog/similarity-metrics-for-vector-search</a></p><p><a href="https://blog.langchain.dev/langchain-state-of-ai-2023/">https://blog.langchain.dev/langchain-state-of-ai-2023/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b9f2042031fe" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[FitVid: AI-powered visual gym]]></title>
            <link>https://medium.com/@datafreakai/fitvid-ai-powered-visual-gym-42336427f5c1?source=rss-71ae604d9df2------2</link>
            <guid isPermaLink="false">https://medium.com/p/42336427f5c1</guid>
            <dc:creator><![CDATA[datafreak]]></dc:creator>
            <pubDate>Wed, 05 Jun 2024 17:05:18 GMT</pubDate>
            <atom:updated>2024-06-05T19:52:02.929Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qKTX7vlVAAwZfCyz" /><figcaption>credits: Prexels</figcaption></figure><h3>Introduction</h3><p>This article explains my latest project: FitVid, an AI-powered visual gym that tracks the movements of the gym exercises, counts the repetitions, and helps improve the form of gym exercises using Pose Estimation.</p><p>This is a pure python project. I have used Mediapipe and openCV for this project.</p><p>Alright without wasting much time, let’s start with it, firstly the main concept behind it.</p><h3>The Concept</h3><p>Firstly, I used Mediapipe’s pose landmark detection, to detect the various joints in the human body. Medipipe’s inbuilt functions and pose detection utilities make it very easy to detect a whole human body and joint landmarks in the body.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/772/0*fl-LZSGWKAq5yNxZ" /><figcaption>BlazePose landmarks</figcaption></figure><p>The above diagram explains how the different joint landmarks are mapped using Mediapipe’s pose estimation model. These joint landmarks are then used to track the movements of the body.</p><p>Once the landmark points of the joints are calculated, angle heuristics and movements between these landmark coordinates are tracked to determine the different movements associated with varying exercises in the gym.</p><p>Fine, we can now track the movement of the body using joint landmarks but how to count the reps or how to improve the ‘form’ or technique of that particular movement? Well for that, angle heuristics have been used. Now what do I mean by ‘angle heuristics’. Angle heuristics simply means the calculation of angles between joints and applying some conditions on the calculated angle. The conditions imposed on the calculated angle help in improving the technique of that particular movement and also help in counting the repetitions.</p><p>For example, to track bicep curls and count the curls, the angle between the joints numbered 11, 13, and 15 (Left Shoulder, Left elbow, Left wrist) is calculated (For now only considered one side for the sake of simplicity). Now the rep will only be counted once the angle is below a certain value let’s say 40 degrees. Thus, the model will track the entire movement and calculate the angle, but the rep will only be counted once the angle is below 40 degrees. other than the angle, we also set a boolean variable named: ‘stage’. Initially, stage is set to 0 (DOWN) as the angle has not met the condition. Once that particular condition is met, stage is set to 1 (UP) and that rep is counted.</p><p>Similar conditions and heuristics are applied for different types of movements and exercises. For example, to track lateral raises, the angle between the hip, shoulder, and wrist is calculated and conditions are imposed accordingly. Similarly, for crunches, the angle between shoulders, hip, and knee is tracked or for squats, the angle between hip, knee, and ankle is tracked and conditions are imposed on the tracked angle accordingly.</p><h3>A very Genuine Question</h3><p>Now while reading my approach, you should ask a question:</p><p>“<em>How is it helpful to improve the technique of that particular gym movement, how does it make sense?”</em></p><p>Well, it’s a genuine question. So, as I mentioned earlier, we are applying certain conditions to the movements of the joints. Now that particular movement has to meet that condition to count it as a full rep or curl. This thing helps in improving the technique of that specific movement and forces that person to improve his/her technique of that movement to contribute it to the total rep count.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-1EuYbFoe-fTPKJJ.jpg" /><figcaption>credits: BarBend</figcaption></figure><h3>Implementation</h3><p>I implemented this project on a virtual environment via vscode. The Python dependencies used for this project are Mediapipe, and OpenCV.</p><p>To install the above dependencies, just execute the following pip command on your virtual environment:</p><pre>pip install mediapipe opencv-python</pre><p>Alright, let’s quickly jump to the exciting part</p><p><strong>Importing dependencies:</strong></p><pre>import mediapipe as mp<br>import cv2<br>import numpy as np<br><br>mp_drawing = mp.solutions.drawing_utils<br>mp_pose = mp.solutions.pose</pre><p>I used Mediapipe to deal with pose estimation stuff and openCV to track and the body movements and to put text. And used numpy for angle heuristics.</p><p><strong>mp_drawing</strong>, <strong>mp_pose</strong> stores all the necessary utilities to track landmark joints and body movements involving the joints.</p><p>Alright, let’s quickly test the pose estimation utilities and track/detect the landmark joints:</p><pre>#Testing detections<br><br>cap = cv2.VideoCapture(0)<br>## Setup mediapipe instance<br>with mp_pose.Pose(min_detection_confidence=0.5, min_tracking_confidence=0.5) as pose:<br>    while cap.isOpened():<br>        ret, frame = cap.read()<br>        <br>        # Recolor image to RGB<br>        image = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)<br>        image.flags.writeable = False<br>      <br>        # Make detection<br>        results = pose.process(image)<br>    <br>        # Recolor back to BGR<br>        image.flags.writeable = True<br>        image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)<br>        <br>        # Render detections<br>        mp_drawing.draw_landmarks(image, results.pose_landmarks, mp_pose.POSE_CONNECTIONS,<br>                                mp_drawing.DrawingSpec(color=(245,117,66), thickness=2, circle_radius=2), <br>                                mp_drawing.DrawingSpec(color=(245,66,230), thickness=2, circle_radius=2) <br>                                 )               <br>        <br>        cv2.imshow(&#39;FitVid&#39;, image)<br><br>        if cv2.waitKey(10) &amp; 0xFF == ord(&#39;q&#39;):<br>            break<br><br>    cap.release()<br>    cv2.destroyAllWindows()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*Q8K2hLt9e4oJFwXcgALHYQ.png" /></figure><p>Fine, we can now detect and track the joint landmarks,</p><p>but how to get the specific coordinates?</p><pre>landmarks = results.pose_landmarks.landmark<br>            <br># Get coordinates<br>shoulder = [landmarks[mp_pose.PoseLandmark.LEFT_SHOULDER.value].x,landmarks[mp_pose.PoseLandmark.LEFT_SHOULDER.value].y]<br>hip = [landmarks[mp_pose.PoseLandmark.LEFT_HIP.value].x,landmarks[mp_pose.PoseLandmark.LEFT_HIP.value].y]<br>wrist = [landmarks[mp_pose.PoseLandmark.LEFT_WRIST.value].x,landmarks[mp_pose.PoseLandmark.LEFT_WRIST.value].y]</pre><p>This above code snippet not only detects the landmark joints but also calculates the coordinates of specific joints and stores them on the list.</p><p><strong>for eg:</strong> The code snippet below gets the coordinates of the left shoulder and stores them on the list named ‘shoulder’. Thus, we can use different code names like ‘RIGHT_WRIST’ or ‘LEFT_KNEE’ to get the coordinates of that specific joint.</p><pre>shoulder = [landmarks[mp_pose.PoseLandmark.LEFT_SHOULDER.value].x,landmarks[mp_pose.PoseLandmark.LEFT_SHOULDER.value].y]</pre><p>Nice, now we can get the specific coordinates of the joints.</p><p>But why we need the coordinates? To calculate the angles between the joints. To understand the logic, let’s dive into the math:</p><p>Suppose we have three points: A ( a[0], a[1]), B(b[0], b[1]), C(c{0], c[1])</p><p>We can use simple vectors and Trig to find the angle A<strong>B</strong>C. We can get the vectors BA and BC through the points A, B, and C and then simply find the angle between <strong>BA</strong> and <strong>BC</strong> by using the dot product.</p><pre>#Logic  to calculate angle between coordinates<br><br>def calculate_angle(a, b, c):<br>    # Convert points to numpy arrays<br>    A = np.array(a)<br>    B = np.array(b)<br>    C = np.array(c)<br>    <br>    # Vectors BA and BC<br>    BA = A - B<br>    BC = C - B<br>    <br>    # Dot product of BA and BC<br>    dot_product = np.dot(BA, BC)<br>    <br>    # Magnitudes of BA and BC<br>    magnitude_BA = np.linalg.norm(BA)<br>    magnitude_BC = np.linalg.norm(BC)<br>    <br>    # Cosine of the angle<br>    cos_theta = dot_product / (magnitude_BA * magnitude_BC)<br>    <br>    # Angle in radians<br>    theta_radians = np.arccos(cos_theta)<br>    <br>    # Convert angle to degrees<br>    theta_degrees = np.degrees(theta_radians)<br>    <br>    return theta_degrees</pre><p>Alright, now we know how to calculate angles. Now we are ready to apply it to track specific gym exercise movements.</p><p>Suppose we want to track bicep curls and measure angles. The code snippet achieves it.</p><pre>cap = cv2.VideoCapture(0)<br>## Setup mediapipe instance<br>with mp_pose.Pose(min_detection_confidence=0.5, min_tracking_confidence=0.5) as pose:<br>    while cap.isOpened():<br>        ret, frame = cap.read()<br>        <br>        # Recolor image to RGB<br>        image = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)<br>        image.flags.writeable = False<br>      <br>        # Make detection<br>        results = pose.process(image)<br>    <br>        # Recolor back to BGR<br>        image.flags.writeable = True<br>        image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)<br>        <br>        # Extract landmarks<br>        try:<br>            landmarks = results.pose_landmarks.landmark<br>            <br>            # Get coordinates<br>            shoulder = [landmarks[mp_pose.PoseLandmark.LEFT_SHOULDER.value].x,landmarks[mp_pose.PoseLandmark.LEFT_SHOULDER.value].y]<br>            hip = [landmarks[mp_pose.PoseLandmark.LEFT_HIP.value].x,landmarks[mp_pose.PoseLandmark.LEFT_HIP.value].y]<br>            wrist = [landmarks[mp_pose.PoseLandmark.LEFT_WRIST.value].x,landmarks[mp_pose.PoseLandmark.LEFT_WRIST.value].y]<br>            <br>            # Calculate angle<br>            angle = calculate_angle(wrist, shoulder, hip)<br>            <br>            # Visualize angle<br>            cv2.putText(image, str(angle), <br>                           tuple(np.multiply(shoulder, [640, 480]).astype(int)), <br>                           cv2.FONT_HERSHEY_SIMPLEX, 0.5, (255, 255, 255), 2, cv2.LINE_AA<br>                                )<br>                       <br>        except:<br>            pass<br>        <br>        <br>        # Render detections<br>        mp_drawing.draw_landmarks(image, results.pose_landmarks, mp_pose.POSE_CONNECTIONS,<br>                                mp_drawing.DrawingSpec(color=(245,117,66), thickness=2, circle_radius=2), <br>                                mp_drawing.DrawingSpec(color=(245,66,230), thickness=2, circle_radius=2) <br>                                 )               <br>        <br>        cv2.imshow(&#39;FitVid&#39;, image)<br><br>        if cv2.waitKey(10) &amp; 0xFF == ord(&#39;q&#39;):<br>            break<br><br>    cap.release()<br>    cv2.destroyAllWindows()</pre><p>This above code is nothing just mixed everything mentioned above. In this code, pose detection utilities are applied, joint landmarks are tracked and the angle is calculated. Here, we are trying to detect the bicep curls only. For other gym exercise movements, we shall consider other landmark joints.</p><p>But how to count the reps? We set a counter variable and initialise it to 0. We also set a variable name ‘stage’ to 0 (down). Now the angle which we calculating using the ‘calculate_angle’ function on the fetched coordinates, has to meet a certain condition. That condition depends upon different Gym movements. Once the condition is met, the counter is incremented. Thus, the reps are counted.</p><p><strong>For Bicep Curls</strong>, this condition needs to be added:</p><pre># Extract landmarks<br>        try:<br>            landmarks = results.pose_landmarks.landmark<br>            <br>            # Get coordinates<br>            shoulder = [landmarks[mp_pose.PoseLandmark.LEFT_SHOULDER.value].x,landmarks[mp_pose.PoseLandmark.LEFT_SHOULDER.value].y]<br>            elbow = [landmarks[mp_pose.PoseLandmark.LEFT_ELBOW.value].x,landmarks[mp_pose.PoseLandmark.LEFT_ELBOW.value].y]<br>            wrist = [landmarks[mp_pose.PoseLandmark.LEFT_WRIST.value].x,landmarks[mp_pose.PoseLandmark.LEFT_WRIST.value].y]<br>            <br>            # Calculate angle<br>            angle = calculate_angle(shoulder, elbow, wrist)<br>            <br>            # Curl counter logic<br>            if angle &gt; 130:<br>                stage = &quot;down&quot;<br>            if angle &lt; 50 and stage ==&#39;down&#39;:<br>                stage=&quot;up&quot;<br>                counter +=1<br>                print(counter)<br>        except:<br>             pass</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/956/1*gXDU3FG04ESWcZdc_tXgkw.png" /></figure><p><strong>NOTE</strong>: The above code snippet is to track and count bicep curls. This much code snippet will change for different gym movements. For example,</p><p><strong>For Tricep Pushdowns</strong></p><pre>#For Tricep Pushdowns<br><br>        try:<br>            landmarks = results.pose_landmarks.landmark<br>            <br>            # Get coordinates<br>            shoulder = [landmarks[mp_pose.PoseLandmark.RIGHT_SHOULDER.value].x,landmarks[mp_pose.PoseLandmark.RIGHT_SHOULDER.value].y]<br>            elbow = [landmarks[mp_pose.PoseLandmark.RIGHT_ELBOW.value].x,landmarks[mp_pose.PoseLandmark.RIGHT_ELBOW.value].y]<br>            wrist = [landmarks[mp_pose.PoseLandmark.RIGHT_WRIST.value].x,landmarks[mp_pose.PoseLandmark.RIGHT_WRIST.value].y]<br>            <br>            # Calculate angle<br>            angle = calculate_angle(shoulder, elbow, wrist)<br>                       <br>            # Curl counter logic<br>            if angle &lt; 50:<br>                stage = &quot;down&quot;<br>            if angle &gt; 130 and stage ==&#39;down&#39;:<br>                stage=&quot;up&quot;<br>                counter +=1<br>                print(counter)<br>                       <br>        except:<br>            pass</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/928/1*6KCybusR2O38m_6jhzRGUA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/914/1*ClkyWXsCO5tQrmyl2DNtIA.png" /></figure><p><strong>For Lateral Raises:</strong></p><pre># For Lateral Raises<br><br># Extract landmarks<br>        try:<br>            landmarks = results.pose_landmarks.landmark<br>            <br>            # Get coordinates<br>            shoulder = [landmarks[mp_pose.PoseLandmark.LEFT_SHOULDER.value].x,landmarks[mp_pose.PoseLandmark.LEFT_SHOULDER.value].y]<br>            hip = [landmarks[mp_pose.PoseLandmark.LEFT_HIP.value].x,landmarks[mp_pose.PoseLandmark.LEFT_HIP.value].y]<br>            wrist = [landmarks[mp_pose.PoseLandmark.LEFT_WRIST.value].x,landmarks[mp_pose.PoseLandmark.LEFT_WRIST.value].y]<br>            <br>            # Calculate angle<br>            angle = calculate_angle(hip, shoulder, wrist)<br>            <br>            # rep counter logic<br>            if angle &lt; 50:<br>                stage = &quot;down&quot;<br>            if angle &gt; 50 and stage ==&#39;down&#39;:<br>                stage=&quot;up&quot;<br>                counter +=1<br>                print(counter)<br>        <br>         except:<br>            pass</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/951/1*7uVkn3TNv64BV05DM67-Pw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/968/1*HC0PIBLkFJuFFIr5qirbig.png" /></figure><p><strong>Similarly for Crunches:</strong></p><pre># For Crunches<br><br>        # Extract landmarks<br>        try:<br>            landmarks = results.pose_landmarks.landmark<br>            <br>            # Get coordinates<br>            hip = [landmarks[mp_pose.PoseLandmark.LEFT_HIP.value].x,landmarks[mp_pose.PoseLandmark.LEFT_HIP.value].y]<br>            knee = [landmarks[mp_pose.PoseLandmark.LEFT_KNEE.value].x,landmarks[mp_pose.PoseLandmark.LEFT_KNEE.value].y]<br>            shoulder = [landmarks[mp_pose.PoseLandmark.LEFT_SHOULDER.value].x,landmarks[mp_pose.PoseLandmark.LEFT_SHOULDER.value].y]<br>            <br>            # Calculate angle<br>            angle = calculate_angle(shoulder, hip, knee)<br>            <br>            # rep counter logic<br>            if angle &lt; 120:<br>                stage = &quot;down&quot;<br>            if angle &gt; 150 and stage ==&#39;down&#39;:<br>                stage=&quot;up&quot;<br>                counter +=1<br>                print(counter)<br>                       <br>        except:<br>            pass</pre><p>Thus, we can write conditions for different gym movements.</p><p>So, the <strong>whole</strong> <strong>code</strong> is (BICEP CURLS):</p><pre>#Rep counter for BicepCurl<br>cap = cv2.VideoCapture(0)<br><br># Curl counter variables<br>counter = 0 <br>stage = None<br><br>## Setup mediapipe instance<br>with mp_pose.Pose(min_detection_confidence=0.5, min_tracking_confidence=0.5) as pose:<br>    while cap.isOpened():<br>        ret, frame = cap.read()<br>        <br>        # Recolor image to RGB<br>        image = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)<br>        image.flags.writeable = False<br>      <br>        # Make detection<br>        results = pose.process(image)<br>    <br>        # Recolor back to BGR<br>        image.flags.writeable = True<br>        image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)<br>        <br>        # Extract landmarks<br>        try:<br>            landmarks = results.pose_landmarks.landmark<br>            <br>            # Get coordinates<br>            shoulder = [landmarks[mp_pose.PoseLandmark.LEFT_SHOULDER.value].x,landmarks[mp_pose.PoseLandmark.LEFT_SHOULDER.value].y]<br>            elbow = [landmarks[mp_pose.PoseLandmark.LEFT_ELBOW.value].x,landmarks[mp_pose.PoseLandmark.LEFT_ELBOW.value].y]<br>            wrist = [landmarks[mp_pose.PoseLandmark.LEFT_WRIST.value].x,landmarks[mp_pose.PoseLandmark.LEFT_WRIST.value].y]<br>            <br>            # Calculate angle<br>            angle = calculate_angle(shoulder, elbow, wrist)<br>            <br>            # Visualize angle<br>            cv2.putText(image, str(angle), <br>                           tuple(np.multiply(elbow, [640, 480]).astype(int)), <br>                           cv2.FONT_HERSHEY_SIMPLEX, 0.5, (255, 255, 255), 2, cv2.LINE_AA<br>                                )<br>            <br>            # Curl counter logic<br>            if angle &gt; 130:<br>                stage = &quot;down&quot;<br>            if angle &lt; 50 and stage ==&#39;down&#39;:<br>                stage=&quot;up&quot;<br>                counter +=1<br>                print(counter)<br>                       <br>        except:<br>            pass<br>        <br>        # Render curl counter<br>        # Setup status box<br>        cv2.rectangle(image, (0,0), (225,73), (245,117,16), -1)<br>        <br>        # Rep data<br>        cv2.putText(image, &#39;REPS&#39;, (15,12), <br>                    cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0,0,0), 1, cv2.LINE_AA)<br>        cv2.putText(image, str(counter), <br>                    (10,60), <br>                    cv2.FONT_HERSHEY_SIMPLEX, 2, (255,255,255), 2, cv2.LINE_AA)<br>        <br>        # Stage data<br>        cv2.putText(image, &#39;STAGE&#39;, (65,12), <br>                    cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0,0,0), 1, cv2.LINE_AA)<br>        cv2.putText(image, stage, <br>                    (60,60), <br>                    cv2.FONT_HERSHEY_SIMPLEX, 2, (255,255,255), 2, cv2.LINE_AA)<br>        <br>        <br>        # Render detections<br>        mp_drawing.draw_landmarks(image, results.pose_landmarks, mp_pose.POSE_CONNECTIONS,<br>                                mp_drawing.DrawingSpec(color=(245,117,66), thickness=2, circle_radius=2), <br>                                mp_drawing.DrawingSpec(color=(245,66,230), thickness=2, circle_radius=2) <br>                                 )               <br>        <br>        cv2.imshow(&#39;FitVid&#39;, image)<br><br>        if cv2.waitKey(10) &amp; 0xFF == ord(&#39;q&#39;):<br>            break<br><br>    cap.release()<br>    cv2.destroyAllWindows()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*5gaCOpnY1tk8UqHys7kN8Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/961/1*kR-cLXKkLGCEAMSFcAAroQ.png" /></figure><p>As I said earlier, you can change the joints and angle conditions to track and count reps for different gym exercises. The above example is for Bicep Curls.</p><h3>Pushing/Pressing Movements (A special case):</h3><p>For Pushing movements, like Dumble press, bench press etc, We cannot apply angle heuristics to ‘improve’ the technique and count reps. For that, we impose conditions on Euclidean distance metric between the fetched coordinates of the joints (especially, the Wrist and shoulder joints). Then apply conditions on the calculated Euclidean distance for rep counting.</p><p>The function for Euclidean distance:</p><pre>#Logic to calculate distance between coordinates<br>def calculate_distance(a,b):<br>    a = np.array(a)<br>    b = np.array(b)<br><br>    distance = np.sqrt((a[0] - b[0])**2 + (a[1] - b[1])**2)<br>    return distance</pre><p>Nice, now we can get the distance between the joints using this function and apply the conditions accordingly to track push/pressing movements.</p><pre>wrist = [landmarks[mp_pose.PoseLandmark.LEFT_WRIST.value].x,landmarks[mp_pose.PoseLandmark.LEFT_WRIST.value].y]<br>shoulder = [landmarks[mp_pose.PoseLandmark.LEFT_SHOULDER.value].x,landmarks[mp_pose.PoseLandmark.LEFT_SHOULDER.value].y]<br># Calculate angle<br>distance = calculate_distance(wrist, shoulder)<br># Visualize angle<br>cv2.putText(image, str(distance),<br>tuple(np.multiply(wrist, [640, 480]).astype(int)),<br>cv2.FONT_HERSHEY_SIMPLEX, 0.5, (255, 255, 255), 2, cv2.LINE_AA<br>)<br># Rep counter logic<br>if distance &lt; 0.50:<br>          stage = &quot;down&quot;<br>if distance &gt; 0.50 and stage ==&#39;down&#39;:<br>          stage=&quot;up&quot;<br>          counter +=1<br>          print(counter)</pre><p>The above code is to track and count the reps for pressing exercises like shoulder presses or bench presses. It fetches the coordinates of the wrist and shoulders, and the Euclidean distance between the coordinates is calculated thus the conditions are imposed accordingly to count the reps and ‘improve’ the form of that particular exercise.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/918/1*VjrrShD2NRY5WZTT7YXCpA.png" /></figure><p>The rest of the code that involved detecting the joint landmarks and openCV remains the same, only the type of joint and specific condition changed.</p><p>PS: Please don’t judge me, I don’t have a dumbell 😭</p><h3>Conclusion</h3><p>And here we go, thus, we created a visual gym that helps in tracking the movements of the exercises and improves the technique of that particular movement based on the conditions imposed on the movement.</p><p>Thank You so much for reading so patiently. Do consider upvoting the blog if you find it useful.</p><p>And to contribute to the project or try it, Here is the link to the GitHub Repo:</p><p><a href="https://github.com/devroopsaha744/FitVid">GitHub - devroopsaha744/FitVid: This is AI powered video based gym trainer, that monitors the form of gym exercises.</a></p><p>Also, Do drop suggestions on how to improve the project or the approach.</p><p>Sayonara!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=42336427f5c1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Comprehensive Guide to CNNs Part-1: The Fundamentals]]></title>
            <link>https://medium.com/@datafreakai/a-comprehensive-guide-to-cnns-part-1-the-fundamentals-f056a9accbbb?source=rss-71ae604d9df2------2</link>
            <guid isPermaLink="false">https://medium.com/p/f056a9accbbb</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[datafreak]]></dc:creator>
            <pubDate>Fri, 26 Jan 2024 10:30:37 GMT</pubDate>
            <atom:updated>2024-01-26T13:40:05.007Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/581/0*048u_UdeXt-xPAIZ.png" /><figcaption>source: <a href="https://medium.com/techiepedia/binary-image-classifier-cnn-using-tensorflow-a3f5d6746697">https://medium.com/techiepedia/binary-image-classifier-cnn-using-tensorflow-a3f5d6746697</a></figcaption></figure><p>So, What is CNN? In fact, what is a Neural Network for that matter? Well, a Neural network is like an artificial brain that can intelligently do stuff.</p><p>Computer Scientists have created various types of Neural Networks to perform various tasks. One such type of Neural Network is called a <strong>Convolutional Neural Network</strong> or <strong>CNN. </strong>In this article, we will discuss the fundamentals of CNN, its components, working, and its architecture.</p><p>Pre-requisites of the Article: Artificial Neural Networks (Working and the math)</p><p>Let’s begin!</p><h3><strong>What is a CONVOLUTIONAL NEURAL NETWORK?</strong></h3><p>A Convolutional Neural Network or CNN is a special type of neural network dealing with image or video data. Any task related to images (or videos) can be Image classification, face recognition, object detection, medical image analysis, any task if it is related to images, CNN will do it!</p><p>As we know it’s called a ‘Convolutional<em>’</em> Neural Network. Well, what is Convolution? In simple terms, a ‘Convolution’ is a mathematical operation between two matrices that generates a new matrix. It is somewhat similar to matrix multiplication. This Convolution operation is what makes CNN so good while dealing with Image data.</p><p>The basic workflow of a CNN is, that we take an image, and apply convolution operation and all that related jazz on that image, due to which the image is converted into such a form that can be put into an Artificial Neural Network. This ability to deal with image-related data is what sets CNN apart from an ANN, as ANN alone cannot handle Image data, it needs the input to be in numerical format.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/0*IsU7wMiC_ZIRn23_.png" /><figcaption>basic workflow of CNN (source: <a href="https://journalofbigdata.springeropen.com/articles/10.1186/s40537-021-00444-8">https://journalofbigdata.springeropen.com/articles/10.1186/s40537-021-00444-8</a>)</figcaption></figure><p>We just put an image into CNN, it figures it automatically how to deal with it. How convenient is it? No boring preprocessing, No OpenCV. I mean there is a valid reason why CNNs have so many real-world applications, from Self-driving cars to Medical imaging, it is used everywhere!</p><p>Ok, so now we know that CNN is good at handling Image-related data, but how does CNN do this? Let’s find out!</p><p>But before understanding how CNN works let&#39;s understand how Images are represented digitally.</p><h3>Images and their representation</h3><p>To put it simply, An Image is a large square divided into several sub-squares, each subsquare is called a P<strong>ixel</strong>. Pixels are the basic building blocks of the image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/451/0*kHNJdeipLcK5Pm6s.jpg" /><figcaption>image representation (source: <a href="https://bournetocode.com/projects/GCSE_Computing_Fundamentals/pages/3-3-6-representing_images.html">https://bournetocode.com/projects/GCSE_Computing_Fundamentals/pages/3-3-6-representing_images.html</a>)</figcaption></figure><p>Every pixel has an assigned pixel value (or set of values). These pixel values describe the intensity and color at that point. The subtle combination of these pixels in a structured manner is what makes an image unique.</p><p>Now, there are two types of images: 1. Grayscale 2. RGB. These are nothing but fancy scientific terms for a Black and white image and a colored image.</p><p>For a grayscale image, Each pixel in a grayscale image is represented by a single value that denotes its brightness or intensity. The value typically ranges from 0 (black) to 255 (white) in an 8-bit grayscale image. Grayscale images have only one channel, representing the intensity of light. It’s like looking at the world through a single lens that captures only the brightness.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/397/0*9iwH0ebD6mQAUFt8" /><figcaption>a grayscale image representation (source: Analytics Vidhya)</figcaption></figure><p>RGB images are full-color images that include red, green, and blue color channels. Each pixel in an RGB image is represented by three values, one for each color channel (red, green, and blue). Each of these values can range from 0 to 255, determining the intensity of that color in the pixel. RGB images have three channels, and when combined, these channels create a wide spectrum of colors. It’s like having three lenses (red, green, and blue) that capture different aspects of the scene. Combinations of pixel values at Red, Blue, and Green channels form a Coloured image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/0*Zi-l-9dPV-XBM5S3" /><figcaption>RGB image representation (source: GeekForGeek)</figcaption></figure><h3>Components of CNN</h3><p>On a bigger picture, a CNN has three layers:</p><ol><li><strong>Convolution Layer</strong></li><li><strong>Pooling Layer</strong></li><li><strong>Fully Connected Layer</strong></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LdLiIZIW1K9huqJe.png" /><figcaption>source: interviewbit.com</figcaption></figure><p>The above 2 layers — The convolution layer and the Pooling layer are what make CNNs different from ANNs. It’s the combination of the first two layers with ANN, ie. the Fully Connected Layer.</p><p>Let’s study each of them one by one in detail.</p><h3>Convolution Layer</h3><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*PU3VnSPfyJ_UnxkB.gif" /><figcaption>Convolution operation (source: <a href="https://towardsdatascience.com/convolutional-neural-networks-explained-9cc5188c4939">https://towardsdatascience.com/convolutional-neural-networks-explained-9cc5188c4939</a>)</figcaption></figure><p>In this layer, that special ‘Convolution’ operation takes place. (Well, duh!)</p><p>But what is Convolution and How it works?</p><p>Suppose we have two matrices M1 and M2. And let’s say the convolution of M1 and M2 produces an output matrix M3. ie:</p><p>M1 * M2 = M3</p><p>So, to calculate M1 * M2 (Convolution of M1 and M2), we place M2 over the top left corner of M1, do element-wise multiplication of those elements that M2 covers at M1, and sum these products. This alone would be the one element of M3. Then we shift M2 by 1 column and do the same thing. This would be the next element of M3. In this way, all the elements of M3 are calculated. This is the essence of Convolution in the context of matrices.</p><p>Now, why are we talking about matrices? Remember, we talked about the representation of images in the above section? Yes, an image (grayscale) is a matrix of pixel values. So, it acts as M1. What about M2? We use a special type of matrix called ‘Filter’ over images, these act as M2. The convolution of Image (I) with a filter (F) results in the formation of Feature Map (M), which acts as M3.</p><p>So, just like the above general example, a filter will be placed over the top left corner of the image, perform element-wise products of coincided elements and then these are summed. Then, the filter will get shifted by one pixel, and do the same thing over the image, to produce a third type of matrix called a Feature Map.</p><p>So: I * F = M (Image * Filter = Feature Map)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*qaQdkZ1xB8lIxmU2.gif" /><figcaption>Animation for Convolution operation</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/839/0*r_wfc7YuiOyI1GG-.png" /><figcaption>math behind convolution. (Source: Deep Learning by Ian Goodfellow, Yoshua Bengio, and Aaron Courville)</figcaption></figure><p>Wait, I get it, so much to take in. First of all, what are filters? Well, Filters (or kernels) act as ‘feature detectors’. These are used to find certain patterns or shapes in an image. These filters can be of different types. Some types of filters are used to find only edges, while others are used to find any corners or other complex patterns in the image.</p><p>By the way, these filters are just some special matrices with fixed numerical pixel values as their elements.</p><p>When these filters are convolved over an image, a feature map is produced, which tells the presence of these features or patterns in these images. These feature maps are the outputs of the convolution layer, while acting as inputs for the next layer.</p><p>In layman’s terms, a filter is used to find patterns in the image while feature maps tell where these patterns are in the image.</p><p><strong>Note</strong>: Multiple Filters or kernels are going to be used to capture every kind of pattern in the image. There are different kinds of filters to identify different kinds of patterns in images.</p><p>These patterns we are talking about are nothing but subtle combinations and arrangements of pixels in there image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*jxRu4tk5jbqkRYnL.png" /><figcaption>source: <a href="https://kharshit.github.io/blog/2018/12/14/filters-in-convolutional-neural-networks">https://kharshit.github.io/blog/2018/12/14/filters-in-convolutional-neural-networks</a></figcaption></figure><p>For example, to detect different types of edges in the image, we use a specific kind of filter called ‘The Sobel Filter’. Similarly, other kinds of filters exist to detect other kinds of patterns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/455/0*LvMES-VBaN_nZLLs.gif" /><figcaption>sobel filter (source: <a href="https://homepages.inf.ed.ac.uk/rbf/HIPR2/sobel.htm">https://homepages.inf.ed.ac.uk/rbf/HIPR2/sobel.htm</a>)</figcaption></figure><p>I won’t go into the details of it, to study this, refer to this article:</p><p><a href="https://kharshit.github.io/blog/2018/12/14/filters-in-convolutional-neural-networks">https://kharshit.github.io/blog/2018/12/14/filters-in-convolutional-neural-networks</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jKSrr-x3vZya9GiN.jpg" /><figcaption>a sample image</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TveFTa7n57lsfB6y.png" /><figcaption>illustration of edge detection in action using Sobel filter</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2k4SET_ykmHqR_sT.png" /><figcaption>source: <a href="https://kharshit.github.io/blog/2018/12/14/filters-in-convolutional-neural-networks">https://kharshit.github.io/blog/2018/12/14/filters-in-convolutional-neural-networks</a></figcaption></figure><p>To study more deeply about the types of filters used, kindly refer to this amazing article: <a href="https://iq.opengenus.org/convolution-filters/">https://iq.opengenus.org/convolution-filters/</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D2AgoAMdgrhkbiqkxHQanw.png" /></figure><p><strong>Note:</strong> In convolutional neural networks, the filters are learned the same way as hyperparameters through backpropagation during the training process.</p><p>Okay, so now we know the fundamentals of Convolution. But this is not the end. To make the model capture complex patterns and features, we need to pass the feature map to an activation function. The most common (in fact, the only) activation function used in the convolution layer is ReLU. Rectified Linear Unit, is an activation function used to capture complex patterns in any type of problem, hence trendy in various Neural network architectures. Mathematically it is defined as:</p><pre>def ReLU(x):<br>if x &gt;= 0:<br> return x<br>else:<br> return 0</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/274/0*0BXU9KHv8sKlkAiH.png" /></figure><p>To study more about ReLU and other activation functions, kindly refer to this article:</p><p><a href="https://www.analyticsvidhya.com/blog/2020/01/fundamentals-deep-learning-activation-functions-when-to-use-them/">https://www.analyticsvidhya.com/blog/2020/01/fundamentals-deep-learning-activation-functions-when-to-use-them/</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w__N1IC_EUZI02FjWkbfbA.png" /><figcaption>Convolution Layer (Source: resaerchgate)</figcaption></figure><p>In the case of RGB images, everything is the same just the dimensions are more. Instead of talking in terms of matrices, we talk with tensors.</p><p>Alright now, we studied different components of the convolution layer. Before jumping into the next layer, we shall also discuss about two important concepts: 1. Padding 2. Striding. These ‘techniques’ can be used to generate more accurate feature maps during the convolution process. Let’s study them in detail.</p><h4>Padding</h4><p>Are you familiar with Padding in CSS? If yes, that’s pretty much what Padding is in the context of Convolutional Neural Networks. Padding, in layman’s terms, is adding ‘zero-valued’ pixels on all four sides of the image, then performing convolution operation over THERE. By this, the features at the edges of the images are more involved and thus are more considered in the feature map. In traditional convolution, the pixels at the borders are less involved in the convolution, thus less prevalent in the feature map, padding solves this problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/721/1*J7vXkIO6FOHAS-fAPtjw5Q.png" /><figcaption>source: d2l.ai</figcaption></figure><p>You have noticed that in traditional convolution, the dimension of the feature map is less than that of the dimension of the image. Now in a practical scenario, we apply multiple convolution layers. Like, if an image is passed through multiple convolution layers, the size of the final feature map will be much smaller. So, to deal with this problem, padding is applied when the filter or kernel is convolved over the image. Thus, the size of the feature map produced is not reduced and the important patterns from the image are not missed. Padding helps control the dimension of the feature map and thus, helps preserve important features and patterns in the image during convolution.</p><p>Generally, one layer of padding on the image is enough for most of the problems. Two or more layers of padding on the image borders are rarely used (or not used at all) for any problem</p><p>Nice, so we discussed about padding. Let’s also discuss another important technique like padding which is used to control the dimension of the feature map: <strong>Striding</strong></p><h4>Striding</h4><p>It decides, by how many units the filter (or kernel) will shift over the image after computing the first set of dot products (element-wise products). In traditional convolutions, it is set to 1. But we can set it to other values and based on that dimensions of the feature map is controlled.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/924/1*ZApmZXVjsSiUaZO-Q9hf6A.png" /><figcaption>courtesy: Ting-Hao-Chen (Machine-Learning-Notes)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/1*d9bEwAdxRtY6DbSA9Y9G-A.png" /><figcaption>courtesy: Ting-Hao-Chen (Machine-Learning-Notes)</figcaption></figure><p>See, it&#39;s as simple as that. However, the default value is still considered for most of the architectures. As we increase striding, the dimension of the feature map keeps getting reduced. For higher values of striding, the filters will miss some pixels while convolving over the image, thus size reduction and potentially missing some important patterns from the image. So, either a proper tradeoff between padding value and stride value is considered or the default is taken (the latter case gives better results, empirically speaking).</p><p>Let’s see how the dimensions of the feature map are controlled by padding and striding mathematically in convolution operation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/1*R6fIM1AljB30mp846MA61g.png" /><figcaption>Dimension of feature when padding and stridiing is involved in convolution</figcaption></figure><p>Here,</p><p>n = &gt; Dimension of image. (n x n)</p><p>f =&gt; Dimension of filter (f x f)</p><p>p =&gt; Padding value (or the number of layers in padding, default = 0)</p><p>s =&gt; Striding value (default = 1)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/459/1*26ZXO5ur6oBLVkP1Z2kNyw.png" /><figcaption>Dimenstion of feature map WITHOUT Padding or Striding</figcaption></figure><p>Here,</p><p>n =&gt; Dimension of image (n x n)</p><p>f =&gt; Dimension of the filter (or kernel, f x f)</p><p>Great, now you are all well versed with how the convolution layer works in a proper sense. Now let’s jump into the next layer in a general CNN architecture, which is…</p><h3>The Pooling Layer</h3><p>Well, the bookish definition is:</p><blockquote>“ Pooling is a down-sampling operation commonly used in convolutional neural networks (CNNs) to reduce the spatial dimensions of the input data while retaining important features. ”</blockquote><p>I know, this definition went over your head….</p><p>To put this into simple words, Pooling is the technique for gaining the most important and useful features from the convolved feature map.</p><p>The Pooling operation is somewhat similar to how filters are involved in convolution. Here, the ‘Pooling’ filter (with a stride value of 2 and filter size of 2x2) interacts with the feature map and produces the ‘Pooled’ feature map. But what is this operation and how this operation is taking place?</p><p>After the image is passed through the convolution layer, a feature map is formed. Now, this special ‘Pooling’ filter (window would be a better word) is placed over the top left corner of the feature map. The pixel values of the feature map that are inside this filter are considered to form the first-pixel value of the ‘Pooled’ feature map. This ‘consideration’ depends upon the type of Pooling we are using.</p><p>If we use <strong>Max Pooling, </strong>the <strong>max</strong> pixel value inside that filter is taken as the pixel value for the pooled output feature map. If we are using <strong>Average Pooling</strong>, then the <strong>average</strong> of the pixel values inside that pooling window becomes the pixel value for the Pooled feature map. After this, the pooling window shifts by 2 pixels and the process repeats over the new set of pixel values inside that window. Thus, the pooled feature map is formed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/517/0*KyqkZFFI1_Kv__K4.png" /><figcaption>source: <a href="https://www.researchgate.net/figure/Illustration-of-Max-Pooling-and-Average-Pooling-Figure-2-above-shows-an-example-of-max_fig2_333593451">https://www.researchgate.net/figure/Illustration-of-Max-Pooling-and-Average-Pooling-Figure-2-above-shows-an-example-of-max_fig2_333593451</a></figcaption></figure><p>This also makes sense. By applying Pooling, we are Pooling out (or extracting) the most standout pixels or an average of the pixels (or patterns) from the feature map, thus it’s aptly named ‘Pooling’ Lol!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PB4336BL2PAuPV06.png" /><figcaption>Pooling operation (source: O’Reilly Media)</figcaption></figure><p>One thing to note here is that in Pooling, the Pooling filter is essentially a window and not a matrix containing pixel values. Instead of an element-wise product, it is the means of gathering the most useful pixels from that specific cluster of pixels inside that window, thereby eliminating useless pixels and features.</p><p>Another important advantage of the Pooling operation is that it is ‘Translation Invariant’. I know, a very fancy term, let me explain.</p><p>Suppose, You have an image of a Cat. So no matter how the cat is positioned in the image. It can be upside down, it can be of small size, slightly tilted, or not aligned at the center, it is still an image of a cat, right? YES. This is what Translational invariance is in the context of images. We can identify through the image that its a cat but how CNN can do it? Well, though Pooling. Thus, it&#39;s an essential layer in CNN.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*7gwMt60SiYSKbiRp.png" /><figcaption>illustration of translation invariance</figcaption></figure><p>So, the main point? Some variations in the image should not hinder classification ability. Thus the term ‘invariance’. This property of CNN helps it to generalize better.</p><p>Alright, the Convolution layer is done. The pooling layer. done. Now what?</p><h3>Flattening</h3><p>After the pooling operation, the feature map is converted into a Pooled feature map. But it is still a matrix (or tensor in the case of RGB images), it is NOT one-dimensional. As we discussed at the beginning that image is first converted in such a form that can act as an input for ANN. But, we cannot put this pooled feature map into the ANN. We first have to make it a one-dimensional format. How do we do this? Through Flattening layer! I mean it was obvious by the name, that we ‘Flatten’ the pooled feature map in the form of one one-dimensional array.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/0*tvdvELVLIPn41NPA.png" /><figcaption>source: superdatascience</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/0*3sqpM_TL9GGsJpCU.png" /><figcaption>source: superdatascience</figcaption></figure><p>Mathematically also, it is not rocket science. it is just re-arranging all the elements of the pooled feature map linearly. THATS IT!</p><p>Now, this flattened array is ready to be put into the ANN or we call it…</p><h3>Fully Connected Layer</h3><p>That’s it we are almost there. We studied the difficult portion. In the above sections, we studied the part of CNN that separates it from an ANN.</p><p>This layer, the Fully Connected layer as we call is just a typical artificial neural network.</p><p>So, the flattened array we generated in the last layer acts as the numerical input for the Fully Connected layer. And, that’s it, from here on onwards the work will be similar to how an ANN responds to numerical inputs. . Yes, those same interconnections of neurons with weights and biases involved blah blah and all that backpropagation stuff. I won’t go into much detail otherwise the article will be too long (It is already very long).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jmYJWGd-Fx6t3Of1.png" /><figcaption>Fully connected layer in CNN (source: InterviewBit)</figcaption></figure><p>It is expected to have some knowledge about ANNs before jumping into CNN even if you want to study the mathematical intricacies of ANN, kindly refer to these articles:</p><ul><li><a href="https://medium.com/analytics-vidhya/an-introduction-to-mathematics-behind-neural-networks-135df0b85fa1">An Introduction To Mathematics Behind Neural Networks</a></li><li><a href="https://raj-aryan001.medium.com/mathematics-behind-neural-nets-e10257ffc09f">Mathematics behind Neural-Nets</a></li><li><a href="https://towardsdatascience.com/understanding-backpropagation-algorithm-7bb3aa2f95fd">Understanding Backpropagation Algorithm</a></li></ul><p><strong>Note:</strong> These are the basic building blocks of CNN. We can have so many variations by tweaking the numbers or number of layers involved and BOOM! A new CNN architecture is born. So many architectures of CNN like LeNet, AlexNet, VGGNet, Inception, etc are formed just by tweaking these building blocks and numbers. In this article, however, we only focused on the basic building blocks, working, and intuition behind CNN.</p><p>Thank You so much for reading this article very patiently, I hope I was able to help you resolve your doubts about the fundamentals of CNN.</p><p>Sorry, if the article was too long. This article was meant to be comprehensive, teaching the reader every single detail about the working of CNN. Regarding the Mathematics part, I will soon release another article where I will be explaining the mathematics of CNN in great detail, so stay tuned for that!</p><p>Have a wonderful day : )</p><h3><strong>References</strong></h3><p><a href="https://www.upgrad.com/blog/basic-cnn-architecture/">https://www.upgrad.com/blog/basic-cnn-architecture/</a></p><ul><li><a href="https://medium.com/@draj0718/convolutional-neural-networks-cnn-architectures-explained-716fb197b243">Convolutional Neural Networks (CNN) — Architectures Explained</a></li><li><a href="https://www.interviewbit.com/blog/cnn-architecture/">CNN Architecture - Detailed Explanation</a></li><li><a href="https://towardsdatascience.com/convolutional-neural-networks-explained-9cc5188c4939">Convolutional Neural Networks, Explained</a></li><li><a href="https://towardsdatascience.com/understand-the-architecture-of-cnn-90a25e244c7">Understand the architecture of CNN</a></li><li><a href="https://medium.com/@koushikkushal95/understanding-convolutional-neural-networks-cnns-in-depth-d18e299bb438">Understanding Convolutional Neural Networks (CNNs) in Depth</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f056a9accbbb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Detailed Explanation, In-depth Intuition, and Python implementation of Bagging and Random Forest]]></title>
            <link>https://medium.com/@datafreakai/detailed-explanation-in-depth-intuition-and-python-implementation-of-bagging-and-random-forest-7cca07c5d0a6?source=rss-71ae604d9df2------2</link>
            <guid isPermaLink="false">https://medium.com/p/7cca07c5d0a6</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[mathematics]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[datafreak]]></dc:creator>
            <pubDate>Sun, 10 Dec 2023 18:41:42 GMT</pubDate>
            <atom:updated>2023-12-11T03:54:09.985Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/612/0*nRNwPOFV-uB5tMog" /><figcaption>credit: gettyimages.com</figcaption></figure><p>In this article, we will dive into the concept of <strong>Bagging </strong>and study one of the most famous and used algorithms in Industry and Kaggle competitions: <strong>Random Forests.</strong></p><p>The only pre-requisite for this article is Decision Trees.</p><p>To study Decision Trees, you can look into this article: <a href="https://towardsdatascience.com/decision-trees-explained-3ec41632ceb6">https://towardsdatascience.com/decision-trees-explained-3ec41632ceb6</a></p><p>But Before, jumping into Bagging and Random Forests, let us understand what are ‘Ensemble Techniques’:</p><h3>Ensemble Learning</h3><p>Ensemble learning is a machine learning approach that combines or aggregates the predictions from multiple models(called ‘weak learners’) to achieve better accuracy and robustness than individual models.</p><p>The main aim of using multiple models for aggregation is to create ‘variety’ so that the bias is reduced and the overall model generalizes better. Thus, the risk of overfitting is reduced.</p><p>This variety can be achieved by using multiple models or training the same model on different subsets of the data. By using ensemble techniques, we can achieve low bias - low variance in models.</p><p>Types of ensemble techniques::</p><ol><li>Voting ensemble</li><li>Bagging ensemble</li><li>Boosting ensemble</li><li>Stacking ensemble</li></ol><p>And many more…..</p><p>Now, Let us discuss about Bagging.</p><h3>Bagging Ensemble</h3><p>Bagging is the portmanteau word for ‘Bootstrapped Aggregating”.</p><p>In this ensemble technique, a base model is trained multiple times on random samples (or chunks) of a dataset. This process of assigning random data samples (or chunks) to each copy of the base model is called ‘Bootstrapping”. The purpose of Bootstrapping is to generate variety while training such that the bias and correlation between the models while training are reduced. This helps in reducing the risk of overfitting.</p><p>After training, the predictions of each model are then aggregated or combined for overall prediction. For classification tasks, ‘Wisdom of Crowd’ (Basically making decisions based on the majority) is used for overall prediction, and for Regression tasks, the predictions of each model are averaged.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DB4Q5eE_RMW1R1ow" /><figcaption>Bagging Flowchart</figcaption></figure><p><strong>Note:</strong> The sampling does not need to be row sampling, column/feature sampling can also be done to generate variety. The special term used for random column sampling is called ‘<strong>Random Subspaces</strong>’. Also, we can perform both row sampling and column sampling simultaneously. It is called ‘<strong>Random Patches</strong>’</p><h4>Major Hyperparameters (in Scikit learn class)</h4><p>There is a separate class used in Scikit Learn called BaggingEstimator(BaggigngClassifier for Classification, BaggingRegressor for Regression)</p><p>Some hyperparameters used are:</p><ol><li><strong>base_estimator:</strong> Estimator used as the base model, the default base model is Decision Tree.</li><li><strong>max_samples: </strong>fraction of rows taken during random sampling</li><li><strong>max_features: </strong>fraction of columns taken during random column sampling.</li><li><strong>n_estimators: </strong>No. of base models to train</li></ol><h4>Implementation using Scikit Learn</h4><pre>#For Classification<br>from sklearn.ensemble import BaggingClassifier<br>from sklearn.model_selection import train_test_split<br><br>x_train, x_test, y_train, y_test = train_test_split(x,y, test_size = 0.25, random_state = 42)<br><br>bag = BaggingClassifier()<br><br>#Fitting the model<br>bag.fit(x_train, y_test)<br><br>#Predictions<br>bag.predict(x_test)</pre><pre>#For Regression<br>from sklearn.ensemble import BaggingRegressor<br>from sklearn.model_selection import train_test_split<br><br>x_train, x_test, y_train, y_test = train_test_split(x,y, test_size = 0.25, random_state = 42)<br><br>bag = BaggingRegressor()<br><br>#Fitting the model<br>bag.fit(x_train, y_test)<br><br>#Predictions<br>bag.predict(x_test)</pre><p><strong>Note:</strong> The above code was to show how bagging is implemented using sk learn, hyperparameter tuning was not done. Performing Hyperparameter tuning is advisable for better performances (You have to perform hyperparameter tuning for the base_estimator used in bagging as well).</p><p>Thus the name <strong>“B</strong>ootstrap + <strong>Agg</strong>regat<strong>ing</strong>”. Bootstrapping helps in achieving bias-variance tradeoff and aggregation helps in achieving better predictions.</p><p>Though any type of base model can be used for bagging using Decision Trees gives the best results and performance. The implementation of Decision Trees in Bagging is what is known as Random Forest (though there are some fundamental differences between random forests and bagging of decision trees, but mostly, they are very similar.)</p><p>Let’s talk about Random Forest in the next section.</p><h3>Random Forests</h3><p>Random Forest is the application of Bagging, where the base estimator is the Decision Tree. Here, multiple Decision Trees are constructed and are trained on the random subset of the training data.</p><p>Then the predictions of each Decision Tree are aggregated accordingly (Outputting the mode of predictions in case of classification tasks and averaging the predictions in case of regression tasks.)</p><p>Random forests are so powerful that even without much pre-processing, it still give great results.</p><h4>Training and Algorithm</h4><p>The training process of a Random Forest involves constructing multiple Decision Trees through a combination of Bootstrapped sampling and Random Feature selection.</p><p>The steps involved are:</p><ol><li>Bootstrapped Sampling: Randomly select a subset of the training data with replacement. This means that some samples may be repeated, while others may be omitted.</li><li>Random Feature Selection: For each tree in the Random Forest, randomly select a subset of features from the full set of features. This introduces diversity among the trees and prevents them from becoming too correlated.</li></ol><p>3. Decision Tree Construction:</p><ul><li>Build a decision tree using the bootstrapped sample and the randomly selected subset of features.</li><li>At each node of the tree, consider a subset of features for splitting, based on the random feature selection.</li><li>Split the node based on a criterion (e.g., Gini impurity for classification, Mean Squared Error for regression).</li><li>Continue recursively until a stopping criterion is met (e.g., maximum depth, minimum samples per leaf).</li></ul><p>4. Repeat the steps 1–3: Repeat the steps 1–3 to create additional trees. Each tree created is constructed and trained independently and parallelly.</p><p>The training process in Random Forests is parallelizable, making it efficient for large datasets.</p><p>The combination of diverse trees, each trained on a different subset of data and features, helps the Random Forest generalize well to unseen data and reduces the risk of overfitting.</p><p>The randomness introduced in both sample selection and feature selection contributes to the robustness of the model.</p><h4>Predictions</h4><p>So, How does the Prediction in Random Forests take place?</p><p>Prediction by the overall random forest is done by aggregating or combining the predictions of individual decision trees.</p><p>For Regression tasks, the output of each decision tree is averaged out. The output of the overall model is the mean of the outputs of the individual decision trees.</p><p>While for Classification tasks, the output of the overall model is based on ‘Voting’. The Class which is being predicted by the majority of the Decision Trees is the class predicted by the overall model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YhLQxLQSYVmMHGaz" /><figcaption>Random Forests Flowchart</figcaption></figure><h4>OOB Score</h4><p>The full form of OOB is ‘Out Of Bag’. So what happens is that during sampling (with replacement), some rows are never taken for training. Such rows are called ‘Out Of Bag’ rows/samples.</p><p>Statistically, it has been seen that 67% of the samples are taken for training and the remaining 33% of samples are never taken for training, thus they are the <strong>‘Out Of Bag’ </strong>samples.</p><p>Since OOB samples are never by the base model during training, they can be used to test the trained model.</p><p>So, the metric used to test the trained random forest on OOB samples is called the OOB score. The value OOB score is somewhat similar to that of other traditional metrics.</p><h4>Feature Importance</h4><p>In a Random Forest, the feature importance of a feature is calculated by each decision Tree and then averaged to get the overall importance of that particular feature by the Random Forest.</p><p>Wait, so how the feature importance is calculated in each Decision Tree?</p><p>For each decision tree, Scikit-learn calculates a ‘<strong>nodes importance’ </strong>using Gini Importance, assuming only two child nodes (binary tree):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1006/0*azcwcfOWTRLmFif9" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1011/0*nkJ89FxD7ixMIigT" /></figure><p>The final feature importance, at the Random Forest level, is its average over all the trees. The sum of the feature importance value on each tree is calculated and divided by the total number of trees:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*T1DxQECU8dTf4UgM" /></figure><h4>Scikit Learn Implementation</h4><pre>#For Classification<br>from sklearn.ensemble import RandomForestClassifier<br>from sklearn.model_selection import train_test_split<br><br>x_train, x_test, y_train, y_test = train_test_split(x,y, test_size = 0.25, random_state = 42)<br><br>rf = RandomForestClassifier()<br><br>#Fitting the model<br>rf.fit(x_train, y_test)<br><br>#Predictions<br>rf.predict(x_test)</pre><pre>#For Regression<br>from sklearn.ensemble import RandomForestRegressor<br>from sklearn.model_selection import train_test_split<br><br>x_train, x_test, y_train, y_test = train_test_split(x,y, test_size = 0.25, random_state = 42)<br><br>rf = RandomForestRegressor()<br><br>#Fitting the model<br>rf.fit(x_train, y_test)<br><br>#Predictions<br>rf.predict(x_test)</pre><p><strong>Note:</strong> In the above code, only the scikit learn implementation of RandomForest was shown, parameter tuning was not done. Performing Hyperparameter tuning is advisable for better performances.</p><h4>Some major Hyperparameters</h4><p>Some Major Hyperparameters used for tuning are:</p><ol><li><strong>n_estimators:</strong> Number of Decision Trees to be used in forest(default = 100).</li><li><strong>max_samples:</strong> Samples to be taken during bootstrapping. (if bootstrap = True)</li><li><strong>bootstrap: </strong>To perform random sampling or not. The default value is True. If false, all the samples are considered for training.</li><li><strong>oob_score: </strong>It tells whether to use OOB samples to validate the degree of generalization of the model (to test the model). The default value is False.</li><li><strong>n_jobs: </strong>Controls the parallel construction of trees while training. The default is 1.</li></ol><p>The rest of the hyperparameters like max_depth, max_features, etc control the individual decision trees in the ensemble.</p><h3>A comparison?</h3><p>A genuine question you might be getting now is that in sci-kit learn, there is already a separate class for Bagging. So we can technically use that class Bagging Estimator class with base_estimator as Decision Tree, then what is the use for a Random Forest estimator? Well, it is because there is a bit of a difference between Bagging of Decision Trees and Random Forest.</p><p>In Random Forests, there is an extra level of randomness. During each split on each tree, a random subset of features is chosen for the split. In Bagging of Trees, the random subset of features is only chosen initially, and then during each split, all the features are considered from that specific random subset of features that were chosen initially for the tree. Thus, there is less degree of randomness in the case of bagging of trees.</p><p>The main reason for this extra randomness in Random Forests was to make it less and less prone to overfitting so that the generalization is better.</p><p>Let’s compare the performance of Random Forest and Bagging on the PIMA Diabetes Dataset using sk learn.</p><p>The dataset: <a href="https://www.kaggle.com/datasets/mathchi/diabetes-data-set">https://www.kaggle.com/datasets/mathchi/diabetes-data-set</a></p><pre>#Importing the models<br>from sklearn.ensemble import BaggingClassifier, RandomForestClassifier<br>from sklearn.model_selection import train_test_split<br><br>#Splitting the dataset into train and test<br>x_train, x_test, y_train, y_test = train_test_split(x,y, test_size = 0.25, random_state)</pre><pre>#For Bagging<br>bag = BaggingClassifier(<br>    base_estimator = DecisionTreeClassifier(max_depth = 8),<br>    n_estimators = 50<br>)<br><br>bag.fit(x_train, y_train)<br>bag.predict(x_test)<br><br>print(&quot;The Accuracy score of the Bagging Classfier is: &quot;, bag.score(x_test, y_test))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/673/1*gXApjO7qq0E36hOGqr7Vdw.png" /><figcaption>Accuracy metric of the Bagging Classifier on that diabetes dataset</figcaption></figure><pre>#For Random Forest<br>rf = RandomForestClassifier(<br>    n_estimators = 50,<br>    max_depth = 8<br>)<br><br>rf.fit(x_train, y_train)<br>rf.predict(x_test)<br><br>print(&quot;The Accuracy for the Random Forest classifier is: &quot;, rf.score(x_test, y_test))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/721/1*qPEqHWNdFLTLWOvQZxs7Ig.png" /><figcaption>Accuracy metric of Random Forest on that dataset</figcaption></figure><p>As we can see Random Forest performs slightly better than the bagging classifier on the same values of the same hyperparameters, which was expected as the extra degree of randomness helps Random Forests to generalize the patterns better.</p><p>However, Random Forest doesn&#39;t need to give better results, it just depends upon the dataset, the hyperparameter tuning done, and how preprocessing was performed. Generally, Random Forests give better results than Bagging of Trees but it is not a hard and fast rule.</p><h3>Conclusions</h3><p>Putting everything aside, we can conclude that Random Forests (in fact Bagging in general) is an excellent technique.</p><p>Without much preprocessing, it still gives excellent and respectable results. The idea of aggregating the predictions of multiple trees improves the performance a lot and also, the degree of randomness in Random Forests, helps in generalizing the patterns in a better manner, thus less prone to overfitting. Another major advantage in the case of Random Forest is that it can be used for feature selection as well (based on calculated feature importances)</p><p>Though there are some drawbacks, which include the lack of interpretability of the model (the decision-making process is a bit confusing) and the requirement of higher computational power for training, but overall, Random Forest is a powerful algorithm.</p><p>I mean, there is a valid reason why it is so popular in industries and Kaggle competitions 😉</p><p>Thank You so much for reading, hope you found this article helpful and learned something from it : )</p><h3>References</h3><p>Some resources I took help from while writing this article:</p><ul><li><a href="https://medium.com/geekculture/random-forest-ensemble-method-860aaf4fcd16">Random Forest — Ensemble method</a></li><li><a href="https://towardsdatascience.com/understanding-random-forest-58381e0602d2">Understanding Random Forest</a></li></ul><p><a href="https://machinelearningmastery.com/bagging-and-random-forest-ensemble-algorithms-for-machine-learning/">https://machinelearningmastery.com/bagging-and-random-forest-ensemble-algorithms-for-machine-learning/</a></p><ul><li><a href="https://towardsdatascience.com/the-mathematics-of-decision-trees-random-forest-and-feature-importance-in-scikit-learn-and-spark-f2861df67e3">The Mathematics of Decision Trees, Random Forest and Feature Importance in Scikit-learn and Spark</a></li><li><a href="https://towardsdatascience.com/ensemble-methods-bagging-boosting-and-stacking-c9214a10a205">Ensemble methods: bagging, boosting and stacking</a></li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fvideoseries%3Flist%3DPLKnIA16_RmvZyqP3WGUo7iVziIIea_1bp&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fplaylist%3Flist%3DPLKnIA16_RmvZyqP3WGUo7iVziIIea_1bp&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FbHK1fE_BUms%2Fhqdefault.jpg%3Fsqp%3D-oaymwEXCOADEI4CSFryq4qpAwkIARUAAIhCGAE%3D%26rs%3DAOn4CLDvvXaRLqe2wjFlCyt0xKenDMpnYQ%26days_since_epoch%3D20148&amp;type=text%2Fhtml&amp;schema=youtube" width="853" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/5d3cb37b75e025c60caf15e496c7f169/href">https://medium.com/media/5d3cb37b75e025c60caf15e496c7f169/href</a></iframe><ul><li><a href="https://builtin.com/data-science/random-forest-algorithm">Random Forest: A Complete Guide for Machine Learning</a></li><li><a href="https://blog.paperspace.com/random-forests/">Random Forests: Consolidating Decision Trees | Paperspace Blog</a></li><li><a href="https://syncedreview.com/2017/10/24/how-random-forest-algorithm-works-in-machine-learning/?ref=blog.paperspace.com">How Random Forest Algorithm Works in Machine Learning | Synced</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7cca07c5d0a6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Categorical Encoding]]></title>
            <link>https://medium.com/@datafreakai/understanding-categorical-encoding-bdd8a17a6d2b?source=rss-71ae604d9df2------2</link>
            <guid isPermaLink="false">https://medium.com/p/bdd8a17a6d2b</guid>
            <category><![CDATA[data-preprocessing]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[datafreak]]></dc:creator>
            <pubDate>Fri, 10 Nov 2023 17:28:56 GMT</pubDate>
            <atom:updated>2023-11-11T03:37:57.800Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/612/0*rWJd01bUdXcHxRsK" /><figcaption>image source: istockphoto.com</figcaption></figure><p>In this article, we will try to understand what is a categorical data, what is categorical Encoding, why to perform encoding and How to perform encoding. I will try to explain all of these concepts by using real world datasets and examples.</p><h3><strong>What is Categorical Data?</strong></h3><p>The textbook definition of categorical data is that it’s a type of data which represents categories or labels. It generally represents non-numeric attributes or characteristics such as Colors, Car Models, Gender, Education qualification etc. Categorical variable are of two types:</p><ol><li>Nominal Data: This type of data has categories with no ‘order’ or ‘ranking’. There is no sense of ranking the labels(or categories) in this type of data. An example would be a column of colors having values like Red, Yellow, Green etc. or a column of cities having values like ‘Kolkata’, ‘Delhi’, ‘New York City’ etc. ranking in these columns won’t make sense at all.</li><li>Ordinal Data: This type of data has categories which can be ordered or ‘ranked’. For example, a column of education qualifications (High school, Bachelors, Masters, PhD) or a column of student performance (Poor, Average, Good, Excellent). In such columns, Ranking the categories actually make sense, We can assign a number to them to properly rank the categories.</li></ol><p>Let’s understand all of this through an example using python:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GSRq5uANLAnpIzbI99VuhQ.png" /><figcaption>A sample dataset of retail customers</figcaption></figure><p>In the above dataset, we see columns like ‘Gender’, ‘Item Purchased’, ‘Category’, ‘Location’, ‘Size’ and many more. All of these columns are categorical columns. How? Cause the values these columns are taking are in string format, taking a discrete value and are repetitive in nature.</p><p>Let’s have a deeper look on the ‘Size’ variable:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/363/1*maCAvb_-tSVj3E6uypTOHw.png" /></figure><p>In the above image, we can clearly see that the ‘Size’ column is categorical column as the values are discrete and repetitive in nature. However, I want you to see that the specific values it can take: ‘S’, ‘M’, ‘L’, ‘XL’; All of these values represent the size of clothing. Interesting part is, we can assign a numerical value to them. That is, we can put 0 instead of ‘S’, 1 instead of ‘M’, 2 instead of ‘L’ and so on according to their size as it would make sense. As size increases, the assigned numerical value also increases. So it’s safe to say that it is an Ordinal variable.</p><p>Similarly, Look at the column ‘Season’:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*JcDR0X1CTTDGruPBRjaRtw.png" /></figure><p>In this column, we are able to see all the unique values this categorical column has. However, it’s bit different than the ‘Size’. Here, assigning numbers to them won’t make sense at all. For example, if we assign number 0 to the value ‘Winter’, then how will you assign the next values? 1 to ‘Summer’? and Why we assigned 0 to ‘Winter’, we can also assign 0 to ‘Fall’ right? Unlike the ‘Size’ column where there was a logical universal ordering (‘L’ size is more than ‘S’ therefore, assigning a greater number to ‘L’), there is no such case in the Season column. Such variables or columns are Nominal variables.</p><p>A categorical data generally contains String like values but can also take numerical values. For example in a House Price dataset, suppose there exists a column of ‘number of rooms’. And suppose it can take values like 1, 2, 3, 4 etc. (basically telling the number of rooms that specific house has). Since the values that column can take are limited, discrete and continuously repetitive, it will be considered as a categorical column even though it has numerical values.</p><p>Let’s understand the above case through a different example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/436/1*AZlbhb5S3xI9ew_aLMFWxw.png" /></figure><p>The above image, shows a column of ‘percent_of_income’ in a loan dataset. This column basically shows percent of loan amount is to that person’s income. In this column, we can see it can only take 4 values: 1,2,3,4. Even though the values it can take are numerical, but these are discrete and repetitive, hence this column is also a categorical variable.</p><p>To see if a column is categorical or not, we have to see how repetitive the values are (that is, how many duplicates of a certain value are present) and also how discrete the values are in that column.</p><h3><strong>What is Categorical Encoding and Why to perform it?</strong></h3><p>Now, What is Categorical Encoding?</p><p>See, In a dataset most of the categorical data is in string format. When we are fitting that data into our model using Scikit learn, it only accepts numerical data (in the format of integer or float).</p><p>So when we try to fit that data in string format using sci-kit learn, our code will not execute and will give error. So before fitting that into the model, we have convert that data in string format into numerical format using various techniques, so that we can fit this converted data to Scikit learn ML models.</p><p>Such techniques are called <strong>Categorical Encoding. </strong>Categorical Encoding is an essential part of Data Preprocessing step (if of course categorical data is present in the dataset).</p><h3>So, How to perform Encoding?</h3><p>There are two popular techniques to handle categorical variables. These are:</p><ol><li><strong>Ordinal Encoding:</strong> As the name suggests, this encoding technique is used to handle ordinal variables (or data).Through this technique, we can rank the different categories on the data. For example, In a column of education levels, we can set the high school to be equivalent to 0, Bachelors to be equivalent to 1, masters to 2 and so on. In this way, each category is ranked based on the numbers assigned to them, like a category whose numerical value is 0 will be ranked below to a category whose assigned numerical value is 1 and so on. We can use direct pandas techniques and methods or even use scikit learn for that.</li><li><strong>One Hot Encoding: </strong>This technique used for nominal variables (the variables which cannot be ‘ranked’ or ‘ordered’). When we apply this technique what happens is a separate extra column is created for each of the category present in that column. For example, in the colors column there are three colors: Red, Blue, Green. Now When we apply one hot encoding, three extra columns of red, blue and green are created. Now suppose a data sample is valued red in the color category, So after applying one hot encoding, the value of red column will be 1 of that sample and rest will be 0. Similarly, if the color of data sample is valued Green, so the value at the green column(which was created after applying one hot encoding) will be 1 and rest will be valued 0. Thus, sort of binary value is encoded to that data sample (By data sample, I mean a row or data point). It’s useful for nominal variables as we didn’t have to rank the categories based on labels as it does not make any sense in case of nominal variables. We can again apply this technique using pandas and scikit learn.</li></ol><p>Let’s see the python implementation using pandas and scikit-learn to apply both of these techniques:</p><h4>Ordinal Encoding Implementation</h4><ol><li>Using Pandas:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/618/1*VR8-2FDp78TdvXV3tEzNww.png" /><figcaption>Implementing Ordinal Encoding using pandas method on the Size column</figcaption></figure><p>2. Using Scikit Learn:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/595/1*TZj8RLeABXBKABNU49diJg.png" /><figcaption>Implementing Ordinal Encoder using Scikit learn on the Size column</figcaption></figure><p><strong>NOTE: </strong>In the scikit learn method, it is important to specify the order before applying the method as if you directly use the method on a column before specifying the order, it will assign labels based on the order of appearance of the category in that column rather than the inherent order of the categories.</p><h4>One Hot Encoding Implementation</h4><ol><li>Using Pandas:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/655/1*jxCX4lxirFWTMUU94VHBAQ.png" /><figcaption>Implementing One Hot Encoding using pandas on Category column</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m0cmIy3_5kVEJjul0sXfyw.png" /><figcaption>The output</figcaption></figure><p>2. Using Scikit Learn:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*xaLZpcx9MtbuQI-M0n0Rzw.png" /><figcaption>Implementing One Hot encoding using scikit learn on Category column</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FRb32ZN0cb7wJVViGJDjFw.png" /><figcaption>The output</figcaption></figure><p>As we can clearly see, extra columns were created on both of the approaches and values were set accordingly.</p><p>In sklearn’s method, a separate numpy array is created while applying the OneHotEncoder method which contains the one hot encoded columns. So it needs to be converted into a Dataframe and then concatenated on the main Dataframe separately. Though it might seem to be less ‘efficient’ than the pandas get_dummies( ) method but it has its own benefits especially while fitting data to ML models using sklearn.</p><h3><strong>When to perform which technique and approach?</strong></h3><p>So, we know that we have to apply Ordinal Encoding for Ordinal data (duh!) and One Hot Encoding for Nominal data.</p><p>But the main question you might be getting is: “What approach to use? Pandas or Scikit learn?”</p><p>Well, this question does not have a concrete answer. It depends upon the dataset/type of data and your own objectives. Pandas approach of encoding is generally used for statisitcal analysis and Scikit learn approach is mainly to make the data be able to fit on sklearn ML models. Both of these approaches are best on their own and there is no universal answer to what approach is better.</p><p>I recommend you to implement these techniques and approaches on your own in python on a real world project, that’s exactly how I learnt this topic.</p><p>Hope You found this article helpful, Thanks for reading this so patiently.</p><p>Happy Learning : )</p><h3>Resources</h3><p>The resources I took help from to write this article:</p><ol><li><a href="https://medium.com/aiskunks/categorical-data-encoding-techniques-d6296697a40f">https://medium.com/aiskunks/categorical-data-encoding-techniques-d6296697a40f</a></li><li><a href="https://medium.com/analytics-vidhya/categorical-variable-encoding-techniques-17e607fe42f9">https://medium.com/analytics-vidhya/categorical-variable-encoding-techniques-17e607fe42f9</a></li><li><a href="https://towardsdatascience.com/all-about-categorical-variable-encoding-305f3361fd02">https://towardsdatascience.com/all-about-categorical-variable-encoding-305f3361fd02</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bdd8a17a6d2b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>