<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Venkat on Medium]]></title>
        <description><![CDATA[Stories by Venkat on Medium]]></description>
        <link>https://medium.com/@venkatpendela96?source=rss-05858c616c3e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*ohDc5ObgQ-v7IENX</url>
            <title>Stories by Venkat on Medium</title>
            <link>https://medium.com/@venkatpendela96?source=rss-05858c616c3e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 04:46:02 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@venkatpendela96/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Understanding Memory Management in Apache Spark]]></title>
            <link>https://medium.com/@venkatpendela96/understanding-memory-management-in-apache-spark-abd1c8bd0903?source=rss-05858c616c3e------2</link>
            <guid isPermaLink="false">https://medium.com/p/abd1c8bd0903</guid>
            <category><![CDATA[out-of-memory]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[memory-management]]></category>
            <category><![CDATA[disk-spill]]></category>
            <category><![CDATA[execution-memory]]></category>
            <dc:creator><![CDATA[Venkat]]></dc:creator>
            <pubDate>Fri, 24 Jan 2025 04:08:55 GMT</pubDate>
            <atom:updated>2025-01-24T04:08:55.561Z</atom:updated>
            <content:encoded><![CDATA[<p>Memory management is a crucial aspect of optimizing Apache Spark applications. Let’s break down how Spark handles memory, using the example of requesting <strong>2 GB executor memory</strong> through the --executor-memory parameter in spark-submit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WWShr_Pf02yxanHKSkX60A.png" /></figure><h3>Types of Memory in Spark Executors</h3><p>When we request <strong>2 GB executor memory</strong>, this amount refers to <strong>heap memory</strong>, managed by the Java Virtual Machine (JVM). In addition to this, Spark provides <strong>overhead memory</strong>, which is non-heap memory which is outside JVM and overhead memory is used for VM related issues and for safegaurding against out of memory issues. The overhead memory is calculated as the <strong>maximum of 384 MB or 10% of the requested memory</strong>.</p><p>So, for <strong>2 GB executor memory</strong>, the total allocated memory turns out to be:<br> <strong>2 GB (heap) + max(384 MB, 10% of 2 GB) = 2.384 GB.</strong></p><h3>Breakdown of Executor Memory</h3><p>Lets have a look at the division of executor memory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/889/1*8EpmPmap7BU_6u6HI1AEsw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/502/1*L2hNeVHgNQM0n66vcK606A.png" /></figure><p>Let us replicate this division in our example of 2 GB executor memory. Out of the requested <strong>2 GB executor memory</strong>, the following allocations are made:</p><p>Complete 2 GB executor memory will be divided into three parts:</p><ul><li>Reserved memory</li><li>Unified area.</li><li>User memory</li></ul><ol><li><strong>Reserved Memory (300 MB):</strong> Dedicated to the Spark engine. This is fixed and cannot be changed.</li><li><strong>Unified Area &amp; User memory:</strong> After excluding reserved memory, we will be left with 1.7 GB, out of this 60% (1.02 GB) would fall under Unified area and remaining 40 % (0.68 GB) would fall under user memory. Unified area refers to combination of storage memory and execution memory. These percentages are configurable based on requirement. Inside unified area there is no hard boundary between execution and storage memory. They can borrow from each other based on availability and requirement.</li></ol><ul><li><strong>Execution Memory:</strong> Used for shuffle, sort, join, and aggregation operations.</li><li><strong>Storage Memory:</strong> Used for caching and persisting RDDs and datasets.</li><li><strong>User Memory: </strong>Used for RDD related operations and usen when dealing with user defined data structures like hash table, broadcast join uses this.</li></ul><h3>Memory Usage and Adjustments in the Unified Area</h3><p>Consider the unified memory as <strong>100%</strong>. By default, it is split as follows:</p><ul><li><strong>50% for Storage Memory</strong></li><li><strong>50% for Execution Memory</strong></li></ul><p>If our use case <strong>2 GB</strong> is the total requested memory, the unified area pool is <strong>1.02 GB</strong>, let us consider as <strong>1GB</strong> for ease of understanding, divided into:</p><ul><li><strong>500 MB (50%) for Storage Memory</strong></li><li><strong>500 MB (50%) for Execution Memory</strong></li></ul><p>Here’s how Spark handles memory adjustments dynamically:</p><ol><li>If storage memory is underutilized, execution memory can borrow from it, and vice versa.</li><li>For instance, if caching requires <strong>700 MB</strong> but storage memory has only <strong>500 MB</strong> free and execution memory is underutilized, the additional <strong>200 MB</strong> can be borrowed from execution memory.</li><li>In the similar way when storage memory is underutilized, and if storage memory requires <strong>more than 500 MB</strong>, it can borrow additional memory from storage memory.</li></ol><p>Let us consider a scenario, initially when execution memory is underutilized, storage memory borrowed 200MB memory from execution memory, and then executor memory came across a task that requires <strong>500 MB.</strong> what would execution memory do now?</p><ul><li>Execution memory can evict storage memory and get back its 500 MB for executing the task.</li></ul><p>In the similar scenario, what if execution memory requires 700 MB to complete the task (provided storage memory borrowed additional memory when execution memory is underutilized)?</p><ul><li>Even in this case, execution can evict storage memory only until 500 MB as we have configured 50 % as threshold while dividing unified area between storage and execution memories.</li><li>By default, the unified area is split <strong>50%-50%</strong>, but this ratio is configurable. (Once we update this configuration, we need to restart the spark application to get this change into effect).</li></ul><p>Does it work the same way when storage memory wants to evict execution memory?</p><ul><li>No, storage memory can’t evict execution memory.</li></ul><p><strong>Execution memory can evict Storage Memory.</strong></p><p><strong>Storage memory cannot evict Execution Memory.</strong></p><h4>What about maximum memory that could be used while running tasks on a CPU core?</h4><p>Let us consider an example of an executor with 4 GB memory and 4 CPU cores.</p><p>For an executor with <strong>4 GB of memory</strong> and <strong>4 CPU cores</strong>, the division works as follows:</p><ul><li>Unified Memory: <strong>2.2 GB (60% of (4 GB-300 MB))</strong></li><li>Per Core: 568<strong> MB per core</strong></li></ul><p>Tasks running on each core cannot use more than 568<strong> MB</strong> of execution memory. To minimize disk spills, it is critical to allocate sufficient execution memory per core.</p><h3>Off-Heap Memory and PySpark Memory</h3><p>In addition to heap memory, Spark also supports <strong>off-heap memory</strong> and <strong>PySpark memory</strong>, which need to be explicitly requested.</p><p><strong>Off-Heap Memory:</strong></p><ul><li>This is outside the JVM and is not subject to garbage collection.</li><li>Useful for avoiding JVM garbage collection overhead by storing objects directly in OS-managed memory.</li><li>Example: If <strong>2 GB off-heap memory</strong> is requested, the total unified memory becomes <strong>912 MB (default unified) + 2 GB (off-heap).</strong></li></ul><p><strong>Note:</strong> Off-heap memory is disabled by default and must be enabled explicitly.</p><p><strong>PySpark Memory:</strong></p><ul><li>Required only when using Spark with Python (not needed for Spark with Scala or Java).</li><li>Allocates memory for Python workers when core Python libraries are used.</li></ul><h3>Spill Mechanism</h3><p>When Spark processes data, it tries to fit as much as possible into memory for faster computation. However, when memory resources are insufficient, <strong>spill mechanisms</strong> come into play to avoid crashing the job.</p><ul><li>We have two types of spill mechanisms, memory spill and disk spill.</li></ul><p><strong>Memory Spill</strong></p><p>Memory spill happens when the amount of data being processed exceeds the available <strong>execution memory</strong> in the <strong>unified memory area</strong>. In such cases:</p><ul><li>Spark writes intermediate computation results to disk.</li><li>Spill occurs <strong>within memory first</strong> before writing to disk.</li><li>This spill ensures the job doesn’t fail due to insufficient execution memory but might slow down performance.</li></ul><p>Example: During operations like <strong>shuffling, sorting, or aggregations</strong>, if the allocated execution memory is insufficient, Spark spills the extra data into memory buffers or disk.</p><h4>Disk Spill</h4><p>Disk spill occurs when:</p><ul><li>Spark exhausts both <strong>execution memory</strong> and the memory buffers allocated for spill.</li><li>The remaining data that cannot fit into memory is written to disk.</li><li>Disk spill involves <strong>serialization</strong> and <strong>compression</strong> of data before writing to disk and <strong>deserialization</strong> when reading it back, this results in disk spill being slower than memory spill.</li><li>Disk I/O is significantly slower than processing in memory, making it a bottleneck in computation.</li></ul><h3>Execution Out Of Memory(OOM):</h3><p>When we have spill mechanisms, why are we coming across Execution OOM error?</p><p>Even with spill mechanisms, <strong>execution out of memory (OOM)</strong> errors can occur under specific circumstances. Here’s why:</p><p><strong>1. Spill Threshold Exceeded</strong></p><ul><li>Spark needs some memory for overheads, managing tasks, and intermediate computations.</li><li>If the combined <strong>storage memory</strong> and <strong>execution memory</strong> in the unified area are insufficient, and the system cannot spill fast enough, an OOM error occurs.</li></ul><p><strong>2. Overloaded Tasks</strong></p><ul><li>Each task running on an executor is allocated memory proportional to the executor’s memory divided by the number of cores.</li><li>If a task processes more data than its memory allocation, it can cause an OOM error, especially during:</li><li><strong>Shuffles</strong>: Too much intermediate data to process.</li><li><strong>Joins</strong>: Large datasets not fitting in execution memory.</li><li><strong>Sorts</strong>: Extensive sorting of data.</li></ul><p><strong>3. Disk I/O Bottlenecks</strong></p><ul><li>Spill to disk requires frequent writing and reading, and if the disk cannot keep up with these operations, Spark may fail due to memory overflow.</li></ul><p><strong>4. Off-Heap Memory Limits</strong></p><ul><li>Off-heap memory can be used to store data outside the JVM’s garbage collection, but if this allocation is exceeded, an OOM error may occur.</li></ul><p><strong>5. Garbage Collection Delays</strong></p><ul><li>Excessive reliance on JVM heap memory can trigger frequent garbage collection (GC), delaying spill operations and leading to an OOM error if Spark cannot spill data in time.</li></ul><h3>Best Practices to Prevent OOM Errors</h3><p><strong>1. Optimize Memory Configuration</strong>:</p><ul><li>Increase <strong>executor memory</strong> (--executor-memory).</li><li>Increase <strong>spark.memory.fraction</strong> to allocate more unified memory.</li></ul><p><strong>2. Adjust Task Parallelism</strong>:</p><ul><li>Reduce data processed by each task by increasing the number of tasks (adjust <strong>spark.sql.shuffle.partitions</strong> or <strong>rdd.repartition</strong>).</li></ul><p><strong>3. Enable Off-Heap Memory</strong>:</p><ul><li>Use off-heap memory (spark.memory.offHeap.enabled and spark.memory.offHeap.size) for temporary storage outside the JVM heap.</li></ul><p><strong>4. Increase Shuffle Memory</strong>:</p><ul><li>Configure <strong>spark.shuffle.spill.compress</strong> to control shuffle compression and reduce disk spill.</li></ul><p><strong>5. Broadcast Smaller Datasets</strong>:</p><ul><li>Use <strong>broadcast joins</strong> for smaller datasets to avoid shuffle-heavy operations.</li></ul><p><strong>6. Optimize Data Storage</strong>:</p><ul><li>Use columnar formats (e.g., <strong>Parquet</strong>, <strong>ORC</strong>) and compression for efficient data reading.</li></ul><p><strong>7. Adjust Unified Memory Ratios</strong>:</p><ul><li>Reconfigure <strong>spark.memory.storageFraction</strong> and <strong>spark.memory.fraction</strong> to better allocate execution and storage memory.</li></ul><h3>Key Takeaways</h3><ul><li>Proper memory management is critical for minimizing disk spills and optimizing Spark performance.</li><li>Storage memory cannot evict execution memory, but execution memory can evict storage memory within the configured threshold.</li><li>Adjusting the unified memory split and ensuring adequate memory per core can significantly reduce spills.</li><li>Off-heap memory and PySpark memory can further enhance performance but require explicit configuration.</li></ul><p>By understanding and configuring these memory components effectively, Spark applications can be optimized to handle large-scale data processing efficiently.</p><p>This blog post is intended to provide a general overview of memory management in Apache Spark.</p><p>Please feel free to leave a comment below if you notice any errors or have suggestions for improvement. Your feedback could be very helpful.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=abd1c8bd0903" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering Image Variations: Building Robust Computer Vision Models for Real-World Scenarios]]></title>
            <link>https://medium.com/@venkatpendela96/mastering-image-variations-building-robust-computer-vision-models-for-real-world-scenarios-a4ce9006a7c9?source=rss-05858c616c3e------2</link>
            <guid isPermaLink="false">https://medium.com/p/a4ce9006a7c9</guid>
            <category><![CDATA[occlusions]]></category>
            <category><![CDATA[overexposed-photos]]></category>
            <category><![CDATA[noise]]></category>
            <category><![CDATA[image-variation]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Venkat]]></dc:creator>
            <pubDate>Wed, 25 Dec 2024 08:24:21 GMT</pubDate>
            <atom:updated>2024-12-25T08:53:17.921Z</atom:updated>
            <content:encoded><![CDATA[<p>When developing computer vision (CV), machine learning (ML), or deep learning (DL) projects, it’s important to understand that the real-world images we work with are not always high-definition (HD) and perfect. Images may come with various variations, and part of the process involves classifying these variations to determine if we need to include them in our training data.</p><p><strong>Important Consideration</strong>: Always assess the client’s environment to understand how objects appear in that setting before choosing training data.</p><p>Let us go through some of the image variations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/964/1*x4gkxv8H0nUVnzVHF-XYRA.png" /></figure><h3>1. Occlusion:</h3><p>This refers to a situation where part of an image is hidden or partially obscured. For example, if an algorithm is trained on perfectly clear images of a person, and it is later given an image where the person is partially covered (e.g., with a hat, glasses, or another object blocking part of the view), the algorithm may struggle to identify or predict accurately. This can lead to reduced performance of the system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/568/1*enjzWW3tTw99OFUhA2Cmzw.jpeg" /><figcaption>Natural occlusion</figcaption></figure><p><strong>Solution</strong>:</p><ul><li>To ensure the robustness of the algorithm, it’s essential to include diverse training examples that account for occlusions and other variations that could be present in real-world scenarios.</li></ul><p>This will help improve the algorithm’s ability to generalize and perform well under a variety of conditions.</p><h3>2. Illumination/Exposure:</h3><p>When training a model for real-world applications, such as a CCTV system for fire detection, it’s crucial to consider variations in illumination and exposure. If a model is trained solely on high-quality images under optimal lighting conditions and then deployed in a CCTV setup, it may perform poorly due to differences in the lighting conditions.</p><p><strong>Challenges:</strong></p><ul><li><strong>Overexposed Images</strong>: When the exposure or highlights are at maximum, the image becomes very bright, and details may be lost. In real-world scenarios, images can be overexposed, leading to challenges in object recognition.</li><li><strong>Underexposed Images</strong>: When brightness, exposure, or highlights are reduced, the image becomes darker and may lose important details, which is common during the evening or in low-light environments.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/568/1*6vbJS41WzhVAxaV8ZeEi3A.jpeg" /><figcaption>Underexposed Image</figcaption></figure><p><strong>Solution</strong>:</p><ul><li>To create a robust model, we need to incorporate both overexposed and underexposed images in the training set. This helps the model learn to handle various lighting conditions and generalize better.</li></ul><p><strong>Use cases:</strong></p><p><strong>Color of Fire</strong>:</p><ul><li><strong>Overexposed Environments</strong>: In an overexposed environment, fire may appear white in images due to the intense lighting.</li></ul><p><strong>Night-time Challenges</strong>:</p><ul><li>At night, many CCTV cameras do not operate in RGB mode; instead, they switch to infrared mode when light levels are too low, capturing greyscale images. Fire may still appear white under infrared lighting due to the way sensors work.</li></ul><h3>3. Scale Variation:</h3><p>In computer vision, scale variation refers to the challenge of detecting and recognizing objects regardless of their size or the distance between the camera and the object. The scale at which objects appear in images can vary widely, which can impact an algorithm’s ability to identify them accurately.</p><p><strong>Challenge</strong>:</p><ul><li>For instance, a model trained to recognize a person or a license plate should be able to identify them at any distance from the camera. This means that even if the number plate is far away on a highway and appears smaller, the algorithm must still detect it correctly.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/1*B6-rxdL_DtSbKzf7gGdX6Q.png" /><figcaption>Original Image</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iYF0Hl20BQZnIaVwXHsH6w.png" /><figcaption>After padding</figcaption></figure><p><strong>Solution:</strong></p><ul><li>To address scale variation, one effective approach is to <strong>pad the image</strong> and place the object of interest (e.g., the number plate) at the center in a larger canvas. This creates an effect where the object appears smaller due to the increased space around it.</li></ul><p>This process simulates the effect of viewing the number plate from a greater distance, helping the algorithm learn to recognize the number plate even when it appears smaller. By training the model on such padded images with the number plate scaled down, it becomes more robust to variations in scale, improving its performance when detecting objects in different camera perspectives or distances.</p><h3>4. Background Variation:</h3><p>Variations in background, such as clutter, patterns, or colors, can confuse models.</p><p><strong>Solution</strong>:</p><ul><li>Collect training images with diverse backgrounds to improve the model’s ability to differentiate objects from their surroundings.</li><li>Use data augmentation techniques to generate varied backgrounds.</li></ul><h3>5. Pose Variation:</h3><p>If we train a model exclusively on images of people standing upright and then use it to classify a person in a sitting position, the model may fail to make accurate predictions because it hasn’t learned the features that distinguish a sitting pose from a standing one.</p><p><strong>Challenge</strong>:</p><ul><li>How can we create a system that reliably determines whether a person is standing or sitting, given that training may only include images of people standing upright?</li></ul><p><strong>Solution:</strong></p><ul><li><strong>Collect a Diverse Dataset</strong>: Ensure the training set includes images of people in various poses, such as standing, sitting, crouching, and other postures. This helps the model learn the visual cues associated with each pose.</li><li><strong>Augmentation Techniques</strong>: Apply data augmentation methods to simulate pose variations. This can include rotation, flipping, and adjustments to the angle of view to make the model more robust.</li></ul><h3>6. Noise and Variations in Image Quality:</h3><p>When training computer vision (CV) models, it’s crucial to consider the impact of various types of noise and image quality variations. If a model is trained exclusively on high-definition (HD) images without noise, it may struggle to identify or process images that have noise or imperfections. This can affect the model’s ability to function effectively in real-world scenarios where images may not be perfect.</p><p><strong>Challenges</strong>:</p><ul><li><strong>Noise</strong>: Random variations in pixel intensity can occur due to various factors such as camera quality, lighting conditions, or transmission errors. If your model is trained only on HD, clear images, it will likely perform poorly when faced with noisy images, which are common in practical applications.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/568/1*_AFoAO8XjVYiZ9CoorFfjw.jpeg" /></figure><p><strong>Solution</strong>:</p><ul><li>By incorporating noisy images, motion-blurred images, and variations in illumination and saturation into the training data, the model can generalize in a better way.</li></ul><h3><strong>Intelligent Datasets</strong>:</h3><p>To create a robust model, the training dataset should include diverse scenarios that represent the real-world conditions the model will face. This means incorporating various types of noise, image qualities, and lighting conditions into the training data.</p><h4>Benefits of Using an Intelligent Dataset:</h4><ul><li><strong>Robustness</strong>: The model becomes more robust and adaptable to real-world scenarios with noise and variations.</li><li><strong>Generalization</strong>: The model can generalize better to new images that it wasn’t explicitly trained on but have similar variations in quality or conditions.</li><li><strong>Improved Performance</strong>: The model’s performance will be more consistent across a variety of images, improving its utility for practical applications.</li></ul><p><strong>Should we include all these variations in our dataset ?</strong></p><ul><li>Not all these variations need to be included in the dataset during model training. The specific variations to address depend entirely on the use case at hand.</li></ul><h3>Conclusion</h3><p>In this blog we covered different image variations to be considered while preparing our training dataset.</p><p>An intelligent dataset — one that includes various scenarios relevant to the use case — enables the model to handle diverse conditions. This helps create a more resilient and capable computer vision system that performs well across different situations and image qualities.</p><h3>Join My Learning Journey in Computer Vision</h3><p>I’m embarking on an exciting journey to explore Computer Vision, and I’m documenting everything I learn through this blog series.</p><p>If you’re interested in diving into this fascinating field together, make sure to <strong>follow</strong> to stay updated with each post.</p><p>Let’s learn, experiment, and grow in Computer Vision as a community!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a4ce9006a7c9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering Text Preprocessing: Essential Techniques for Clean and Efficient Data]]></title>
            <link>https://medium.com/@venkatpendela96/mastering-text-preprocessing-essential-techniques-for-clean-and-efficient-data-76a36890a4fb?source=rss-05858c616c3e------2</link>
            <guid isPermaLink="false">https://medium.com/p/76a36890a4fb</guid>
            <category><![CDATA[text-processing]]></category>
            <category><![CDATA[emoji]]></category>
            <category><![CDATA[punctuation]]></category>
            <category><![CDATA[stop-word]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Venkat]]></dc:creator>
            <pubDate>Tue, 24 Dec 2024 02:24:05 GMT</pubDate>
            <atom:updated>2024-12-24T02:29:05.531Z</atom:updated>
            <content:encoded><![CDATA[<p>Leveraging neural networks in deep learning, systems can handle both images and natural language processing (NLP). When the focus is on images, it’s termed as Computer Vision, when the focus is on languages, it’s NLP.</p><h4>What are we gonna solve using NLP?</h4><p>Use cases of NLP could be summarized into 4 categories on a high level overview:</p><ol><li>Summarization</li><li>Text Classification</li><li>Translation</li><li>Text Generation</li></ol><p>When it comes to NLP, the text we provide needs to be of high quality or preprocessed to have meaningful analysis and get useful insights out of it.</p><h4>What do we mean by text preprocessing?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xMQmZIG4zUEX0WaVbsB3Nw.png" /></figure><p>Text preprocessing involves handling elements such as HTML tags, URLs, stop words, emojis, and punctuation in the text to prepare it for analysis.</p><p>How can we remove them or eliminate them?</p><p>Are we supposed to go through these text preprocessing steps even after the development of advanced models that are available today?</p><h4>Is Text Preprocessing mandatory?</h4><p>When working with advanced technologies like Large Language Models (LLMs), text preprocessing is often not essential, as these models are designed to handle raw and noisy data effectively. However, preprocessing may still be necessary when the data is particularly messy to ensure better results.</p><p>Large Language Models (LLMs) leverage Deep Learning with transformer architectures to process and understand natural language. On the other hand, when working with classical NLP models such as RNNs, LSTMs, or GRUs — commonly referred to as Language Models (LMs) — text preprocessing is a critical step before analysis to ensure the models perform effectively.</p><h4>What are we gonna solve using NLP?</h4><p>Use cases of NLP could be summarized into 4 categories on a high level overview:</p><ol><li>Summarization</li><li>Text Classification</li><li>Translation</li><li>Text Generation</li></ol><p>Let us discuss about some of the text preprocessing techniques that can be used in different scenarios.</p><p>To understand these better lets choose one sample dataset and work on it.</p><p>Dataset in consideration: <a href="https://raw.githubusercontent.com/Ankit152/IMDB-sentiment-analysis/master/IMDB-Dataset.csv">https://raw.githubusercontent.com/Ankit152/IMDB-sentiment-analysis/master/IMDB-Dataset.csv</a></p><p>Lets create a pandas dataframe out of it.</p><pre>import pandas as pd<br>data=pd.read_csv(&quot;https://raw.githubusercontent.com/Ankit152/IMDB-sentiment-analysis/master/IMDB-Dataset.csv&quot;)</pre><p>Lets have a look athe data.</p><pre>data.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/1*9hb0dYC4gLV5hzypOQ0NXA.png" /></figure><p>It is clear from the sample data that we have text data in review column of data frame ‘data’.</p><p>And we can observe that we have html tags included in the data[‘review’]</p><p>Lets remove those html tags.</p><h4>Task 1: Remove HTML tags</h4><p>We can remove HTML tags using regular expressions(regex).</p><pre>import re<br><br>def remove_html_tag(text):<br>    pattern=re.compile(&#39;&lt;.*?&gt;&#39;)<br>    return pattern.sub(&quot;&quot;,text)</pre><p>Before applying on our column lets test the function with some text.</p><pre>sample_text=&quot;&quot;&quot;&lt;!DOCTYPE html&gt;&lt;html lang=&quot;en&quot;&gt;&lt;head&gt;&lt;meta charset=&quot;UTF-8&quot;&gt;&lt;meta http-equiv=&quot;X-UA-Compatible&quot; content=&quot;IE=edge&quot;&gt;&lt;meta name=&quot;viewport&quot; content=&quot;width=device-width, initial-scale=1.0&quot;&gt;&lt;title&gt;Welcome to My Website&lt;/title&gt;&lt;style&gt;body{font-family:&#39;Arial&#39;,sans-serif;background-color:#f0f0f0;color:#333;margin:20px}h1{color:#007bff}p{line-height:1.5}&lt;/style&gt;&lt;/head&gt;&lt;body&gt;&lt;header&gt;&lt;h1&gt;Welcome to My Awesome Website!&lt;/h1&gt;&lt;/header&gt;&lt;main&gt;&lt;p&gt;This is a sample HTML document created for demonstration purposes.&lt;/p&gt;&lt;p&gt;Feel free to explore and enjoy the content on this website.&lt;/p&gt;&lt;/main&gt;&lt;footer&gt;&lt;p&gt;&amp;copy; 2024 My Website. All rights reserved.&lt;/p&gt;&lt;/footer&gt;&lt;/body&gt;&lt;/html&gt;&quot;&quot;&quot;<br><br>updated_text=remove_html_tag(sample_text)<br><br>print(updated_text)</pre><p><strong>Output</strong></p><pre>Welcome to My Websitebody{font-family:&#39;Arial&#39;,sans-serif;background-color:#f0f0f0;color:#333;margin:20px}h1{color:#007bff}p{line-height:1.5}Welcome to My Awesome Website!This is a sample HTML document created for demonstration purposes.Feel free to explore and enjoy the content on this website.&amp;copy; 2024 My Website. All rights reserved.</pre><p>We can observe that we are able to remove html tags from our text, so lets apply the same function on our review column.</p><pre>data[&quot;review&quot;]=data[&quot;review&quot;].apply(remove_html_tag)</pre><pre>data.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/641/1*aX0aiPWr8Fm43nliDWv68g.png" /></figure><p>We removed html tags from our data, we can observe the same in 2nd column.</p><h4>Task 2: Remove URL data</h4><p>We can also remove URLdata available in our text using regex.</p><pre>text_with_url=&#39;Check out my youtube channel: https://www.youtube.com/@sunnysavita10&#39;<br><br>def remove_url(https://medium.com/text):<br>    pattern=re.compile(r&#39;https?://\S+|www\.\S+&#39;)<br>    return pattern.sub(&quot;&quot;,text)<br><br>print(remove_url(https://medium.com/text_with_url))</pre><p><strong>Output</strong></p><pre>Check out my youtube channel: </pre><p>Our URL is removed from our sample text.</p><h4>Task 3: To convert all the text into lower case or upper case</h4><p>Lets convert the text in review column of our dataframe into lower case</p><pre>data.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/641/1*aX0aiPWr8Fm43nliDWv68g.png" /></figure><pre>data[&#39;review&#39;]=data[&#39;review&#39;].str.lower()</pre><pre>data.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/633/1*hOEox-nCKd73WssjLX_JHg.png" /></figure><p>If we wish to convert all the text into upper case, we would just replace lower() with upper().</p><pre>data[&#39;review&#39;]=data[&#39;review&#39;].str.upper()</pre><h4>Task 4: Remove Punctuations</h4><p>We can get list of punctuations from string module.</p><pre>import string<br><br>punctuations=string.punctuation<br>print(punctuations)</pre><p><strong>Output</strong></p><pre>!&quot;#$%&amp;\&#39;()*+,-./:;&lt;=&gt;?@[\\]^_`{|}~</pre><p>Lets define a simple function to remove punctuation out of text provided.</p><pre>def remove_punctuation(text):<br>    for punct in punctuations:<br>        text=text.replace(punct,&quot;&quot;)<br>    return text<br><br>sample_text=&quot;Hello, world! Welcome to NLP—Natural Language Processing. Isn&#39;t it amazing?&quot;<br><br>remove_punctuation(sample_text)</pre><p><strong>Output</strong></p><pre>Hello world Welcome to NLP—Natural Language Processing Isnt it amazing</pre><p>When we wish to convert all the text into lower case or upper case based on requirement.</p><p>We would use lower() or upper() string function to convert the text as required.</p><p>Lets apply the same on our column.</p><pre>data.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/633/1*hOEox-nCKd73WssjLX_JHg.png" /></figure><pre>data[&quot;review&quot;]=data[&quot;review&quot;].apply(remove_punctuation)<br><br>data.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*7F2gTgqa0UDw0KA0ZlztmQ.png" /></figure><p>We can observe the change in 4th and 5th column.</p><h4>Task 5: Handling Abbreviations</h4><p>In some use cases we might need to replace abbreviations with their actual full form, lets look at this now.</p><pre># Lets create a dict with Abbreviations as keys and their full forms as values<br>chat_words={<br>&quot;AFAIK&quot;:&quot;As Far As I Know&quot;,<br>&quot;AFK&quot;: &quot;Away From Keyboard&quot;,<br>&quot;ASAP&quot;:&quot;As Soon As Possible&quot;,<br>&quot;BTW&quot;:&quot;By The Way&quot;,<br>&quot;B4&quot;:&quot;Before&quot;,<br>&quot;LAMO&quot;:&quot;Laugh My A.. Off&quot;,<br>&quot;FYI&quot;:&quot;For your information&quot;  <br>}<br><br><br># Lets use this sample text to replace abbreviations<br>text_1=&quot;FYI this is not true&quot;<br>text_2=&quot;I want it ASAP&quot;<br><br><br># create function to replace abbreviations<br>def chat_conversion(text):<br>    new_text=[]<br>    for w in text.split():<br>        if w.upper() in chat_words:<br>            new_text.append(chat_words[w.upper()])<br>        else:<br>            new_text.append(w)<br>    return &quot; &quot;.join(new_text)<br><br>print(chat_conversion(text_1))<br>print(chat_conversion(text_2))</pre><p><strong>Output</strong></p><pre>For your information this is not true<br>I want it As Soon As Possible</pre><h4>Task 6: Remove emoji’s from our text</h4><p>We can use emoji library and identify and remove emoji’s from our text.</p><pre>import emoji<br><br>def remove_emojis(text):<br>    return emoji.replace_emoji(text, replace=&#39;&#39;)  # Replace emojis with an empty string<br><br><br>sample_text = &quot;Hello 🌟! NLP is fun 🚀🔥. Let&#39;s preprocess text data 😊🎉.&quot;<br><br># Remove emojis<br>cleaned_text = remove_emojis(sample_text)<br>print(cleaned_text)</pre><p><strong>Output</strong></p><pre>Hello ! NLP is fun . Let&#39;s preprocess text data .</pre><h4>Task 7: Remove stop words</h4><p>What are stop words?</p><p>Words like ‘is’, ‘the’, ‘and’, ‘of’, and ‘to’ occur frequently. These words add little meaning to the analysis, yet they dominate the text. If stop words are not removed, they can skew results. The focus should be on meaningful words. By removing stop words, we make the data more concise and relevant for machine learning models. Without this step, algorithms may learn irrelevant patterns. Thus, stopword removal is critical for accurate results.</p><pre>import nltk<br>from nltk.corpus import stopwords<br><br>nltk.download(&quot;stopwords&quot;)  # download stopwords from nltk<br><br>#creating a function to remove stop words<br>def remove_stopword(text):<br>    new_text=[]<br>    for word in text.split():<br>        if word in stopwords.words(&quot;english&quot;):     # stopwords.words(&quot;english&quot;) gives u list of all the stopwords in english.<br>            new_text.append(&quot;&quot;)<br>        else:<br>            new_text.append(word.strip())<br>    return &quot; &quot;.join(new_text)<br><br><br>sample_text=&quot;Hello, world! Welcome to NLP-Natural Language Processing. Isn&#39;t it amazing?&quot;<br><br>cleaned_text=remove_stopword(sample_text)<br>print(cleaned_text)</pre><p><strong>Output</strong></p><pre>Hello, world! Welcome  NLP-Natural Language Processing. Isn&#39;t  amazing?</pre><p>We can observe that the words like ‘to’, ‘it’ are removed, yet we retained the context of the text.</p><h4>Conclusion</h4><p>In this blog we explored essential text processing techniques like removing punctuations, emojis, stop words, HTML tags, URLs. We also covered converting text to lower or upper case and replacing abbreviations with their full forms. These preprocessing steps play a crucial role in preparing text data for Natural Language Processing tasks to ensure meaningful analysis.</p><h4>Join My Learning Journey in Computer Vision</h4><p>I’m embarking on an exciting journey to explore NLP, and I’m documenting everything I learn through this blog series.</p><p>If you’re interested in diving into this fascinating field together, make sure to <strong>follow</strong> to stay updated with each post.</p><p>Let’s learn, experiment, and grow in Computer Vision as a community!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=76a36890a4fb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[RGB vs HSV: Understanding Color Models and Conversion with Ease]]></title>
            <link>https://medium.com/@venkatpendela96/rgb-vs-hsv-52e55e504120?source=rss-05858c616c3e------2</link>
            <guid isPermaLink="false">https://medium.com/p/52e55e504120</guid>
            <category><![CDATA[hsv]]></category>
            <category><![CDATA[rgb2hsv]]></category>
            <category><![CDATA[rgb]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[hue]]></category>
            <dc:creator><![CDATA[Venkat]]></dc:creator>
            <pubDate>Mon, 23 Dec 2024 03:10:34 GMT</pubDate>
            <atom:updated>2024-12-23T03:37:07.656Z</atom:updated>
            <content:encoded><![CDATA[<p>RGB and HSV are two color models and both of them reads images in their own ways, lets explore these two color models.</p><p>Before understanding color models like RGB, HSV, if you wish to look into basics of Images and basic usage of Pillow and OpenCV libraries, have a look at this blog: <a href="https://medium.com/@venkatpendela96/introduction-to-computer-vision-155f12e91a4e">Introduction to Image Processing with Pillow and OpenCV</a></p><h4>RGB:</h4><p>In RGB color model, different colors are represented using three channels Red, Green and Blue.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/445/1*Uvwl-noTtc9tPNjtuUw0ug.png" /></figure><p>And each pixel of image carries these three channels with their respective values and their values represent the contribution of the respective channel to the final color of that pixel.</p><p>Let us have a look at a situation where one channel contributes to the max and other two channels doesn’t contribute anything to the color of pixel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O_bs-jyDCnEnHctNIQ0JYw.png" /></figure><p>Lets have a look at different combinations of values and their resultant as final color.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LtndXx2IBBVle7rlphHySw.png" /></figure><p>Sometimes, we might not be able to identify the variation between different combinations, but it is a different color for system.</p><p><strong>you can play around different combinations of values in each channel </strong><a href="https://www.rapidtables.com/web/color/RGB_Color.html"><strong>here</strong></a><strong>.</strong></p><p>The range of values in all the three channels is 0 to 255 (Channels are typically represented using 8 bit value, so the range turns out to be 0 to 255).</p><p>0 represents no contribution of that channel in final color.</p><p>255 represents maximum contribution from that channel.</p><p>How are we going to get white color?</p><p>We get white color when all three channels contribute to their maximum capacity(R-255, G-255, B-255).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0aDjWOy6kDZyQDSir1mMQA.png" /></figure><p>What happens if no channel contributes to the final color?</p><p>We get pure black when we don’t have any contribution from all the three channels.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XibPxkwtelIFFCDRY5bUFw.png" /></figure><p>Each combination of the values from three channels represents an unique color and thus RGB represents (256 x 256 x 256) different colors.</p><p>And hence RGB is also called as additive model, as a slight change in value of a single channel can vary the color being represented.</p><h3>HSV(Hue Saturation Value)</h3><p>HSV is another popular color model.</p><p>H represents Hue, S represents Saturation, V represents Value.</p><p>What does hue, saturation mean?</p><p>What does this value mean?</p><p>Lets discuss in detail.</p><p>RGB can be represented as a cube, while HSV can be represented as a cone.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p5vzzsMEJgw-yZav5Zgdgw.png" /></figure><p>This image represents complete information about HSV.</p><h4><strong>Hue(H)</strong></h4><p>Base of the cone represents Hue, it represents different colors available and we can access different colors by giving different values for hue in the form of angle.</p><p>From the image we can see that Red is represented around 0 Degree, as we move clock wise we get Green at 120 Degrees and similarly Blue can be seen around 240 Degrees.</p><p>And we can understand that between these colors we have different colors representing different combinations of those colors.</p><p>So, the <strong>range of Hue is (0 to 360)</strong> and each value represents different color.</p><p>we could also refer to range of hue as (0 to 359) as 0 and 360 refers to the same degree in a circle.</p><h4>Saturation(S)</h4><p>Saturation is amount of color / pigment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KDKdfAde5fZsYZue3O9LzA.png" /></figure><p>For example consider an apple, we all know the color of an apple is red.</p><p>Saturation speaks about how red the apple is — bright red or dull red.</p><p>Let us take another example.</p><p>Consider a box of crayons. Saturation is like how <strong>bright and bold</strong> the color is.</p><ul><li>If crayon is super bright, like a shiny red, it has <strong>high saturation</strong>.</li><li>If the same red looks pale or grayish, like it’s been mixed with a lot of white or gray, it has <strong>low saturation</strong>.</li></ul><p>Let us look this difference pictorially.</p><p>Lets fix Hue at 0 and value at 100(max value) and play with saturation value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pv-BmNrBiu3ClR_UbZVmXg.png" /></figure><p>We can see the difference between two variants of red when we change saturation from 100 to 50.</p><p>And we can also see that <strong>range of saturation is (0 to 100.)</strong></p><h4>Value(V)</h4><p>Value represents brightness of color.</p><p>If we consider our previous examples of apple and crayon, we can say the value represents the amount of light available in the room where we stored apple or drawing crayons or amount of light shining on our apple or crayon.</p><p>If light is bright, we can say that the colors look <strong>bright and we find it easy easy to see the color.</strong></p><p>If light is dim, we can still see the color, but not as good as before.</p><p>If we put off the light in the room, it doesn’t matter whether apple or crayon is red or green, all we can see is a black object.</p><p>Lets fix Hue at 0 Degree and Saturation at 100(max value) and play with Value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YPeL7ltY4z5oJIcaqyuEeA.png" /></figure><p>We can see the difference clearly.</p><p>And the <strong>range of value is (0 to 100)</strong>.</p><h4>You can play with different combinations of hue, saturation and value <a href="https://www.selecolor.com/en/hsv-color-picker/">here</a>.</h4><p>What happens if saturation and value goes to zero, i.e their minimum values?</p><p>Lets have a look at it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NUIcb2xYICQKHRauAc11zA.png" /></figure><p>We can observe that if we make saturation 0, we don’t have any importance to hue, as we can see that Hue is set to 0 and that means we are selecting Hue as Red but as soon as we make saturation as 0, hue completely goes blank and turns into white through out the width, and this can be read as if we remove the pigment of the color from object, any object would appear white(not taking Value into consideration here).</p><h4>Without Saturation, Hue doesn’t holds any importance.</h4><p>If we observe second case where we make value zero, we can see that both Saturation and Hue went blank and turned themselves to black, and this can be read as irrespective of the color of the object in a dark room, it appears black to us.</p><h4>Without Value, Hue and Saturation doesn’t hold any importance.</h4><p>At a high level, we could say we can differentiate colors easily using HSV when compared to the same exercise in RGB.</p><h4>HSV with OpenCV</h4><p>When we intend to work with HSV images using OpenCV library, we should remember that ranges of Hue, saturation, value are slightly tweaked in OpenCV.</p><p>Hue: 0 to 180 (we could say 0 to 179 as 0 and 180 are same)</p><p>Saturation: 0 to 255</p><p>Value: 0 to 255</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/672/1*00nW0OuEP5n8E-7AlsAfLQ.png" /></figure><p>The way visualize am image via RGB color model vs HSV color model is different. We are used to look at images in RGB, let us look at difference between RGB vs HSB of an image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vb_49FQtrclzNi70K36Oiw.jpeg" /></figure><p>Left half represent RGB image and right half represents HSV image.</p><h4>How can we convert an RGB image to HSV image?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PAvm6Ew_tujroxYJAMHQHA.png" /></figure><p>We can convert an RGB image to HSV using OpenCV library.</p><p>Before looking at that, lets look the logic behind this conversion.</p><p>Lets take a sample pixel and RGB values looks like this.</p><p>R-200, G-100, B-150</p><p>Can we directly shift these values to hue, saturation, value respectively and call it the HSV replication of the pixel?</p><p>No, it is clear from the nomenclature itself that in RGB we represent contribution of different colors by those values, and in HSV the values represent completely different parameters.</p><p>So, lets look at the logic behind the conversion of RGB to HSV.</p><p>Let r, g, b be the respective values of Red, Green, Blue channels respectively in a pixel.</p><p>Lets look at the algorithm to convert this to HSV values.</p><ol><li>Divide r, g, b by 255</li><li>Compute cmax (max( r, g, b), cmin (min(r, g, b), difference (cmax-cmin)</li><li>Hue(h) calculation : <br>- if cmax and cmin are equal, then h = 0<br>- if cmax equal r then compute h = (60 * ((g — b) / diff) + 360) % 360<br>- if cmax equal g then compute h = (60 * ((b — r) / diff) + 120) % 360<br>- if cmax equal b then compute h = (60 * ((r — g) / diff) + 240) % 360</li><li>Saturation(s) computation : <br>- if cmax = 0, then s = 0<br>- if cmax does not equal 0 then compute s = (diff/cmax)*100</li><li>Value(v) computation<br>- v = cmax*100</li></ol><p>We can get HSV values of a pixel for given RGB values by following these steps.</p><h4>Conversion of RGB to HSV using OpenCV</h4><p>Now lets look into how to convert an RGB image to HSV image using OpenCV library.</p><pre>import cv2<br>import numpy as np<br><br># Load an image in RGB format<br>image_rgb = cv2.imread(&#39;image.jpg&#39;)  # Replace &#39;image.jpg&#39; with your image path<br><br># Convert the image from BGR to RGB (OpenCV loads images in BGR by default)<br>image_rgb = cv2.cvtColor(image_rgb, cv2.COLOR_BGR2RGB)<br><br># Convert the RGB image to HSV<br>image_hsv = cv2.cvtColor(image_rgb, cv2.COLOR_RGB2HSV)</pre><p>image_hsv gives us HSV image after converting image_rgb to HSV.</p><h3>Conclusion</h3><p>In this blog we covered differences between RGB color model and HSV color model and logic behind conversion of an image from from RGB to HSV.</p><h3>Join My Learning Journey in Computer Vision</h3><p>I’m embarking on an exciting journey to explore Computer Vision, and I’m documenting everything I learn through this blog series.</p><p>If you’re interested in diving into this fascinating field together, make sure to <strong>follow</strong> to stay updated with each post.</p><p>Let’s learn, experiment, and grow in Computer Vision as a community!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=52e55e504120" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to Image Processing with Pillow and OpenCV]]></title>
            <link>https://medium.com/@venkatpendela96/introduction-to-computer-vision-155f12e91a4e?source=rss-05858c616c3e------2</link>
            <guid isPermaLink="false">https://medium.com/p/155f12e91a4e</guid>
            <category><![CDATA[pillow]]></category>
            <category><![CDATA[opencv]]></category>
            <category><![CDATA[opencv-python]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Venkat]]></dc:creator>
            <pubDate>Sun, 22 Dec 2024 06:33:41 GMT</pubDate>
            <atom:updated>2024-12-23T03:24:50.973Z</atom:updated>
            <content:encoded><![CDATA[<p>Computer Vision deals with how systems process and interpret images. Leveraging neural networks in deep learning, systems can handle both images and natural language processing (NLP). When the focus is on images, it’s termed as Computer Vision; when on languages, it’s NLP.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3rWE7LCHjvSBEIMZ3FQkUw.png" /></figure><h3>Image</h3><p>Images are composed of pixels, each representing color information at a specific point in the image.</p><h3>Types of Images</h3><p><strong>Grayscale Image: </strong>Black and white images with varying shades of gray. They have a single channel representing light intensity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zgxMgMb2IPLENZh8WfnwlQ.jpeg" /></figure><p><strong>Color Images</strong>: These images have three channels — RGB (Red, Green, Blue). Each pixel contains values for these channels, with the combination representing the final color of the pixel.</p><p>Range of values accepted in each channel of pixel: 0 to 255</p><p>0 represents no contribution of channel</p><p>255 represents maximum contribution of channel</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CnpWI0FUSwR2BISWuFWHGA.png" /></figure><p>Lets have a look at the difference between black and white image and color image in a single frame.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FKhd9y_5IH54JUVBPBeo-Q.jpeg" /></figure><h4>Other Image Models</h4><p>Apart from RGB, other color models include:</p><ol><li>CMYK (used in printing)</li><li>HSV</li><li>HSL</li><li>YIQ</li></ol><h3>Libraries for Computer Vision</h3><p>Commonly used python libraries include:</p><ul><li><strong>Pillow</strong>: For simple tasks like reading, writing, and displaying images.</li><li><strong>OpenCV</strong>: For advanced tasks like numpy operations, image augmentation, and real-time image processing.</li></ul><h3>Creating Images</h3><p>Lets create simple images using pillow and opencv libraries to get better understanding of channels.</p><h4>Using Pillow</h4><pre>pip install pillow<br><br>from PIL import Image<br><br># Creating a grayscale image<br>image_pill_bw = Image.new(&quot;L&quot;, (100, 100), (255))<br># &quot;L&quot;: Lightness (grayscale), (100, 100): Size, (255): Full light intensity<br><br># Creating a color image<br>image_pill_color = Image.new(&quot;RGB&quot;, (100, 100), (255, 0, 0))<br># &quot;RGB&quot;: Color image, (255, 0, 0): Pure red<br><br># Metadata<br>print(image_pill_bw.size)<br>print(image_pill_bw.mode)<br>print(len(image_pill_bw.getbands()))</pre><pre>(100, 100)<br>L<br>1</pre><h4>Using OpenCV</h4><p>We would take help of numpy matrices to create images using OpenCV library.</p><p>We should remember one more thing when it comes to processing images using OpenCV, it reads images as BGR rather than RGB sequence, so we might need to make necessary adjustments for correct results.</p><pre>import cv2<br>import numpy as np<br><br># Creating a color image<br>image_cv = np.zeros((500, 500, 3), np.uint8)<br>image_cv[:] = (0, 0, 255)  # Filling red color<br><br><br># Creating a grayscale image<br>image_cv_bw = np.zeros((500, 500, 1), np.uint8)<br>image_cv_bw[:] = 255  # Full light intensity <br># giving same intensity in all pixels for better understanding<br><br><br>print(image_cv_bw.shape)  # Shape of the grayscale image</pre><pre>(500, 500, 1)</pre><h3><strong>How can we display the image created ?</strong></h3><p><strong>Using Pillow</strong></p><pre>pip install pillow<br><br>from PIL import Image<br><br># Creating a grayscale image<br>image_pill_bw = Image.new(&quot;L&quot;, (100, 100), (255))<br># &quot;L&quot;: Lightness (grayscale), (100, 100): Size, (255): Full light intensity<br><br>image_pill_bw.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/585/1*isRFJ2hZp-3wZ-PqNq2P4A.png" /></figure><p><strong>Using OpenCV</strong></p><pre>import cv2<br>import numpy as np<br><br># Creating a color image<br>image_cv = np.zeros((100, 100, 3), np.uint8)<br>image_cv[:] = (0, 0, 255)  # Filling red color<br><br>cv2.imshow(&#39;Grayscale Image&#39;, image_cv) # Grayscale Image is title of image<br>cv2.waitKey(0) <br>cv2.destroyAllWindows()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*YkXqDICBooPzanzycUQBJA.png" /></figure><p>Now let us look into some of the image types available.</p><ul><li><strong>JPEG (.jpg, .jpeg, .pjp)</strong>: Used for camera pictures. Not the highest quality.</li><li><strong>PNG (.png)</strong>: High-quality images with lossless compression and transparency support. Prefferd over JPEG when we need more precise reproduction of source image.</li><li><strong>HEIC (.heic)</strong>: Used in Apple’s latest technology.</li><li><strong>GIF (.gif)</strong>: For animations.</li><li><strong>TIFF (.tiff, .tif)</strong>: High-quality format, often used for medical imaging like Brain MRIs.</li></ul><h3><strong>How can we write these images created using Pillow and OpenCV libraries to disk?</strong></h3><p><strong>Saving an image using Pillow</strong></p><pre>image_pill.save(&#39;path.jpeg&#39;, quality=95)  <br><br># Quality ranges from 0(worst)-95(best)<br># Values above 95 should be avoided</pre><p>To understand more about optimizations while saving a file using Pillow refer <a href="https://pillow.readthedocs.io/en/stable/handbook/image-file-formats.html">here</a>.</p><p><strong>Saving an image using OpenCV</strong></p><pre>cv2.imwrite(&#39;path.jpeg&#39;, image_cv, [cv2.IMWRITE_JPEG_QUALITY, 100])  <br># JPEG quality : 0(worst)-100(best)<br># Default: 95<br><br><br>cv2.imwrite(&#39;path.png&#39;, image_cv, [cv2.IMWRITE_PNG_COMPRESSION, 3])  <br># PNG compression : 0(least compression)-9(highest compression)<br># Default: 3<br></pre><p>To understand more about cv2.write refer <a href="https://www.geeksforgeeks.org/python-opencv-cv2-imwrite-method/?ref=lbp">here</a>.</p><h3>How can we read images from disk using Pillow and OpenCV libraries?</h3><p><strong>Using Pillow</strong></p><pre>from PIL import Image<br>img = Image.open(&#39;path&#39;) </pre><p><strong>Using OpenCV</strong></p><pre>import cv2<br>img = cv2.imread(&#39;path&#39;)</pre><h3>How can we get metadata details of an image?</h3><p>We can get metadata of image using its exif details.</p><p>EXIF metadata provides details like:</p><ul><li>Orientation</li><li>Device used</li><li>Technology</li><li>Shutter speed</li><li>Dimensions (width, length)</li><li>GPS info</li></ul><p>Lets see how to retrieve Exif details of an image.</p><pre>from PIL import Image, ExifTags<br>img = Image.open(&#39;path&#39;)<br>exif_data = img._getexif()<br>for tag, value in exif_data.items():<br>    tag_name = ExifTags.TAGS.get(tag, tag)<br>    print(f&quot;{tag_name}: {value}&quot;)</pre><h3>When are we supposed to use pillow?</h3><p>For simple tasks like reading, writing, and displaying images.</p><h3>When are we supposed to use OpenCV?</h3><p>For advanced tasks like real-time processing, augmentations, and camera integrations.</p><h3>Conclusion</h3><p>In this blog we covered basic operations like creating, writing, reading images using Pillow and OpenCV libraries.</p><h3>Join My Learning Journey in Computer Vision</h3><p>I’m embarking on an exciting journey to explore Computer Vision, and I’m documenting everything I learn through this blog series.</p><p>If you’re interested in diving into this fascinating field together, make sure to <strong>follow</strong> to stay updated with each post.</p><p>Let’s learn, experiment, and grow in Computer Vision as a community!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=155f12e91a4e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Types of Algorithms in Machine Learning]]></title>
            <link>https://medium.com/@venkatpendela96/types-of-algorithms-in-machine-learning-cd3d713ae344?source=rss-05858c616c3e------2</link>
            <guid isPermaLink="false">https://medium.com/p/cd3d713ae344</guid>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Venkat]]></dc:creator>
            <pubDate>Fri, 20 Dec 2024 12:19:00 GMT</pubDate>
            <atom:updated>2024-12-20T12:19:00.673Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Algorithm in ML:</strong></h3><p>In machine learning (ML), an <strong>algorithm</strong> is a set of rules or procedures that a model follows to learn patterns from data and make predictions or decisions. It’s essentially the “recipe” that tells the computer how to process input data to generate the desired output. In ML, these algorithms can vary widely based on the type of problem they are designed to solve, such as classification, regression, clustering, or recommendation tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*08GeDlsdfnSnIUF3UYnCAQ.png" /></figure><h3><strong>Types of ML Algorithms</strong></h3><p><strong>1. Supervised Learning Algorithms</strong>: These algorithms learn from labeled data, where the input data is paired with the correct output. Examples include:</p><p>- Linear Regression: Predicts continuous values.</p><p>- Decision Trees: A tree-based approach for both classification and regression.</p><ul><li>Support Vector Machines (SVM): Classifies data by finding the best hyperplane.</li></ul><p><strong>2. Unsupervised Learning Algorithms</strong>: These algorithms find patterns in unlabeled data, where the output isn’t provided. Examples include:</p><p>- K-Means Clustering: Groups similar data points into clusters.</p><ul><li>Principal Component Analysis (PCA): Reduces the dimensionality of data.</li></ul><p><strong>3. Reinforcement Learning Algorithms</strong>: These algorithms learn by interacting with an environment, taking actions, and receiving feedback in the form of rewards or penalties. Examples include:</p><p>- Q-Learning: Used in environments where the agent learns to optimize long-term rewards.</p><ul><li>Deep Q-Networks (DQN): Combines neural networks with Q-learning for complex tasks.</li></ul><p><strong>4. Deep Learning Algorithms</strong>: These algorithms are based on neural networks and are particularly useful for handling large, complex datasets (like images or text).</p><p>- Convolutional Neural Networks (CNN): Primarily used for image processing tasks.</p><p>- Recurrent Neural Networks (RNN): Often used for sequential data, like time series or language processing.</p><p>Each algorithm has its strengths and weaknesses, and the choice of algorithm depends on the data type, problem requirements, and goals of the model.</p><p>This blog post is intended to provide a general overview of Artificial Intelligence, Machine Learning, Data Science, and Deep Learning.</p><p>Please feel free to leave a comment below if you notice any errors or have suggestions for improvement. Your feedback could be very helpful.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cd3d713ae344" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Algorithm and Model in Machine Learning]]></title>
            <link>https://medium.com/@venkatpendela96/algorithm-and-model-in-machine-learning-247636cfcbf9?source=rss-05858c616c3e------2</link>
            <guid isPermaLink="false">https://medium.com/p/247636cfcbf9</guid>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[model]]></category>
            <dc:creator><![CDATA[Venkat]]></dc:creator>
            <pubDate>Fri, 20 Dec 2024 11:56:41 GMT</pubDate>
            <atom:updated>2024-12-20T11:56:41.877Z</atom:updated>
            <content:encoded><![CDATA[<p>In machine learning (ML), <strong>models</strong> and <strong>algorithms</strong> are related but distinct concepts, and understanding the difference is key to grasping how ML works.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-XldtvH04u5KA740GZXlpg.png" /></figure><p><strong>1. Algorithm</strong></p><p>An algorithm is the procedure or step-by-step method used to train a model. It’s essentially the “learning method” that a model uses to recognize patterns and relationships in data. Think of it as a recipe that specifies how to process the data, adjust model parameters, and improve predictions. Examples include linear regression, decision trees, and neural networks.</p><p><strong>2. Model</strong></p><p>A model is the result of applying an algorithm to a specific dataset. It’s the final outcome — a program that can take in new data and make predictions or decisions. The model consists of learned parameters that the algorithm has adjusted to best fit the training data.</p><p><strong>How They Work Together ?</strong></p><p>When you choose an algorithm and feed it a dataset, the algorithm processes the data, finding patterns and tuning parameters. Once the training is complete, what you get is the model, which can then be used to make predictions on new, unseen data.</p><p><strong>Example</strong></p><ul><li>Algorithm: Linear Regression ( Used to find linear relation between different variables or columns of training dataset)</li><li>Model: After training, the model is a line that best fits the data points, with specific slope and intercept values adjusted to minimize prediction errors.</li></ul><p>This blog post is intended to provide a general overview of Artificial Intelligence, Machine Learning, Data Science, and Deep Learning.</p><p>Please feel free to leave a comment below if you notice any errors or have suggestions for improvement. Your feedback could be very helpful.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=247636cfcbf9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI, ML, DS, and DL: A Beginner’s Guide to the Future of Technology]]></title>
            <link>https://medium.com/@venkatpendela96/ai-ml-ds-and-dl-a-beginners-guide-to-the-future-of-technology-d3f3c1adbe0e?source=rss-05858c616c3e------2</link>
            <guid isPermaLink="false">https://medium.com/p/d3f3c1adbe0e</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Venkat]]></dc:creator>
            <pubDate>Fri, 20 Dec 2024 11:47:02 GMT</pubDate>
            <atom:updated>2024-12-20T12:05:31.996Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Understanding Machine Learning and Its Connections to AI, Data Science, and Deep Learning</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zfhw1hWGjTYeW7RNwvl0cQ.png" /></figure><p>In the realm of technology today, terms like Machine Learning (ML), Artificial Intelligence (AI), Data Science, and Deep Learning are often used interchangeably, but they represent different concepts. Let’s explore these foundational ideas to clarify their meanings and interrelations.</p><p>To better understand these concepts, consider the <strong>importance of experience in real life</strong>.</p><p>For example, when choosing between two doctors — one with 65 years of experience and another just starting out — most would likely prefer the experienced doctor. Why? Because <strong>experience translates to knowledge</strong> about diagnosing and treating a wide array of symptoms effectively.</p><p>Here, we can think of the <strong>knowledge he possesses as data</strong>: his understanding of various medical conditions based on past cases. The process involves <strong>applying this knowledge to diagnose</strong> current patients accurately and provide appropriate treatments.</p><p>This analogy emphasizes the significance of data and the ability to process it for better decision-making.</p><p>Today, we generate data at an unprecedented rate, and the challenge is how to effectively analyze it.</p><p>Given the sheer volume of data generated daily, it’s impractical for individuals or teams to process it manually. Just as calculators revolutionized mathematics, Machine Learning allows us to automate data processing. It applies statistical and mathematical techniques to extract insights and make predictions based on patterns in the data.</p><p>So, <strong>how do we make predictions using data</strong>? Think about how we can reliably predict the time of sunrise or the order of seasons. These predictions are based on patterns observed over years. For instance, if spring, summer, autumn, and winter have followed a consistent order for several years, we can confidently expect this pattern to continue. However, there’s always uncertainty — floods or droughts can disrupt even the most reliable forecasts.</p><p>Machine Learning models must be updated with new data over time to improve their predictive accuracy.</p><p>What if we could create a machine that updates itself without human intervention? This concept falls under the umbrella of Artificial Intelligence. AI aims to develop applications that can perform tasks autonomously.</p><p>Examples include self-driving cars and recommendation systems, like those used by Netflix, which suggest movies based on your viewing history without human oversight.</p><p>Let us try to define these terms in better words.</p><h3><strong>What is Machine Learning?</strong></h3><p>Machine Learning is a subset of AI focused on enabling machines to learn from data and make predictions or decisions without explicit programming. It involves algorithms that analyze and interpret complex datasets to uncover patterns and insights.</p><h3><strong>The Relationship Between Machine Learning and Other Fields</strong></h3><p><strong>1. Data Science (DS):</strong></p><p>Data Science involves the extraction of knowledge and insights from structured and unstructured data. ML plays a crucial role in data science, providing the tools and methodologies for analyzing large volumes of data to generate predictions and insights.</p><p><strong>2. Deep Learning (DL):</strong></p><p>Deep Learning is a specialized subset of ML that utilizes neural networks with many layers. It excels at processing complex data types such as images, audio, and text, making it a powerful tool within the ML framework.</p><p><strong>3. Artificial Intelligence (AI):</strong></p><p>Machine Learning is a key component of AI, which encompasses any technique that allows machines to mimic human intelligence. While AI includes ML, it also covers other areas like rule-based systems and expert systems.</p><h3><strong>Distinguishing the Terms:</strong></h3><p>While these terms are interconnected, they do not represent the same thing.</p><p>Here’s a brief distinction:</p><p><strong>Data</strong> <strong>Science(DS)</strong>: A field focused on extracting knowledge from data, utilizing techniques from statistics, ML, and computer science.</p><p><strong>Machine Learning(ML)</strong>: A method of enabling machines to learn from data and make predictions or decisions without explicit programming.</p><p><strong>Deep</strong> <strong>Learning(DL)</strong>: A specific approach within ML that employs neural networks for high-level data analysis.</p><p><strong>Artificial Intelligence (AI)</strong>: The broad concept of machines performing tasks intelligently.</p><p>This blog post is intended to provide a general overview of Artificial Intelligence, Machine Learning, Data Science, and Deep Learning.</p><p>Please feel free to leave a comment below if you notice any errors or have suggestions for improvement. Your feedback could be very helpful.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d3f3c1adbe0e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering Virtual Environments in Python: A Comprehensive Guide]]></title>
            <link>https://medium.com/@venkatpendela96/mastering-virtual-environments-in-python-a-comprehensive-guide-927c61cd060e?source=rss-05858c616c3e------2</link>
            <guid isPermaLink="false">https://medium.com/p/927c61cd060e</guid>
            <dc:creator><![CDATA[Venkat]]></dc:creator>
            <pubDate>Wed, 04 Dec 2024 15:21:10 GMT</pubDate>
            <atom:updated>2024-12-20T11:23:30.475Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Introduction to Virtual Environments</strong></p><p>Virtual environments are a crucial aspect of Python development, allowing you to create isolated environments for your projects. This ensures that dependencies and libraries do not conflict across different projects. In this blog, we’ll cover how to set up and manage virtual environments using Python’s venv module, Conda, and Linux commands.</p><p><strong>Using Python’s venv Module</strong></p><ol><li>Create a Virtual Environment:</li></ol><pre>python -m venv venv_name</pre><p>A folder named `venv_name` will be created in your working directory.</p><p>2. Activate the Virtual Environment:</p><pre>venv_name\Scripts\activate</pre><p>3. Install Libraries:<br> Once activated, you can install libraries as and when required for your project using pip.</p><pre>pip install library_name</pre><p>4. Check Python Version:</p><pre>python --version</pre><p>By default, it uses the Python version installed on your system.</p><p>5. Deactivate the Virtual Environment:</p><pre>deactivate</pre><p><strong>Using Conda for Virtual Environments</strong></p><p>1. Create a Virtual Environment with a Custom Python Version:</p><pre>conda create -p venv_name python=3.11 -y</pre><p>2. Activate the Virtual Environment:</p><pre>conda activate ./venv_name</pre><p>3. Deactivate the Virtual Environment:</p><pre>conda deactivate</pre><p><strong>Using Linux Commands for Virtual Environments</strong></p><p>1. Install Virtualenv:</p><pre>pip install virtualenv</pre><p>2. Create a Virtual Environment:</p><pre>virtualenv -p python3 virtual_env</pre><p>3. Activate the Virtual Environment:</p><pre> virtual_env\Scripts\activate</pre><p>4. Deactivate the Virtual Environment:</p><pre>deactivate</pre><p><strong>Conclusion</strong></p><p>Virtual environments are a vital part of Python development, enabling smooth project management and dependency handling. Whether you prefer Python’s built-in venv, Conda, or Linux commands, the choice depends on your specific use case and project requirements.</p><p>Start creating virtual environments today and experience the benefits of isolated project environments!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=927c61cd060e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>