<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nabeel Khan on Medium]]></title>
        <description><![CDATA[Stories by Nabeel Khan on Medium]]></description>
        <link>https://medium.com/@nbeel.original?source=rss-54000852e720------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*gCxDIpP2lO_nioXsP47RDw.jpeg</url>
            <title>Stories by Nabeel Khan on Medium</title>
            <link>https://medium.com/@nbeel.original?source=rss-54000852e720------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 05:06:29 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nbeel.original/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Growing Role of Distributed Computing in Modern AI]]></title>
            <link>https://ai.plainenglish.io/the-growing-role-of-distributed-computing-in-modern-ai-b921bc3829c8?source=rss-54000852e720------2</link>
            <guid isPermaLink="false">https://medium.com/p/b921bc3829c8</guid>
            <dc:creator><![CDATA[Nabeel Khan]]></dc:creator>
            <pubDate>Sun, 07 Sep 2025 18:22:32 GMT</pubDate>
            <atom:updated>2025-09-07T18:22:32.694Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ykPgfBtt-KVRLJ7JRHGtaA.jpeg" /><figcaption>Figure : 01 (Generated By AI)</figcaption></figure><h3>How Distributed Computing is Shaping the Future of AI</h3><p>It’s been almost two years since my last post and things have really been progressing in AI especially LLM’s. It seems like yesterday, when the first version of GPT emerged bringing along a promise of improved computer-centric interactions and saving us countless hours of repetitive task executions with seamless responses. This promise didn’t just meet expectations but also delivered more then anticipated, that too in just a short time span. As new competitors emerged, a whole new race began, which brought to the table features that were once thought far-fetched, decades away or at best straight out of a sci-fi. But this progress came at a cost which includes high compute demands, ginormous data and hard-wares were capable of executing these workloads.</p><p>With the premise set, let us understand how the backstory is at relevance with the scope of our article. The past couple of years I’ve been engaged with all-things AI so had little bandwidth for my writing escapades which was “worth it”. As during the break, I had the incredible opportunity to enrich my experience by working inside a high-performance AI data centre which as the name implies is a place buzzing with GPUs, clusters, and servers humming 24/7. Watching massive models train across tens if not hundreds of machines in real time, while being a living testimony of how tiny delays or misconfigurations could ripple into hours of lost work was inspiring. It was in this environment that the idea for this article came to me, that is to share how distributed computing is not just supporting AI, but actively shaping its future.</p><h3>The AI Evolution and Computational Demands</h3><p>Over the years, AI has grown exponentially. Early models could be trained on a single machine, sometimes in just a few hours. Today, the training of models like GPT-5 or advanced image recognition networks can involve hundreds of GPUs and petabytes of data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/807/1*0u3o7ZJPIKY4f6pU_9fo4Q.png" /><figcaption>Figure : 02 (Evolution of Artificial Intelligence in the last two decades)</figcaption></figure><p>During my time at the data centre, I remember standing beside a 32-GPU cluster and watching one of the engineer optimize every detail including data loading pipelines, network bandwidth and synchronization strategies just to shave off a few hours of training time. It was clear that the traditional centralized systems are no longer enough. The sheer scale of data and computations demanded a distributed approach.</p><h3>Understanding Distributed Computing</h3><p>Distributed computing is all about dividing tasks across multiple machines, each contributing CPU, GPU, and memory resources. Think of it as a team of chefs in a massive kitchen. Instead of one chef cooking everything, one handles desserts, another handles appetizers, and another mains but on individual cooking ranges. As the similar analogy is often used for explaining concurrent processing which differs in availability of only a single cooking range for multiple workers. So in crux, the result from the distributed system is faster, more efficient, and less prone to mistakes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*9ux2bme2RXV3iZNVIjId0A.png" /><figcaption>Figure : 03 ( Use-case and Limitations of Centralized vs Distributed System)</figcaption></figure><h4>Key Advantages</h4><p>From what I understood during my interaction with these large stacks of Machines, the foundational principle is akin to a colloquially used idiom “There is strength in numbers”. Hence below are the few features that make distributed computing standout:</p><ol><li><strong>Scalability:</strong> Add more machines, handle larger datasets.</li><li><strong>Fault tolerance:</strong> If a node fails, others pick up the slack.</li><li><strong>Efficiency:</strong> Tasks run in parallel, dramatically reducing time.</li></ol><p>Distributed computing doesn’t just make workloads faster, it enables AI at a scale we couldn’t dream of before.</p><ul><li><strong>Training large models:</strong> Massive neural networks can span multiple GPUs and servers.</li><li><strong>Real-time inference:</strong> For applications like autonomous vehicles or live translation, distributed systems process data with minimal latency.</li></ul><p>I remember training an LLM model across 64 GPUs where each node performed like a musician in an orchestra where if one played out of sync, the melody suffered. The same applies to the synchronization of each node as proper coordination and synergy between nodes is required for optimal performance and reducing overheads.</p><p>Despite being powerful, these large scale distributed systems come with their own set of limitations owing to which we start getting diminishing returns. These limitations and their possible solutions will be discussed later on in a different setting.</p><h3>Parallel and Distributed Architectures</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*D8WREH1M2DvtV88--6SZsg.png" /><figcaption>Figure : 04 (Comparison between Data and Model Level Parallelism)</figcaption></figure><p>I’ve observed people using parallel and distributed interchangeably in reference to distributed systems. Though it is technically correct, but their is a subtle difference in these two. Parallelism is usually associated with GPU’s because of their structural support for parallel processing wherein one can execute multiple set of instructions simultaneously on a single device within a shared memory paradigm. Where as distributed systems allow parallelism by dividing the workload across different devices achieving the same results. The load distribution for distributed systems can be across both CPU and GPU’s.</p><p>So in context of Large AI models (especially LLM’s), the terms data parallelism and model parallelism are usually a connotation to a distributed system. Wherein either the data is fed in a parallel capacity to multiple machines in chunks (data parallelism) or the model is split in parts and each device in the respective distributed system hosts a portion of the model (model parallelism). In the following parts of the article, the respective jargons will be explained concisely to get a better understanding of core concepts and at the same time improve upon the distributed computing vocabulary.</p><h4>Data Parallelism vs. Model Parallelism</h4><ol><li><strong>Data Parallelism:</strong> Each node uses a full model on a portion of the data.</li><li><strong>Model Parallelism:</strong> Different layers or parts of a model run on different nodes.</li></ol><h4>Data Distribution and Sharding</h4><ol><li><strong>Sharding:</strong> Splitting datasets across nodes to balance load.</li><li><strong>Replication:</strong> Duplicating data to maintain reliability in case of failures.</li></ol><h4>Communication Protocols and Synchronization</h4><ol><li>Efficient message passing ensures nodes share updates quickly.</li><li>Synchronization keeps training consistent across the distributed system.</li></ol><h4>Federated Learning</h4><p>One of the most emerging trend I’ve seen is federated learning, where the training of models is done locally on devices and weight updates are aggregated centrally. This preserves privacy and allows AI to learn across distributed data sources without ever centralizing sensitive data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*4v-NUAC49DiWK7xWyowqpQ.png" /><figcaption>Figure : 05 (Illustration of Federated Learning Architecture)</figcaption></figure><h4>Optimizing Resource Usage</h4><p>During training, even small inefficiencies can snowball, which we remedy by load balancing, memory optimization, and gradient compression to prevent bottlenecks and maximize throughput.</p><h3>Deployment and Productization</h3><p>Once the model is trained, the next critical phase is serving to the end-user. This is where complications arise, mostly due to the environments, packages or other incompatibilities. Therefore, it is imperative to create an image of the model and containerize it for streamlined and scalable use. This is where the deployment and productization pipeline steps in, with each part working in tandem to deliver optimal performance and a seamless experience. The deployment pipeline spans multiple stages starting with containerization, orchestration, CI/CD automation and then concluding with monitoring. Each of these stage is concisely explained in the following section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/764/1*f-MclIA1Isdm0gKZm9fYyw.png" /><figcaption>Figure : 06 (Chronological Flow of AI Deployment Pipeline)</figcaption></figure><h4>Containerization and Orchestration</h4><p><strong>Docker:</strong> Packages AI models with all dependencies for seamless deployment.</p><p><strong>Kubernetes:</strong> Orchestrates containers across clusters, enabling scalability and reliability.</p><h4>CI/CD for AI</h4><p>Automating model deployment ensures updates happen smoothly. Watching a trained model roll out to hundreds of nodes automatically reminded me how far AI operations have come since I first started.</p><h4>Monitoring and Maintenance</h4><ol><li>Real-time monitoring helps detect drifts in model performance.</li><li>Automated retraining keeps AI systems accurate over time.</li></ol><h4>Use Cases and Emerging Trends</h4><ol><li><strong>OpenAI:</strong> GPT models trained on massive multi-node cloud clusters.</li><li><strong>Tesla:</strong> Distributed AI enables real-time decision-making for autonomous driving.</li></ol><h4>Emerging Trends</h4><ol><li><strong>Edge AI:</strong> Pushing intelligence closer to users.</li><li><strong>Multi-cloud AI:</strong> Balancing workloads across multiple cloud providers.</li><li><strong>Decentralized learning:</strong> Collaborative AI training without centralizing data.</li></ol><h3>Challenges and Future Directions</h3><ol><li><strong>Network latency:</strong> Even milliseconds matter in large systems.</li><li><strong>Fault tolerance:</strong> Systems must continue running despite failures.</li><li><strong>Energy efficiency:</strong> Large-scale training consumes significant energy.</li><li><strong>Data consistency:</strong> Synchronizing distributed updates is tricky.</li></ol><p>The road ahead points to fully decentralized intelligence, where AI systems can self-organize and collaborate independently, opening possibilities we can only imagine today.</p><h3>Key Takeaways</h3><ol><li>Distributed computing is essential for modern AI.</li><li>It enables faster training, real-time inference, and scalable deployment.</li><li>Trends like federated learning and edge AI are shaping the next frontier.</li></ol><p>Reflecting on my time in the AI data centre, I can say with conviction that distributed computing is no longer just a backend convenience but rather a backbone of AI innovation. By embracing these architectures, we’re not just making AI faster or larger; we’re reshaping the very way intelligence is built, deployed, and scaled.</p><p>That’s about it, feel free to drop in a 👏 if the article piqued a bit of interest in this emerging domain. Stay connected, as I intend to expand on D-AI series by bringing in hand’s on tutorials for parallel and multi-node systems.</p><p>For updates, follow:</p><ul><li><a href="https://github.com/nbeeeel">nbeeeel - Overview</a></li><li><a href="https://medium.com/@nbeel.original">Nabeel Khan - Medium</a></li></ul><h3><strong>References:</strong></h3><ol><li>Verbraeken, J., Schoonjans, J., Simoens, P., De Turck, F., &amp; Dhoedt, B. (2020). “A Survey on Distributed Machine Learning.” <em>ACM Computing Surveys</em>, 53(2), 1–47.</li><li>McMahan, H. B., Moore, E., Ramage, D., Hampson, S., &amp; y Arcas, B. A. (2017). “Communication-Efficient Learning of Deep Networks from Decentralized Data.” In <em>Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS)</em>, 54, 1273–1282.</li><li>Koubaa, A., Al-Ma’ashri, A., Al-Fuqaha, A., &amp; Chaari, I. (2023). “Distributed artificial intelligence: Taxonomy, review, framework and reference architecture for DAI-as-a-service.” <em>Journal of King Saud University — Computer and Information Sciences</em>, 35(4), 101509.</li><li>Stone, P., &amp; Veloso, M. (2000). “Multiagent systems: A survey from a machine learning perspective.” <em>Artificial Intelligence</em>, 117(2), 187–221.</li></ol><h3>A message from our Founder</h3><p><strong>Hey, </strong><a href="https://linkedin.com/in/sunilsandhu"><strong>Sunil</strong></a><strong> here.</strong> I wanted to take a moment to thank you for reading until the end and for being a part of this community.</p><p>Did you know that our team run these publications as a volunteer effort to over 3.5m monthly readers? <strong>We don’t receive any funding, we do this to support the community. ❤️</strong></p><p>If you want to show some love, please take a moment to <strong>follow me on </strong><a href="https://linkedin.com/in/sunilsandhu"><strong>LinkedIn</strong></a><strong>, </strong><a href="https://tiktok.com/@messyfounder"><strong>TikTok</strong></a>, <a href="https://instagram.com/sunilsandhu"><strong>Instagram</strong></a>. You can also subscribe to our <a href="https://newsletter.plainenglish.io/"><strong>weekly newsletter</strong></a>.</p><p>And before you go, don’t forget to <strong>clap</strong> and <strong>follow</strong> the writer️!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b921bc3829c8" width="1" height="1" alt=""><hr><p><a href="https://ai.plainenglish.io/the-growing-role-of-distributed-computing-in-modern-ai-b921bc3829c8">The Growing Role of Distributed Computing in Modern AI</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[4 Classical Feature Extraction Techniques You Must Try!]]></title>
            <link>https://ai.plainenglish.io/4-classical-feature-extraction-techniques-you-must-try-7ea0969e9c23?source=rss-54000852e720------2</link>
            <guid isPermaLink="false">https://medium.com/p/7ea0969e9c23</guid>
            <category><![CDATA[image-processing]]></category>
            <category><![CDATA[image-recognition]]></category>
            <category><![CDATA[feature-engineering]]></category>
            <category><![CDATA[image-classification]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Nabeel Khan]]></dc:creator>
            <pubDate>Mon, 24 Jul 2023 14:16:57 GMT</pubDate>
            <atom:updated>2023-07-24T14:16:57.860Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ww2QckWK1rbBKJp7yg1Ebw.jpeg" /><figcaption>Image : 1 By <a href="https://unsplash.com/@kmaksimi?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Krzysztof Maksimiuk</a> on <a href="https://unsplash.com/photos/eIcW5IUhpjg?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Image classification has been a hot topic in computer vision domain for an extended period now and will hopefully remain so. Why shouldn’t it be? As time after time has it equipped machines to answer a very fundamental question! <strong><em>“What am I seeing?”</em></strong>. Riveting as it may sound, the answer to this question not only satiates a thirst for something emulating our own perceptual and decision making capabilities but opens myriad realms of creative possibilities. Recent advances in <strong><em>Generative AI</em></strong> shifted the paradigm by harnessing these discerning capabilities to create means of innovation and creativity, achieving apparently insurmountable feats with shear grace. However, every advance in this domain centers around the concept of visual understanding i.e., the ability to effectively discern.</p><p>Feature extraction plays an important role in determining the performance of the discerning model. CNN based feature extraction provides an automated framework, however it lacks explainability and is computationally expensive. Traditional feature extraction techniques on the contrary are labor-intensive and may requires domain knowledge in order to be a better alternative to CNN in specific cases. In this article, we’ll be exploring 4 classical feature extraction techniques for image classification.</p><ol><li><strong>Histograms Of Oriented Gradients</strong></li><li><strong>Local Binary Patterns</strong></li><li><strong>Gabor Filters</strong></li><li><strong>Gray Level Co-occurrence Matrix</strong></li></ol><p>For a simplified understanding the step-wise illustration of the technique along with a descriptive explanation is given below.</p><h3><strong>Histogram Of Oriented Gradients (HOG)</strong></h3><p>HOG represents the distribution of local gradients in an image. It calculates the gradient magnitude and orientation in small image patches and forms a histogram of these orientations, which can be used as features for classification.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/609/1*QjDpgJTlo_giq72iPUqXkg.png" /><figcaption>Image : 2— By Author</figcaption></figure><ul><li>Divide the image into small cells.</li><li>Compute gradient (magnitude and orientation) for each pixel in each cell.</li><li>Create histograms of gradient orientations within each cell.</li><li>Normalize the histograms to account for local variations in lighting and contrast.</li><li>Concatenate the normalized histograms to form the final feature vector representing the image.</li></ul><h3><strong>Local Binary Patterns (LBP)</strong></h3><p>LBP captures the local texture information in an image. It defines a binary pattern for each pixel based on the comparison of its intensity value with the neighboring pixels. Histograms of these patterns can be used as features for classification.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/1*DA2wQRhtBHkFy1ZvXi2UGw.png" /><figcaption>Image : 3 — By Author</figcaption></figure><ul><li>Divide the image into small regions (e.g., pixels in a 3x3 neighborhood).</li><li>Threshold the center pixel value and compare it with its neighboring pixels.</li><li>If a neighbor’s value is greater or equal, assign it a binary value of 1; otherwise, assign 0.</li><li>Convert the binary pattern to a decimal number.</li><li>Construct a histogram of the decimal values across the image to create the LBP feature representation.</li></ul><h3>Gabor Filters</h3><p>Gabor filters are a set of linear filters tuned to different frequencies and orientations. They capture local frequency and orientation information in an image. Gabor filter responses can be used as features for texture classification.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/1*iMdMP3U-BEn1idUw3BksdQ.png" /><figcaption>Image : 4 — By Author</figcaption></figure><ul><li>Define a set of Gabor filters, which are complex sinusoidal wave patterns modulated by a Gaussian envelope.</li><li>Convolve each Gabor filter with the input image.</li><li>The Gabor filter response highlights different frequency and orientation information in the image.</li><li>Extract features from the filtered responses, typically using techniques like mean, variance, or histograms.</li><li>The resulting feature representation captures texture and structural information present in different frequencies and orientations.</li></ul><h3>Gray Level Co-occurrence Matrix (GLCM)</h3><p>Gray Level Co-occurrence Matrix (GLCM) is another classical feature extraction technique for image classification. GLCM captures the spatial relationships between pixels based on their intensity values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/641/1*l_yl9FBBsdz1PakoNGYysw.png" /><figcaption>Image : 5 — By Author</figcaption></figure><ul><li>Convert the image to grayscale if it’s not already.</li><li>Define a distance and angle (or set of angles) to specify pixel pairs’ relative positions.</li><li>Create a co-occurrence matrix by counting the occurrences of pixel intensity pairs at the specified distances and angles.</li><li>Normalize the matrix to get probabilities of pixel intensity co-occurrences.</li><li>Extract statistical features (e.g., contrast, energy, entropy) from the normalized GLCM to represent the image.</li></ul><h3>Key Takeaways</h3><ol><li>HOG (Histogram of Oriented Gradients)</li></ol><ul><li>Effective for detecting object edges and shapes in images.</li><li>Robust to changes in illumination and contrast.</li><li>Widely used in pedestrian detection and object recognition tasks.</li><li>Provides a simple and efficient representation of image gradients.</li></ul><p>2. LBP (Local Binary Pattern)</p><ul><li>Efficient for describing local texture patterns in images.</li><li>Insensitive to changes in lighting and contrast.</li><li>Useful for facial recognition, texture classification, and object detection in textured backgrounds.</li><li>Easy to compute and requires minimal computational resources.</li></ul><p>3. GLCM (Gray-Level Co-occurrence Matrix):</p><ul><li>Captures second-order statistical information about texture and structure in images.</li><li>Sensitive to spatial relationships between pixel intensities.</li><li>Commonly used in texture analysis, remote sensing, and medical imaging.</li><li>Provides features like contrast, energy, and entropy, which help distinguish different textures.</li></ul><p>4. Gabor Filters</p><ul><li>Excellent for capturing texture and frequency information in images.</li><li>Emphasizes specific frequency and orientation components of an image.</li><li>Useful for tasks like texture segmentation, fingerprint recognition, and face recognition.</li><li>Can be combined to form a feature set that encodes various frequency and orientation characteristics.</li></ul><h3>Conclusion</h3><p>In this article, we’ve discussed four classical feature extraction techniques that can be used as an effective alternative to computationally expensive convolutional techniques. In addition, the use-case, strengths and weakness of these techniques are briefly touched. The afore-mentioned techniques can be used individually or in combination to give exceptional results. So you should give these techniques a try and make them a part of your arsenal.</p><p>Follow on <a href="https://medium.com/@nbeel.original">Medium</a>, for more content related to computer vision.</p><p>Till next time, Stay Blessed!</p><h3>References</h3><ol><li>N. Dalal and B. Triggs, “Histograms of oriented gradients for human detection,” 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), San Diego, CA, USA, 2005, pp. 886–893 vol. 1, doi: 10.1109/CVPR.2005.177.</li><li>A. Hadid, “The Local Binary Pattern Approach and its Applications to Face Analysis,” 2008 First Workshops on Image Processing Theory, Tools and Applications, Sousse, Tunisia, 2008, pp. 1–9, doi: 10.1109/IPTA.2008.4743795.</li><li>Haralick, Robert &amp; Shanmugan, K &amp; Dinstein, Itshak. (1973). Texture for image classification. IEEE Trans Syst Man Cyber. 3.</li><li>KAFUO, ASMAA &amp; DIAF, SAMERA &amp; GONIFEDA, AISHA &amp; ALSHATOURI, ZAMZAM &amp; Baba, Abdullatif. (2017). A Literature survey of Gabor filter and its application. 10.13140/RG.2.2.11079.50085.</li></ol><p><em>More content at </em><a href="https://plainenglish.io/"><strong><em>PlainEnglish.io</em></strong></a><em>.</em></p><p><em>Sign up for our </em><a href="http://newsletter.plainenglish.io/"><strong><em>free weekly newsletter</em></strong></a><em>. Follow us on </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>Twitter</em></strong></a>, <a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a><em>, </em><a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong><em>YouTube</em></strong></a><em>, and </em><a href="https://discord.gg/GtDtUAvyhW"><strong><em>Discord</em></strong></a><strong><em>.</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7ea0969e9c23" width="1" height="1" alt=""><hr><p><a href="https://ai.plainenglish.io/4-classical-feature-extraction-techniques-you-must-try-7ea0969e9c23">4 Classical Feature Extraction Techniques You Must Try!</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fuel Your Artistic Vision: A 4-Minute Guide to Neural Style Transfer]]></title>
            <link>https://levelup.gitconnected.com/fuel-your-artistic-vision-a-4-minute-guide-to-neural-style-transfer-7f1ec0e458bc?source=rss-54000852e720------2</link>
            <guid isPermaLink="false">https://medium.com/p/7f1ec0e458bc</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[art]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Nabeel Khan]]></dc:creator>
            <pubDate>Thu, 29 Jun 2023 18:33:33 GMT</pubDate>
            <atom:updated>2023-06-29T22:58:03.532Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qw8L8mNILalPQAMcY3Hqqw.jpeg" /><figcaption>Fig : 01 — Image By Author Generated Using AI</figcaption></figure><p><strong><em>Neural Style Transfer</em></strong> is a fascinating technique, that compliments the <strong><em>Generative AI Family</em></strong> by combining the style of one image with the content of another, creating visually stunning artworks. In just a few minutes, you can transform your dull and lifeless images into captivating pieces that blend different artistic styles. In this quick guide, I’ll walk you through the process of implementing neural style transfer using a pre-build model while providing step-by-step instructions to help you bring your art to life.</p><p>So what would we accomplish in this article? Let’s see!</p><ol><li><strong><em>Implementation Of Fast Neural Style Transfer.</em></strong></li><li><strong><em>Explore effective use cases of this technique.</em></strong></li></ol><h3>Pythonic Implementation</h3><pre>import tensorflow as tf<br>import tensorflow_hub as hub<br><br>import matplotlib.pyplot as plt</pre><p>Importing required libraries.</p><pre>from google.colab import drive<br>drive.mount(&#39;/content/drive&#39;)</pre><p>Mounting the drive, the above code is required for GoogleColab if you want to access the files from your system or drive.</p><pre>def tensor_to_image(tensor):<br>  tensor_shape = tf.shape(tensor)<br>  number_elem_shape = tf.shape(tensor_shape)<br>  if number_elem_shape &gt; 3:<br>    assert tensor_shape[0] == 1<br>    tensor = tensor[0]<br>  return tf.keras.preprocessing.image.array_to_img(tensor) <br><br><br>def load_img(path_to_img):<br>  max_dim = 512<br>  image = tf.io.read_file(path_to_img)<br>  image = tf.image.decode_jpeg(image)<br>  image = tf.image.convert_image_dtype(image, tf.float32)<br><br>  shape = tf.shape(image)[:-1]<br>  shape = tf.cast(tf.shape(image)[:-1], tf.float32)<br>  long_dim = max(shape)<br>  scale = max_dim / long_dim<br><br>  new_shape = tf.cast(shape * scale, tf.int32)<br><br>  image = tf.image.resize(image, new_shape)<br>  image = image[tf.newaxis, :]<br>  image = tf.image.convert_image_dtype(image, tf.uint8)<br><br>  return image<br><br><br>def load_images(content_path, style_path):<br>  content_image = load_img(&quot;{}&quot;.format(content_path))<br>  style_image = load_img(&quot;{}&quot;.format(style_path))<br><br>  return content_image, style_image<br><br><br>def imshow(image, title=None):<br>  if len(image.shape) &gt; 3:<br>    image = tf.squeeze(image, axis=0)<br><br>  plt.imshow(image)<br>  if title:<br>    plt.title(title)<br>    <br>    <br>def show_images_with_objects(images, titles=[]):<br>  if len(images) != len(titles):<br>    return<br><br>  plt.figure(figsize=(20, 12))<br>  for idx, (image, title) in enumerate(zip(images, titles)):<br>    plt.subplot(1, len(images), idx + 1)<br>    plt.xticks([])<br>    plt.yticks([])<br>    imshow(image, title)</pre><p>The above code snippet defines multiple auxiliary functions to load, process and show the image outputs. Below is a brief explanation of each function given above.</p><h4><em>Function tensor_to_image</em></h4><ol><li>Receives an image tensor.</li><li>Check its shape for batch images or individual.</li><li>If images in batch, indexes the images and retrieves individual image tensors.</li><li>Converts tensor into an image object and returns.</li></ol><h4>Function load_img</h4><ol><li>Reads an image file and decodes into respective array format.</li><li>Changes datatype to float, adds in a batch dimension and normalizes the images between 0 and 255.0.</li><li>Scales the image according to the scaling factor.</li><li>Converts image to uint8 format and returns.</li></ol><h4>Function load images</h4><ol><li>Loads the content and style images from their respective paths.</li><li>Returns the content and style images</li></ol><h4>Function im_show and show_images_with_objects</h4><ol><li>Display the image outputs.</li></ol><pre>content_path = &#39;/content/drive/MyDrive/content.jpg&#39;<br>style_path = &#39;/content/drive/MyDrive/style.jpg&#39;</pre><p>Setting up content and style path variables to fetch the content and style image.</p><pre>content_image, style_image = load_images(content_path, style_path)<br>show_images_with_objects([content_image, style_image], <br>                         titles=[f&#39;content image: {content_path}&#39;,<br>                                 f&#39;style image: {style_path}&#39;])</pre><p>Displaying the content and style images for sanity check as shown in figure below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/949/1*NyhAplscV9rgxT8-IyJQew.png" /><figcaption>Fig 02 — image credits (<a href="http://freepik.com/">Freepik</a>)</figcaption></figure><pre>hub_module = hub.load(&#39;https://tfhub.dev/google/magenta/arbitrary-image-stylization-v1-256/2&#39;)</pre><p>Downloading the model from TensorFlow hub.</p><pre>stylized_image = hub_module(tf.image.convert_image_dtype(content_image, tf.float32), <br>                            tf.image.convert_image_dtype(style_image, tf.float32))[0]<br><br>tensor_to_image(stylized_image)</pre><p>Stylizing the content image and displaying the final output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/1*vaZl-Zrl3PpQ0z4BlFaB8w.png" /><figcaption>Fig 03 — Stylized Output From Neural Style Transfer.</figcaption></figure><p>You can play with various combinations of styles and content images. Although using the tf-hub model provides a quick implementation, but there is limited control over the stylizing. To remedy this, we’ll work on an end-to-end style transfer algorithm in some other setting. Meanwhile, let’s look at the potential applications of this algorithm.</p><h3>Use-case of Fast-Neural Style Transfer</h3><ol><li><strong><em>Artistic Rendering</em></strong><em>: Transforming images into unique artistic renditions inspired by famous paintings or styles.</em></li><li><strong><em>Visual Effects in Films and Games:</em></strong><em> Applying specific visual styles to create immersive and visually appealing effects in movies and games.</em></li><li><strong><em>Virtual Reality and Augmented Reality:</em></strong><em> Enhancing the visual aesthetics of VR and AR environments with artistic filters and style transformations.</em></li><li><strong><em>Photo Editing and Filters:</em></strong><em> Developing photo editing applications with personalized artistic filters for enhancing images.</em></li><li><strong><em>Fashion and Design:</em></strong><em> Inspiring designers by applying artistic styles to clothing, textiles, and product designs.</em></li><li><strong><em>Advertisement and Marketing:</em></strong><em> Creating visually captivating imagery with unique artistic styles for advertising and marketing campaigns.</em></li><li><strong><em>Personalization and Customization: </em></strong><em>Allowing individuals to personalize visual content, such as artwork and merchandise, with their preferred artistic styles.</em></li></ol><h3>Conclusion</h3><p>Here in this article, we implemented a fast neural style transfer algorithm courtesy of TensorFlow hub. Moreover, we’ve also explored certain use-case scenarios of this technique to tap into the application side of this phenomenal technique. The fast-style transfer provides fast implementation, however it gives little control over the saturation and the extent of transformation. Therefore, it’s essential to get our hands dirty with the end-to-end implementation. Which I’ll try to bring in future.</p><p>I hope this content was of value, and feel free to appreciate the content for more content of this lineament. Follow on <a href="https://medium.com/@nbeel.original">Medium</a>.</p><p>Till next time, Stay Blessed!</p><h3>References</h3><ol><li>Golnaz Ghiasi, Honglak Lee, Manjunath Kudlur, Vincent Dumoulin, Jonathon Shlens. <a href="https://arxiv.org/abs/1705.06830">Exploring the structure of a real-time, arbitrary neural artistic stylization network</a>. Proceedings of the British Machine Vision Conference (BMVC), 2017.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7f1ec0e458bc" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/fuel-your-artistic-vision-a-4-minute-guide-to-neural-style-transfer-7f1ec0e458bc">Fuel Your Artistic Vision: A 4-Minute Guide to Neural Style Transfer</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Effectively compress your deep learning models with this powerful technique.]]></title>
            <link>https://levelup.gitconnected.com/effectively-compress-your-deep-learning-models-with-this-powerful-technique-c723ee8696df?source=rss-54000852e720------2</link>
            <guid isPermaLink="false">https://medium.com/p/c723ee8696df</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Nabeel Khan]]></dc:creator>
            <pubDate>Sun, 25 Jun 2023 21:00:19 GMT</pubDate>
            <atom:updated>2023-06-25T21:00:19.286Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sPBSrXMPkjykmcAQVGzlEg.png" /><figcaption>Figure 01 — Image by author generated using AI</figcaption></figure><p>Not so long ago, I researched for effective means to reduce the parameters of my model while retaining performance. Although, there were some over the counter methods, but I was looking for something different. Something that wasn’t a cliched size versus accuracy trade-off. So I kept looking and looking, until I landed on a study whose mere title made more sense to me than my accumulated literature study on parametric reduction. The seminal paper by <strong><em>Geoffrey Hinton et. al</em> </strong>titled <strong>“Distilling the knowledge in a Neural Network”<em> </em></strong>provided a novel take on the compression method for deep learning models that i merrily utilized for my intended application.</p><p>The all mysterious and incredible compression technique that we were discussing about is <strong><em>“Knowledge Distillation” </em></strong>as suggested by the title of the paper. Its an effective and proven method, described justly by the analogy of a teacher-student paradigm. We know that <strong><em>“Applied Knowledge is Power”</em></strong> or at-least now we know! Therefore, I’ll be <strong><em>distilling </em></strong>this comprehensive topic to a more condensed and concise list of keypoints for a quick grasp of the concept prior to moving towards its implementation on Cifar-10 dataset.</p><p>For some added clarity, lets see what we’ll be covering in this article</p><ol><li><strong>Introduction To Knowledge Distillation</strong></li><li><strong>Pythonic Implementation of Knowledge Distillation on Cifar-10 Dataset.</strong></li></ol><h3>Knowledge Distillation</h3><p>Knowledge Distillation involves a sharing of knowledge between a teacher <strong><em>(Larger Model)</em></strong> and a student <strong><em>(Smaller Model)</em></strong> while maintaining a similar performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*p_LLEIXaQVtz29p-IJkTYA.png" /><figcaption>Figure 02 — Image by author.</figcaption></figure><p>Here’s a quick reference to a general working of knowledge distillation process :</p><ol><li><strong>Hierarchical</strong> <strong>Knowledge Sharing : T</strong>raining a larger and complex model called a teacher model to train a smaller student model.</li><li><strong>Probability Distribution Based Targets </strong>: Student model is trained on probability distribution of classes obtained via teacher model instead of one-hot labels. These probability values are also called soft-targets.</li><li><strong>Loss Function</strong> : Student model’s predicted probabilities are optimized to minimize its difference with the soft-targets. Kulback Divergence is commonly used for its ability to measure difference between two probability distributions.</li><li><strong>Knowledge Transfer </strong>: During the replication of the probability distribution, the student model learns the underlying patterns and crucial decision making embedded within the teacher model. Hence replicating or sometimes enhancing the performance of the student model.</li></ol><p>Now as we’ve introduced ourselves with the Knowledge Distillation process, we’ll head to our coding part. So without wasting much time let dig into it.</p><h3>Pythonic Implementation</h3><pre>import tensorflow as tf<br>from tensorflow import keras</pre><p>Importing the required libraries.</p><pre># Loading and preprocessing the dataset<br>(x_train, y_train), (x_test, y_test) = keras.datasets.cifar10.load_data()<br>x_train = x_train.astype(&quot;float32&quot;) / 255.0<br>x_test = x_test.astype(&quot;float32&quot;) / 255.0</pre><p>Downloading and preprocessing the dataset from Keras for the classification pipeline.</p><pre>teacher_model = keras.applications.vgg16.VGG16(<br>    weights=&quot;imagenet&quot;, include_top=False, input_shape=(32, 32, 3)<br>)<br># Freeze the layers of the pre-trained VGG-16<br>for layer in teacher_model.layers:<br>    layer.trainable = False</pre><p>Downloading VGG-16 model for feature extraction and as a teacher model. The layers are frozen to allow transfer learning.</p><pre># Add a classifier on top of the pre-trained layers<br>flatten_layer = keras.layers.Flatten()(teacher_model.output)<br>dense_layer = keras.layers.Dense(256, activation=&quot;relu&quot;)(flatten_layer)<br>output_layer = keras.layers.Dense(10, activation=&quot;softmax&quot;)(dense_layer)<br><br># Create the teacher model<br>teacher_model = keras.models.Model(teacher_model.input, output_layer)<br><br># Compile and train the teacher model<br>teacher_model.compile(<br>    optimizer=&quot;adam&quot;,<br>    loss=&quot;sparse_categorical_crossentropy&quot;,<br>    metrics=[&quot;accuracy&quot;]<br>)<br>teacher_model.fit(<br>    x_train, y_train,<br>    batch_size=32,<br>    epochs=25,<br>    validation_data=(x_test, y_test)<br>)</pre><p>Defining and compiling the teacher model for training. Due to the demonstrative nature of this code, the parameters are not changed. The number of epochs is kept at 25 for a quick training. You can tune the parameters for desired results.</p><p>After being trained for 25 epochs the performance of the model can be seen in the snippet below. The validation accuracy is at 0.6024 and the loss at 1.4020. For reminder, we’re not concerned with the evaluation results, just the implementation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FpeKTrnGVwpLZBt_3fqxwg.png" /><figcaption>Figure 03— Evaluation of Teacher Model at 25th Epoch</figcaption></figure><pre># Generate soft targets using the teacher model<br>soft_targets_train = teacher_model.predict(x_train)<br>soft_targets_test = teacher_model.predict(x_test)</pre><p><strong>Here comes the actual part that concerns the distillation process. </strong>After training the teacher model in the same way we normally do our training drills, we’ll extract the probability distribution also called the soft targets for each class using the predict method.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/509/1*iij3tXNGTqN1HaJ5hRfwoQ.png" /><figcaption>Figure 04 — Soft Target Extraction</figcaption></figure><pre>student_model = keras.models.Sequential([<br>  keras.layers.Conv2D(32, (3, 3), activation=&quot;relu&quot;, input_shape=(32, 32, 3)),<br>  keras.layers.Conv2D(16,(3,3,),activation=&quot;relu&quot;),<br>  keras.layers.MaxPooling2D(),<br>  keras.layers.Flatten(),<br>  keras.layers.Dense(64, activation=&quot;relu&quot;),<br>  keras.layers.Dense(10, activation=&quot;softmax&quot;)<br>])</pre><p>After extracting the soft targets i.e., the class probabilities from the Teacher Model, we’ll define a custom architecture for the student model.</p><pre>#Defining distillation loss<br>def soft_loss(y_true, y_pred, temperature=1.0):<br>    y_true = tf.nn.softmax(y_true / temperature)<br>    y_pred = tf.nn.softmax(y_pred / temperature)<br>    return tf.keras.losses.KLDivergence()(y_true, y_pred)</pre><p>Prior to compiling, we’ll define a custom soft loss function that we’ll be passing into the loss parameter during compilation. The soft loss function takes in three parameters i.e., ground truths, predictions and the temperature. The temperature parameter represents the softness or hardness of the target distribution and its value range is (1 — 20). The higher the temperature value the smoother will be the distribution and the smaller value gives much sharper probabilities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*yHhZXsok0z2qXh3HTztQSw.png" /><figcaption>Figure 05 — Image By Author ( Effect Of Temperature On Probability Distribution)</figcaption></figure><p>By increasing to a higher temperature value we can create a uniformity in probability distribution as can be seen in the figure above. This can cause the model to be less confident about its predictions and more focused on exploring other nuanced relationships between the classes also referred to as <strong><em>Dark Knowledge</em></strong>. This knowledge helps the student model achieve better performance.</p><p>As for the <strong><em>y_true</em></strong> and <strong><em>y_pred</em></strong> in the soft loss function, both these values are divided by the <strong><em>temperature parameter</em></strong> to adjust the softness. After which, a softmax probability is applied to convert the outputs into respective probability distributions. These probability distributions are then fed to a Kulback Divergence Loss function which calculates the discrepancy between the predicted and ground truth probability values.</p><pre>student_model.compile(<br>    optimizer=&quot;adam&quot;,<br>    loss=[lambda y_true, y_pred: soft_loss(y_true, y_pred, temperature=5.0)],<br>    metrics=[&quot;accuracy&quot;]<br>)<br>student_model.fit(<br>    x_train, soft_targets_train,<br>    batch_size=32,<br>    epochs=10,<br>    validation_data=(x_test, soft_targets_test)<br>)</pre><p>After setting up the loss function and the student model architecture, the model is compiled using the soft loss function we defined earlier. Additionally, the model is trained on the same input given to the teacher model but this time the <strong><em>hard labels</em></strong> i.e., the one-hot-encoded values are replaced by the respective <strong><em>soft labels</em></strong> or the probability distribution values of the logits. Below is the performance of the distilled model after being trained for <strong><em>10 epochs</em></strong>. Honestly, the model was not improving on validation accuracy so I preferred to keep the training under 10 epochs. You can play with the student model’s architecture and the parameters of the teacher model to improve the performance.</p><pre>test_loss, test_acc = student_model.evaluate(x_test, soft_targets_test)<br>print(&quot;Test loss:&quot;, test_loss)<br>print(&quot;Test accuracy:&quot;, test_acc)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*cI6yOhH19PQ8F4IflQWSqw.png" /><figcaption>Figure 06 — Evaluation of Student Model after being trained for 10 epochs.</figcaption></figure><h3>Potential Shortcomings</h3><p>Knowledge Distillation is indeed a powerful compression methodology, however it may fall victim to these drawbacks.</p><ol><li>High dependency on teacher model’s performance.</li><li>Computational Complexity due to the training of two models.</li><li>Optimal tuning of various hyperparameters required for better performance.</li><li>May lack interpretability.</li><li>Risks of overfitting by excessive mimicking of teacher model.</li></ol><h3>Conclusion</h3><p>Knowledge Distillation like most other efficient techniques, needs a scenario based requirement and optimal parametric configuration to shine. Here in this article we introduced ourselves to the basic working of Knowledge Distillation process along with its pythonic implementation. The motive was to learn and implement, however the practical nature of this session constrained the in-depth analysis of the technique. Hence, we’ll be exploring the subject meticulously in another setting.</p><p>Do let me know if you have any suggestions and connect! <a href="https://medium.com/@nbeel.original">Medium</a></p><h3>References</h3><p>Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean. “Distilling the knowledge in a neural network.” <em>arXiv preprint arXiv:1503.02531</em> (2015)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c723ee8696df" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/effectively-compress-your-deep-learning-models-with-this-powerful-technique-c723ee8696df">Effectively compress your deep learning models with this powerful technique.</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Easily build a Web App with Haar Cascades and OpenCV.js]]></title>
            <link>https://medium.com/@nbeel.original/easily-build-a-web-app-with-haar-cascades-and-opencv-js-aa46be637096?source=rss-54000852e720------2</link>
            <guid isPermaLink="false">https://medium.com/p/aa46be637096</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[face-recognition]]></category>
            <dc:creator><![CDATA[Nabeel Khan]]></dc:creator>
            <pubDate>Thu, 22 Jun 2023 19:21:44 GMT</pubDate>
            <atom:updated>2023-06-22T19:21:44.728Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HPvIOX9kLTp4-YOFNC9cLQ.png" /><figcaption>Fig — 01 Image By Author Inspired By (<a href="https://www.freepik.com/free-vector/face-recognition-biometric-scan-cyber-security-technology-blue-tone_16406566.htm#query=facial%20recognition&amp;position=33&amp;from_view=keyword&amp;track=ais">rawpixel</a>)</figcaption></figure><p>I’ve been building computer vision based applications for some hands on practice and curious indulgences, but they were mostly desktop native. However, just recently I came across a client who needed an application built using opencv.js. It was a fairly complex problem, which needed days to figure out considering my average java-scripting experience.</p><p>Although, I pulled through but it was a challenging ordeal. So I thought to work on my web-development skills before landing another order. After some experience, I learned that the most effective method to enhance my skills was to adopt a project-based approach and assist fellow learners in the process.</p><p>Here I am, starting the journey by trying to build a face detection web application using none other then Haar cascade and our very own OpenCV.js.</p><p>So without wasting any time, lets dive into the coding part!</p><h3>Setting up the VS code editor</h3><p>Before actually commencing our coding expedition, we need to setup our editor. Here’s what we’ll do</p><ol><li>Install the Live Server extension.</li><li>Install Auto Rename Tag</li><li>Install Prettier</li></ol><p>We don’t need to hard code the installation process, for those who are new to the platform just navigate to the top left — just below the Run and Debug you’d see the “<strong>Extensions”</strong> icon. From there you’d know what to do!</p><h3>Downloading Requisites</h3><p>After setting the code editor, we need to download opencv.js package and the Haar cascade xml files. To download the opencv.js package files, copy the entire script from <a href="https://docs.opencv.org/3.4.0/opencv.js">here</a> and save it on a notepad as a opencv.js file. Move the saved opencv.js file to your project directory. Now download the Haar-cascade trained xml’s for face and eyes detection from this <a href="https://github.com/opencv/opencv/tree/master/data/haarcascades">git repo</a>. Similarly, move the downloaded xml file to the project directory.</p><h3>JavaScripting</h3><h3>Utility.js</h3><p>Now we’re all set to start the coding process. We’ll be dividing the java-scripting in two files. In the first file named “Utils.js”, we’ll be setting up the library, file fetching, canvas display and camera initialization process. The second file would focus solely on defining the logic of Haar cascade and related operations.</p><pre>function Utils(errorOutputId) { <br>    let self = this;<br>    this.errorOutput = document.getElementById(errorOutputId);</pre><p>Initializing a Utils(utility) Class and assigning an error output element to display the error messages.</p><pre>const OPENCV_URL = &#39;opencv.js&#39;;<br>    this.loadOpenCv = function(onloadCallback) {<br>        let script = document.createElement(&#39;script&#39;);<br>        script.setAttribute(&#39;async&#39;, &#39;&#39;);<br>        script.setAttribute(&#39;type&#39;, &#39;text/javascript&#39;);<br>        script.addEventListener(&#39;load&#39;, () =&gt; {<br>                // Checking the loading of the buildInformation<br>           if (cv.getBuildInformation)<br>            {<br>                console.log(cv.getBuildInformation());<br>                onloadCallback();<br>            }<br>            else<br>            {<br>                // Web Assembly check<br>                cv[&#39;onRuntimeInitialized&#39;]=()=&gt;{<br>                    console.log(cv.getBuildInformation());<br>                    onloadCallback();<br>                }<br>            }<br>        });<br>        script.addEventListener(&#39;error&#39;, () =&gt; {<br>            self.printError(&#39;Failed to load &#39; + OPENCV_URL);<br>        });<br>        script.src = OPENCV_URL;<br>        let node = document.getElementsByTagName(&#39;script&#39;)[0];<br>        node.parentNode.insertBefore(script, node);<br>    };</pre><p>Dynamically loading the OpenCV.js library in the browser. It creates a Script element and sets some required attributes. Event Listeners are added to handle the error reports and the loading of the script. The conditional statement <strong>“if-else”</strong> checks the “<strong><em>cv.getBuildInformation”</em></strong> if it exists or not. If it checks positive, the <strong>“<em>onloadCallback()</em>”</strong> function is invoked else it indicates the library is using Web-Assembly(WASP) so it creates an <strong><em>“onRuntimeInitialized”</em></strong> callback to be called when Runtime is initialized. In short the above snippets ensures appropriate loading of the opencv.js library.</p><pre>this.createFileFromUrl = function(path, url, callback) {<br>        let request = new XMLHttpRequest();<br>        request.open(&#39;GET&#39;, url, true);<br>        request.responseType = &#39;arraybuffer&#39;;<br>        request.onload = function(ev) {<br>            if (request.readyState === 4) {<br>                if (request.status === 200) {<br>                    let data = new Uint8Array(request.response);<br>                    cv.FS_createDataFile(&#39;/&#39;, path, data, true, false, false);<br>                    callback();<br>                } else {<br>                    self.printError(&#39;Failed to load &#39; + url + &#39; status: &#39; + request.status);<br>                }<br>            }<br>        };<br>        request.send();<br>    };</pre><p>This snippet is useful for fetching image or video data from external sources, but we’ll be using the live webcam to capture the results in Realtime. The <strong><em>createFileFromUrl</em></strong> method fetches a file from a given URL and creates a virtual file in the virtual file system. It uses an <strong>XMLHttpRequest</strong> to retrieve the file as an array buffer. On successful retrieval, it converts the response to a <strong>Uint8Array </strong>and creates the virtual file using <strong><em>“cv.FS_createDataFile”</em></strong>. The callback function is invoked once the file creation is complete.</p><pre>this.loadImageToCanvas = function(url, cavansId) {<br>        let canvas = document.getElementById(cavansId);<br>        let ctx = canvas.getContext(&#39;2d&#39;);<br>        let img = new Image();<br>        img.crossOrigin = &#39;anonymous&#39;;<br>        img.onload = function() {<br>            canvas.width = img.width;<br>            canvas.height = img.height;<br>            ctx.drawImage(img, 0, 0, img.width, img.height);<br>        };<br>        img.src = url;<br>    };</pre><p>The above snippet is responsible for creating a canvas object to display the image. The <strong><em>“loadImageToCanvas”</em></strong> function takes two elements i.e., a url and a canvas Id. The let canvas variable fetches the canvas element from the html document using the <strong><em>“canvasId”</em></strong> passed as a parameter. To draw on the canvas, we’ll use the <strong><em>.getContext(‘2d’)</em></strong> method to define the rendering context. We’ll then create an image object to represent the image that’ll be loaded, followed by defining crossOrigin property to <strong><em>‘anonymous’</em>. </strong>This ensures that images can be accessed from different domains. Then we have an onload event handler, that defines the post-loading events for an image. The succeeding lines equal the image width and height to that of the canvas, followed by a <strong><em>drawImage</em></strong> function to overlay the image on a specified coordinate and shape. Finally the <strong><em>img.src </em></strong>fetches the image from the url.</p><pre>this.executeCode = function(textAreaId) {<br>        try {<br>            this.clearError();<br>            let code = document.getElementById(textAreaId).value;<br>            eval(code);<br>        } catch (err) {<br>            this.printError(err);<br>        }<br>    };<br>this.clearError = function() {<br>        this.errorOutput.innerHTML = &#39;&#39;;<br>    };<br>    this.printError = function(err) {<br>        if (typeof err === &#39;undefined&#39;) {<br>            err = &#39;&#39;;<br>        } else if (typeof err === &#39;number&#39;) {<br>            if (!isNaN(err)) {<br>                if (typeof cv !== &#39;undefined&#39;) {<br>                    err = &#39;Exception: &#39; + cv.exceptionFromPtr(err).msg;<br>                }<br>            }<br>        } else if (typeof err === &#39;string&#39;) {<br>            let ptr = Number(err.split(&#39; &#39;)[0]);<br>            if (!isNaN(ptr)) {<br>                if (typeof cv !== &#39;undefined&#39;) {<br>                    err = &#39;Exception: &#39; + cv.exceptionFromPtr(ptr).msg;<br>                }<br>            }<br>        } else if (err instanceof Error) {<br>            err = err.stack.replace(/\n/g, &#39;&lt;br&gt;&#39;);<br>        }<br>        this.errorOutput.innerHTML = err;<br>    };</pre><p>This part of the code is intended for exception handling. The excuteCode function takes <strong><em>“textAreaId”</em></strong> as a parameters and executes the code while the catch block holds any error associated with the execution. The <strong><em>clearError</em></strong> function set the <strong><em>errorOutput</em></strong> element to an empty string. The entire snippet follows the same evaluation format by catching any errors that occur, and formats the error messages based on the type of error encountered before displaying them in the error output element.</p><pre>this.loadCode = function(scriptId, textAreaId) {<br>        let scriptNode = document.getElementById(scriptId);<br>        let textArea = document.getElementById(textAreaId);<br>        if (scriptNode.type !== &#39;text/code-snippet&#39;) {<br>            throw Error(&#39;Unknown code snippet type&#39;);<br>        }<br>        textArea.value = scriptNode.text.replace(/^\n/, &#39;&#39;);<br>    };</pre><p>The above code snippet defines a function that loads code from a script element and populates <strong><em>textArea</em></strong> element with that code. the <strong><em>loadCode</em></strong> function takes two parameters i.e., the <strong><em>scriptId</em></strong> and the <strong><em>textAreaId</em></strong> which are later fetched from html and stored in <strong><em>scriptNode</em></strong> and textArea using the <strong><em>.getElementById</em></strong>. Later an exception check is initiated by ensuring the <strong><em>scriptNode.type</em></strong> to be code-snippet. The text content of the script element after successful conformation gets assigned to the value property of the <strong><em>textArea</em></strong>.</p><pre>function onVideoCanPlay() {<br>        if (self.onCameraStartedCallback) {<br>            self.onCameraStartedCallback(self.stream, self.video);<br>        }<br>    };</pre><p>This snippet defines a function <strong><em>onVideoCanPlay()</em></strong> that is typically used as an event handler. When the video can be played, it checks if the<strong><em> “onCamerStartedCallback”</em></strong> function is defined. If it is, the <strong><em>“onCamerStartedCallback”</em></strong> function is called, passing the <strong><em>self.stream</em></strong> and <strong><em>self.video</em></strong> as arguments. The purpose of this function is to trigger a callback function when the video is ready to play, allowing further actions or processing to be performed.</p><pre>this.startCamera = function(resolution, callback, videoId) {<br>        const constraints = {<br>            &#39;qvga&#39;: {width: {exact: 320}, height: {exact: 240}},<br>            &#39;vga&#39;: {width: {exact: 640}, height: {exact: 480}}};<br>        let video = document.getElementById(videoId);<br>        if (!video) {<br>            video = document.createElement(&#39;video&#39;);<br>        }<br>let videoConstraint = constraints[resolution];<br>        if (!videoConstraint) {<br>            videoConstraint = true;<br>        }<br>        navigator.mediaDevices.getUserMedia({video: videoConstraint, audio: false})<br>            .then(function(stream) {<br>                video.srcObject = stream;<br>                video.play();<br>                self.video = video;<br>                self.stream = stream;<br>                self.onCameraStartedCallback = callback;<br>                video.addEventListener(&#39;canplay&#39;, onVideoCanPlay, false);<br>            })<br>            .catch(function(err) {<br>                self.printError(&#39;Camera Error: &#39; + err.name + &#39; &#39; + err.message);<br>            });<br>    };<br>    this.stopCamera = function() {<br>        if (this.video) {<br>            this.video.pause();<br>            this.video.srcObject = null;<br>            this.video.removeEventListener(&#39;canplay&#39;, onVideoCanPlay);<br>        }<br>        if (this.stream) {<br>            this.stream.getVideoTracks()[0].stop();<br>        }<br>    };<br>};</pre><p>The code defines two functions, “<strong><em>startCamera”</em></strong> and “<strong><em>stopCamera”</em></strong>, which are responsible for controlling the camera stream.</p><p>The “<strong><em>startCamera”</em></strong> function takes three parameters: “<strong><em>resolution”</em></strong>, “<strong><em>callback”</em></strong>, and “<strong><em>videoId”</em></strong>. It sets up the desired resolution options and creates a video element for displaying the camera stream. It then uses the “<strong><em>navigator.mediaDevices.getUserMedia”</em></strong> method to request access to the camera’s video stream. If the request is successful, the stream is assigned to the video element’s “<strong><em>srcObject”</em></strong> property, and the stream starts playing. The function also assigns the video element, stream, and callback to corresponding properties for later use. Additionally, an event listener is added to the video element for the <strong><em>“canplay”</em></strong> event, which triggers the <strong><em>onVideoCanPlay</em></strong> function.</p><p>On the other hand, the “<strong>stopCamera”</strong> function doesn’t take any parameters. It checks if the video element exists and, if so, pauses the video playback, clears the <strong><em>srcObject</em></strong> property to stop the camera stream, and removes the <strong>“<em>canplay</em>”</strong> event listener. Furthermore, if the stream exists, it stops the video tracks associated with the stream.</p><p>With this comes the end of our utils.js file, now we’ll move towards the haar.js file to finalize our Haar detector. But before that, here’s the entire snippet of the Utils.js.</p><pre>function Utils(errorOutputId) { <br>    let self = this;<br>    this.errorOutput = document.getElementById(errorOutputId);<br>const OPENCV_URL = &#39;opencv.js&#39;;<br>    this.loadOpenCv = function(onloadCallback) {<br>        let script = document.createElement(&#39;script&#39;);<br>        script.setAttribute(&#39;async&#39;, &#39;&#39;);<br>        script.setAttribute(&#39;type&#39;, &#39;text/javascript&#39;);<br>        script.addEventListener(&#39;load&#39;, () =&gt; {<br>            if (cv.getBuildInformation)<br>            {<br>                console.log(cv.getBuildInformation());<br>                onloadCallback();<br>            }<br>            else<br>            {<br>                cv[&#39;onRuntimeInitialized&#39;]=()=&gt;{<br>                    console.log(cv.getBuildInformation());<br>                    onloadCallback();<br>                }<br>            }<br>        });<br>        script.addEventListener(&#39;error&#39;, () =&gt; {<br>            self.printError(&#39;Failed to load &#39; + OPENCV_URL);<br>        });<br>        script.src = OPENCV_URL;<br>        let node = document.getElementsByTagName(&#39;script&#39;)[0];<br>        node.parentNode.insertBefore(script, node);<br>    };<br>    this.createFileFromUrl = function(path, url, callback) {<br>        let request = new XMLHttpRequest();<br>        request.open(&#39;GET&#39;, url, true);<br>        request.responseType = &#39;arraybuffer&#39;;<br>        request.onload = function(ev) {<br>            if (request.readyState === 4) {<br>                if (request.status === 200) {<br>                    let data = new Uint8Array(request.response);<br>                    cv.FS_createDataFile(&#39;/&#39;, path, data, true, false, false);<br>                    callback();<br>                } else {<br>                    self.printError(&#39;Failed to load &#39; + url + &#39; status: &#39; + request.status);<br>                }<br>            }<br>        };<br>        request.send();<br>    };<br>    this.loadImageToCanvas = function(url, cavansId) {<br>        let canvas = document.getElementById(cavansId);<br>        let ctx = canvas.getContext(&#39;2d&#39;);<br>        let img = new Image();<br>        img.crossOrigin = &#39;anonymous&#39;;<br>        img.onload = function() {<br>            canvas.width = img.width;<br>            canvas.height = img.height;<br>            ctx.drawImage(img, 0, 0, img.width, img.height);<br>        };<br>        img.src = url;<br>    };<br>    this.executeCode = function(textAreaId) {<br>        try {<br>            this.clearError();<br>            let code = document.getElementById(textAreaId).value;<br>            eval(code);<br>        } catch (err) {<br>            this.printError(err);<br>        }<br>    };<br>    this.clearError = function() {<br>        this.errorOutput.innerHTML = &#39;&#39;;<br>    };<br>    this.printError = function(err) {<br>        if (typeof err === &#39;undefined&#39;) {<br>            err = &#39;&#39;;<br>        } else if (typeof err === &#39;number&#39;) {<br>            if (!isNaN(err)) {<br>                if (typeof cv !== &#39;undefined&#39;) {<br>                    err = &#39;Exception: &#39; + cv.exceptionFromPtr(err).msg;<br>                }<br>            }<br>        } else if (typeof err === &#39;string&#39;) {<br>            let ptr = Number(err.split(&#39; &#39;)[0]);<br>            if (!isNaN(ptr)) {<br>                if (typeof cv !== &#39;undefined&#39;) {<br>                    err = &#39;Exception: &#39; + cv.exceptionFromPtr(ptr).msg;<br>                }<br>            }<br>        } else if (err instanceof Error) {<br>            err = err.stack.replace(/\n/g, &#39;&lt;br&gt;&#39;);<br>        }<br>        this.errorOutput.innerHTML = err;<br>    };<br>    this.loadCode = function(scriptId, textAreaId) {<br>        let scriptNode = document.getElementById(scriptId);<br>        let textArea = document.getElementById(textAreaId);<br>        if (scriptNode.type !== &#39;text/code-snippet&#39;) {<br>            throw Error(&#39;Unknown code snippet type&#39;);<br>        }<br>        textArea.value = scriptNode.text.replace(/^\n/, &#39;&#39;);<br>    };<br>    this.addFileInputHandler = function(fileInputId, canvasId) {<br>        let inputElement = document.getElementById(fileInputId);<br>        inputElement.addEventListener(&#39;change&#39;, (e) =&gt; {<br>            let files = e.target.files;<br>            if (files.length &gt; 0) {<br>                let imgUrl = URL.createObjectURL(files[0]);<br>                self.loadImageToCanvas(imgUrl, canvasId);<br>            }<br>        }, false);<br>    };<br>    function onVideoCanPlay() {<br>        if (self.onCameraStartedCallback) {<br>            self.onCameraStartedCallback(self.stream, self.video);<br>        }<br>    };<br>    this.startCamera = function(resolution, callback, videoId) {<br>        const constraints = {<br>            &#39;qvga&#39;: {width: {exact: 320}, height: {exact: 240}},<br>            &#39;vga&#39;: {width: {exact: 640}, height: {exact: 480}}};<br>        let video = document.getElementById(videoId);<br>        if (!video) {<br>            video = document.createElement(&#39;video&#39;);<br>        }<br>        let videoConstraint = constraints[resolution];<br>        if (!videoConstraint) {<br>            videoConstraint = true;<br>        }<br>        navigator.mediaDevices.getUserMedia({video: videoConstraint, audio: false})<br>            .then(function(stream) {<br>                video.srcObject = stream;<br>                video.play();<br>                self.video = video;<br>                self.stream = stream;<br>                self.onCameraStartedCallback = callback;<br>                video.addEventListener(&#39;canplay&#39;, onVideoCanPlay, false);<br>            })<br>            .catch(function(err) {<br>                self.printError(&#39;Camera Error: &#39; + err.name + &#39; &#39; + err.message);<br>            });<br>    };<br>    this.stopCamera = function() {<br>        if (this.video) {<br>            this.video.pause();<br>            this.video.srcObject = null;<br>            this.video.removeEventListener(&#39;canplay&#39;, onVideoCanPlay);<br>        }<br>        if (this.stream) {<br>            this.stream.getVideoTracks()[0].stop();<br>        }<br>    };<br>};</pre><h3>Haar.js</h3><p>As we’re done with the utility setup, we’ll code the detection part by creating a file named haar.js. We’ll start the coding now, without any delays!</p><pre>let isFaceDetection = true; // Flag to indicate whether face detection is enabled<br>function switchDetection() {<br>    isFaceDetection = !isFaceDetection; // Toggle the flag<br>}</pre><p>The above code snippet initializes with a detection check for the face strictly followed by a toggle function <strong><em>switchDetection()</em></strong>. This switch function is used to add an additional toggling feature in the application, such that the user can navigate between the face detection or the face and eyes detection by clicking the switch button. The function of the switch button is thoroughly explained in the code below.</p><pre>function addNavigationButtons() {<br>    // Create a button for switching detection<br>    let switchButton = document.createElement(&#39;button&#39;);<br>    switchButton.textContent = &#39;Switch Detection&#39;;<br>    switchButton.addEventListener(&#39;click&#39;, switchDetection);<br>// Append the button to the body<br>    // Get the button container element<br>    let buttonContainer = document.getElementById(&#39;buttonContainer&#39;);<br>    // Append the button to the button container<br>    buttonContainer.appendChild(switchButton);<br>}</pre><p>This code is quite simple to follow. The <strong><em>addNavigationButtons() </em></strong>function first creates a “<strong><em>switchButton”</em></strong> variable to create a button element and add in the text content of choice. The next line is the event handling for the switch button by passing in the event name <strong><em>‘click’ </em></strong>and the callback function <strong><em>‘switchDetection()’</em> </strong>that we created above. Next we fetch the button container through the id defined in the html and store it inside the <strong><em>buttonContainer</em></strong> variable. Don’t worry we’ll get to see the html part in the final stages of this project. The last thing we’ll do is add our <strong><em>switchButton</em></strong> as a child element to the fetched <strong><em>“buttonContainer”</em></strong> element from the html. This is done to allow the access inside the html editor for style or other required changes to this button element.</p><pre>function openCvReady() {<br>    cv[&#39;onRuntimeInitialized&#39;] = () =&gt; {<br>        let video = document.getElementById(&quot;cam_input&quot;); // video is the id of video tag<br>        navigator.mediaDevices.getUserMedia({ video: true, audio: false })<br>        .then(function(stream) {<br>            video.srcObject = stream;<br>            video.play();<br>        })<br>        .catch(function(err) {<br>            console.log(&quot;An error occurred! &quot; + err);<br>        });<br>        let src = new cv.Mat(video.height, video.width, cv.CV_8UC4);<br>        let dst = new cv.Mat(video.height, video.width, cv.CV_8UC1);<br>        let gray = new cv.Mat();<br>        let cap = new cv.VideoCapture(cam_input);<br>        let faces = new cv.RectVector();<br>        let eyes = new cv.RectVector();<br>        let faceClassifier = new cv.CascadeClassifier();<br>        let eyeClassifier = new cv.CascadeClassifier();<br>        let utils = new Utils(&#39;errorMessage&#39;);<br>        let faceCascadeFile = &#39;haarcascade_frontalface_default.xml&#39;; // path to face cascade xml<br>        let eyeCascadeFile = &#39;haarcascade_eye.xml&#39;; // path to eye cascade xml<br>        utils.createFileFromUrl(faceCascadeFile, faceCascadeFile, () =&gt; {<br>            faceClassifier.load(faceCascadeFile); // in the callback, load the face cascade from file<br>        });<br>        utils.createFileFromUrl(eyeCascadeFile, eyeCascadeFile, () =&gt; {<br>            eyeClassifier.load(eyeCascadeFile); // in the callback, load the eye cascade from file<br>        });<br>        const FPS = 24;</pre><p>Here the <strong><em>opencvReady()</em></strong> function is a callback which initiates with the loading of the <strong><em>OpenCV</em></strong> library. <strong><em>onRuntimeInitialized </em></strong>is an event that is triggered when the library is initialized and a class-back function is assigned to it using an arrow. Inside the callback the video element is retrieved using the ID <strong><em>“cam_input”</em></strong>. Then a promise based function <strong><em>“navigator.mediaDevices.getUserMedia” </em></strong>is used to request access to user’s camera which upon access returns a promise using <strong><em>then()</em></strong> function. It assigns the stream to the <strong><em>“srcObject”</em></strong> property of the video element and starts playing the video. The <strong><em>catch( ) </em></strong>method is used to log errors. Next, several variables and objects are initialized. These include <strong><em>src</em></strong> , <strong><em>dst </em></strong>and <strong><em>gray</em></strong>, which are instances of OpenCV matrices used for image processing. <strong><em>cap</em></strong> is a <strong><em>VideoCapture </em></strong>object that provides access to video frames. <strong><em>faces</em></strong> and <strong><em>eyes </em></strong>are <strong><em>RectVector </em></strong>objects that will store the detected faces and eyes, respectively. <strong><em>faceClassifier</em></strong> and <strong><em>eyesClassifier </em></strong>are instances of <strong><em>CascadeClassifier </em></strong>, which are used to load and apply trained cascade classifiers for face and eye detection. An instance of the <strong><em>Utils </em></strong>class is created, which provides utility functions. The constructor of <strong><em>Utils</em></strong> takes an argument, “<strong><em>errorMessage”</em></strong>, which is the ID of an HTML element used to display error messages. The <strong><em>utils.createFileFromUrl()</em></strong> function is called to asynchronously load the cascade classifier XML files. It takes the file path, a callback function, and optional parameters. In the callback function, the XML file is loaded using <strong><em>faceClassifer.load() </em></strong>and <strong>eyeClassifier.load() </strong>methods. Finally, <strong><em>FPS </em></strong>is set to 24, indicating the desired frames per second for video processing.</p><pre>function processVideo() {<br>            let begin = Date.now();<br>            cap.read(src);<br>            src.copyTo(dst);<br>            cv.cvtColor(dst, gray, cv.COLOR_RGBA2GRAY, 0);<br>if (isFaceDetection) {<br>                try {<br>                    faceClassifier.detectMultiScale(gray, faces, 1.1, 3, 0);<br>                    console.log(faces.size());<br>                } catch (err) {<br>                    console.log(err);<br>                }<br>                for (let i = 0; i &lt; faces.size(); ++i) {<br>                    let face = faces.get(i);<br>                    let point1 = new cv.Point(face.x, face.y);<br>                    let point2 = new cv.Point(face.x + face.width, face.y + face.height);<br>                    cv.rectangle(dst, point1, point2, [255, 0, 0, 255]);<br>                }<br>            } else {<br>                try {<br>                    faceClassifier.detectMultiScale(gray, faces, 1.1, 3, 0);<br>                    eyeClassifier.detectMultiScale(gray, eyes, 1.1, 3, 0);<br>                    console.log(eyes.size());<br>                } catch (err) {<br>                    console.log(err);<br>                }<br>                for (let i = 0; i &lt; faces.size(); ++i) {<br>                    let face = faces.get(i);<br>                    let point1 = new cv.Point(face.x, face.y);<br>                    let point2 = new cv.Point(face.x + face.width, face.y + face.height);<br>                    cv.rectangle(dst, point1, point2, [255, 0, 0, 255]);<br>                }<br>                for (let i = 0; i &lt; eyes.size(); ++i) {<br>                    let eye = eyes.get(i);<br>                    let point1 = new cv.Point(eye.x, eye.y);<br>                    let point2 = new cv.Point(eye.x + eye.width, eye.y + eye.height);<br>                    cv.rectangle(dst, point1, point2, [0, 255, 0, 255]);<br>                }<br>            }<br>            cv.imshow(&quot;canvas_output&quot;, dst);<br>            // schedule next one.<br>            let delay = 1000 / FPS - (Date.now() - begin);<br>            setTimeout(processVideo, delay);<br>        }<br>        // Add navigation buttons<br>        addNavigationButtons();<br>        // schedule first one.<br>        setTimeout(processVideo, 0);<br>    };</pre><p>The <strong><em>“processVideo()” </em></strong>function is responsible for processing video frames in real-time. It starts by capturing a frame from the video source and converting it to grayscale. The function then determines whether to perform face detection or both face and eye detection based on the <strong><em>“isFaceDetection”</em></strong> flag. If face detection is enabled, it uses a face classifier to detect faces in the frame and draws rectangles around them. If both face and eye detection are enabled, it also uses an eye classifier to detect eyes within the detected faces and draws rectangles around them. The processed frame with the drawn rectangles is displayed on a canvas. The function then calculates the delay required to achieve a desired frame rate and schedules the next frame processing using <strong><em>setTimeout()</em></strong>. Additionally, the <strong><em>addNavigationButtons()</em></strong> function is called to add buttons for toggling between face and eye detection. Overall, this code allows for real-time video processing with face and eye detection capabilities.</p><p>With this comes the end of the JavaScripting part. Before moving forward to the html let’s just take a look at the entire Haar.js part.</p><pre>let isFaceDetection = true; // Flag to indicate whether face detection is enabled<br>function switchDetection() {<br>    isFaceDetection = !isFaceDetection; // Toggle the flag<br>}<br>function addNavigationButtons() {<br>    // Create a button for switching detection<br>    let switchButton = document.createElement(&#39;button&#39;);<br>    switchButton.textContent = &#39;Switch Detection&#39;;<br>    switchButton.addEventListener(&#39;click&#39;, switchDetection);<br>    // Append the button to the body<br>    // Get the button container element<br>    let buttonContainer = document.getElementById(&#39;buttonContainer&#39;);<br>    // Append the button to the button container<br>    buttonContainer.appendChild(switchButton);<br>}<br>function openCvReady() {<br>    cv[&#39;onRuntimeInitialized&#39;] = () =&gt; {<br>        let video = document.getElementById(&quot;cam_input&quot;); // video is the id of video tag<br>        navigator.mediaDevices.getUserMedia({ video: true, audio: false })<br>        .then(function(stream) {<br>            video.srcObject = stream;<br>            video.play();<br>        })<br>        .catch(function(err) {<br>            console.log(&quot;An error occurred! &quot; + err);<br>        });<br>        let src = new cv.Mat(video.height, video.width, cv.CV_8UC4);<br>        let dst = new cv.Mat(video.height, video.width, cv.CV_8UC1);<br>        let gray = new cv.Mat();<br>        let cap = new cv.VideoCapture(cam_input);<br>        let faces = new cv.RectVector();<br>        let eyes = new cv.RectVector();<br>        let faceClassifier = new cv.CascadeClassifier();<br>        let eyeClassifier = new cv.CascadeClassifier();<br>        let utils = new Utils(&#39;errorMessage&#39;);<br>        let faceCascadeFile = &#39;haarcascade_frontalface_default.xml&#39;; // path to face cascade xml<br>        let eyeCascadeFile = &#39;haarcascade_eye.xml&#39;; // path to eye cascade xml<br>        utils.createFileFromUrl(faceCascadeFile, faceCascadeFile, () =&gt; {<br>            faceClassifier.load(faceCascadeFile); // in the callback, load the face cascade from file<br>        });<br>        utils.createFileFromUrl(eyeCascadeFile, eyeCascadeFile, () =&gt; {<br>            eyeClassifier.load(eyeCascadeFile); // in the callback, load the eye cascade from file<br>        });<br>        const FPS = 24;<br>        function processVideo() {<br>            let begin = Date.now();<br>            cap.read(src);<br>            src.copyTo(dst);<br>            cv.cvtColor(dst, gray, cv.COLOR_RGBA2GRAY, 0);<br>            if (isFaceDetection) {<br>                try {<br>                    faceClassifier.detectMultiScale(gray, faces, 1.1, 3, 0);<br>                    console.log(faces.size());<br>                } catch (err) {<br>                    console.log(err);<br>                }<br>                for (let i = 0; i &lt; faces.size(); ++i) {<br>                    let face = faces.get(i);<br>                    let point1 = new cv.Point(face.x, face.y);<br>                    let point2 = new cv.Point(face.x + face.width, face.y + face.height);<br>                    cv.rectangle(dst, point1, point2, [255, 0, 0, 255]);<br>                }<br>            } else {<br>                try {<br>                    faceClassifier.detectMultiScale(gray, faces, 1.1, 3, 0);<br>                    eyeClassifier.detectMultiScale(gray, eyes, 1.1, 3, 0);<br>                    console.log(eyes.size());<br>                } catch (err) {<br>                    console.log(err);<br>                }<br>                for (let i = 0; i &lt; faces.size(); ++i) {<br>                    let face = faces.get(i);<br>                    let point1 = new cv.Point(face.x, face.y);<br>                    let point2 = new cv.Point(face.x + face.width, face.y + face.height);<br>                    cv.rectangle(dst, point1, point2, [255, 0, 0, 255]);<br>                }<br>                for (let i = 0; i &lt; eyes.size(); ++i) {<br>                    let eye = eyes.get(i);<br>                    let point1 = new cv.Point(eye.x, eye.y);<br>                    let point2 = new cv.Point(eye.x + eye.width, eye.y + eye.height);<br>                    cv.rectangle(dst, point1, point2, [0, 255, 0, 255]);<br>                }<br>            }<br>            cv.imshow(&quot;canvas_output&quot;, dst);<br>            // schedule next one.<br>            let delay = 1000 / FPS - (Date.now() - begin);<br>            setTimeout(processVideo, delay);<br>        }<br>        // Add navigation buttons<br>        addNavigationButtons();<br>        // schedule first one.<br>        setTimeout(processVideo, 0);<br>    };<br>}</pre><h3>Html</h3><p>We’ll now be coding the html part for basic interfacing and getting things together.</p><pre>&lt;!DOCTYPE html&gt;<br>&lt;html lang=&quot;en&quot;&gt;<br>&lt;head&gt;<br>    &lt;meta charset=&quot;UTF-8&quot;&gt;<br>    &lt;meta name=&quot;viewport&quot; content=&quot;width=device-width, initial-scale=1.0&quot;&gt;<br>    &lt;title&gt;Opencv JS&lt;/title&gt;<br>    &lt;style&gt;<br>        body {<br>            font-family: Arial, sans-serif;<br>            background-color: #f4f4f4;<br>            margin: 0;<br>            padding: 0;<br>            display: flex;<br>            justify-content: center;<br>            align-items: center;<br>            height: 100vh;<br>            flex-direction: column;<br>        }<br>        h2, h3 {<br>            text-align: center;<br>            color: #333;<br>            margin: 0;<br>            padding: 10px;<br>        }<br>        video, canvas {<br>            display: block;<br>            margin: 0 auto;<br>            background-color: #000;<br>            box-shadow: 0 2px 6px rgba(0, 0, 0, 0.3);<br>        }<br>        #buttonContainer {<br>            padding: 7px 15px;<br>            font-size: 10px;<br>            background-color: #4CAF50;<br>            color: #fff;<br>            border: none;<br>            border-radius: 2px;<br>            cursor: pointer;<br>            transition: background-color 0.3s ease;<br>        }<br>        #buttonContainer:hover {<br>            background-color: #45a049;<br>        }<br>    &lt;/style&gt;<br>    &lt;script async src=&quot;js/opencv.js&quot; onload=&quot;openCvReady();&quot;&gt;&lt;/script&gt;<br>    &lt;script src=&quot;js/haar.js&quot;&gt;&lt;/script&gt;<br>    &lt;script src=&quot;js/utils.js&quot;&gt;&lt;/script&gt;<br>&lt;/head&gt;<br>&lt;body&gt;<br>    &lt;video id=&quot;cam_input&quot; height=&quot;320&quot; width=&quot;480&quot;&gt;&lt;/video&gt;<br>    &lt;canvas id=&quot;canvas_output&quot;&gt;&lt;/canvas&gt;<br>    &lt;h2&gt;Haar Cascade Based Face and Eyes Detection&lt;/h2&gt;<br>    &lt;h3&gt;Click the button to toggle between Simple Face Detection and Face + Eyes Detection.&lt;/h3&gt;<br>    &lt;div id=&quot;buttonContainer&quot;&gt;<br>    &lt;/div&gt;<br>    &lt;/body&gt;<br>&lt;/html&gt;</pre><p>The code is basic html coding and nothing so complex to need explanation. However, its important to understand the path to the opencv.js library file that we downloaded earlier. As you can see, the opencv.js, haar.js and utils.js files are all in the same folder <strong><em>“js”</em></strong>, whereas<strong><em> </em></strong>the html file for the above code is placed alongside the “<strong><em>js”</em></strong> directory. You can adjust the paths according to your requirements.</p><p>Below is the output for both the face detection and face and eyes detection. Pardon the background</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J4N66T4cocHmQo-LqTduGg.png" /><figcaption>Fig — 02 Image By author. (Haar Cascade Face Detection)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1-XNtPPutS0uph77-T2T0A.png" /><figcaption>Fig — 03 Image By author. (Haar Cascade Face and Eyes Detection)</figcaption></figure><p>Download the entire code and auxiliary files from my <a href="https://github.com/nbeeeel/haar-cascade-web-application-using-opencv.js">Github repo</a>.</p><h3>Conclusion</h3><p>In this tutorial we implemented Haar cascade face and eye detection using opencv.js on our web application. We also went through the details of the code in order to learn the purpose of each method. This way we’ll be learning both computer vision and web development. I hope this tutorial adds some value to your learning journey and if some stuff is missed or you have some suggestions, do let me know. If you want the code and the related files do visit my <a href="https://github.com/nbeeeel/haar-cascade-web-application-using-opencv.js">Github</a> for this project.</p><p><strong><em>Till next time….. Stay Blessed!</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aa46be637096" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting Started with Depth Estimation using MiDaS and Python]]></title>
            <link>https://medium.com/@nbeel.original/getting-started-with-depth-estimation-using-midas-and-python-d0119bfe1159?source=rss-54000852e720------2</link>
            <guid isPermaLink="false">https://medium.com/p/d0119bfe1159</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Nabeel Khan]]></dc:creator>
            <pubDate>Mon, 12 Jun 2023 09:23:22 GMT</pubDate>
            <atom:updated>2023-06-12T11:23:31.985Z</atom:updated>
            <content:encoded><![CDATA[<p>Getting Started with Depth Estimation using MiDaS and Python</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GOIdx95IUMsHNP7YnIkzdw.png" /><figcaption>Fig — 01 Image by Author Inspired by <a href="https://pixabay.com/users/sambeetarts-1339110/">Sambeetarts</a> on <a href="https://pixabay.com/illustrations/geometric-design-computer-1732847/">Pixabay</a></figcaption></figure><p>Measuring<strong> </strong>distance of an object from camera poses a significant challenge within the computer vision domain, due to the lack of inherent depth information in 2D images, perspective distortion, variable object sizes, camera calibration requirements, and occlusion in complex scenes. Distance estimation via perspective projection, for instance, relies on variables such as the size of the sensor, focal length, and the actual height of the object. The computation of these unknown variables adds to the complexity of the task.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/1*tTdzp9ChZ6uT47gCXMzAmg.png" /><figcaption>Fig — 02 (Formula for Distance to Object)</figcaption></figure><p>A series of traditional and deep learning based approaches have been in service for quite some time to provide an effective solution to distance estimation. Solutions involving stereo vision have proven to be effective and accurate in depth calculation, however there’s always a search for a much efficient and inexpensive alternative. Deep learning has shown itself to thrive in such constraints, pushing the boundaries of possibilities to a whole new level and bringing to reality depth estimation models for monocular vision — one of which we’ll be exploring in this article.</p><p>Following my recent <a href="https://medium.com/artificialis/swift-and-simple-calculate-object-distance-with-ease-in-just-few-lines-of-code-38889575bb12?source=your_stories_page-------------------------------------">distance measurement using media-pipe</a> tutorial, I intended to build upon the notion of monocular depth estimation by bringing all possible approaches — to provide you with best options for your fun projects.</p><p>In this article, I’ll be estimating the distance of an object using a hybrid of media-pipe pose estimation module and MiDaS depth estimation model. But before that, let’s just take a quick overview of what we’ll be covering in this article.</p><ol><li><strong>MiDaS overview.</strong></li><li><strong>Distance measurement using Media-pipe landmarks and MiDaS depth map.</strong></li></ol><h3>MiDaS</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iHcmPCWez148p32CP2nToA.png" /><figcaption>Fig — 03 (MiDaS Depth Estimation) Image by Author Inspired by <a href="https://pixabay.com/users/mikes-photography-1860391/">Mikes-Photography</a> on <a href="https://pixabay.com/videos/cars-motorway-speed-motion-traffic-1900/">Pixabay</a></figcaption></figure><p>MiDaS(Multiple Depth Estimation Accuracy with Single Network) is a deep learning based residual model built atop Res-Net for monocular depth estimation. MiDaS is known to have shown promising results in depth estimation from single images. Below is the generic overview of the MiDaS architecture:</p><ol><li><strong>Encoder-Decoder Architecture</strong></li></ol><p>MiDaS is based on an encoder-decoder architecture, where the encoder part is responsible for high level feature extraction and the decoder generates the depth map from these features via up-sampling.</p><p>2. <strong>Backbone</strong></p><p>MiDaS typically uses a residual network (ResNet-50 or ResNet-101) for feature extraction because it is robust to vanishing gradients. Allowing MiDaS to extract multi-channeled feature maps from input images, capturing hierarchical information at varying scales.</p><p>3. <strong>Multi-Scale Feature Fusion</strong></p><p>Skip connections and feature fusion is incorporated within MiDaS to allow accurate depth estimation. Feature maps from earlier layers are connected to the later layers via skip connections for accessing low level details during up-sampling. With feature fusion, the multi-scale feature maps are combined to ensure the effective exploitation of both local and global information for depth estimation.</p><p>4. <strong>Up-Sampling and Refinement</strong></p><p>The final depth map is generated using up-sampling. The general techniques used for the up-sampling are bi-linear interpolation or transposed convolutions to increase the spatial resolution of the feature maps. Feature fusion is employed to combine the depth maps with corresponding skip connections in order to refine the depth estimation.</p><h3>Python Code for distance measurement</h3><pre>import cv2<br>import torch<br>import mediapipe as mp<br>import numpy as np<br>from scipy.interpolate import RectBivariateSpline</pre><pre>mp_pose = mp.solutions.pose<br>pose = mp_pose.Pose(static_image_mode=False)</pre><p>Importing required packages and initializing media-pipe pose estimation class <strong><em>mp_pose.Pose</em></strong>.</p><pre>#Downloading the model from TorchHub.<br>midas = torch.hub.load(&#39;intel-isl/MiDaS&#39;,&#39;MiDaS_small&#39;)<br>midas.to(&#39;cpu&#39;)<br>midas.eval()</pre><p>Downloading MiDaS_small model from torch hub. You can download the python executable for MiDaS through <a href="https://github.com/isl-org/MiDaS">Github</a> once, instead. There are three variants of MiDaS available on torch hub and can be downloaded by substituting <strong>‘<em>DPT_Large</em>’</strong> or <strong>‘<em>DPT_Hybrid</em>’ </strong>for<strong> ‘<em>MiDaS_small</em>’</strong>. The general performance of all three variants is given below :</p><ol><li><strong>Small variant </strong>: Lowest accuracy and high inference rate.</li><li><strong>Hybrid variant</strong> : Medium accuracy and medium inference speed</li><li><strong>Large variant </strong>: Highest accuracy with lowest inference speed.</li></ol><p>If you’ve a <strong>Cuda</strong> compatible GPU, then you can replace the <strong><em>midas.to(‘cpu’)</em></strong> with <strong><em>midas.to(“cuda”)</em></strong> to maximize the inference speed.</p><pre>#Performing preprocessing on input for small model <br>transforms = torch.hub.load(&#39;intel-isl/MiDaS&#39;,&#39;transforms&#39;)<br>transform = transforms.small_transform</pre><pre>#Converting Depth to distance<br>def depth_to_distance(depth_value,depth_scale):<br>  return -1.0/(depth_value*depth_scale)</pre><p>Applying the requisite preprocessing on the input image/ video frame for MiDaS small model. Followed by a function <strong>‘depth_to_distance’</strong> defined for converting the calculated depth values into the respective distance values.</p><pre>cap = cv2.VideoCapture(&#39;&#39;)<br>while cap.isOpened():<br>  ret, frame = cap.read()<br><br>  img = cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)<br><br>  cv2.imshow(&#39;Walking&#39;,img)<br><br>  if cv2.waitKey(2) &amp;0xFF == ord(&#39;q&#39;):<br>    cap.release()<br>    cv2.destroyAllWindows()</pre><p>Reading the video input and applying color space transformation using <strong>cv2.cvtColor</strong> function. As cv2 reads images in BGR format we need to convert it to RGB for standard visuals. Let run the code to check if its working right so far.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/1*tc3crTchEIlC3uUj7crQJA.gif" /><figcaption>Fig — 04 (A person walking in the woods) Gif by Author Inspired by <a href="https://pixabay.com/users/matthias_groeneveld-4535957/">Matthias Groeneveld</a> on <a href="https://pixabay.com/videos/hiking-walking-man-alone-nature-109277/">Pixabay</a></figcaption></figure><p>Next we will extract the landmarks from the video frame using Media-Pipe using the below code.</p><pre># Detect the body landmarks in the frame<br>    results = pose.process(img)<br><br>    # Check if landmarks are detected<br>    if results.pose_landmarks is not None:<br>        # Draw Landmarks<br>        mp_drawing = mp.solutions.drawing_utils<br>        mp_drawing.draw_landmarks(img, results.pose_landmarks, mp_pose.POSE_CONNECTIONS)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/1*BAvpgsjucGY1nanjac5BzQ.gif" /><figcaption>Fig — 05 (Keypoints Detected)</figcaption></figure><p>After successfully drawing the detected landmarks, we’ll now extract the waist landmarks as a reference for the distance estimation. Lets write the code to extract the waist landmarks.</p><pre># Extract Landmark Coordinates<br>        landmarks = []<br>        for landmark in results.pose_landmarks.landmark:<br>            landmarks.append((landmark.x, landmark.y, landmark.z))<br><br>        # Extract left and right waist Landmarks<br>        waist_landmarks = [results.pose_landmarks.landmark[mp_pose.PoseLandmark.LEFT_HIP],<br>                           results.pose_landmarks.landmark[mp_pose.PoseLandmark.RIGHT_HIP]]<br>        #Finding midpoint from waist    <br>        mid_point = ((waist_landmarks[0].x + waist_landmarks[1].x) / 2, (waist_landmarks[0].y + waist_landmarks[1].y) / 2)<br>        mid_x , mid_y = mid_point</pre><p>Extracting the x and y coordinate values for both landmarks and calculating the midpoint. You can pick any landmark of your choice depending on the use-case from the pose-landmark list on <a href="https://developers.google.com/mediapipe/solutions/vision/pose_landmarker">Media-pipe</a>.</p><p>Moving forward, we’ll pass our video through MiDaS depth estimation model to get the depth map.</p><pre>imgbatch = transform(img).to(&#39;cpu&#39;)<br><br>    # Making a prediction<br>    with torch.no_grad():<br>        prediction = midas(imgbatch)<br>        prediction = torch.nn.functional.interpolate(<br>            prediction.unsqueeze(1),<br>            size=img.shape[:2],<br>            mode=&#39;bicubic&#39;,<br>            align_corners=False<br>        ).squeeze()<br><br>    output = prediction.cpu().numpy()<br>    #Normalizing the output predictions for cv2 to read.<br>    output_norm = cv2.normalize(output, None, 0, 1, norm_type=cv2.NORM_MINMAX, dtype=cv2.CV_32F)<br><br>    cv2.imshow(&#39;Walking&#39;,output_norm)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/1*hWbpd1NzmCVJ1HHG4lJc4A.gif" /><figcaption>Fig — 06 (MiDaS Depth Map)</figcaption></figure><p>Above is the depth map extracted through MiDaS, also change the <strong><em>waitKey </em></strong>value to<strong><em> </em></strong>1 to decrease the frame delay. You can also change the output of the depth map to a colored map using the code below, but we’ll be using the standard black and white map in this project.</p><pre>#Colored Depth map<br>output_norm = (output_norm*255).astype(np.uint8)<br>output_norm = cv2.applyColorMap(output_norm, cv2.COLORMAP_MAGMA)</pre><p>We’ll then use the from waist landmarks extracted earlier, to calculate the depth value using MiDaS.</p><pre>#Creating a spline array of non-integer grid<br>        h , w = output_norm.shape<br>        x_grid = np.arange(w)<br>        y_grid = np.arange(h)<br><br>        # Create a spline object using the output_norm array<br>        spline = RectBivariateSpline(y_grid, x_grid, output_norm)</pre><p>The use of a spline array in above code snippet enables the creation of a smooth and continuous representation of the <strong><em>output_norm</em></strong> array on a non-integer grid. The need for a spline array arises from the normalization of the output predictions, resulting in an array with floating-point values. By utilizing a spline object, more accurate and versatile computations or visualizations can be performed by interpolating values based on the given data.</p><pre>#Passing the x and y co-ordinates distance function to calculate distance.<br>#Tweak with the depth scale to see what suits you!<br>depth_scale = 1<br>depth_mid_filt = spline(mid_y,mid_x)<br>depth_midas = depth_to_distance(depth_mid_filt, depth_scale)<br><br>#Displaying the distance.<br>cv2.putText(img, &quot;Depth in unit: &quot; + str(<br>            np.format_float_positional(depth_mid_filt , precision=3)), (20, 50), cv2.FONT_HERSHEY_SIMPLEX,<br>                    1, (255, 255, 255), 3)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*V689v_yP8e6aNJ4HK7DOAA.gif" /><figcaption>Fig — 07 (Depth Calculation)</figcaption></figure><p>The depth values are fluctuating a bit. To stabilize the values we’ll use a exponential mean average filter on the depth values and then see the improvements.</p><pre>#Adjust alpha values to suit your need<br>alpha = 0.2<br>previous_depth = 0.0<br><br>#Applying exponential moving average filter<br>def apply_ema_filter(current_depth):<br>    global previous_depth<br>    filtered_depth = alpha * current_depth + (1 - alpha) * previous_depth<br>    previous_depth = filtered_depth  # Update the previous depth value<br>    return filtered_depth</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*u82MQbed3TBJJeSCVxl-gg.gif" /><figcaption>Fig — 08 (Applying EMA Filter)</figcaption></figure><p>Here we can witness considerable difference in the fluctuation after applying Exponential Mean Average Filter on distance value.</p><p>Lets just take a look at the entire code snippet.</p><pre>import cv2<br>import torch<br>import matplotlib.pyplot as plt<br>import mediapipe as mp<br>import numpy as np<br>import shutil<br>from scipy.interpolate import RectBivariateSpline<br><br>#To Clear the model cache<br># shutil.rmtree(torch.hub.get_dir(), ignore_errors=True)<br><br>#Initializing the body landmarks detection module<br>mp_pose = mp.solutions.pose<br>pose = mp_pose.Pose(static_image_mode=False)<br><br># #download the model<br>midas = torch.hub.load(&#39;intel-isl/MiDaS&#39;,&#39;MiDaS_small&#39;)<br>midas.to(&#39;cpu&#39;)<br>midas.eval()<br><br>#Process image<br>transforms = torch.hub.load(&#39;intel-isl/MiDaS&#39;,&#39;transforms&#39;)<br>transform = transforms.small_transform<br><br>alpha = 0.2<br>previous_depth = 0.0<br>depth_scale = 1.0<br><br>#Applying exponential moving average filter<br>def apply_ema_filter(current_depth):<br>    global previous_depth<br>    filtered_depth = alpha * current_depth + (1 - alpha) * previous_depth<br>    previous_depth = filtered_depth  # Update the previous depth value<br>    return filtered_depth<br><br><br>#Define depth to distance<br>def depth_to_distance(depth_value,depth_scale):<br>    return 1.0 / (depth_value*depth_scale)<br><br>def depth_to_distance1(depth_value,depth_scale):<br>    return -1.0 / (depth_value*depth_scale)<br><br>cap = cv2.VideoCapture(&#39;distance1.mp4&#39;)<br>while cap.isOpened():<br>    ret, frame = cap.read()<br><br>    img = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)<br><br>    # Detect the body landmarks in the frame<br>    results = pose.process(img)<br><br>    # Check if landmarks are detected<br>    if results.pose_landmarks is not None:<br>        # Draw Landmarks<br>        # mp_drawing = mp.solutions.drawing_utils<br>        # mp_drawing.draw_landmarks(img, results.pose_landmarks, mp_pose.POSE_CONNECTIONS)<br><br>        # Extract Landmark Coordinates<br>        landmarks = []<br>        for landmark in results.pose_landmarks.landmark:<br>            landmarks.append((landmark.x, landmark.y, landmark.z))<br><br>        waist_landmarks = [results.pose_landmarks.landmark[mp_pose.PoseLandmark.LEFT_HIP],<br>                           results.pose_landmarks.landmark[mp_pose.PoseLandmark.RIGHT_HIP]]<br><br>        mid_point = ((waist_landmarks[0].x + waist_landmarks[1].x) / 2, (waist_landmarks[0].y + waist_landmarks[1].y) / 2,(waist_landmarks[0].z + waist_landmarks[1].z) /2)<br>        mid_x,mid_y = mid_point<br><br>        <br>        imgbatch = transform(img).to(&#39;cpu&#39;)<br><br>        # Making a prediction<br>        with torch.no_grad():<br>            prediction = midas(imgbatch)<br>            prediction = torch.nn.functional.interpolate(<br>                prediction.unsqueeze(1),<br>                size=img.shape[:2],<br>                mode=&#39;bicubic&#39;,<br>                align_corners=False<br>            ).squeeze()<br><br>        output = prediction.cpu().numpy()<br>        output_norm = cv2.normalize(output, None, 0, 1, norm_type=cv2.NORM_MINMAX, dtype=cv2.CV_32F)<br><br>        # Creating a spline array of non-integer grid<br>        h, w = output_norm.shape<br>        x_grid = np.arange(w)<br>        y_grid = np.arange(h)<br><br>        # Create a spline object using the output_norm array<br>        spline = RectBivariateSpline(y_grid, x_grid, output_norm)<br>        depth_mid_filt = spline(mid_y,mid_x)<br>        depth_midas = depth_to_distance(depth_mid_filt, depth_scale)<br>        depth_mid_filt = (apply_ema_filter(depth_midas)/10)[0][0]<br>        <br>        cv2.putText(img, &quot;Depth in unit: &quot; + str(<br>            np.format_float_positional(depth_mid_filt , precision=3)) , (20, 50), cv2.FONT_HERSHEY_SIMPLEX,<br>                    1, (255, 255, 255), 3)<br>        <br>        cv2.imshow(&#39;Walking&#39;,img)<br><br>    if cv2.waitKey(1) &amp;0xFF == ord(&#39;q&#39;):<br>        cap.release()<br>        cv2.destroyAllWindows()</pre><h3>Conclusion</h3><p>In this article we calculated the distance of an object from the camera using the MiDaS depth estimation model while leveraging the reference landmark extracted from Media-pipe. The following approach can be used to detect distance of multiple objects/people and integrated into mini projects based on proximity.</p><p>You can download the source code using <a href="https://github.com/nbeeeel/Distance-Measurement-Using-MiDaS"><strong>Github </strong></a>link<strong>.</strong></p><h3>References</h3><p>[1] Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler and Vladlen Koltun, <a href="https://arxiv.org/abs/1907.01341">Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer</a> (2020), IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI).</p><p>[2] Ranftl, Alexey Bochkovskiy and Vladlen Koltun, <a href="https://arxiv.org/abs/2103.13413">Vision Transformers for Dense Prediction</a> (2021), ArXiv preprint.</p><p>[3] Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Guang Yong, Juhyun Lee, Wan-Teh Chang, Wei Hua, Manfred Georg and Matthias Grundmann, <a href="https://arxiv.org/abs/1906.08172">MediaPipe: A Framework for Building Perception Pipelines</a> (2019), ArXiv preprint.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d0119bfe1159" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Swift and Simple: Calculate Object Distance with ease in just few Lines of Code]]></title>
            <link>https://medium.com/@nbeel.original/swift-and-simple-calculate-object-distance-with-ease-in-just-few-lines-of-code-38889575bb12?source=rss-54000852e720------2</link>
            <guid isPermaLink="false">https://medium.com/p/38889575bb12</guid>
            <category><![CDATA[object-detection]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[cameras]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Nabeel Khan]]></dc:creator>
            <pubDate>Fri, 26 May 2023 20:22:04 GMT</pubDate>
            <atom:updated>2023-05-31T14:34:06.396Z</atom:updated>
            <content:encoded><![CDATA[<p>So the past few days, I’ve rummaged through various internet sources looking for ways to calculate object distance using monocular vision. Along the way, I found deep learning based monocular depth estimation models and some landmarks based distance approximation methodologies. These were the only options considering my obstinacy towards a low-resource alternative to this problem.</p><p>Generally, depth or distance is calculated using stereo vision approaches. Stereo vision is a powerful technique inspired by human vision, it uses binocular disparity concept to precisely approximate the distance of object from camera. However, it requires stereoscopic cameras capable of depth estimation by capturing two slightly offset images of a scene simultaneously. I was lacking this luxury, so I focused my efforts towards a landmarks based distance approximation approach.</p><p>So that was a bit of a background, and without wasting one more minute we’ll go through the code:</p><pre>import mediapipe as mp<br>import cv2<br>import numpy as np<br>import math<br><br>mp_pose = mp.solutions.pose<br>pose = mp_pose.Pose()</pre><p>Importing the dependencies, instantiating the media-pipe class instance <strong><em>mp_pose.Pose() </em></strong>and assigning it to a variable <strong><em>pose</em></strong>.</p><pre>cap = cv2.VideoCapture(&#39;distance.mp4&#39;)<br>while cap.isOpened():<br>  ret,img = cap.read()<br><br>#Converting to RGB<br>  img = cv2.cvtColor(img,cv2.COLOR_BGR2RGB)<br>  results = pose.process(img)<br><br>#check to see if bodylandmarks are being detected   <br>  if results.pose_landmarks is not None:<br>    mp_drawing = mp.solutions.drawing_utils<br>    mp_drawing.draw_landmarks(img,results.pose_landmarks, mp_pose.POSE_CONNECTIONS)<br><br>    cv2.imshow(&#39;ImgWindow&#39;,img)<br><br>  if cv2.waitKey(1) &amp; 0xFF == ord(&#39;q&#39;):<br>    cap.release()<br>    cv2.destroyAllWindows() </pre><p>Loading the Video Capture window from OpenCV, and converting the image to RGB as cv2 loads the images in BGR format. Next we draw the extracted landmarks on image for a sanity check. Here’s the result so far :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/1*F0rnCed3mVKfsyYwvw72uQ.gif" /><figcaption>Fig : 02 — Landmark Detection</figcaption></figure><p>Now the landmarks are being detected, the next step is to extract the landmark co-ordinates that we’ll use as a reference point for detecting distance. We’ll be taking the nose landmark as reference, although the body is only visible from behind — the model is still able to extrapolate the nose key-point.</p><pre>#Extracting the nose landmark. <br>landmarks = []<br>for landmark in results.pose_landmarks.landmark:<br>    landmarks.append((landmark.x, landmark.y, landmark.z))<br>        <br>nose_landmark = landmarks[mp_pose.PoseLandmark.NOSE.value]<br>_,_, nose_z = nose_landmark</pre><p>We extracted the landmarks and appended all detected landmarks within the <strong><em>landmarks </em></strong>list<em>. </em>From there we extract the <strong><em>nose_landmark </em></strong>which encapsulates three values for x, y and z axis. The x-axis and y-axis values provide the position of the point along the x and y plane while the z axis provides the relative depth/ distance information from all the other keypoints. By leveraging this depth information we can estimate the distance of the object from the camera. We can take any key-point for reference and the below figure from the official media-pipe website provides all the key-point locations with their respective names and numbers to help you select.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*uxBqmSr-XSH5uYr2.png" /><figcaption>Fig : 03 — Landmark List (<a href="https://developers.google.com/static/mediapipe/images/solutions/pose_landmark_index.png">credits</a>)</figcaption></figure><pre>#Calculating distance from the z-axis<br>#Set the depth_scale to 1<br>depth_scale = 1<br>def depth_to_distance(depth_value,depth_scale):<br>    return -1.0 / (depth_value*depth_scale)</pre><pre>distance = depth_to_distance(nose_z,depth_scale)<br>cv2.putText(img, &quot;Depth in unit: &quot; + str(np.format_float_positional(distance, precision=2)),(20,50),cv2.FONT_HERSHEY_SIMPLEX,1,(255,255,255),3)<br>cv2.imshow(&#39;Video&#39;,img)</pre><p>The function <strong><em>depth_to_distance </em></strong>converts the <strong><em>z-axis</em></strong> value into distance value. The parameter <strong><em>depth_value</em></strong> is the value obtained from the nose landmark, the <strong><em>depth_scale</em></strong> is used to adjust the depth values to desired unit of measurement. The depth scale is usually provided by the algorithm. In this scenario, depth scale is in meters as mentioned on the model’s documentation site. Furthermore, the <strong><em>constant</em></strong> <strong><em>-1.0</em></strong> is used to invert the depth value, as depth is usually in the negative.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/1*ZvoviXCCkSb-FaE4VI-RGg.gif" /><figcaption>Fig : 04 — Distance Measurement</figcaption></figure><p>The above depth values are fluctuating at a rapid rate. Moreover, there are random spikes within the values. The random spikes are due to the camera not being stationery, causing occasional glitches in key-point detection — resulting in negative or very large values. To stabilize the fluctuations in distance we can use an exponential moving average filter. The filter will approximate the values to a more stable state reducing the fluctuations. Explanation of the filtering technique is beyond the scope of this article. So we’ll keep it for some other day and for some other application. Here’s the code to apply this filtering technique and get better results.</p><pre>#Tweak the alpha value to suit your needs<br>alpha = 0.6<br>previous_depth = 0.0<br><br>def apply_ema_filter(current_depth):<br>    global previous_depth<br>    filtered_depth = alpha * current_depth + (1 - alpha) * previous_depth<br>    previous_depth = filtered_depth  # Update the previous depth value<br>    return filtered_depth</pre><pre>filter = apply_ema_filter(nose_z)<br>distance = depth_to_distance(filter,depth_scale)</pre><p>Here’s the result of the video, after applying the filter. The value is a bit stable then before however there are still some random negative and positive peaks due to the mobility of the camera as discussed before. A stationery camera will give much better results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/1*KpHeIyvXUBMyDUKzg1LrLA.gif" /></figure><p>Here’s the entire code :</p><pre>import mediapipe as mp<br>import cv2<br>import numpy as np<br><br>mp_pose = mp.solutions.pose<br>pose = mp_pose.Pose(static_image_mode=False)<br><br>#Tweak this parameter to suit your own need<br>alpha = 0.6<br>previous_depth = 0.0<br><br>def apply_ema_filter(current_depth):<br>    global previous_depth<br>    filtered_depth = alpha * current_depth + (1 - alpha) * previous_depth<br>    previous_depth = filtered_depth  # Update the previous depth value<br>    return filtered_depth<br><br>#Play with this constant value and vary it to check what suits your needs.<br>def depth_to_distance(depth_value, depth_scale):<br>  return -1.0 / (depth_value * depth_scale)<br><br>#Change it to your own camera feed<br>cap = cv2.VideoCapture(&#39;distance.mp4&#39;)<br>while cap.isOpened():<br>  ret,frame = cap.read()<br><br>#Grayscaling the image<br>  img = cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)<br>  results = pose.process(img)<br><br>#check to see if bodylandmarks are being detected<br>  if results.pose_landmarks is not None:<br>    landmarks = []<br>    for landmark in results.pose_landmarks.landmark:<br>      landmarks.append((landmark.x, landmark.y, landmark.z))<br><br>    nose_landmark = landmarks[mp_pose.PoseLandmark.NOSE.value]<br>    _, _, nose_z = nose_landmark<br><br>    img = cv2.cvtColor(img,cv2.COLOR_BGR2RGB)<br>    filter = apply_ema_filter(nose_z)<br>    distance = depth_to_distance(filter,1)<br>#Convert the distance to your own requirement.<br>    cv2.putText(img, &quot;Depth in unit: &quot; + str(np.format_float_positional(distance, precision=2)),(20,50),cv2.FONT_HERSHEY_SIMPLEX,1,(255,255,255),3)<br>    cv2.imshow(&#39;ImgWindow&#39;,img)<br><br>  if cv2.waitKey(1) &amp; 0xFF == ord(&#39;q&#39;):<br>   cap.release()<br>   cv2.destroyAllWindows()</pre><p>Conclusion:</p><p>Although this monocular method of finding distance is an effective way to play around with the concept. However, the results from the dedicated stereo vision equipment are recommended for applications needing high accuracy and precision. I hope, this satisfies your monocular vision based distance estimation needs. Give me a 👏if you like this. I’ll try to bring more refined versions of this project in near future. <br>Till then …. Take care and stay blessed !</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=38889575bb12" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Harmonizing Chaos: Exploring the Fusion of Imagination and Generative AI to Cultivate Creative…]]></title>
            <link>https://medium.com/@nbeel.original/harmonizing-chaos-exploring-the-fusion-of-imagination-and-generative-ai-to-cultivate-creative-2c6041665c0e?source=rss-54000852e720------2</link>
            <guid isPermaLink="false">https://medium.com/p/2c6041665c0e</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[imagination]]></category>
            <category><![CDATA[creativity]]></category>
            <category><![CDATA[generative-ai]]></category>
            <category><![CDATA[philosophy-of-science]]></category>
            <dc:creator><![CDATA[Nabeel Khan]]></dc:creator>
            <pubDate>Sat, 20 May 2023 13:55:59 GMT</pubDate>
            <atom:updated>2023-05-20T13:55:59.608Z</atom:updated>
            <content:encoded><![CDATA[<h3>Harmonizing Chaos: Exploring the Fusion of Imagination and Generative AI to Cultivate Creative Order.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ThjNVeCYpDw202suf4Gndw.png" /><figcaption>Fig : 1 — Sand Animation Art</figcaption></figure><p>In the ever-evolving landscape of technology, imagination stands as a driving force behind groundbreaking innovation. It ignites our creative process, pushing boundaries and transcending limitations. Today, we embark on a journey to explore the captivating fusion of imagination and generative AI — a realm where chaos finds order, and creativity reaches new heights. Let us delve into this fascinating subject, combining the perspectives of both the observer and the explorer.</p><h3>Chaos and Order: Unleashing Imagination’s Potential</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/564/0*GwqQwu_CbWxMpEVG.jpg" /><figcaption>Fig : 2 — Chaos and Order (<a href="https://i.pinimg.com/564x/57/48/4f/57484f39c71e18ebddead0f2434e1e8e.jpg">credits</a>)</figcaption></figure><p>From the extreme disarray of particles to a captivating illustration, sand animation art exemplifies “Order within chaos.” It amazes me how, despite lacking meaning, structure, and a semblance of synergy, it still converges into a harmonious composition. This phenomenon emphasizes that the journey towards equilibrium, rather than the achieved state, creates a mesmerizing trail of wonders.</p><p>As Carl Jung eloquently puts it,</p><blockquote>“In all Chaos there is a cosmos, In all disorder a secret order.”</blockquote><p>Chaos plays a pivotal role in fostering innovation by disrupting our conventional thinking patterns. As creatures of habit, our thoughts and perceptions often conform to established norms and beliefs. However, to unlock the full power of our imagination, we must liberate ourselves from the confines of complacency and explore uncharted territories. It takes courage and effort to challenge these norms and venture into the untapped realms of possibility lurking within our subconscious. By doing so, we open the doors to innovative breakthroughs and embrace the audacity to ask bold questions like <strong><em>“What if?”</em></strong> and <strong>“Why not?”</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*GJc2gUfz47P0DzhQ.jpg" /><figcaption>Fig : 3 — Balance (<a href="https://pixabay.com/illustrations/yin-yang-karma-meditation-7483695/">credits</a>)</figcaption></figure><p>On the other side, when our innate imagination flows without restraint, the line between reality and fantasy becomes blurred. It is both exhilarating and overwhelming. Ideas and possibilities flood our minds, but if left unorganized, they can result in a chaotic barrage that leads nowhere but down a rabbit hole. Finding a balance between chaos and order becomes crucial in fostering innovation. By harnessing the creative energy of chaos while implementing structured frameworks and organization, we can channel our imaginative potential into meaningful and impactful outcomes. This delicate equilibrium allows us to navigate the realm of innovation, transforming chaotic ideas into purposeful and transformative innovations.</p><blockquote>“There is a need for chaos and order. Too much order and you fossilise. Too much chaos and you fall apart. These things are always in fruitful harmony.”</blockquote><blockquote>Dr. Lain McGilchrist</blockquote><h3>Imagination and Generative AI: Exploring Possibilities</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*e0QWtQbuBapHQf7z.jpg" /><figcaption>Fig : 4 — Imagination (credits)</figcaption></figure><p>As humans, we possess an incredible gift — the power of imagination. It empowers us to explore infinite possibilities and fuels our desire to create something in our own image. This very capacity led to the development of Artificial Intelligence, which attempts to mimic the intricate decision-making processes of our cognitive systems, free from emotional affiliations. However, entrusting decision-making solely to factual algorithms raises concerns. It makes me ponder the implications of a world without humans. <strong>Bard </strong>answered a question to a world without humans in an Idealistic Manner.</p><blockquote>A world without humans would be a world of peace and tranquility. The air would be clean and the water would be clear. The forests would be lush and green, and the animals would roam free. The Earth would be able to heal itself and return to its natural state.</blockquote><p>The response given by Bard sparked contemplation within me, raising questions about our role in the grand scheme of things.</p><blockquote><strong>Do we represent chaos or order?</strong></blockquote><p>If we align with chaos, the future prospects for the flourishing of humanity appear bleak. However, it is crucial to recognize that everything eventually returns to its natural order. This prompts us to reflect on the part we play in shaping the world.</p><blockquote><strong>Are we catalysts of disorder or agents of harmony?</strong></blockquote><p>By embracing our capacity for creativity and conscious action, we have the power to influence the direction of our collective existence and contribute to a future that thrives.</p><p>To truly understand and nurture creative endeavors, we must delve into the sources of inspiration. Many innovations replicate or emulate natural phenomena. As we observe ourselves and the world around us with an inquisitive mind, we unlock the padlocks of mediocrity and dormant innovation. Consider Artificial Neural Networks, which are modeled after the human nervous system. They exemplify the concept of learning from the world around us. It is our responsibility to pay heed to the mysteries within and surrounding us to find answers and motivation.</p><h3>Guided Augmentation of GAI’s Capabilities: Human Intervention in the Imagination Process</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/1*-gvaZwmId62sf7XEttqw8g.gif" /><figcaption>Fig : 5— Generative AI (<a href="http://makeagif.com/">credits</a>)</figcaption></figure><p>Generative Artificial Intelligence (AI) has made significant strides in a short period, yet it has inherent limitations. As an approximation of the human imagination process, its outputs tend to rely heavily on learned experiences, often resembling what it has been trained on. This is where human intervention becomes essential.</p><p>We, as humans, have a vital role in enhancing the imagination process of generative models. We provide creative input, curate imaginative datasets, guide the training process, engage in co-creation and collaboration, and interpret the outputs. Through our creative ideas, we influence the model’s output, fostering the emergence of novel and imaginative creations. By curating diverse and imaginative datasets, we shape the learning process, encouraging the models to generate imaginative outputs. Through active guidance and feedback during training, we refine the models’ imaginative capabilities.</p><p>Collaboration and co-creation between humans and generative models result in the production of novel and imaginative outputs. Furthermore, by interpreting and interacting with the generated content, we discover meaning, inspiration, and gain new perspectives. This interaction ignites our own imagination, generating further creative ideas. Thus, we actively contribute to and facilitate the imagination process of generative models, complementing their capabilities and expanding the boundaries of creative potential.</p><h3>Conclusion</h3><p>Generative AI’s capacity to surpass human expertise in innovation and creativity fuels our curiosity. We question the sources of inspiration behind this profound development — whether it stems from the stroke of a brush on a spilled canvas, opening a rift into a world of vibrant dreams, or the transformation of scattered sand into an expression of utmost beauty by a mere humane touch. By embracing our imagination and exploring these inspirations, we unlock limitless possibilities for future research and innovation, shaping a world of generative wonders. Together, we bridge the realms of imagination and technology, harmonizing chaos, and cultivating creative order.</p><h3>References</h3><p><a href="https://www.linkedin.com/pulse/ai-order-chaos-why-time-hype-might-justified-zach-reeder/">https://www.linkedin.com/pulse/ai-order-chaos-why-time-hype-might-justified-zach-reeder/</a><br><a href="https://www.informationweek.com/big-data/before-chaos-there-was-ai">https://www.informationweek.com/big-data/before-chaos-there-was-ai</a></p><p><a href="https://cosmosmagazine.com/technology/artificial-intelligence-at-the-edge-of-chaos/">https://cosmosmagazine.com/technology/artificial-intelligence-at-the-edge-of-chaos/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2c6041665c0e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Navigating Math for Computer Vision: Your Ultimate Roadmap]]></title>
            <link>https://medium.com/@nbeel.original/navigating-math-for-computer-vision-your-ultimate-roadmap-8389a0d7b7be?source=rss-54000852e720------2</link>
            <guid isPermaLink="false">https://medium.com/p/8389a0d7b7be</guid>
            <category><![CDATA[linear-algebra]]></category>
            <category><![CDATA[mathematics]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Nabeel Khan]]></dc:creator>
            <pubDate>Mon, 15 May 2023 10:55:40 GMT</pubDate>
            <atom:updated>2023-05-31T14:34:35.124Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4wD92Z8uTSzKQ8S9fHmPBQ.png" /></figure><p>I got myself occupied with developing an understanding of Convolutional Neural Networks, as part of my final year project themed around object detection. However, absorbing the mathematical principles defining the fundamentals of the subject was not an easy feat. I had to joust with the underlying concepts and for the first time see numbers as a representation of images, which to be honest was not easy for me to get my head around. Anyways, I did eventually made sense of the maths behind the representation and manipulation of images albeit after some sleepless nights.</p><p>Coming to the point, learning algebra, calculus or other numerical concepts to develop an understanding of computer vision or image processing would in itself be a debilitating ordeal, unless a clarity in relation between images and their numeric representation is established. Many of us are familiar with the vital role mathematics plays within the computer vision domain while those of you who aren’t, would be vaguely familiarized through this post along with a roadmap to conquer your fear of sophisticated Greek and Latin representations that contribute a sizeable proportion of modern day mathematics. So its been established, conquering mathematical methods is an imperative objective in our quest to becoming a CV expert and there is no other way. Hence we should embrace it as a powerful language to explain densely intricate phenomenon, an archaic tool to decrypt the mysteries of visual data and a pathway to innovation. One of the many effective ways to demystify the language of the Gods for CV whilst being creative and robust in our comprehension of it is to leverage python, as it will aid greatly in closely observing the image response to subsequent mathematical operations while also providing high dimensional visualizations via its arsenal of libraries.</p><p>After introduction to my hard-earned epiphany, let me walk you through a generic overview of mathematical concepts that play a critical role in image processing and computer vision.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/259/1*FjRRyfGC58ua7iBgnU_3Pg.jpeg" /><figcaption><a href="https://ai.stanford.edu/~syyeung/cvweb/tutorial1.html">credits</a></figcaption></figure><h3>Goal of the article</h3><ol><li>Familiarize with the role of various mathematical methods in Computer Vision.</li><li>Provide a complete mathematical roadmap for Computer Vision research and development.</li></ol><h3><strong>Linear Algebra</strong></h3><p>Linear Algebra comes into play as we represent images in the form of vectors, matrices and tensors.</p><h3><strong>Calculus</strong></h3><p>Calculus helps derive and optimize mathematical models for image processing and computer vision tasks.</p><h3>Probability and Statistics</h3><p>Probability and Statistics help us model and analyse image data, including feature extraction, image segmentation and object detection.</p><h3>Signal Processing</h3><p>To filter and transform images for artifacts and noise removal as well as to extract meaningful information in time-frequency domain, we employ signal processing techniques such as Fourier Analysis and wavelet transforms.</p><h3>Differential Equations</h3><p>Differential Equations are leveraged to model Dynamic Systems such as optical flow, motion estimation and image registration.</p><h3>Geometry</h3><p>Geometry is important for spatial transformations and 3D reconstruction of objects in computer vision.</p><h3>Optimization</h3><p>Optimization is used to develop algorithms and models for image denoising, deblurring, and super-resolution.</p><h3>Mathematics Roadmap For Image Processing and Computer Vision</h3><p>Below is the roadmap of mathematical methods for computer vision that will contribute sufficiently in your computer vision research and development journey.</p><h3><strong>Linear Algebra</strong></h3><ul><li>Vector spaces and subspaces: Understanding properties and operations.</li><li>Matrix factorizations: Singular value decomposition (SVD) and eigenvalue decomposition.</li><li>Linear regression: Modeling relationships between variables for regression tasks.</li><li>Principal Component Analysis (PCA): Dimensionality reduction technique.</li></ul><h3>Calculus</h3><ul><li>Multivariable calculus: Partial derivatives, gradients, and optimization in multiple dimensions.</li><li>Chain rule: Calculating derivatives in composite functions.</li><li>Hessian matrix: Analyzing curvature and optimization in higher dimensions.</li><li>Variational calculus: Euler-Lagrange equations for energy minimization problems.</li></ul><h3>Probability and Statistics</h3><ul><li>Random processes: Modeling temporal and spatial uncertainty in computer vision.</li><li>Markov chains: Analyzing sequential and temporal data.</li><li>Statistical pattern recognition: Statistical techniques for object recognition and classification.</li><li>Bayesian decision theory: Decision-making under uncertainty.</li></ul><h3>Signal Processing</h3><p><strong>Sampling Theory</strong></p><ul><li>Nyquist-Shannon sampling theorem: Principles of converting continuous signals to discrete signals.</li><li>Aliasing: Understanding the effects of under-sampling and frequency folding.</li></ul><p><strong>Image Filtering</strong></p><ul><li>Linear filters: Convolution, correlation, and their applications for noise reduction and image enhancement.</li><li>Non-linear filters: Median filtering, bilateral filtering, and their use in preserving edges and reducing noise.</li></ul><p><strong>Frequency Domain Analysis</strong></p><ul><li>Discrete Fourier Transform (DFT): Transforming signals from time domain to frequency domain.</li><li>Fast Fourier Transform (FFT): Efficient algorithms for computing the DFT.</li><li>Power spectra: Analyzing signal content and identifying dominant frequencies.</li></ul><p><strong>Wavelet Theory</strong></p><ul><li>Continuous Wavelet Transform (CWT): Analysis of signals at different scales and resolutions.</li><li>Discrete Wavelet Transform (DWT): Decomposing signals into wavelet coefficients for efficient representation.</li><li>Wavelet packet analysis: Further analysis and decomposition of signals using wavelet packets.</li></ul><p><strong>Filter Design</strong></p><ul><li>Finite Impulse Response (FIR) filters: Designing filters with finite-duration impulse responses.</li><li>Infinite Impulse Response (IIR) filters: Designing filters with infinite-duration impulse responses.</li><li>Filter banks: Constructing filter banks for multi-resolution analysis and synthesis.</li></ul><p><strong>Image Compression</strong></p><ul><li>Transform coding: Applying transformations like Discrete Cosine Transform (DCT) for efficient data representation.</li><li>Quantization: Reducing precision while preserving essential image information.</li><li>Entropy coding: Techniques like Huffman coding and arithmetic coding for further compression.</li></ul><p><strong>Image Restoration</strong></p><ul><li>Inverse problems: Modeling image deblurring, super-resolution, and image reconstruction.</li><li>Regularization: Balancing fidelity to observed data and prior assumptions in restoration tasks.</li></ul><h3>Differential Equations</h3><ul><li>Partial differential equations (PDEs) in computer vision: Heat equation, wave equation, and diffusion equation for image analysis and restoration.</li><li>Level set methods: Implicit representation of curves and surfaces for segmentation tasks.</li><li>Active contours (Snakes): Contour evolution for object boundary detection.</li></ul><h3>Geometry</h3><ul><li>3D geometry: Representing and transforming 3D objects and scenes.</li><li>Camera geometry: Pinhole camera model, intrinsic and extrinsic camera parameters.</li><li>Structure from motion: Estimating 3D structure from 2D image sequences.</li><li>3D reconstruction: Techniques for building 3D models from multiple images.</li></ul><h3>Optimization</h3><ul><li>Nonlinear optimization: Techniques like Newton’s method and Levenberg — Marquardt algorithm.</li><li>Convex hulls: Convex polytopes and their applications in computer vision.</li><li>Graph cuts: Energy minimization for image segmentation and object recognition.</li><li>Combinatorial optimization: Solving NP-hard problems in computer vision.</li></ul><h3>Conclusion</h3><p>Computer Vision has grown rapidly in the last decade, owing to the vehement endeavors of the research community. To keep pace with the research and publications within the domain, an understanding of governing mathematical principles is imperative. Hence this article, intended to unravel the roadmap required for a career as a researcher or even a developer in Computer Vision domain.</p><p>I hope the aforementioned roadmap would navigate your computer vision journey. If some facets were left unattended, do let me know in the comment section.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8389a0d7b7be" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>