<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Praveen Kumar on Medium]]></title>
        <description><![CDATA[Stories by Praveen Kumar on Medium]]></description>
        <link>https://medium.com/@inboxpraveen?source=rss-f1a8d777cc44------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*0jXqPKfxcTX0TWK2OcsFLA.png</url>
            <title>Stories by Praveen Kumar on Medium</title>
            <link>https://medium.com/@inboxpraveen?source=rss-f1a8d777cc44------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 09:25:28 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@inboxpraveen/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[FIFA Player Recommendation System]]></title>
            <link>https://medium.com/@inboxpraveen/fifa-player-recommendation-system-7868ada8e87d?source=rss-f1a8d777cc44------2</link>
            <guid isPermaLink="false">https://medium.com/p/7868ada8e87d</guid>
            <category><![CDATA[football]]></category>
            <category><![CDATA[scikit-learn]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[recommendation-system]]></category>
            <dc:creator><![CDATA[Praveen Kumar]]></dc:creator>
            <pubDate>Sun, 03 May 2026 14:04:54 GMT</pubDate>
            <atom:updated>2026-05-03T14:04:54.266Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://github.com/inboxpraveen/FIFA-Player-Recomendation">Project Code Link</a></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PLuZZDFAzoYOJFSk1Ivq9w.png" /></figure><h3>1. Project Overview</h3><p>This is a web-based player recommendation system built on FC 25 FIFA stats. You search for a player, and the app finds the most similar players from the dataset using their in-game attributes. It supports both male and female players, lets you compare up to four players side by side, and visualizes the comparisons using radar charts.</p><p>The app runs entirely in the browser against a Flask backend. There is no database. All recommendations come from a precomputed similarity model built at training time.</p><h3>2. Why I Built This</h3><p>I built this as a side project to work on something concrete with recommender systems. Most tutorials on recommendation engines use movies or books. I wanted to apply the same ideas to a domain that is more interesting to me, and one where the features are richer and more structured.</p><p>This project also gave me a reason to build a full end-to-end ML system from scratch: data cleaning, feature engineering, model training, a REST API, and a frontend. I published it open source so that other developers learning about content-based filtering or ML systems have a real working example they can read, run, and modify.</p><h3>3. Problem It Solves</h3><p>Finding players with similar play styles in FIFA is not straightforward. The game has hundreds of attributes per player, and comparing them manually does not scale.</p><p>This project reduces that problem by:</p><p>- Computing how similar any two players are across 34 key attributes<br>- Giving you the top N most similar players for any player you search<br>- Letting you compare multiple players visually with radar charts<br>- Supporting filters like position category and age gap so results stay relevant</p><p>It is a practical example of how content-based filtering works when your “items” have structured numeric features.</p><h3>4. Key Features</h3><p>- <strong>Player search</strong> with filters for name, position, overall rating, nationality, league, and team<br>- <strong>Similar player recommendations</strong> using cosine similarity on 34 player attributes<br>- <strong>Gender toggle</strong> with separate trained models for male and female players<br>- <strong>Multi-player comparison</strong> for 2 to 4 players at once with a radar chart<br>- <strong>Player detail view</strong> with a stats card and individual radar chart<br>- <strong>Autocomplete</strong> for player name search<br>- <strong>Top players</strong> listing and dataset statistics</p><h3>5. How It Works</h3><p>1. <strong>Data preparation</strong>: The training script reads a CSV file of FC 25 player stats. It cleans the data, removes duplicates, and assigns each player a position category.</p><p>2. <strong>Feature extraction</strong>: 34 numeric attributes are selected per player, including pace, shooting, passing, dribbling, defending, and specific sub-stats like sprint speed, vision, and composure.</p><p>3. <strong>Normalization</strong>: Each player’s feature vector is min-max normalized so that all attributes contribute equally to the similarity score.</p><p>4. <strong>Similarity matrix</strong>: The full pairwise cosine similarity matrix is computed over all players at training time. This makes query-time lookups very fast. The matrix, the original DataFrame, and feature names are bundled and saved as a `.pkl` file using `joblib`.</p><p>5. <strong>Serving</strong>: At startup, Flask loads the pickled models into memory. All API calls work against this in-memory data. There is no live database.</p><p>6. <strong>Recommendation</strong>: When you request similar players for a name, the app finds the player’s row index, reads their similarity scores against all other players, filters by any constraints you set, and returns the top N results sorted by score.</p><p>7. <strong>Frontend</strong>: A single-page HTML/JS app calls the Flask JSON API and renders results using Chart.js radar charts.</p><h3>6. Tech Stack</h3><p><strong>Backend</strong><br>- Python 3.8+<br>- Flask (web framework and REST API)<br>- scikit-learn (`cosine_similarity`)<br>- pandas and numpy (data processing)<br>- joblib (model serialization)<br>- gunicorn (production server)</p><p><strong>Frontend</strong><br>- HTML, CSS, vanilla JavaScript<br>- Chart.js 4.4.0 (radar charts)<br>- Font Awesome 6.4.0 (icons)<br>- Google Fonts (Inter)</p><p><strong>Data</strong><br>- FC 25 FIFA player CSVs (male and female)</p><h3>7. Core Code Snippets</h3><p>The complete source code is available on <a href="https://github.com/inboxpraveen/FIFA-Player-Recomendation">[GitHub]</a>. The snippets below show only the core logic.</p><p><strong>The 34 attributes used to measure player similarity</strong></p><p>These features form the input vectors for cosine similarity. Every recommendation comes down to how close two players’ normalized versions of these values are.</p><pre>FEATURE_COLUMNS = [<br>    &#39;PAC&#39;, &#39;SHO&#39;, &#39;PAS&#39;, &#39;DRI&#39;, &#39;DEF&#39;, &#39;PHY&#39;,<br>    &#39;Acceleration&#39;, &#39;Sprint Speed&#39;, &#39;Positioning&#39;, &#39;Finishing&#39;,<br>    &#39;Shot Power&#39;, &#39;Long Shots&#39;, &#39;Volleys&#39;, &#39;Penalties&#39;,<br>    &#39;Vision&#39;, &#39;Crossing&#39;, &#39;Free Kick Accuracy&#39;, &#39;Short Passing&#39;,<br>    &#39;Long Passing&#39;, &#39;Curve&#39;, &#39;Dribbling&#39;, &#39;Agility&#39;, &#39;Balance&#39;,<br>    &#39;Reactions&#39;, &#39;Ball Control&#39;, &#39;Composure&#39;, &#39;Interceptions&#39;,<br>    &#39;Heading Accuracy&#39;, &#39;Def Awareness&#39;, &#39;Standing Tackle&#39;,<br>    &#39;Sliding Tackle&#39;, &#39;Jumping&#39;, &#39;Stamina&#39;, &#39;Strength&#39;, &#39;Aggression&#39;<br>]</pre><p><strong>Precomputing the full similarity matrix at training time</strong></p><p>This is a deliberate trade-off: spend time and memory once at training so that every recommendation at query time is just an array lookup.</p><pre># Precompute similarity matrix for fast recommendations<br>print(&quot;Computing similarity matrix...&quot;)<br>self.similarity_matrix = cosine_similarity(normalized_features)<br>print(f&quot;Similarity matrix computed: {self.similarity_matrix.shape}&quot;)</pre><p><strong>Filtering and ranking recommendations</strong></p><p>After reading the similarity row for a player, the system skips the player itself, applies optional position and age filters, and returns the top N results sorted by score.</p><pre>similarity_scores = self.similarity_matrix[player_idx]<br><br>for idx, score in enumerate(similarity_scores):<br>    if idx == player_idx:  # Skip the player itself<br>        continue<br><br>    if same_position and player_position:<br>        if candidate.get(&#39;Position_Category&#39;) != player_position:<br>            continue<br><br>    # ... age gap filter applied here ...<br><br>candidates.sort(key=lambda x: x[1], reverse=True)<br><br>for idx, score in candidates[:n_recommendations]:<br>    # build and return result list</pre><h3>9. Challenges and Learnings</h3><p><strong>Scaling the similarity matrix</strong></p><p>The precomputed similarity matrix is O(n²) in memory. For a large player dataset, this gets heavy fast. I kept it as a deliberate design choice for query speed, but it made me think about when this trade-off makes sense versus using approximate nearest neighbor methods.</p><p><strong>Consistent normalization between training and inference</strong></p><p>When I added a feature to recommend players by custom attribute values (not from the dataset), I had to ensure the input was normalized the same way as the training data. It is easy to get this subtly wrong, and the recommendations would silently degrade without an obvious error.</p><p><strong>Name resolution ambiguity</strong></p><p>Player names in the CSV are not always unique or clean. The lookup function falls back to substring matching, which means it can return a different player than intended. Handling name resolution reliably turned out to be trickier than expected.</p><p><strong>Separating male and female models cleanly</strong></p><p>The two player pools have different data distributions. Keeping them as completely separate trained models was the simplest and most reliable approach, even though it doubles the memory footprint at runtime.</p><h3>10. Screenshots or Demo</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YXvlcBJ5X2S2couQwMzE5Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TYzm_IRE9Y4hom4E-_Iycg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pPUTdo_FH5SzHTZqsHPU9w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mn0NDYcJKHtu2F32fo1TQw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PLuZZDFAzoYOJFSk1Ivq9w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ORFv5bBEePy_y6q-ISEazg.png" /></figure><p>- Repository: <a href="https://github.com/inboxpraveen/FIFA-Player-Recomendation">[GitHub]</a></p><h3>11. Future Improvements</h3><p>- Replace the precomputed O(n²) matrix with an approximate nearest neighbor index (like FAISS or Annoy) to reduce memory and support larger datasets<br>- Add a proper name normalization step so player lookups are more reliable<br>- Support dataset updates without retraining from scratch<br>- Add a Docker setup for easier one-command deployment<br>- Expose a simple public API so other developers can build on top of the recommendation engine</p><h3>12. Closing Note</h3><p>This project is fully open source under the MIT license. It was built as a personal learning project and shared so others can use it, study it, or improve it. If you find it useful, run into a bug, or have an idea to make it better, feel free to open an issue or a pull request on GitHub.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7868ada8e87d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[1. Python’s love — Objects & Abstract Concept — What, why, and how.]]></title>
            <link>https://medium.com/practical-python/1-pythons-love-objects-abstract-concept-what-why-and-how-9f113c8a293e?source=rss-f1a8d777cc44------2</link>
            <guid isPermaLink="false">https://medium.com/p/9f113c8a293e</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[development]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Praveen Kumar]]></dc:creator>
            <pubDate>Mon, 24 Jun 2024 07:05:40 GMT</pubDate>
            <atom:updated>2024-06-24T08:16:56.567Z</atom:updated>
            <content:encoded><![CDATA[<h3>1. Python’s love — Objects &amp; Abstract Concept — What, why, and how.</h3><p>Hi… Let’s look at an example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*syCUMtG6n5KczB8bcHzmHQ.png" /></figure><p>That is all there is to objects. Let’s remove all the values from one of the boxes. What do we get? The following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/594/1*7n-yl4Ha-JyFecwthBdHNQ.png" /></figure><p>Now this is more like a blank canvas. This is called as <strong><em>abstract</em></strong> representation. 👈 Let’s explore each section. Let’s label these boxes.</p><p>For those trying to quickly glance some parts, here’s a table for you to go your favorite section faster.</p><h3>THE WHAT SECTION</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/1*Ay4dDuC5rHnScA5DU6ljzA.png" /></figure><p><strong><em>Box 1:</em></strong> This is just for our reference in this blog. However, as we move ahead, we will only use this box and assume the other boxes we are keeping in our mind. Whenever in future we want to say we create an object, we will just draw this box, and the other boxes, we will assume that those are inside it.</p><p><strong><em>Box 2:</em></strong> This is the value box of an object. When we define something, we must give it a value. Why? Think logically, everything around you, is something because of a value given to it. Similar way, every object has some value. Even having ‘no value’ also means ‘None’ object. Python decided to name it ‘None’, so we use it as it to represent something that has “no value” or “Nothing”. A 5 is a 5 only if its value is 5. Same way, everything else, has a value.</p><p><strong><em>Box 3:</em></strong> The type box tells you what kind of stuff is inside. A type that tells Python (and us) what kind of data the object represents. For example, Apple, Mango, Avocado, etc. are of type ‘Fruits’. Similarly, all numbers from -infinity to +infinity are Integer, all text are of type String, etc. This box is very important as it tells Python what all it can do with it. For example, if it an integer type, then it can perform addition, subtraction, or any other arithmetic operations. We will learn about so often error related to type arise in program just because of this box not clearly looked up by programmers.</p><p><strong><em>Box 4:</em></strong> The final box is ID box. Think of this box as the houses on our planet. Every house (does not matter small/big/apartments) has an address. At a time, the house can belong to one family or a person. If someone moves out, someone else may move it to replace. The same concept is there in python. At a given instance in time, there can only be one object residing at one ID. You can think of ID as an address (except instead of planet, its inside your system RAM). Another interesting thing to know is, a house could be identical to a newly constructed house someplace else but surely their address will be different. This concept is very helpful in understanding why Python sometimes behave weirdly.</p><h3>THE WHY SECTION</h3><p>The why is simple: The way python is developed, it wanted to the programmers to not worry about defining or keeping these 3 boxes (2, 3, and 4) in mind. What does it mean? Let’s look at this code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/518/1*DPnL0gE-30fBprcGZj5LoQ.png" /></figure><p>We tried 3 different types of values, and without a worry, we did not need to know anything else. This type of behavior is called “Dynamically typed”. That means, it will dynamically take the shape of the underlying object and then handle memory on behalf of us without having to define it manually unlike other programming languages such as C, C++, or Java.</p><h3>THE HOW SECTION</h3><p>Now we know what and Why objects are there in python. Now using the above code example, we will show how it is created inside memory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UdPm3dsOv1MvqQq-jViOfQ.png" /></figure><p>The empty canvas we created above is called an “Abstract” Representation. What does it mean? It means that Python only keep in mind you will have “some value” of “some type” at “some location” in the memory. If you create it, the “some” part will be replaced with what you create. Until then, python don’t know what you will be creating. So, this idea is called an abstract.</p><p>In our code example, we create 3 values of variable “a”. If you look at the above diagram, it shows 3 different values, different types, and address. Now let’s focus on this step by step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/783/1*sS-OhMn89pJ0WkJYRblcMA.png" /></figure><p>The same thing happens for any other value in our code.</p><p>Now there’s lot more to cover about what happens if you create same variable twice, thrice, different types, etc. That’s for you to explore until we meet next time. See you around.</p><p>If you enjoyed it, do subscribe, follow me, and give a clap 👏 so it can reach to more people. Appreciate your time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9f113c8a293e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/practical-python/1-pythons-love-objects-abstract-concept-what-why-and-how-9f113c8a293e">1. Python’s love — Objects &amp; Abstract Concept — What, why, and how.</a> was originally published in <a href="https://medium.com/practical-python">Practical Python</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding How Programs Run — Part 1]]></title>
            <link>https://medium.com/@inboxpraveen/understanding-how-programs-run-part-1-5f93c727061d?source=rss-f1a8d777cc44------2</link>
            <guid isPermaLink="false">https://medium.com/p/5f93c727061d</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Praveen Kumar]]></dc:creator>
            <pubDate>Thu, 11 Jan 2024 09:08:38 GMT</pubDate>
            <atom:updated>2024-01-11T09:08:38.877Z</atom:updated>
            <content:encoded><![CDATA[<h3>Understanding How Programs Run — Part 1</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wCzmanRkcfXvTr5Yl8mF9w.png" /></figure><p>While many of us are great programmers, expanding our horizons to learn different programming languages, several core low-level constructs are carried out when you run the programs you have written. Understanding them helps us to understand how things work, and how to unlock the true powers of programming. The most fundamental concepts lie beneath 5 major hardware components inside any system. These are — CPU, RAM, ROM, Bus, and I/O. Additionally, we have an Operating system that orchestrates the functional behavior of programs.</p><p>In this blog, we will cover 3 things to understand how programs run — <strong>What </strong>is involved, <strong>How</strong> it is involved, and <strong>Why</strong> it is involved, really the 3 most definite questions you must have to understand almost anything.</p><blockquote>Before we start — My Name is Praveen Kumar, Senior AI Engineer with 5+ years of experience. I want to share foundational knowledge to help you succeed at programming, AI (CV, NLP, and Speech) &amp; in general, any Computer Science concept. Hit the claps and subscribe if you love and gain knowledge.</blockquote><p>We won&#39;t dive into the running programs directly. Let’s first understand what hardware is involved and unwrap its use in running any program.</p><p>1. Secondary Storage: The secondary storage is your actual storage unit, storing all your files and folders. It is primarily your hard disk drive (HDD) or Solid State Drive (SSD). If you have wallpapers, audio songs, video files, movies, program scripts, etc, all of these are stored in this component. It’s why secondary storage components are called “Non-volatile” storages or “Read-Only” storages because even after you shut down your system, upon the next reboot, all your files and programs are still available for you to see and use. Whenever you write any piece of program, you save your program into a file along with an extension. Once you finish saving, it is written into this storage. Every save you make propagates the updates to the location where it is stored.</p><p>2. Primary Storage: The primary storage (RAM) is your go-to memory whenever you want to do anything inside your system. When you say you have 100 songs, and 10 movies in your system, those are only visible to you because your system shows where it is located. For your system to show that you have those many files and folders, it loads the primary view into the memory. When you want to play a movie or a song, you tell your system “Hey I am opening this file” and your system takes the reference location where that file is there in the secondary storage, loads it into primary storage (RAM), and then the file is opened and then seen by you. For you to see, there is another component involved which is referred to as “Output”. Whenever you want to view something on your monitor screen, an Output is called, which renders you the view. That’s why sometimes we get frustrated when Chrome hangs our systems because they are loaded into RAM and consume a lot of memory, thus, we have less vacant space to view or do other tasks.</p><p>3. CPU: The Central Processing Unit (CPU) is often referred to as the brain of the computer. It’s where the actual processing of your program happens. When you execute a program, it’s the CPU that takes the instructions from your saved file in the secondary storage and loads them into the RAM. It then reads these instructions, executes them, and sends the necessary information to the Output devices for display or further processing. The CPU performs these operations through a cycle of fetching (getting the instruction), decoding (understanding what the instruction means), and executing (performing the instruction). Modern CPUs are incredibly fast and can perform billions of such cycles per second. Also you need to fundamentally question yourself — <strong>how does it even run? like it needs something to first store and then do something right?</strong> if you are thinking this, great. Because even “Cloud Computing” is not in clouds. Nothing happens on the fly. Even CPU has so called “memory” which is provided using “Registers”. Imagine Registers like hands to humans. whatever you want to do, you first use your hands to store it, and then do something with it.</p><p>4. Bus: The Bus in a computer system is like the highways connecting different cities. In technical terms, it’s the communication system that transfers data between components inside a computer, or between computers. The bus manages the data flow from your CPU to RAM, from RAM to the storage devices, and to the I/O devices. It ensures that the right piece of data gets to the right place at the right time. Without the bus, your CPU, RAM, and other components would be unable to communicate with each other, making them pretty much useless.</p><p>5. I/O Devices: Input and Output devices are your interfaces with the computer. Input devices like keyboards, mice, and scanners allow you to enter data into the computer, while output devices like monitors and printers let you receive information from the computer in a human-readable form. In the context of running a program, these devices play a crucial role in allowing you to interact with the program and see the results of its execution.</p><p>Now that we have a basic understanding of the hardware involved, let’s delve into the Operating System’s role. The OS is like the conductor of an orchestra, ensuring that each component plays its part at the right time. It manages the CPU’s time, controls the memory, and orchestrates the data flow through the bus. The OS also handles I/O operations, making sure that when you run a program, it interacts seamlessly with the hardware. So, long story short, your OS manages everything you do with your system, and your CPU manages everything you do within your program.</p><p>To combine all these, we get this blog. 😝Just kidding. We see a program running. Understanding these components is crucial because it helps you realize how your code interacts with the hardware.</p><p>Quick Summary: <em>You work on your computer with the help of </em><strong><em>OS</em></strong><em>. It will allow you to save your program in a file. Whenever you want to run a program, you tell you </em><strong><em>CPU</em></strong><em> “it’s time to run it”, it uses the </em><strong><em>Bus</em></strong><em> to take the program from </em><strong><em>ROM</em></strong><em> and load into </em><strong><em>RAM</em></strong><em>. Once loaded, it takes each line of code from your program, and then executes it (Not entirely true but should give you idea). CPU processes your instructions, and then shows you the final result of the program using </em><strong><em>I/O</em></strong><em>.</em></p><p>In the next section, we will look at how these components work individually to run a program, and why this process is designed the way it is. Stay tuned, and don’t forget to hit the claps and subscribe for more insights into the world of programming and AI!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5f93c727061d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unfolding the Realm of Natural Language Processing: A Practical Dive Series]]></title>
            <link>https://medium.com/@inboxpraveen/unfolding-the-realm-of-natural-language-processing-a-practical-dive-series-79572cbeeec3?source=rss-f1a8d777cc44------2</link>
            <guid isPermaLink="false">https://medium.com/p/79572cbeeec3</guid>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Praveen Kumar]]></dc:creator>
            <pubDate>Mon, 06 Nov 2023 09:03:46 GMT</pubDate>
            <atom:updated>2023-11-06T09:03:46.004Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o-nQnH8IGe0G5FES-11A_g.png" /></figure><p>In an age where data is the new oil, the ability to extract meaningful insights from data, particularly textual data, is a critical asset. Natural Language Processing (NLP), a subset of Artificial Intelligence (AI), bridges the chasm between human communication and computer understanding, unlocking a plethora of opportunities. This article series aims to unravel the core of NLP, Transformers, LLM’s, its applications, components, challenges, and the tools that make NLP a reality in today’s tech-savvy world.</p><p>Here is a high-level overview of the whole series and the topics we will be covering.</p><h3>Part 1: Introduction to NLP</h3><ol><li>Introduction to NLP</li></ol><ul><li>Definition and Importance</li><li>Real-world Applications of NLP</li><li>Overview of the Course Structure</li></ul><p>2. Basic Text Processing</p><ul><li>Tokenization</li><li>Stopword Removal</li><li>Stemming and Lemmatization</li></ul><p>3. Text Representation Techniques</p><ul><li>Bag of Words (BoW)</li><li>TF-IDF</li><li>Word Embeddings (Word2Vec, GloVe)</li></ul><p>4. Basic Math for NLP</p><ul><li>Linear Algebra</li><li>Probability and Statistics</li><li>Information Theory</li></ul><h3>Part 2: Core NLP Tasks</h3><ol><li>Syntax Analysis</li></ol><ul><li>Part-of-Speech Tagging</li><li>Parsing Techniques (Dependency Parsing, Constituency Parsing)</li></ul><p>2. Semantics and Pragmatics</p><ul><li>Word Sense Disambiguation</li><li>Semantic Role Labeling</li><li>Anaphora and Coreference Resolution</li></ul><p>3. Information Retrieval</p><ul><li>Boolean Retrieval</li><li>Ranked Retrieval</li><li>Evaluation Metrics in IR</li></ul><p>4. Information Extraction</p><ul><li>Named Entity Recognition</li><li>Relation Extraction</li></ul><h3>Part 3: Machine Learning and Deep Learning in NLP</h3><ol><li>Supervised, Unsupervised, and Semi-supervised Learning in NLP</li></ol><ul><li>Classification Tasks (Sentiment Analysis, Spam Detection)</li><li>Clustering and Topic Modeling</li></ul><p>2. Sequence Modeling</p><ul><li>Hidden Markov Models</li><li>Recurrent Neural Networks (RNNs)</li><li>Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRUs)</li></ul><p>3. Advanced Attention Mechanisms</p><ul><li>Attention is all you need!</li><li>Understanding Various Attention Mechanisms</li></ul><p>4. Transformers and Recent Advancements</p><ul><li>Introduction to Transformers</li><li>Architectural Improvements and Optimizations</li><li>Self-Supervised Learning and Architecture Search​</li></ul><p>5. Latest Transformer Models</p><ul><li>BERT, GPT-3, T5, and GPT-4</li><li>Applications in Natural Language Generation, Conversational AI, Content Generation, and more​</li></ul><h3>Part 4: Advanced Topics and Recent Trends</h3><ol><li>Transfer Learning and Few-shot Learning in NLP</li></ol><ul><li>Pre-training and Fine-tuning</li><li>Few-shot and Zero-shot Learning</li></ul><p>3. Multimodal Learning</p><ul><li>Visual, Audio, and Language Representation Learning</li><li>The Gato model by DeepMind​</li></ul><p>4. Large Language Models (LLMs)</p><ul><li>Understanding LLMs and Their Applications</li><li>Generative AI and AI Chatbots​<a href="https://beebom.com/best-large-language-models-llms/#:~:text=12%20Best%20Large%20Language%20Models,fold">8</a>​</li></ul><p>4. Ethics in NLP</p><ul><li>Bias and Fairness</li><li>Privacy and Security</li></ul><p>5. State-of-the-art Models and Case Studies</p><ul><li>Exploration of Top NLP Algorithms and Models of 2023</li><li>Building End-to-End NLP Applications​</li></ul><h3>Part 5: Practical Projects and Capstone</h3><ol><li>Mini-Projects</li></ol><ul><li>Covering a variety of NLP tasks and applications</li></ul><p>2. Capstone Project</p><ul><li>Designing and Building an End-to-End NLP Application</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=79572cbeeec3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Natural Language Processing — The Next Wave to Prepare for in 2024–25]]></title>
            <link>https://medium.com/analytics-vidhya/natural-language-processing-the-next-wave-to-prepare-for-in-2024-25-b432a1345494?source=rss-f1a8d777cc44------2</link>
            <guid isPermaLink="false">https://medium.com/p/b432a1345494</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Praveen Kumar]]></dc:creator>
            <pubDate>Mon, 15 May 2023 03:20:52 GMT</pubDate>
            <atom:updated>2023-05-15T03:20:52.086Z</atom:updated>
            <content:encoded><![CDATA[<h3>Natural Language Processing — The Next Wave to Prepare for in 2024–25</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uFXvJU9PYtRVKpubaldmAA.jpeg" /></figure><p>Welcome, dear reader! Today we’re going to talk about the magical world of Natural Language Processing (NLP). I hope you brought your sense of humor, because let’s face it, without it, we’d all be bots.</p><p>NLP is a branch of artificial intelligence that deals with the interaction between computers and humans using natural language. In simpler terms, it’s teaching computers how to understand and communicate with us, pesky humans and our inconsistent language. And believe me, it’s not an easy task, even for the most advanced algorithms out there.</p><p>NLP has the potential to bend the future in ways we can’t even imagine yet. As technology continues to advance, NLP will play an increasingly important role in how we interact with that technology. It will enable us to communicate with machines in a more natural and intuitive way, making technology more accessible and user-friendly for everyone.</p><p>So why is it important to learn NLP? For starters, NLP is an in-demand skill in the tech industry. Learning NLP can open up a wide range of job opportunities, from working on chatbots and virtual assistants to developing language models for search engines and social media platforms. But more than that, NLP has the potential to improve our lives in countless ways, from healthcare and education to business and beyond. By learning NLP, you’ll be able to contribute to the development of new tools and technologies that have the potential to make a real difference in the world.</p><p>One area where NLP will have a huge impact is in the field of healthcare. Imagine a world where doctors can use NLP-powered tools to analyze patient data and make more accurate diagnoses. Or where patients can use chatbots to get personalized medical advice without having to leave their homes. NLP can also be used to analyze medical records and research papers, helping researchers discover new treatments and cures for diseases.</p><p>Another area where NLP will be important is in the field of education. NLP can be used to create intelligent tutoring systems that can adapt to a student’s individual learning style and pace. It can also be used to analyze student writing and provide feedback on grammar and style. And imagine being able to learn a new language using an NLP-powered app that can understand your mistakes and provide personalized feedback.</p><p>NLP will also be important in the business world. Companies can use NLP to analyze customer feedback and sentiment, helping them make more informed decisions about product development and marketing. It can also be used to automate customer service, freeing up human agents to focus on more complex issues.</p><p>The applications and use cases of NLP are vast and varied. From sentiment analysis and chatbots to machine translation and speech recognition, NLP has the potential to revolutionize the way we interact with technology.</p><p>Let’s start with sentiment analysis. This technique uses NLP algorithms to identify the sentiment or emotion behind a piece of text. Imagine the possibilities — analyzing customer feedback to improve products and services, monitoring social media to track brand reputation, or even predicting market trends based on news articles. And if you’re feeling down, sentiment analysis can even detect that and cheer you up with some funny cat videos. Ok, that last part might not be true, but one can always hope.</p><p>Next, let’s talk about chatbots. Chatbots are computer programs that simulate human conversation, often used in customer service or online shopping. NLP is used to understand the user’s message and generate a response. It’s like having your own personal assistant, except it’s a robot that never gets tired and doesn’t need coffee breaks. Although, come to think of it, I could use a coffee break right now.</p><p>Machine translation is another use case for NLP. This technique uses algorithms to translate text from one language to another. It’s not perfect yet, but it’s come a long way from the days of Google Translate spitting out gibberish. And who knows, maybe one day we’ll be able to communicate seamlessly with people from all over the world, without having to resort to pantomime and exaggerated hand gestures.</p><p>Speech recognition is also a major application of NLP. It’s what allows us to speak to our devices and have them respond, like saying “Hey Siri” or “Ok Google” 🔊. Speech recognition has come a long way, but it’s not perfect yet, especially if you have a thick accent or are in a noisy environment. But hey, if you ever want to have some fun, try talking to your voice assistant in a ridiculous accent and see what happens. 😛</p><p>Now, you may be thinking, “This all sounds great, but why should I follow someone with three years of experience in NLP, Large Language Models (LLM’s) &amp; Self Attention Networks?” Well, let me tell you. NLP is a rapidly evolving field, with new algorithms and techniques being developed all the time. Having someone with experience can help you stay up to date with the latest developments and best practices. Plus, can help you navigate the confusing and often overwhelming world of NLP. And I might even share some funny cat videos with you. 😆</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eNMbFZpcLnB87UTlnfQGIg.jpeg" /></figure><p>Lets face it. This is just one side of the coin. On the other side, there are deep concerns and is one of the main reasons why larger companies will soon start to slow down LLM’s experiment.</p><p>As with any technology, AI and NLP also have some potential side effects, particularly in the area of ethics. One of the most significant ethical concerns associated with AI is bias. Machine learning algorithms can only learn from the data they are given, and if that data is biased, the resulting AI system will also be biased.</p><p>For example, if an AI system is trained on data that is primarily collected from one specific demographic group, it may not be able to accurately predict outcomes for people from other demographic groups. This can lead to unfair treatment and discrimination in areas like hiring, lending, and healthcare.</p><p>Another ethical concern related to AI is privacy. NLP systems often rely on vast amounts of personal data, such as email content, social media posts, and chat transcripts. If this data falls into the wrong hands or is misused, it can lead to serious privacy violations and breaches of trust.</p><p>AI can also lead to job displacement, as automation can replace certain types of human labor. This can have significant economic and social consequences, particularly for workers in industries that are particularly susceptible to automation.</p><p>Finally, there is also the risk of AI being used for malicious purposes, such as creating deepfake videos or spreading misinformation. As AI and NLP systems become more advanced, it may become increasingly difficult to distinguish between real and fake information, making it easier for bad actors to spread false information and manipulate public opinion.</p><p>It is important for developers, policymakers, and society as a whole to consider these ethical concerns as AI and NLP continue to evolve and become more integrated into our daily lives. Proper safeguards and regulations can help ensure that these technologies are developed and used in a responsible and ethical manner, for the benefit of all.</p><p>Thank you for reading! Hope you enjoyed it. Follow me for more great and insightful learning.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b432a1345494" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/natural-language-processing-the-next-wave-to-prepare-for-in-2024-25-b432a1345494">Natural Language Processing — The Next Wave to Prepare for in 2024–25</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python Data Models Explained — Object, Type & Identity]]></title>
            <link>https://medium.com/analytics-vidhya/python-data-models-explained-object-type-identity-8232d174cca1?source=rss-f1a8d777cc44------2</link>
            <guid isPermaLink="false">https://medium.com/p/8232d174cca1</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[model]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Praveen Kumar]]></dc:creator>
            <pubDate>Sat, 26 Nov 2022 11:32:12 GMT</pubDate>
            <atom:updated>2022-11-26T11:32:12.756Z</atom:updated>
            <content:encoded><![CDATA[<h3>Python Data Models Explained — Object, Type &amp; Identity</h3><p>Every python data is an object! What do I mean by that? Let’s find out.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/1*ykstS46OiF7eBfaKybq6Lg.png" /></figure><p>The official documentation says that “Objects are Python’s abstraction of data”. 😅 Meaning? To understand such a statement let’s know what abstraction actually means. Abstract means “just the idea but not as a physical thing”. So in simpler terms, without knowing every detail of how things can be done, abstraction is a way to represent an idea. Thus, the statement “Objects are Python’s abstraction of data” means that “Python language uses the concept of Objects to represent all their data”.</p><p>Simple enough? Let’s move on.</p><p>Now we know that every data representation can be understood by its underlying object representation. So, if we create an integer variable in python, it would be called an integer object. The same applies to all the different variables and data types you can construct or think of.</p><p>Whenever we create an object, we associate 3 properties with it.</p><ol><li>The value of an object</li><li>The location or Identity of an object</li><li>The type of an object</li></ol><p>For example, when we define: A = 10, we have all these 3 properties bound with it. <strong>Note that A is just a variable(reference/pointer) &amp; not an object. 10 is the object. </strong>In python, 10 (an object) is first created into the Heap Segment (Since it’s a global variable), and then a reference variable ‘A’ is created which points to the memory address where 10 is created. Thus A is just a mere reference to the object. The same principle is followed across every python object.</p><p>When ‘10&#39; was created, the 3 properties are defined as follows:</p><ol><li>Value: 10</li><li>Location: 0x238A352 (Some memory address)</li><li>Type: Integer</li></ol><p>This is the reason python only creates a single object of 10. If later you create another variable with the same value, the address of the of 10 is again assigned to the new variable without having to create another object of the same value. In CPython implementation, there is an inbuild method <strong><em>id(variable/object)</em></strong> which is just to find out the memory address of a variable/object. let’s look at the following code example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/341/1*74lY2XTyNUYGSGqN666RdQ.png" /></figure><p>This shows ‘a’ and 10 and ‘c’, all of them have the same memory location. This shows that only 1 instance of 10 was created, and then it was assigned to ‘a’ &amp; ‘c’.</p><p>Python allows us to compare two <strong><em>values</em></strong> with the ‘==’ operator. However, to compare if two variables are pointing/referencing to the same memory location, we have to use the <strong><em>is </em></strong>operator.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/328/1*yGM_pUDEUXh4NmKtn-4GuQ.png" /></figure><p>Here, when we do <strong><em>a == c</em></strong>, the values are compared. However, when we do <strong><em>a is c</em></strong>, it internally compares id(a) == id(c).</p><p>Along with the identity/location of an object, the type of the object is helpful in deciding the operations, limitations, and methods that are associated with the object. For example, the length can be calculated for a string object but not for float or integer type.</p><p>Objects whose values can be changed are called <strong>mutable </strong>objects and objects whose values can not be changed once created are called <strong>immutable </strong>objects. However, immutability is not just the essence of unchangeable objects, it&#39;s more subtle. Let’s take an example to explain what I mean by that.</p><p>Official python documentation states the following data types.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/558/1*lBwZpOHu2AF-gFhq2KdIYA.png" /><figcaption>Python Data Types &amp; their Mutability</figcaption></figure><p>Consider the following program now…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/345/1*hHJlemJvUHhCKH9ustSBFA.png" /></figure><p>if we see, we have a tuple object and it contains a list object. This is an example of an Immutable container (A container is an object which can contain references to other objects) containing a mutable object. We can change the inner list but we can not still change the immutable container which contains it. Even though you may feel that the tuple is changed, it is not the tuple but the mutable object inside it changed. This is why python is a live language because it can allow you to such complex behavior in a more subtle way.</p><p>Learning always pays off. Stay patient &amp; keep learning. 📚</p><p>If you got value from this article, share it on your Social Media handles. 😃</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8232d174cca1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/python-data-models-explained-object-type-identity-8232d174cca1">Python Data Models Explained — Object, Type &amp; Identity</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Movie Recommendation System | Python & Flask | Web Application | Heroku Deployment]]></title>
            <link>https://medium.com/analytics-vidhya/movie-recommendation-system-python-flask-web-application-heroku-deployment-7e39492b640c?source=rss-f1a8d777cc44------2</link>
            <guid isPermaLink="false">https://medium.com/p/7e39492b640c</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[recommendations]]></category>
            <category><![CDATA[movies]]></category>
            <category><![CDATA[heroku]]></category>
            <dc:creator><![CDATA[Praveen Kumar]]></dc:creator>
            <pubDate>Thu, 30 Jan 2020 11:06:43 GMT</pubDate>
            <atom:updated>2023-11-13T15:22:19.010Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*510YIZPxybErPCE_h3SeQw.png" /></figure><h3>Movie Recommendation System | Python AI Web Application | Cloud Deployment</h3><blockquote>The final Web-Application can we checked <a href="https://movie-recommendation-8g56.onrender.com/">HERE</a>. (Be patient — the clound run is slow)</blockquote><blockquote>The complete code is on Github. Check <a href="https://github.com/inboxpraveen/recommendation-system">HERE</a>.</blockquote><p>Welcome, cinephiles and coders alike! Today, we’re diving into the fascinating world of recommendation systems. Specifically, we’re constructing a Movie Recommendation System using Python, leveraging the power of Kaggle datasets. This system is designed to suggest movies to users based on their viewing history and preferences. So, grab your popcorn, and let’s get coding!</p><p>In this blog, I am going to explain the following:</p><ol><li>Data Preprocessing</li><li>Building Movie Recommender Machine Learning Model</li><li>Creating Web Pages and connecting them to Django Rendering</li><li>Create a complete interface and exception handling using Python along with form validation</li><li>Deploying to Render.com</li></ol><p>So, without further ado, let’s get started. Also, feel free to skip any section if you are already familiar with it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*nAC9rB79M5U--Cx6xavDTQ.gif" /></figure><h3>Setting the Stage: Importing Libraries and Datasets</h3><p>First things first, we need to set up our environment by installing necessary packages and importing our data. We’ll use pandas for data manipulation, numpy for numerical operations, and several other tools for specific tasks like feature extraction and similarity computation.</p><pre>!pip install --quiet fastparquet<br>!pip install --quiet pyarrow<br><br>import pandas as pd<br>import numpy as np<br>from sklearn.feature_extraction.text import CountVectorizer<br>from sklearn.metrics.pairwise import cosine_similarity<br>import pyarrow as pa<br>import pyarrow.parquet as pq<br>import warnings<br>warnings.simplefilter(&#39;ignore&#39;)</pre><p>We load our movie datasets — movies_metadata.csv, credits.csv, keywords.csv, and links.csv. These datasets provide a comprehensive look at various aspects of the movies, including metadata, cast, crew, and keywords associated with each movie.</p><h3>Data Limitations</h3><p>Let’s look at the dataset first. We are going to use <a href="https://www.kaggle.com/tmdb/tmdb-movie-metadata">TMDB5000 DATASET</a> from <a href="https://www.kaggle.com/">Kaggle.com</a>. The dataset consists of 2 files, namely, tmdb_5000_credits.csv &amp; tmdb_5000_movies.csv. There is another dataset called <a href="https://www.kaggle.com/rounakbanik/the-movies-dataset">THE MOVIES DATASET</a> which has more than a million movie reviews and ratings. However, I did not use it for 2 reasons.</p><ol><li>The dataset is too large for the system &amp; requires an estimate of 45–50GB RAM.</li><li>The machine learning model produced is also too large for Render. Render does not allow us to store more than 250MB on a free account.</li></ol><h3>Data Preprocessing: The Art of Cleaning Data</h3><p>Data preprocessing is crucial in any data science project. We start by dropping rows with inconsistent data types, extracting relevant information, and merging datasets to create a master dataset. This master dataset will serve as the backbone of our recommendation system.</p><h3>Key Steps in Data Preprocessing:</h3><ol><li>Extracting Genres: Convert genres from a string representation to a list.</li><li>Type Conversion: Ensure the ‘id’ column across different datasets is of the same type for successful merging.</li><li>Merging Datasets: Combine movies_dataset, credits, and keywords to form a unified dataset.</li><li>Parsing Columns: Convert the stringified lists back into lists for columns like cast, crew, and keywords.</li><li>Extracting Director’s Name: A function get_director is created to extract the director&#39;s name from the crew information.</li><li>Cleaning and Trimming Data: Focus on top cast members, stem keywords, and lowercase transformations to maintain uniformity and relevance.</li><li>Creating a Soup Feature: A combination of keywords, cast, director, and genres for each movie. This ‘soup’ becomes the basis of our content-based recommendation system.</li></ol><pre>movies_dataset[&#39;genres&#39;] = movies_dataset[&#39;genres&#39;].fillna(&#39;[]&#39;).apply(literal_eval).apply(lambda x: [i[&#39;name&#39;] for i in x] if isinstance(x, list) else [])<br>movies_dataset[&#39;id&#39;] = movies_dataset[&#39;id&#39;].astype(&#39;int&#39;)<br>master_dataset = movies_dataset.merge(credits, on=&#39;id&#39;).merge(keywords, on=&#39;id&#39;)</pre><p>we transform the genres column into a list of genre names. We also ensure the id column across datasets is of the same data type (int) for smooth merging. Finally, we merge the movies_dataset with credits and keywords datasets to form a comprehensive master_dataset.</p><h3>Feature Engineering: The Secret Sauce</h3><p>Feature engineering involves transforming raw data into features that better represent the underlying problem, improving the accuracy of machine learning models. In our case, we create a ‘soup’ feature that amalgamates all the significant elements of a movie into a single string.</p><pre>master_dataset[&#39;cast&#39;] = master_dataset[&#39;cast&#39;].apply(lambda x: [str.lower(i.replace(&quot; &quot;, &quot;&quot;)) for i in x][:3])<br>master_dataset[&#39;director&#39;] = master_dataset[&#39;crew&#39;].apply(get_director).astype(&#39;str&#39;).apply(lambda x: str.lower(x.replace(&quot; &quot;, &quot;&quot;)))<br>master_dataset[&#39;soup&#39;] = master_dataset[&#39;keywords&#39;] + master_dataset[&#39;cast&#39;] + master_dataset[&#39;director&#39;] + master_dataset[&#39;genres&#39;]<br>master_dataset[&#39;soup&#39;] = master_dataset[&#39;soup&#39;].apply(lambda x: &#39; &#39;.join(x))</pre><h3>Building the Recommendation Engine</h3><p>Once our data is prepped and ready, we focus on the core of our project — the recommendation engine. This engine is based on the concept of content-based filtering.</p><h3>Steps to Build the Engine:</h3><ol><li>Count Vectorization: We use CountVectorizer to convert the soup of words into a matrix of token counts.</li><li>Cosine Similarity: This metric helps us in determining how similar the movies are to each other based on their soup feature.</li><li>Storing Results: We store the master dataset and similarity matrix using pyarrow, which is efficient for handling large datasets.</li></ol><pre>count = CountVectorizer(analyzer=&#39;word&#39;, ngram_range=(1, 2), min_df=2, stop_words=&#39;english&#39;)<br>count_matrix = count.fit_transform(master_dataset[&#39;soup&#39;])<br>cosine_sim = cosine_similarity(count_matrix, count_matrix)<br><br>table = pa.Table.from_pandas(pd.DataFrame(cosine_sim))<br>pq.write_table(table, &#39;/content/model.parquet&#39;)</pre><p>With this, we finish building our recommender engine. It is saved to the model.parquet.</p><h3>Integration with Django Web Application</h3><h3>The Django Setup</h3><p>We have a Django project named movie_recommendation a single app called recommender. This structure is clean and typical for a focused Django application.</p><h3>Project Structure</h3><ul><li>The movie_recommendation directory contains global settings and URLs.</li><li>The recommender app is where the magic happens—it&#39;s responsible for generating movie recommendations.</li><li>Static files, including our precomputed movie dataset and similarity matrix, reside in the static directory.</li></ul><h3>Deploying to Render.com</h3><p>Before we begin deployment, create an account on <a href="https://dashboard.render.com/">render.com</a>. Simple steps for creating an account are:</p><ol><li>Visit <a href="https://dashboard.render.com/">Render’s website</a> and click on the “Sign Up” button.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HRP5g48OfDYdk7TWXnibWw.png" /></figure><p>2. Enter your email address, create a password, and agree to the terms of service.</p><p>3. Verify your email address by clicking on the link sent to your inbox.</p><p>4. Once verified, you will be redirected to your Render dashboard.</p><p>Then, once you are on your dashboard, select the top right-hand side toggle button to create a new web service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-Yuq9xIKJWmfoMTH3AXWsA.png" /></figure><p>Once you choose, it will ask you to choose your code base. As we have already pushed our code on GitHub, we will authorize it to our repository. If you have not logged in through GitHub, it will prompt you to provide authorization to all your GitHub repositories.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CIzAE0lyOs136sFuocqt6Q.png" /></figure><blockquote>VERY VERY IMPORTANT STEP BEFORE YOU AUTHORIZE YOUR REPOSITORY.</blockquote><blockquote>You need to create a build.sh file at the root level which contains the following code.</blockquote><pre>#!/usr/bin/env bash<br>set -o errexit<br>pip install -r requirements.txt<br>python manage.py collectstatic --no-input</pre><blockquote>This is used by Render.com to build your application automatically. It has commands which need to be run in-order to run your application smoothly. So it installs all your dependencies and then collects all your static code files and model files, and then starts the final build to cloud deploy.</blockquote><p>This file is located in my repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/928/1*jRAdRmFWOl6HaaV-nrvSHQ.png" /></figure><p>Currently, we do not use db.sqlite3 database file but it is also necessary to be available before the build. Once you complete this step, then you can proceed to connect your repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OukFvVE8NcVT_NFaACjgKw.png" /></figure><p>Once you select, you will be popped with a screen with inputs required before the final build. It contains things like the name of your web application, environments, build commands, cloud computing to use, etc. The most important of all of these are 3 things — The name of your web service (this will be shown on your dashboard and not the domain), the GitHub branch on which the code is there, and the build command. I have highlighted them here for example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lKiA6x1KS5Pb2heqmSVA-g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u9IEZjs-FBNCxtpgJwtOcQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HNAlD0pe21rw_qfqFlvOwQ.png" /></figure><p>You are expected to choose instance type, but in our case, we are only using the free tier so we continue with that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q74oJrJ1qRefH0g1uucKLA.png" /></figure><p>Once done, choose “Create Web Service” and you will be redirected to this page which shows the name of the web service, the URL on which your application will be hosted, and logs that show the current status of the application build.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZIEDZIxxZJ_LH3jNlC-XiA.png" /></figure><p>The default option to deploy is automatically, triggered every time you make changes to the branch. With this, your application will be launched successfully. You can visit the URL seen on the dashboard.</p><p>Hooray!!! Congratulations on building your application. These steps can be used to build any Web Application using the Python-Django Framework. You may need to add slight tweaks accordingly, but the fundamentals will be the same.</p><blockquote>The Final application can we checked <a href="http://recmovie.herokuapp.com/">HERE</a>. (Be patient — the clound run is slow)</blockquote><blockquote>The complete code is on Github. Check <a href="https://github.com/inboxpraveen/recommendation-system">HERE</a>.</blockquote><blockquote>If you like it, make sure to follow me for more &amp; upcoming awesome articles for <strong>free</strong>.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7e39492b640c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/movie-recommendation-system-python-flask-web-application-heroku-deployment-7e39492b640c">Movie Recommendation System | Python &amp; Flask | Web Application | Heroku Deployment</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Real-World Problems & How Machine Learning comes into the picture]]></title>
            <link>https://medium.com/ai-techsystems/real-world-problems-how-machine-learning-comes-into-the-picture-719463910d39?source=rss-f1a8d777cc44------2</link>
            <guid isPermaLink="false">https://medium.com/p/719463910d39</guid>
            <category><![CDATA[problem-solving]]></category>
            <category><![CDATA[solutions]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Praveen Kumar]]></dc:creator>
            <pubDate>Thu, 25 Jul 2019 10:53:06 GMT</pubDate>
            <atom:updated>2019-07-29T02:53:58.781Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-sa/4.0/</cc:license>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*MczzhtfHoMxvXk2kaJba2A.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z4vu2kw1bvcPrrV1y8qH-g.png" /></figure><p>Written by <a href="https://medium.com/u/f1a8d777cc44">Praveen Kumar</a> for blogs at <a href="https://medium.com/u/1ff8967cb825">AI Technology &amp; Systems</a></p><p>It’s an era of tremendous &amp; electrically fast technological changes. Within a flick of an eye, the way we read, understand, share, communicate, &amp; interact with technology changes. We are in the 4th industrial phase in which use cyber-physical systems to monitor, analyze, and automate business. With the help of Machine learning, we are now able to gain insights from historical data &amp; make predictions to solve problems which were not possible in any of the industry-phases before. We can now store a tremendous amount of data as well as make large computations on it. Machine learning has come up with solutions from data which were not humanly possible. However, we still have a long way to go because we still have problems which we face real-world life community.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*FkB8Qxx80ZN3zDtAvSoXIA.png" /></figure><p>Anyone who is objectively searching for answers to AI/ML problems knows that we are still just tackling low hanging fruits. That makes a lot of sense since AI is vaguely defined and if at all you wish to make any progress in AI, then you should learn to simplify your objectives or goals. If you read media outlets, it may seem like researchers are sweeping the floor clean with deep learning (DL), solving ML problems one after the other leaving no stones unturned. In reality, they are not, researchers attack relatively simpler problems in the hope of collectively solving the bigger problems that is just how research works. For example, object detection is mostly about the recovery of 2D bounding boxes with some limitations on the various objects can undergo visually. Though, in reality, the problem is harder than that, object poses are not in 2D but they are in 3D and it is not trivial to extend a 2D bounding box based object detection algorithm to 3D precise object detection problem. However, simple datasets like MNIST are considered solved but that is a toy problem when it comes to real-world variation in digit appearance. ImageNet is also solved but it is still not as challenging as everyday vision problems human visual systems solve. So even when you read about algorithms beating humans on ImageNet, that is just a small part of the full picture. There is a lot of research work needed to be done in:</p><p>Computer Vision such as object detection: The so-called state-of-the-art object detection algorithms make silly errors a human would never make. In the vision, we have more problems like video understanding, segmentation, image-to-image translation and many more.</p><p>Language translation: We do have better translation algorithms thanks to long-short-term-memory (LSTM) and convolutional networks than say, 5 years ago but trust me, humans do far better in translation.</p><p>Adversarial examples: Even towards the end of 2018, we still haven’t solved the adversarial examples problem yet. ML models can still be fooled by adversarial attacks. It is interesting to note that work coming from Google AI shows that even time-limited humans</p><p>can be affected by some adversarial examples. But given enough time, humans are extremely robust to such tricks.</p><p>Unsupervised/reinforcement learning: ML algorithms are highly dependent on clean and labeled data but we know that a lot more information can be gathered by agents interacting and experimenting with the environment all by themselves. There has been a lot of work in the area of reinforcement learning (RL) especially from DeepMind and OpenAI teams.</p><p>Generalization: As others have pointed out here, ML models find it hard to generalize well to novel conditions that deviate from optimal performance conditions.</p><p>Zero/one/few-shot learning: Humans, animals, and insects can learn from very little information in a very complex setting. Trying to emulate such capabilities in machines has remained elusive even to this very day (the time of writing of this answer).</p><p>Some real-life scenarios:</p><p>1. Climate &amp; Forest classification:</p><p>Studying &amp; understanding patterns in weathers, climate changes, Atmospheric changes, lot of lives can be saved. Forest classification datasets are present on Kaggle as well as with the governing agencies, but yet, there have no steps taken forward to analyze Flora &amp; Fauna classification based on regions &amp; states. If this is done, we will be able to grow plants where we have scarcity and being able to make balances of all the types of plants in almost every region.</p><p>2. Accidents:</p><p>According to the WHO annual global road crash statistics, nearly 1.25M million people die in road crashes, on average 3287 deaths a day. An additional 20–50M are injured or disabled [ref: <a href="https://www.asirt.org/safe-travel/road-safety-facts/].">https://www.asirt.org/safe-travel/road-safety-facts/].</a> We need to build systems on streets which are able to detect any of these before happening and also build autonomous systems targeting accidents as one of the most important factors.</p><p>3. Privacy &amp; Security:</p><p>Privacy &amp; Security are the most important aspects of a person’s life. Personal Information must be kept to ones and ones only. No one has the right to acknowledge some else’s private data &amp; and other information. Privacy &amp; Security must be taken care of. Machine learning systems should be able to protect user’s data if anyone is trying to access by illegal means and authorization.</p><p>4. Ledgers &amp; Online transactions: Blockchain is a technology that can help track transactions between users in a public ledger which was developed originally for Bitcoin. It is now used commercially for various applications such as tracking ownership documents, digital assets or voting rights. Combining ML Systems with Blockchain will not just eliminate 3rd party indulgence, but also will be able to provide a stamp of trust on each legal online transaction. It will also help in maintaining anonymity throughout the web and no breach of privacy &amp; security can be exploited.</p><p>We are in one of the biggest transitions phases and with each passing day, old technologies are eliminated and new solutions are built which are smarter, better &amp; secure. We are still a long way away from the target of making the world a better place but well on course of doing so. We have problems, but we also keep digging for the solution and it’s about to happen in some time sooner. Some of the problems mentioned above are already been under tremendous success but yet to be completed or deployed on a large scale, others might just need some more time &amp; research. While we solve problems, it’s also important we also make efforts in adapting new methods, technological solutions as well as move ahead with the technology by supporting it to make it happen on a global scale.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=719463910d39" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai-techsystems/real-world-problems-how-machine-learning-comes-into-the-picture-719463910d39">Real-World Problems &amp; How Machine Learning comes into the picture</a> was originally published in <a href="https://medium.com/ai-techsystems">AITS Journal</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>