<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Learn with Nas on Medium]]></title>
        <description><![CDATA[Stories by Learn with Nas on Medium]]></description>
        <link>https://medium.com/@learnwithnas?source=rss-cbae1ad7c072------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*wNXvyIhoQPEpWY1ylrP02A.jpeg</url>
            <title>Stories by Learn with Nas on Medium</title>
            <link>https://medium.com/@learnwithnas?source=rss-cbae1ad7c072------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 04:32:18 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@learnwithnas/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[I Built an ML Prediction System with AI Agents]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/womenintechnology/building-an-ml-prediction-system-with-ai-from-data-pipelines-to-product-design-44d6a8fb9c7d?source=rss-cbae1ad7c072------2"><img src="https://cdn-images-1.medium.com/max/1414/1*IUt_HNu01nYFPCCl8NiDzg.png" width="1414"></a></p><p class="medium-feed-snippet">How I used Claude to build a machine learning pipeline, then turned its outputs into an application using mockups, requirements, and AI&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/womenintechnology/building-an-ml-prediction-system-with-ai-from-data-pipelines-to-product-design-44d6a8fb9c7d?source=rss-cbae1ad7c072------2">Continue reading on Women in Technology »</a></p></div>]]></description>
            <link>https://medium.com/womenintechnology/building-an-ml-prediction-system-with-ai-from-data-pipelines-to-product-design-44d6a8fb9c7d?source=rss-cbae1ad7c072------2</link>
            <guid isPermaLink="false">https://medium.com/p/44d6a8fb9c7d</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[machine-learning-ai]]></category>
            <category><![CDATA[claude]]></category>
            <dc:creator><![CDATA[Learn with Nas]]></dc:creator>
            <pubDate>Sun, 01 Nov 2026 17:24:38 GMT</pubDate>
            <atom:updated>2026-11-02T14:18:21.755Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[How to Preprocess, Visualise, and Analyse Spatial Data: A Step by Step Case Study on Urban…]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/womenintechnology/how-to-preprocess-visualise-and-analyse-gis-data-a-step-by-step-case-study-on-urban-d2949c9e1f6c?source=rss-cbae1ad7c072------2"><img src="https://cdn-images-1.medium.com/max/1944/1*loPr7ungJ0Pz4ufpmS4G-g.png" width="1944"></a></p><p class="medium-feed-snippet">To read this story for free, click here.</p><p class="medium-feed-link"><a href="https://medium.com/womenintechnology/how-to-preprocess-visualise-and-analyse-gis-data-a-step-by-step-case-study-on-urban-d2949c9e1f6c?source=rss-cbae1ad7c072------2">Continue reading on Women in Technology »</a></p></div>]]></description>
            <link>https://medium.com/womenintechnology/how-to-preprocess-visualise-and-analyse-gis-data-a-step-by-step-case-study-on-urban-d2949c9e1f6c?source=rss-cbae1ad7c072------2</link>
            <guid isPermaLink="false">https://medium.com/p/d2949c9e1f6c</guid>
            <category><![CDATA[gis]]></category>
            <category><![CDATA[data-processing]]></category>
            <category><![CDATA[women-in-tech]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[qgis]]></category>
            <dc:creator><![CDATA[Learn with Nas]]></dc:creator>
            <pubDate>Mon, 01 Dec 2025 09:22:50 GMT</pubDate>
            <atom:updated>2025-12-13T02:39:10.309Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[How to Preprocess, Visualise, and Analyse Spatial Data: A Step by Step Case Study on Urban…]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/womenintechnology/how-to-preprocess-visualize-and-analyze-gis-data-a-step-by-step-case-study-on-urban-4867bc24af21?source=rss-cbae1ad7c072------2"><img src="https://cdn-images-1.medium.com/max/1874/1*UM-E7FiOtbjKP9tTB6zRkQ.png" width="1874"></a></p><p class="medium-feed-snippet">In Part 1, we covered the Preprocessing and Analysis steps.
In this part, we shift our focus to how the results are visualised and how to&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/womenintechnology/how-to-preprocess-visualize-and-analyze-gis-data-a-step-by-step-case-study-on-urban-4867bc24af21?source=rss-cbae1ad7c072------2">Continue reading on Women in Technology »</a></p></div>]]></description>
            <link>https://medium.com/womenintechnology/how-to-preprocess-visualize-and-analyze-gis-data-a-step-by-step-case-study-on-urban-4867bc24af21?source=rss-cbae1ad7c072------2</link>
            <guid isPermaLink="false">https://medium.com/p/4867bc24af21</guid>
            <category><![CDATA[data-processing]]></category>
            <category><![CDATA[qgis]]></category>
            <category><![CDATA[women-in-tech]]></category>
            <category><![CDATA[gis]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[Learn with Nas]]></dc:creator>
            <pubDate>Mon, 01 Dec 2025 09:17:52 GMT</pubDate>
            <atom:updated>2025-12-13T02:39:06.908Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Common Causes of Software Failures and How Clear Requirements Can Help: A Sample Product…]]></title>
            <link>https://learnwithnas.medium.com/common-causes-of-software-failures-and-how-clear-requirements-can-help-a-sample-product-be7b9a6c601b?source=rss-cbae1ad7c072------2</link>
            <guid isPermaLink="false">https://medium.com/p/be7b9a6c601b</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[product-requirements]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[product-requirement-docs]]></category>
            <category><![CDATA[prd]]></category>
            <dc:creator><![CDATA[Learn with Nas]]></dc:creator>
            <pubDate>Wed, 30 Oct 2024 04:01:52 GMT</pubDate>
            <atom:updated>2024-10-30T04:01:52.825Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NR2KA-bQZ2OgPtACOmQK4Q.png" /></figure><h3>Common Causes of Software Failures and How Clear Requirements Can Help: A Sample Product Requirement Document for an Intelligent Tutoring System</h3><p><em>This post is brought to you by my enthusiasm for analyzing software systems. Have ideas you’d like to transform into a specification document? I’m available for new projects. Drop a message at </em><a href="/service/mailto:nasywa.ibtisamah@outlook.com"><em>nasywa.ibtisamah@outlook.com</em></a></p><p>Having worked for five years as a system analyst and re-skilling in the database engineering field, which provided a broader perspective on software systems design, I want to address one of the top issues in software development failure. While software development failures do happen, they’re often avoidable. It may not be hopeful to expect a 100% success rate (or even ideal), but it’s important to be aware of common project pitfalls to give your work the best chance of success.</p><p>For the case study, I find it interesting to reflect on AI in the context of the learning process as I have been practicing in the educational domain for several years. The study The Impact of an Intelligent Tutoring System (ITS) on the Academic Progress and Success of EFL Jordanian Students serves as a basis for creating a sample Product Requirement Document. Its findings regarding the effectiveness of ITS in the improvement of English language skills provide a foundation for designing a system that delivers similar learning outcomes. Using techniques of AI, these ITSs create a suitable learning environment tailored to the students’ needs [1].</p><p>I quoted Frederick Brooks, an American computer architect, software engineer, and computer scientist: “The hardest single part of building a software system is deciding what to build. No part of the work so cripples the resulting system if done wrong. No other part is more difficult to rectify later.” [2]</p><p>Brooks published a book on software engineering and project management in 1975. What ideas were presented? I categorize them into three groups: people-related, project management-related, and system-related.</p><p><strong>People-related:</strong></p><ol><li>The Mythical Man-Month: Adding manpower to a late software project only makes it later [2].</li><li>The Second-System Effect: When an architect (this is more about a ‘Business System Analyst’ than an ‘Architect’ as we would describe it today) designs a second system, it is the most dangerous system they will ever design. This is because they tend to incorporate all the additions they originally left out of the first system due to inherent time constraints [2].</li><li>Communication: To avoid disaster, all teams working on a project should remain in contact with each other through as many channels as possible (e-mail, phone, meetings, memos, etc.). Instead of making assumptions, implementers should ask the architect(s) to clarify their intent for a feature before proceeding, as an incorrect assumption could lead to significant issues [2].</li><li>The Surgical Team: It seems reasonable to have a highly skilled programmer develop critical system components while the rest of the team provides necessary support at the right time [2].</li></ol><p><strong>Project Management/Process-related:</strong></p><ol><li>Progress tracking: Incremental slippages on many fronts eventually accumulate, resulting in a large overall delay [2].</li><li>Formal documents: Every project manager should create a small core set of formal documents that define project objectives, how they are to be achieved, who will achieve them, when they will be achieved, and how much they will cost. These documents may also reveal inconsistencies that are otherwise difficult to detect [2].</li><li>Code freeze and system versioning: Software is intangible, so many things only become apparent once a certain amount of work has been done, allowing users to experience it. This experience often provides insights that may change the user’s needs or the perception of those needs. The system should, therefore, be adjusted to meet these changing requirements — but only up to a certain point. If changes continue indefinitely, the system may never be completed. At a certain point, no further changes should be allowed, and the code should be frozen. All requests for changes should be postponed until the next version of the system [2].</li><li>Project estimation: When estimating project timelines, remember that programming products (which can be sold to paying customers) and programming systems are both three times as difficult to write as simple, independent in-house programs [2].</li><li>The manual: The chief architect produces a manual of system specifications. It should describe the external specifications of the system in detail — that is, everything the user will see [2].</li></ol><p><strong>System/Technology-related:</strong></p><ol><li>No silver bullet: There is no single development, whether in technology or management technique, that can solve all problems [2].</li><li>Conceptual integrity: To create a user-friendly system, the system must have conceptual integrity, which can only be achieved by separating architecture (this is more about interface design than ‘architecture’ as we would describe it today) from implementation. A single chief architect (or a small group of architects) should decide what goes into the system and what stays out. The key point is that if a system is too complicated to use, many features will go unused because users don’t have the time to learn them [2].</li><li>Tendency toward irreducible errors: In a suitably complex system, there will be a certain irreducible number of errors that cannot be eliminated [2].</li><li>The pilot system: When designing a new type of system, the team will often create a throw-away system. This system serves as a “pilot plan,” revealing techniques that will later lead to a complete redesign of the final system [2].</li><li>Specialized tools: Each team should have a designated tool-maker who can create tools highly customized for the specific tasks the team is working on [2].</li></ol><h3>Is it still relevant in 2024?</h3><p>It depends on which aspect you are referring to. If you’re talking about the importance of communication, coordination, and collaboration, then yes, this book is still relevant.</p><p>However, if you’re referring to programmer productivity, in today’s environment, the widespread availability of online module libraries (such as Pandas for Python) can produce the kind of significant increase in productivity that Brooks felt was once out of reach.</p><p>To understand current trends and issues, I highlighted some key insights from the annual Project Management Institute study, Pulse of the Profession:</p><p><strong>2018 report — Success in Disruptive Times:</strong> This report emphasized the importance of managing projects effectively during disruptive times. Agile methods gained popularity, and the need for organizations to adapt quickly to change was highlighted [3].</p><p><strong>Key Insight:</strong> High-performing organizations prioritized agility, innovation, and continuous improvement. Project success was tied to balancing innovation with strong project management practices [3].</p><p><strong>2019 report — The Rise of PMTQ (Project Management Technology Quotient):</strong> This year introduced the concept of Project Management Technology Quotient (PMTQ), which is the ability of project managers to adapt, manage, and integrate technology within projects. The report revealed that organizations wasted nearly 12% of their project investment due to poor performance, which had barely improved over the past five years. The reasons for these failures included poor requirements management, inadequate communication, and misalignment between project goals and organizational strategy [4].</p><p><strong>Key Insight:</strong> As digital transformation accelerated, organizations needed project managers who could blend technical skills with traditional project management expertise. Around 80% of organizations had implemented disruptive technologies, but only 25% realized tangible benefits, signaling the need for better PMTQ [4].</p><p><strong>2020 report — Future-Focused Culture: </strong>This report highlighted the need for project managers to foster a future-oriented culture in organizations. Agility, adaptability, and empathy were seen as crucial skills in navigating the rapidly changing technological landscape. The report also emphasized that 53% of organizations prioritized building a culture that embraces change, but failure occurred when there was resistance to change or when organizations lacked agility [5].</p><p><strong>Key Insight:</strong> Successful organizations invested in digital skills and were agile in their approaches. Interpersonal skills, such as empathy, were emphasized alongside technical skills as critical to fostering strong project teams and achieving project success [5].</p><p><strong>2021 report — Beyond Agility: </strong>This report, published amid the pandemic, explored the concept of gymnastic enterprises — organizations that are flexible and able to pivot swiftly to meet new challenges [6].</p><p><strong>Key Insight:</strong> The pandemic reshaped project management, placing greater emphasis on adaptability, with hybrid and agile methodologies becoming standard practice. Organizations that combined flexibility with structured governance achieved the best project outcomes [6].</p><p><strong>2023 report — Power Skills:</strong> Redefining Project Success This report emphasized the growing importance of power skills (e.g., leadership, communication, problem-solving) in achieving project success. These skills help project managers engage stakeholders and lead teams effectively, particularly in evolving hybrid work environments [7].</p><p><strong>Key Insight:</strong> Organizations that invest in power skills see higher rates of project success, showcasing the need for project managers to balance both technical and interpersonal skills [7].</p><p><strong>2024 report — The Future of Project Work:</strong> Moving Past Office-Centric Models This report focuses on how organizations are adapting to new ways of working, including remote, hybrid, and on-site models. It highlights that flexibility in work arrangements does not negatively impact project performance [8].</p><p><strong>Key Insight: </strong>project success is more about choosing the right project management approach — whether predictive, agile, or hybrid — based on the specific needs of the project, rather than adhering to a single work location model [8].</p><p>These reports highlight how critical <strong>well-defined requirements, proper communication, and adaptability are to project success.</strong> Failing to address these factors contributes significantly to project failure.</p><p>Given my expertise, I will focus on creating a requirement document. Identifying requirements is the starting point for all software development projects, regardless of the development method being used. While changes in requirements are inevitable, we can handle them effectively if every change is well-documented, specifying why it’s being made and how it impacts the project. Clear documentation keeps the team aligned.</p><p>Let’s start with a basic question: What is a requirement? A requirement is something the product must do or a quality it must have. A requirement exists either because the type of product demands certain functions or qualities, or because the client wants it to be part of the delivered product [9].</p><p>I will use the Product Requirement Document format to outline the product overview, background, scope, and requirements of the IELTS online learning system.</p><p>Let’s roll up our sleeves and put it into practice!</p><h3>Business Requirement</h3><h3>1. Product Overview</h3><p>According to [10], Intelligent Tutoring Systems (ITS) can be defined as computer-based educational systems with independent databases or knowledge bases for educational content and teaching strategies. These systems aim to conclude the learner’s ability to understand topics and identify their weaknesses and strengths so they can dynamically adapt to the learning process.</p><p>These systems can determine the learning path, recommend suitable learning content, offer support and guidance to students, engage them in meaningful dialogues, and simulate one-to-one tutoring, among other functionalities [11]. Furthermore, ITSs can deliver personalized experiences tailored to different students, teachers, and tutors [12].</p><p><strong>For this sample Product Requirement Document (PRD), I will not deep-dive into the concept of ITS. The ITS will be implemented for the IELTS online learning system and will cover only specific features and not the entire system. It’s intended to provide a focused view on key functionalities while leaving other parts of the system out of scope.</strong></p><h3>2. Background</h3><p>According to [13], traditional education is unable to accommodate the varied learning styles and levels of preparation among students. In this approach, a single instructor is responsible for a large number of students, making it difficult to address each student’s individual needs. As a result, some students may feel dissatisfied and unable to achieve their academic objectives.</p><p>In conventional education, individualization is lost in large groups because a teacher can’t create a personalized study plan for each student. Moreover, traditional education offers fewer teaching methods. Tools like pictures, videos, and discussion boards are not always available, which may hinder learning and disrupt effective study [14].</p><p>According to [15], students who used intelligent tutoring systems performed better academically than those who relied solely on conventional education.</p><h3>3. Stakeholders</h3><p>I will focus on the student side as the main customers of ITS who will benefit from personalized and adaptive learning. We will also need the development team who is Responsible for the actual implementation and maintenance of the system.</p><p>Ideally, there are other stakeholders, such as:</p><ul><li>Teachers: Secondary users who can track the progress of students and upload supplementary material</li><li>Administrators: Interested in overall management of the system, such as preparation of contents and user management.</li><li>Parents: Progress reports so that they can see and follow the child’s progress.</li><li>Marketing Team: Responsible for promotions to educational institutions and potential users of ITS.</li></ul><p>However, we will exclude them in this sample PRD.</p><h3>4. Scope</h3><ul><li>Registration</li><li>Learning path: the system will create a personalized learning path based on the current score, targeted score and the duration inputted by the student.</li><li>Initial Test: This feature assesses the student’s current proficiency level in a subject, such as IELTS, before beginning any personalized lessons or practice. The results of this test will help the system understand the student’s strengths and weaknesses, allowing it to tailor the learning content and suggest a study plan accordingly.</li><li>Practice Test: it helps them practice the required skills for each section. Students receive instant feedback on their answers, helping them learn from mistakes in real-time. Detailed explanations and audio transcripts are available to reinforce learning. The test also includes tips on how to approach each question, making it easier for students to improve their test-taking strategies.</li><li>Report: It includes scores for different sections, highlights areas of strength and weakness, and provides personalized feedback to help students focus on improving. This feature is designed to help students reflect on their performance and plan their next steps in studying.</li><li>Timeline: The Timeline feature helps students organize their study plan based on their target exam date. It provides a visual schedule that includes important milestones, such as practice test dates and study sessions. The timeline adjusts based on the student’s progress, ensuring they stay on track. It helps students manage their time effectively and ensures they are preparing at the right pace to achieve their goals.</li></ul><h3>5. Requirement</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GUQb082t5AXn59NpCoYJwQ.png" /></figure><h3>6. Mockup</h3><p>Registration</p><p>Web-app:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WabOzdye_yUNqz0Zkv-jjQ.png" /></figure><p>Mobile-app:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/393/1*yTRM1pj4vg9hMKivcKsVGA.png" /></figure><p>Learning Path</p><p>Slide 1:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uL7nvzEeeBHeaJ6BtjCXaA.png" /></figure><p>Slide 2:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZSF7l_Et-zxIPOAGbc6WxQ.png" /></figure><p>Slide 3:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NR2KA-bQZ2OgPtACOmQK4Q.png" /></figure><p>Initial Test:</p><p>Question reference: <a href="https://ieltstutors.org/courses/guided-listening-test-2/modules/guided-listening-test-2/lessons/audio-guide-section-1/">https://ieltstutors.org/courses/guided-listening-test-2/modules/guided-listening-test-2/lessons/audio-guide-section-1/</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y_X_t8kD7J520TYaKei8Hg.png" /></figure><p>Practice Test:</p><p>Before the student submits the answer:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kKRi0447Lj-7OBnDc3tRtQ.png" /></figure><p>After the student submitted the answer:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M7yeeOIQAlgUrXoWHx6Gvw.png" /></figure><p>Report:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iscIIvh8Utmcew8n12IU8Q.png" /></figure><p>Timeline:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pfGgLbKM8HI5UPiYx5FCvg.png" /></figure><p>In conclusion, developing a clear requirement document is essential for the success of any system. By doing that, we ensure a targeted approach to meet learners’ needs.</p><p>Thank you for investing your time in reading this article! Stay connected for my upcoming articles :)</p><p><strong>Reference:</strong></p><ol><li>Peter Stone, Rodney Brooks, Erik Brynjolfsson, Ryan Calo, Oren Etzioni, Greg Hager, Julia Hirschberg, Shivaram Kalyanakrishnan, Ece Kamar, Sarit Kraus, Kevin Leyton-Brown, David Parkes, William Press, AnnaLee Saxenian, Julie Shah, Milind Tambe, and Astro Teller. “Artificial Intelligence and Life in 2030.” One Hundred Year Study on Artificial Intelligence: Report of the 2015–2016 Study Panel, Stanford University, Stanford, CA, September 2016. Doc: <a href="http://ai100.stanford.edu/2016-report">http://ai100.stanford.edu/2016-report</a>.</li><li>Frederick P. Brooks (1995). The Mythical Man-Month</li><li>Pulse of the Profession 2018 (2018)</li><li>Pulse of the Profession 2019 (2019)</li><li>PMI (2020). Ahead of the Curve: Forging a Future-Focused Culture. Pulse of the Profession.</li><li>PMI (2021). Beyond Agility. Pulse of the profession.</li><li>PMI (2023). Power skills, Redefining Project Success. Pulse of the profession</li><li>PMI (2024). The Future of Project Work: Moving Past Office-Centric Models</li><li>International Institute of Business Analysis (2009). A Guide to the Business Analysis Body of Knowledge Version 2.0</li><li>Alkhatlan, A., &amp; Kalita, J. Intelligent tutoring systems: A comprehensive historical survey with recent developments. arXiv preprint arXiv:.09628, 2018.</li><li>Zawacki-Richter, O., Marín, V. I., Bond, M., &amp; Gouverneur, F. (2019). Systematic review of research on artificial intelligence applications in higher education–where are the educators? International Journal of Educational Technology in Higher Education, 16(1), 1–27.</li><li>Churi, P. P., Joshi, S., Elhoseny, M., &amp; Omrane, A. (Eds.). (2022). Artificial intelligence in higher education: A practical approach (1st ed.). CRC Press.</li><li>Naser, S. S. A. Developing an intelligent tutoring system for students learning to program in C++. Information Technology Journal, Vol. 7, №7, 2008, pp. 1051–1060.</li><li>Chichekian, T., &amp; Benteux, B. The potential of learning with (and not from) artificial intelligence in education. Artificial Intelligence for Education, Vol. 5, 2022, doi:10.3389/frai.2022.903051</li><li>Karaci, A., Akyüz, H. I., Bilgici, G., &amp; Arici, N. (2018). Effects of Web-Based Intelligent Tutoring Systems on Academic Achievement and Retention. International Journal of Computer Applications, Vol. 181, №16, pp. 35–41.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=be7b9a6c601b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Preprocessing Steps for Machine Learning in Python (Part 2)]]></title>
            <link>https://medium.com/womenintechnology/data-preprocessing-steps-for-machine-learning-in-phyton-part-2-7cbf5856c757?source=rss-cbae1ad7c072------2</link>
            <guid isPermaLink="false">https://medium.com/p/7cbf5856c757</guid>
            <category><![CDATA[data-preprocessing]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[women-in-tech]]></category>
            <dc:creator><![CDATA[Learn with Nas]]></dc:creator>
            <pubDate>Thu, 05 Oct 2023 18:27:39 GMT</pubDate>
            <atom:updated>2023-10-30T04:24:06.497Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://learnwithnas.medium.com/data-preprocessing-steps-for-machine-learning-in-phyton-part-1-18009c6f1153">In the first part of this article,</a> we covered the data preprocessing process, demonstrating how to collect data, clean data including handling missing values, outliers, and duplicate data. Additionally, we presented data transformation techniques, utilizing diverse pandas functions like groupby and pivot_table. The fourth step encompassed a detailed exploration of Feature Engineering, specifically scaling, normalization, and standardization.</p><p>In this following segment, we will deeply explore feature selection, the management of imbalanced datasets, encoding features, and data splitting.</p><h3>Step 5: Feature Selection</h3><p>Feature selection involves the careful selection of optimal features for your model. While the specific technique may vary, the primary objective remains consistent: to identify the features that exert a greater influence on your model’s performance [12].</p><h4>Feature selection techniques in machine learning can be broadly categorized into the following groups:</h4><ol><li><strong>Supervised Techniques:</strong> These methods are applicable to labeled data and help identify pertinent features to enhance the performance of supervised models such as classification and regression. Examples include linear regression, decision trees, and SVM [13].</li><li><strong>Unsupervised Techniques:</strong> These methods are suitable for unlabeled data. Examples include K-Means Clustering, Principal Component Analysis, and Hierarchical Clustering [13].</li></ol><h4>Feature selection methods:</h4><ol><li><strong>Filter methods </strong>identify the inherent characteristics of features through univariate statistics rather than relying on cross-validation performance. They are quicker and less computationally intensive compared to wrapper methods. When working with high-dimensional data, filter methods are a more computationally efficient choice [13].</li><li><strong>Wrapper methods</strong> necessitate a mechanism to explore the entire array of potential feature subsets, evaluating their efficacy through training and assessing a classifier with that particular subset of features. This approach hinges on a specific machine learning algorithm tailored to the dataset at hand. It involves an exhaustive search, assessing all feasible feature combinations against the evaluation criterion. Typically, wrapper methods yield superior predictive accuracy compared to filter methods [13].</li><li><strong>Embedded methods</strong> combine the advantages of both wrapper and filter methods by considering feature interactions while still keeping computational costs manageable. These methods operate iteratively, managing each iteration of the model training process and selectively extracting features that significantly contribute to training for a given iteration [13].</li></ol><p>You can consult the diagram below for insights into each method’s technique:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IDxglXM97EPez-eIzljYZw.png" /></figure><p>While I won’t delve deeply into each technique here, you can explore a comprehensive article by Analytics Vidya that provides detailed descriptions of each technique: <a href="https://www.analyticsvidhya.com/blog/2020/10/feature-selection-techniques-in-machine-learning/">https://www.analyticsvidhya.com/blog/2020/10/feature-selection-techniques-in-machine-learning/</a></p><p><strong>Implementing Correlation Coefficient Technique to Detect Anomalous Data in Python:</strong></p><p>To detect the presence of anomalous or outlier data, the ‘pyod’ library will be used by importing the KNN algorithm. This algorithm can be used to identify data that is unusual or uncommon in a dataset.</p><pre>plt.figure(figsize=(20,15))<br>heatmap = sns.heatmap(df_final.corr(), annot=True)<br>heatmap.set_title(&#39;Correlation Heatmap&#39;, fontdict={&#39;fontsize&#39;:12});</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ybIrmmwomIe2Egpg-r1D-w.png" /></figure><p><strong>Findings</strong></p><p>The correlation coefficient between the features sum_active_power and sum_reactive_power is very high (0.96). I will remove sum_reactive_power from the model to avoid overfitting.</p><pre># drop sum_reactive_power column<br>df_final = df_final.drop(&#39;sum_reactive_power&#39;, axis=1)</pre><p>The target variable is ‘end_temp’. In the next process, I will evaluate the correlation between features and the target.</p><pre>df_final.corr()[&#39;end_temp&#39;].sort_values()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/426/1*KVLMlnwauqpMMinKp8od0g.png" /></figure><p><strong>Findings</strong></p><p>1. From the correlation coefficient values above, out of 25 columns, it can be observed that the average correlation values are below 0.1 and also above -0.1. This indicates that the relationship between the target and other features is relatively weak.</p><p>2. Due to this condition, only features with correlation values greater than 0.1 and less than -0.1 will be used.</p><p><strong>Action to do</strong></p><p>Use the features which have a correlation greater than 0.1 and less than -0.1 to check anomalous data.</p><pre>df_final.corr()[&#39;end_temp&#39;][df_final.corr()[&#39;end_temp&#39;]&gt;0.1]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/418/1*EOsRNo2t6wTIszdW4yN3Qw.png" /></figure><pre>df_final.corr()[&#39;end_temp&#39;][df_final.corr()[&#39;end_temp&#39;]&lt;-0.1]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/410/1*wWRYjKvPB2EDcjN9HRgOvw.png" /></figure><pre>outliers = df_final[[&#39;sum_active_power&#39;,<br>                     &#39;start_temp&#39;,<br>                     &#39;end_temp&#39;,<br>                     &#39;bulk_12&#39;,<br>                     &#39;bulk_6&#39;,<br>                     &#39;wire_2&#39;,<br>                     &#39;wire_4&#39;,<br>                     &#39;wire_7&#39;]]</pre><pre>model = KNN()<br>model.fit(outliers)<br>outliers[&#39;is_outlier&#39;] =  model.predict(outliers) == 1<br>outliers_knn = outliers[&#39;is_outlier&#39;].sum()<br>print(&quot;Number of Anomalies: &quot;, outliers_knn)</pre><p>Result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/392/1*94R2GqwRwU6DJ0MGo5qUVg.png" /></figure><p>These anomalous data will be dropped from the dataset.</p><pre>outlier_keys = list(outliers[outliers[&#39;is_outlier&#39;] == 1].index)<br>good_keys = list(set(outliers.index) - set(outlier_keys))</pre><pre>df_final = df_final.drop(outlier_keys)<br>df_final.shape</pre><p><strong>You can visit my GitHub account to access the complete code related to the above example:</strong></p><p><a href="https://github.com/nasywa-ibtisamah/Optimizing_Steel_Processing_Efficiency-Predicting_Temperature_to_Reduce_Electricity_Consumption">GitHub - nasywa-ibtisamah/Optimizing_Steel_Processing_Efficiency-Predicting_Temperature_to_Reduce_Electricity_Consumption</a></p><h3>Step 6: Handling Imbalanced Data</h3><p>Imbalanced data denotes datasets in which the distribution of observations within the target class is uneven. In other words, one class label has a significantly higher number of observations, while the other has a notably lower number [14].</p><h4>Some approaches to handling imbalanced data set problem:</h4><ol><li><strong>Choose Proper Evaluation Metric:</strong> An initial consideration is the nature of the challenge you aim to address in your Machine Learning endeavor. Does it involve classifying data, predicting distinct labels like “spam” or “not spam”? Alternatively, is it a regression task, entailing predictions of continuous values like housing prices or customer ratings? Your choice of evaluation metrics, encompassing accuracy, precision, recall, or prediction error, will depend on the specific problem type [15].</li><li><strong>Resampling (Oversampling or Undersampling)</strong> involves adjusting the sample size of either the minority or majority class, either increasing or decreasing it. In the case of an imbalanced dataset, we can use oversampling to increase the representation of the minority class with replacement. Conversely, undersampling entails randomly removing rows from the majority class to align its size with that of the minority class. Following this sampling process, we obtain a well-balanced dataset with comparable sizes for both major and minor classes. This balance ensures that the classifier assigns equal significance to both classes when they have a similar number of records in the dataset [14].</li><li><strong>SMOTE (Synthetic Minority Oversampling Technique)</strong> is an alternative approach for oversampling the minority class. Mere duplication of minority class records doesn’t usually bring new insights to the model. In SMOTE, new instances are created by synthesizing data from the existing records. To put it plainly, SMOTE examines instances in the minority class, selects a random nearest neighbor using k-nearest neighbor, and generates a synthetic instance randomly within the feature space [14].</li></ol><p><strong>Implementing Upsampling and Downsampling in Python:</strong></p><pre># check data composition of target column in percentage<br>df[&#39;exited&#39;].value_counts()/df.shape[0] * 100</pre><p>Result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/1*sKW4XSKAti-S2imdTXZ4rg.png" /></figure><p>The composition of the target data is not ideal due to an imbalance. This implies that, since the majority of the target data is 1, there is a tendency for the model to predict the value 1. This can result in poor model performance and low accuracy. To address this imbalance, techniques like upsampling (increasing the frequency of value 0) or downsampling (reducing the frequency of value 1) can be employed. However, both upsampling and downsampling might lead to the introduction of synthetic data points.</p><p><strong>Upsampling:</strong></p><pre># function to upsample<br>def upsample (features, target, repeat):<br>    features_zeros = features[target == 0]<br>    features_ones = features[target == 1]<br>    target_zeros = target[target == 0]<br>    target_ones = target[target == 1]<br>    <br>    features_upsampled = pd.concat([features_zeros] + [features_ones] * repeat)<br>    target_upsampled = pd.concat([target_zeros] + [target_ones] * repeat)<br>    <br>    features_upsampled, target_upsampled = shuffle (features_upsampled, target_upsampled, random_state = 42)<br>    <br>    return features_upsampled, target_upsampled</pre><p>Before upsampling:</p><pre># check data composition<br>target_train.value_counts()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/362/1*9WSx9Nnl1aEWh8KekAlelw.png" /></figure><pre># apply the function<br>features_upsampled, target_upsampled = upsample(features_train, target_train, 4)</pre><p>After upsampling:</p><pre># data composition after upsampling<br>target_upsampled.value_counts()</pre><p>Result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/398/1*d8y86cnUb4rHtUY-7fp3VQ.png" /></figure><p><strong>Downsampling:</strong></p><pre># function to downsample<br>def downsample (features, target, fraction):<br>    features_zeros = features[target == 0]<br>    features_ones = features[target == 1]<br>    target_zeros = target[target == 0]<br>    target_ones = target[target == 1]<br>    <br>    features_downsampled = pd.concat([features_zeros.sample(frac=fraction, random_state = 42)] + [features_ones])<br>    target_downsampled = pd.concat([target_zeros.sample(frac=fraction, random_state = 42)] + [target_ones])<br>    <br>    features_downsampled, target_downsampled = shuffle (features_downsampled, target_downsampled, random_state = 42)<br>    <br>    return features_downsampled, target_downsampled</pre><p>Before downsampling:</p><pre># check data composition<br>target_train.value_counts()</pre><p>Result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/354/1*yZw_T37Yq2Q66p2rKKbmBA.png" /></figure><pre># apply the function<br>features_downsampled, target_downsampled = downsample(features_train, target_train, 0.3)</pre><p>After upsampling:</p><pre># check data composition after upsampling<br>target_downsampled.value_counts()</pre><p>Result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*kAyCNkygoVHeFZYKyAjjEg.png" /></figure><p><strong>You can visit my GitHub account to access the complete code related to the above example:</strong></p><p><a href="https://github.com/nasywa-ibtisamah/Predicting-Customer-Churn-for-Cost-Effective-Retention-in-A-Bank-using-F1-Score-and-AUC-ROC">GitHub - nasywa-ibtisamah/Predicting-Customer-Churn-for-Cost-Effective-Retention-in-A-Bank-using-F1-Score-and-AUC-ROC</a></p><h3><strong>Step 7: Encoding Categorical Features</strong></h3><p>Encoding categorical features involves transforming categorical data into integer format, enabling its utilization in various models. Categorical data typically exists in the form of strings or object data types. However, machine learning algorithms exclusively operate on numerical values. Hence, it’s crucial to convert categorical features or data into a numeric representation [16].</p><h4>How to Encode Categorical Features</h4><p><strong>Ordinal Encoding</strong></p><p>When converting ordinal data to numeric form, it’s crucial to preserve the inherent order. In this process, each category is assigned a numerical value ranging from 0 to the total number of categories. For instance, if we have three categories like ‘bad,’ ‘average,’ and ‘good,’ they would be encoded as 0, 1, and 2 respectively, ensuring the order is preserved [16].</p><p><strong>Nominal Encoding</strong></p><p>Nominal data lacks any inherent order. Encoding categories in an ordinal manner (e.g., using numbers like 0, 1, 2) can mislead machine learning algorithms into assigning unwarranted importance (e.g., assigning higher importance to 2 over 0 and 1) when, in fact, the data is nominal and lacks such inherent order. Consequently, this approach is not suitable for nominal data. For nominal data, a better approach involves transforming each category into a distinct column and assigning binary values (0 or 1) based on the presence of the respective category. Consequently, this transformation increases the number of features, with each category represented as a separate column [16].</p><p><strong>Implementing Nominal Encoding in Pyton:</strong></p><p>the dataset:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/758/1*Fl0Q5CCi2Ru92qFX81XD7Q.png" /></figure><pre># define nominal features<br>df_categorical = [&#39;geography&#39;, &#39;gender&#39;]</pre><p>We will implement get_dummies from the Pandas Library for nominal encoding, specifically a technique called one-hot encoding. One-hot encoding is a common method to convert categorical data (including nominal data) into a numerical format suitable for machine learning [17].</p><pre>df = pd.get_dummies(df, drop_first=True, columns = df_categorical)</pre><p>The output shows how each of the categories is transformed into a column.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/1*UfFIS8oMoWihi__YuviksA.png" /></figure><p><strong>You can visit my GitHub account to access the complete code related to the above example:</strong></p><p><a href="https://github.com/nasywa-ibtisamah/Predicting-Customer-Churn-for-Cost-Effective-Retention-in-A-Bank-using-F1-Score-and-AUC-ROC">GitHub - nasywa-ibtisamah/Predicting-Customer-Churn-for-Cost-Effective-Retention-in-A-Bank-using-F1-Score-and-AUC-ROC</a></p><h3>Step 8: Data Splitting</h3><p>In the realm of data science and machine learning, data splitting is a crucial step. It involves partitioning the given dataset into two or more subsets and facilitating model training, validation, and testing. This process is fundamental, especially when building models reliant on the dataset. Typically, we divide the main dataset into two or three parts to achieve this [18].</p><h4>How does Data Splitting work?</h4><p>When engaged in supervised machine learning tasks, it’s advisable to partition the data into three distinct sets: the training set, the testing set, and the validation set. But what exactly does this entail?</p><ol><li><strong>Training Set:</strong> This subset of the main dataset is utilized to educate the model, enabling it to grasp patterns and relationships within the data for accurate predictions. When selecting training data from the entire dataset, it’s essential to prioritize high data representativeness, ensuring an adequate representation for each class. Additionally, the quality of the extracted data should encompass impartiality, as biased data could compromise the accuracy of the model [18].</li><li><strong>Validation Set: </strong>This set is used to understand the performance of the model in comparison to different models and hyperparameter choices. During the process of constructing a machine learning model, our approach typically involves training multiple models by adjusting parameters or employing various algorithms. For instance, while developing a decision tree model for our dataset, we engaged in hyperparameter tuning, revealing multiple well-performing models under varying conditions. Consequently, the task at hand is to select the optimal model by considering different parameters. A critical observation is that employing the same data for both training and model tuning can result in overfitting, rendering the model incapable of generalization. This is where the validation set plays a crucial role, serving as independent and unbiased data. It facilitates a fair comparison of model performance, aiding in the selection of the best model algorithm or parameters. Once this data aids in identifying the most promising model and parameter combinations, we proceed to put the model into production, estimating its performance. However, it’s advisable not to employ the test data for evaluating the model before finalizing the optimal choice [18].</li><li><strong>Test set: </strong>As mentioned earlier, following the steps of training, validation, and model selection, the next phase involves putting the selected model into production after evaluating its performance on a specific subset of data known as the test set. Exercise caution during this phase, as premature execution can result in overfitting, yielding an unreliable model performance. The test set should be employed as the ultimate evaluation stage, once validation set usage is concluded, and the final model is chosen [18].</li></ol><p><strong>Implementing Data Splitting in Python:</strong></p><pre># Split dataset<br>df_train_valid, df_test = train_test_split(df, test_size=0.2)<br><br># Apply `random_state 42` to obtain the same results across different executions.<br>df_train, df_valid = train_test_split(df_train_valid, test_size=0.25)# untuk membuat model<br><br># features and target for training dataset<br>features_train = df_train.drop(&#39;is_ultra&#39;, axis=1)<br>target_train = df_train[&#39;is_ultra&#39;]<br><br># features and target for validation dataset<br>features_valid = df_valid.drop(&#39;is_ultra&#39;, axis=1)<br>target_valid = df_valid[&#39;is_ultra&#39;]<br><br># features and target for test dataset<br>features_test = df_test.drop(&#39;is_ultra&#39;, axis=1)<br>target_test = df_test[&#39;is_ultra&#39;]</pre><p>Result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/305/1*63wfUQtRmhhR5473uRFjfQ.png" /></figure><p>After completing the data preprocessing steps, you’re prepared to commence model training. It’s important to emphasize that data preprocessing encompasses various tasks, including data formatting and feature creation, tailored to the specifics of your AI project.</p><p>Thank you for investing your time in reading this article! Stay connected for more enriching content in my forthcoming articles :)</p><p><a href="https://www.linkedin.com/in/nasywa/"><strong>Connect with me on Linkedin</strong></a></p><p><strong>References:</strong></p><p>13. Aman Gupta, Feature Selection Techniques in Machine Learning (2023)</p><p><a href="https://www.analyticsvidhya.com/blog/2020/10/feature-selection-techniques-in-machine-learning/">Feature Selection Techniques in Machine Learning</a></p><p>14. Saikat Mazumder, 5 Techniques to Handle Imbalanced Data For a Classification Problem (2023)</p><p><a href="https://www.analyticsvidhya.com/blog/2021/06/5-techniques-to-handle-imbalanced-data-for-a-classification-problem/">5 Techniques to Handle Imbalanced Data For a Classification Problem</a></p><p>15. Linkedin, What’s the best way to pick an evaluation metric for your Machine Learning project?</p><p><a href="https://www.linkedin.com/advice/0/whats-best-way-pick-evaluation-metric-your-machine">What&#39;s the best way to pick an evaluation metric for your Machine Learning project?</a></p><p>16. Gowtham S R, Encoding Categorical Data- The Right Way (2022)</p><p><a href="https://towardsai.net/p/l/encoding-categorical-data-the-right-way#:~:text=Encoding%20categorical%20data%20is%20a,can%20work%20only%20on%20numbers">https://towardsai.net/p/l/encoding-categorical-data-the-right-way#:~:text=Encoding%20categorical%20data%20is%20a,can%20work%20only%20on%20numbers</a>.</p><p>17. Pandas Documentation</p><p><a href="https://pandas.pydata.org/docs/reference/api/pandas.get_dummies.html">pandas.get_dummies - pandas 2.1.1 documentation</a></p><p>18. Data Science Wizards, A Guide to Data Splitting in Machine Learning (2022)</p><p><a href="https://medium.com/@datasciencewizards/a-guide-to-data-splitting-in-machine-learning-49a959c95fa1#:~:text=What%20is%20Data%20Splitting%3F,get%20trained%2C%20tested%20and%20evaluated">https://medium.com/@datasciencewizards/a-guide-to-data-splitting-in-machine-learning-49a959c95fa1#:~:text=What%20is%20Data%20Splitting%3F,get%20trained%2C%20tested%20and%20evaluated</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7cbf5856c757" width="1" height="1" alt=""><hr><p><a href="https://medium.com/womenintechnology/data-preprocessing-steps-for-machine-learning-in-phyton-part-2-7cbf5856c757">Data Preprocessing Steps for Machine Learning in Python (Part 2)</a> was originally published in <a href="https://medium.com/womenintechnology">Women in Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Preprocessing Steps for Machine Learning in Python (Part 1)]]></title>
            <link>https://medium.com/womenintechnology/data-preprocessing-steps-for-machine-learning-in-phyton-part-1-18009c6f1153?source=rss-cbae1ad7c072------2</link>
            <guid isPermaLink="false">https://medium.com/p/18009c6f1153</guid>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[data-preprocessing]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Learn with Nas]]></dc:creator>
            <pubDate>Sat, 30 Sep 2023 05:28:13 GMT</pubDate>
            <atom:updated>2023-10-29T22:00:09.098Z</atom:updated>
            <content:encoded><![CDATA[<p>Data Preprocessing, also recognized as Data Preparation or Data Cleaning, encompasses the practice of identifying and rectifying erroneous or misleading records within a dataset. This involves pinpointing flawed, incomplete, or irrelevant segments of the data and subsequently modifying, substituting, or eliminating this impure or coarse data [1]. Data Preprocessing techniques have been adapted to train AI models, including machine learning models. The techniques are generally used at the early stages to ensure accurate results [2]. Please be aware that data preprocessing is a comprehensive term covering a wide range of tasks, spanning from formatting the data to creating features, all depending on the nature of your AI project.</p><p>This preparatory phase not only enhances the overall quality of the data but also streamlines the modelling process, ultimately leading to more reliable and accurate predictive models. This article delves into the vital role that Data Preprocessing plays in the context of Machine Learning, shedding light on its various aspects and emphasizing its necessity for achieving meaningful and impactful results.</p><h3>Why is it important?</h3><p>The significance of Data Preprocessing in Machine Learning cannot be overstated, as it forms the cornerstone of any successful data analysis or machine learning endeavour. In the realm of data-driven technologies, the quality and suitability of data directly influence the outcomes and effectiveness of machine learning models.</p><h3>Data Preprocessing involves a series of steps such as:</h3><ol><li>Data Collection</li><li>Data Cleaning</li><li>Data Transformation</li><li>Feature Engineering: Scaling, Normalization and Standardization</li><li>Feature Selection</li><li>Handling Imbalanced Data</li><li>Encoding Categorical Features</li><li>Data Splitting</li></ol><h4>Step 1: Data Collection</h4><p>The cornerstone of machine learning is rooted in data. Collecting data involves gathering information aligned with the goals and objectives of your AI project. If you feed subpar or low-quality data into your model, it will not produce satisfactory outcomes. This holds true regardless of the model’s complexity, the expertise of the data scientist, or the financial investment in the project [3].</p><p>While some companies have been accumulating data for years, ensuring a steady supply for machine learning, those lacking sufficient data can turn to reference datasets available online to complete their AI projects. Discovering new data and sharing it can be achieved through three methods: collaborative analysis (DataHub), web (Google Fusion Tables, CKAN, Quandl, and Data Market), and a combination of collaboration and web use (Kaggle). Additionally, there are specialized data retrieval systems, including data lakes (Google Data Search) and web-based platforms (WebTables) [3].</p><p>Suppose we have all the necessary data; we can proceed with creating a dataset.</p><pre># import library<br>import pandas as pd<br>import matplotlib.pyplot as plt<br>import seaborn as sns</pre><pre># load the data<br>df = pd.read_csv(&#39;data/credit_scoring_eng.csv&#39;)</pre><pre>df.head(5)</pre><p>Result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1018/1*CRbgg4XyhexVi8Kqdfrm-g.png" /></figure><p>Data Description</p><ul><li>children - number of children in the family</li><li>days_employed - number of days employed</li><li>dob_years - client&#39;s age in years</li><li>education - client&#39;s education level</li><li>education_id - education identifier</li><li>family_status - marital status</li><li>family_status_id - marital status identifier</li><li>gender - client&#39;s gender</li><li>income_type - type of employment</li><li>debt - whether the client has a loan debt</li><li>total_income - monthly income</li><li>purpose - purpose of the loan application</li></ul><h4>Step 2: Data Cleaning</h4><p>This involves identifying and correcting errors or inconsistencies in the data, such as missing values, outliers and duplicates. Various techniques can be used for data cleaning, such as imputation, removal or transformation [4].</p><p><strong>Implementing in phyton:</strong></p><p>2-a: Handling missing values</p><pre># check the dataset information<br>df.info()</pre><p>Result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/706/1*bzB0xcpNRTvxboXATGC_PA.png" /></figure><p><strong>Findings:</strong></p><p>There are missing values in the columns days_employed and total_income, because the number of rows should be 21,525</p><pre># check the percentage<br>df.isna().sum() / len(df)</pre><p>Result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/448/1*SB3LvAXGwt2DSruiXMh-Vg.png" /></figure><p><strong>Findings:</strong></p><p>The missing value percentage for both columns are around 10%</p><pre># Visualizing Missing Data using a seaborn heatmap.<br>plt.figure(figsize=(10,6))<br>sns.heatmap(df.isna().transpose(),<br>            cmap=&quot;YlGnBu&quot;,<br>            cbar_kws={&#39;label&#39;: &#39;Missing Data&#39;})</pre><p>Result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S_g_8R2UzWvkqQxTO3s8tA.png" /></figure><p><strong>Findings:</strong></p><p>1. Missing values form a pattern. The missing values are caused by job types where clients with the job types ‘student’ and ‘unemployed’ do not have any income, leading them to leave the ‘days_employed’ and ‘total_income’ columns empty.</p><p>2. This conclusion is reinforced by the pattern shown in the seaborn heatmap, indicating that when the value in the ‘days_employed’ column is missing, the data in the same row for ‘total_income’ is also missing (symmetrical).</p><p>3. Since the missing values are only present in the ‘days_employed’ and ‘total_income’ columns, and both of these columns have float data types, which fall under the Numeric/Ratio category, the missing data will be filled using statistical calculations (such as Mean, Median).</p><p>4. Median is chosen to fill in missing values because it can prevent the occurrence of outliers [5]</p><pre>#  function to fill in missing values using median<br>def data_imputation(data, column_grouping, column_selected):<br>    # Parameter meaning<br>    # data =&gt; The name of the dataframe to be processed<br>    # column_grouping =&gt; The column used to group values and take the median<br>    # column_selected =&gt; The column in which we will fill its NaN values<br><br>    # Get unique category groups<br>    group = data[column_grouping].unique()<br><br>    # Loop through each value in the group category<br>    for value in group:<br>        # get median<br>        median = data.loc[(data[column_grouping]==value) &amp; ~(data[column_selected].isna()), column_selected].median()<br><br>        # change missing value<br>        data.loc[(data[column_grouping]==value) &amp; (data[column_selected].isna()), column_selected] = median<br><br>    # Return the dataframe after filling the missing values<br>    return data</pre><pre># apply the function to &#39;total_income&#39; column<br>df = data_imputation(data=df, column_grouping=&#39;age_category&#39;, column_selected=&#39;total_income&#39;)<br></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/492/1*F4byTMI71dg6IEzx72wJuA.png" /></figure><pre># apply the function to &#39;days_employed&#39; column<br>df = data_imputation(data=df, column_grouping=&#39;age_category&#39;, column_selected=&#39;days_employed&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/1*9Fcg0rn4soWKfOoku-nqaQ.png" /></figure><pre># check the statistical<br>df.info()</pre><p>Result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/1*NLfsboGrUucRk2zeSHMOHg.png" /></figure><p>2-b: Handling outliers</p><pre># check outlier in children column<br>sns.boxplot(df[&#39;children&#39;])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*8vM0npuaKs9JeqC9XzB44g.png" /></figure><pre># check statistical data in children column<br>df[&#39;children&#39;].describe()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/420/1*CzqS-2HlWyS4SGtk6VrOTg.png" /></figure><p><strong>Findings:</strong></p><p>1. Based on the statistical data above, I will replace the value 20 with the value 2, assuming it was an input error.</p><p>2. I will remove the minus sign (-), assuming it was an input error</p><pre># replace the value 20 with the value 2<br>condition_children = df[&#39;children&#39;]==20<br>df[&#39;children&#39;] = df[&#39;children&#39;].mask(condition_children, 2)</pre><pre># remove minus sign<br>df[&#39;children&#39;] = abs(df[&#39;children&#39;])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/476/1*fjzau2eLJTml005bRqMG1g.png" /></figure><pre># check outliers in days_employed column<br>sns.boxplot(df[&#39;days_employed&#39;])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*LwvlaP4-L5GUxOPHN90l8Q.png" /></figure><pre># check percentage<br>len(df.loc[(df[&#39;days_employed&#39;] &lt; 0 ) | (df[&#39;days_employed&#39;] &gt; 200000)]) / len(df)</pre><p>Result:</p><p>0.8990011614401858</p><p><strong>Findings:</strong></p><ol><li>There are 2 issues identified in the ‘days_employed’ column:</li></ol><ul><li>Too many digits after the decimal point.</li><li>Existence of negative values and outliers, with a high percentage of rows having these conditions, approximately 89%.</li></ul><p>2. The steps to solve these issues are as follows:</p><ul><li>Remove the minus sign (-).</li><li>Perform rounding.</li><li>Replace the outlier values.</li></ul><pre># remove minus sign (-), assuming it was an input error<br>df[&#39;days_employed&#39;] = abs(df[&#39;days_employed&#39;])</pre><pre># round <br>df[&#39;days_employed&#39;] = round(df[&#39;days_employed&#39;],0)</pre><pre># check data distribution<br>df[&#39;days_employed&#39;].describe()</pre><p>Result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/462/1*LgYlz2ycJXrCGsthBnvtOg.png" /></figure><p><strong>Findings:</strong></p><p>The mean value does not represent the data as it is mixed with outliers. Therefore, the replacement of outliers will be done using the median value.</p><pre># Replace outlier with median<br>condition_de = (df[&#39;days_employed&#39;] &gt; 200000) &amp; (df[&#39;days_employed&#39;].notnull())<br>df[&#39;days_employed&#39;] = df[&#39;days_employed&#39;].mask(condition_de, df[&#39;days_employed&#39;].median())</pre><pre># verify the result<br>sns.boxplot(df[&#39;days_employed&#39;])</pre><p>Result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/1*_JLZinWfD9OZyN0OeNDorw.png" /></figure><p>2-c: Handling Duplicates</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/426/1*-ZoT8l6WDSnbFTUJRfAMqA.png" /></figure><p><strong>Findings</strong></p><p>1. There are 72 identified duplicate data entries.</p><p>2. These duplicate data entries will be removed, and the index will be reset.</p><pre># remove duplicate data and do reset index<br>df = df.drop_duplicates().reset_index(drop=True)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/356/1*hK7sJ3GxDRfZb_-uNyuIvQ.png" /></figure><p><strong>You can visit my GitHub account to access the complete code related to the above example:</strong></p><p><a href="https://github.com/nasywa-ibtisamah/Analyzing-Borrower-Default-Risk-and-Its-Correlation-with-Marital-Status-and-Children-for-Bank-Credit">GitHub - nasywa-ibtisamah/Analyzing-Borrower-Default-Risk-and-Its-Correlation-with-Marital-Status-and-Children-for-Bank-Credit</a></p><h4>Step 3: Data Transformation</h4><p>Data transformation involves technically converting data from one format, standard, or structure to another, without changing the dataset’s content. This is typically done to prepare the data for consumption by an application or user, or to enhance data quality. The specifics of data transformation can vary based on the techniques employed. In this article, I will utilize a data aggregation technique for data transformation [6].</p><p>Data aggregation is a method used to present data in a summarized form. Given the likelihood of data originating from diverse sources, combining all incoming data into a cohesive description is the essence of data aggregation. This facet of data processing holds significance as it hinges on the quality and quantity of the data at hand. An illustrative example of this process is generating an annual sales report by consolidating quarterly or monthly data [7].</p><p>There are many ways to aggregate data in Pandas, including:</p><p><strong>a. Utilizing the groupby() function: </strong>Grouping involves breaking down a dataset into smaller subsets depending on specific variables. This approach is widely employed for data exploration and analysis purposes. The pandas’ groupby() function is highly versatile and allows for the grouping of data based on one or more columns. By using the groupby() function, we can group data according to selected variables and subsequently apply a range of aggregation functions to these groups [8].</p><p><strong>Implementing in phyton:</strong></p><p>We will use groupby() to analyze whether the user review and professional review will influence platform sales</p><pre># preparing the dataset<br>top2_ref_df = reference_df.groupby([&#39;platform&#39;, &#39;name&#39;])[[&#39;total_sales&#39;, &#39;critic_score&#39;, &#39;user_score&#39;]].sum().query(&#39;platform == &quot;PS3&quot; &amp; critic_score &gt; 0 &amp; user_score &gt; 0&#39;).reset_index()<br>top2_ref_df = top2_ref_df[[&#39;name&#39;, &#39;platform&#39;, &#39;total_sales&#39;, &#39;critic_score&#39;, &#39;user_score&#39;]]<br>top2_ref_df</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/1*XVGUdmV0wgLw6dUoUNPQYQ.png" /></figure><p><strong>b. using pivot_table() function:</strong> We’ve explored how the GroupBy concept enables us to investigate connections within a dataset. A pivot table is a similar operation commonly encountered in spreadsheet software and other programs working with tabular data. When using a pivot table, input data in a column-wise format is organized into a two-dimensional table, offering a multidimensional summary of the information. Distinguishing between pivot tables and GroupBy can be confusing at times. It’s helpful to consider pivot tables as essentially a multidimensional form of GroupBy aggregation. In other words, you perform the split-apply-combine process, but in this case, both the splitting and combining occur not along a one-dimensional index, but across a two-dimensional grid [9].</p><p><strong>Implementing in phyton:</strong></p><p>In order to analyze top 5 platforms in NA, EU and JP regions and visualize the variation in market share from one region to another</p><pre># data aggregation of sales for each platform in NA, EU, JP regions<br>agg_selected_region_platform = pd.pivot_table(data=reference_df, index=&#39;platform&#39;, values = [&#39;na_sales&#39;, &#39;eu_sales&#39;, &#39;jp_sales&#39;], aggfunc = &#39;sum&#39;).sort_values(by=&#39;jp_sales&#39;)<br>agg_selected_region_platform</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/240/1*uvZGFLb2JlQZ3Q1sn2fU1A.png" /></figure><p>We can visualize the data:</p><pre># visualizing the sales for each platform in NA, EU, JP regions<br>plt.figure(figsize=(20,6))<br>plt.title(&#39;Distribution of game sales on each platform in the EU, NA, and JP regions&#39;)<br>sns.lineplot(data=agg_selected_region_platform)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p3JImukZ3f6rqRY22_9C1w.png" /></figure><p><strong>You can visit my GitHub account to access the complete code related to the above example:</strong></p><p><a href="https://github.com/nasywa-ibtisamah/Identifying-Patterns-in-Games-User-Profiling-for-Platform-and-Genre-Insights-Review-Impact-Analysis">GitHub - nasywa-ibtisamah/Identifying-Patterns-in-Games-User-Profiling-for-Platform-and-Genre-Insights-Review-Impact-Analysis</a></p><h3>Step 4: Feature Engineering: Scaling, Normalization and Standardization</h3><p>Feature engineering constitutes a pivotal stage in the creation of accurate and efficient machine learning models. A significant facet of feature engineering involves scaling, normalization, and standardization, encompassing the alteration of data to enhance its suitability for modeling. Employing these methods can enhance model accuracy, mitigate the influence of outliers, and ensure uniformity in data scale. This article delves into the fundamentals of scaling, normalization, and standardization [10].</p><h4>Feature Scaling</h4><p>Feature scaling is a crucial step in data preprocessing, aiming to standardize the values of features or variables within a dataset to a uniform scale. The primary objective is to ensure that all features have a fair influence on the model, avoiding the dominance of features with higher values. The necessity for feature scaling arises when working with datasets that encompass features having diverse ranges, units of measurement, or orders of magnitude. In such scenarios, discrepancies in feature values can introduce bias in model performance or hinder the learning process. Through the application of feature scaling, the features in a dataset can be harmonized to a consistent scale, simplifying the construction of precise and efficient machine learning models. Scaling promotes meaningful feature comparisons, enhances model convergence, and prevents specific features from dominating others solely based on their magnitude [10].</p><h4>Why Should We Use Feature Scaling?</h4><p>Certain machine learning algorithms exhibit sensitivity to feature scaling, whereas others remain mostly unaffected by it. Let’s delve into a detailed examination of this aspect.</p><ol><li><strong>Gradient Descent Based Algorithms</strong></li></ol><p>Machine learning algorithms that use gradient descent as an optimization technique (like linear regression, logistic regression, etc) require data to be scaled [10]</p><p><strong>2. Distance-Based Algorithms</strong></p><p>Algorithms based on distance metrics, such as K-nearest neighbors (KNN), K-means clustering, and support vector machines (SVM), are highly influenced by the range of features. This is because these algorithms rely on calculating distances between data points to ascertain their similarity [10].</p><p><strong>Implementing in Phyton:</strong></p><p>A function will be created to calculate the distance using the k-nearest neighbors algorithm based on two distance metrics: Euclidean and Manhattan. We will then compare the distance results on both unscaled and scaled data.</p><p>the df dataset:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/810/1*YQqZGUV9xlSsOp-2nYEmVw.png" /></figure><pre># function for calculating kNN distance<br>def get_knn(df, n, k, metric):<br>    <br>    &quot;&quot;&quot;<br>    Display k nearest neighbors:<br>    param df: Pandas DataFrame used to find similar objects within it<br>    param n: number of the object for which k nearest neighbors are sought<br>    param k: number of k nearest neighbors to be displayed<br>    param metric: name of the distance metric<br>    &quot;&quot;&quot;<br><br>    nbrs = sklearn.neighbors.NearestNeighbors(n_neighbors = k, metric = metric, algorithm = &#39;brute&#39;)<br>    nbrs.fit(df[feature_names])<br>    nbrs_distances, nbrs_indices = nbrs.kneighbors([df.iloc[n][feature_names]], k, return_distance=True)<br>    <br>    df_res = pd.concat([<br>        df.iloc[nbrs_indices[0]], <br>        pd.DataFrame(nbrs_distances.T, index=nbrs_indices[0], columns=[&#39;distance&#39;])<br>        ], axis=1)<br>    <br>    return df_res<br></pre><p><strong>Using unscaled data (df):</strong></p><pre># euclidean metric - unscaled data<br>get_knn(df, 1, 50, &#39;euclidean&#39;)</pre><p>Result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/932/1*Ss5OaN4Hamk-N9_3nH5RMw.png" /></figure><pre># manhattan metric - unscaled data<br>get_knn(df, 1, 50, &#39;manhattan&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*383YlOQ4sBjgqGoQyzZcpw.png" /></figure><p><strong>Findings:</strong></p><p>When using unscaled data, the results are the same (referring to the generated indices — at index 1, it has a similar classification to the following indices: 3920, 4948, 2528, 3593 -) for both distance metrics.</p><p><strong>Using scaled data:</strong></p><p>For instance, age and income have different scales (age = years, income = dollars), hence data scaling is necessary.</p><p>MaxAbsScaler is utilized to scale data to its maximum value; that is, dividing each observation by the maximum value of the variable: The result of the previous transformation is a distribution where values roughly vary within the range of -1 to 1.</p><pre># scalling the data using MaxAbsScaler<br>feature_names = [&#39;gender&#39;, &#39;age&#39;, &#39;income&#39;, &#39;family_members&#39;]<br><br></pre><pre>transformer_mas = sklearn.preprocessing.MaxAbsScaler().fit(df[feature_names].to_numpy())<br>df_scaled = df.copy()<br>df_scaled.loc[:, feature_names] = transformer_mas.transform(df[feature_names].to_numpy())</pre><p>the df_scaled dataset:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/874/1*-RqrrCqpT_PbQPfZQmLdPQ.png" /></figure><pre># euclidean metric - scaled data<br>get_knn(df_scaled, 1, 10, &#39;euclidean&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/986/1*xeyF3hwP68WWSO_TbPZrxQ.png" /></figure><pre># manhattan metric - scaled data<br>get_knn(df_scaled, 1, 10, &#39;manhattan&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/970/1*VuSZwsM2UK9fy_0cBEl8Kg.png" /></figure><p><strong>The question is: Does non-scaled data affect the kNN algorithm? If it does, how does it affect it?</strong></p><p>Yes, when data is not scaled, the results will be the same (regardless of the metric used). Therefore, the results might be inaccurate due to differences in the scales used in each column.</p><p>In calculations, it is important to maintain a consistent scale as much as possible. For example: age and income have different scales (age = years, income = dollars).</p><p><strong>You can visit my GitHub account to access the complete code related to the above example:</strong></p><p><a href="https://github.com/nasywa-ibtisamah/Developing-kNN-Machine-Learning-Model-and-Analyzing-RMSE-and-R-square-for-Masked-and-Unmasked-Data">GitHub - nasywa-ibtisamah/Developing-kNN-Machine-Learning-Model-and-Analyzing-RMSE-and-R-square-for-Masked-and-Unmasked-Data</a></p><h4>Normalization</h4><p>Normalization, a data preprocessing approach, standardizes feature values within a dataset to a consistent scale. This is carried out to streamline data analysis and modeling, mitigating the influence of disparate scales on machine learning model accuracy [10].</p><p><strong>Implementing in Phyton:</strong></p><pre># import library<br>from sklearn.preprocessing import MinMaxScaler<br><br># fit scaler on training data<br>std = MinMaxScaler().fit(X_train)<br><br># transform training data<br>X_train_std = std.transform(X_train)<br><br># transform testing data<br>X_test_std = std.transform(X_test)</pre><h4>Standardization</h4><p>Standardization, a form of scaling, involves centering values around the mean and adjusting the standard deviation to one unit. Consequently, the attribute’s mean becomes zero, and the resulting distribution maintains a unit standard deviation [10].</p><p>Now, let’s proceed with utilizing scikit-learn’s StandardScaler for standardizing features. This process involves eliminating the mean and adjusting the scale to unit variance, ultimately resulting in a mean of 0 and a standard deviation of 1. This aligns the data with a standard normal distribution. [11].</p><p><strong>Implementing in Phyton:</strong></p><p>We will compare the metric results before and after implementing StandardScaler.</p><p>Before normalizing the features:</p><pre>beforeScaling_lr = LogisticRegression(random_state = 42)</pre><pre># train model on training set<br>beforeScaling_lr.fit(features_train, target_train)<br>#  predict using validation set<br>y_predict_valid_lr = beforeScaling_lr.predict(features_valid)<br># measuring probability using validation set<br>y_probability_valid_lr = beforeScaling_lr.predict_proba(features_valid)[:, 1]<br># test performance algorithm using F1 score and auc_score<br>print(&#39;F1 score =&#39;, f1_score(target_valid, y_predict_valid_lr))<br>print(&#39;AUC-ROC score =&#39;, roc_auc_score(target_valid, y_probability_valid_lr))</pre><p>Result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/478/1*w6lhIZRCu4N1pERnZvbeMw.png" /></figure><p>Normalizing the features:</p><pre># normalizing the features using StandardScaler<br>scaler = StandardScaler()</pre><pre>features_train[df_numerical] = scaler.fit_transform(features_train[df_numerical])<br>features_valid[df_numerical] = scaler.transform(features_valid[df_numerical])<br>features_test[df_numerical] = scaler.transform(features_test[df_numerical])</pre><p>After normalizing the features:</p><pre>afterScaling_lr = LogisticRegression(random_state = 42)</pre><pre># train model on training set<br>afterScaling_lr.fit(features_train, target_train)<br>#  predict using validation set<br>y_predict_valid_lr = afterScaling_lr.predict(features_valid)<br># measuring probability using validation set<br>y_probability_valid_lr = afterScaling_lr.predict_proba(features_valid)[:, 1]<br># test performance algorithm using F1 score and auc_score<br>print(&#39;F1 score =&#39;, f1_score(target_valid, y_predict_valid_lr))<br>print(&#39;AUC-ROC score =&#39;, roc_auc_score(target_valid, y_probability_valid_lr))</pre><p>Result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/492/1*CbJYsEH5Fa_YDp3-J0TVpA.png" /></figure><p><strong>Findings:</strong></p><p>Both the F1 score and AUC-ROC score are increasing after standardizing the features.</p><p><strong>You can visit my GitHub account to access the complete code related to the above example:</strong></p><p><a href="https://github.com/nasywa-ibtisamah/Predicting-Customer-Churn-for-Cost-Effective-Retention-in-A-Bank-using-F1-Score-and-AUC-ROC">GitHub - nasywa-ibtisamah/Predicting-Customer-Churn-for-Cost-Effective-Retention-in-A-Bank-using-F1-Score-and-AUC-ROC</a></p><p><a href="https://learnwithnas.medium.com/data-preprocessing-steps-for-machine-learning-in-phyton-part-2-7cbf5856c757">In Part 2, I will delve into topics such as Feature Selection, handling imbalanced dataset, Encoding Features and Data Splitting. </a>Keep an eye out for this continuation where we’ll explore these essential steps in detail!</p><p><strong>References:</strong></p><p>1. Shaomin Wu, A review on coarse warranty data and analysis (2013)</p><p>2. George Lawton, Data Preprocessing (2022)</p><p><a href="https://www.techtarget.com/searchdatamanagement/definition/data-preprocessing">https://www.techtarget.com/searchdatamanagement/definition/data-preprocessing</a></p><p>3. Yuliia Kniazieva, What is Data Collectio in Machine Learning (2022)</p><p><a href="https://labelyourdata.com/articles/data-collection-methods-AI">https://labelyourdata.com/articles/data-collection-methods-AI</a></p><p>4. Deepak Jain, Data Preprocessing in Data Mining (2023)</p><p><a href="https://www.geeksforgeeks.org/data-preprocessing-in-data-mining/">https://www.geeksforgeeks.org/data-preprocessing-in-data-mining/</a></p><p>5. <a href="https://stats.stackexchange.com/questions/143700/which-is-better-replacement-by-mean-and-replacement-by-median">https://stats.stackexchange.com/questions/143700/which-is-better-replacement-by-mean-and-replacement-by-median</a></p><p>6. Chiradeep BasuMallick, What Is Data Transformation? Types, Tools, and Importance (2022)</p><p><a href="https://www.spiceworks.com/tech/big-data/articles/what-is-data-transformation/">https://www.spiceworks.com/tech/big-data/articles/what-is-data-transformation/</a></p><p>7. Data Science Wizards, Introduction to Data Transformation (2023)</p><p><a href="https://medium.com/@datasciencewizards/introduction-to-data-transformation-935dbbfd3968">Introduction to Data Transformation</a></p><p>8. <a href="https://jakevdp.github.io/PythonDataScienceHandbook/03.08-aggregation-and-grouping.html">https://jakevdp.github.io/PythonDataScienceHandbook/03.08-aggregation-and-grouping.html</a></p><p>9. <a href="https://jakevdp.github.io/PythonDataScienceHandbook/03.09-pivot-tables.html">https://jakevdp.github.io/PythonDataScienceHandbook/03.09-pivot-tables.html</a></p><p>10. Aniruddha Bhandari, Feature Engineering: Scalling, Normalization and Standardization (2023)</p><p><a href="https://www.analyticsvidhya.com/blog/2020/04/feature-scaling-machine-learning-normalization-standardization/">What is Feature Scaling and Why is it Important?</a></p><p>11. Scikit-learn documentation, Standard Scaler</p><p><a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html">https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html</a></p><p>12. Nate Rosidi, Advanced Feature Selection Techniques for Machine Learning Models (2023)</p><p><a href="https://www.kdnuggets.com/2023/06/advanced-feature-selection-techniques-machine-learning-models.html">Advanced Feature Selection Techniques for Machine Learning Models - KDnuggets</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=18009c6f1153" width="1" height="1" alt=""><hr><p><a href="https://medium.com/womenintechnology/data-preprocessing-steps-for-machine-learning-in-phyton-part-1-18009c6f1153">Data Preprocessing Steps for Machine Learning in Python (Part 1)</a> was originally published in <a href="https://medium.com/womenintechnology">Women in Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Evaluating Whether Implementing Matrix Multiplication (Dot Product) for Data Manipulation Affects…]]></title>
            <link>https://learnwithnas.medium.com/evaluating-whether-implementing-matrix-multiplication-dot-product-for-data-manipulation-affects-6b0f1fc5d567?source=rss-cbae1ad7c072------2</link>
            <guid isPermaLink="false">https://medium.com/p/6b0f1fc5d567</guid>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[matrix-multiplication]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Learn with Nas]]></dc:creator>
            <pubDate>Sun, 24 Sep 2023 05:43:24 GMT</pubDate>
            <atom:updated>2023-09-24T09:30:26.748Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Evaluating Whether Implementing Matrix Multiplication (Dot Product) for Data Manipulation Affects the Performance of a Predictive Model</strong></h3><p>In the realm of data-driven insights and predictive modelling, the integrity and quality of data are paramount. Data manipulation can be used to protect sensitive information by altering or modifying it. In the context of data security and privacy, altering the sensitive data is crucial. However, a pertinent concern arises: Does data manipulation impact the performance of a predictive model? Before we start investigating this, let’s first understand why data manipulation is important.</p><p>By implementing data manipulation, sensitive data columns can be modified/masked. If an attacker compromises an internal account, they will not be able to understand the sensitive data.</p><p>According to data from IBM, the global average total cost of a data breach in 2023 was 4.45 million dollars, reflecting a 15% increase over three years. This indicates that manipulating data is one process that needs to be carefully considered.</p><p>In this article, we will implement matrix multiplication (dot product) as a technique to manipulate the data.</p><h4><strong>What is Matrix Multiplication and Dot Product?</strong></h4><p>Matrix multiplication specifies a set of rules for multiplying matrices together to produce a new matrix.</p><p>Not all matrices are eligible for multiplication. In addition, there is a requirement on the dimensions of the resulting matrix output.</p><ol><li>The number of columns of the 1st matrix must equal the number of rows of the 2nd</li><li>The product of an M x N matrix and an N x K matrix is an M x K matrix. The new matrix takes the rows of the 1st and columns of the 2nd</li></ol><p>The dot product is a specific type of matrix multiplication applicable when dealing with vectors (1D matrices). Alternatively, when working with 2D matrices, you can reshape them into 1D vectors and then compute the dot product.</p><p>In summary, while matrix multiplication is a broader concept applicable to matrices of varying dimensions, the dot product is a specific case of matrix multiplication when dealing with vectors or flattened matrices.</p><p>Here are examples of both 1D (vectors) and 2D matrices for illustrating matrix multiplication and dot product:</p><p><strong>One Dimensional Matrices (vectors)</strong></p><pre>A = [1, 2, 3]<br>B = [4, 5, 6]pu</pre><p>Dot Product:</p><p>For the dot product of vectors A and B:</p><p>A • B (dot product) = (1 * 4) + (2 * 5) + (3 * 6) = 32</p><p><strong>Two Dimensional Matrices</strong></p><pre>A = [[1, 2, 3],<br>    [4, 5, 6]]<br><br>B = [[7, 8],<br>    [9, 10],<br>    [11, 12]]</pre><p>Matrix Multiplication:</p><p>Matrix multiplication of A and B (resulting in a 2x2 matrix):</p><pre>Result = [[58, 64],<br>         [139, 154]]</pre><h3><strong>How Matrix Multiplication (dot product) can be used in a process to mask data in a Deep Learning Model?</strong></h3><p>Matrix multiplication (dot model) is not a conventional data masking technique by itself. It is a mathematical operation used for various purposes in linear algebra and data manipulation. However, in some specific contexts, matrix multiplication can be used in a process that could be considered a form of data masking.</p><p>For instance, matrix multiplication can be used to apply a transformation to the data, such as encryption or obfuscation. This transformation could be viewed as a form of data masking if it alters the original data in a way that hides its true nature or obscures sensitive information.</p><p>Let’s roll up our sleeves and go through the process.</p><p>Suppose you have completed the data preprocessing step and the exploratory data analysis (EDA) step, and you are now ready to proceed with model training and evaluation. Iwill compare the performance results of two methods: one with data masking and one without. I am using RMSE as the performance metric.</p><ol><li><strong>Create a function to measure the performance metric</strong></li></ol><pre>def eval_regressor(y_true, y_pred):<br>    <br>    rmse = math.sqrt(sklearn.metrics.mean_squared_error(y_true, y_pred))<br>    print(f&#39;RMSE: {rmse:.2f}&#39;)<br>    <br>    r2_score = math.sqrt(sklearn.metrics.r2_score(y_true, y_pred))<br>    print(f&#39;R2: {r2_score:.2f}&#39;)    </pre><p><strong>2. Data Transformation and Creating Matrix P2</strong></p><pre># creation of matrix P2 as a constant<br>rng = np.random.default_rng(seed=42)<br>P2 = rng.random(size=(X.shape[1], X.shape[1]))</pre><pre>P2_inverse = np.linalg.inv(P2)<br>P2_inverse</pre><p><strong>3. Training the model without masking the data</strong></p><pre>reg_withoutmasking = LinearRegression()<br>reg_withoutmasking.fit(X, y)<br><br>y_test_pred_withoutmasking = reg_withoutmasking.predict(X)<br>eval_regressor(y, y_test_pred_withoutmasking)</pre><p>The result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/872/1*JQFoSs-ZUSjlR4K4dsK5Wg.png" /></figure><p><strong>4. Data Manipulation Process</strong></p><pre>X_mask = X.dot(P2)</pre><p><strong>5. Training the model with data that has been masked</strong></p><pre>reg_masking = LinearRegression()<br>reg_masking.fit(X_mask, y)<br><br>y_test_pred_masking = reg_masking.predict(X_mask)<br>eval_regressor(y, y_test_pred_masking)</pre><p>Result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/858/1*6ItzaJ_396XJH-foz5ApyQ.png" /></figure><h3>Conclusion</h3><p>The RMSE value in the LinearRegression model with or without masking<strong> is the SAME.</strong></p><p>This proves that using matrix multiplication (Dot Product) for Data manipulation <strong>will not affect</strong> the Performance of a Predictive Model.</p><p>Thank you for dedicating your time to read this article! Stay tuned for more insightful content in my upcoming articles :)</p><p><strong>You can visit my GitHub account for the complete code:</strong></p><p><a href="https://github.com/nasywa-ibtisamah/Developing-kNN-Machine-Learning-Model-and-Analyzing-RMSE-and-R-square-for-Masked-and-Unmasked-Data">GitHub - nasywa-ibtisamah/Developing-kNN-Machine-Learning-Model-and-Analyzing-RMSE-and-R-square-for-Masked-and-Unmasked-Data</a></p><p><a href="https://www.linkedin.com/in/nasywa/"><strong>Connect with me on Linkedin</strong></a></p><p><strong>Reference:</strong></p><p>1. <a href="https://www.ibm.com/reports/data-breach">https://www.ibm.com/reports/data-breach</a></p><p>2. <a href="https://pathlock.com/learn/5-data-masking-techniques-and-why-you-need-them/">https://pathlock.com/learn/5-data-masking-techniques-and-why-you-need-them/</a></p><p>3. <a href="https://builtin.com/data-science/dot-product-matrix">https://builtin.com/data-science/dot-product-matrix</a></p><p>4. <a href="https://ml-cheatsheet.readthedocs.io/en/latest/linear_algebra.html#matrix-multiplication">https://ml-cheatsheet.readthedocs.io/en/latest/linear_algebra.html#matrix-multiplication</a></p><p>5. <a href="https://medium.com/data-science-bootcamp/understand-dot-products-matrix-multiplications-usage-in-deep-learning-in-minutes-beginner-95edf2e66155">https://medium.com/data-science-bootcamp/understand-dot-products-matrix-multiplications-usage-in-deep-learning-in-minutes-beginner-95edf2e66155</a></p><p>6. <a href="https://u-next.com/blogs/data-science/data-manipulation/">https://u-next.com/blogs/data-science/data-manipulation/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6b0f1fc5d567" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Find the Best parameter for each model: A Sample Project of Predicting Steel's Optimal Temperature…]]></title>
            <link>https://learnwithnas.medium.com/find-the-best-parameter-for-each-model-a-sample-project-of-predicting-steels-optimal-temperature-f98547878bf?source=rss-cbae1ad7c072------2</link>
            <guid isPermaLink="false">https://medium.com/p/f98547878bf</guid>
            <category><![CDATA[loop]]></category>
            <category><![CDATA[gridsearchcv]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[regression]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Learn with Nas]]></dc:creator>
            <pubDate>Thu, 21 Sep 2023 12:50:51 GMT</pubDate>
            <atom:updated>2023-09-24T09:32:34.546Z</atom:updated>
            <content:encoded><![CDATA[<h3>Find the Best parameter for each model: A Sample Project of Predicting Steel&#39;s Optimal Temperature to Reduce Electricity Consumption (contd)</h3><p>I wrote another article focused on “<a href="https://medium.com/@learnwithnas/tuning-multiple-ml-models-using-gridsearchcv-a-sample-project-of-predicting-steel-optimal-8d1c86f7f4f0">Tuning Multiple ML Models using GridSearchCV</a>”. To effectively compare the performance metric results for each model, it’s essential to identify the optimal parameters for each one. Detailed information regarding the sample project, including the problem definition, data definition, and data preprocessing, is available in that article. If you’re interested in understanding more about the project, please refer to the link provided above.</p><h3>How to find the best parameter for each model?</h3><p>Let’s roll up our sleeves!</p><ol><li><strong>Define the hyperparameter</strong></li></ol><pre>params = [<br>    {<br>        &#39;Regressor&#39; : [RandomForestRegressor()],<br>        &#39;Regressor__criterion&#39;: [&#39;friedman_mse&#39;],<br>        &#39;Regressor__n_estimators&#39;: [50, 100, 500],<br>    },<br>    {<br>        &#39;Regressor&#39; : [DecisionTreeRegressor()],<br>        &#39;Regressor__criterion&#39; : [&#39;friedman_mse&#39;],<br>        &#39;Regressor__max_depth&#39; : [None, 1, 5, 10, 15],<br>    },<br>    {<br>        &#39;Regressor&#39; : [XGBRegressor()],<br>        &#39;Regressor__learning_rate&#39; : [0.01, 0.1, 0.2, 0.5],<br>    },<br>    {<br>        &#39;Regressor&#39; : [LGBMRegressor()],<br>        &#39;Regressor__num_leaves&#39; : [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20],<br>        &#39;Regressor__learning_rate&#39; : [0.01, 0.1, 0.2, 0.5],<br>    },<br>    {<br>        &#39;Regressor&#39; : [LinearRegression()],<br>    },<br>    {<br>        &#39;Regressor&#39; : [CatBoostRegressor()],<br>    }<br>]</pre><p><strong>2. Training Model using For loop</strong></p><pre>%%time<br><br>models = [<br>    RandomForestRegressor(random_state=random_state),<br>    DecisionTreeRegressor(random_state=random_state),<br>    XGBRegressor(random_state=random_state),<br>    LGBMRegressor(random_state=random_state),<br>    LinearRegression(),<br>    CatBoostRegressor(verbose=False, random_state=random_state)<br>]<br><br>best_params_list = []<br>mae_list = []<br><br>for i, model in enumerate(models):<br>    print(f&quot;Model {i+1}: {type(model).__name__}&quot;)<br>    <br>    pipe = Pipeline([(&#39;Regressor&#39;, model)])<br>    param_grid = params[i]<br>    grid_search = GridSearchCV(pipe, param_grid, cv=cv,scoring=&#39;neg_mean_absolute_error&#39;, n_jobs=-1)<br>    grid_search.fit(features_train, target_train)<br>    <br>    # CatBoost handling<br>    if isinstance(model, CatBoostRegressor):<br>        best_params_cbr = CatBoostRegressor(verbose=False, random_state=random_state).fit(features_train, target_train).get_all_params()<br>        <br>    else:<br>        best_params = grid_search.best_params_<br>        best_params_list.append(best_params)<br>    <br>    mae = grid_search.best_score_<br>    mae_list.append(mae)<br>    <br>        <br>    print(&quot;Best parameters:&quot;, best_params)<br>    print(&quot;Mean value:&quot;,  mae)<br>    print()</pre><p>During this process, I saved the best parameters and MAE values for each model in ‘best_params_list’ and ‘mae_list’. For ‘CatBoostRegressor’, given its unique process compared to other models, I implemented an If-Else condition to handle it appropriately and saved the best parameters in a variable named ‘best_params_cbr’.</p><p>Here is the data of best_params_list. You can see the best parameter for each model, except for CatBoostRegressor.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FwgYOqCUaZiIsZed3Fcang.png" /></figure><p>For the CatBoostRegressor, you can refer to the variable named as ‘best_params_cbr’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/1*c33O65rnY2dDT5Ie9whT-Q.png" /></figure><p><strong>3. Comparing the MAE Value on the Train Set</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/436/1*INhhmZZIQGpFZ7BbU82aJA.png" /></figure><p>The neg_mean_absolute_error scoring is essentially the negative of the Mean Absolute Error (MAE). It’s designed this way because scikit-learn’s cross-validation and grid search functions aim to maximize scoring values, so they use the negation of error metrics to find the best model. A lower (more negative) neg_mean_absolute_error indicates better model performance.</p><p>The MAE values for each model on the train set are as follows:</p><ul><li>RandomForestRegressor: 5.946547832399344</li><li>DecisionTreeRegressor : 6.8359288354882795</li><li>XGBRegressor : 5.777462005615234</li><li>LGBMRegressor : 5.894216192872837</li><li>LinearRegression : 7.466512240607064</li><li>CatBoostRegressor : 5.705407348020233</li></ul><p>From the results above, it can be observed that all models meet the minimum MAE value requirement (&lt;=8.7), and four other models (RandomForestRegressor, CatBoostRegressor, XGBRegressor, and LGBMRegressor) have ideal MAE values &lt; 6.</p><p>The model with the lowest MAE value is CatBoostRegressor with a value of 5.705407348020233</p><p><strong>4. Cleansing Best Parameter List</strong></p><p>Before I proceeded to implement the best parameter in the Test Set, I removed the key-value pair associated with ‘Regressor’. If the ‘Regressor’ key is not found in the dictionary, it does nothing (‘None’ is returned).</p><pre>for params_dict in best_params_list:<br>    params_dict.pop(&#39;Regressor&#39;, None)</pre><p>Here is the result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FgvtDnVFh2pQXLyyHjQwag.png" /></figure><p><strong>5. Applying the trained model to the Test dataset</strong></p><pre>%%time<br><br>models = [<br>    RandomForestRegressor(random_state=random_state),<br>    DecisionTreeRegressor(random_state=random_state),<br>    XGBRegressor(random_state=random_state),<br>    LGBMRegressor(random_state=random_state),<br>    LinearRegression(),<br>    CatBoostRegressor()<br>]<br><br>test_mae_list = []<br><br># Loop through the models and evaluate them<br>for i, model in enumerate(models):<br>    print(f&quot;Model {i+1}: {type(model).__name__}&quot;)<br>    <br>    # CatBoost handling<br>    if isinstance(model, CatBoostRegressor):<br>        model.set_params(**best_params_cbr)  # Set the best parameters for CatBoost<br>    elif params:<br>        model_params = {key.split(&#39;__&#39;)[1]: value for key, value in best_params_list[i].items()}<br>        model.set_params(**model_params)  # Set the best parameters for other models<br>    <br>    # Train and evaluate the model<br>    model.fit(features_train, target_train)<br>    target_predict = model.predict(features_test)<br>    test_mae = mean_absolute_error(target_predict, target_test)<br>    test_mae_list.append(test_mae) <br>    <br>    print(f&#39;MAE Score in test dataset: {test_mae}&#39;)</pre><p>In this procedure, I utilize the optimal parameters for each model, previously stored in ‘best_params_list’. Following this, I store the Mean Absolute Error (MAE) value for each model in a variable titled ‘test_mae_list’</p><p>Here is the MAE result for each model</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/420/1*QftKqCD356RTSJqXaV8O6Q.png" /></figure><p>The Mean MAE values on the test set with the best parameters applied are as follows:</p><ul><li>RandomForestRegressor: 5.348324022346371</li><li>DecisionTreeRegressor: 6.26299664493562</li><li>XGBRegressor: 5.3495793</li><li>LGBMRegressor: 5.583030851802979</li><li>LinearRegression: 7.2683977438694845</li><li>CatBoostRegressor: 5.39874692476624</li></ul><p>Similar to the MAE values on the train set, all models meet the minimum MAE value requirement (&lt;=8.7), and four other models (RandomForestRegressor, CatBoostRegressor, XGBRegressor, and LGBMRegressor) have ideal MAE values &lt; 6.</p><p>The model with the lowest MAE value is XGBRegressor with a value of 5.3495793</p><p>Furthermore, the close similarity in MAE values between the train set and test set demonstrates that there is no overfitting in the model, and the model is sufficiently accurate.</p><p><strong>Conclusion: </strong>The decision to use a loop or not depends on your specific requirements — whether you seek only the best parameter of the top model or you want to assess the performance metric results for each model.</p><p>Thank you for dedicating your time to read this article! Stay tuned for more insightful content in my upcoming articles :)</p><p><strong>You can visit my GitHub account for the complete code:</strong></p><p><a href="https://github.com/nasywa-ibtisamah/Find_the_Best_parameter_for_each_model-Predicting_temperature_to_reduce_electricity_consumption">GitHub - nasywa-ibtisamah/Find_the_Best_parameter_for_each_model-Predicting_temperature_to_reduce_electricity_consumption</a></p><p><a href="https://www.linkedin.com/in/nasywa/">Connect with me on Linkedin</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f98547878bf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tuning Multiple ML Models using GridSearchCV: A Project of Predicting Steel’s Optimal Temperature…]]></title>
            <link>https://learnwithnas.medium.com/tuning-multiple-ml-models-using-gridsearchcv-a-sample-project-of-predicting-steel-optimal-8d1c86f7f4f0?source=rss-cbae1ad7c072------2</link>
            <guid isPermaLink="false">https://medium.com/p/8d1c86f7f4f0</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[gridsearchcv]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[regression]]></category>
            <dc:creator><![CDATA[Learn with Nas]]></dc:creator>
            <pubDate>Sun, 17 Sep 2023 11:19:35 GMT</pubDate>
            <atom:updated>2023-10-23T08:44:58.900Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Tuning Multiple ML Models using GridSearchCV: A Project of Predicting Steel’s Optimal Temperature to Reduce Electricity Consumption</strong></h3><p>In the rapidly evolving field of machine learning, the quest for optimal model performance is unending. It’s crucial to understand that the success of any machine learning or deep learning project often hinges on finding the right set of hyperparameters. These parameters serve as the dials and knobs that can significantly impact a model’s performance, from accuracy to efficiency. To achieve the highest levels of accuracy and efficiency, data scientists and engineers often find themselves exploring multiple machine learning models [1]. However, fine-tuning these models for the best results can be a complex and time-consuming task. In this article, we delve into GridSearchCV to simplify the process of tuning multiple models.</p><h3><strong>Let’s refresh our memories, what is a hyperparameter?</strong></h3><p>Hyperparameters represent the specific parameters configured prior to initiating the machine learning model’s training process. They function as controls and levers that can be fine-tuned to various configurations with the aim of enhancing the model’s learning capabilities. In contrast to standard parameters, hyperparameters are not acquired or adjusted by the model autonomously [1].</p><p>If we draw a comparison with everyday situations, it’s like desiring to craft a burger recipe based on your father’s instructions. While all the ingredients and their amounts have been clearly laid out, you opt to introduce an extra layer of cheese, incorporate an additional meat slice, and omit the vegetables. This is similar to adjusting the burger’s hyperparameters.</p><p>For example, the learning rate (one of the hyperparameters that define the adjustment in the weights of our network with respect to the loss gradient descent) of a linear regression model determines how quickly it learns. If the learning rate is set too high, the model may learn quickly but make more mistakes. With a high learning rate, the model will start to look more exponential. Higher learning rates will decay the loss faster, but they get stuck at worse values of loss (as seen in the green line). On the other hand, if the learning rate is set too low, the model may learn slowly but make fewer mistakes. The quality of the predictions made by the model is measured by the model’s score, which can vary depending on the chosen hyperparameters [2].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/459/1*nM0iXoX5cWIROpbgWi4PQA.jpeg" /><figcaption>Effects of different learning rates (image credit: <a href="https://cs231n.github.io/neural-networks-3">cs231n</a>)</figcaption></figure><h3><strong>How to tune the hyperparameters?</strong></h3><p>Let’s roll up our sleeves and put Hyperparameter Tuning into practice in a sample project. Here’s a step-by-step guide on how to tune hyperparameters effectively:</p><h4><strong>1. Understand the Problem and Data</strong></h4><p>According to Cameron Warren, “<em>… the number one most important skill for a data scientist above any technical expertise is the ability to clearly evaluate and define a problem” </em>[3].</p><p>Understanding the problem and the data is a crucial first step in any data science or machine learning project. There’s a great value in being able to translate a business idea or question into a clearly formulated problem statement. And being able to effectively communicate whether or not that problem can be solved by applying appropriate machine learning algorithm. It sets the foundation for making informed decisions throughout the project [3].</p><p><strong>Implementation in the sample project:</strong></p><p><em>Introduction</em></p><p>A plant aims to reduce its electricity consumption during the steel processing stage to lower production costs. This is achieved by determining the optimal temperature at the steel processing stage.</p><p><em>Problem</em></p><p>Predicting the temperature of metal in the steel processing stage is crucial for reducing energy consumption and optimizing production processes for several reasons:</p><ol><li>Efficient Energy Usage: Maintaining the metal at the correct temperature is essential for the various processes involved in steel production, such as melting, alloying, and casting. Accurate temperature control ensures that energy is used efficiently, avoiding overheating or underheating that would result in energy wastage.</li><li>Quality Control: The quality of the final steel product is highly dependent on its temperature during processing. Accurate temperature control helps ensure that the steel meets the required quality standards, reducing the need for rework or scrap, which can be both energy-intensive and costly.</li><li>Process Optimization: Understanding and predicting the metal’s temperature allows for better process optimization. It enables the adjustment of parameters like heating time, alloy composition, and cooling rates to achieve the desired product characteristics while minimizing energy consumption.</li><li>Reduced Heat Loss: By predicting and controlling the temperature accurately, companies can minimize heat loss throughout the production process. This includes reducing heat loss through the walls of the furnaces or other processing equipment, which can significantly impact energy efficiency.</li><li>Environmental Impact: Lowering energy consumption not only reduces costs but also has a positive environmental impact by decreasing greenhouse gas emissions and resource consumption. It aligns with sustainability goals and regulatory requirements.</li><li>Cost Savings: Energy costs can be a significant portion of a steel production facility’s operating expenses. Predicting and optimizing temperature control can lead to substantial cost savings over time, improving the company’s overall financial performance.</li><li>Consistency and Safety: Consistent temperature control ensures a safer working environment for employees, as extreme temperature variations can pose safety risks. Predictive modeling helps maintain a stable and safe processing environment.</li></ol><p>In summary, predicting the temperature of metal during steel processing is essential for improving energy efficiency, reducing production costs, maintaining product quality, and aligning with environmental and sustainability goals. It allows companies to make informed decisions and adjustments to their processes, ultimately leading to more efficient and sustainable steel production.</p><p><em>Data Description</em></p><p>There are 7 datasets, as follows</p><p><strong>a. Df_arc</strong>: contains the information about:</p><ul><li>Arc Heating start- date and time when the intense heat is generated by creating an electric arc, which is a discharge of electric current through a gap in a circuit. arc heating is used to achieve high temperatures for tasks like melting metals or fusing materials.</li><li>Arc Heating end- date and time when the intense heat is concluded or terminated. This marks the completion of this heat generation phase and often indicates that the process is transitioning to a different stage or task.</li><li>Active power- the real power consumed by an electrical system. the active power is the component that contributes to the actual functioning or performance of the system.</li><li>Reactive power- the power that oscillates back and forth between the source and the load in an alternating current (AC) electrical system. it is required to establish and maintain the electromagnetic fields in devices like motors, transformers, and solenoids. a balance between active power and reactive power is necessary to maintain efficiency and reliability.</li></ul><p><strong>b. Df_bulk</strong>: contains the information about bulk material supply data (volume).</p><p><strong>c. Df_bulk_time</strong>: contains the information about bulk material delivery data (date and time).</p><p><strong>d. Df_gas</strong>: contains the information data for gas purge.</p><p><strong>e. Df_temp</strong>: contains the information about:</p><ul><li>Sampling time- date and time when doing measurement</li><li>Temperature- temperature when doing measurement</li></ul><p><strong>f. Df_wire</strong>: contains the information about wire materials data (volume)</p><p><strong>g. Df_wire_tim</strong>e: contains the information about wire materials data (date and time)</p><p>Based on the Data Overview process, there are several issues:</p><ol><li>A lot of missing data in df_bulk, df_bulk_time df_wire df_wire_time, df_temp.</li><li>Missing values in df_bulk_time follow the same pattern as df_bulk.</li><li>Missing values in df_wire_time follow the same pattern as df_wire.</li><li>Incorrect data types &gt; date and time values need to be converted to the datetime format and temperature colum should be converted to integer data type.</li><li>Inconsistent column names.</li></ol><h4>2. Data Transformation</h4><p>I will transform raw data into features that are more useful by adding new aggregate columns, as follows:</p><ol><li>summarising sum_active_power and sum_reactive power, indexed by ‘key’</li></ol><pre>df_arc = df_arc.query(&#39;reactive_power &gt; 0&#39;)<br>df_arc_sum = pd.pivot_table(df_arc, index = &#39;key&#39;, values = [&#39;active_power&#39;, &#39;reactive_power&#39;],<br>                            aggfunc={&#39;active_power&#39;: np.sum,<br>                                      &#39;reactive_power&#39;: np.sum})</pre><pre>df_arc_sum = df_arc_sum.rename(columns = {<br>    &quot;active_power&quot; : &quot;sum_active_power&quot;,<br>    &quot;reactive_power&quot; : &quot;sum_reactive_power&quot;<br>})</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/1*Nc5kQKJG-QgSx4PkxuXiyA.png" /></figure><p>2. exluding bad criteria of df_temp data</p><p>I will categorize the following conditions as ‘excluded_key’ based on either of the following criteria:</p><p>1. The maximum measurement time in df_temp for a given key is less than the maximum ‘end_of_arc_heating’ time in df_arc for the same key.</p><p>2. The maximum and minimum measurement times are the same for a given key in df_temp.</p><pre>excluded_key = []<br><br>for key in list(df_temp[&#39;key&#39;].unique()):<br>    try:<br>        if ((df_temp[df_temp[&#39;key&#39;] == key][&#39;measurement_time&#39;].max() &lt; df_arc[df_arc[&#39;key&#39;] == key][&#39;end_of_arc_heating&#39;].max()) <br>            or<br>           (df_temp[df_temp[&#39;key&#39;] == key][&#39;measurement_time&#39;].max() == df_temp[df_temp[&#39;key&#39;] == key][&#39;measurement_time&#39;].min())):<br>            excluded_key.append(key)<br>    except:<br>        excluded_key.append(key)</pre><pre>df_temp = df_temp.query(&#39;key not in @excluded_key&#39;)<br>df_temp = df_temp.dropna()<br>df_temp.info()</pre><pre>counting_excluded_key = (df_temp[&#39;key&#39;].value_counts() &lt; 2).sum() <br>good_keys = list(df_temp[&#39;key&#39;].value_counts().index[:-counting_excluded_key])<br>df_temp = df_temp.query(&#39;key in @good_keys&#39;)<br>df_temp[&#39;key&#39;].value_counts()</pre><p>3. Finding start temperature and end temperature , based on ‘min’ and ‘max’ temperature time, indexed by ‘key’</p><pre>agg_df_temp = pd.pivot_table(df_temp,values=&#39;measurement_time&#39;, index=&#39;key&#39;, aggfunc={&#39;measurement_time&#39;: [np.min, np.max]})<br>agg_df_temp</pre><pre># create list<br>start_time_list = list(agg_df_temp[&#39;amin&#39;])<br>end_time_list = list(agg_df_temp[&#39;amax&#39;])</pre><pre>start_temp = df_temp.query(&#39;measurement_time in @start_time_list &#39;)<br>start_temp = start_temp.set_index(&#39;key&#39;)<br>end_temp = df_temp.query(&#39;measurement_time in @end_time_list &#39;)<br>end_temp = end_temp.set_index(&#39;key&#39;)</pre><pre>df_temp_final = start_temp.copy()<br>df_temp_final.columns = [&#39;measurement_time&#39;,&#39;start_temp&#39;]<br>df_temp_final[&#39;end_temp&#39;] = end_temp[&#39;temperature&#39;]</pre><pre>df_temp_final = df_temp_final.drop(&#39;measurement_time&#39;, axis=1)<br>df_temp_final.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/1*ArG59dw67tXpqpwatFWkFg.png" /></figure><h4>3. Data Preprocessing</h4><p>I did Pre-processing Data as follows:</p><ol><li>Standardized Column Names</li><li>Dropped Unused Columns</li><li>Prepared Final Dataset</li></ol><pre>df_final = pd.concat([df_arc_sum, df_temp_final, df_bulk, df_gas, df_wire], axis=1, sort=False)</pre><pre>df_final = df_final.dropna(subset=[&#39;end_temp&#39;])</pre><p>Columns in df_final</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/628/1*RqJJNrqj8vE8UzmX38bUGA.png" /></figure><p>4. Filled Missing Values</p><p>5. Fixed Data Types</p><h4>4. Choose the right Algorithm and Model</h4><p>Since the output of the model trained in this project is to predict metal temperature (continuous value), this falls into a regression problem, and an accurate and efficient regression model is needed.</p><p><strong>Implementation in the sample project:</strong></p><p>The chosen regression models are:</p><p>- Linear Regression</p><p>- Decision Tree Regression</p><p>- Random Forest Regression</p><p>- CatBoostRegressor</p><p>- XGBRegressor</p><p>- LGBMRegressor</p><h4>5. Split the data</h4><p>Implementation in the sample project:</p><pre>random_state = 43<br><br>features = df_final.drop(&#39;end_temp&#39;, axis=1)<br>target = df_final[&#39;end_temp&#39;]<br><br>features_train, features_test, target_train, target_test = train_test_split(<br>                                                            features, <br>                                                            target, <br>                                                            test_size=0.25, <br>                                                            random_state=random_state)<br><br>cv = 6</pre><h4>6. Select hyperparameters to tune</h4><p><strong>Implementation in the sample project:</strong></p><pre>pipe = Pipeline([(&#39;Regressor&#39;, RandomForestRegressor(random_state=random_state))])<br><br>params = [<br>    {<br>        &#39;Regressor&#39; : [RandomForestRegressor(random_state=random_state)],<br>        &#39;Regressor__criterion&#39;: [&#39;friedman_mse&#39;],<br>        &#39;Regressor__n_estimators&#39;: [50, 100, 500],<br>    },<br>    {<br>        &#39;Regressor&#39; : [DecisionTreeRegressor(random_state=random_state)],<br>        &#39;Regressor__criterion&#39; : [&#39;friedman_mse&#39;],<br>        &#39;Regressor__max_depth&#39; : [None, 1, 5, 10, 15],<br>    },<br>    {<br>        &#39;Regressor&#39; : [XGBRegressor(random_state=random_state)],<br>        &#39;Regressor__learning_rate&#39; : [0.01, 0.1, 0.2, 0.5],<br>    },<br>    {<br>        &#39;Regressor&#39; : [LGBMRegressor(random_state=random_state)],<br>        &#39;Regressor__num_leaves&#39; : [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20],<br>        &#39;Regressor__learning_rate&#39; : [0.01, 0.1, 0.2, 0.5],<br>    },<br>    {<br>        &#39;Regressor&#39; : [CatBoostRegressor(verbose=False, random_state=random_state)],<br>    },<br>    {<br>        &#39;Regressor&#39; : [LinearRegression()],<br>    }<br>]</pre><h4>7. Choose a Search Strategy and Set a Performance Metric</h4><p>There are several strategies for hyperparameter tuning:</p><ul><li>Grid Search: This involves specifying a set of hyperparameter values to explore exhaustively [4].</li><li>Random Search: Randomly samples hyperparameters from predefined ranges, which can be more efficient than grid search [4].</li><li>Bayesian Optimization: Utilizes a probabilistic model to determine the next set of hyperparameters to evaluate [4].</li><li>Gradient-Based Optimization: Involves using optimization algorithms like gradient descent to search for optimal hyperparameters [4]</li></ul><p>On the other hand, performance metrics play a crucial role in every machine learning pipeline, providing insights into progress by quantifying it. Regardless of the model, be it linear regression or classification, assessing performance requires these metrics. Metrics are used to monitor and measure the performance of a model (during training and testing) [5].</p><p><em>Regression Metrics</em></p><p>Regression models have continuous output. In order to evaluate regression models, the following are the metrics you can use based on your needs [5]:</p><ul><li>Mean Absolute Error (MAE),</li><li>Mean Squared Error (MSE),</li><li>Root Mean Squared Error (RMSE),</li><li>R² (R-Squared).</li></ul><p><em>Classification Metrics</em></p><p>Classification models have discrete outputs, so we need a metric that compares discrete classes in some form<em>. </em>Classification metrics assess a model’s performance, providing insights into the quality of classification. However, each metric evaluates the performance in a distinct manner. In order to evaluate classification models, the following are the metrics you can use based on your needs [5]:</p><ul><li>Accuracy</li><li>Confusion Matrix (not a metric but fundamental to others)</li><li>Precision and Recall</li><li>F1-score</li><li>AU-ROC</li></ul><p>You can click <a href="https://neptune.ai/blog/performance-metrics-in-machine-learning-complete-guide">here</a> for further explanation of the above metrics.</p><p><strong>Implementation in the sample project:</strong></p><p>I chose GridSearchCV as it tries all the combinations of the values passed in the dictionary and evaluates the model for each combination using the <a href="https://www.mygreatlearning.com/blog/cross-validation/">Cross-Validation</a> method. Hence after using this function, we get accuracy/loss for every combination of hyperparameters and we can choose the one with the best performance.</p><p>For the performance metric, since the output of the model trained in this project is to predict metal temperature (continuous value), this falls into <strong>a regression problem </strong>and the used metric is Mean Absolute Error (MAE). It measures the average absolute difference between the predicted values and the actual values. In the context of predicting metal temperature, I would aim to minimize the MAE to ensure that your model’s predictions are as close as possible to the actual temperature values.</p><h4>8. Model Training and Evaluation</h4><p><strong>Implementation in the sample project:</strong></p><pre>grid_search = GridSearchCV(pipe, params, cv = cv, scoring = &#39;neg_mean_absolute_error&#39;, verbose = 2, n_jobs= -1)<br><br>grid_search.fit(features_train, target_train)<br><br>best_model = grid_search.best_estimator_<br>best_params = grid_search.best_params_<br>best_score = grid_search.best_score_</pre><pre># save the result<br>cv_results_df = pd.DataFrame(grid_search.cv_results_)</pre><pre># filter the mean_test_score value<br>cv_results_df.loc[cv_results_df[&#39;mean_test_score&#39;].between(-5.9, -5.7)]</pre><p>The neg_mean_absolute_error scoring is essentially the negative of the Mean Absolute Error (MAE). It’s designed this way because scikit-learn’s cross-validation and grid search functions aim to maximize scoring values, so they use the negation of error metrics to find the best model. A lower (more negative) neg_mean_absolute_error indicates better model performance.</p><p>When I filtered the MAE value between -5.7 and -5.9, the top three models were as follows:</p><p>CatBoostRegressor: 5.762954</p><p>XGBRegressor: 5.777462</p><p>LGBMRegressor: 5.894216</p><p>From the results above, it can be observed that all models meet the minimum MAE value requirement (&lt;= 8.7). Three models (CatBoostRegressor, XGBRegressor, and LGBMRegressor) have ideal MAE values of less than 6. The model with the lowest MAE value is CatBoostRegressor with a value of 5.762954.</p><h4>9. <strong>Applying the trained model to Test dataset</strong></h4><p>Evaluate the model’s performance on a separate test dataset to assess how well it generalizes to new, unseen data. This step helps ensure that your model isn’t overfitting to the training data.</p><p><strong>Implementation in the sample project:</strong></p><pre>%%time<br><br>model = best_model<br><br>model.set_params(**best_params)<br>model.fit(features_train, target_train)<br>target_predict = model.predict(features_test)<br><br>test_mae_dtr = mean_absolute_error(target_predict, target_test)<br>print(&#39;MAE scores&#39;, test_mae_dtr)</pre><p><strong>Conclusion:</strong></p><p>The Mean MAE value on the test set, obtained by applying the best model with the best parameters, is 5.39874692476624.</p><p>This model meets the minimum MAE value requirement (&lt;= 8.7) and has an ideal MAE value of &lt; 6.</p><p>Furthermore, the close similarity in MAE values between the training set and the test set demonstrates that there is no overfitting in the model, indicating its accuracy.</p><p>This sample project has successfully trained a model capable of predicting the optimal end temperature of metal during steel processing to enhance energy efficiency and reduce production costs. The plant can utilize the CatBoostRegressor model with the known best parameters to predict the optimal end temperature. With an MAE value of 5.39, it indicates that the average error between the model’s predictions and the actual values in the test data is approximately 5.39 degrees Celsius.</p><p><strong>Recommendations:</strong></p><p>This project only uses one measurement metric, which is MAE. It would be better if other metrics, such as R2, MSE, and RMSE, were also used for measurement.</p><p>By using the RMSE metric, it is possible to measure the average error between the model’s predictions and the actual values in the same units as the target variable. Additionally, R-squared does not provide information about how well the model fits the data.</p><p>Thank you for dedicating your time to read this article! Stay tuned for more insightful content in my upcoming articles :)</p><p><strong>You can visit my GitHub account for the complete code:</strong></p><p><a href="https://github.com/nasywa-ibtisamah/Optimizing_Steel_Processing_Efficiency-Predicting_Temperature_to_Reduce_Electricity_Consumption">GitHub - nasywa-ibtisamah/Optimizing_Steel_Processing_Efficiency-Predicting_Temperature_to_Reduce_Electricity_Consumption</a></p><p><a href="https://www.linkedin.com/in/nasywa/">Connect with me on Linkedin</a></p><h3>Another inquiry: When utilizing GridSearchCV, can we obtain the best parameters for each model?</h3><p>Yup! We can obtain the best parameter for each model. <a href="https://learnwithnas.medium.com/find-the-best-parameter-for-each-model-a-sample-project-of-predicting-steels-optimal-temperature-f98547878bf">This will be discussed here</a>.</p><p><strong>References:</strong></p><ol><li>Umer Irshad, Hyperparameter tuning using GridSearchCV<br><a href="https://www.educative.io/answers/hyperparameter-tuning-using-gridsearchcv">https://www.educative.io/answers/hyperparameter-tuning-using-gridsearchcv</a></li><li>cs231n, Convolutional Neural Networks for Visual Recognition<br><a href="https://cs231n.github.io/neural-networks-3/">https://cs231n.github.io/neural-networks-3/</a></li><li>Cameron Waren, Don’t do data science, solve business problems (2019)<br><a href="https://towardsdatascience.com/dont-do-data-science-solve-business-problems-6b70c4ee0083">https://towardsdatascience.com/dont-do-data-science-solve-business-problems-6b70c4ee0083</a></li><li>Shantababu Pandian, A Comprehensive Guide on Hyperparameter Tuning and its Techniques (2022)<br><a href="https://www.analyticsvidhya.com/blog/2022/02/a-comprehensive-guide-on-hyperparameter-tuning-and-its-techniques/">https://www.analyticsvidhya.com/blog/2022/02/a-comprehensive-guide-on-hyperparameter-tuning-and-its-techniques/</a></li><li>Aayush Bajaj, Performance metrics in machine learning [Complete Guide] (2023)<br><a href="https://neptune.ai/blog/performance-metrics-in-machine-learning-complete-guide">https://neptune.ai/blog/performance-metrics-in-machine-learning-complete-guide</a></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/2/1*W7VC7ds6nwA_NRjzmm5VXg.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8d1c86f7f4f0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Sentiment Analysis: Concept and Practical Use Case for Detecting Negative Reviews]]></title>
            <link>https://learnwithnas.medium.com/sentiment-analysis-model-comparison-in-detecting-negative-reviews-using-nlp-7b8973b7a980?source=rss-cbae1ad7c072------2</link>
            <guid isPermaLink="false">https://medium.com/p/7b8973b7a980</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[sentiment-analysis]]></category>
            <dc:creator><![CDATA[Learn with Nas]]></dc:creator>
            <pubDate>Mon, 04 Sep 2023 14:43:16 GMT</pubDate>
            <atom:updated>2023-11-30T13:15:12.355Z</atom:updated>
            <content:encoded><![CDATA[<p>Let’s take a moment to ponder the quirks of human communication. Have you ever thought about how a simple text can carry so many different emotions, like happiness, annoyance, or just not caring? It’s like figuring out a secret code, right? Well, that’s where sentiment analysis comes in. It helps us uncover the hidden feelings in what people write or say. So, get ready to dig in like a detective (you can even grab a magnifying glass if you’re in a Sherlock Holmes mood), as we explore how to understand emotions in text. And don’t worry, we’ll make analyzing data more fun than a knock-knock joke.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y71fooL9MdIrzYbXcnoA3w.png" /><figcaption>Source: LOLJokesforKids</figcaption></figure><h3><strong>How do computers understand Natural Language as we humans do?</strong></h3><p>Natural Language Processing (NLP) refers to the branch of AI — concerned with giving computers the ability to understand text and spoken words in much the same way human beings can [1]. Similar to how humans have different sensors such as eyes and ears, computers have microphones to collect audio and programs to read and process inputs. The inputs are then converted into code that the computer can understand [2]. Sentiment Analysis, a sub-field of NLP, plays an essential role in connecting human emotional comprehension with machine learning technology.</p><h4><strong>The next question is, what is Sentiment Analysis?</strong></h4><p>Humans communicate with each other in a variety of languages, and any language is just a mediator or a way in which we try to express ourselves. And, whatever we say has a sentiment associated with it. It might be positive, negative, or even neutral [3]. Sentiment Analysis is the process of analyzing the emotions or intent behind a piece of text, speech, or any mode of communication [4]. In this article, we will focus on the sentiment analysis of text data.</p><p>Suppose there is an online database that provides information about movies, television shows, actors, directors, producers, and other individuals and entities related to the entertainment industry, called IMDb. IMDb allows registered users to write reviews for movies and TV shows, indicating whether they liked the movies or not. These reviews help users get an idea of the quality and popularity of a particular title.</p><p>I pasted some reviews of ‘Split’ Movie from IMDB website here:</p><p>User Review 1: A fantastic performance by the film’s star, James McAvoy is reason alone to watch this film. Every personality on display is distinct to the other, and he is so interesting to watch. Anya who was breath-taking in The Witch does a fine job here too. This is a film where M. Night Shyamalan reasserts himself as a serious director following a string of poor films. I absolutely cannot wait for the sequel and the conclusion of the Unbreakable series arriving in January. 8.5/10.</p><p>User Review 2: This is not a good movie. This is a very bad movie. It would only scary some one who is easy to scary. The story line is awful. The ending is awful. I do not know why this got a 7.5. That is just overrating it. I give it 4 out of 10. I you want get scared see The Exorcist. That is a very scary movie. This is just pooh pooh. Do not see it. The Exorcist II is also a lot better. It did have some good acting. And the actors were wasting there time being in this awful movie. Do not see this movie. Do not waste your time. Do not waste your money.</p><p>User Review 3: “Split” is not a bad movie by any stretch of the imagination. The acting, the directing, the production value, and the music are all great, and at times are truly masterful. However, the relentlessly bleak ending and unfinished feel of the film kept me from enjoying it.</p><p>The first review is definitely a positive one, signifying that the reviewer was impressed with the movie. The second one is a negative comment, as evident from the reviewer discouraging others from watching the movie. As for the third review, while the reviewer gives compliments, there is a sense that the film feels incomplete, which prevents the reviewer from fully enjoying it. Hence, we can consider this a neutral statement.</p><p>But now, a problem arises: there will be hundreds and thousands of reviews for movies, and after a point in time, it will become nearly impossible to scan through each user review and reach a conclusion. Therefore, this is where the Sentiment Analysis Model comes into play, which processes a vast corpus of data containing reviews, identifies patterns, and derives conclusions based on real evidence, rather than assumptions made on a small sample of data [3].</p><h3>Which libraries were utilized?</h3><ol><li>Basic Python</li></ol><ul><li>Pandas</li><li>Matplotlib</li><li>Seaborn</li></ul><p>2. Text Preprocessing</p><ul><li>For tokenization, I utilized NLTK and spaCy and then compared the results between them. What distinguishes them? NLTK takes strings as input and returns strings or lists of strings as output. Whereas, spaCy uses an object-oriented approach. When we parse a text, spaCy returns a document object whose words and sentences are objects themselves [5].</li><li>I utilized stopwords to remove low-information words and they can also become powerful in adding context [6].</li><li>For text vectorization, I utilized the TfidfVectorizer from Sklearn. TF-IDF is an algorithm that transforms text into a meaningful numerical representation, which is then used to train machine learning algorithms for prediction. TfidfVectorizer considers how often words appear in documents (i.e., relative to corpus) and assigns them weights based on their importance [7].</li></ul><p>3. Evaluation Metrics</p><ul><li><strong>F1</strong> measures a model’s accuracy. It combines the precision and recall scores of a model. The accuracy metric calculates the number of correct predictions made by a model across the entire dataset [8].</li><li><strong>average_precision_score</strong> computes the average precision (AP) from the prediction score. AP summarizes a precision-recall curve as the weighted mean of precisions achieved at the threshold, with the increase in recall from the previous threshold used as the weight [9].</li><li><strong>AUC-ROC </strong>stands for Area Under the Curve (AUC) of the Receiver Operating Characteristics (ROC). The ROC curve is an evaluation metric for binary classification problems. It is a probability curve that plots the TPR (true positive rate) against FPR (false positive rate) at various threshold values and essentially separates the ‘signal’ from the ‘noise.’ In other words, it shows the performance of a classification model at all classification thresholds. The AUC is a measure of the ability of a binary classifier to distinguish between classes and is used as a summary of the ROC curve. The higher the AUC, the better the model’s performance at distinguishing between the positive and negative classes [10].</li></ul><h3>Implementing in Python:</h3><p>Let’s jump in on the first step — understanding the problem and the data.</p><h4>1. Data Overview</h4><p><strong>Introduction</strong></p><p>For a company that collects reviews for movies, TV series, and other infotainment content, manually classifying reviews as positive or negative has become a challenge due to the sheer volume of data. To streamline this process, implementing a machine learning model capable of automatically detecting sentiment from review data could provide a solution.</p><p><strong>Problem</strong></p><p>Predicting the sentiment of review data is driven by several problems and needs:</p><ol><li>Large Volume of Text Data: With the proliferation of user-generated review data, there is a massive volume of text data to process. Manually analyzing all this data for sentiment would be time-consuming and impractical.</li><li>Market Research: Companies need to stay competitive and understand market trends. Sentiment analysis provides insights into public opinion about movies that is trends, helping organizations make data-driven decisions.</li><li>Brand Reputation Management: Monitoring sentiment around a brand is crucial for managing reputation. Detecting negative sentiment early allows companies to respond and mitigate potential damage.</li><li>Content Recommendation: Understanding user preferences and emotions helps offer more relevant content.</li></ol><p><strong>Objective</strong></p><p>To classify positive and negative reviews, a classification model will be trained. The selected model should achieve a minimum F1 score of 0.85.</p><p><strong>The dataset</strong></p><p>The IMDB movie review dataset with polarity labelling will be used. The following are the first few rows of the DataFrame.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*CYF1D_wcGNT5a64BDS78iw.png" /><figcaption>the first 5 rows of the dataframe</figcaption></figure><p><strong>Data description:</strong></p><p>Features:</p><ul><li>tconst: alphanumeric unique identifier of the title</li><li>title_type: the type/format of the title (e.g. movie, short, TV series, TV episode, video, etc.)</li><li>primary_title: the more popular title / the title used by the filmmakers on promotional materials at the point of release</li><li>original_title: original title in the original language</li><li>start_year: represents the release year of a title. In the case of TV Series, it is the series start year</li><li>end_year: TV Series end year. ‘\N’ for all other title types</li><li>runtime_minutes: primary runtime of the title, in minutes</li><li>is_adult: 0: non-adult title; 1: adult title</li><li>genres: includes up to three genres associated with the title</li><li>average_rating: weighted average of all the individual user ratings</li><li>votes: number of votes the title has received</li><li>review: review text</li><li>rating: rating score</li><li>ds_part: train/test for part of the dataset (train/test dataset split)</li></ul><p>Target:</p><ul><li>pos: 0 for negative and 1 for positive.</li></ul><h4><strong>2. Exploratory Data Analysis</strong></h4><ol><li>Check the count of movies and reviews over the course of several years</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1rOs4kvQ83pQu1vSJV3ZDA.png" /><figcaption>The count of reviews over the course of several years</figcaption></figure><p><strong>Findings</strong></p><ol><li>From the distribution plot of the number of reviews per movie above, it can be observed that the highest number of reviews per movie is the value 1, with a count of around 1300.</li><li>From the distribution plot of the KDE for the number of reviews per movie, it can be seen that the value 1 has the highest density value, with a density of around 0.12.</li><li>As additional information, KDE plots are used to visualize the probability distribution of continuous data. KDE is employed to estimate the probability density function of unknown data.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vjKgPYpEZ-7QAgutg6Btdw.png" /><figcaption>The count of ratings over the course of several years</figcaption></figure><p><strong>Findings</strong></p><ol><li>From the two graphs above, both the Train Set and Test Set have the same distribution of data.</li><li>The highest rating is 1, with a count of nearly 5000, followed by rating 10 with a count around 4500.</li><li>The next rankings are 8 and 4.</li><li>Rankings 3 and 7 have a difference in order between the train set and test set.</li><li>In the train set, rating 7 is higher than rating 3, but in the test set, rating 3 is higher than rating 7. Following those, we have rating 2 and lastly, rating 9.</li></ol><p>2. Check the distribution of the number of reviews per movie with accurate counting and KDE (Kernel Density Estimation) (to understand the difference from accurate counting)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8lwm9KvY5zsHBRc9gQSfmw.png" /><figcaption>The distribution of the number of reviews per movie with accurate counting and KDE (Kernel Density Estimation)</figcaption></figure><p><strong>Findings</strong></p><ol><li>From the comparison graphs of the Train Set and Test Set on the distribution of the number of reviews from different polarities per year, it can be observed that both datasets share a common trend, where negative reviews are more prevalent than positive reviews.</li><li>However, there are similarities and differences in the graphs depicting the number of reviews from different polarities per year. In the first rank, the year 2006 emerges as the year with the highest number of reviews in both datasets, with a count above 1600. However, the difference is apparent in the second rank, where in the Train Set, the second rank is in the year 2005. On the other hand, in the Test Set, the second rank is in the year 2007.</li></ol><h4><strong>3. Data Preprocessing</strong></h4><p>Assumption: all models below accept text in lowercase and without numbers, punctuation, etc.</p><p>Data normalization:</p><pre>df_reviews[&#39;review_norm&#39;] = df_reviews[&#39;review&#39;].str.lower().str.replace(&#39;[^a-zA-Z]&#39;, &#39; &#39;)</pre><h4><strong>4. Model Training and Evaluation</strong></h4><p>Function to evaluate the model and visualize the evaluation metric results in a graph.</p><pre>import sklearn.metrics as metrics<br><br>def evaluate_model(model, train_features, train_target, test_features, test_target):<br>    <br>    eval_stats = {}<br>    <br>    fig, axs = plt.subplots(1, 3, figsize=(20, 6)) <br>    <br>    for type, features, target in ((&#39;train&#39;, train_features, train_target), (&#39;test&#39;, test_features, test_target)):<br>        <br>        eval_stats[type] = {}<br>    <br>        pred_target = model.predict(features)<br>        pred_proba = model.predict_proba(features)[:, 1]<br>        <br>        # F1<br>        f1_thresholds = np.arange(0, 1.01, 0.05)<br>        f1_scores = [metrics.f1_score(target, pred_proba&gt;=threshold) for threshold in f1_thresholds]<br>        <br>        # ROC<br>        fpr, tpr, roc_thresholds = metrics.roc_curve(target, pred_proba)<br>        roc_auc = metrics.roc_auc_score(target, pred_proba)    <br>        eval_stats[type][&#39;ROC AUC&#39;] = roc_auc<br><br>        # PRC<br>        precision, recall, pr_thresholds = metrics.precision_recall_curve(target, pred_proba)<br>        aps = metrics.average_precision_score(target, pred_proba)<br>        eval_stats[type][&#39;APS&#39;] = aps<br>        <br>        if type == &#39;train&#39;:<br>            color = &#39;blue&#39;<br>        else:<br>            color = &#39;green&#39;<br><br>        # Scor F1 <br>        ax = axs[0]<br>        max_f1_score_idx = np.argmax(f1_scores)<br>        ax.plot(f1_thresholds, f1_scores, color=color, label=f&#39;{type}, max={f1_scores[max_f1_score_idx]:.2f} @ {f1_thresholds[max_f1_score_idx]:.2f}&#39;)<br>        # Setting thresholds for multiple crossovers.<br>        for threshold in (0.2, 0.4, 0.5, 0.6, 0.8):<br>            closest_value_idx = np.argmin(np.abs(f1_thresholds-threshold))<br>            marker_color = &#39;orange&#39; if threshold != 0.5 else &#39;red&#39;<br>            ax.plot(f1_thresholds[closest_value_idx], f1_scores[closest_value_idx], color=marker_color, marker=&#39;X&#39;, markersize=7)<br>        ax.set_xlim([-0.02, 1.02])    <br>        ax.set_ylim([-0.02, 1.02])<br>        ax.set_xlabel(&#39;threshold&#39;)<br>        ax.set_ylabel(&#39;F1&#39;)<br>        ax.legend(loc=&#39;lower center&#39;)<br>        ax.set_title(f&#39;F1 Score&#39;) <br><br>        # ROC<br>        ax = axs[1]    <br>        ax.plot(fpr, tpr, color=color, label=f&#39;{type}, ROC AUC={roc_auc:.2f}&#39;)<br>        # Setting thresholds for multiple crossovers.<br>        for threshold in (0.2, 0.4, 0.5, 0.6, 0.8):<br>            closest_value_idx = np.argmin(np.abs(roc_thresholds-threshold))<br>            marker_color = &#39;orange&#39; if threshold != 0.5 else &#39;red&#39;            <br>            ax.plot(fpr[closest_value_idx], tpr[closest_value_idx], color=marker_color, marker=&#39;X&#39;, markersize=7)<br>        ax.plot([0, 1], [0, 1], color=&#39;grey&#39;, linestyle=&#39;--&#39;)<br>        ax.set_xlim([-0.02, 1.02])    <br>        ax.set_ylim([-0.02, 1.02])<br>        ax.set_xlabel(&#39;FPR&#39;)<br>        ax.set_ylabel(&#39;TPR&#39;)<br>        ax.legend(loc=&#39;lower center&#39;)        <br>        ax.set_title(f&#39;ROC curve&#39;)<br>        <br>        # PRC<br>        ax = axs[2]<br>        ax.plot(recall, precision, color=color, label=f&#39;{type}, AP={aps:.2f}&#39;)<br>        # Setting thresholds for multiple crossovers.<br>        for threshold in (0.2, 0.4, 0.5, 0.6, 0.8):<br>            closest_value_idx = np.argmin(np.abs(pr_thresholds-threshold))<br>            marker_color = &#39;orange&#39; if threshold != 0.5 else &#39;red&#39;<br>            ax.plot(recall[closest_value_idx], precision[closest_value_idx], color=marker_color, marker=&#39;X&#39;, markersize=7)<br>        ax.set_xlim([-0.02, 1.02])    <br>        ax.set_ylim([-0.02, 1.02])<br>        ax.set_xlabel(&#39;recall&#39;)<br>        ax.set_ylabel(&#39;precision&#39;)<br>        ax.legend(loc=&#39;lower center&#39;)<br>        ax.set_title(f&#39;PRC&#39;)        <br><br>        eval_stats[type][&#39;Accuracy Score&#39;] = metrics.accuracy_score(target, pred_target)<br>        eval_stats[type][&#39;F1&#39;] = metrics.f1_score(target, pred_target)<br>    <br>    df_eval_stats = pd.DataFrame(eval_stats)<br>    df_eval_stats = df_eval_stats.round(2)<br>    df_eval_stats = df_eval_stats.reindex(index=(&#39;Accuracy score&#39;, &#39;F1&#39;, &#39;APS&#39;, &#39;ROC AUC&#39;))<br>    <br>    print(df_eval_stats)<br>    <br>    return</pre><p><strong>Model 0: Dummy Classifier Model (baseline)</strong></p><pre>df = DummyClassifier(strategy = &#39;constant&#39;, constant = 1)<br>model_0 = df.fit(X = df_reviews_train, y = train_target)</pre><p>Using model_0 to make predictions on both training and test datasets and then evaluate the model’s performance using some evaluation metrics</p><pre>evaluate_model(model_0, np.empty(len(train_target)), train_target, np.empty(len(test_target)), test_target)</pre><p>Results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2FUsBxHfIP1oEwUFEF7nDQ.png" /></figure><p><strong>Model 1: Logistic Regression Model, utilizing NLTK and TF-IDF for text preprocessing</strong></p><p>Begin by removing common English stop words and subsequently converting the text to lowercase:</p><pre>tfidf_nltk = TfidfVectorizer(stop_words = stopwords.words(&#39;english&#39;), lowercase = True)</pre><p>Transforming the text data into a TF-IDF matrix. The TF-IDF Vectorizer used for both datasets has already been configured to apply the same text preprocessing steps as specified earlier. This is important to ensure consistency between the training and test text preprocessing. The resulting train_features_tfidf_nltk and test_features_tfidf_nltk variables will contain the TF-IDF representations of the text data, which can be used as input features in the machine learning model:</p><pre>train_features_tfidf_nltk = tfidf_nltk.fit_transform(df_reviews_train[&#39;review_norm&#39;])<br><br>test_features_tfidf_nltk = tfidf_nltk.transform(df_reviews_test[&#39;review_norm&#39;])</pre><p>Training the Logistic Regression model on the training data.</p><pre>lr = LogisticRegression()<br>model_1 = lr.fit(train_features_tfidf_nltk, train_target)</pre><p>Using model_1 to make predictions on both training and test datasets and then evaluate the model’s performance using some evaluation metrics</p><pre>evaluate_model(model_1, train_features_tfidf_nltk, train_target, test_features_tfidf_nltk, test_target)</pre><p>Results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vFtuidJSTr7QS1aVrsw5ig.png" /></figure><p>Finding:</p><p>The accuracy values on the train set and test set have a difference of about 6 points. This could be one indicator that the model is experiencing overfitting.</p><p><strong>Model 2: Utilizing the same variables of preprocessed text data as in Model 1, but in Model 2, I performed Hyperparameter Tuning</strong></p><p>L1 regularization using the ‘liblinear’ solver and a regularization parameter C of 0.85 was applied to the training dataset.</p><pre>tuned_lr = LogisticRegression(penalty = &#39;l1&#39;, solver = &#39;liblinear&#39;, C = 0.85)<br>model_2 = tuned_lr.fit(train_features_tfidf_nltk, train_target)</pre><p>Using model_2 to make predictions on both training and test datasets and then evaluate the model’s performance using some evaluation metrics</p><pre>evaluate_model(model_2, train_features_tfidf_nltk, train_target, test_features_tfidf_nltk, test_target)</pre><p>Results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BTfEv0ZSRsym6vQG7bvKNA.png" /></figure><p>Findings:</p><p>From the results above, it can be observed that the model is no longer experiencing overfitting, as the difference between the train set and test set is small.</p><p><strong>Model 3: Logistic Regression Model, utilizing spaCy and TF-IDF for text preprocessing</strong></p><p>I created a function for text processing. It uses spacy to process the input text. The ‘nlp’ object is used to analyze the text. The code tokenizes the text and performs lemmatization, which is the process of reducing words to their base or dictionary form (lemmas). For each token in the document, it first lemmatizes the token, and then it lemmatizes the token again without considering stop words. The resulting lemmatized tokens are stored in the tokens list. Finally, the function joins the lemmatized tokens into a single string using spaces and returns the preprocessed text.</p><pre>def text_preprocessing_3(text):<br>    <br>    doc = nlp(text)<br>    tokens = [token.lemma_ for token in doc if not token.is_stop]<br>    tokens = [token.lemma_ for token in doc]<br>    <br>    return &#39; &#39;.join(tokens)</pre><p>Applied the above function to preprocess the ‘review’ text in the two data frames (df_reviews_train and df_reviews_test). It applies a lambda function to each row in the ‘review’ column. The preprocessed text is stored in new ‘review_tokenizer3’ columns in the respective DataFrames.</p><pre>df_reviews_train[&#39;review_tokenizer3&#39;] = df_reviews_train[&#39;review&#39;].apply(lambda x: text_preprocessing_3(x))<br><br>df_reviews_test[&#39;review_tokenizer3&#39;] = df_reviews_test[&#39;review&#39;].apply(lambda x: text_preprocessing_3(x))<br><br></pre><p>Text data is prepared for machine learning by converting it into numerical features using TF-IDF vectorization technique. It computes TF-IDF values for both the training and test datasets, which can then be used as input features for the machine-learning model.</p><pre>tfidf_spacy = TfidfVectorizer()<br><br>train_features_tfidf_spacy = tfidf_spacy.fit_transform(df_reviews_train[&#39;review_tokenizer3&#39;])<br><br>test_features_tfidf_spacy = tfidf_spacy.transform(df_reviews_test[&#39;review_tokenizer3&#39;])</pre><p>Training the Logistic Regression model on the training data.</p><pre>lr3 = LogisticRegression()<br>model_3 = lr3.fit(train_features_tfidf_spacy, train_target)</pre><p>Using model_3 to make predictions on both training and test datasets and then evaluate the model’s performance using some evaluation metrics.</p><pre>evaluate_model(model_3, train_features_tfidf_spacy, train_target, test_features_tfidf_spacy, test_target)</pre><p>Results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o9UqrVm6VeGVaNQH7sLXYw.png" /></figure><p>Findings:</p><p>The accuracy values on the train set and test set have a difference of about 5 points. This could be one indicator that the model is experiencing overfitting.</p><p><strong>Model 4: Utilizing the same variables of preprocessed text data as in Model 3, but in Model 4, I performed Hyperparameter Tuning</strong></p><p>Training the logistic regression model with hyperparameter: L1 (Lasso) regularization, ‘liblinear’ solver, which is suitable for L1 regularization and sets the regularization parameter to 0.85.</p><pre>tuning_lr3 = LogisticRegression(penalty = &#39;l1&#39;, solver = &#39;liblinear&#39;, C = 0.85)<br>model_4 = tuning_lr3.fit(train_features_tfidf_spacy, train_target)</pre><p>Using model_4 to make predictions on both training and test datasets and then evaluate the model’s performance using some evaluation metrics.</p><pre>evaluate_model(model_4, train_features_tfidf_spacy, train_target, test_features_tfidf_spacy, test_target)</pre><p>Results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zvH5bMtBEAZp2egPGJxkbw.png" /></figure><p>Findings:</p><p>it can be seen that the values in Model 4 (after hyperparameter tuning in Model 3) are the same as those in Model 2, but with significantly different processing times.</p><p>When using the spaCy library, lemmatization processing takes a long time. Considering this, Model 2 is preferable over Model 4.</p><p><strong>Model 5: LGBMClassifier model, utilizing the same variables of preprocessed text data as in Model 3</strong></p><p>Training the LGBM Classifier model with hyperparameter: Learning Rate = 0.01 and max depth = 10.</p><pre>model_5 = LGBMClassifier(learning_rate = 0.01, max_depth = 10)<br>model_5.fit(train_features_tfidf_spacy, train_target)</pre><p>Using model_5 to make predictions on both training and test datasets and then evaluate the model’s performance using some evaluation metrics.</p><pre>evaluate_model(model_5, train_features_tfidf_spacy, train_target, test_features_tfidf_spacy, test_target)</pre><p>Results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EbOSVlqxvNLwGy0YOTxdcg.png" /></figure><p>Findings:</p><p>Model 4 and Model 2 have higher evaluation scores than Model 5.</p><h4><strong>5. Used the trained models to make predictions on the self-created reviews dataset</strong></h4><p>The self-created dataset after I implemented text preprocessing:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6_ppId2xq5QPi6Xrh5yIFw.png" /></figure><p><strong>Used Model 3 (Logistic Regression Model using spaCy and TF-IDF for text preprocessing) to make predictions</strong></p><pre>texts = my_reviews[&#39;review_norm&#39;]<br><br>my_reviews_pred_prob = model_3.predict_proba(tfidf_spacy.transform(texts.apply(lambda x: text_preprocessing_3(x))))[:, 1]<br><br>for i, review in enumerate(texts.str.slice(0, 100)):<br>    print(f&#39;{my_reviews_pred_prob[i]:.2f}:  {review}&#39;)</pre><p>Result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3THiNQMIp2z5j9f1Y5KznQ.png" /></figure><p>6 out of 8 are correct. Pretty accurate.</p><p><strong>Used Model 5 (LGBMClassifier Model, utilizing spaCy and TF-IDF for text preprocessing) to make predictions</strong></p><pre>texts = my_reviews[&#39;review_norm&#39;]<br><br>tfidf_vectorizer_4 = tfidf_spacy<br>my_reviews_pred_prob = model_5.predict_proba(tfidf_vectorizer_4.transform(texts.apply(lambda x: text_preprocessing_3(x))))[:, 1]<br><br>for i, review in enumerate(texts.str.slice(0, 100)):<br>    print(f&#39;{my_reviews_pred_prob[i]:.2f}:  {review}&#39;)</pre><p>Result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1M04X1YaNYNsCDciJOuMfA.png" /></figure><p>The accuracy is worse than the Logistic Regression Model.</p><h3>General Conclusion</h3><p>Good accuracy in machine learning is subjective. According to a reference, a model with a 70% is considered to have good performance. In fact, an accuracy measure falling between 70%-90% is not only ideal but also realistic. If the accuracy falls below this range, it may be worth consulting a data scientist to understand what’s happening [11].</p><p>If we examine the model performance results below, it becomes evident that all trained models demonstrate good performance. However, a notable drop in the F1 score is observed between the training and testing datasets for Models 1 and 3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hR6PvNelnbyOGdoUItr8aw.png" /></figure><p>While the disparity does not conclusively prove overfitting, it does suggest that these models might not generalize as effectively to new, unseen data compared to their performance on the training data.</p><p>In Models 2 and 4, the application of L1 regularization penalty parameters during hyperparameter tuning assists in narrowing the gap by encouraging the model to generalize better.</p><p>L1 regularization (Lasso) adds a penalty to the loss function based on the absolute value of the coefficients of the features. This penalty encourages the model to reduce the coefficients of less important features down to zero, effectively performing feature selection [12].</p><p>Using the ‘liblinear’ solver in logistic regression is suitable for L1 regularization. This solver works well with L1 regularization and is efficient for problems where the dataset is not extremely large [13].</p><p>Setting the regularization parameter to 0.85 indicates a higher penalty for the magnitude of the coefficients. This stronger regularization encourages the model to reduce the impact of less important features even more, effectively simplifying the model by pushing more coefficients toward zero [12].</p><p>Here’s a refined version:</p><p>Models 2 and 4 exhibit identical performance but differ in the libraries they utilize. Model 2 utilizes NLTK, while Model 4 utilizes spaCy. Both NLTK and spaCy serve as robust libraries for constructing NLP systems.</p><p>In comparison to NLTK, spaCy offers enhanced utility in development and production environments due to its notably rapid and precise semantic analysis capabilities. While researchers often favour NLTK for its diverse range of algorithms, simplifying the execution of specific tasks, spaCy stands out for its efficiency and accuracy in real-world applications [14].</p><p>The selection of the optimal library for your project depends entirely on your use case and the context in which it will be applied.</p><p>Thank you for dedicating your time to reading this article! Stay tuned for more insightful content in my upcoming articles :)</p><p>You can visit my GitHub account for the complete code:</p><p><a href="https://github.com/nasywa-ibtisamah/Sentiment-Analysis-Model-Comparison-Detecting-Negative-Reviews-with-NLP-and-Multiple-Metrics">GitHub - nasywa-ibtisamah/Sentiment-Analysis-Model-Comparison-Detecting-Negative-Reviews-with-NLP-and-Multiple-Metrics</a></p><p><a href="https://www.linkedin.com/in/nasywa/">Connect with me on Linkedin</a></p><p>Reference:</p><ol><li>IBM, What is Natural Language Processing?<br><a href="https://www.ibm.com/id-en/topics/natural-language-processing">https://www.ibm.com/id-en/topics/natural-language-processing</a></li><li>Ben Luketvich, Definition of Natural Language Processing<br><a href="https://www.techtarget.com/searchenterpriseai/definition/natural-language-processing-NLP">https://www.techtarget.com/searchenterpriseai/definition/natural-language-processing-NLP</a></li><li>Nikhil Raj, Starters Guide to Sentiment Analysis using Natural Language Processing (2023)<br><a href="https://www.analyticsvidhya.com/blog/2021/06/nlp-sentiment-analysis/">https://www.analyticsvidhya.com/blog/2021/06/nlp-sentiment-analysis/</a></li><li>Shashank Gupta, Sentiment Analysis: Concept, Analysis and Applications (2018)<br><a href="https://towardsdatascience.com/sentiment-analysis-concept-analysis-and-applications-6c94d6f58c17">https://towardsdatascience.com/sentiment-analysis-concept-analysis-and-applications-6c94d6f58c17</a></li><li>Akanksha Malhotra, Introduction to Libraries of NLP in Python — NLTK vs. spaCy (2018)<br><a href="https://medium.com/@akankshamalhotra24/introduction-to-libraries-of-nlp-in-python-nltk-vs-spacy-42d7b2f128f2#:~:text=NLTK%20is%20a%20string%20processing,and%20sentences%20are%20objects%20themselves">https://medium.com/@akankshamalhotra24/introduction-to-libraries-of-nlp-in-python-nltk-vs-spacy-42d7b2f128f2#:~:text=NLTK%20is%20a%20string%20processing,and%20sentences%20are%20objects%20themselves</a>.</li><li>Kavita Ganeshan, What Are Stop Words?<br><a href="https://www.opinosis-analytics.com/knowledge-base/stop-words-explained/#:~:text=Stop%20words%20are%20a%20set,carry%20very%20little%20useful%20information">https://www.opinosis-analytics.com/knowledge-base/stop-words-explained/#:~:text=Stop%20words%20are%20a%20set,carry%20very%20little%20useful%20information</a>.</li><li>Mukesh Chaudhary, TF-IDF scikit learn (2020)<br><a href="https://medium.com/@cmukesh8688/tf-idf-vectorizer-scikit-learn-dbc0244a911a">https://medium.com/@cmukesh8688/tf-idf-vectorizer-scikit-learn-dbc0244a911a</a></li><li>Ruhut Kundu, F1 Score in Machine Learning: Intro &amp; Calculation (2022)<br><a href="https://www.v7labs.com/blog/f1-score-guide#:~:text=for%20Machine%20Learning-,What%20is%20F1%20score%3F,prediction%20across%20the%20entire%20dataset">https://www.v7labs.com/blog/f1-score-guide#:~:text=for%20Machine%20Learning-,What%20is%20F1%20score%3F,prediction%20across%20the%20entire%20dataset</a>.</li><li>Sklearn documentation, average precision score metric<br><a href="https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html">https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html</a></li><li>Aniruddha Bhandari, Guide to AUC ROC Curve in Machine Learning: What Is Specificity? (2023)<br><a href="https://www.analyticsvidhya.com/blog/2020/06/auc-roc-curve-machine-learning/">https://www.analyticsvidhya.com/blog/2020/06/auc-roc-curve-machine-learning/</a></li><li>Kirsten Barkved, How To Know if Your Machine Learning Model Has Good Performance (2022)<br><a href="https://www.obviously.ai/post/machine-learning-model-performance">https://www.obviously.ai/post/machine-learning-model-performance</a></li><li>Dinesh Kumar, A Complete Understanding of LASSO Regression (2023)<br><a href="https://www.mygreatlearning.com/blog/understanding-of-lasso-regression/">https://www.mygreatlearning.com/blog/understanding-of-lasso-regression/</a></li><li>Sklearn Documentation, Logisctic Regression CV<br><a href="https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegressionCV.html#:~:text=This%20class%20implements%20logistic%20regression,only%20for%20the%20L2%20penalty">https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegressionCV.html#:~:text=This%20class%20implements%20logistic%20regression,only%20for%20the%20L2%20penalty</a>.</li><li>seaflux, NLTK vs spaCy — Python based NLP libraries and their functions<br><a href="https://www.seaflux.tech/blogs/NLP-libraries-spaCy-NLTK-differences">https://www.seaflux.tech/blogs/NLP-libraries-spaCy-NLTK-differences</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7b8973b7a980" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>