<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by misun_song on Medium]]></title>
        <description><![CDATA[Stories by misun_song on Medium]]></description>
        <link>https://medium.com/@msong507?source=rss-f052cdb2ea66------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*qy7uIQpZY_sktgJECtbEDQ.jpeg</url>
            <title>Stories by misun_song on Medium</title>
            <link>https://medium.com/@msong507?source=rss-f052cdb2ea66------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 19:48:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@msong507/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[AI and the Workforce Analysis with NLP]]></title>
            <link>https://medium.com/@msong507/ai-and-the-workforce-analysis-with-nlp-bff1ce0783c9?source=rss-f052cdb2ea66------2</link>
            <guid isPermaLink="false">https://medium.com/p/bff1ce0783c9</guid>
            <category><![CDATA[workforce-development]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[future-of-work]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <dc:creator><![CDATA[misun_song]]></dc:creator>
            <pubDate>Thu, 01 Feb 2024 04:59:31 GMT</pubDate>
            <atom:updated>2024-02-01T04:59:31.947Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Insights and Future Directions</strong></p><h3>The Code Behind the Project</h3><p><a href="https://github.com/msong507/nlp_ai_job">GitHub - msong507/nlp_ai_job: Analyzing AI&#39;s Industry Impact and Employment via Natural Language Processing</a></p><h3><strong><em>AI’s Impact on Industry and Employment</em></strong></h3><p>In exploring the integration of AI into industries, a critical insight emerges: by examining historical and current data, we can predict the potential impact on our workforce. This article discusses an NLP project that analyzed 200,000 news articles to identify tasks and professions on the brink of AI transformation. Through this analysis, we aim to strategize on job automation and productivity enhancements, offering insight into the evolving employment landscape in the AI era.</p><h3><strong>Source Data Overview</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/594/1*Ddy-GJMyC1oI-Vuld3GUQw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/510/1*fL0TPu4829QD7WBFfUXNKg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/536/1*-1uxNy-XzLQsfkX_THhl3g.png" /><figcaption>The news article count increased each year.</figcaption></figure><ol><li><strong><em>Total Entries:</em></strong> 199,677</li><li><strong><em>Columns</em></strong>: URL, Date, Language, Title, Text</li><li><strong><em>Data Types:</em></strong> Textual: Object (4 columns), Temporal: DateTime (1 column)</li><li><strong><em>Coverage Period:</em></strong> January 1, 2020, to October 16, 2023 (1,095 days)</li></ol><h3><strong>Data Clean-Up and Filtering</strong></h3><p>Following initial data preparation, we implemented various levels of data cleansing, resulting in <em>three distinct datasets</em>, each tailored for specific analytical objectives.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y_vYC11d8AXmtnf4ogVLzQ.png" /></figure><h3><strong>Topic Detection</strong></h3><h4><strong>LDA Analysis with Gensim</strong></h4><p>We observed an <em>extensive impact of AI </em>on both the job market and industrial landscape through Latent Dirichlet Allocation (LDA) analysis with Gensim.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/628/1*N-5h6ovUPGuhPQSNwOUu-A.png" /></figure><p><strong><em>Detected Keywords:</em></strong> ai, data, technology, business, company, human, work, chatgpt, solution, platform, research, forecast, intelligence, stock, financial, investment, fund, health, school, policy</p><p><strong><em>Detected Topics: </em></strong>AI and Workforce, AI and Technology, AI and Business, AI and Market Intelligence, AI and Finance, AI and Healthcare, AI and Education</p><h4>LDA Analysis with <strong>Ktrain</strong></h4><p>Key industrial trends in AI’s impact on <em>healthcare, finance, and security, </em>and the rising role of <em>generative AI in content creation, </em>are evident through time-based analysis using LDA with Ktrain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u8f_0hUevFXilZtOCDYV8w.png" /></figure><h4>— 2020</h4><p><strong><em>Detected Keywords:</em></strong><em> </em>Market, Business, Work Evolution, Cloud Computing, Cyber(Digital) Security</p><p><strong><em>Detected Topics:</em></strong><em> </em>AI’s impact on market analysis and business strategy, AI’s effect on the job market and the evolution of work, AI’s integration in cloud computing and security</p><h4>— 2021</h4><p><strong><em>Detected Keywords:</em></strong><em> </em>Market Intelligence, Healthcare (Diagnostics), Stock Market, Finance</p><p><strong><em>Detected Topics:</em></strong><em> </em>AI’s role in market intelligence and predictive analytics, AI’s advancement of healthcare diagnostics and patient care, AI’s automation of stock market trading and financial analysis</p><h4>— 2022</h4><p><strong><em>Detected Keywords:</em></strong><em> </em>Cyber(Digital) Security, Healthcare (Diagnostics), Finance</p><p><strong><em>Detected Topics:</em></strong><em> </em>AI’s strengthening of digital security and user privacy, AI’s enhancement of medical diagnostics and treatment, AI’s automation of financial trading and market analysis</p><h4>— 2023</h4><p><strong><em>Detected Keywords:</em></strong><em> </em>Generative AI, Content Creation, Cyber(Digital) Security, Finance, Entertainment, Healthcare</p><p><strong><em>Detected Topics:</em></strong><em> </em>Generative AI in content creation, AI norms in legal and cybersecurity, AI in financial analytics and investing, AI and personalized entertainment experiences, and AI in healthcare communication</p><h3><strong>Sentiment Analysis</strong></h3><h4><strong>Model Comparison</strong></h4><p><em>VADER</em> was chosen for its better handling of <em>diverse expressions </em>in news articles, compared to TextBlob and Yelp models. A model trained on Yelp reviews may not perform well with news articles due to different language styles and contexts. TextBlob is limited by its predefined lexicon and may struggle with complex language and context in news.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-jbIRSRHRHsRA6-EoIOaNg.png" /></figure><h4>Overtime View with <strong>Positive News Articles</strong></h4><p>Sentiment analysis over time reveals key patterns, suggesting the need to combine it with topic modeling to uncover the root causes of these sentiment trends.</p><p>News articles with positive sentiment have shown an increase over time, surging by 192% from 2022 to 2023.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T4-Gwjtn2BvxRm1I2apuvg.png" /></figure><p>This uptrend is likely linked to the emergence of Generative AI, such as ChatGPT, and its growing applications in sectors like finance and healthcare, as revealed by our LDA analysis.</p><p>Averaging positive and negative scores separately can provide insights into the intensity and prevalence of each sentiment type within the data.</p><p>The monthly average compound scores from news articles, generally between 0.97 and 0.99, indicate a strong positive sentiment over the years. However, a yearly downward trend has emerged since 2021.</p><h4>Overtime View with <strong>Negative News Articles</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y1IoqgSuf1iRPLI25RiiMQ.png" /></figure><p>The proportion of negative sentiment in the data is small, accounting for 5% compared to 95% for positive sentiment.</p><p>Despite this, the incidence of negative sentiment in news articles has risen sharply, increasing by 350% from 2022 to 2023.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y1IoqgSuf1iRPLI25RiiMQ.png" /></figure><h3><strong>Topic Modeling</strong></h3><p>After analyzing sentiments with VADER, we used BERTopic for topic modeling. BERTopic’s transformer-based approach provides deeper, context-aware insights compared to traditional bag-of-words models.</p><p>Combining BERTopic with sentiment analysis, we gained clearer insights into major topics: Positive sentiments focused on AI’s transformative impact on industries, while negative sentiments centered on ethical and moral concerns.</p><h4><strong>Positive Sentiment</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KWkoutCJb2m4ySplcScyZg.png" /></figure><h4><strong>Negative Sentiment</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sIqlemfhuAFxTqWRZE5IbQ.png" /></figure><h3><strong>Entity Identification</strong></h3><p>Through Named Entity Recognition (NER with spaCy), we effectively pinpointed crucial entities pertaining to <em>organizations (ORG)</em>, <em>geographical locations (GPE)</em>, and <em>individuals (PERSON). Each entity type reflects a growing interest in generative AI, especially ChatGPT.</em></p><p>Utilizing NER, we identified key companies, academic institutions, and government entities driving AI advancements that reshape employment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tddipLZAjFQIwHDcR3hyYg.png" /></figure><h4><strong>Organization</strong></h4><ul><li><strong><em>Recognized Entities: </em></strong>Google, Bard, Microsoft, ChatGPT, Meta, NVIDIA, IBM, Amazon, YouTube, Apple, Tesla</li><li><strong><em>Description: </em></strong>The frequency of organizations named about generative AI has significantly increased since 2022.</li></ul><h4><strong>Location</strong></h4><ul><li><strong><em>Recognized Entities: </em></strong>US, India, China, UK, Russia, Canada, Taiwan, Germany, France, Japan</li><li><strong><em>Description: </em></strong>The United States leads by a significant margin, followed by India and China. Notably, both ChatGPT and Bard, leading Generative AI models, were developed in the U.S.</li></ul><h4><strong>Individuals</strong></h4><ul><li><strong><em>Recognized Entities: </em></strong>Elon Musk (Tesla CEO), Sam Altman (OpenAI CEO), Mark Zuckerberg (Meta CEO), Jesen Huang (NVIDIA CEO), Satya Nadella (Microsoft CEO)</li><li><strong><em>Description: </em></strong>The increased prominence of Sam Altman of OpenAI and Satya Nadella of Microsoft correlates with the rise of ChatGPT and generative AI.</li></ul><h4><strong>AI Keywords</strong></h4><ul><li><strong><em>Recognized Entities: </em></strong>AI, Generative AI, ML, Data Science, Big Data</li><li><strong><em>Description: </em></strong>The emergence of AI in discussions has spiked following the introduction of generative AI.</li></ul><h3><strong>Entity Identification with Topic Modeling</strong></h3><p>Technically, AI can revolutionize content creation, model building, image generation, and digital security, but requires careful navigation of ethical and regulatory landscapes for responsible use.</p><h4><strong><em>What types of companies are planning to invest in AI?</em></strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TzPFhQXU1Hfd42OLbfyJbA.png" /></figure><h4><strong>— Technology</strong></h4><ul><li>Google (Bard), Microsoft (ChatGPT), Amazon, Meta, IBM, SAS, Oracle, Quantum</li></ul><h4><strong>— Healthcare</strong></h4><ul><li>FDA, NHS, Aidoc, WAYCEN, WHO, Novant Health, Google, KLAS, AstraZeneca, Videra Health</li></ul><h4><strong>— Business Intelligence</strong></h4><ul><li>Microsoft, Google, Amazon, IBM, SAS, Oracle</li></ul><h4><strong>— Hardware / Consumer Electronics</strong></h4><ul><li>Apple, NVIDIA, Intel, Samsung</li></ul><h4><strong>— Entertainment</strong></h4><ul><li>Google (YouTube), Meta, Apple, Sony, TikTok, Instagram</li></ul><h4><strong><em>What types of applications cannot currently be transformed by AI, based on today’s state of technology?</em></strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sM0tg-zshOhaL6b8GipPAg.png" /></figure><h4><strong>— Ethical Judgement and Privacy Concerns</strong></h4><ol><li>Generative AI, such as ChatGPT, raises concerns over <strong>cheating and plagiarism </strong>in content creation.</li><li>Gebru’s dismissal from Google spotlighted <strong>AI model biases, </strong>emphasizing the need for AI fairness.</li><li>The rise of <strong>deepfake pornography, </strong>unaddressed by specific laws, poses a significant threat to personal privacy.</li></ol><h4><strong>— Digital Security</strong></h4><ul><li>AI’s reliance on historical data makes it vulnerable to new, unrecognized <strong>cyber threats.</strong></li></ul><h3><strong>Targeted Sentiment Identification</strong></h3><h4>Positive Sentiment</h4><p>Leveraging topics identified by BERTopic / Ktrain and associated articles, we pinpointed three prime sectors for AI integration: <em>Healthcare, Finance, and Business Intelligence.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g72Yy3kTr_L4jgUrjU1Dyg.png" /></figure><h4><strong>— Healthcare</strong></h4><ul><li><strong><em>Job Enhancement:</em> </strong>Medical researchers, bioinformaticians, and radiologists</li><li><strong><em>Implication: </em></strong>AI integration in healthcare promises improved patient outcomes through <strong><em>advanced diagnostics </em></strong>and <strong><em>personalized treatment </em></strong>but raises concerns about data privacy and the need for ethical guidelines.</li></ul><h4><strong>— Finance</strong></h4><ul><li><strong><em>Job Enhancement: </em></strong>Financial analysts and risk management professionals</li><li><strong><em>Implication: </em></strong>AI’s role in finance, particularly in algorithmic <strong><em>trading and risk assessment, </em></strong>enhances efficiency but also necessitates stringent regulatory compliance to prevent malpractices and ensure financial stability.</li></ul><h4><strong>— Business Intelligence</strong></h4><ul><li><strong><em>Job Enhancement: </em></strong>Data analysts, market researchers, and strategic planners</li><li><strong><em>Implication: </em></strong>AI’s role in business intelligence revolutionizes data analysis and <strong><em>decision-making,</em></strong> leading to more informed strategies but also requiring sophisticated data handling and interpretation skills.</li></ul><h4><strong>Negative Sentiment</strong></h4><p>We identified key AI integration technologies: <em>Generative AI, AI Governance and Compliance, and Entertainment. </em>AI Governance and Compliance, uniquely analyzed with negative sentiment like cheating, bias, and manipulation, is anticipated to expand as it tackles these specific challenges.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5o-1BgmbEknwrgWpoHlqmg.png" /></figure><h4>— Generative AI</h4><ul><li><strong><em>Job Enhancement: </em></strong>AI Model Developers, AI Researchers, Product Managers, Data Analysts</li><li><strong><em>Implication: </em></strong>This technology streamlines the <em>creative process </em>by autonomously producing <em>complex content, </em>which could drastically alter the landscape for creators and content-related services.</li></ul><h4>— AI Governance and Compliance</h4><ul><li><strong><em>Job Enhancement: </em></strong>Compliance Managers, Legal Advisors, Ethical AI Advisors</li><li><strong><em>Implication: </em></strong>As AI integrates deeper into various sectors, the need for robust governance and compliance frameworks to ensure <em>ethical, fair, and legal AI usage</em> becomes critical.</li></ul><h4>— Entertainment (Music)</h4><ul><li><strong><em>Job Enhancement: </em></strong>Music Producers, Marketing Analysts, Content Curators</li><li><strong><em>Implication: </em></strong>AI redefines the music landscape, <em>augmenting creativity and personalization,</em> which may shift traditional roles towards tech-centric music production and curation.</li></ul><h4><strong><em>ChatGPT</em> and <em>Bard</em></strong></h4><p>In our BERTopic analysis, we recognized generative AI as pivotal. Integrating NER, we pinpointed <em>ChatGPT</em> and <em>Bard</em> as key players in this domain. Leveraging these key entities, we advanced our analysis to track sentiment trends over time, focusing specifically on these industry leaders.</p><p>The average compound score shows public sentiment towards ChatGPT and Bard is similar. <strong><em>However, ChatGPT has a more significant media presence, both positive and negative, suggesting a higher potential to lead in generative AI. </em></strong>Meanwhile, Bard may face reputation challenges, evidenced by increasing negative media coverage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*REctodmz_BhCqcTnNdDP_A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*KXitUElnv2F2QGaFCxjIHA.png" /></figure><h3><strong><em>Key Insights</em></strong></h3><h4>AI Integration Hotspots</h4><ul><li><strong><em>Healthcare, Finance, and Business Intelligence</em></strong> as early-stage leaders in AI adoption.</li><li>While Finance and Business Intelligence show a cooling trend, Healthcare’s engagement with AI continues to surge.</li></ul><h4>AI’s Innovative Applications</h4><ul><li>The introduction of <strong><em>generative AI </em></strong>like ChatGPT has revolutionized content personalization, expanding into sectors like <strong><em>entertainment,</em></strong> while also giving rise to crucial sectors such as <strong>AI governance and digital security</strong> due to emerging ethical and security challenges.</li></ul><h4>AI Development Accelerators</h4><ul><li>Tech giants like <strong><em>Google</em></strong> and <strong><em>Microsoft</em></strong> are at the forefront of AI evolution, with their generative AI tools leading the charge.</li><li>Influential institutions, such as <strong><em>MIT,</em></strong> and <strong><em>NASA,</em></strong> are vital to this advancement, showcasing a diverse ecosystem driving AI forward.</li></ul><h4>AI Investment Dynamics</h4><ul><li>Tech giants <strong><em>(Google, Microsoft, Amazon, Meta, IBM, Intel)</em></strong> are set to advance AI integration, with <strong><em>NVIDIA’s</em></strong> semiconductors boosting computational needs.</li><li>Notable players also include consumer goods <strong><em>(Apple, Samsung),</em></strong> social media <strong><em>(YouTube, TikTok, Instagram),</em></strong> and <strong><em>Tesla</em></strong> in the auto industry.</li></ul><h4>AI’s Current Boundaries</h4><ul><li>The technology faces challenges such as maintaining <strong><em>content integrity,</em></strong> preventing <strong><em>bias,</em></strong> regulating <strong><em>image/video manipulation,</em></strong> and defending against <strong><em>cyber threats.</em></strong></li><li>These areas underscore the vital necessity for human intervention alongside AI.</li></ul><h3><strong>Actionable Recommendations</strong></h3><h4><strong>General Recommendations</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gOdn1L_cmskWn_4LyX0EvQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5M5lGFBamtLmQd2s1RmV2A.png" /></figure><h4>— Embrace AI Momentum</h4><ul><li>Recognize the swift adoption of AI as a critical business driver and invest in AI technologies to stay ahead.</li></ul><h4>— Leverage Positive AI Sentiments</h4><ul><li>Capitalize on the prevailing positive sentiment by promoting AI’s benefits in your services and products.</li></ul><h4>— Prioritize AI in Key Sectors</h4><ul><li>Focus AI development in healthcare, finance, and business intelligence for maximum impact and ROI.</li></ul><h4>— Expand into Generative AI</h4><ul><li>Incorporate generative AI, especially in fields like entertainment for innovative customer experiences.</li></ul><h4>— Engage with AI Innovations</h4><ul><li>Stay informed by leading AI firms and consider adopting their tools and platforms to your business operations.</li></ul><h4>— Develop AI Risk Management</h4><ul><li>Establish AI governance and risk management strategies to mitigate ethical and security challenges.</li></ul><h4><strong>Industry-Specific Recommendations</strong></h4><h4><strong>— Generative AI</strong></h4><ul><li><em>Generative AI </em>holds the transformative potential to automate complex tasks, innovate solutions, and personalize experiences across all industries.</li><li>Automate content creation to reduce production times.</li><li>Generate synthetic datasets while maintaining privacy.</li><li>Offer customized user experiences in digital services.</li></ul><h4><strong>— Healthcare</strong></h4><ul><li>Deploy AI for swift and accurate medical diagnoses.</li><li>Optimize patient scheduling and medication management.</li><li>Enhance telemedicine with AI for initial assessments.</li></ul><h4><strong>— Finance</strong></h4><ul><li>Utilize AI for market analysis and risk management.</li><li>Strengthen fraud detection with pattern recognition algorithms.</li><li>Integrate AI chatbots for efficient customer service.</li></ul><h4><strong>— Business Intelligence</strong></h4><ul><li>Automate data analytics for insightful business reporting.</li><li>Predict market trends to inform proactive strategies.</li><li>Streamline business processes for increased productivity.</li></ul><h4><strong>— Entertainment</strong></h4><ul><li>Deploy AI for customized user and music recommendations on streaming services.</li><li>Create AI-powered interactive media for personalized storytelling and music experiences.</li></ul><h4><strong>— AI Governance and Compliance</strong></h4><ul><li>Create ethical AI guidelines to guide responsible usage.</li><li>Monitor compliance with automated AI tools.</li><li>Regularly audit AI systems for bias and performance.</li></ul><h3>Conclusion</h3><p>In summary, AI integration is advancing swiftly, with healthcare leading the way and finance and business intelligence sectors experiencing a slower pace. The introduction of generative AI, like ChatGPT, has revolutionized content personalization, particularly in entertainment. To capitalize on this momentum, businesses should prioritize investments in key sectors, expand into generative AI, and develop robust governance strategies to navigate ethical and security challenges effectively. These actions will ensure that businesses stay competitive and unlock new opportunities for growth in the evolving AI landscape.</p><p><em>In preparing my article, I consulted ChatGPT, a large language model developed by OpenAI (OpenAI. (2024). ChatGPT [Large language model]. </em><a href="https://chat.openai.com/"><em>https://chat.openai.com</em></a><em>), for additional insights and information.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bff1ce0783c9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Digital Transforming the Workforce]]></title>
            <link>https://medium.com/@msong507/digital-transforming-the-workforce-f9b6f5dd3803?source=rss-f052cdb2ea66------2</link>
            <guid isPermaLink="false">https://medium.com/p/f9b6f5dd3803</guid>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[book-review]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[digital-transformation]]></category>
            <dc:creator><![CDATA[misun_song]]></dc:creator>
            <pubDate>Sun, 28 Jan 2024 05:28:22 GMT</pubDate>
            <atom:updated>2024-01-28T05:29:59.184Z</atom:updated>
            <content:encoded><![CDATA[<p>Integrating Fundamental Principles with Practical Experience</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hxKwvPtv7RhPSTwC3L4GWw.jpeg" /><figcaption>By Eugène Delacroix — Erich Lessing Culture and Fine Arts Archives via artsy.net, Public Domain, <a href="https://commons.wikimedia.org/w/index.php?curid=27539198">https://commons.wikimedia.org/w/index.php?curid=27539198</a></figcaption></figure><h3><strong><em>Introduction to ‘Easy Data Analytics Projects’</em></strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/529/1*FEr7lcQtW5nx-9kB0kiQ2Q.jpeg" /><figcaption><a href="https://kbookstore.com/104985346">https://kbookstore.com/104985346</a></figcaption></figure><p>Korean publications on data science exhibit a unique trait: a knack for near-perfect summarization that cuts to the core of key principles. Upon delving into <strong><em>‘Easy Data Analytics Projects’ authored by Young Jin Yoon and Jae Jin Hwang,</em></strong> I was struck by this distinct quality. It not only motivated me but also prompted a reflection on my own real-world experiences, offering valuable insights into how we can effectively spearhead the digital transformation of our workforce.</p><p>Today, we’ll explore strategies for digitally transforming our workforce by drawing insights from both the book and real-world experiences.</p><h3>Navigating the 6 Steps of Data Analytics Projects</h3><p>In our endeavor to digitally transform our workforce, data analytics serves as the cornerstone. The book has outlined “<strong><em>six pivotal steps</em></strong> <strong><em>for data analytics projects”</em></strong> tailored to our business environment based on the <a href="https://en.wikipedia.org/wiki/Cross-industry_standard_process_for_data_mining"><strong><em>CRISP-DM</em></strong></a> principles.</p><p><a href="https://medium.com/@msong507/crisp-dm-paving-the-path-to-a-successful-data-mining-project-fe171057778d">CRISP-DM: Paving the Path to a Successful Data Mining Project</a></p><h4><strong><em>1. Understanding Business Goals</em></strong></h4><ul><li>Delve into the current business landscape to discern overarching objectives.</li><li>Clearly define the final goals and identify the necessary data to achieve them.</li></ul><h4>2. <strong><em>Data Analytics Project Planning</em></strong></h4><ul><li>Establish a comprehensive timetable aligning with project milestones.</li><li>Tailor each step of the plan to address specific business needs while adhering to data analytics principles.</li></ul><h4><strong>3. Data Collection and Preprocessing</strong></h4><ul><li>Anticipate imperfections in collected data and prepare for preprocessing.</li><li>Ensure data integrity through meticulous cleaning and preparation processes.</li></ul><h4><strong>4. Data Analytics</strong></h4><ul><li>Explore the spectrum of data insights, including <strong><em>descriptive, diagnostic, predictive, and prescriptive analytics.</em></strong></li><li>Harness each type of analytics to extract actionable insights vital for informed decision-making.</li></ul><h4><strong>5. Evaluation</strong></h4><ul><li>Assess the effectiveness and relevance of the analyses conducted.</li><li>Validate findings against business objectives to ensure alignment and accuracy.</li></ul><h4><strong>6. Visualization and Presentation</strong></h4><ul><li>Translate complex data insights into compelling visualizations.</li><li>Craft presentations that effectively communicate findings to key stakeholders, driving actionable outcomes.</li></ul><p>While each step merits detailed exploration, <strong><em>today’s focus is on utilizing the four types of data analytics to digitize our workforce effectively.</em></strong></p><h3>Four Types of Data Analytics</h3><p>In the data analytics framework, the book classifies “<strong><em>data</em></strong> <strong><em>analytics into four distinct types,”</em></strong> each offering unique insights into business dynamics.</p><h4>Descriptive Analytics (Past)</h4><ul><li>This type elucidates ‘<strong><em>what’ happened in the past,</em></strong> offering clarity on current business situations with minimal complexity and cost.</li><li>Examples include monthly income statements and demographic analyses of customers.</li></ul><h4>Diagnostic Analytics (Past)</h4><ul><li>Moving beyond mere description, diagnostic analytics <strong><em>delves into the ‘why’ behind past events.</em></strong> Being more intricate than descriptive analytics, it aims to uncover<strong><em> correlations and causations</em></strong> within collected data.</li><li>An example is a dashboard dissecting a company’s revenue by quarter, month, and product.</li></ul><h4>Predictive Analytics (Future)</h4><ul><li>Looking toward the future, predictive analytics utilizes AI, machine learning, or deep learning <strong><em>to forecast outcomes</em></strong> based on historical data.</li><li>While not infallible, it enhances decision-making by <strong><em>reducing uncertainty.</em></strong></li></ul><h4>Prescriptive Analytics (Future)</h4><ul><li>This type goes a step further, <strong><em>offering actionable insights</em></strong> based on predictions based on the patterns from past data.</li><li>By presenting multiple alternative solutions, prescriptive analytics empowers decision-makers to take informed actions.</li></ul><h3>Real-World Challenges in Data Analytics Implementation</h3><p>In the realm of data science, real-world applications often diverge from academic definitions, particularly in the context of data analytics.</p><h4>Diagnostic Analytics</h4><p>While <strong><em>descriptive analytics</em></strong>, often mastered by companies through modern data collection systems like ERP and POS, is a foundational type, the implementation of <strong><em>diagnostic analytics</em></strong> presents persistent challenges.</p><p>For instance, consider a company launching a new product with sales falling below expectations. Despite access to real-time sales data, reliance on subjective insights from sales personnel can hinder accurate analysis, leading to potentially biased or time-consuming assessments.</p><h4>Predictive Analytics &amp; Prescriptive Analytics</h4><p>Furthermore, the transition to <strong><em>predictive analytics</em></strong> poses its own set of hurdles. Setting future goals, while evolving from traditional top-down methods to more practical bottom-up approaches, still often relies on intuitive estimates — 5%, 10%, sometimes 20% — rather than algorithmic precision. External pressures may also force unrealistic targets, undermining the accuracy of resulting action plans.</p><p>In the face of these challenges, bridging the gap between academic theory and practical implementation is crucial for leveraging the full potential of data analytics in business decision-making.</p><h3>Bridging Theory and Practice</h3><p>While challenges persist in advancing data analytics, there are solutions that bring us closer to achieving a digitally transformed workforce.</p><h4>Diagnostic Analytics</h4><p>To enhance diagnostic analytics, it’s essential to grasp the diverse contributors and their relationships to outcomes. The book proposes “<strong><em>three key steps: anomaly detection, drilling down, and correlation analysis.”</em></strong></p><ol><li><strong><em>Anomaly detection</em></strong> serves as the initial step in diagnosing a business issue, involving the identification of deviations from the norm. For instance, a daily sales figure double the average over the past three months warrants analysis as a potential anomaly.</li><li><strong><em>Drilling down</em></strong> involves exploring various contributing factors comprehensively. This entails examining factors such as time periods (daily, monthly, quarterly, yearly), product groups, sales channels, or promotions that may have influenced the observed outcome, such as rapid sales growth.</li><li>Finally, <strong><em>correlation analysis</em></strong> integrates the insights gained from examining these factors to understand their relationship with the desired outcome. It necessitates <strong><em>a holistic perspective</em></strong> to uncover meaningful connections. For instance, identifying the product group with the largest sales portion may not suffice if its growth rate remains constant. However, detecting a newly added product group with a significant sales increase highlights its critical contribution to overall sales growth. <strong><em>This process relies not only on data analysis but also on a deep understanding of the business context and background.</em></strong></li></ol><p>To optimize diagnostic analytics efficiency, the book recommends leveraging business intelligence tools to construct <strong><em>purpose-driven dashboards.</em></strong> These dashboards facilitate easy observation and integration of pertinent data, enabling thorough analysis of diverse contributing factors.</p><h4>Predictive Analytics &amp; Prescriptive Analytics</h4><p>For more accurate predictive analytics, advanced algorithms like linear and non-linear regression can be leveraged. Time series analysis, in particular, proves efficient and practical for prediction, considering real-world data often deviates from ideal assumptions for regression analysis.</p><p><strong><em>While objective predictive techniques may yield less optimistic forecasts, prescriptions based on these precise insights hold greater significance and impact. </em></strong>For instance, employing ARIMAX in time series analysis allows for understanding both predicted future sales and the external factors influencing them. By accurately pinpointing underlying issues, viable solutions can be formulated to address them effectively.</p><h3>Conclusion</h3><p>Our exploration of data analytics for workforce transformation reveals critical insights and challenges. By merging academic principles with practical experiences, we identify hurdles and solutions.</p><p>Each step in the analytics process offers potential, yet translating theory to practice poses persistent challenges. Implementing diagnostic analytics effectively and leveraging predictive and prescriptive analytics requires purpose-driven approaches and advanced techniques.</p><p>By embracing these strategies, organizations can navigate digital transformation, empower their workforce, and achieve sustainable growth in today’s business landscape.</p><p><em>While this article primarily explores the data analytics aspect, the book offers a comprehensive guide to executing data analytics projects in the real world. Korean readers are encouraged to explore </em><a href="https://product.kyobobook.co.kr/detail/S000001952216"><em>‘가볍게 떠먹는 데이터 프로젝트 (제이펍, 2021)’</em></a><em> for further insights.</em></p><p><em>In preparing my article, I consulted ChatGPT, a large language model developed by OpenAI (OpenAI. (2024). ChatGPT [Large language model]. </em><a href="https://chat.openai.com/"><em>https://chat.openai.com</em></a><em>), for additional insights and information.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f9b6f5dd3803" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hypothesis Test and P-Value]]></title>
            <link>https://medium.com/@msong507/hypothesis-test-and-p-value-3b2d2e28d843?source=rss-f052cdb2ea66------2</link>
            <guid isPermaLink="false">https://medium.com/p/3b2d2e28d843</guid>
            <category><![CDATA[p-value]]></category>
            <category><![CDATA[confidence-interval]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[hypothesis-testing]]></category>
            <dc:creator><![CDATA[misun_song]]></dc:creator>
            <pubDate>Sun, 24 Sep 2023 07:37:32 GMT</pubDate>
            <atom:updated>2023-09-24T07:37:32.513Z</atom:updated>
            <content:encoded><![CDATA[<h4>Discerning Real Differences with P-values</h4><h3>Hypothesis Testing</h3><p>Whenever we make an assumption or pose a hypothesis, it’s essential to verify its validity. Let’s take a scenario: a new drug, “X,” is developed to treat disease “Y” and has shown positive results in curing “N” number of patients. The immediate impulse might be to start mass-producing and advertising the drug, touting its effectiveness. However, there’s an essential step to consider first: <strong><em>hypothesis testing.</em></strong></p><p>Scientific assumptions and hypotheses differ from mathematical proofs. While a mathematical proof deals with certainties, <strong><em>scientific experiments typically deal with samples rather than entire populations.</em></strong> This introduces <strong><em>potential errors.</em></strong> In our drug example, the fact that the drug was effective on a sample group doesn’t guarantee its effectiveness on the entire global population. There might be variations due to genetics, environment, or other factors that weren’t present in the initial sample.</p><blockquote>A population encompasses the entire group under study, while a sample is a smaller, representative subset of that group. Researchers typically study samples because it’s often impractical (or impossible) to study the entire population, then use statistics to infer conclusions about the population based on the sample.</blockquote><p>Hypothesis testing is a tool used to statistically assess whether <strong><em>the observed results in our sample</em></strong> are <strong><em>likely to generalize to a broader population</em></strong> or if they <strong><em>might have occurred purely by chance.</em></strong> It’s crucial to conduct this test to avoid making potentially incorrect assumptions about the entire population based on a limited sample.</p><h3>Null Hypothesis vs. Alternative Hypothesis</h3><h4>Null Hypothesis <strong><em>(H0) </em></strong>vs. Alternative Hypothesis <strong><em>(H1)</em></strong></h4><p>At the core of hypothesis testing lie two opposing hypotheses: the null hypothesis and the alternative hypothesis.</p><p><strong><em>The alternative hypothesis (H1)</em></strong> <strong><em>represents a new claim or assertion suggesting a significant effect or difference in the population.</em></strong> For this to be validated statistically, we use sample data. Referring back to our drug scenario, the alternative hypothesis might state: “The new drug X is effective in curing disease Y.”</p><p><strong><em>In contrast, the null hypothesis (H0) suggests that there’s no significant effect or change, thereby opposing the alternative hypothesis.</em></strong> Using the same example, the null hypothesis would be: “The new drug X has no significant effect in curing disease Y, or its effect is the same as the existing treatment.”</p><p>The goal of hypothesis testing is to determine which hypothesis is more supported by the data. Based on the results, we either <strong><em>reject the null hypothesis</em></strong> (giving credence to the alternative hypothesis) or <strong><em>fail to reject the null hypothesis</em></strong> (suggesting there’s not enough evidence for the alternative hypothesis).</p><h4>Type 1 and Type 2 Error</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/0*WgahLPGOJrFXvYg2.jpg" /><figcaption><a href="https://www.simplypsychology.org/type_i_and_type_ii_errors.html">https://www.simplypsychology.org/type_i_and_type_ii_errors.html</a></figcaption></figure><p>To understand Type I and Type II errors more intuitively, let’s use a framework similar to the confusion matrix. Consider our null hypothesis: it could be true (<em>O</em>) or false (<em>X</em>). Likewise, our decision might be to accept it (<em>O</em>) or reject it (<em>X</em>). This creates four potential outcomes: (<em>O</em>,<em>O</em>), (<em>X</em>,<em>X</em>), (<em>O</em>,<em>X</em>), and (<em>X</em>,<em>O</em>).</p><p><a href="https://medium.com/@msong507/understanding-the-confusion-matrix-without-confusion-126b25dd773c">Understanding the Confusion Matrix without Confusion</a></p><p>The first two combinations, (<em>O</em>,<em>O</em>) and (<em>X</em>,<em>X</em>), represent correct decisions, but the latter two introduce errors:</p><ol><li>(<em>O</em>,<em>X</em>) — This scenario arises when the null hypothesis is true (<em>O</em>), but we wrongly reject it (<em>X</em>). This error is known as a Type I error. To draw from our previous example, this would be akin to mistakenly determining that Drug X is effective and then starting its mass production, thereby expending resources based on a false premise.</li><li>(<em>X</em>,<em>O</em>) — Here, the null hypothesis is false (<em>X</em>), but we incorrectly accept it (<em>O</em>). This is a Type II error. In the context of Drug X, this implies we would wrongly assume the drug isn’t effective, missing out on a potentially beneficial opportunity.</li></ol><blockquote>It’s worth noting that statisticians often avoid saying they “accept” a hypothesis — instead, they focus on rejecting or failing to reject, as this better reflects the inherent uncertainty in the process. However, for the sake of clarity and intuitive understanding in this context, I’ve employed the term “accept.”</blockquote><h3>Hypothesis Testing: A Step-by-Step Guide</h3><p>Hypothesis testing is a systematic method used in statistics to determine whether a hypothesis about a population parameter is valid based on sample data.</p><h4>1. Define the Population Parameter</h4><ul><li>Begin by deciding which parameter you’re interested in testing. It could be the mean, proportion, variance, etc. For this example, let’s consider we’re testing the mean (<em>μ</em>) of a population.</li></ul><h4>2. State the Hypotheses</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/0*2iyAfAZAdw1LwM_L.jpeg" /><figcaption><a href="https://www.ztable.net/z-test/#google_vignette">https://www.ztable.net/z-test/#google_vignette</a></figcaption></figure><blockquote>Two-Tailed Test evaluates if <em>μ</em> is either significantly greater or less than <em>μ</em>0​ while One-Tailed Test checks if <em>μ</em> is significantly greater or less than <em>μ</em>0​, but not both. The direction must be specified.</blockquote><ul><li>Null Hypothesis (<em>H</em>0​): This is a statement that there is no effect or no difference, and it serves as the default assumption. For our mean, it might look like <em>H</em>0​: <em>μ</em>=<em>μ</em>0​, where <em>μ</em>0​ is a known value.</li><li>Alternative Hypothesis (<em>H</em>1​): This is what you want to prove. Depending on the test, it could be <em>μ</em>=<em>μ</em>0​ (two-tailed), <em>μ</em>&gt;<em>μ</em>0​ (one-tailed, right), or <em>μ</em>&lt;<em>μ</em>0​ (one-tailed, left).</li></ul><h4>3. Choose the Significance Level (<em>α</em>)</h4><ul><li>Often set at <em>α</em>=0.05 (which becomes 0.025 for each side in a two-tailed test), this represents the probability of rejecting a true null hypothesis.</li><li>It can also be set at <em>α</em>=0.01 (or 0.005 in a two-tailed test), making it more stringent to reject the null hypothesis.</li></ul><h4>4. Select the Appropriate Test and Compute the Test Statistic</h4><ul><li>Depending on your data and sample size, you might choose a t-test, Z-test, chi-squared test, etc.</li><li>Calculate the test statistic using the appropriate formula. For instance, in a Z-test for means, the test statistic <em>Z</em> is calculated using:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/0*eIKn3BCYDHCuzFXr" /><figcaption><a href="https://www.fireblazeaischool.in/blogs/hypothesis-testing-using-z-test/">https://www.fireblazeaischool.in/blogs/hypothesis-testing-using-z-test/</a></figcaption></figure><h4>5. Determine the P-Value:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/825/0*TS_TZ-vyc0RqwlPQ.png" /><figcaption><a href="https://www.wallstreetmojo.com/z-test/">https://www.wallstreetmojo.com/z-test/</a></figcaption></figure><ul><li>Using statistical tables (or software), determine the probability (p-value) of observing a test statistic as extreme as, or more extreme than, the one calculated, given that <em>H</em>0​ is true.</li></ul><blockquote>Suppose we conduct a Z-test and obtain a Z-score of 1.97. Referring to the <a href="https://statisticsbyjim.com/hypothesis-testing/z-table/">Z-score table</a>, the cumulative probability up to this value is 0.97558. This means that the probability of getting a Z-score greater than 1.97 (in one tail) is <em>α</em>/2 = 1−0.97558 = 0.02442. For a two-tailed test, the combined probability becomes <em>α </em>= 2×0.02442 = 0.04884. Since this p-value is less than our predetermined significance level of 0.05, we have sufficient evidence to reject the null hypothesis.</blockquote><h4>6. Make a Decision:</h4><ul><li>If <em>p</em>−<em>value </em>≤ <em>α</em>: Reject <em>H</em>0​. The results are statistically significant at the <em>α</em> level, and there’s evidence for <em>H</em>1​. A smaller p-value provides stronger evidence against the null hypothesis.</li><li>If <em>p</em>−<em>value </em>&gt; <em>α</em>: Fail to reject <em>H</em>0​. There’s not enough evidence at the <em>α</em> level to support <em>H</em>1​.</li></ul><h3>Conclusion</h3><p>Using samples instead of whole populations can lead to errors in our theories. Hypothesis testing helps us determine the accuracy of these theories. We set a significance level (usually denoted as α) and compare it to a p-value obtained from statistical tests, like the Z-test. If the p-value is smaller than α, we can reject our initial theory or the null hypothesis. Delving into the underlying principles of hypothesis testing and p-value can empower us to hone our predictive models for more accurate outcomes.</p><h3>Additional Topic: Confidence Interval</h3><blockquote>Both confidence intervals and hypothesis testing are grounded in the theory of probability and share common concepts, such as the standard error and the central limit theorem. They offer different perspectives and approaches to the same underlying question: based on a sample, what can we infer about the population?</blockquote><p>A confidence interval in statistics provides <strong><em>a range within which we expect a population parameter, such as the true mean (the actual average of the entire population), to fall.</em></strong> Commonly used confidence levels are 95% and 99%. It’s crucial to underscore that confidence intervals convey the probability that the true mean lies within the specified range, rather than the reliability of that range.</p><p>Let’s use the estimation of the population’s true mean for clarity. Imagine testing a weight-loss drug on mice. Instead of specifying a single expected weight, like 70g, after injection, <strong><em>it’s more practical to provide a range.</em></strong> If you suggest that the true mean weight after the drug injection is likely <strong><em>between 68g and 72g with 95% confidence, this means that if you were to repeat the experiment 100 times, you’d expect the true mean weight to fall within this range in 95 of those repetitions, and outside of it in 5.</em></strong></p><p>The breadth of the confidence interval can differ. A wider interval, say from 60g to 80g, might come with a higher confidence level, like 99%. However, more extensive doesn’t always mean better. <strong><em>While it captures a wider set of potential true means, it’s less precise.</em></strong> On the other hand, a tighter interval, like 69g to 70g at 90% confidence, provides <strong><em>a sharper focus on the estimated true mean but has a higher risk of missing it.</em></strong> Balancing precision and confidence is a key reason many opt for a 95% confidence interval.</p><p><em>For a detailed walkthrough, check out the accompanying video!</em></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FvemZtEM63GY%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DvemZtEM63GY&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FvemZtEM63GY%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/8468e47f2235851e459c5a50ebfeb37f/href">https://medium.com/media/8468e47f2235851e459c5a50ebfeb37f/href</a></iframe><p><em>In preparing my article, I consulted ChatGPT, a large language model developed by OpenAI (OpenAI. (2023). ChatGPT [Large language model]. </em><a href="https://chat.openai.com/"><em>https://chat.openai.com</em></a><em>), for additional insights and information.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3b2d2e28d843" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding the ROC-AUC Curve]]></title>
            <link>https://medium.com/@msong507/understanding-the-roc-auc-curve-cc204f0b3441?source=rss-f052cdb2ea66------2</link>
            <guid isPermaLink="false">https://medium.com/p/cc204f0b3441</guid>
            <category><![CDATA[logistic-regression]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[confusion-matrix]]></category>
            <category><![CDATA[roc-curve]]></category>
            <category><![CDATA[classification]]></category>
            <dc:creator><![CDATA[misun_song]]></dc:creator>
            <pubDate>Sat, 23 Sep 2023 05:58:46 GMT</pubDate>
            <atom:updated>2023-09-23T06:01:48.604Z</atom:updated>
            <content:encoded><![CDATA[<h4>Evaluating Classification Model Performance Simply</h4><h3>Laying the Foundation for ROC-AUC Curve</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/754/0*8_gokeSGvFThxqXm" /><figcaption><a href="https://encord.com/glossary/confusion-matrix/">https://encord.com/glossary/confusion-matrix/</a></figcaption></figure><p><a href="https://medium.com/@msong507/understanding-the-confusion-matrix-without-confusion-126b25dd773c">Understanding the Confusion Matrix without Confusion</a></p><p>The journey to the ROC-AUC curve begins with <strong><em>the confusion matrix,</em></strong> a foundational tool to assess classification model performance. Here, we consider four critical elements: True Positives (TP), True Negatives (TN), False Positives (FP), and False Negatives (FN). With these, we evaluate the efficacy of binary or multiclass classification models using metrics like precision, recall (sensitivity), specificity, and NPV — Negative Predictive Value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*5d8Uym7AVLOGvcEn.png" /><figcaption><a href="https://www.javatpoint.com/logistic-regression-in-machine-learning">https://www.javatpoint.com/logistic-regression-in-machine-learning</a></figcaption></figure><p><a href="https://medium.com/@msong507/logistic-regression-explained-2d1b8babe6c1">Logistic Regression Explained</a></p><p>Our exploration then takes us to <strong><em>logistic regression,</em></strong> a staple in the classification model realm. Unlike linear regression’s straight line, logistic regression uses a sigmoid function to predict outcomes, which are typically binary (0 or 1). The logistic model builds upon the core concepts of linear regression, introducing the notion of odds and log odds (logit). This serves as a link function, enabling us to interpret our model’s coefficients.</p><h3>The Role of Threshold in Model Performance</h3><h4>How Does Threshold Impact Decisions?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/0*iNA7A66m_4ijau6-.png" /><figcaption><a href="https://stackoverflow.com/questions/30499018/why-is-logistic-regression-called-regression">https://stackoverflow.com/questions/30499018/why-is-logistic-regression-called-regression</a></figcaption></figure><p><strong><em>In binary classification, the threshold is indeed a pivotal aspect. </em></strong>When you’re predicting the probability of an instance belonging to a particular class, a threshold determines at what probability you classify an instance as the positive class (1) or the negative class (0).</p><p>Imagine your email service has a spam filter that uses a classification model to decide whether an incoming email is spam or not. The model computes a score (probability) for each email, representing the likelihood it’s spam.</p><ul><li>If you set <strong><em>a high threshold</em></strong>, say 95%, the model will only mark very obvious spam emails, ensuring your important emails don’t accidentally get flagged. However, this might mean some less-obvious spam emails might slip through into your main inbox.</li><li>On the other hand, if you set a <strong><em>lower threshold</em></strong>, say 50%, the filter will catch a broader range of spam emails, but there’s also a higher chance it might incorrectly classify a legitimate email as spam.</li></ul><p><strong><em>In this scenario, the threshold determines the balance between ensuring you don’t miss important emails (False Positives) and ensuring spam doesn’t clutter your main inbox (True Positives). Adjusting this threshold affects the balance, representing the trade-off in decision-making.</em></strong></p><h3>True Positive Rate and False Positive Rate</h3><h4>Introducing TPR and FPR</h4><p>To determine the optimal threshold for a classification model, we study its performance across multiple thresholds, particularly focusing on <strong><em>how True Positives and False Positives shift with threshold changes.</em></strong> This examination involves two key metrics: True Positive Rate (TPR) and False Positive Rate (FPR).</p><p>For those unfamiliar with these terms:</p><ul><li><strong><em>True Positive Rate (TPR):</em></strong> Defined as <em>TPR</em>=<em>TP</em>/(<em>TP</em>+<em>FN</em>). Also known as recall or sensitivity, TPR gauges the proportion of correctly identified positive instances out of all actual positive instances.</li><li><strong><em>False Positive Rate (FPR):</em></strong> Defined as <em>FPR</em>=<em>FP</em>/(<em>FP</em>+<em>TN</em>). FPR measures the proportion of actual negatives mistakenly identified as positives. Another way to compute FPR is by taking 1−specificity, focusing on the negative class.</li></ul><h4>Balancing TPR and FPR</h4><p>The relationship between TPR and FPR reveals a model’s behavior across different thresholds.</p><ul><li><strong><em>A lower threshold:</em></strong> classifies more instances as positive, leading to a higher TPR, but also a higher FPR as more negatives are mistakenly identified.</li><li><strong><em>A higher threshold:</em></strong> becomes stricter, classifying fewer instances as positive. This results in a lower TPR (missing some actual positives) and a lower FPR (fewer negatives are misclassified).</li></ul><p>The ROC curve shows how TPR and FPR change as we adjust the threshold. <strong><em>Even though both TPR and FPR can rise or fall together, the key is in how fast or slow they change compared to each other. </em></strong>This difference in their rates is the trade-off we often talk about.</p><p>A classifier’s performance is considered optimal when it achieves a high TPR without a significant increase in FPR. Hence, the top-left corner of the ROC curve, which signifies <strong><em>a high TPR and a low FPR, represents the ideal balance or “sweet spot”.</em></strong></p><h3>ROC (Receiver Operating Characteristic Curve)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9h-FJb3PclpQDVQF.png" /><figcaption>By cmglee, MartinThoma — Roc-draft-xkcd-style.svg, CC BY-SA 4.0, <a href="https://commons.wikimedia.org/w/index.php?curid=109730045">https://commons.wikimedia.org/w/index.php?curid=109730045</a></figcaption></figure><h4>What is the ROC curve?</h4><p>The ROC curve is a graph that shows how well a classification model distinguishes between two classes (like “spam” and “not spam”) <strong><em>at various thresholds.</em></strong> It lets you visualize the model’s performance at ALL possible thresholds, not just a single one.</p><h4>How to create a ROC curve</h4><ol><li><strong><em>Predict Probabilities:</em></strong> When you ask a classification model whether something is “spam” or “not spam”, it doesn’t simply guess. Internally, the model calculates <strong><em>a probability.</em></strong></li><li><strong><em>Test Different Thresholds:</em></strong> If the model directly classifies based on its internal score, <strong><em>we’d have to set a threshold.</em></strong> For instance, you might say “If the score is above 0.8, call it spam.” But is 0.8 the right threshold? What if 0.7 is better? Or 0.9?</li><li><strong><em>Track TPR and FPR: TPR (True Positive Rate)</em></strong> is about of all actual spam emails, how many did we correctly call “spam.” <strong><em>FPR (False Positive Rate)</em></strong> is about of all actual “not spam” emails, how many did we wrongly call “spam”?</li><li><strong><em>Plot: </em></strong>For each threshold, <strong><em>plot TPR on the y-axis and FPR on the x-axis.</em></strong> Connect the dots to form a curve.</li></ol><h4>How to interpret it:</h4><ol><li><strong><em>Top-Left Corner is Best:</em></strong> If the curve is closer to the top-left corner, that’s great! It means we’re correctly identifying spam without mislabeling many genuine emails. (Maximum True Positives, minimum false positives)</li><li><strong><em>Above the Diagonal Line:</em></strong> If the curve is above the diagonal line (from bottom-left to top-right), our model is better than just random guessing because the diagonal line means where TPR = FPR.</li><li><strong><em>Area Under Curve (AUC):</em></strong> The bigger the area under the curve (closer to 1), the better our model is.</li></ol><h3>AUC (Area Under the Curve)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7-qtTcMwGcljLm0MSrpuOw.png" /><figcaption><a href="https://www.youtube.com/watch?app=desktop&amp;v=4jRBRDbJemM">https://www.youtube.com/watch?app=desktop&amp;v=4jRBRDbJemM</a></figcaption></figure><h4>What is AUC?</h4><p><strong><em>AUC, which stands for “Area Under the ROC Curve,”</em></strong> quantifies a classifier’s performance. The ROC Curve plots the True Positive Rate (TPR) against the False Positive Rate (FPR) over various thresholds. Instead of assessing a model through the ROC curve visually, <strong><em>AUC summarizes it into a single numerical value, with a higher AUC indicating superior model performance.</em></strong></p><h4>Why is AUC Important?</h4><ol><li><strong><em>Comparison of Models:</em></strong> A single AUC value offers a quick comparison for multiple models. The one with a higher AUC generally excels in classification when evaluated on the same dataset.</li><li><strong><em>Effective with Imbalanced Datasets:</em></strong> In datasets where one class significantly dominates, metrics like accuracy might not provide an accurate picture. AUC, which incorporates both TPR and FPR, serves as a more reliable metric in such instances.</li></ol><h4>Interpreting AUC Values</h4><ul><li><strong><em>AUC = 0.5:</em></strong> This implies that the classifier’s performance is equivalent to <strong><em>random guessing</em></strong> (akin to a coin toss). It corresponds to the diagonal line from (0,0) to (1,1) on the ROC plot.</li><li><strong><em>0.5 &lt; AUC &lt; 1:</em></strong> Indicates that the classifier outperforms random guessing. <strong><em>The closer the value is to 1, the better the classifier is</em></strong> at differentiating between positive and negative instances.</li><li><strong><em>AUC = 1:</em></strong> <strong><em>A perfect classifier</em></strong> that flawlessly distinguishes between positive and negative classes. It’s a theoretical value and seldom achieved in real-world applications.</li></ul><h3>Conclusion</h3><p>While generating an ROC-AUC curve for evaluating binary classification models is straightforward, grasping its underlying principles demands a deeper dive. This journey begins with the foundational concepts of the confusion matrix, which helps gauge the performance of a classification model, and extends to the nuances of logistic regression, a cornerstone of classification modeling. Armed with this understanding, <strong><em>we’re now adept at comparing models,</em></strong> empowering us to choose the most suitable one for our specific challenges!</p><p><em>Note: The ROC-AUC curve is traditionally associated with binary classification, but it has been extended for use in multiclass classification problems as well. It’s worth noting that it might not always be the most intuitive method to evaluate multiclass classifiers.</em></p><p><em>For a detailed walkthrough, check out the accompanying video!</em></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F4jRBRDbJemM&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D4jRBRDbJemM&amp;image=http%3A%2F%2Fi.ytimg.com%2Fvi%2F4jRBRDbJemM%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/11ce8c4ece5e33ba14ef2dcbb33767a2/href">https://medium.com/media/11ce8c4ece5e33ba14ef2dcbb33767a2/href</a></iframe><p><em>In preparing my article, I consulted ChatGPT, a large language model developed by OpenAI (OpenAI. (2023). ChatGPT [Large language model]. </em><a href="https://chat.openai.com/"><em>https://chat.openai.com</em></a><em>), for additional insights and information.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cc204f0b3441" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Logistic Regression Explained]]></title>
            <link>https://medium.com/@msong507/logistic-regression-explained-2d1b8babe6c1?source=rss-f052cdb2ea66------2</link>
            <guid isPermaLink="false">https://medium.com/p/2d1b8babe6c1</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[likelihood]]></category>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[logistic-regression]]></category>
            <dc:creator><![CDATA[misun_song]]></dc:creator>
            <pubDate>Wed, 20 Sep 2023 20:08:25 GMT</pubDate>
            <atom:updated>2023-09-24T01:13:42.434Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Moving Beyond Linear Predictions</em></h4><h3>Linear Regression Overview</h3><p>Linear regression models the relationship between a dependent variable and one or more independent variables <strong><em>by fitting a linear equation</em></strong> to observed data. The goal is to minimize <strong><em>the sum of squared errors</em></strong> — the squared differences between observed values and the values predicted by the model. The R-squared value represents the proportion of variance in the dependent variable that’s explained by the independent variables. A higher R-squared generally suggests a better fit, but it’s essential to check the residuals to ensure the model adequately captures underlying data patterns. Once the model is established, it can be used to predict <strong><em>continuous outcomes</em></strong> based on new input values for the independent variables.</p><p><a href="https://medium.com/@msong507/linear-regression-fundamentals-489e6d60d5cc">Linear Regression Fundamentals</a></p><h3>Logistic Regression: The Classification Perspective</h3><h4>Understanding Classification in Logistic Regression</h4><p>Unlike linear regression, which predicts continuous outcomes, when predicting categorical outcomes, logistic regression is often the model of choice. While it’s particularly well-suited for binary outcomes, like churn or not, infected or uninfected, and alive or deceased, it can also handle multi-class problems. <strong><em>Regardless of the number of classes, tasks that involve predicting categories are termed classification problems.</em></strong></p><h4>The Role of the Sigmoid Function</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*c47mNr66HttGJB2x.png" /><figcaption><a href="https://www.oreilly.com/library/view/artificial-intelligence-for/9781788472173/bc945010-2715-42f9-8934-b2e08b53c34e.xhtml">https://www.oreilly.com/library/view/artificial-intelligence-for/9781788472173/bc945010-2715-42f9-8934-b2e08b53c34e.xhtml</a></figcaption></figure><p>For binary outcomes, a straight-line prediction, as done in linear regression, isn’t appropriate. This is where logistic regression comes into play, using the sigmoid (or logistic) function. <strong><em>This function constrains any input value into the range of 0 to 1.</em></strong> Such a feature is invaluable because it guarantees that predictions align with binary outcomes.</p><p><strong><em>Furthermore, the sigmoid function is naturally robust against outliers, enhancing the reliability of our model:</em></strong> extremely low inputs are mapped close to 0, and extremely high inputs are mapped close to 1, reducing their influence on the overall model.</p><blockquote>For a deeper dive into the subject, further material is available below (please note that the detailed content is in Korean).</blockquote><p><a href="https://sosoeasy.tistory.com/335">logistic regression 에서 sigmoid함수를 쓰는 이유</a></p><h4>Predicting Outcomes with Probabilities</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*eV5-mCe4E--5guZw.png" /><figcaption><a href="https://www.javatpoint.com/logistic-regression-in-machine-learning">https://www.javatpoint.com/logistic-regression-in-machine-learning</a></figcaption></figure><p>We now understand the importance of the sigmoid function in logistic regression, but how does it facilitate predictions? Visualize the sigmoid curve: the x-axis represents the actual values from our dataset, <strong><em>while the y-axis depicts the corresponding probability for each input value.</em></strong> Conventionally, a threshold of 0.5 is utilized; <strong><em>if the probability is above this, the outcome is labeled as 1 (yes). If below, it’s categorized as 0 (no).</em></strong></p><p>However, this threshold is adaptable. For instance, during the pandemic’s height, detecting infections was prioritized, even at the risk of more false positives. This required adjusting the threshold downwards. Now, with a focus on precise diagnosis, there’s merit in revisiting and potentially raising the threshold.</p><h3>Logistic Regression: The Regression Perspective</h3><h4>Odds: Measuring the Likelihood of an Event</h4><p><strong><em>Odds represent the ratio of the probability of an event happening to it not happening.</em></strong> In other words, Odds can provide an intuitive measure of the strength of an association. <strong><em>For a binary event, the odds tell us how many times more likely the event is to happen than not.</em></strong> For instance, consider the risk of a smoker developing a certain disease. If the probability (P) of a smoker getting the disease is 0.6, then the odds of them developing the disease are:</p><ul><li>Odds = P / (1-P) = 0.6 / 0.4 = 1.5.</li><li>This implies that smokers are 1.5 times more likely to develop the disease than to avoid it.</li></ul><h4>Using Logit (Log Odds) as a Link Function</h4><p>Why, after discussing the sigmoid function and its role in classification probabilities, are we turning our attention to odds?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/0*Eqwbxw_9_z8nx95C.png" /><figcaption><a href="https://www.theanalysisfactor.com/what-is-logit-function/">https://www.theanalysisfactor.com/what-is-logit-function/</a></figcaption></figure><p><strong><em>In logistic regression, we’re not directly modeling the probability of an outcome. Instead, we rely on the logit (log odds), which is essentially the natural logarithm of odds, as our link function.</em></strong></p><p>Think of the link function as a bridge or translator. When our predictors and the outcome operate in different domains or ‘languages’, the link function helps translate between them, ensuring they’re compatible. For logistic regression, this is crucial. <strong><em>Our outcome lies in a constrained space (between 0 and 1 since it’s a probability), while our predictors might span different ranges. </em></strong>The logit function, acting as this translator, modifies the probabilities, allowing them to be more harmoniously modeled alongside the predictors with the following characteristics:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dov1EODdA371l4qSiH9XiQ.png" /><figcaption><a href="https://www.youtube.com/watch?v=vN5cNN2-HWE&amp;list=RDCMUCtYLUTtgS3k1Fg4y5tAhLbw&amp;index=2">https://www.youtube.com/watch?v=vN5cNN2-HWE&amp;list=RDCMUCtYLUTtgS3k1Fg4y5tAhLbw&amp;index=2</a></figcaption></figure><ol><li><strong><em>Transformability: </em></strong>It’s tricky to directly model outcomes like probabilities, which are bounded between 0 and 1, using linear approaches. The logit function liberates these values, placing them on an unrestricted continuous scale.</li><li><strong><em>Linearity: </em></strong>With the logit transformation, there’s a more linear relationship between our predictors and the outcome. It lets us apply linear regression principles, even if the original outcome isn’t inherently linear.</li></ol><p>In summary, the logit function equips logistic regression to leverage the strengths of linear models, making our predictions both strong and interpretable.</p><h3>Maximum Likelihood Estimation (MLE)</h3><h4>Probability vs. Likelihood</h4><p>In everyday language, we often use “probability” and “likelihood” interchangeably. However, in the realm of statistics, they have distinct meanings.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FpYxNSUDSFH4%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DpYxNSUDSFH4&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FpYxNSUDSFH4%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/62ab055961e3b2afdfb62293b8b06f08/href">https://medium.com/media/62ab055961e3b2afdfb62293b8b06f08/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b5Q4gvcG6_ryWCyQ1cB0sA.png" /><figcaption><a href="https://www.youtube.com/watch?v=pYxNSUDSFH4">https://www.youtube.com/watch?v=pYxNSUDSFH4</a></figcaption></figure><ul><li><strong><em>Probability describes the chance of a specific event happening. </em></strong>For instance, if you flip a fair coin, the probability of getting heads is 0.5, since there’s an equal chance for both outcomes. Probabilities always range between 0 (event will not happen) and 1 (event will definitely happen).</li><li><strong><em>Likelihood, on the other hand, gauges how well a certain set of data supports a specific statistical model or a parameter within that model.</em></strong> It’s not about the “probability” of the model or parameter itself but the compatibility of observed data with it. For instance, if you observed 7 heads in 10 coin flips, you’d assess the likelihood of this data given a model where the coin is fair.</li></ul><h4>Optimizing Predictions with Maximum Likelihood</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/0*Eqwbxw_9_z8nx95C.png" /><figcaption><a href="https://www.theanalysisfactor.com/what-is-logit-function/">https://www.theanalysisfactor.com/what-is-logit-function/</a></figcaption></figure><p>Logistic regression is primarily designed to predict categorical outcomes. For the binary context in this article, <strong><em>we employ the logit function to frame these outcomes within a regression structure.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dov1EODdA371l4qSiH9XiQ.png" /><figcaption><a href="https://www.youtube.com/watch?v=vN5cNN2-HWE&amp;list=RDCMUCtYLUTtgS3k1Fg4y5tAhLbw&amp;index=2">https://www.youtube.com/watch?v=vN5cNN2-HWE&amp;list=RDCMUCtYLUTtgS3k1Fg4y5tAhLbw&amp;index=2</a></figcaption></figure><p>The logit function transforms the probabilities of these outcomes <strong><em>from the range [0,1] to the entire real number line (-∞, ∞).</em></strong></p><p>In traditional linear regression, we calculate <strong><em>the sum of squared residuals</em></strong> to find the best-fitting line. The residual is the difference between the observed value and the predicted value, representing the “error” for each data point. However, in the context of logistic regression, since our outcomes after the logit transformation stretch to positive and negative infinity, <strong><em>we can’t compute residuals in the usual way.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/604/0*fUAJr82-FEc2-wtb.png" /><figcaption><a href="https://blog.finxter.com/logistic-regression-in-one-line-python/">https://blog.finxter.com/logistic-regression-in-one-line-python/</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/604/0*kQ88eIA2ZxUSQthD.png" /><figcaption><a href="https://blog.finxter.com/logistic-regression-in-one-line-python/">https://blog.finxter.com/logistic-regression-in-one-line-python/</a></figcaption></figure><blockquote>The first image illustrates the calculation of likelihood using the logistic (sigmoid) function. In the second image, as the sigmoid curve shifts to the right, the value of the maximum likelihood increases.</blockquote><p>Given this challenge, we turn to an alternative method: maximum likelihood estimation (MLE). At its core, MLE estimates the likelihood by multiplying the individual likelihoods of each data point. <strong><em>The goal is to find the parameters (e.g., weights and bias) that maximize this overall likelihood. </em></strong>This ensures that, under our logistic regression model, the observed data is as probable as it can be.</p><h3>Translating Log-Odds into Probabilities</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/0*Eqwbxw_9_z8nx95C.png" /><figcaption><a href="https://www.theanalysisfactor.com/what-is-logit-function/">https://www.theanalysisfactor.com/what-is-logit-function/</a></figcaption></figure><p>In fine-tuning our logistic regression model by playing with MLE, we identify the coefficients and intercept that offer the best predictions. But it’s crucial to note that, <strong><em>fundamentally, our model predicts the log-odds, not direct probabilities.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/324/0*lT9TUQpEzFHrZD3a.gif" /><figcaption>http://faculty.cas.usf.edu/mbrannick/regression/Logistic.html</figcaption></figure><p>For actionable insights, <strong><em>we must convert these log-odds back to probabilities. </em></strong>This conversion involves the logit function, which is based on the natural logarithm with base <em>e</em> (Euler’s constant). While we often simplify the expression by not explicitly referencing <em>e</em>, it’s integral to the transformation. By exponentiating our logit output (using base <em>e</em>), we can determine the corresponding probability <em>p</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*57Sbq8VdKRS_EuSZfdzA3A.png" /><figcaption><a href="http://faculty.cas.usf.edu/mbrannick/regression/Logistic.html">http://faculty.cas.usf.edu/mbrannick/regression/Logistic.html</a></figcaption></figure><p>This probability then allows us to classify observations as either 0 or 1, depending on a predetermined threshold.</p><h3>Interpreting Coefficient</h3><h4>In Linear Regression</h4><p>Regression provides a clear way to understand the relationship between dependent and independent variables. Consider a linear regression model relating temperature (independent variable) to ice cream sales (dependent variable). If we determine a coefficient of 10 for temperature, it means that, all else being equal, a one-unit increase in temperature would result in a 10-unit increase in ice cream sales.</p><h4>In Logistic Regression</h4><p>Logistic regression works similarly, with coefficients indicating the influence on the probability of an outcome being classified as either 0 or 1. <strong><em>However, there’s a key distinction: these coefficients are associated with the log-odds, not the probability directly.</em></strong> Therefore, interpreting the coefficients isn’t as straightforward.</p><p>In other words, a one-unit increase in a predictor variable doesn’t lead to a uniform increase or decrease in the outcome’s probability. Instead, it changes the log-odds. And because of the sigmoidal shape of the logistic function, this change in log-odds translates to a non-linear change in probability.</p><ul><li><strong><em>A positive coefficient:</em></strong> as the predictor goes up, the log-odds of the outcome being 1 goes up, making it more probable.</li><li><strong><em>A negative coefficient:</em></strong> as the predictor goes up, the log-odds of the outcome being 1 goes down, making it less probable.</li></ul><p>While the coefficients in logistic regression provide invaluable insights, they demand a nuanced understanding, which is a departure from the straightforward interpretation in linear regression. <strong><em>Specifically, if βi​ is the coefficient of predictor xi​, then for a one-unit increase in xi​, the odds will multiply by e^βi​.</em></strong> This is because the relationship is exponential due to the logarithmic link.</p><p>When we’re looking at how a change in <em>xi</em>​ affects the log-odds, we’re examining the multiplicative effect of that predictor on the odds of the outcome. Given the logarithmic transformation, this multiplicative effect becomes an additive effect on the log scale. When other predictors are held constant, their contributions remain unchanged. Thus, the only effect we observe due to a one-unit increase in <em>xi</em>​ is an exponential change in the odds, represented by <em>e^βi</em>​​</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FyhogDBEa0uQ%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DyhogDBEa0uQ&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FyhogDBEa0uQ%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/917d6bf12e64656ba8db55d2bd7de945/href">https://medium.com/media/917d6bf12e64656ba8db55d2bd7de945/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rdq5l2MKBnYhyaZx86eIGQ.png" /><figcaption><a href="https://www.youtube.com/watch?v=yhogDBEa0uQ">https://www.youtube.com/watch?v=yhogDBEa0uQ</a></figcaption></figure><p>To illustrate this more clearly, let’s consider a straightforward example. In this case above, the coefficient <em>β</em>1​ is 2.75, a one-unit increase in <em>xi</em>​ multiplies the odds of the outcome by e^2.75 or approximately 15.6, assuming all other variables are held constant.</p><h3>Conclusion</h3><p>Logistic Regression is a cornerstone for grasping classification models. While it echoes linear regression in its predictive framework, it differentiates itself by employing the sigmoid function.</p><p>This function maps input values into probabilities, which are then used to determine predictions as either binary outcomes (0 or 1) or categories in multi-class scenarios. Moreover, while logistic regression employs the logit as its link function introduces the complexity of converting logits to probabilities and nuances in interpreting the coefficients, while it addresses constraints associated with direct probability estimation.</p><p>These transitions between underlying principles can appear daunting initially. However, by bridging these gaps with your own understanding and insights, you’ll be well-positioned to tackle even more complex machine-learning concepts in the future.</p><p><em>For a deeper understanding of how to evaluate the performance of various logistic models, refer to the articles below.</em></p><ul><li><a href="https://medium.com/@msong507/understanding-the-confusion-matrix-without-confusion-126b25dd773c">Understanding the Confusion Matrix without Confusion</a></li><li><a href="https://medium.com/@msong507/understanding-the-roc-auc-curve-cc204f0b3441">Understanding the ROC-AUC Curve</a></li></ul><p><em>Please review the attached videos for a comprehensive understanding of logistic regression. The first video is part of a series that provides a straightforward explanation of logistic regression, complemented by captivating visuals. The second video, also part of a series, offers a detailed perspective reminiscent of a statistics textbook.</em></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FyIYKR4sgzI8%3Flist%3DPLblh5JKOoLUKxzEP5HA2d-Li7IJkHfXSe&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DyIYKR4sgzI8&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FyIYKR4sgzI8%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/995998e3738fa760a3add256bcc58476/href">https://medium.com/media/995998e3738fa760a3add256bcc58476/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FyhogDBEa0uQ%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DyhogDBEa0uQ&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FyhogDBEa0uQ%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/917d6bf12e64656ba8db55d2bd7de945/href">https://medium.com/media/917d6bf12e64656ba8db55d2bd7de945/href</a></iframe><p><em>In preparing my article, I consulted ChatGPT, a large language model developed by OpenAI (OpenAI. (2023). ChatGPT [Large language model]. </em><a href="https://chat.openai.com/"><em>https://chat.openai.com</em></a><em>), for additional insights and information.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2d1b8babe6c1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding the Confusion Matrix without Confusion]]></title>
            <link>https://medium.com/@msong507/understanding-the-confusion-matrix-without-confusion-126b25dd773c?source=rss-f052cdb2ea66------2</link>
            <guid isPermaLink="false">https://medium.com/p/126b25dd773c</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[logistic-regression]]></category>
            <category><![CDATA[confusion-matrix]]></category>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[model-performance]]></category>
            <dc:creator><![CDATA[misun_song]]></dc:creator>
            <pubDate>Fri, 15 Sep 2023 05:57:21 GMT</pubDate>
            <atom:updated>2023-09-15T06:02:05.226Z</atom:updated>
            <content:encoded><![CDATA[<h4>A Primer for Classification in Machine Learning</h4><h3>Regression and Classification</h3><p>At the heart of machine learning are two core tasks: <strong><em>regression and classification.</em></strong> While regression predicts continuous values (think of predicting house prices with linear regression), classification assigns data to specific categories, often as 0 or 1 (like deciding if an email is spam with logistic regression). But once a classification model is built, how do we gauge its accuracy? Enter the confusion matrix — a vital tool for assessing classification model performance.</p><h3>The Origins of the ‘Confusion’ Matrix</h3><p>The term “confusion matrix” might sound a bit puzzling at first, but its name gives insight into its purpose. A confusion matrix helps us understand where our classification model is getting “confused” or making mistakes.</p><ol><li><strong><em>False Positive (FP):</em></strong> Instances which are actually negative, but the model mistakenly predicts them as positive.</li><li><strong><em>False Negative (FN):</em></strong> Instances which are actually positive, but the model mistakenly predicts them as negative</li></ol><p>The matrix, therefore, provides a consolidated view of where the model is correct and where it is mistaken, hence aiding in diagnosing the specific types of errors it is making.</p><p>Armed with this foundational understanding, let’s delve deeper into the entire matrix and its individual components</p><h3>Deciphering the ‘Confusion’ in Confusion Matrix</h3><p>While the confusion matrix appears as a mere 2x2 grid, it can be daunting due to its combinations and relevant terminologies. Yet, comprehending its core structure is crucial, as it’s a universal and fundamental tool in every classification problem.</p><p>This time, aside from the conventional approach starting from the confusion matrix, we will begin with an intuitive understanding of the concept.</p><h4>Breaking Down the Fundamentals</h4><p><strong><em>The matrix reflects two perspectives: the unchangeable actual class and the model’s prediction.</em></strong> Each perspective has two possible outcomes: true or false. Using “T” and “F” for the actual class and “t” and “f” for model predictions, <strong><em>we have four possible combinations based on the model’s prediction:</em></strong></p><ul><li><strong><em>(T,t) and (F,f):</em></strong> Here, the model nailed it! It predicted correctly.</li><li><strong><em>(T,f) and (F,t):</em></strong> In these cases, the model <strong><em>got confused</em></strong>, predicting incorrectly.</li></ul><p>From these combinations, we derive metrics to gauge the model’s accuracy, always focusing on (T,t) or (F,f) as these represent correct predictions:</p><ul><li><strong><em>Accuracy for Positive Predictions (T,t):</em></strong> (T,t) / [(T,t) + (F,t)], (T,t) / [(T,t) + (T,f)]</li><li><strong><em>Accuracy for Negative Predictions (F,f):</em></strong> (F,f) / [(F,f) + (T,f)], (F,f) / [(F,f) + (F,t)]</li><li><strong><em>Overall Accuracy:</em></strong> (T,t + F,f) / [(T,t) + (F,f) + (T,f) + (F,t)]</li></ul><h4>Unraveling the Terminology</h4><p>Now that we’ve understood the underlying concept, let’s enhance our clarity by introducing the official terminology.:</p><ul><li><strong><em>(T,t) as True Positive:</em></strong> The model predicted positive, and it’s indeed positive.</li><li><strong><em>(F,f) as True Negative:</em></strong> The model predicted negative, and it’s truly negative.</li><li><strong><em>(F,t) as False Positive:</em></strong> The model predicted positive when it’s actually negative. Thus, it’s aptly termed <strong><em>‘False’</em></strong> positive.</li><li><strong><em>(T,f) as False Negative:</em></strong> A slip-up, where the model predicted negative, but it’s positive. Hence, it’s rightly called <strong><em>‘False’</em></strong> negative.</li></ul><p>Now, it’s time to replace the foundational understanding with these terms, and you have the well-known confusion matrix as shown below.</p><h3>The Confusion Matrix Estimates</h3><p>Using these elements, we will now delve into the confusion matrix to evaluate the performance of our classification model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/754/0*P7ZXAWS1QJl9k7h1" /><figcaption><a href="https://encord.com/glossary/confusion-matrix/">https://encord.com/glossary/confusion-matrix/</a></figcaption></figure><h4>Type 1 vs. Type 2 Errors: Understanding the Difference</h4><p>Before diving deeper into each estimate consisting of the elements we saw above, let’s learn about Type 1 and Type 2 Errors. <strong><em>Type 1 and Type 2 Errors arise when a model makes a false prediction.</em></strong> They reflect instances when the model’s prediction is the opposite of the truth.</p><ul><li><strong><em>Type 1 Error:</em></strong> Occurs when the model predicts a positive outcome, but in reality, it’s negative. <strong><em>Essentially, it’s a false alarm.</em></strong></li><li><strong><em>Type 2 Error:</em></strong> Represents the opposite scenario. The model predicts a negative outcome, but the reality is positive. <strong><em>This is akin to a missed opportunity.</em></strong></li></ul><p>While the goal is to minimize both errors, can we prioritize one over the other? <strong><em>Traditionally, Type 1 Error is perceived as more detrimental.</em></strong> Consider an investment scenario. A Type 1 Error, or false positive, might prompt us to invest in a venture with no actual potential. Meanwhile, a Type 2 Error, or false negative, could deter us from a golden opportunity. While the latter only results in a missed chance, the former can lead to <strong><em>tangible losses</em></strong>, making it, in many contexts, the graver error.</p><h4>True Positives: Precision vs. Sensitivity (Recall)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A6ks84hIiCEiFKn-FlvAAg.png" /><figcaption><a href="https://www.v7labs.com/blog/f1-score-guide">https://www.v7labs.com/blog/f1-score-guide</a></figcaption></figure><p><strong><em>Precision</em></strong> indicates accuracy of the model’s positive predictions. It asks among the instances the model predicted as positive, how many were actually positive.</p><p><strong><em>Sensitivity</em></strong><em> r</em>efers to the completeness of capturing actual positive cases. It asks among all the actual positive instances, how many the model correctly predicted as positive.</p><h4>True Negatives: Negative Predictive Value (NPV) vs. Specificity</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*cmxkr5wCep2q2Zfs.jpg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*1CDjpSKbswUN3co8.jpg" /><figcaption>https://geekymedics.com/sensitivity-specificity-ppv-and-npv/</figcaption></figure><p><strong><em>NPV</em></strong><em> r</em>eflects accuracy of the model’s negative predictions. Specifically, of the instances the model labeled as negative, how many were truly negative?</p><p><strong><em>Specificity</em></strong> measures how well the model recognizes actual negative cases. It asks from all the actual negative instances, how many were correctly predicted as negative by the model.</p><h4>Overall Performance: Accuracy</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SGVNCB-Hh-ehtI2xKiOurw.png" /><figcaption><a href="https://hasty.ai/docs/mp-wiki/metrics/accuracy">https://hasty.ai/docs/mp-wiki/metrics/accuracy</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jI0yveiUdV2aiR3zjDMHsQ.png" /><figcaption><a href="https://hasty.ai/docs/mp-wiki/metrics/accuracy">https://hasty.ai/docs/mp-wiki/metrics/accuracy</a></figcaption></figure><p><strong><em>Accuracy</em></strong><em> </em>represents<em> </em>the overall correctness of the model’s predictions, both positive and negative.</p><p><strong><em>Remember, while accuracy gives a broad overview of performance, it might not capture the complete picture, especially in imbalanced datasets.</em></strong> In such cases, considering metrics like the F1 Score can provide more insight.</p><h3>Precision-Recall Trade-Off</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jd4YztqFrbY1NunE.gif" /><figcaption><a href="https://www.analyticsvidhya.com/blog/2020/10/how-to-choose-evaluation-metrics-for-classification-model/">https://www.analyticsvidhya.com/blog/2020/10/how-to-choose-evaluation-metrics-for-classification-model/</a></figcaption></figure><p><strong><em>Precision and recall, two pivotal metrics in classification, often stand in opposition.</em></strong> At its core, precision is concerned with how many of the predicted positive cases are truly positive. The higher the precision, the fewer false positives there are. Recall, on the other hand, focuses on capturing as many of the actual positive cases as possible, even if it means wrongly classifying some negatives as positives.</p><p>When we <strong><em>lower our model’s threshold</em></strong> to detect more true positives (boosting recall), we also increase the risk of falsely classifying negatives as positives, which decreases precision. Conversely, <strong><em>raising the threshold</em></strong> to enhance precision may lead to missing out on some actual positive cases, hence reducing recall. This balancing act between precision and recall is what we term the “precision-recall trade-off.” Making informed decisions in the context of this trade-off is crucial, as different scenarios might demand prioritizing one metric over the other.</p><h3>Imbalanced Data Set and the Role of F1 Score</h3><h4>F1 Score</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yjlhty-7CWdC2FlqNJaiZg.png" /><figcaption><a href="https://www.v7labs.com/blog/f1-score-guide">https://www.v7labs.com/blog/f1-score-guide</a></figcaption></figure><p>The harmonic mean, utilized in computing the F1 score, is distinct from the arithmetic mean. Instead of averaging two numbers directly, it averages their reciprocals and then takes the reciprocal of that result.</p><p><strong><em>This method is inherently more “punitive.”</em></strong> If either precision or recall is exceptionally low, the F1 score will also be markedly reduced due to the nature of the harmonic mean. An elevated F1 score is an indicator that both precision and recall are commendable.</p><h4>Assessing Model Performance in Imbalanced Datasets</h4><p>Imbalanced datasets, where one class far outnumbers the other, are common in areas like <strong><em>disease detection, fraud identification, and customer churn prediction.</em></strong> These pose unique challenges in assessing model performance.</p><p>When a model leans heavily towards the majority class, it may inaccurately favor that class. Consider a dataset where 95% belong to the negative class. If a model labels everything as negative, it may seem impressively accurate at 95%. <strong><em>However, this isn’t genuine proficiency but a bias towards the majority.</em></strong></p><p>In such skewed scenarios, precision often appears high, primarily because there are fewer instances of the minority class, reducing the chance of false positives. But recall, which aims to identify all true instances of the minority class, can struggle, leading to its decrease. This tension between precision and recall results in a lower F1 score.<strong><em> Thus, while accuracy may appear high, the F1 score offers a more truthful measure of a model’s performance in imbalanced settings.</em></strong></p><h3>Conclusion</h3><p>The confusion matrix, while deceptively simple, unlocks deep insights into machine learning’s intricacies. It reveals the nuances of predictions and teaches us that raw accuracy isn’t always the ultimate benchmark, especially with imbalanced datasets. As we navigate this realm, the matrix serves as a reminder that understanding our models is as pivotal as their creation.</p><p>For a detailed walkthrough, check out the accompanying video!</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FKdsp6soqA7o%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DKdsp6soqA7o&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FKdsp6soqA7o%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/491aec55b7f5aeddebfd36d5a03e2b6b/href">https://medium.com/media/491aec55b7f5aeddebfd36d5a03e2b6b/href</a></iframe><p>Want a catchy way to remember the confusion matrix? Tune into our song!</p><p><a href="https://www.youtube.com/shorts/PWvfrTgaPBI">The Sensitivity, Specificity, Precision, Recall Sing-a-Long!!!</a></p><p><em>In preparing my article, I consulted ChatGPT, a large language model developed by OpenAI (OpenAI. (2023). ChatGPT [Large language model]. </em><a href="https://chat.openai.com/"><em>https://chat.openai.com</em></a><em>), for additional insights and information.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=126b25dd773c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bridging Data Science with Your Business Acumen]]></title>
            <link>https://medium.com/@msong507/bridging-data-science-with-domain-expertise-77196e96f1ff?source=rss-f052cdb2ea66------2</link>
            <guid isPermaLink="false">https://medium.com/p/77196e96f1ff</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[business]]></category>
            <category><![CDATA[career-advice]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[career-change]]></category>
            <dc:creator><![CDATA[misun_song]]></dc:creator>
            <pubDate>Tue, 12 Sep 2023 04:49:17 GMT</pubDate>
            <atom:updated>2023-09-14T19:26:57.560Z</atom:updated>
            <content:encoded><![CDATA[<h4>The Transformative Power of Your Professional Journey</h4><h3>The Evolution of the Data Scientist</h3><blockquote>Digital transformation has amplified the role of data scientists, enabling them to apply domain expertise across diverse data science areas.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OlP9X2GTqoG4Pmq8.jpg" /><figcaption><a href="https://www.thoughtco.com/sentence-combining-grammar-1692085">https://www.thoughtco.com/sentence-combining-grammar-1692085</a></figcaption></figure><p><strong><em>Data science, in its essence, has always been with us.</em></strong> Historically, Doctors, statisticians, and engineers, among others, diligently applied analytical skills, albeit often within their specific domain of expertise. So, why the sudden buzz around <a href="https://www.hbs.edu/faculty/Pages/item.aspx?num=43110">“Data Scientist: The Sexiest Job of the 21st Century”</a>? The digital transformation has democratized data storage and analysis tools, broadening the scope of data science. <strong><em>In defining the ‘modern’ data scientist, while they may begin with expertise in a particular domain, they possess the unique ability to apply their business acumen universally across multiple areas within data science.</em></strong> Let’s explore further.</p><h3>Business Acumen 1: Navigating Beyond Precision</h3><blockquote>Data science should prioritize context and purpose over mere precision, using the right analytical methodologies for each unique situation, guided by domain expertise.</blockquote><p>Data science, equipped with advanced machine-learning tools, offers unparalleled precision in analysis. Yet, precision alone is not always the end goal. More critical is understanding the ‘why’ behind the analysis. Without this clear purpose, even the most sophisticated tools can miss the mark, providing answers that may be technically accurate but contextually misplaced.</p><p>Consider the simplicity of linear regression, a topic we’ve delved into before (<a href="https://medium.com/@msong507/linear-regression-fundamentals-489e6d60d5cc"><em>Linear Regression Fundamentals</em></a>). Advanced algorithms might offer higher accuracy, but for certain sectors, like manufacturing, it’s not just about precision. It’s about understanding the relationship between factors. In this context, linear regression might be more suited, as it helps manufacturers understand how different variables relate, potentially cutting down defects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/897/0*Zh1VlRhIA50lVvQ6.jpg" /><figcaption><a href="https://www.frontiersin.org/articles/10.3389/fpubh.2017.00307/full">https://www.frontiersin.org/articles/10.3389/fpubh.2017.00307/full</a></figcaption></figure><p>Take another example: COVID-19 testing. During the pandemic’s peak, the primary goal was speed. Rapid tests, even with their occasional errors, were vital to quickly identify and isolate cases. As the urgency of the situation reduced, the focus shifted to precision. The ‘why’ behind the testing changed, and so did the methods and thresholds.</p><p>This emphasizes a crucial point: <strong><em>Analysis isn’t just about finding the most accurate answer, but the right answer for the given context. And that’s where domain expertise comes into play. </em></strong>Professionals experienced in a particular field can choose the right analytical tools to meet their specific needs, ensuring results are both accurate and meaningful for their unique situations.</p><h3>Business Acumen 2: Grounding Decisions in Reality</h3><blockquote>Domain knowledge and experience are crucial for accurate data interpretation, as raw data and logic can be misleading without real-world context.</blockquote><p>Raw data and pure logic, on their own, can often paint a misleading picture. In many cases, it’s our hands-on experience and deep understanding of a specific field that provide the necessary context to interpret data accurately.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*vmrlKTT28Xwo7OPQBaJ4lQ.png" /><figcaption><a href="https://campus.datacamp.com/courses/data-visualization-in-power-bi/less-is-more?ex=6">https://campus.datacamp.com/courses/data-visualization-in-power-bi/less-is-more?ex=6</a></figcaption></figure><p>Consider the example from a <a href="https://campus.datacamp.com/courses/data-visualization-in-power-bi/less-is-more?ex=6">DataCamp course</a>. Here, students are tasked with creating a KPI dashboard for a hypothetical clothing company, focusing on tracking returned items. On the surface, setting a concrete target for returns might seem logical. However, in the intricate dynamics of real-world retail, setting such a target is less meaningful than monitoring trends in returns. Observing patterns, like a consistent high return rate compared to the previous season or a sudden spike compared to the previous day, provides more actionable insights than a static goal. Although this example was created for educational purpose, it mirrors the complexities and nuances professionals often grapple with in the field.</p><p>Such examples spotlight the inherent limitations of relying purely on data and logic. <strong><em>Without the guidance of domain knowledge and field experience, even the most sophisticated data analysis can be off-target.</em></strong> It’s our familiarity with industry-specific intricacies that helps us avoid these pitfalls, ensuring our data-driven decisions align with the real-world scenarios.</p><h3>Business Acumen 3: Choosing the Optimal Toolset</h3><blockquote>In data science, choosing the right tool — be it Python, R, Tableau, or Power BI — depends on the specific task and context, with deep domain knowledge guiding optimal decisions.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LrETuEPTStdbnTqT.jpg" /><figcaption><a href="https://www.bridge-global.com/blog/r-and-python/">https://www.bridge-global.com/blog/r-and-python/</a></figcaption></figure><p>In the dynamic realm of data science, tools and methodologies are abundant, yet the key to successful analysis often lies in choosing the right instrument for the job. Python, with its vast applications, has emerged as a go-to for many due to its adaptability. On the flip side, when the task calls for deep statistical dives, R steps up, tailor-made to distill complex problems into comprehensible insights.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OlB0ud-0KzvnO6Lw.png" /><figcaption><a href="https://unitedtraining.com/resources/blog/microsoft-power-bi-vs-tableau">https://unitedtraining.com/resources/blog/microsoft-power-bi-vs-tableau</a></figcaption></figure><p>This principle of selecting the ideal tool extends to data visualization as well. Tableau, with its flair for vivid, custom visuals, contrasts with Power BI, which aligns perfectly with those seeking professional reports, especially within businesses already rooted in the Microsoft landscape.</p><p><strong><em>However, the heart of these choices is always context.</em></strong> If you find yourself sifting through customers’ reviews, aiming to capture sentiments using Natural Language Processing, Python becomes indispensable. Yet, when the rhythm of seasonal sales patterns beckons, R stands out as the fitting contender. I’ve felt the weight of such decisions firsthand. At my previous company, we gravitated towards Tableau, drawn in by its widespread praise. Yet, our needs mostly revolved around straightforward data tabulations rather than elaborate visual representations. In retrospect, Power BI’s seamless integration with our other Microsoft tools would have served us better.</p><p>In essence, the landscape of data tools is vast, but one’s path becomes clearer with deep domain knowledge. <strong><em>Truly understanding the specifics of your tasks and objectives is the compass that directs you to the optimal analytical instrument.</em></strong></p><h3>Business Acumen 4: Turning Insights into Impact</h3><blockquote>Effective data analysis requires translating raw data into actionable insights, with industry-specific experience guiding decisions beyond mere technical proficiency.</blockquote><p>During my internship, our team produced a textbook-perfect Tableau dashboard spotlighting search keyword volumes in the fashion industry. While our visualization was technically flawless, it failed to bring new revelations: the fact that searches for jumpers ascend in November or that short pants are a summer favorite wasn’t groundbreaking to our client. They sought fresh insights, those emerging trends not yet in the mainstream. Thankfully, by contrasting absolute search volumes with the sudden upticks in specific keywords, we were able to pivot and win back their trust. (<a href="https://medium.com/@msong507/trend-analytics-fashions-digital-dive-450764452cf5"><em>Trend Analytics: Fashion’s Digital Dive</em></a>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-tL9f2r57whjYPCT.png" /><figcaption><a href="https://medium.com/@msong507/dispelling-the-10-growth-plan-part1-114a68689cc3">https://medium.com/@msong507/dispelling-the-10-growth-plan-part1-114a68689cc3</a></figcaption></figure><p>Another client, also in the fashion sector, requested we integrate weather data into our dashboard. The world of meteorological information is vast, encompassing metrics from temperature and rainfall to wind speed and humidity. Yet, with an understanding that the crux for this industry was optimizing winter inventory rotation, our focus narrowed down to temperature — emphasizing its regional variances and forward-looking predictions.</p><p><strong><em>At its core, data analysis is about translating raw figures into actionable business insights.</em></strong> Aesthetically pleasing visualizations or intricate algorithms might dazzle, but without alignment to a concrete goal, they risk redundancy in practical applications.<strong><em> This is where pure domain experience, transcending mere knowledge, becomes paramount. </em></strong>It’s an area that isn’t simply satisfied by a data scientist’s creativity, but requires a tangible touch grounded in the realities of the industry.</p><h3>Conclusion</h3><blockquote>Data science, when paired with domain knowledge, drives transformative change.</blockquote><p>Data science offers an insightful perspective, enabling us to understand our surroundings, influence others, and make visionary decisions. However, its potency is magnified when paired with domain knowledge. Previously, you might’ve felt moments of frustration in a non-data-centric environment, but it’s precisely in such realms that the union of field experience and data science can bring about revolutionary change. Embrace that unique blend, for it holds the key to a data-driven transformation.</p><p><em>In preparing my article, I consulted ChatGPT, a large language model developed by OpenAI (OpenAI. (2023). ChatGPT [Large language model]. </em><a href="https://chat.openai.com/"><em>https://chat.openai.com</em></a><em>), for additional insights and information.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=77196e96f1ff" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Linear Regression Fundamentals]]></title>
            <link>https://medium.com/@msong507/linear-regression-fundamentals-489e6d60d5cc?source=rss-f052cdb2ea66------2</link>
            <guid isPermaLink="false">https://medium.com/p/489e6d60d5cc</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[predictions]]></category>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[misun_song]]></dc:creator>
            <pubDate>Tue, 29 Aug 2023 22:47:17 GMT</pubDate>
            <atom:updated>2023-09-14T19:27:17.817Z</atom:updated>
            <content:encoded><![CDATA[<h4>Understanding the Core Concepts and Applications</h4><h3>Data Science 101: Linear Regression</h3><p>In any foundational data science course, one of the first topics you’ll encounter is linear regression. While it may not dominate conversations in the age of AI and deep learning, it remains a cornerstone of predictive analytics. Today, we’ll delve into the significance of linear regression and its core principles, steering clear of dense mathematical jargon.</p><h3>Importance of Linear Regression</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9wBPf1N-_HbbVxUM.jpg" /><figcaption><a href="https://www.formica.ai/blog/which-ai-is-learn-by-its-own">https://www.formica.ai/blog/which-ai-is-learn-by-its-own</a></figcaption></figure><p>Predicting the future based on past data is nothing short of remarkable. With advancements in machine learning algorithms, predictions have become astonishingly accurate. However, this often leads to a misconception: that complexity equates to better performance. This isn’t always the case, and linear regression is a prime example.</p><p><strong><em>Linear regression draws a straight line that best fits the data.</em></strong> Its performance is measured using R-squared, a straightforward metric. The beauty of linear regression is its transparency: it reveals the weight of each factor on the outcome. While advanced algorithms might offer pinpoint predictions, they often work as <strong><em>‘black boxes’</em></strong>, obscuring how decisions are made.</p><p>Sometimes, understanding the <strong><em>‘why’</em></strong> behind an outcome is more crucial than the ‘how likely’. Take a factory setting, for instance. Boosting the predictive accuracy of defect occurrences from 95% to 97% may not drastically change operations. However, identifying and addressing the root cause of these defects could be game-changing.</p><h3>Elements of Linear Regression</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3XzCjKuUyBIN8_O1.jpg" /><figcaption><a href="https://booksite.elsevier.com/9780128017128/chapter1_7.php">https://booksite.elsevier.com/9780128017128/chapter1_7.php</a></figcaption></figure><p>Data science essentially involves using past data to predict future trends. One of the primary tools for this task is ‘linear regression’. Picture it as drawing a line that best fits a scatter plot of data points. This line can then be used to make predictions.</p><h4>Key Terms in Linear Regression:</h4><ol><li><strong>Independent Variable:</strong> <em>The factor you base your prediction on.</em> For instance, ‘age’ could be used to estimate someone’s weight.</li><li><strong>Dependent Variable: </strong><em>What you’re trying to predict.</em> Using the previous example, ‘weight’ is what we’re trying to estimate based on age.</li><li><strong>Weight: </strong><em>This represents the change in the dependent variable when the independent variable increases by one unit</em>. So, if this value is 2, and age increases by a year, the predicted weight might rise by 2 pounds.</li><li><strong>Intercept:</strong> <em>The baseline or starting value.</em> In our weight-age example, it’s the foundational weight before age is taken into account.</li><li><strong>Error:</strong> <em>The difference between our predicted and the actual value.</em> If we predict a weight of 150 pounds based on age but the actual weight is 160 pounds, the error is 10 pounds.</li></ol><p>You might notice terms like <strong><em>‘beta zero’</em></strong> for the intercept and <strong><em>‘beta one’ </em></strong>for the weight. These are ways to indicate coefficients, and <strong><em>as we add more independent variables, we get beta two, three, and so forth.</em></strong> When using just one independent variable to predict the dependent variable, it’s termed <strong><em>‘simple linear regression’.</em></strong> If more than one independent variable is used, it’s <strong><em>‘multiple linear regression’.</em></strong> While linear regression might remind some of the basic linear equations, understanding the differences between the two is pivotal.</p><h3>Linear Regression vs. Linear Equation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/420/0*7X04I2yUHlq2rdiU.jpg" /><figcaption><a href="https://www.mathematics-monster.com/lessons/linear_equations_line_between_points.html">https://www.mathematics-monster.com/lessons/linear_equations_line_between_points.html</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/0*2Ht62YclrNc3jMgl.jpg" /><figcaption><a href="https://www.mathematics-monster.com/lessons/linear_equations_line_between_points.html">https://www.mathematics-monster.com/lessons/linear_equations_line_between_points.html</a></figcaption></figure><p>With just one data point, establishing a trend is impossible. However, introduce a second point, and we can define <strong><em>a straight line with its slope (weight) and intercept,</em></strong> a basic principle many are familiar with.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZPjT9Uw55CzLDn2y.png" /><figcaption><a href="https://paperswithcode.com/method/linear-regression">https://paperswithcode.com/method/linear-regression</a></figcaption></figure><p>But what if we have an <a href="https://en.wikipedia.org/wiki/Overdetermined_system">“overdetermined situation”</a> with more than two data points? In many real-world scenarios, we’re faced with a multitude of data points. <strong><em>Yet, with linear regression, our objective is to encapsulate this data using a single line.</em></strong> While linear algebra delves deeper into understanding such situations, we’ll keep that discussion for another day. The key takeaway here is this: when we strive for the best-fitting line in linear regression, <strong><em>it’s uncommon for that line to perfectly pass through every data point.</em></strong></p><h3>Best-Fitting Line</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Z5Y2J0dEIPeVuwEE.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Mj3PDsBKFLWrcEb0.png" /><figcaption>https://learningstatisticswithr.com/book/regression.html</figcaption></figure><p>As previously mentioned, the line in linear regression seldom passes through every data point; instead, our aim is to find the most optimal line that best represents the data. How do we define the <strong><em>“best”</em></strong> in the best-fitting line? In linear regression, the primary goal is <strong><em>to minimize the difference between observed (actual) and predicted values.</em></strong> The data points not on the line signify prediction discrepancies. These discrepancies are termed <strong><em>“residuals”.</em></strong> However, it’s crucial to distinguish between residuals and errors:</p><ul><li><strong><em>Residuals: </em></strong>These represent the difference between the observed values and the values predicted by the model in your sample data.</li><li><strong><em>Errors:</em></strong> Refers to the difference between the observed values and the values that would be predicted by the true regression line (a line we can’t see because we only have sample data, not the entire population).</li></ul><p>In practice, we often hear both terms used interchangeably, but “error” is more general, while “residual” is specific to the observed sample.</p><h3>SSE vs. MSE vs. RMSE in Linear Regression</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*29QqqNyOOm2mWU9Y.png" /><figcaption><a href="https://www.statology.org/residuals/">https://www.statology.org/residuals/</a></figcaption></figure><blockquote>The difference between the prediction and the observed value is the residual. If we plot the observed values and overlay the fitted regression line, the residuals for each observation would be <strong>the vertical distance between the observation and the regression line </strong>— What Are Residuals in Statistics? by <a href="https://www.statology.org/author/admin/">ZACH</a> from Statology</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/397/0*y1WsIDc7fPp9yJ4s.png" /><figcaption><a href="https://blog.csdn.net/qq_31020665/article/details/100080181">https://blog.csdn.net/qq_31020665/article/details/100080181</a></figcaption></figure><p>In linear regression, the goal is to minimize the errors between actual and predicted values. This ‘best fitting line’ allows us to make accurate future predictions. The discrepancies can be quantified using SSE, MSE, and RMSE, metrics that are pivotal not just in linear regression but also in various machine learning models.</p><h4>SSE (Sum of Squared Errors)</h4><p>The direct summation of residuals (differences between predicted and actual values) is flawed, <strong><em>as positive and negative residuals cancel each other out, resulting in zero.</em></strong> Hence, to compare models effectively, <strong><em>residuals are squared before being summed, giving us the SSE. </em></strong>It measures the model’s total error. Larger SSE values signify poor model fit. However, its sheer magnitude makes it hard to interpret, especially across models with varying data points.</p><p>For instance, a model with 100,000 data points might inherently have a larger SSE than one with just 10 data points purely because of the larger number of observations, even if the model with fewer data points is less accurate on a per-point basis. This highlights the difficulty of comparing SSE values across models with vastly different sample sizes.</p><h4>MSE (Mean Squared Error)</h4><p>MSE addresses SSE’s interpretability issue. <strong><em>By averaging the squared residuals, we get a clearer understanding of the model’s error.</em></strong> It’s a favored metric in regression because it provides a standardized error measure across models. However, both SSE and MSE amplify the scale of the errors because of squaring, which might distort their representation, especially for large errors.</p><h4>RMSE (Root Mean Squared Error)</h4><p>Enter RMSE, the antidote to the scale distortion in MSE. <strong><em>By taking the square root of MSE, we revert the error metric to the original scale of the data.</em></strong> This makes RMSE more interpretable, and it retains an emphasis on larger errors, penalizing them more than smaller ones.</p><p>In essence, while all these metrics quantify the prediction errors, the choice between them hinges on the specific context and interpretability needs.</p><h3>R-Squared</h3><p>Linear regression is like a detective trying to predict an outcome (dependent variable) based on clues (independent variables). While our detective can make several educated guesses using various clues, not all are equally good. In a world filled with countless data points, our prediction line won’t touch them all. <strong><em>So, which prediction, or model, should we trust? This is where R-squared steps in.</em></strong></p><p>Imagine you’re trying to figure out how advertising affects sales. If advertising is a crucial factor, it should explain most of the changes in sales. R-squared quantifies this relationship: a higher value indicates that our chosen clue (like advertising) effectively predicts the outcome.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/625/0*Kt4wOvzkScgrPOfp" /><figcaption><a href="http://www.fairlynerdy.com/what-is-r-squared/">http://www.fairlynerdy.com/what-is-r-squared/</a></figcaption></figure><p>To get to R-squared, we start with a simple prediction:<strong><em> the average of all data points, our baseline.</em></strong> If our detective’s educated guess isn’t much better than this average, then maybe the clues chosen aren’t that helpful.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/998/0*JA1T-lzC9Asljsn7.png" /><figcaption><a href="https://begin4learn.gitbooks.io/begin-to-learn-statistics/content/Statistics/Coefficient_of_determination.html">https://begin4learn.gitbooks.io/begin-to-learn-statistics/content/Statistics/Coefficient_of_determination.html</a></figcaption></figure><p>In technical terms, we measure the distance of actual values from this average, known as <strong><em>the Total Sum of Squares (SST).</em></strong> Then, we see how our predicted values, using the clues, differ from this average, termed the <strong><em>Explained Sum of Squares (RSS).</em></strong> The difference between SST and RSS is <strong><em>the Residual Sum of Squares (SSE)</em></strong> — the bit our model hasn’t quite figured out. In essence, R-squared captures how much of the entire story our clues tell us. A higher R-squared? Our detective’s on the right track.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/963/0*m3-A09IN8T6-KcNB.png" /><figcaption><a href="https://begin4learn.gitbooks.io/begin-to-learn-statistics/content/Statistics/Coefficient_of_determination.html">https://begin4learn.gitbooks.io/begin-to-learn-statistics/content/Statistics/Coefficient_of_determination.html</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/562/0*WYLQPmOzJINc_-ix.png" /><figcaption><a href="https://begin4learn.gitbooks.io/begin-to-learn-statistics/content/Statistics/Coefficient_of_determination.html">https://begin4learn.gitbooks.io/begin-to-learn-statistics/content/Statistics/Coefficient_of_determination.html</a></figcaption></figure><p>After we’ve unraveled the intricacies of SSE, MSE, and RMSE, the natural question arises: “Why introduce R-squared into the mix?”</p><ol><li><strong>Comparability:</strong> R-squared provides a level playing field. It allows for <strong><em>model comparisons irrespective of the scale of the data or the size of the dataset,</em></strong> making it a versatile metric. In contrast, metrics like SSE can be misleading due to varying sample sizes or magnitudes.</li><li><strong>Interpretability:</strong> While SSE, MSE, and RMSE give absolute measures of model error, their values can sometimes be daunting to interpret. R-squared offers clarity — <strong><em>it’s a relative measure that presents the proportion of variance</em></strong> in the dependent variable explained by the independent ones.</li><li>Standardization: The beauty of R-squared lies in its universality. It’s unitless, meaning <strong><em>it doesn’t hinge on the specific measurement units of the dependent variable,</em></strong> making it applicable across varied problems and datasets.</li></ol><h3>Adjusted R-Squared</h3><p>In linear regression, adding more predictors (independent variables) typically causes the R-squared value to rise, even if these predictors don’t necessarily improve the model’s predictive power. (For a deeper dive into this topic, please refer to the video linked below.)</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F4otEcA3gjLk%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D4otEcA3gjLk&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F4otEcA3gjLk%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/f12952724cc643b5500079e99fb8ad49/href">https://medium.com/media/f12952724cc643b5500079e99fb8ad49/href</a></iframe><p>However, incorporating too many predictors can introduce multiple complications:</p><ol><li><strong>Overfitting: </strong>A model with numerous predictors might closely fit the training data, but it risks failing to generalize effectively to new, unseen data.</li><li><strong>Multicollinearity:</strong> As the number of predictors grows, the potential for correlations between them increases. In linear regression, it’s crucial that predictors remain independent to clearly interpret their individual effects on the dependent variable.</li><li><strong>Elevated Variance: </strong>While an increased number of predictors grants the model greater flexibility, it also amplifies the model’s sensitivity to minor variations in the data.</li><li><strong>Compromised Interpretability: </strong>One of linear regression’s advantages is its transparency in elucidating the relationship between predictors and the dependent variable. However, with too many predictors, this transparency diminishes, obscuring the understanding of predictor effects.</li></ol><p>This is where Adjusted R-Squared steps in. Unlike its counterpart, R-squared, which invariably rises with the inclusion of more predictors, Adjusted R-squared introduces a penalty for unnecessary complexity. It offers several key benefits:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/319/0*fTk9pWS1_cr5gJ_D.png" /><figcaption><a href="https://datastoriesweb.wordpress.com/2017/01/15/interpreting-statistical-values/">https://datastoriesweb.wordpress.com/2017/01/15/interpreting-statistical-values/</a></figcaption></figure><ol><li><strong>Punishing Complexity:</strong> Adjusted R-squared provides a more conservative measure of fit. <strong><em>It penalizes the introduction of superfluous predictors, </em></strong>ensuring that only variables enhancing the model’s predictive power are rewarded.</li><li><strong>Facilitating Model Comparison:</strong> When evaluating multiple models, each might have <strong><em>a different number of predictors.</em></strong> Adjusted R-squared enables a more equitable comparison by accounting for model complexity.</li><li><strong>Mitigating Overfitting:</strong> Adjusted R-squared serves as a check against the inclination to over-complicate a model. By incorporating a penalty for extraneous predictors, <strong><em>it encourages parsimony,</em></strong> ensuring the model retains its generalization capabilities.</li></ol><p>In summary, while R-squared provides an initial assessment of model fit, Adjusted R-squared delivers a deeper, more discerning evaluation. It’s imperative to consider both metrics when finalizing model selection.</p><h4>P-Value</h4><pre># Create vectors for Advertising and Sales data<br>advertising &lt;- c(80000, 80000, 90000, 100000, 100000, 110000, 120000, 120000, 130000, 140000)<br>sales &lt;- c(499000, 477000, 546000, 614000, 623000, 653000, 747000, 714000, 785000, 858000)<br><br># Fit the linear regression model<br>model &lt;- lm(sales ~ advertising)<br><br># Display the model&#39;s summary<br>summary(model)</pre><pre>Call:<br>lm(formula = sales ~ advertising)<br><br>Residuals:<br>     Min       1Q   Median       3Q      Max <br>-16732.3  -9960.6    929.1   9679.1  16826.8 <br><br>Coefficients:<br>             Estimate Std. Error t value Pr(&gt;|t|)    <br>(Intercept) 4.882e+03  2.302e+04   0.212    0.837    <br>advertising 6.044e+00  2.116e-01  28.561 2.44e-09 ***<br>---<br>Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1<br><br>Residual standard error: 13060 on 8 degrees of freedom<br>Multiple R-squared:  0.9903, Adjusted R-squared:  0.9891 <br>F-statistic: 815.7 on 1 and 8 DF,  p-value: 2.442e-09</pre><p>Imagine you’re testing a new plant food to see if it boosts plant height. You give some plants the new food, but not others. A few weeks later, those with the new food tower over the rest. In this scenario, the p-value serves as our confidence checker. A low p-value indicates we can trust the new food truly benefits the plants. A high one? The growth might just be random.</p><p>In research, a low p-value suggests our observations are likely genuine, not just flukes. It helps determine the trustworthiness of factors in our predictions. The given example helps illuminate this concept.</p><p>From the provided output:</p><ul><li>For the (Intercept): p-value is 0.837.</li><li>For advertising: p-value is 2.44e-09 (which means 2.44×10−92.44×10−9).</li></ul><p>The p-value is an indicator of the evidence against a null hypothesis. Smaller p-values suggest stronger evidence against the null hypothesis. Here’s a general guideline:</p><ul><li><em>p </em>&lt; 0.05: statistically significant</li><li><em>p </em>&lt; 0.01: strongly significant</li><li><em>p </em>&lt; 0.001: very strongly significant</li></ul><p>Now, let’s interpret the p-values:</p><ol><li>For the (Intercept): The p-value is 0.837, which is greater than 0.05. This implies that there is no statistically significant evidence to reject the null hypothesis for the intercept, meaning the intercept may not be significantly different from zero.</li><li>For advertising: The p-value is extremely small (2.44e-09), way less than 0.001. This implies that there is very strong evidence against the null hypothesis for the advertising coefficient. In other words, advertising has a statistically significant effect on sales.</li></ol><p>In simple terms, for every unit increase in advertising, sales are expected to increase by an estimated 6.044 units, <strong><em>and this effect is highly significant.</em></strong> The advertising expenditure significantly predicts sales and the likelihood of observing this effect if the true coefficient was zero is extremely low (essentially, almost zero).</p><h3>Conclusion</h3><p>We’ve delved into the essentials of linear regression, yet much remains to be explored. While we’ve touched upon determining the best-fit line, we haven’t dived into the methods behind it, such as the numerical solution using the normal equation or the iterative approach via gradient descent <a href="https://medium.com/@msong507/gradient-descent-intuitive-overview-43e7869e1545">(For a deeper dive, check out, “Gradient Descent — Intuitive Overview”)</a>. After constructing a model, it’s crucial to validate it by examining the residuals — through plotting, assessing homoscedasticity, testing for normality, and checking for autocorrelation. Recall that once all patterns in the data are identified, residuals should appear entirely random. As we move forward, we’ll delve deeper into these facets since they intertwine with other data science topics.</p><p>Yet, this discussion should serve as a reliable springboard as you dive deeper into the world of data science. As you embark on your data science endeavors, remember to build on this foundation. Happy linear regression!</p><p><em>Please see the attached videos for a more detailed explanation of linear regression.</em></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FeYTumjgE2IY%3Fstart%3D2%26feature%3Doembed%26start%3D2&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DeYTumjgE2IY&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FeYTumjgE2IY%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/5c36cb550276c707129925e171ace5e0/href">https://medium.com/media/5c36cb550276c707129925e171ace5e0/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FPaFPbb66DxQ%3Fstart%3D43%26feature%3Doembed%26start%3D43&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DPaFPbb66DxQ&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FPaFPbb66DxQ%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/a0e28fc39043aca0a7b7412c4d7a7192/href">https://medium.com/media/a0e28fc39043aca0a7b7412c4d7a7192/href</a></iframe><p><em>In preparing my article, I consulted ChatGPT, a large language model developed by OpenAI (OpenAI. (2023). ChatGPT [Large language model]. </em><a href="https://chat.openai.com/"><em>https://chat.openai.com</em></a><em>), for additional insights and information.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=489e6d60d5cc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Trend Analytics: Fashion’s Digital Dive]]></title>
            <link>https://medium.com/@msong507/trend-analytics-fashions-digital-dive-450764452cf5?source=rss-f052cdb2ea66------2</link>
            <guid isPermaLink="false">https://medium.com/p/450764452cf5</guid>
            <category><![CDATA[fashion-trends]]></category>
            <category><![CDATA[business-analytics]]></category>
            <category><![CDATA[business-strategy]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[fashion]]></category>
            <dc:creator><![CDATA[misun_song]]></dc:creator>
            <pubDate>Tue, 22 Aug 2023 19:36:58 GMT</pubDate>
            <atom:updated>2023-09-14T19:27:41.216Z</atom:updated>
            <content:encoded><![CDATA[<h4>Using Search Volume Analysis to Unveil Upcoming Fashion Trends</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/568/1*dTKzsgqYe2h9_LvOUe5vYw.jpeg" /></figure><h3>Fashion and Tech Collaboration: Beyond the Surface</h3><blockquote>Fashion’s inherent structure and data science’s predictive prowess make for a transformative collaboration.</blockquote><p>At first glance, fashion dazzles with creativity and flair. Yet beyond its glittering surface lies a structured world. Similarly, data science does more than just gaze into the future — it’s a continuum that begins with deciphering past patterns and seamlessly progresses to connecting the dots for future insights. By recognizing and intertwining these deep-seated aspects, fashion and data science are set for a groundbreaking collaboration.</p><h3>Reframing the Fashion Process: Present and Future</h3><blockquote>Amid the digital shift in fashion, with significant online transactions, the industry must use data analytics to blend real-time trends with its inherent forward-looking nature.</blockquote><p>In the digital age, the fashion industry finds itself grappling with intensified immediacy. The overwhelming shift towards online commerce — evidenced by data such as 13.5% of South Korea’s online transactions being fashion-centric <a href="https://www.cosinkorea.com/news/article.html?no=47367">(amounting to an impressive 28 trillion KRW or approximately 21 billion USD, according to KOSIS) </a>— highlights the urgent need to analyze and adapt to present trends.</p><p>However, at its core, fashion thrives on foresight. Typically demanding up to two years from idea conception to market release, <strong><em>it’s an industry built on predicting the future.</em></strong> Herein lies the unique challenge and opportunity: to interpret the present, informed by data analytics, while always casting an eye toward tomorrow’s horizons. Data analytics, in this landscape, serves as a bridge, seamlessly merging real-time insights with strategic vision.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/884/1*Fha28NkpBV_QD01OGH0xbw.png" /><figcaption><a href="https://fashiondlab.com/fashion/">https://fashiondlab.com/fashion/</a></figcaption></figure><h3>Search Volume: A Digital Lens into Fashion’s Tomorrow</h3><blockquote>Search volume in the digital age provides broad insights into global fashion trends, complementing the immediacy of traditional runways and informing strategic decisions.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2ADfmDes_2LdbAP66qQFdg.png" /><figcaption><a href="https://www.bbc.com/culture/article/20180301-are-these-the-weirdest-fashion-trends-ever">https://www.bbc.com/culture/article/20180301-are-these-the-weirdest-fashion-trends-ever</a></figcaption></figure><p>Paris and Milan’s Fashion Weeks serve as vibrant stages for the world’s freshest fashion trends. Yet, navigating through an array of eclectic and sometimes avant-garde styles to pinpoint the next significant trend can be intricate. <strong><em>Enter the digital age, where technology equips us with a new lens: search volume.</em></strong></p><p>Harnessing search patterns allows us to peek into global fashion inclinations, offering an avenue that’s both broad and insightful. It might not rival the immediacy of the runway, but it presents valuable, timely intelligence, crucial for shaping strategic fashion business moves. Through this digital lens, we gain access to wide-reaching trends, informed by the collective, rather than a selective few.</p><h3>Effective Search Volume Strategies</h3><blockquote>Using search volume effectively in fashion requires timing promotions with peak interest, monitoring subtle emerging trends, and integrating external data for richer insights and strategic actions.</blockquote><p>Fashion operates in seasonal cycles. Searches for shorts naturally peak in summer and down jackets trend in winter. However, these clear-cut patterns aren’t always enlightening for industry insiders. This prompts the question: How can search volume data offer deeper, actionable insights into upcoming fashion trends?</p><h4>Capitalizing on Peak Interest</h4><p>Predictable annual spikes in search volumes, such as those for seasonal items, might seem mundane, but they hide rich marketing opportunities. For fashion marketers, these surges signal the perfect time to unveil promotional events. Consider the late October to November window — essential for winter fashion campaigns. Start your promotions too soon, and consumers, still feeling autumn’s warmth, may ignore them. Delay and rivals might swoop in on your target audience. The solution? <strong><em>Plan early, but launch when search trends climax, maximizing engagement.</em></strong></p><h4>Spotlight on the Underdogs</h4><p>While dominant search trends have their merits, they might not pique the interest of visionary professionals like designers and merchandisers. These mavens often forecast trends up to two years in advance, sculpting the market landscape. Their focus? The subtle cues: nascent, yet promising trends. Every major fashion movement, transient or long-lasting, begins inconspicuously. Thus, rather than chasing the obvious, the real strategy lies in unearthing these burgeoning patterns. <strong><em>Monitor keywords on the rise or those just making an appearance; they may well hint at the next big wave.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xq5fjiGOMi2GklKw.png" /><figcaption><a href="https://www.semanticscholar.org/paper/Consumer-Behavior-Characteristics-in-Fast-Fashion-Wang/1deefe1dac0c07263a7072e35a1dd34619075f12/figure/1">https://www.semanticscholar.org/paper/Consumer-Behavior-Characteristics-in-Fast-Fashion-Wang/1deefe1dac0c07263a7072e35a1dd34619075f12/figure/1</a></figcaption></figure><h4>Integrating External Insights</h4><p>Marrying search volume analysis with external data can be intricate but yields richer insights. Envision meshing demographics, like gender and age, with search trends. Such a blend can reveal unexpected findings. Case in point, as a business analyst, I once unearthed an odd search pattern among women in their 20s: a surge in searches for men’s wallets. The twist? It was driven by a quest for fatherly gifts. Recognizing this, I spearheaded a gift-themed campaign, reaping significant success. The potential is vast; intertwining data from regions, weather conditions, and search volumes, for example, can offer strategies to smartly rotate stock and trim inventory excesses.</p><h3>Conclusion</h3><blockquote>In fashion, blending intuitive style with data analytics, especially search trends, is key to both immediate decision-making and anticipating future shifts in the digital age.</blockquote><p>In the fiercely competitive world of fashion, intuitive style isn’t enough; strategic foresight, grounded in the digital age’s insights, is vital. The merger of fashion expertise with data analytics, particularly trend analysis via search volume, has emerged as our guiding star. This isn’t about numbers alone — it’s about a profound comprehension of fashion’s ever-evolving landscape.</p><p>The interplay between the present and the future is our new focal point. Real-time search trends inform our immediate moves, such as targeted marketing. Yet, the deeper magic lies in recognizing nascent patterns and integrating diverse datasets — from demographics to climates. This rich, future-oriented approach doesn’t just keep us in the game — it sets us ahead, promising a vanguard position for those ready to harness both tradition and innovation in fashion’s digital era.</p><p><em>In preparing my article, I consulted ChatGPT, a large language model developed by OpenAI (OpenAI. (2023). ChatGPT [Large language model]. </em><a href="https://chat.openai.com/"><em>https://chat.openai.com</em></a><em>), for additional insights and information.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=450764452cf5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[South Korean Fashion Market Overview with Time Series Analysis: Part2]]></title>
            <link>https://medium.com/@msong507/dispelling-the-10-growth-plan-part2-e56350e0384b?source=rss-f052cdb2ea66------2</link>
            <guid isPermaLink="false">https://medium.com/p/e56350e0384b</guid>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[fashion]]></category>
            <category><![CDATA[south-korea]]></category>
            <category><![CDATA[online-shopping]]></category>
            <dc:creator><![CDATA[misun_song]]></dc:creator>
            <pubDate>Thu, 03 Aug 2023 16:58:28 GMT</pubDate>
            <atom:updated>2023-09-14T19:26:39.658Z</atom:updated>
            <content:encoded><![CDATA[<h4>Understanding the Ascending Fashion Market with a 2023 Forecast</h4><p><a href="https://github.com/msong507/fashion_timeseries">GitHub - msong507/fashion_timeseries: 2023 Forecast: South Korean Online Fashion Sales Volume - A Time Series Analysis</a></p><h3>Embrace the Hard Truth</h3><p>As we draw close to the year’s end, nestled among the Christmas celebrations, we encounter an important task: to predict next year’s sales. The number 10% often appears like an ideal target — neither too small nor too large. However, the time has come to say goodbye to the myth of an annual 10% growth. It’s time to face the facts.</p><blockquote>Note: This article primarily provides an in-depth discussion and implementation of time series analysis. For a foundational understanding of time series analysis and an exploratory data analysis (EDA) of the chosen dataset, please refer to our next article linked below.</blockquote><p><a href="https://medium.com/@msong507/dispelling-the-10-growth-plan-part1-114a68689cc3">South Korean Fashion Market Overview with Time Series Analysis: Part1</a></p><h3>Time Series Analysis: An Overview</h3><h3>Why Do We Need It?</h3><p>There’s no shortage of statistical models for predicting next year’s sales, but they come with strings attached. They need our data to fulfill certain assumptions, such as <strong><em>independent observations, constant variance, and no correlation between error terms and independent variables.</em></strong> In reality, our data often break these rules, which is where Time Series Analysis can step in.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/0*w-1IZKHddOCjBtjq.gif" /></figure><p><a href="https://condor.depaul.edu/sjost/it223/documents/regress.htm">https://condor.depaul.edu/sjost/it223/documents/regress.htm</a></p><ol><li><strong><em>Violation of Independence</em></strong>: Traditional linear regression requires each data point to be independent, disconnected from those before it. But time series data often show <strong><em>autocorrelation</em></strong>, where observations are linked to past ones.</li><li><strong><em>Inconstant Variance</em></strong>: Linear regression expects the error’s variance to remain the same across all independent variables, known as ‘Homoscedasticity.’ However, in time series data, this variance may change over time, leading to what’s known as <strong><em>heteroscedasticity</em></strong>.</li><li><strong><em>Autocorrelation</em></strong>: In linear regression, the error terms shouldn’t correlate with independent variables. But with time series, past values of the dependent variable are often used as independent variables, which can lead to a <strong><em>correlation</em></strong>.</li></ol><h3>What Do We Need for It?</h3><p>While Time Series Analysis is free from these traditional assumptions, it still requires something — <strong><em>stationarity</em></strong>. This means the mean, variance, and covariance of our data need to stay constant over time. Although no real-world data perfectly achieve this, we can <strong><em>modify</em></strong> our data to approximate it using various techniques. This allows us to make more reliable future predictions based on past data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pTO03q-X3wjUliv_.png" /><figcaption><a href="https://blogs.oregonstate.edu/gemmlab/2022/02/28/wavelet-analysis-to-describe-biological-cycles-and-signals-of-non-stationarity/">https://blogs.oregonstate.edu/gemmlab/2022/02/28/wavelet-analysis-to-describe-biological-cycles-and-signals-of-non-stationarity/</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*N3QPi0CVYzyqdFzf.png" /><figcaption><a href="https://otexts.com/fpp3/stationarity.html">https://otexts.com/fpp3/stationarity.html</a></figcaption></figure><ul><li>Most of our data sets, namely (a), (c), (d), (e), (f), (h), and (i), exhibit non-stationarity due to the influence of trends, changing levels, and clear seasonality. These factors contribute to the disruption of stationarity in these data sets.</li><li>Conversely, the data set (b) demonstrates stationarity as it remains largely unaffected by trends, changing levels, or seasonality.</li><li>Although data set (g) displays aperiodic cycles that could create variability over time, it retains its stationarity in the long term because the timing of these cycles is unpredictable.</li></ul><h3>Our Approach</h3><p>While this article doesn’t aim to dive into the intricate details of time series analysis, its main goal is to demonstrate how this tool can be pragmatically applied to real-world data. Used judiciously, time series analysis can empower us to make informed decisions by accurately predicting future trends.</p><ul><li><strong><em>In Part 1 (previous article)</em></strong>, we will explore ‘Domestic Online Shopping Transactions by Channel and Product Category’ data. This exploration will provide a basis for our time series analysis through an extensive exploratory data analysis (EDA) from various angles.</li><li><strong><em>In Part 2 (this article)</em></strong>, leveraging our understanding of the dataset, we will employ multiple time series models to forecast the online fashion transaction volume for 2023 in South Korea.</li></ul><h3>Summary: The South Korean Online Fashion Industry</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/568/1*jKwhNZDHzgOvkdjNiP8ENQ.png" /></figure><h4><strong><em>Preliminary Note</em></strong></h4><ul><li>Our analysis utilizes the ‘Domestic Online Shopping Transaction by Channel and Product Category’ data set by KOSIS (Korean Statistical Information Service).</li><li>From the broader data set, we exclusively focus on the fashion category, which includes four sectors: clothing, bags, accessories, and shoes.</li><li>The data primarily presents transaction amounts. Making direct profitability inferences could be challenging.</li><li>The data was collated monthly, spanning from January 2017 to the present date.</li><li>All figures, unless otherwise specified, are denoted in the unit of 100 million KRW.</li></ul><h4><strong><em>Market Growth</em></strong></h4><p>From 2017 to 2022, the industry saw substantial growth, with transactions rising from 17.542 trillion KRW (approximately 13.496 billion USD) to 29.661 trillion KRW (approximately 22.815 billion USD).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/1*sOORcxQGyND5H8oC1fZj4Q.png" /></figure><h4><strong><em>Sector Dominance</em></strong></h4><p>Through the years, the ‘Clothing’ category consistently led the market, generating 19.297 trillion KRW (approximately $14.844 billion USD) in transactions in 2022, followed by ‘Shoes’, ‘Accessories’, and ‘Bags’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/653/1*DeV16aKj7UBz0Coa4cHNKA.png" /></figure><h4><strong><em>Shopping Platforms</em></strong></h4><p>In 2022, mobile transactions represented 75% of all fashion purchases, amounting to 22.386 trillion KRW (approximately 17.22 billion USD). Among these, the ‘Bags’ category had the highest mobile purchase rate at 74%, higher than other categories.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/1*J4wBcM6rG0C_WHONOc1iRw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/1*DqVDc7rlu_2c1NKVDMFQkg.png" /></figure><h4><strong><em>Seasonality</em></strong></h4><p>Transaction volumes show clear seasonal patterns, peaking in March and November, with a dip during the summer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KvRNG6WBuuSNprjFoNsZZw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qw8cX4OXI-kPqQq5TNH3tA.png" /></figure><h4><strong><em>Demographic Shifts</em></strong></h4><p>Increased online shopping activity among the 40s and 50s demographic is evident, alongside the growing luxury market engagement of younger consumers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3FBIMjM3gzdJ_HWkruj6ug.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XytBTtgu8O6bCpGqUoi6nw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HG1J9HoyjdwxbzsBmrTZiQ.png" /></figure><h3>Time Series Analysis Challenges</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AdT0-9AcOZlJjLXf9MsyMw.png" /></figure><p>Forecasting future trends through data analysis relies on discerning patterns within historical data. The accuracy and dependability of these predictions, however, rest on meeting specific criteria like <strong><em>independence, uniform variability, and stationarity</em></strong>. Traditional predictive models, such as linear regression, often encounter limitations when applied to time series data due to factors like <strong><em>autocorrelation, non-stationarity, seasonality, and trends</em></strong>. Real-world data frequently contradicts these assumptions, with fashion sales serving as a key example. This article explores the unique time series factors that shape fashion sales and proposes strategies to mitigate these forecasting challenges.</p><h3>Time Series Factors Influencing Fashion Sales</h3><h4><strong><em>Seasonality</em></strong></h4><p>Fashion trends and demand vary with the seasons, showing distinct sales patterns related to spring, summer, fall, and winter collections.</p><h4><strong><em>Year-Round Events</em></strong></h4><p>Special days, holidays, and sales events like Black Friday and Boxing Day significantly impact fashion purchasing trends.</p><h4><strong><em>Trend Lifecycles</em></strong></h4><p>Fashion is highly sensitive to trends. As new trends emerge and old ones fade, buying patterns adapt accordingly.</p><h4><strong><em>Economic Factors</em></strong></h4><p>Broader economic conditions, such as economic growth or downturns, can influence consumer spending on fashion items.</p><h3>Time Series Prediction Models</h3><h4><strong><em>Seasonal Naïve</em></strong></h4><ol><li>Type: Simple, quick baseline model.</li><li>Assumption: Upcoming season mirrors the last.</li><li>Use Case: Works best with strong, stable seasonal patterns.</li><li>Simplicity: No need for parameter estimation, thus low computational cost.</li><li>Caveat: Struggles with data showing trends or non-seasonal elements.</li></ol><h4><strong><em>ETS (Error, Trend, Seasonality)</em></strong></h4><ol><li>Type: An enhanced multiple linear regression model.</li><li>Assumption: Time series comprises Error, Trend, and Seasonality.</li><li>Use Case: Ideal for data with trends and seasonal patterns.</li><li>Complexity: Requires parameter estimation (additive, multiplicative).</li><li>Caveat: Limited in capturing non-linear patterns.</li></ol><h4><strong><em>ARIMA / SARIMA</em></strong></h4><ol><li>Type: Handles both linear and non-linear patterns.</li><li>Assumption: Presumes data stationarity (applies differencing for non-stationary data).</li><li>Use Case: Suitable for trends, and seasonality, and assumes independent errors.</li><li>Complexity: Requires estimating several parameters; SARIMA also considers seasonal elements.</li><li>Caveat: Performance can vary greatly with different parameter settings.</li></ol><h4><strong><em>TBATS</em></strong></h4><ol><li>Type: Similar to ARIMA but manages multiple seasonalities.</li><li>Assumption: Potential for multiple seasonality patterns.</li><li>Use Case: Ideal for time series data with more than one seasonal pattern.</li><li>Complexity: Estimates seasonal levels, periods, and trends simultaneously.</li><li>Caveat: Complexity may result in longer computation times.</li></ol><h3>Step 1: Prepare Time Series Data</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bULRW6cLv-98RTgEMcTSOQ.png" /></figure><ol><li>Data Name: Domestic Online Shopping Transaction by Channel and</li><li>Product Category</li><li>Data Source: KOSIS (Korean Statistical Information Service)</li><li>Survey Period: January 2017 — December 2022 (collected monthly)</li><li>Sample Size: 72 months</li><li>Data Split: Train Dataset from January 2017 to December 2021 (60 months), Test Dataset from January 2022 to December 2022 (12 months)</li><li>Channels: Online (PC), Mobile</li><li>Product Categories: Clothing, Bags, Accessories, and Shoes</li><li>Unit: 100 million KRW</li></ol><h3>Step 2: Check Data Stationarity</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yxNV70oXO5dZN3DeBUqf9A.png" /></figure><h4>Visual Inspection</h4><p>Through a relatively simple visual inspection, it is apparent that the given time series data does not meet the stationarity criteria.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F3m9UnPOszw2Qnz6QzAvCw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KF-eWwKv6xhbAtuqxa6UFQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Prt0j0qAHxXgmhe5iiySNg.png" /></figure><h4>Check Data Stationarity: KPSS and ADF Tests</h4><p>The time series data, though passing the Augmented Dickey-Fuller (ADF) test, still leaves a shadow of doubt regarding non-stationarity, especially considering it failed the Kwiatkowski-Phillips-Schmidt-Shin (KPSS) test.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LBrTRP5-9vXErxYHbUQlaA.png" /></figure><h4>Check Data Stationarity: Seasonality</h4><p>The periodogram and seasonality plot reveal periodic fluctuations occurring at six-month intervals. This discernable pattern is a strong indicator of non-stationarity within our time series data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HbLxCyLX8gWDGH5U1K8Sig.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mg2RLdQSC03qLUAMCSYo2Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5eye2AsNdBddNB562xSm7g.png" /></figure><h4>Achieve Data Stationarity</h4><ol><li>Differencing: Calculate the difference between each data point and the one before it to remove certain patterns or trends in the data.</li><li>First-order differencing: Subtract the previous point from each data point to remove overall trends in the data.</li><li>Seasonal differencing: Subtract the data point from an earlier season from each current data point to remove seasonal patterns. (e.g. Subtract data from 12 months ago to remove the yearly pattern)</li></ol><p>First-order and seasonal differencing, individually or combined, can help remove trends and autocorrelation from data. It&#39;s crucial to account for any trends or seasonal patterns when crafting models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SOVErZUUkOhwP0M_MEIHDw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BfO9JRm42J_VGe9mih0sMA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Nc2E9jC1SPP8ZNRr7dbdeQ.png" /></figure><h3>Step 3: Model Design and Evaluation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sPw7R48kt9IBsN912mb3Tw.png" /></figure><h4>Model 1. Seasonal Naïve</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UlOX6Vj1aVbuosyhTOLsXQ.png" /></figure><ol><li>Training data: 01/2017–12/2021 (60 months)</li><li>Test data: 01/2022–12/2022 (12 months)</li><li>Ljung-Box test p-value: 8.129e-06 (&lt;0.05)</li><li>RMSE (Training): 24.04 billion KRW (18.49 million USD)</li><li>RMSE (Test): 29.03 billion KRW (22.33 million USD)</li></ol><h4>Model Evaluation 2. ETS (Error, Trend, Seasonality)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V8nJ5OexfLbuInbfQXYaTA.png" /></figure><ol><li>Training data: 01/2017–12/2021 (60 months)</li><li>Test data: 01/2022–12/2022 (12 months)</li><li>Ljung-Box test p-value: 0.345</li><li>AICc: 825.18</li><li>RMSE (Training): 8.47 billion KRW (6.52 million USD)</li><li>RMSE (Test): 10.10 billion KRW (7.77 million USD)</li></ol><h4>Model 3. ARIMA (Auto Regressive + Moving Average)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V8nJ5OexfLbuInbfQXYaTA.png" /></figure><ol><li>Training data: 01/2017–12/2021 (60 months)</li><li>Test data: 01/2022–12/2022 (12 months)</li><li>Ljung-Box test p-value: 0.345</li><li>AICc: 825.18</li><li>RMSE (Training): 10.23 billion KRW (7.87 million USD)</li><li>RMSE (Test): 16.38 billion KRW (12.6 million USD)</li></ol><h4>Model 4. TBATS</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ogMsxPz-Cd0OJoRcorbD5A.png" /></figure><ol><li>Training data: 01/2017–12/2021 (60 months)</li><li>Test data: 01/2022–12/2022 (12 months)</li><li>Ljung-Box test p-value: 0.843</li><li>RMSE (Training): 8.32 billion KRW (6.4 million USD)</li><li>RMSE (Test): 15.93 billion KRW (12.25 million USD)</li></ol><h4>Model Evaluation</h4><ol><li>Ljung-Box Test: Checks residual independence. A p-value under 0.05 signals non-independent residuals, suggesting some time series information may have been overlooked during modeling.</li><li>AICc: Measures model fit relative to complexity. A lower AICc indicates a better-fitting model but doesn’t assure test data performance.</li></ol><ul><li>Upon thorough consideration of performance evaluation factors for time series data, the ETS model was selected.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9yql1yLRv6qWXh9yIj5-JA.png" /></figure><h3>Step 4: Model Selection and Deployment</h3><h4>Choice of ETS(m,ad,m) Model</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oXUQmvYfXCG9qjbFNM_rDA.png" /></figure><ol><li><strong><em>ETS(m,ad,m) Model</em></strong></li></ol><ul><li><strong><em>Multiplicative Errors</em></strong>: Variance in observations increases over time.</li><li><strong><em>Additive Trend</em></strong>: Data exhibits a linear trend.</li><li><strong><em>Multiplicative Seasonality</em></strong>: The strength of seasonal patterns escalates in relation to the time series level.</li></ul><p>2. <strong><em>Trend and Seasonality</em></strong></p><ul><li>The ETS model is proficient when <strong><em>clear trend and seasonality</em></strong> elements are evident in the data.</li><li>The given time series data displays <strong><em>an ascending trend </em></strong>and <strong><em>biannual seasonality</em></strong>.</li></ul><p>3. <strong><em>Data Stability</em></strong></p><ul><li>The used time series data shows <strong><em>minimal shifts</em></strong> in statistical attributes like mean and variance over time.</li><li>Although we applied a Box-Cox Transformation with a lambda value of -0.512, which is more pronounced than a logarithmic transformation, its impact on the data was relatively minimal.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*loBCKUnHZie-mNSR8XHkIA.png" /></figure><h4>Model Selection and Deployment</h4><p>The estimated transaction volume for online fashion in Korea in 2023, predicted through the ETS model, is approximately 31.7 trillion KRW (approximately 24.38 billion USD). This suggests a slowdown in growth compared to previous years.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9eLJg3W1O_ShvzUeHz-0GA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1jZJ870WUL11QfAw5m2qYg.png" /></figure><h3>Conclusion</h3><p>A comfortable 10% growth figure has long been touted as the ideal, a target that feels attainable. However, our research into the Korean online fashion industry uncovers a sobering reality: continuous growth is a myth, and we’re witnessing a slowdown in the growth rate.</p><p>Our exploration began with the deployment of an ETS model to decode this slowdown. Highly efficient at capturing clear trends and seasonality, the model helped us peer into 2023’s outlook. We foresee a slower growth rate with the online fashion transaction volume predicted to reach approximately <strong><em>31.7 trillion KRW (approximately 24.38 billion USD)</em></strong>.</p><p>While insightful, our study is not without limitations. Our reliance on a mere 60 months of train data indicates a clear scope for improvement. Future endeavors should incorporate larger historical datasets to fortify our forecasts’ accuracy and reliability.</p><p>In an increasingly interconnected world, we must also consider the impact of global events such as pandemics, or broader economic scenarios, on our industry. These factors can be woven into our models as <strong><em>exogenous variables</em></strong> using tools like an extended ARIMA (ARIMAX) model or a Vector Autoregressive (VAR) model, offering a more nuanced perspective of the industry.</p><p>In closing, while our current model offers useful insights, it underscores the need for ongoing refinement in our forecasting methods. This integrative approach will help us refine our forecasting precision, ensuring our growth predictions are firmly grounded in market realities.</p><p><em>In preparing my article, I consulted ChatGPT, a large language model developed by OpenAI (OpenAI. (2023). ChatGPT [Large language model]. </em><a href="https://chat.openai.com/"><em>https://chat.openai.com</em></a><em>), for additional insights and information.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e56350e0384b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>