<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mae Shahvirdi on Medium]]></title>
        <description><![CDATA[Stories by Mae Shahvirdi on Medium]]></description>
        <link>https://medium.com/@mae.shahvirdi?source=rss-bf48ae07f02f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*obpSneQLD-eUCCQxf4x-_w.png</url>
            <title>Stories by Mae Shahvirdi on Medium</title>
            <link>https://medium.com/@mae.shahvirdi?source=rss-bf48ae07f02f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 18:45:37 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mae.shahvirdi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Is This Course Even Real? My Journey Cleaning Udemy’s Dataset]]></title>
            <link>https://medium.com/@mae.shahvirdi/detecting-dirty-data-a-case-study-using-udemy-courses-6adac9806678?source=rss-bf48ae07f02f------2</link>
            <guid isPermaLink="false">https://medium.com/p/6adac9806678</guid>
            <category><![CDATA[critical-thinking]]></category>
            <category><![CDATA[data-anaytics]]></category>
            <category><![CDATA[data-cleaning]]></category>
            <category><![CDATA[free-udemy-course]]></category>
            <category><![CDATA[outlier-detection]]></category>
            <dc:creator><![CDATA[Mae Shahvirdi]]></dc:creator>
            <pubDate>Fri, 04 Apr 2025 14:33:55 GMT</pubDate>
            <atom:updated>2025-04-04T14:49:16.359Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qbupVhHoOPB-SO12I-0jvA.jpeg" /><figcaption>A sketch of a girl holding an eraser, saying “So, I should clean it here, right?!” in front of a messy “Outliers Lists” — a playful take on data cleaning.</figcaption></figure><p>When I began a project on E-learning trends, my goal was simple: to understand what makes online courses successful. I wanted insights into pricing strategies, course structure, and what “popular” actually looks like on platforms like Udemy, Coursera, and edX.</p><p>To start, I picked Udemy’s public dataset. It had a nice structure, a manageable size, and most importantly — it gave me a realistic playground to practice cleaning and exploring real-world data.</p><h3>Starting with the Basics</h3><p>Like most data projects, I began by cleaning the obvious stuff — removing duplicates, checking for nulls, and doing a basic .describe() to get a feel for the dataset. That’s when I noticed something odd: in many columns, like num_subscribers and num_reviews, the <strong>mean was much higher than the median</strong>, and the <strong>standard deviation was off the charts</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/722/1*h82W-yyLc3dlBfaIM4UO9w.png" /><figcaption>A summary table showing descriptive statistics for Udemy courses. It highlights strong outliers, with num_subscribers having a mean of 3,190 but a median of 912, and num_reviews showing a similar skew. num_lectures and content_duration also show wide ranges, confirming data skew and the presence of outliers.</figcaption></figure><p>Clearly, I had a bunch of outliers.</p><p>But I paused. My main objective wasn’t to study course popularity (yet) — it was to focus on <strong>valid courses only</strong>, and make sure that <strong>what I was analyzing actually existed and made sense</strong>.</p><h3>Step One: Content Duration vs Number of Lectures</h3><p>I turned my attention to the more structural elements of a course: num_lectures and content_duration. These two go hand in hand—if a course has <strong>0 lectures</strong> or <strong>0 total duration</strong>, it’s a dead entry. Gone.</p><p>So I visualized the distribution of lectures (box plot )</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/523/1*6lLKxZq2kvKvy3bnI5fFkA.png" /><figcaption>A box plot visualizing the distribution of num_lectures in Udemy courses. Most values cluster below 100, with numerous outliers stretching beyond 100 and one extreme outlier near 779 lectures.</figcaption></figure><p>and calculated the <strong>upper bound using IQR</strong>. Outliers made up <strong>7.9%</strong> of my data — well above the commonly accepted 5% threshold. That was my signal to investigate further, rather than blindly cutting them out.</p><p>Sure enough, I found some oddities. One course had <strong>779 lectures</strong>, while the next highest had 544. That’s a huge gap. I decided to drop that one extreme outlier, along with any courses that had 0 lectures or 0 duration. Those were clearly broken entries.</p><h3>The main Question: What makes a Course Suspicious?</h3><p>Some courses had 300+ lectures but only 5 hours of content. That felt… off. So I asked myself, <strong>what’s the normal length of a lecture</strong>?</p><p>Rather than going with gut instinct, I checked <strong>E-learning platform guidelines</strong>. Udemy encourages instructors to keep lectures short — <strong>2 to 10 minutes</strong>, to retain student engagement. Coursera and others say the same: <strong>shorter videos = better attention span</strong>.</p><p>I then checked my own dataset:</p><ul><li><strong>Median lecture length</strong> = ~5.6 minutes</li><li><strong>75th percentile</strong> = ~7.8 minutes</li></ul><p>So I set my logic: if a course had <strong>more than 100 lectures</strong> and an average lecture time of <strong>less than 5 minutes</strong>, and was within the outliers previously detected, it deserved scrutiny.</p><p>Using that threshold, I flagged <strong>167 suspicious courses</strong> (4.5% of the dataset) and removed them. They likely represented bloated, broken, or filler-heavy entries.</p><h3>Now the Dataset is Clean — Mostly</h3><p>At this point, I felt confident in my dataset’s integrity. I had removed entries that clearly violated basic logic.</p><p>But what about courses with <strong>200,000+ subscribers</strong> or <strong>15,000 reviews</strong>?</p><p>Here’s the thing: those are outliers too — but I <strong>chose to keep them</strong>.</p><h3>Why I Didn’t Delete Popular or Unpopular Courses</h3><p>Just because a course has strange-looking numbers doesn’t mean it’s invalid. Consider these possibilities:</p><ul><li><strong>Heavy promotions</strong> or discounts driving large enrollments</li><li><strong>Viral popularity</strong> from social media, influencers, or YouTube</li><li><strong>Corporate training programs</strong> enrolling in bulk</li><li><strong>Older courses</strong> naturally accumulating more students</li><li><strong>Niche topics</strong> that are high-quality but less popular</li></ul><p>Removing these entries could mean deleting <strong>legitimate success stories</strong> or <strong>authentic niche content</strong>. While they skew the averages, they still reflect <strong>real platform behavior</strong>.</p><h3>Being Honest: I Did Consider It</h3><p>Yes, I considered trimming extreme values in num_subscribers and num_reviews. But unless I had stronger evidence of fraud—like a course with <strong>200,000 subscribers and 2 reviews</strong>, or <strong>no lectures but thousands of ratings</strong>—I couldn’t justify removing them. That would be assumption-driven, not data-driven.</p><p>That said, if I later move into <strong>predictive modeling</strong>, I’ll revisit how to handle these outliers. But for now, in the realm of <strong>exploratory data analysis</strong>, they’re staying.</p><h3>What’s Next?</h3><p>Now that I’ve narrowed down to a clean, valid subset of courses, I can begin exploring pricing patterns, subscriber trends, and subject-based popularity — with confidence in my data.</p><p>Let the fun part begin!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6adac9806678" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Analysis Creates More Questions Than Answers]]></title>
            <link>https://medium.com/@mae.shahvirdi/why-analysis-creates-more-questions-than-answers-244278260e6d?source=rss-bf48ae07f02f------2</link>
            <guid isPermaLink="false">https://medium.com/p/244278260e6d</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Mae Shahvirdi]]></dc:creator>
            <pubDate>Mon, 03 Mar 2025 14:02:27 GMT</pubDate>
            <atom:updated>2025-03-03T14:02:27.034Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kg8QePNo8ILzpGsXIxfTfA.jpeg" /></figure><p>One of my dear friends asked me about the challenges of analyzing a marketing project, and it got me thinking.</p><p>One major challenge is knowing when to stop looking for trends and patterns. Assuming the dataset is clean and correct, we search for patterns that make sense. If they don’t, there must be an underlying reason behind the numbers. The real question is: where should we look, and how should we approach the analysis? Often, the key is not just in the data itself but in understanding the external factors that may not be explicitly present in the dataset. Recognizing when to expand our view beyond the immediate numbers is crucial in making sense of seemingly unexplainable trends.</p><p>Another challenge is the lack of domain understanding. To properly analyze a dataset, we need general domain knowledge to interpret results with common sense. Without it, data can easily be misinterpreted or lead to conclusions that do not reflect reality. For instance, sales trends in different locations cannot be fully understood without knowledge of local market dynamics, consumer behavior, and economic conditions. Even with a comprehensive dataset, an analyst must step into the context of the problem and ask the right questions to uncover hidden insights.</p><p>Imagine two cities in the same country with similar cultures and populations. The marketing team has implemented the same strategy in both places, yet the difference in sales is significant and cannot be ignored. Our dataset does not contain any variable that can be identified as an indicator or catalyst. In such situations, we need to take a broader perspective and see the bigger picture because this significant difference doesn’t make common sense at first glance. This is where real analytical thinking comes in — beyond just working with numbers, we need to consider economic conditions, customer sentiment, and even potential regional biases that could influence purchasing behavior.</p><p>To better understand this gap, we need to ask the right questions:</p><ul><li>Do these two cities have similar purchasing power?</li><li>Is the cost of living significantly different?</li><li>Are there local competitors in one of the cities?</li><li>How is the brand perceived in each location?</li><li>Does one city have better distribution logistics than the other?</li><li>Where are the stores located, and is product availability the same in both cities?</li></ul><p>Asking these questions leads to deeper insights. Often, the difference in sales between two locations is not due to a single variable but rather a combination of multiple subtle influences that, when put together, create a significant impact. Recognizing these nuances is what sets apart an average analysis from an insightful one.</p><p>This is why I love analysis. Coming from a UX design background, I have learned that “I don’t know everything, and research can reveal facts that are completely different from what I initially assumed.” Thinking from the perspective of people interacting with the product or service is crucial. It’s essential to always question, look for missing variables, and identify factors that are overlooked as insignificant. This mindset of curiosity and openness allows analysts to refine their findings continually and make informed decisions based on both qualitative and quantitative data.</p><p>Breaking down complex topics into their fundamental components is where true understanding begins. Whether it’s marketing, science, or philosophy, analysis serves as a tool for explanation and insight. However, it is not always about finding a direct answer — often, analysis uncovers more layers of complexity, leading to new questions and areas of exploration.</p><p>Even if we analyze every element individually, we will uncover deeper interconnections, uncertainties, and sometimes interdependencies. Recognizing these relationships can shift the focus of the analysis and lead to discoveries that were not initially apparent. The process of analysis is not a straight path but rather a cycle of continuous refinement and questioning.</p><p>Our interpretations are subjective — especially in fields like philosophy, literature, and social sciences. New variables constantly emerge, and subjects often turn out to be far more complex than they initially appear. As a result, new questions arise despite our original goal of finding answers. This is not a flaw in analysis but rather its most valuable trait: the ability to evolve, adapt, and challenge assumptions.</p><p>This continuous cycle of questioning pushes analysts to refine their arguments, challenge opposing viewpoints, and remain as objective as possible. Being willing to accept new perspectives and adjust conclusions based on new information is what makes an analyst effective. The ability to ask “why?” repeatedly is what leads to breakthroughs and genuine understanding.</p><p>In marketing, especially for global products or services, defining the success factors correctly is critical. Each region has unique characteristics, and applying a one-size-fits-all goal could lead to misinterpreting the marketing team’s efforts. Sometimes achieving moderate sales in one region is a greater challenge than achieving high sales in another. Understanding these regional nuances ensures that performance is measured fairly and that strategies are adapted to suit the needs of different markets.</p><p>Now that I’ve written my thoughts, another question comes to mind: how does analysis deal with localization? Do we even consider it in our variables? Is there a way to normalize it? If so, what would that normalization process look like across different cultural and economic landscapes?</p><p>Well, maybe that’s a question for another time — but it’s certainly one worth exploring :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=244278260e6d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What do you need to know about database types as a Junior Data Analyst?]]></title>
            <link>https://medium.com/@mae.shahvirdi/what-do-you-need-to-know-about-database-types-as-a-junior-data-analyst-53d55b983797?source=rss-bf48ae07f02f------2</link>
            <guid isPermaLink="false">https://medium.com/p/53d55b983797</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[structured-data]]></category>
            <category><![CDATA[semi-structured-data]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[unstructured-data]]></category>
            <dc:creator><![CDATA[Mae Shahvirdi]]></dc:creator>
            <pubDate>Wed, 04 Sep 2024 12:51:56 GMT</pubDate>
            <atom:updated>2024-09-04T12:51:56.417Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lFsNVyWAjYEEiIM_6-JOBA.jpeg" /><figcaption>An image of different databases</figcaption></figure><p>In my mind, I have made databases seem much more difficult to understand than they actually are, so let’s break them down and see that they are not as intimidating as they may appear.</p><p>A database is a collection of stored data, and there are several ways to categorize this data based on how it is stored and organized: Structured, Semi-Structured, and Unstructured data.</p><p>Structured Data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DnroRZmNs6TrovgxavvzuA.jpeg" /><figcaption>Structured database</figcaption></figure><p>The most common approach to storing data in a database is to use columns and rows in a tabular format. The intersection of each column and row is called a cell, and every cell stores a data element, which can be numeric, text, etc.</p><p>Sounds familiar to an Excel spreadsheet? well, there are some key differences. An Excel spreadsheet holds all the data in one place. In contrast, a database allows you to retrieve specific records using queries and supports multiple users executing these queries simultaneously on the same database.</p><p>To manipulate structured data Relational Database Management Systems (RDBMS) are designed. (though they can also accommodate some aspects of semi-structured data, however with certain limitations.)</p><p>Examples of RDBMS include MySQL, PostgreSQL, Oracle Database, and Microsoft SQL Server. The standard query language for relational databases is SQL (Structured Query Language). Still, programming languages like Python and R can also be used to interact with and manipulate data in relational databases.</p><p>Semi-structured Data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fCCEKSc4mkfq-HitwXpSFA.jpeg" /><figcaption>Semi-structured database</figcaption></figure><p>Semi-structured data falls between structured and unstructured data. It has some organization but is more flexible than fully structured data. Unlike structured data, which fits neatly into rows and columns, semi-structured data does not conform to a rigid form. However, it contains tags or markers that give some level of organization to it and make it easier to analyze compared to unstructured data.</p><p>For example, HTML is a type of semi-structured data. If you understand the HTML tags and their meanings, you can interpret the data presented through those tags. Similarly, formats like JSON and XML are also considered semi-structured because they have hierarchical or tagged data that allows for flexible data representation and retrieval.</p><p>Unstructured Data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kCIj6G3nISnlJk9vYlE5Tg.jpeg" /><figcaption>Unstructured database</figcaption></figure><p>An unstructured database is not stored in a tabular format and does not have a predefined form or structure. It includes various types of data that do not fit neatly into rows and columns, such as emails, text messages, images, and audio files. Unlike structured data, which is highly organized and easily searchable, unstructured data lacks a specific form and requires more complex methods for processing and analysis. To handle it, we use various techniques: text processing helps analyze written content, while machine learning can find patterns and classify data or specific tools for image and video analysis recognize objects, and extract text from visuals.</p><p>I hope now you have a clear understanding of the initial differences between these data types, there are more to come :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=53d55b983797" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My Experience in Learning with AI]]></title>
            <link>https://medium.com/@mae.shahvirdi/my-experience-in-learning-with-ai-7e14fc15f09b?source=rss-bf48ae07f02f------2</link>
            <guid isPermaLink="false">https://medium.com/p/7e14fc15f09b</guid>
            <category><![CDATA[learning-data]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Mae Shahvirdi]]></dc:creator>
            <pubDate>Thu, 27 Jun 2024 13:00:25 GMT</pubDate>
            <atom:updated>2024-06-27T13:00:25.844Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="A person telling AI that: you are gonna help me, Chat GPT Answers: Yes! I am there 24/7, no coffee break! well, I lack common sense!" src="https://cdn-images-1.medium.com/max/1024/1*uNCC-3fWAc2ht3X99YR55g.jpeg" /></figure><p>Six months ago, I began learning about data analysis through the CareerFoundry program. Coming from a UX background, having studied mathematics in high school, and taken some online courses in quantitative research, many concepts were new to me. Instead of googling every topic, I started asking questions from ChatGPT, which proved to be very helpful. If you are learning something new and eager to understand it rather than just memorizing it, here are some tips:</p><p><strong>Ask ChatGPT to Explain Like You’re a 5-Year-Old:</strong></p><p>It did a good job at it, giving easily understandable examples, and simplifying it. From there you can ask other leading questions to get familiar with the more complex sides of the concepts, so start simple, then ask more complex questions as you go.</p><p><strong>Be Careful with Your Prompts:</strong></p><ul><li>Talk in a clear way, and be concise, if you are referring to a variable.</li><li>choose a proper name for it, you don’t want to see an answer like: ‘variable a’ has a positive correlation with ‘rental_prices’</li><li>If a variable has changed during the process and a new value is embedded there, there is a chance that it uses the first value instead of the updated one, always check</li></ul><p><strong>Use Correct Terminologies:</strong></p><p>Chat GPT does a great job of understanding your question, but if you are not sure about a specific terminology, make sure before adding it to your prompt. I once couldn’t remember the terminology and I just pressed some random keys on the keyboard (something like fjkhdgkdfh :D ), and it actually told me what I was searching for and typed the terminology for me! Yep! I was impressed! After that whenever I cannot remember the exact term I do this, but always check!</p><p><strong>It Doesn’t Have Feelings:</strong></p><p>Once I felt something was wrong despite checking outliers and data types over and over, and the results didn’t make sense! I wrote: <em>‘I feel sth is wrong there’</em>, and the answer was: <em>‘I don’t have feelings, emotions, or consciousness. I’m a program designed to process and generate human-like text based on the input I receive.’</em><br>Yep! Trust your gut feelings! I checked them all again, and there was a small discrepancy while combining 2 data sets, which was overlooked in the data cleaning part. It doesn’t have the same common sense that we do.</p><p><strong>It Makes Mistakes!</strong></p><p>Check, double-check, triple-check! Especially if you are new to data and your talk with chat GPT was long and you typed many prompts about the same subject in a raw.</p><p>These checks also help you to critically analyze the process and evaluate different options for handling the same task.</p><p><strong>In Comparison with Copilot:</strong></p><p>I used both, but I find chat GPT way better in understanding the prompt, it doesn’t need much effort and input to understand your prompt and to tell you the process, but CoPilot needs to have the prompt written in a more detailed and concise way.</p><p><strong>Ask for ‘Why’ in Syntax:</strong></p><p>If you are learning the syntax of SQL or Python, and you need a certain code for a certain task, it gives you the reason why it is generating that specific code, but better to ask it to include the meanings of the syntax, it is very helpful in understanding the logic in your code, and to get over the ‘coding is a monster with 7heads’ fright. For me, personally and shockingly the coding part was very pleasant!</p><p><strong>All in all:</strong></p><p>Use it! It speeds things up for you! But don’t let it think for you!</p><p><strong>Critical thinking, attention to detail, and understanding relationships between variables are all essential and human-powered (still!).</strong></p><p>Embrace these aspects and enjoy the learning journey!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7e14fc15f09b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>