<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by UCLA Data Science Union on Medium]]></title>
        <description><![CDATA[Stories by UCLA Data Science Union on Medium]]></description>
        <link>https://medium.com/@ucladsu?source=rss-e0043e83850a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*Vpt4ErzqndQgx9vg</url>
            <title>Stories by UCLA Data Science Union on Medium</title>
            <link>https://medium.com/@ucladsu?source=rss-e0043e83850a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 23:11:56 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ucladsu/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Social Intelligence: Emotion Prediction in Videos]]></title>
            <link>https://medium.com/@ucladsu/social-intelligence-emotion-prediction-in-videos-01f704a9bfbb?source=rss-e0043e83850a------2</link>
            <guid isPermaLink="false">https://medium.com/p/01f704a9bfbb</guid>
            <dc:creator><![CDATA[UCLA Data Science Union]]></dc:creator>
            <pubDate>Tue, 02 Jan 2024 09:29:06 GMT</pubDate>
            <atom:updated>2024-01-02T20:21:17.025Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Project Lead</strong>: <em>Yiling Yun </em><strong>&amp;</strong><em> </em><strong>Project Members</strong>: <em>Jonah Jung, Madelaine Leitman, Ved Phadke, Aaron Tae</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/965/1*sZhAJhw7T4UqH73Ml-Muxw.jpeg" /><figcaption>Source: <a href="https://www.unite.ai/what-is-emotion-ai-why-does-it-matter/">https://www.unite.ai/what-is-emotion-ai-why-does-it-matter/</a></figcaption></figure><h3>Introduction</h3><p>We are very good at connecting with others’ emotions. We feel the sorrow when Rose cries to watch Jack slowly sink into the ocean in <em>Titanic</em>, and we feel the joy when we see a toddler first time knows how to walk. This almost innate ability to recognize and share the emotions of other people defines us as social beings. While recent developments in machine learning have been very successful in identifying objects and generating texts, the exploration of predicting the social aspects of human capability has been somewhat overlooked. In this project, we created an interface to predict the emotions of every frame in a video using a series of deep-learning models designed and trained on a dataset called VEATIC. In addition, we evaluated the model from VEATIC by examining the prediction performance in a variety of videos.</p><h3>VEATIC</h3><h4>Dataset</h4><p>Inferring others’ emotions and reactions is integral to how humans communicate with one another, so a multitude of past studies have focused on the mechanisms of emotion recognition. However, such studies have focused on facial expression, which excludes a large aspect of emotional recognition– contextual evidence. The context of facial cues can drastically influence how emotion is perceived. For example, the look of excitement may be confused with shock or grief without considering the context (Figure 1).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/754/1*jUtIMSRL6jzG5isO8G1ZSA.png" /><figcaption>Figure 1. a) From the facial expression alone, you might think the woman is sad or in grief. b) With the context, you correctly see that she is happy or excited.</figcaption></figure><p>A group of researchers from various universities across the United States collaborated to introduce the Video-based Emotion and Affect Tracking in Context (VEATIC) Dataset (Ren et al., 2023). The VEATIC dataset consists of 124 video clips that can track emotions using both facial features and context to benefit research groups in Psychology or Computer Vision. The videos consist of 104 clips from Hollywood movies, 5 clips from documentaries, and 15 clips from home videos, and their lengths range from 10 seconds to 2 minutes and 37 seconds. The videos were gathered from YouTube and were specifically selected for the emotion in the clips to vary throughout the video. There are zero to multiple interacting characters in the videos, and all sound was removed.</p><p>Each video was rated based on two dimensions: emotional valence and emotional arousal. These two dimensions have been widely used in Psychology and Neuroscience literature to describe a wide range of emotions (Figure 2): valence describes the emotion from negative to positive where happy and depressed will be on the two ends; arousal describes the intensity of the emotion where clam and angry will be on the two ends. To create the VEATIC dataset, 192 human annotators were recruited. Each of them rated all video clips throughout numerous sessions, where they rated the valence and arousal of the pre-specified character throughout each video.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/623/0*cLovsyzIPU5ELxT2" /><figcaption>Figure 2. We can describe any emotion based on two dimensions: emotional valence and emotional arousal.</figcaption></figure><h4>Framework</h4><p>To test the validity of the VEATIC dataset, the authors created a model and trained it with their dataset.The model architecture can be broken down into three main sections: a convolutional neural network (CNN) for feature extraction, a visual transformer (ViT) for processing temporal (time-series) data, and finally a multilayer perceptron (MLP) head to output the predicted valence and arousal of a given frame (Figure 3).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*p2FSi2XXkAJH4kk6" /><figcaption>Figure 3. Model framework to train the VEATIC dataset.</figcaption></figure><p>The CNN model structure and weights were adopted from a pre-trained module, Resnet50 (He et al., 2016), which is a CNN with 50 layers that is trained on more than 1 million images from the ImageNet database (Russakovsky et al., 2015). Unlike typical image-processing CNNs, the CNN trained on the VEATIC dataset used images that have not been extensively preprocessed as inputs. Thus, the training process was much more computationally heavy than conventional CNNs, but these image inputs retained more important contextual information in exchange. After each frame of a particular video is fed into the CNN, it outputted a vector of extracted features, which was then moved into the visual transformer.</p><p>These sets of vectors of extracted features were first position embedded and fed into the encoder of a visual transformer (ViT) (Dosovitskiy et al., 2020). Through a set of self-attention modules with layer normalization and multi-head dot-product attention in ViT, information from consecutive video frames were combined.Finally, the layers were normalized and fed into a multilayer perceptron (MLP) head that outputed the predicted valence and arousal value of the given frame. The model performance was compared by feeding in different datasets to train. The model trained on the VEATIC dataset performed the best compared to the same model architecture trained on other datasets.</p><h3>Demo</h3><h4>Training</h4><p>To optimize model performance, we tested different model parameters to see which performed the best in predicting the valence and arousal values of a given frame. We tested six different sets of parameters on different training processes: one trained and tested on the 98 videos after 20 epochs provided by the authors (Original), five trained on all 124 videos after 1, 2, 3, 4, and 5 epochs obtained by ourselves (New1, New2, New3, New4, New5). To compare the outputs given from these model parameters, we used each of the six sets of parameters to predict valence and arousal ratings from preselected videos from the dataset. We then compared these results with the actual valence and arousal ratings given by the human annotators, and used the Pearson correlation coefficient as our metric of prediction accuracy.</p><p>Due to the inherent computationally intensive nature of the training process, we decided to test out the parameters and just three videos each and perform further testing on just two best-performing parameters. These two parameters were Original and New1 by a fairly large margin, which we then tested on a different set of 10 videos. The correlations between the predictions given by these parameters and the actual human ratings are in Figure 4.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cir7Y6Xh-qh2MNLm" /><figcaption>Figure 4. Correlation between model predictions and human responses for Original and New1.</figcaption></figure><p>We can see that albeit a very small difference, the Original parameter demonstrates a higher correlation with the given human ratings. Thus, we proceeded with our next steps with the Original parameter.</p><h4>Illustration</h4><p>Using the decided set of parameters, we created an interface on Streamlit to easily predict emotional arousal and valence for each frame in a video. Screenshots of the interface are in Figure 5.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7p5j1nMocTEN9-p-g3KvwQ.png" /><figcaption>Figure 5. a) Select a video on your local site to upload. b) Select a frame you would like to predict emotions. c) Display predicted emotional valence and arousal throughout all frames in a video.</figcaption></figure><h3>Model Evaluation</h3><h4>Our Video Selection</h4><p>To further test the performance of the model that the VEATIC authors claimed to be the best, we compared its performance across various video inputs. Specifically, our group selected 14 videos. The videos were selected to test how the model responded to video inputs of a single character per frame vs. multiple characters per frame, animals vs. human subjects, cartoons vs. live action,, and landscape vs. portrait.</p><h4>Findings</h4><p><strong><em>Single character per frame vs. Multiple characters per frame</em></strong></p><p>a) Single character: Brad Pitt smiling at an award show</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1sSE-y2y_Lr8G37tpjezFA.png" /></figure><p>b) Single character: Girl giving a monologue showing little change in emotion</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EGkeaGMIzoR350CFtKQ7sA.png" /></figure><p>c) Multiple characters: Wizards of Waverly Place laughing</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pg_dXeH2jUuKrIeaJqtoMA.png" /><figcaption>Figure 6. Emotion predictions of new videos outside of the VEATIC dataset with a single character or multiple characters per frame.</figcaption></figure><p>Examining the results of the Wizards of Waverly Place clip provides a compelling analysis (Figure 6c). The arousal chart reveals two distinct spikes, coinciding with moments in the clip where a character delivers a joke, eliciting strong reactions from the others. The abrupt change in energy and intensity of the character expressions leads to a rapid increase in arousal ratings. Concurrently, the valence chart maintains a close proximity to zero but experiences a sharp dip at the same instances of heightened arousal. This indicates that the model successfully captured the intense emotions of multiple characters and interpreted the overall valence as negative or unpleasant. However, several considerations come into play. Despite the scene being intended for comedic relief in the actual show, the model may have focused predominantly on a character who was upset, potentially misinterpreting the mother’s laughter as a crying expression, among other nuances. In contrast, when analyzing clips featuring single characters, CNN’s ability to concentrate on a sole character enhances the accuracy of valence and arousal outputs. Despite the multiple-character video yielding impressive results, it is essential to acknowledge the limitation in testing due to the constrained number of available videos, each lacking ideal features. For instance, the Brad Pitt clip (Figure 6a) displays minimal fluctuation in arousal and valence, forming a more linear trend. This suggests that the model effectively captured Brad Pitt’s gradual emotional changes throughout his speech. In summary, the study’s findings underscore the significance of scene complexity and the number of characters in influencing the accuracy of emotion capture. While the model excelled in certain scenarios, the limitations in available testing data highlight the need for a more diverse and extensive dataset to refine the model’s performance across a broader range of contexts.</p><p><strong><em>Animals &amp; animations</em></strong></p><p>a) Lions documentary</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mlxfO15yxGGI9p557KfRUg.png" /></figure><p>b) Polar bear cartoon Ad</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A31tjoysG5EZsgSAchTBEw.png" /></figure><p>c) Goose animation</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_AbOvTSb_B0qs_k1kjcItQ.png" /><figcaption>Figure 7. Emotion predictions of new videos outside of the VEATIC dataset with animals and animations.</figcaption></figure><p>Analyzing the charts for cartoons, specifically the goose (Figure 7c) and polar bear cartoons (Figure 7b), reveals a notable challenge in capturing emotions. A rapid fluctuation in arousal levels is observed, oscillating between positive and negative ratings. This erratic pattern suggests that the model encounters difficulties in accurately capturing the facial expressions of cartoon characters, given their simplistic features. Unlike human faces, cartoon characters lack the nuanced and distinctive facial features that the model may rely on for emotion recognition. The inherent simplicity in cartoon facial expressions poses a challenge for the model to interpret and assign consistent arousal ratings. Similarly, when examining videos featuring animals, such as the lion clip, challenges arise due to the inherent difficulty in reading the facial expressions of non-human entities. The lion video (Figure 7a), in particular, transitions quickly from baby lions to adult lions roaring with open mouths. This rapid shift contributes to the fluctuating and swiftly dipping arousal ratings. Animals, much like cartoons, present a unique set of challenges for the model, as their facial expressions may not align with the human-centric cues the model is trained to recognize. The complexities introduced by cartoon characters and animals in the emotion analysis process highlight the limitations of the model when faced with non-human or simplified facial features. Future improvements may involve training the model on a more diverse dataset that includes a wider range of facial expressions, encompassing both human and non-human entities, to enhance its capability to accurately capture and interpret emotions across varied visual contexts.</p><p><strong><em>Landscape orientation vs Portrait orientation</em></strong></p><p>a) Saturday Night Live Bereal landscape</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_czfK_TIuWYDXLpvt5NBKA.png" /></figure><p>b) Saturday Night Live Bereal portrait</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WZw0vGBncL6LSDqjZq7evw.png" /></figure><p>c) Harry Potter landscape</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VH0AUfZOP59jWe_X_MEl0Q.png" /></figure><p>d) Harry Potter portrait</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WIlCRwEcj6XZgetrW_fRoA.png" /><figcaption>Figure 7. Emotion predictions of new videos outside of the VEATIC dataset with the same videos in landscape vs. portrait orientation.</figcaption></figure><p>Analyzing videos in both portrait and landscape orientation reveals interesting observations regarding the impact of framing on emotion capture. In the portrait versions of both videos (Figure 7b, 7d), there is a discernible increase in fluctuations and changes in both arousal and valence. This can be attributed to the model’s focused attention on the character’s facial expressions, minimizing consideration of external factors. Conversely, the landscape-oriented model (Figure 7a, 7c) demonstrates a higher degree of accuracy in interpreting the emotional data. This effectiveness may stem from the broader perspective provided by the landscape format, allowing the model to incorporate environmental cues and body language. The landscape model’s ability to capture a more comprehensive view contributes to a more nuanced understanding of the emotional dynamics within the scenes. However, the limitations of the portrait videos become apparent. The narrower frame in portrait mode may lead to characters being cut out of the videos, potentially contributing to the observed fluctuations in arousal and valence ratings. The restricted view in portrait clips may hinder the model’s capacity to contextualize emotions within the broader visual and environmental context. In conclusion, the choice between portrait and landscape orientations significantly influences the model’s performance in emotion capture. While portrait formats offer a more focused analysis of facial expressions, landscape formats provide a more holistic view, leading to a potentially more accurate interpretation of emotional dynamics. Careful consideration of framing is crucial in optimizing the model’s performance across varied visual contexts.</p><h3>Discussion</h3><p>Constraints within our model evaluation revolved around the small set of videos due to resource constraints. To accommodate the computational expenses associated with long videos, the Streamlit platform, employed for our analysis, necessitated the condensation of many clips. An additional challenge arose from videos that frequently switched frame perspectives, potentially introducing inaccuracies in the captured emotional data.</p><p>Turning our attention to valence and arousal, the former reflects the pleasantness or unpleasantness of an emotion. Positive valence encompasses emotions such as happiness, joy, and love, signifying pleasant and desirable states. Conversely, negative valence is associated with emotions like sadness, anger, and fear, representing unpleasant and generally undesirable feelings. Regarding arousal, it measures the intensity or activation level of an emotion. Arousal is always represented as a positive number on a scale from low to high intensity. For example, a relaxed state corresponds to low arousal, while excitement or anger signifies high arousal.</p><p>Analyzing the results, it’s notable that many valence charts exhibited fluctuations between positive and negative values, often starting at zero. This pattern suggests a certain level of ambivalence in the emotions observed. External factors such as noise, environmental influences, or other small perturbations may contribute to these minor fluctuations in emotional expression. Recognizing and accounting for these factors is crucial for a more nuanced interpretation of the emotional dynamics captured in the analysis.</p><p>In an optimal testing scenario, the ideal video setup would involve a stable background without alterations, allowing CNNs to more accurately capture the character’s emotions, eliminating extraneous noise or interfering factors. It’s important to note that our analysis did not incorporate sound, a significant oversight given its potential to reveal crucial information about emotions.</p><h3>References</h3><p>Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., … &amp; Houlsby, N. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. <em>arXiv preprint arXiv:2010.11929</em>.</p><p>He, K., Zhang, X., Ren, S., &amp; Sun, J. (2016). Deep residual learning for image recognition. In <em>Proceedings of the IEEE conference on computer vision and pattern recognition</em> (pp. 770–778).</p><p>Ren, Z., Ortega, J., Wang, Y., Chen, Z., Whitney, D., Guo, Y., &amp; Yu, S. X. (2023). VEATIC: Video-based Emotion and Affect Tracking in Context Dataset. <em>arXiv preprint arXiv:2309.06745.</em></p><p>Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., … &amp; Fei-Fei, L. (2015). Imagenet large scale visual recognition challenge. <em>International journal of computer vision</em>, <em>115</em>, 211–252.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=01f704a9bfbb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[News Generation]]></title>
            <link>https://medium.com/@ucladsu/news-generation-191087f08a7d?source=rss-e0043e83850a------2</link>
            <guid isPermaLink="false">https://medium.com/p/191087f08a7d</guid>
            <dc:creator><![CDATA[UCLA Data Science Union]]></dc:creator>
            <pubDate>Mon, 04 Dec 2023 05:04:19 GMT</pubDate>
            <atom:updated>2023-12-06T09:47:06.515Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Project Lead</strong>: <em>Dylan Winward </em><strong>Project Members</strong>: <em>Leo Cardozo, Rithwik Narendra, Defne Tanyildiz, Nathan Taylor Trux, Emily Widjaja</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zBHqi7jhxvhqfyHS.png" /><figcaption>Pictured is a newsroom staffed entirely by robots. Image Credit: Dall-E 2.</figcaption></figure><h3>Introduction:</h3><p><strong>Why do this project?</strong></p><p>News journalism is a fast developing field with the need to meet deadlines quickly and summarize information in a fast format. Typically, journalists are rushing for three things: accuracy, speed to publication and SEO optimization.</p><p>Lots of the reporting that most outlets use is not their own — instead coming from the wire services, like Reuters or the Associated Press. However, articles from wire services are often rewritten to provide more detail and to make them more interesting</p><p>What if this could be done using artificial intelligence?</p><h3>What were we hoping to do?</h3><p>The goal of the project is to basically regurgitate news wire briefs so that they are more interesting, without losing too much factual accuracy. We are hoping to do this on an experimental basis, rather than creating a product output.</p><p>We’re also not going to deal with any ethical ramifications of the work we do. Journalistic ethics are a complicated and nuanced field — rather than proposing a product for rollout, we instead want to analyze the status quo feasibility of this idea.</p><h3>Background</h3><p>The project also comes at an interesting time for the intersection of artificial intelligence and news generation. In amidst a climate of anti-AI strikes, allegations of fake news and a serious investigation at Sports Illustrated about the use of AI, the project has continued relevance with current news today.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NFjP3HEneEN4QhLb.png" /><figcaption>Pictured is a headline about the recent artificial intelligence scandal at Sports Illustrated, a leading sports news magazine. Image Credit: The Guardian</figcaption></figure><h3><strong>Data Acquisition:</strong></h3><p><strong>Strategic Overview:</strong></p><p>Overall, we set out to try to get data from two different wire services with different formats so we could compare the outcomes. In the end, we scraped data from the Associated Press and from Reuters.</p><p>Although we initially aimed to get samples from the same timeframe, it did not prove feasible during scraping, so the AP samples were from 2023 and the Reuters samples were from 2022.</p><p>In the end, we scraped 8 samples from each site.</p><p><strong>Acquisition of Reuters Dataset:</strong></p><p>We primarily used beautifulsoup to acquire the Reuters data. While the AP dataset consisted of longer articles, we wanted to gauge how generation would work with respect to stubs, or press releases. Therefore, we randomly selected 60 Reuters newswires.</p><p>Reuters had a blocker in place which prevented users from accessing certain portions of their HTML. In order to bypass this blocker, we used a header in our request.</p><p>Then, we went through various fields in the html until we were able to extract just the article text. We added that text to a csv.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*B7sHDDafVR4o_beh.png" /><figcaption>Pictured is the first page of code to scrape samples from Reuters.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/0*pyNimjz49RDeYsWC.png" /><figcaption>Pictured is the second page of code to scrape samples from Reuters.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/0*gn9PWD5ZY5qWihIF.png" /><figcaption>Pictured is the final page of code to scrape samples from Reuters.</figcaption></figure><p><strong>Acquisition of Associated Press Dataset:</strong></p><p>Articles from the AP Wire were collected using an automated web scraper tool called Apify. Apify allows you scrape article text from websites provided a link, and we were able to take advantage of a 7-day free trial for the purposes of collecting our data. The tool allows you filter by the maximum amount of articles you want, and you can limit the date range on how far back articles will be scraped.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/0*4BXV8g2Z0ttv2NCN.png" /><figcaption>Pictured is an explanation of how the Apify tool was used to scrape content from the Associated Press website.</figcaption></figure><p>Initially, we started out by plugging in just the general AP website, however this would only allow us to scrape articles from the day of, which a large language model may not be up to date on. Instead, we used articles from the archive, which were primarily produced around roughly June 2023.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NJD71Y9ClTgzUcjU.png" /><figcaption>Pictured is a Bruin Bear in a data center. Image Credit: Dall-E-2.</figcaption></figure><h3>Generation</h3><p><strong>Strategic Overview:</strong></p><p>Having received 16 samples from scraping, we fed them all into ChatGPT and Llama. The point of us doing this was to create 4 different datasets (AP-ChatGPT, Reuters-ChatGPT, AP-Llama, Reuters-Llama) so we could cross-compare them.</p><p><em>Editors Note: The word Llama will be used in this project to signify Llama-2, which was the actual module used. Llamas as a plural form is also used in some graphics to signify the reference to multiple samples generated.</em></p><p><strong>Generation via Chat GPT:</strong></p><p>Although Chat GPT 3.5 is not able to intake uploaded files, we were still able to generate using the csvs we had created after scraping the AP and Reuters sources. Chat GPT 3.5 is able to parse comma separated fields that are copy and pasted from a csv into the prompt field. It is not necessary for the user to tell Chat GPT that the data came from a csv, but we explicitly told it. Then, we asked it to generate a news article with the following prompt: Rewrite the following sample into an interesting, readable and engaging news article of about 500 words. Afterwards, we asked it to give the piece a title.</p><p><strong>Generation via Llama:</strong></p><p>We generated the Llama sample by using a program called LM Studio, after finding it easier than doing it directly in the Command line.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JDO7Mgx7T-YKLViC.png" /><figcaption>Pictured is LM Studio, which was used for the Llama text generation.</figcaption></figure><h3>Evaluation Metrics:</h3><p><strong>Evaluation by AI:</strong></p><p>The first criteria we evaluated was the factual accuracy of the AI generated articles. In order to do this, we used Longshot, an AI tool that identifies misleading or inaccurate claims in text. The output was either a ‘True’ rating if all claims were completely clear, or a ‘False’ rating if one or more claims were inaccurate or misleading. An explanation was provided in the reasoning for each rating, and this explanation was used to produce a numerical score of an article’s accuracy, with articles most heavily punished for factually incorrect claims, followed by misleading claims, and lastly claims that AI was unable to confirm. Articles with a ‘True’ rating automatically received a numerical score of 10.</p><p>The second factor we evaluated was the adherence of the AI generated news article to the source material. To do so, we put the original text and the generated news article in a two text plagiarism detector and got a plagiarism percentage, the amount of words shared, and a breakdown of how many sentences were kept exactly the same or were paraphrased.</p><p>The third criterion that we evaluated was the quality of writing of the AI generated news articles from both sources. To do so, we input the generated articles into ChatGPT and asked the AI tool to rate the quality of writing from 1 to 10 and to produce a short paragraph as to why it provided such a rating.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JIBRzBdP8ElWZQg9.png" /><figcaption>Pictured is a conceptual image of a man fighting against a robot. Image Credit: DALL-E 2.</figcaption></figure><p><strong>Evaluation by Hand:</strong></p><p>In order to evaluate factual accuracy by hand, specific claims from generated articles were compared to their original counterparts, and imagined claims by the generative models were compared to reliable internet sources. Quotes and numerical figures were more intensely examined, and word choice was scrutinized for sensitive descriptions of policies and lead-ins to quotes. In addition to false claims, contextual information imperative to articles was also searched for to make sure generated articles included the full story. Points were redacted based on the impact of the falsehood: 1 point was redacted for missing contextualization, 2 points were redacted for misleading commentary and less impactful lies, 5 points were redacted for detrimental fallacies.</p><p>In a similar fashion to the evaluation by AI, the second factor we explored by hand was the extent to which the generated articles reflected or adhered to original news pieces. Here, we looked for similar structure, facts being mentioned, usage of similar language, and the incorporation of unique words. We also checked for quotes or similar paraphrasing. We gave a score 1 for no adherence and 10 for complete adherence. Many of the articles built on and fleshed out some of the points or stubs in the original article, mixing adherence with imagination in some instances.</p><p>The third factor we were evaluating by hand was the “quality of writing” of the pieces. To do this without it becoming too overbearing from a time-perspective, we broke this down into an assessment of different categories of evaluation, which we scored on a scale of 0–2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/0*Z3LWzlULIhvLcvpl.png" /></figure><h3>Quantitive Analysis of Scores:</h3><p><strong>AP vs Reuters:</strong></p><p>There was generally speaking a preference for articles generated using samples from the Associated Press (which received an average score of 7.32) as opposed to those scraped from Reuters (which received an average score of 5.34).</p><p>That preference for the Associated Press as source material also existed across both generation language learning models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/0*hCRPAo4xSoGizynm.png" /><figcaption>Pictured is a bar chart comparing the total average scores of articles generated using original samples from the Associated Press as compared to those generated by Reuters. Image Credit: Google Sheets</figcaption></figure><p><strong>ChatGPT vs Llama:</strong></p><p>Overall, there was a preference for articles created by ChatGPT compared to those created by Llama.</p><p>Furthermore, both types of evaluation showed a clear preference for ChatGPT, with evaluation by hand demonstrating it as having a 10.1% advantage and machine-based evaluation showing it as having a 10.05% advantage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/886/0*iZsAh5I4hH4Ymq5M.png" /><figcaption>Pictured is a bar chart comparing the total average scores of articles generated using Chat GPT against those generated by Llama. Image Credit: Google Sheets</figcaption></figure><p>However, to get a more nuanced understanding of the strengths and weaknesses of each mode, we need to delve into how ChatGPT scored when compared to Llama in each individual evaluation category.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/884/0*tjauOPZcBla-Ff77.png" /><figcaption>Pictured is a bar chart depicting how different language learning models fared when looking at each category of assessment. Image Credit: Google Sheets</figcaption></figure><p>For an even more nuanced understanding, we can look to the different evaluation types as well. From the results, it appears that while quality of writing aggregates to be a similar value, different evaluation metrics yield significantly different results.</p><p>When we look to the quality of writing of AI generated pieces, it appears that humans assess Llama to be superior at writing. However, an AI based assessment of the quality of writing prefers things written by ChatGPT. Does this mean that Llama is the more “natural” of the two language learning models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/890/0*vDlvtvoEDHWRJe23.png" /><figcaption>Pictured is a bar chart depicting how different language learning models scored in different categorical breakdown. Image credit: Google Sheets</figcaption></figure><h3>Qualitative Observations from Evaluation:</h3><p><strong>Evaluation by AI:</strong></p><p>On average, ChatGPT performed better for factual accuracy. Interestingly, ChatGPT performed much better on Reuters articles, while Llama performed better on AP articles. The cause of the difference between news sources and LLM’s is a question for future work.</p><p>When looking at the plagiarism scores across the news articles generated from the Reuters pieces, scores would be one of two extremes. Generally, either the pieces got a 0% or above a 100%, with two exceptions being 17.9% and 59%. This is likely due to the shortness of the Reuters pieces and how the AI distributed the information given by the source material. With pieces that scored 100%, the Reuter’s source material would be present in the introduction of the piece in paraphrased form or with a few identical sentences. Pieces that scored 0% had fully changed the sentence structure of the Reuters source information and scattered it around the piece and received a 0% on all factors.</p><p><strong>Evaluation by Hand:</strong></p><p>In general, articles generated based on the AP pieces adhered more to the original than did articles generated based on the Reuters newswires. This makes sense, as for the newswires, the AI was asked to write over 500 words based on newswires which were only one paragraph in length. By contrast, the AI had much more to work with for the AP pieces, leading to more copying. Chat GPT tended to use the same quotes, while Llama avoided using the quotes given in the original wire service samples, instead making up quotes. This led to more flavorful writing in the Chat GPT versions of the AP pieces compared to the Llama pieces. While the original AP articles contained conclusions which elicited thought and discussion, Chat GPT concluded its generations with “In conclusion” a plurality of the time. Llama, too, erred on the side of “school paper” by simply summarizing the article or making an abstract philosophical argument and call to action which had nothing to do with the piece.</p><p>One conclusion from the by-hand analysis of the quality of writing of the samples is that they tended to do better at providing contextual information than information that would typically come from sourcing.</p><p>This probably should come as no surprise, given that it is easier for the models to find general historical information about the world from their training data than it is for them to find specific statements people have made about something.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nBL8HEbTBBhTetnZ.png" /><figcaption>Pictured is a bar chart comparing how different factors compared to each other in a by-hand analysis of the quality of writing of the samples. In this analysis, the ‘lede’ described whether the article had a precice introduction. The tense checked whether the article was written in the right tense. The nutgraf checked if appropriate contextual detail was given, whereas quote and paraphrase checked if those elements of ordinary news articles were shared. The factors were all scored out of 2. Image Credit: Google Sheets.</figcaption></figure><p>Interestingly, the by-hand quality of writing analysis showed a preference for Llama as opposed to Chat GPT, which bucked the overall trend. In fact, Llama received a significantly higher average score in this type of analysis, receiving an average of 6.8 points as opposed to 5.3.</p><p>This difference appears to have been driven primarily by Llama better ability at providing quotes and in summarizing information precisely.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vtoSKXdjvxZ7eLz7.png" /><figcaption>Pictured is a bar chart comparing average scores of both language learning models when it comes to different types of by-hand writing analysis comparisons. Image Credit: Google Sheets</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=191087f08a7d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Healthy Communties]]></title>
            <link>https://medium.com/@ucladsu/healthy-communties-b8e02cc8ec81?source=rss-e0043e83850a------2</link>
            <guid isPermaLink="false">https://medium.com/p/b8e02cc8ec81</guid>
            <dc:creator><![CDATA[UCLA Data Science Union]]></dc:creator>
            <pubDate>Wed, 07 Jun 2023 08:41:23 GMT</pubDate>
            <atom:updated>2023-06-07T23:19:14.276Z</atom:updated>
            <content:encoded><![CDATA[<h3>Healthy Communities</h3><p><strong>Project Lead</strong>: <em>Rithwik Narendra</em> &amp; <strong>Project Members</strong>: <em>Emily Widjaja, Maia Smolyanov, Natalie Chapin, Riley Leong</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*sZ8yTrJxleDtLce2uTOIow.png" /></figure><h3><strong>Introduction:</strong></h3><h4>Health Outcomes</h4><p>As health research has advanced in the past few decades with the advent of global health technologies, streamlining of government funding and grant approvals, and a newfound scientific interest in global health, scientists and government officials alike have developed metrics to measure health outcomes and performance. Key health outcomes include, but are not limited to, life expectancy and mortality rates.</p><p>Developments in science and technology have increased countries’ ability to improve their public health outcomes, and many have. Yet the United States continues to lag behind many of the countries it claims to be peers with when it comes to these outcomes. We seek to understand why by analyzing the data behind-the-scenes. What drives these outcomes? What is most important? And most importantly, what should our communities focus on moving forward?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/303/1*_gz9_cAqjiKGPzaTlDH-xA.png" /></figure><h4>Our Dataset(s)</h4><p>Thankfully, all of this data is easily accessible! The University of Wisconsin Madison <a href="https://www.countyhealthrankings.org/explore-health-rankings/rankings-data-documentation">publishes </a>national health rankings and outcomes data with extensive documentation. Based on the specific subproject, we used different datasets within the national data section of the health rankings.</p><p>Our aim was to analyze these outcomes on a county-by-county level to obtain local specificity with our conclusions. We also sought to observe geographic, cultural, and political trends, to see if anything external was driving any of the observed relationships. Because of this, it made most sense to use the County Health Rankings (CHR) datasets. We also made extensive use of the technical documents, data documentation, and the analytic data.</p><h4>Our Subprojects</h4><p>The dataset is comprehensive, extensive, and covers a wide array of health data. Therefore, it made the most sense to explore different aspects of it via different subprojects.</p><ol><li>An exploration of teen birth rate</li><li>What predicts different levels of physical activity?</li><li>The geographic relationship between obesity and a number of predictors</li><li>The life expectancy question</li></ol><h3>The Sub Projects</h3><h4>An Exploration of Teen Birth Rate — Natalie Chapin</h4><p>I began my project with some initial cleaning. First, I put the datasets “Ranked Measure Data” and “Additional Measure Data” together. I then split that master dataset in two: one with county data and one with state summary data. The remainder of my project uses the county dataset.</p><p>I then focused my efforts on examining the “Teen Birth Rate” datapoint, which is expressed per 1,000 females aged 15 to 19.</p><p>I started by plotting teen birth rate against factors with which I assumed it would share a causal relationship. Below are two such plots.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*SrMcYaRv6pi21FPgmmX3NA.png" /></figure><p>I thought school funding would correlate with teen birth rate because I assumed schools with more money would have the proper resources to prevent teen pregnancies and keep students on track for graduation. The students at those schools would also have more money in general, and therefore have more resources at their disposal to prevent disruptive events like teen pregnancies. I reasoned that the gender pay gap would correlate with teen birth rate because I thought teen girls who became pregnant were less likely to finish school, and therefore earn less in their careers. However, as evidenced above, neither school funding nor the gender pay gap seemed to correlate with the teen birth rate.</p><p>Since my first instincts proved incorrect, I widened my search and began plotting teen birth rate against any factor that could be vaguely correlated. From this, I got much more promising results, a few of which are pictured below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*FY-AkCcal-SiFMTtoyjJLQ.png" /></figure><p>Based on my scatterplots, I then made a new dataset with the 23 factors that seemed the most correlated with teen birth. A heatmap of those factors is pictured below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/765/1*PacNEEYXVM0IaizCoFvTGw.png" /></figure><p>Next, I imported a linear regression model and fit it to that dataset. The model reported a coefficient of determination of 0.82 and mean absolute error of 2.98. Not bad!</p><p>Finally, I scaled my data and performed PCA. For my first principal component — which explained about 50% of the variance in teen birth rate — the percentage of adults with diabetes, percentage of people in frequent physical distress, and percentage of children in single-parent households were the most influential factors. For my second principal component — which accounted for another 10% of the data’s variance — the age-adjusted suicide rate, percentage of people in frequent mental distress, homicide rate were the most influential.</p><p>All in all, this project was a great learning experience for me as I got my first exposure to working with a large dataset, insight into the modeling process, and a crash course on the intuition behind PCA.</p><h4>What Predicts Physical Activity? — Riley Leong</h4><p>Physical activity has been shown to reduce the risk of heart disease, prevent Type II Diabetes, and prevent many forms of cancers. Inversely, a lack of physical activity can have harmful impacts to the human body.</p><p>In an attempt to build a model to predict physical inactivity in different regions, we can begin with PCA. The dataset used contains 11 different components that each affect our target variable. The first step is to reduce the dimensions of our data into two components. After our initial PCA, a variance ratio of [0.84147548, 0.15852451] is observed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/933/1*rpn3xjEpck1EZ9kJzYHMbg.png" /></figure><p>Let’s break down what each of these numbers means:</p><ul><li><strong>0.84147538</strong> — The first and most important component our PCA algorithm computed represents our original data with an accuracy of about 85%</li><li><strong>0.15852451</strong> — The second and less important component our PCA algorithm computed represents our original data with an accuracy of about 15%</li></ul><p>We just significantly reduced the dimensionality of the dataset and retained most of the information. Now, we can invert this process and see how the original data from our dataset compares to our transformed, then reconstructed data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/605/1*4efKAJi8VtwjCxHByodqiA.png" /></figure><p>The above graph displays the relationship between the original dataset, and the most important component after PCA. As explained by our variance ratio, the first component of the principal component analysis only retained about 85% of the original data, so the discrepancies in the graph are expected. This is important because it tells us that the variance in our data can be effectively represented by this one component, and will help us choose our model.</p><p>Next, we can perform more exploratory data analysis to determine what factors our PCA algorithm determined to be most influential to our target variable. After exploring the data we determined that the unemployment rate and urban sprawl rating(inefficient use of land) in each state were significantly correlated to physical inactivity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/538/1*WLmpEJxrdVrxXC4gYiS2_w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/543/1*1abszMFncrH4JZ8fBQcSfg.png" /></figure><p>After determining potential independent variables, we can begin to build a model. Since we want to predict physical inactivity, and wish to use multiple components to do this, a multiple linear regression model is a logical choice for our model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/593/1*LPFOA3v-l0a7T6NV6urHlQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/453/1*qflJ6YefuZ6lDwr4Wypa6w.png" /></figure><p>Our model achieved a score of 0.8185 when trying to predict the physical inactivity of a state. While trained on limited information, our model is showing that a large proportion of variability in physical inactivity can be determined from unemployment rate and an urban sprawl rating. Some future directions to improve this model would be to have a larger dataset, have more recent urban sprawl data, and investigate different components determined by the initial PCA process.</p><h4>The Geographic Relationship Between Obesity and a Number of Predictors — Maia Smolyanov</h4><p>Every year, the University of Wisconsin, Madison releases public health data about different Counties in the United States. One health attribute that is observed about each county in the percentage of adults with obesity in each county. Obesity is known as a common risk factor for many health conditions such as type II diabetes, heart disease, cancer, and many more. Therefore, understanding what could be contributing to rising obesity rates in the United States could help us develop policies to curb the rise of these other diseases. In this section of the project, we will observe which other factors observed in the University of Wisconsin health data seem to affect obesity rates.</p><p>Since the original dataset is so large, we separated out only the columns that we will be observing and loaded them into a separate csv file. The specific columns that we will observe are as follows: % Adults with Obesity, Food Environment Index, % Physically Inactive, # Limited Access to Healthy Foods, % Frequent Physical Distress, % Frequent Mental Distress, % With Access to Exercise Opportunities, Life Expectancy, Median Household Income, # Uninsured Adults, and % Excessive Drinking. In order to prepare the data for a proper exploratory data analysis, we needed to change the # Limited Access to Healthy Foods and # Uninsured Adults Columns to percentages in order to be able to compare counties and draw better insights. In order to do this, we combined our data with population data from the US Census Bureau using the MATCH function in Excel. We also dropped rows that contained missing values or presumably incorrect values such as percentages above 100. This process left us with 2,712 counties to observe.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KESBBRTv957E23OoJ-bFww.png" /></figure><p>From the visualization above, we can see that states in the southern region of the United States suffer from the highest average rates. Even though obesity rates are relatively high throughout the country, this map tells us that the southern region would benefit most from policy aiming to lower obesity rates.</p><p>Individual factors that seem to correlate very linearly with the percentage of adults with obesity are % Physically Inactive, % Frequent Physical Distress, % Frequent Mental Distress, Life Expectancy, and Median Household Income. Here is a heatmap of the correlation between % Adults with Obesity and the variables we are observing:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/461/1*TzWrPhjKBfk1w4lfX6UJpw.png" /></figure><p>As you can see, % Physically Inactive, % Frequent Physical and Mental Distress, Life Expectancy, and Median Household Income are the only variables that have a correlation of above 0.5 or below -0.5.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/933/1*kbV41t11iddq1n5ZPjg8nQ.png" /></figure><p>Since many of the variables exhibited linear correlations with % Adults with Obesity, we used multiple linear regression to try to predict the % of Adults with Obesity in a county based on different factors. Our first model used all of the variables, resulting in an R-squared value of 0.65 and a mean squared error of 2.035, meaning about 65% of the variation in % Adults with Obesity is explained by the other variables and the model was on average 2% off from the actual percentage of adults with obesity. The equation for this linear model is the following:</p><blockquote>% Adults with Obesity = (3.69e-1)(Food Environment Index)</blockquote><blockquote>+ (6.75e-1)(% Physically Inactive)</blockquote><blockquote>+ (3.58e-2)(% Limited Access to Healthy Foods)</blockquote><blockquote>+ (-4.62e-1)(% Frequent Physical Distress)</blockquote><blockquote>+ (7.18e-2)(% Frequent Mental Distress)</blockquote><blockquote>+ (-1.57e-2)(% with Access to Exercise Opportunity)</blockquote><blockquote>+ (-2.57e-1)(Life Expectancy)</blockquote><blockquote>+ (-4.83e-5)(Median Household Income)</blockquote><blockquote>+ (-2.52e-2)(% Uninsured Adults)</blockquote><blockquote>+ (5.89e-2)(% Excessive Drinking)</blockquote><p>We developed a second model using only the variables that had a correlation of greater</p><p>than 0.5 or less than -0.5 with % Adults with Obesity. However, this model resulted in an Rsquared value of 0.64 and a mean squared error of 1.97, which is not much of an improvement from the previous model. Here is the equation for the second linear model:</p><blockquote>% Adults with Obesity = (6.83e-1)(% Physically Inactive)</blockquote><blockquote>+ (-6.29e-1)(% Frequent Physical Distress)</blockquote><blockquote>+ (9.54e-2)(% Frequent Mental Distress)</blockquote><blockquote>+ (-2.44e-1)(Life Expectancy)</blockquote><blockquote>+ (-5.41e-5)(Median Household Income)</blockquote><p>Using principal component analysis, we turned our 10 variables into 2 principal components. Here is a scatter plot of our two principal components:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/663/1*IPZDzQxO0Eev8DVzyKeHYA.png" /></figure><p>To see which of our variables contributed the most to each of our principal components, here is a heatmap showing the importance of each of our 10 variables in both principal components:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/594/1*bn3HAamXcoV5uZ39ppv14w.png" /></figure><p>From the scatter plot, we can see that the first principal component correlates with % Adults with Obesity much better and from the heatmap, we can see that the first principal component is mostly made up of Life Expectancy, Frequent Physical and Mental Distress, % Physically Inactive, and Median Household Income.</p><h4>The Life Expectancy Question — Emily Widjaja</h4><p>In my analysis of the overall “health” of states across the US, I aimed to explore the interplay between socioeconomic and physical factors in determining the well-being of individuals. I used life expectancy as a measure of health and chose to analyze the following predictor variables:</p><p><strong><em>Socioeconomic factors:</em></strong></p><ol><li><strong>“% Unemployed”</strong>: Unemployment rates can impact various aspects of individuals’ lives, including access to healthcare, financial stability, and overall well-being. Higher unemployment rates may lead to decreased life expectancy due to associated stress, limited access to healthcare, and increased poverty rates.</li><li><strong>“Income Ratio”</strong>: Income inequality is a crucial socioeconomic factor influencing health outcomes. Higher income inequality can be attributed to disparities in access to healthcare, education, and other resources, which can impact life expectancy.</li><li><strong>“Median Household Income”</strong>: Median household income measures economic well-being and can influence access to quality healthcare, education, nutrition, and other resources. Higher median household income is generally associated with better overall health outcomes and longer life expectancy.</li><li><strong>“% Food Insecure”</strong>: Food insecurity, defined as limited or uncertain access to adequate food, can have detrimental effects on health. It can lead to poor nutrition, higher risk of chronic diseases, and compromised immune systems, all of which can impact life expectancy.</li></ol><p><strong><em>Health-related factors:</em></strong></p><ol><li><strong>“% Adults with Diabetes”</strong>: Diabetes has been increasingly prevalent in recent years and is a significant risk factor for various health complications. Higher rates of diabetes within a population can contribute to lower life expectancy due to associated cardiovascular diseases, organ damage, and other complications.</li><li><strong>“Child Mortality Rate”</strong>: Child mortality rate is a vital indicator of a population’s overall health and well-being. Higher child mortality rates often indicate challenges in healthcare systems, social infrastructure, and environmental factors affecting life expectancy.</li><li><strong>“% Insufficient Sleep”</strong>: Sleep, though often overlooked, plays a vital role in maintaining overall health and well-being. Chronic insufficient sleep can contribute to various health issues, such as obesity, cardiovascular diseases, and mental health disorders, affecting life expectancy.</li><li><strong>“# Drug Overdose Deaths”</strong>: Drug overdose deaths, particularly from opioids, have become a public health crisis in many regions. Higher rates of drug overdose deaths can significantly impact life expectancy, reflecting challenges related to substance abuse, healthcare access, and social issues.</li></ol><p><strong>Exploratory Data Analysis:</strong></p><p>After extensive data cleaning and running the correlation function on the data, the picture below shows the correlation coefficients between each pair of variables in the data set:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zbsQhM2GQGcGzwtxJte3ug.png" /></figure><p>From the results above, the following correlation coefficients stood out:</p><ol><li>Median Household Income ~ Life Expectancy -&gt; 0.7489375</li></ol><p>2. % Food Insecure ~ Life Expectancy -&gt; -0.7326181</p><p>3. % Insufficient Sleep ~ Life Expectancy -&gt; -0.6516089</p><p>These relatively high correlation coefficients indicate that the associated predictor variable has a positive or negative linear relationship with life expectancy.</p><p><strong>Linear Regression:</strong></p><p>After splitting the data into a train and test set (7: 3), I built a linear regression model for the training set data. The picture below summarizes the model:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*I19eSc62c1Mq8pImQI0-xQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/998/1*F0RNZrjlDUmo90yj09zQBw.png" /><figcaption>Residuals Histogram</figcaption></figure><p>Based on the provided output, the predictor variables with the most significant impact on Life Expectancy are:</p><p>1. Child Mortality Rate: This variable has a coefficient estimate of -3.664e-02, a low standard error, and a highly significant t-value (t value = -11.529). The p-value is also &lt; 2e-16, indicating a highly significant relationship. Therefore, a decrease in the child mortality rate is associated with an increase in life expectancy.</p><p>2. % Insufficient Sleep: This variable has a coefficient estimate of -1.938e-01, a low standard error, and a highly significant t-value (t value = -10.174). The p-value is &lt; 2e-16, indicating a highly significant relationship. Thus, a decrease in the percentage of people with insufficient sleep is associated with an increase in life expectancy.</p><p>3. % Food Insecure: This variable has a coefficient estimate of -2.138e-01, a low standard error, and a highly significant t-value (t value = -8.238). The p-value is &lt; 2e-16, indicating a highly significant relationship. Therefore, a decrease in the percentage of food-insecure individuals is associated with an increase in life expectancy.</p><p>4. % Unemployed: This variable has a coefficient estimate of 1.440e-01, a relatively low standard error, and a significant t-value (t value = 4.486). The p-value is also significant (p &lt; 0.001). Thus, a decrease in the percentage of unemployed individuals is associated with an increase in life expectancy, although the effect may be relatively smaller compared to the previous variables.</p><p>The multiple R-squared and adjusted R-squared values measure the model’s overall performance. The multiple R-squared value is 0.7404, indicating that approximately 74.04% of the variability in Life Expectancy can be explained by the predictor variables. The adjusted R-squared (0.7385) is slightly lower due to the penalty for including additional variables in the model. The F-statistic tests the overall significance of the model. In this case, the F-statistic is 388.2, with a p-value smaller than 2.2e-16. This suggests that the linear regression model as a whole is statistically significant in predicting Life Expectancy.</p><p><strong>PCA:</strong></p><p>After running PCA on the predictor variables, the following image summarizes the results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aUEljDZzi6WrxZwC_rpnsQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QgngHqDiIhFe_x96j4UsxA.png" /></figure><p>Analyzing the first and most significant principle component:</p><ol><li>% Unemployed has a positive loading of 0.2238, indicating a moderate positive relationship with PC1.</li><li>Income Ratio has a positive loading of 0.3103, suggesting a moderate positive relationship with PC1.</li><li>Child Mortality Rate has a positive loading of 0.3661, indicating a relatively strong positive relationship with PC1.</li><li>% Adults with Diabetes has a positive loading of 0.4427, implying a relatively strong positive relationship with PC1.</li><li>% Food Insecure has a positive loading of 0.4483, indicating a relatively strong positive relationship with PC1.</li><li>% Insufficient Sleep has a positive loading of 0.3754, suggesting a relatively strong positive relationship with PC1.</li><li>Drug Overdose Deaths has a negative loading of -0.0435, indicating a weak negative relationship with PC1.</li><li>Median Household Income has a negative loading of -0.4241, suggesting a moderate negative relationship with PC1.</li></ol><p>The variables with higher positive loadings are “% Food Insecure,” “% Insufficient Sleep,” “% Adults with Diabetes,” and “Child Mortality Rate.” This suggests that PC1 captures a pattern related to health and well-being factors.</p><p><strong>Conclusion:</strong></p><p>The loadings in the PCA results indicate the relationship between the original variables and the principal components. Each principal component is a linear combination of the original variables, with the loadings indicating the strength and direction of the relationship. For example, in PC1, the variables <strong>% Food Insecure,” “% Insufficient Sleep,” “% Adults with Diabetes,” and “Child Mortality Rate” </strong>have relatively high loadings, indicating their strong influence on the overall variation captured by PC1.</p><p>The linear regression results show the relationship between the predictor variables and the response variable, “Life Expectancy.” The coefficients represent the estimated effects of each predictor variable on life expectancy. Significant coefficients suggest a statistically significant relationship. In this case, variables such as<strong> “% Unemployed,” “Child Mortality Rate,” “% Adults with Diabetes,” “% Food Insecure,” “% Insufficient Sleep,” and “Median Household Income”</strong> have significant coefficients, indicating their impact on life expectancy.</p><p>Based on these findings, it can be concluded that the analyzed socioeconomic and physical factors play a significant role in determining life expectancy in the US. More specifically, combining the results from PCA and linear regression, unemployment, food insecurity, insufficient sleep, diabetes and child mortality rate are associated with variations in life expectancy. These findings highlight the importance of addressing <strong>both socioeconomic and physical determinants of health</strong> to improve overall well-being and life expectancy and showcase how both are highly intertwined with one another.</p><h3><strong>Conclusion</strong></h3><p>As expected, the drivers behind our health outcomes are diverse and regionally specific. PCA and linear regression has shown that the data can be condensed into just a few principle components, which should make it easier to identify what actually matters, and what communities should emphasize.</p><p>Interestingly, building more hospitals or increasing the number of doctors in a given region would not be our recommendations. Instead, as can be seen by our analysis, we urge governments to focus on making general improvements, such as reducing income inequality, increasing median household income, combating mental distress, reducing unemployment, reducing the homicide rate, and improving mental health offerings. It appears that socioeconomic variables should be addressed first and foremost.</p><p>These findings provide a clearer picture of the state of American health, and what we must do to improve it moving forward. EDA and data cleaning has enabled us to find the signal in the noise. More research is needed to better understand the drivers behind child mortality and other regional factors impacting health outcomes. We would also like to better understand how trends change over time and in response to significant changes in national health and economic policy and conditions. Stay tuned for the 2024 dataset!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b8e02cc8ec81" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Text Summarization]]></title>
            <link>https://medium.com/@ucladsu/text-summarization-42a4e31b453f?source=rss-e0043e83850a------2</link>
            <guid isPermaLink="false">https://medium.com/p/42a4e31b453f</guid>
            <dc:creator><![CDATA[UCLA Data Science Union]]></dc:creator>
            <pubDate>Tue, 06 Jun 2023 11:26:24 GMT</pubDate>
            <atom:updated>2023-06-06T19:13:18.669Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Project Leads: </strong><em>Aaron Tae, Kevin Hamakawa </em>&amp; <strong>Project Members:</strong><em> Emily Gong, Emily Huang, Tony Lei, Ved Phadke, Vivian Lee, Victor Shi</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F-eGGbWRxyXiyHY7IfJpJQ.png" /></figure><h3>Introduction:</h3><h4>Different Methods of Text Summarization</h4><p>In the broad picture of text summarization, there are ultimately two strategies that you can take: <em>extractive</em> summarization and <em>abstractive</em> summarization. <em>Extractive</em> summarization is a strategy that requires you to find the most “important” sentences within a body of text, and combine the top “n” sentences together in order to make a summary of the text. <em>Abstractive </em>summarization utilizes neural networks or transformers in order to generate text that would best capture the meaning of any body of text. Throughout this article, we will demonstrate the similarities and differences between the two methods and showcase their strengths and limitations.</p><h4><strong>Our Dataset</strong></h4><p>We ultimately decided on two separate datasets for our text summarization project, both being found from the HuggingFace dataset collection. Our first dataset, labeled “billsum” on the HuggingFace website, is a dataset on US Congressional and California state bills, and contains 18.9k rows and 3 columns. These 3 columns include the entirety of the bill, the summary of the bill, and the title of the bill. As we will soon find out, extractive summarization doesn’t necessarily require you to have a summary for you to “train” a model, but abstractive text summarization does have this requirement.</p><p>The second dataset, labeled “xsum” on the HuggingFace website is a dataset that showcases news articles and their summaries, and contains 204k rows and 2 columns. The reason for our choice to utilize and examine two different datasets is because of the several different limitations of text summarization. For extractive summarization, a lot of the code required for preprocessing the data can be very specific to the type of text that is being read in, as several of the documents in the “billsum” dataset contain very specific punctuation and symbols that would only be present in this specific dataset. We wanted to showcase our strategies of text summarization with two different datasets to show how the approach may differ from dataset to dataset.</p><h3><strong>Extractive:</strong></h3><h4><strong>Text Preprocessing (String Manipulation)</strong></h4><p>Text preprocessing is an integral part of natural language processing (NLP), as it helps algorithms to analyze text more precisely and efficiently. Some of the reasons why we preprocess text data include:</p><ol><li>Reducing data size<br>a. stop words accounts 20–30% of total word counts in a particular text documents<br>b. stemming may reduce indexing size as much as 40–50%</li><li>Improving the efficiency and effectiveness<br>a. Stop words provides minimal information and may even confuse algorithms at times<br>b. After stemming, which we will cover in the next section, we can better match words</li></ol><blockquote><a href="https://www.researchgate.net/profile/Vairaprakash-Gurusamy/publication/273127322_Preprocessing_Techniques_for_Text_Mining/links/54f8319e0cf210398e949292/Preprocessing-Techniques-for-Text-Mining.pdf">Source</a></blockquote><h4><strong>Tokenizing/Stemming</strong></h4><p>Stemming is the process of reducing words to their root form by removing prefixes and suffixes from a word. This reduces the amount of total unique input information by essentially eliminating redundant words, reducing the complexity of the input space and giving certain stems the correct amount of importance relative to other ones. Stemming ensures that variations of the same word are treated as identical during analysis, enabling better comparison and identification of important content for summarization. There are many different algorithms that can be used for stemming, which each have their own purpose and degree of complexity. We used porter stemming for our project, widely considered the standard when it comes to NLP text pre-processing due to its simplicity and efficiency.</p><p>Tokenization, another important part of the text preparation process, involves dividing a text document into individual units, usually words or subwords, referred to as tokens. This process is necessary because computers understand and process individual units of text better than raw text. Tokenization allows the NLP system to treat each token as a separate entity, enabling subsequent analysis and manipulation. Additionally, tokenization helps in segmenting sentences, which is crucial for extractive text summarization. By breaking down a document into tokens, it becomes easier to identify important sentences and extract relevant information for summarization purposes.</p><h4><strong>How We Processed Our Datasets</strong></h4><p>For the project, we used the regex package to perform string manipulation and the Natural Language Toolkit (NLTK) to remove stop words and stem. Here’s a <em>general</em> procedure we took:</p><ul><li>For each text file<br>- Split the file into sentences (sentence tokenization)</li><li>For each sentence<br>- Remove newlines ‘\n’, tabs, extra white spaces, useless punctuations, lowering all text, etc.<br>- Split the sentence into individual words (word tokenization)<br>- Remove stop words<br>- Stem each word</li><li>Return cleaned text :D</li></ul><p>More specifically, for this project, we first started with the Bills dataset. Bills are inherently very complex. (Otherwise we wouldn’t need lawyers…) One issue we encountered very early on was that bills could not be split based on periods, unlike many other documents. For example, “SECTION 1. ENVIRONMENTAL INFRASTRUCTURE.” should be interpreted as one sentence, despite it having 2 periods. We also notice that bills cannot be split based on newlines, as that would cut “(a) Jackson …” into 3 different sentences. There’s also different levels of indentations, which makes things messy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k594ANJh9B905BMCSheaJA.png" /></figure><p>After looking for some patterns and testing on several bills in the test dataset, I think it’s safe to say that we got 90% of them to work. The algorithm we developed was very specific to the Bills dataset, however. One area of improvement is generalizing the algorithm to other types of text data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1L7P9eiRUhh2aRH_2vxYDw.png" /></figure><h4><strong>Utilizing TFIDF to Find the Most Important Sentences</strong>:</h4><p>TF-IDF (term frequency-inverse document frequency) is a measure that quantifies the importance/relevance of string representations in a document amongst a collection of documents.</p><p>Term frequency (TF) is how common a word is. The more times a word occurs, the higher its term frequency for that document.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/562/1*w-u4oHmlQl2W8G5B-Xz_BQ.png" /></figure><p>Inverse document frequency (IDF) is how unique or rare a word is. The more rare a word is, the higher its value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/536/1*c2ZPzgRrJjc3cDSKtd_Z5Q.png" /></figure><p>The TF and IDF values are multiplied together to form the TF-IDF score for each word in a document. This numerical representation can be used to extract keywords and sentences from a document and compare their relevance across a collection of documents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tscWlhvQ8ReEkEEIC-u3bQ.png" /></figure><p>For each sentence in a bill document, the TF-IDF values of all the words in the sentence were averaged over the total number of words in the sentence. This average TF-IDF value was the score given to that particular sentence, where a higher value meant a more important sentence. The mean of all the sentence scores was then found to determine a threshold TF-IDF value for the bill. If an individual sentence’s score was above this threshold, then the corresponding un-stemmed sentence was added to the summary. The sentence scores and original sentences were both iterated through in the order they showed up in the bill so that the final summary made sequential sense.</p><p><em>Example generated summary:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hFEbywwayK9Zlzv49VIEOw.png" /><figcaption>Example generated</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/272/1*3qcjuS_nGJZ3xaRBAT1SFg.png" /><figcaption>Rouge Score</figcaption></figure><p><em>Human written summary:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zQAtWjQCz1G0G-05DM7b2A.png" /><figcaption>Human written</figcaption></figure><h4><strong>Rouge Score</strong></h4><p>As we know, extracting key insights through extractive summarization from a vast amount of text has been a challenging but rewarding process. Extractive summarization, a branch of natural language processing, offers a promising solution by condensing lengthy documents into concise summaries. Among the various evaluation metrics for assessing the quality of these summaries, ROGUE-L, ROGUE-1, and ROGUE-2 have emerged as valuable tools as we have experimented with these three metrics during our project. We will explore the significance of these three metrics in evaluating the effectiveness of summarization algorithms in the following paragraphs.</p><p>ROUGE stands for Recall-Oriented Understudy for Gisting Evaluation. It is essentially a set of metrics for evaluating automatic summarization of texts as well as machine translations.</p><p>Simply put, <em>recall</em> (in the context of ROUGE) refers to how much of the <strong>reference summary</strong><em> </em>the <strong>system summary</strong> is recovering or capturing like indicated in this formula.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*rv5qHeXJnS1wrpunr8U7_Q.png" /></figure><p><strong>ROGUE-L</strong> (Recall-Oriented Understudy for Gisting Evaluation — Longest Common Subsequence) is a metric commonly used to evaluate the quality of extractive summaries. It measures the recall of the generated summary by comparing it to a set of reference summaries. ROGUE-L computes the longest common subsequence (LCS) between the generated summary and the reference summaries, providing a measure of how well the summary captures the essential information. The higher the ROGUE-L score, the better the summary is deemed to be.</p><p>An advantage of using LCS is that it does not require consecutive matches but in-sequence matches that reflect sentence level word order. Since it automatically includes longest in-sequence common n-grams, you don’t need a predefined n-gram length.</p><p><strong>ROGUE-1</strong> is a precision-based metric that focuses on the precision of extractive summaries. It calculates the ratio of the number of overlapping n-grams (n consecutive words) between the generated summary and the reference summaries to the total number of n-grams in the generated summary. ROGUE-1 penalizes longer summaries, as they have a higher chance of including irrelevant or redundant information. This metric provides a quantitative measure of how accurately the generated summary captures the key points from the source material.</p><p><strong>ROGUE-2</strong> evaluates the quality of summaries by considering the co-occurrence of two consecutive words, known as bigrams. It measures the precision of extractive summaries by computing the ratio of overlapping bigrams between the generated summary and the reference summaries to the total number of bigrams in the generated summary. ROGUE-2 offers additional insights into the coherence and fluency of the summary, as it captures the contextual relationship between adjacent words.</p><p><strong><em>Significance and Limitations:</em></strong></p><p>ROGUE-L, ROGUE-1, and ROGUE-2 play a crucial role in evaluating extractive summarization algorithms. They provide quantitative measures of the quality, recall, precision, and coherence of generated summaries. These metrics enable researchers and developers to compare and improve their summarization models, fostering advancements in the field of natural language processing. However, it is important to acknowledge that these metrics have limitations, such as their dependency on the reference summaries and their inability to assess the semantic understanding and informativeness of the summaries.</p><p>Extractive summarization techniques hold immense potential in tackling the information overload challenge. ROGUE-L, ROGUE-1, and ROGUE-2 are valuable evaluation metrics that shed light on the quality, precision, recall, and coherence of extractive summaries. By utilizing these metrics, researchers and developers can fine-tune their algorithms, contributing to the evolution of text summarization systems.</p><p>In the case of our results, we ended up choosing the ROGUE-L metric as this metric led to the highest score at about 0.5628. As the generated summary is compared with the reference summary already provided in the second column of the BIll-SUM data set, we found out that using this metric in which in-sequence matches that reflect sentence level word order are utilized has led to the highest percentage as it exceeds the 0.5 mark. However, it is evident that there is a certain limit to how well extractive summarization works as the score is not as high as we ideally want our model to be.</p><h3><strong>Abstractive:</strong></h3><p>Abstractive Summarization is a summarization technique that is similar to paraphrasing. Unlike extractive summarization, extraction summarization attempts to provide a summary using new and contextual phrases.</p><h4><strong>Seq2Seq</strong></h4><p>One of the most commonly used abstractive text summarization models is the Sequence to Sequence model (Seq2Seq). It is a RNN (recurrent neural network) model, which is a specific class of neural networks. What differentiates RNNs from regular neural networks is that prior inputs can influence the current output. For example, consider the following text:</p><blockquote>“President Roosevelt was one of the most influential presidents in American history. However, Roosevelt Street in Manhattan was not named after him.”</blockquote><p>In the first sentence, Roosevelt should be labeled as a person entity, but in the second sentence it should be labeled as a street name or a location. Knowing these distinctions is not possible without taking into account the words before them (“President”), and after them (“Street”). <em>For more information, please refer to: </em><a href="https://www.telusinternational.com/insights/ai-data/article/difference-between-cnn-and-rnn"><em>https://www.telusinternational.com/insights/ai-data/article/difference-between-cnn-and-rnn</em></a></p><p>Seq2Seq models use an encoder-decoder architecture, where it takes in a sequence of items and outputs another sequence of items (in our case, it would be a sequence of words). Using an encoder-decoder system solves the problem of having different lengths of both the input and output sequences. Let’s explain this encoder-decoder architecture by breaking it down into two parts.</p><p>First there is the <strong>training phase</strong>, where we train the model to acquire a target sequence. The encoder reads each word in the given sequence one by one at each timestep. At each timestep, the encoder captures the contextual information of the word in the input sequence. The decoder then takes in the ‘encoded’ sequence by each word, and predicts the same sequence but offset by one timestep. This means that the decoder is <strong>predicting the next word in the sequence</strong> given the previous word.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YisJEj45m2tzmqecvqUTBA.png" /><figcaption>Image: <a href="https://towardsdatascience.com/sequence-to-sequence-model-introduction-and-concepts-44d9b41cd42d">https://towardsdatascience.com/sequence-to-sequence-model-introduction-and-concepts-44d9b41cd42d</a></figcaption></figure><p>The next part is the <strong>inference phase, </strong>where we test the trained model by inputting new sequences without knowing what our target sequence is. Based on the information it has gathered during the training phase, the model will predict what the target sequence is, which in our case would be the summary of our input text. This is possible because we can specify a maximum length of the output sequence.</p><h4><strong>Transformers</strong></h4><p>Another method of abstractive text summarization we explored was the usage of <strong>transformers:</strong></p><p>Over the recent years, the most popular and efficient models for abstractive summarization are transformers. Some notable examples include BERT (Bidirectional Encoder Representations from Transformers) and GPT (Generative Pre-Trained Transformer) which are all the craze these days. Transformer models aim to solve sequence-to-sequence tasks while being able to handle long range dependencies. They mainly differ from other deep learning models by its use of self-attention. Similar to how it sounds, self-attention is the ability to devote more focus to parts of the data while diminishing the importance of other parts. In our case with abstractive summarization, the model is attentive to the main idea of the sentence and generates phrases and words surrounding it. For example, if the phrase imputed was “Justin ate a turkey leg and it was delicious.”, the model will be smart enough to relate “it” to the turkey leg. The model we will be using for our summarization is called PEGASUS (Pre-training with Extracted Gap-sentences for Abstractive Summarization).</p><p><strong>PEGASUS</strong></p><p>In the model, the pre-training task uses Gap Sentences Generation and Masked Language Modeling where we mask out specific tokens and have the model predict the missing tokens as well as whole sentences and have the model fill those missing sentences and tokens.</p><p>The results are shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z8Z2xtjFR10Za9tZV53BRw.png" /></figure><h3><strong>Conclusion</strong></h3><p>As you can see, extractive summarization definitely has its limitations. Since we are not generating new text and are instead relying on the pre-existing sentences, extractive summarization requires the “input” text to be relatively clean and already make a decent amount of sense. Practically speaking, extractive summarization should not be used to take a large piece of confusing text and try to generate a summary that is cohesive and comprehensible. As a result, it is clear to see some of the struggles that extractive summarization has with the bills dataset, a dataset that is not necessarily the cleanest or easiest to understand. On the other hand, it works amazingly with the article dataset, and can allow the user to find the most important parts of the article in a quick summary.</p><p>With abstractive summarization, you can definitely see a clear difference in performance from extractive. Utilizing deep learning techniques to develop a summary by “predicting” the best next word for a summary is the clear-cut best method of text summarization — but is it always necessary? Abstractive summarization requires a lot of computing power, and isn’t always needed for simple summarization problems. It is up for the user to decide which method they would like to deploy in their own text summarization problems, considering different factors such as computing power, importance of cohesiveness, and overall “pretiness” of the summary.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=42a4e31b453f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to Web Scraping!]]></title>
            <link>https://medium.com/@ucladsu/introduction-to-web-scraping-4d5add4391bb?source=rss-e0043e83850a------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d5add4391bb</guid>
            <category><![CDATA[beautifulsoup]]></category>
            <category><![CDATA[selenium]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[scrapy]]></category>
            <category><![CDATA[web-scraping-tools]]></category>
            <dc:creator><![CDATA[UCLA Data Science Union]]></dc:creator>
            <pubDate>Thu, 09 Mar 2023 01:01:45 GMT</pubDate>
            <atom:updated>2023-03-09T01:01:45.575Z</atom:updated>
            <content:encoded><![CDATA[<p>Web scraping is a critical skill for data scientists due to its ability to collect data from various sources on the internet, which would otherwise be difficult or impossible to obtain. With web scraping tools, data scientists can extract and analyze data to identify patterns, trends, and insights that may not be apparent from a single source. Web scraping also facilitates data cleaning and preprocessing, enabling data scientists to organize and structure data in a way that is suitable for analysis. Furthermore, web scraping can be used to enrich existing datasets by collecting additional data from the web. Automation is another key benefit of web scraping, allowing data scientists to collect data at regular intervals and track changes over time. Ultimately, web scraping is a powerful tool that enables data scientists to collect, explore, enrich, and automate data from a variety of sources, making it an essential skill for any data science project.</p><p>We have put together a collection of resources to help people get started with web scraping, including popular tools such as Beautiful Soup, Scrapy, and Selenium. These tools offer different advantages and use cases, and it’s important to understand their pros and cons before deciding which one to use for a particular project. Beautiful Soup is a Python library that is easy to use and suitable for small to medium-sized scraping tasks. Scrapy, on the other hand, is a more advanced web scraping framework that is better suited for larger and more complex projects. Selenium is a tool that can be used to automate web browsers and interact with dynamic web pages, making it ideal for scraping data from sites with heavy JavaScript usage. Each tool has its own strengths and weaknesses, and we have outlined these in our resources to help users choose the right tool for their specific needs. The resources we have compiled are available on our Github page, and we encourage anyone interested in web scraping to check them out.</p><p><a href="https://github.com/the-data-science-union/DSU-W2023-Web-Scraping">https://github.com/the-data-science-union/DSU-W2023-Web-Scraping</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IPynVtowJ-qO3pCe2VHBOw.jpeg" /><figcaption><a href="https://oxylabs.io/blog/beautiful-soup-parsing-tutorial">https://oxylabs.io/blog/beautiful-soup-parsing-tutorial</a></figcaption></figure><p><strong>Beautifulsoup</strong></p><p>Beautiful soup is a python package library used to pull HTML or XML files from the web. It is essentially a toolkit for dissecting a document and extracting specific information you need, such as extracting texts from the html tags. It does not take much coding to write an application. It’s perfect for beginners and a starting point to learn about web scraping! Beautifulsoup is an easy web-scraping tool in python compared to Scrapy and Selenium. However, Beautifulsoup does not have the ability to scrape javascript-driven websites and needs another web scraping tool. This web scraping tool also requires dependencies or packages to send requests to websites for content. This can cause Beautiful to run slower compared to other web scraping programs. Overall, Beautifulsoup is a great and easy web scraping tool to familiarize with first before getting into complex web scraping tools.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_d-rwtSIGG7hmxj4FGfQEA.jpeg" /><figcaption><a href="https://const.fr/blog/tech/scrapy-how-to-crawl-pages/">https://const.fr/blog/tech/scrapy-how-to-crawl-pages/</a></figcaption></figure><p><strong>Scrapy</strong></p><p>Scrapy is a Python package used for web-crawling to effectively extract and process data from websites and store them in a structured data format. With Scrapy, you create your own “Spider” which is a class you define with rules to help scrape websites and extract the data you want. It is especially useful for scraping large sets of data as Scrapy is much faster than most other web scraping tools. Since you can define your own “Spider” to crawl the web, Scrapy has advantages in creating complex web crawling and scraping. Scrapy is best suited for static websites, and is more beneficial to use than BeautifulSoup when scraping large datasets. It also consumes less CPU. However, it has a <strong>steep learning curve </strong>and is not particularly beginner friendly. But, if you do learn to design spiders, Scrapy can be one of the most effective web scraping tools. Scrapy would be better suited if you already know some HTML, understand what a shell is, are comfortable using Terminal, and know how to write a class without using Google. When using Scrapy, to filter through data, using selectors is necessary (html tags for different sections of a website). SelectorGadget is recommended to identify selectors, as it is more efficient and less confusing. In addition, compared to the other two scraping methods listed, Scrapy is not particularly notebook friendly. It is recommended to use PyCharm and Terminal. Overall, if you are confident in HTML and your understanding of code, and are looking for a challenge/have a complex web-scraping project, Scrapy could be beneficial.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NO6d1BsPWWmZHeAXWxjGEA.png" /><figcaption><a href="https://www.scrapingbee.com/blog/selenium-python/">https://www.scrapingbee.com/blog/selenium-python/</a></figcaption></figure><p><strong>Selenium</strong></p><p>Selenium is a Python package that is used primarily for website testing and automation, though its capabilities also make it a useful web scraping tool. Using Selenium, you can simulate a user’s experience and interact with live websites, performing actions such as clicking buttons, selecting options from dropdown menus, typing entries into search bars, and more. These actions make Selenium a good choice to scrape dynamic websites, or interactive sites where the content or data displayed is dependent on user actions and other factors such as time and date or user demographics. It is also a very flexible tool capable of scraping a wide variety of websites using many different browsers and platforms (i.e. Chrome, Firefox, Safari). However, because Selenium is primarily a website automation tool, using it for web scraping requires some background knowledge of HTML code and web development. In addition, Selenium is the slowest out of the three main Python web scraping packages. The best use case for Selenium is scraping data from a dynamic website that requires clicking buttons or other interactive elements to access the data. For static websites and larger web scraping projects though, BeautifulSoup and Scrapy are probably more efficient and effective options.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d5add4391bb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Enhancing Spotify Playlists using their Audio Features with Classical and Deep Learning Methods]]></title>
            <link>https://medium.com/@ucladsu/enhancing-spotify-playlists-using-their-audio-features-with-classical-and-deep-learning-methods-f8c0e7e878cf?source=rss-e0043e83850a------2</link>
            <guid isPermaLink="false">https://medium.com/p/f8c0e7e878cf</guid>
            <category><![CDATA[spotify]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[UCLA Data Science Union]]></dc:creator>
            <pubDate>Thu, 09 Mar 2023 00:40:48 GMT</pubDate>
            <atom:updated>2023-03-09T00:51:09.655Z</atom:updated>
            <content:encoded><![CDATA[<p>Project Leads<em>: Lukas Hager, Ved Phadke </em>&amp; Project Members:<em> Justin Gong, Chloe Li, Anish Ravilla, Ethan Warren</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*i1M42WWlwRSPq4lU" /><figcaption>Spotify Logo</figcaption></figure><p><strong>Introduction</strong></p><p>In today’s music streaming landscape, platforms like Spotify are becoming increasingly relevant. Part of what makes Spotify such a dominant platform in the space is the recommendation algorithm it uses. Hundreds of millions of users on the platform use the recommendation algorithm to both expand their tastes and get more of what they love. Using data on over a million songs and their features, as well as thousands of playlists, we were able to develop two distinct algorithms to develop our own recommendation engine. The two models we made utilize two very different approaches; one is a classic KNN clustering algorithm, while the other is a deep learning approach leveraging a neural network.</p><h3>KNN Model</h3><p><strong>Data Gathering &amp; API Usage</strong></p><p>In order to provide a cohesive recommendation system, we would need a lot of data — data consisting of features from millions of songs — from Spotify, so that we can train our models on these songs to provide accurate recommendations. So we turned to an API to collect this data for us. The Spotipy API is a powerful tool for accessing data from the Spotify music streaming service. With the Spotipy API, you can extract detailed information about tracks, albums, artists, and playlists. This data can be used to create datasets that are useful for a wide range of purposes, including music analysis and recommendation systems.</p><p>​​Step 1: Installing Spotipy</p><p>The first step in using the Spotipy API is to install the Spotipy library. You can do this using pip, the Python package manager, by running the following command in your terminal:</p><pre>pip install spotipy</pre><p>Step 2: Creating a Spotify Developer Account</p><p>To use the Spotipy API, you need to have a Spotify developer account. You can create one by going to the Spotify Developer Dashboard and clicking on the “Log in” button. Once you are logged in, you can create a new app by clicking on the “Create an App” button.</p><p>Step 3: Obtaining Spotify API Credentials</p><p>After creating an app, you need to obtain your Spotify API credentials, which include the client ID and client secret. You can find these credentials on your app dashboard. Make sure to keep these credentials secret, as they allow access to your Spotify account.</p><p>Step 4: Setting Up Spotipy</p><p>Now that you have Spotipy installed and your Spotify API credentials, you can start using the Spotipy API. To do this, you need to import the Spotipy library and authenticate with your API credentials. Here is an example of how to do this:</p><pre>import spotipy<br>from spotipy.oauth2 import SpotifyClientCredentials<br>client_id = &quot;your_client_id_here&quot;<br>client_secret = &quot;your_client_secret_here&quot;<br>client_credentials_manager = SpotifyClientCredentials(client_id=client_id, client_secret=client_secret)<br>sp = spotipy.Spotify(client_credentials_manager=client_credentials_manager)</pre><p>Step 5: Extracting Data</p><p>With Spotipy set up, you can now start extracting data from Spotify. Spotipy has a wide range of methods for extracting data, including methods for getting track information, album information, artist information, and more. Here is an example of how to extract track information:</p><pre>results = sp.search(q=year:2022, type=’track’, limit=50)<br>tracks = results[&#39;tracks&#39;][&#39;items&#39;]<br>for track in tracks:<br>  print(track[&#39;name&#39;], track[&#39;album&#39;][&#39;name&#39;], track[&#39;artists&#39;][0][&#39;name&#39;])<br></pre><p>This code searches for the top 50 tracks from 2022 and prints out the track name, album name, and artist name for each track. From there you can store this information and write it to a CSV file for future reference.</p><p>Our initial objective was to use a 1.2 million song dataset we found on Kaggle, and append many dataframes of songs from a variety of artists, from 2021 to the present. However, due to limitations with both the Spotify Web API and the sheer amount of music that has been published within the given time frame, it was very difficult to add to this dataset, so we opted to just use the data from the 1.2 million song dataset and train our models on this data. This dataset was also compiled using the Spotipy API, so we still made use of the features that were included in this, with the intention of standardizing and vectorizing these features to see how similar songs from this dataset are compared to songs from a playlist the user inputs. We’ll go into this process in more detail with our discussion of the models we created.</p><p><strong>Model Building</strong></p><p>With a completed dataset to retrieve songs, we decided to approach song recommendations through two methods: classical machine learning and neural networks. The first approach was through classical machine learning by leveraging the KNN (k-Nearest Neighbors) algorithm. The KNN algorithm is oftentime utilized in regression or classification models, where it performs distance calculations to find the <em>k </em>nearest data points to the input in high-dimensional space. It then utilizes a metric to aggregate the <em>k </em>nearest data points to predict the desired response variable for the input. However, for this model, we will not be aggregating the data, and instead, use the <em>k </em>nearest neighbors as potential recommendations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/591/0*lw3kFG8A11_qWyUD" /><figcaption>Credit: <a href="https://towardsdatascience.com/knn-visualization-in-just-13-lines-of-code-32820d72c6b6">https://towardsdatascience.com/knn-visualization-in-just-13-lines-of-code-32820d72c6b6</a></figcaption></figure><p>Before creating the model, we needed a pipeline that could convert a spotify playlist into a single vector of data that can be the input to the model. To achieve this, we used the Spotipy package again to retrieve song information from a playlist. The function takes in a spotify playlist link as input and outputs a vector in the same format as the reference dataset that the model will be trained on. The function first uses string manipulation to retrieve the playlist URI, and then uses spotipy to loop through each track in the playlist to find the audio features and genre(s). Below are lines of code that was used during this process.</p><pre>playlist_uri = link.split(&quot;?&quot;)[0].strip(&quot;https://open.spotify.com/playlist/&quot;)<br>    for track in sp.playlist(playlist_uri)[&quot;tracks&quot;][&quot;items&quot;]:<br>        output_dict[&quot;name&quot;].append(track[&#39;track&#39;][&#39;name&#39;])<br>        output_dict[&#39;artist&#39;].append(track[&#39;track&#39;][&#39;artists&#39;][0][&#39;name&#39;])<br>        output_dict[&#39;released_year&#39;].append(track[&#39;track&#39;][&#39;album&#39;][&quot;release_date&quot;].split(&#39;-&#39;)[0])<br>        <br>        song_uri = track[&#39;track&#39;][&#39;uri&#39;]<br>        filtered_dict = {k:v for k,v in sp.audio_features(song_uri)[0].items() if k in audio_keys}<br>        for audio_feature in filtered_dict.keys():<br>            output_dict[audio_feature].append(filtered_dict[audio_feature])</pre><p>This retrieves all the songs in the playlist and returns the necessary data.</p><pre>for genres in y[&#39;genres&#39;]:<br>                if genres not in output_dict[&quot;genres&quot;]:<br>                    output_dict[&quot;genres&quot;].append(genres)<br>for x in features: <br>        output_dict[x] = np.mean(df[x])</pre><p>This takes the dataframe from the previous function call and finds the averages of every column and aggregates the genres.</p><p>Note: This is not all the code involved, but rather a shortened version of the function definitions to display the logic behind the pipeline.</p><p>Once we created a pipeline to convert a spotify playlist link into a vector of average data values, we needed to construct the kNN model by fitting it to the reference data. Since we did not follow a purely supervised approach, we did not split the data into train and test sets. This is because we are <em>not </em>predicting a response variable, and thus, do not need to test the accuracy of the results using a train test split method. Instead, we fit the kNN model to the entire dataset. This was performed using sci-kit learn’s Nearest Neighbors function. The model was trained to find the <strong>15</strong> closest neighbors using euclidean distance calculations and a brute-force algorithm. These were the parameters that resulted in the most accurate song recommendations. We also did not standardize the values before fitting it to the dataset, which will be further discussed in the next section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/0*zNME3vjjAPWHDhXH" /><figcaption><em>Plot of a subset of songs after clustering algorithm and dimensionality reduction</em></figcaption></figure><p>With the kNN model fully trained with our song bank, we were able to combine it with our pipeline to create predictions. We used the song vector as input to find the 15 songs in the song bank that have the most similar audio features and release year to the average song in a given playlist. With these 15 songs, we filtered out all songs that were not the same genre as any of the songs in the inputted playlist.</p><p>With the remaining songs, we calculated a recommendation score for each of them. This score was calculated by measuring how close each song was to the inputted vector, using the distance outputs of the kNN model. Additionally, we adjusted the weight of the recommendation score based on which features were more important (i.e. released year weighed less). Thus, this allowed us to create a model that takes a spotify playlist link as input and outputs a list of recommended songs and its relative recommendation scores.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SIBiwz1fsUVPGLjP" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-9z0SFCvR7not0sK" /></figure><p><strong>Limitations and Areas for Improvement</strong></p><p>In order to extract a good/usable recommendation from this algorithm, it is all but necessary for the playlist to have a central theme or idea. Playlists which are extremely varied and different are difficult for the algorithm to categorize, and as such do not generate the best results. With using a classical approach and a clustering algorithm, especially on a data set this vast, there are of course concessions we had to make. The algorithm’s focus on indexing by genre is not as clean as we would have hoped, and we can chalk this up to both the computationally difficult process of obtaining genre information by song with the API and the mislabeling of some songs’ genre information within our dataset. In an ideal scenario, we would have liked to build this dataset ourselves from the ground up, extracting only the essential features, then perhaps applying one-hot encoding to genres such as to quantify the relative frequencies and relationships between different genres for songs. This would also allow us to recommend songs that were released post-2021.</p><h3>Deep Learning Model</h3><p><strong>Predicting Song Qualities</strong></p><p>For our second recommendation algorithm, we wanted to create a model that looks at the audio features of a given playlist and then recommends another song with similar characteristics. Using TensorFlow, this is very easy to implement with a neural network. We simply take a sample of twenty songs from a playlist and train a neural network to guess a different hidden song that was also taken from the same playlist. By training over many playlists, we should be able to teach our model the general structure of what falls in a playlist!</p><p><strong>Data Gathering</strong></p><p>The challenge in building a model this way is that you need a good amount of data. In our case, we needed user-generated playlists, as well as the audio features of the songs within those playlists. Thankfully, Spotify released a dataset containing a million playlists back in 2018 <a href="https://www.aicrowd.com/challenges/spotify-million-playlist-dataset-challenge">here</a> as part of the Million Playlist Dataset Challenge. While this dataset is 5 years old, our model only looks at the high-level audio features of a given playlist. We can also presume that people have not significantly changed their playlist habits in the past five years.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yLcyvjF3FNwaYHbU" /></figure><p>For a given playlist, we needed the audio features of our 20 “sample” tracks and our 1 ‘hidden” track. In theory, we could have been able to just iterate through our Million Playlist dataset and find the audio features from Spotify.</p><p>This turned out to be by far the most significant hurdle we faced. We used a song’s URI to get its audio features directly from the Spotify API. We would then use a modified sigmoid function to normalize the features between -1 and 1. Unfortunately, each song look-up ended up taking about a tenth of a second and each playlist look-up took about 2 seconds. That meant that it took about 15 hours of data scraping time to get our final set of playlists. However, this also does not account for the further issues we ran into such as songs being taken down from Spotify and the general unreliability of the API which repeatedly took our scraping program offline. Thankfully, after many hours of runtime, we were able to get the audio features of 27,000 playlists. While this amount is far from ideal, it is certainly enough to build our model.</p><p><strong>Model Building</strong></p><p>Neural networks mimic the human brain by creating and weaving together layers of neurons. The neurons communicate with each other by receiving inputs which are then multiplied by randomly assigned weights. The neurons apply an activation function on the sum of the weighted inputs, before passing the output to the next layer of neurons. Over time, the model adjusts the weights and the network learns to minimize the difference between its predicted output and the actual output. Thankfully, using Tensorflow makes training a neural network extremely easy with the Keras API.</p><p>Before going into how our neural network model works, it is important to discuss the way our data is set up. We gathered around 30 playlists resulting in a total of 27,201 songs. For each playlist, one song is hidden so that the model can learn from the remaining songs to guess the hidden song. Our model ended up consisting of three layers: two dense layers of 200 neurons with a ReLU activation function, and one dense layer of 11 neurons with a hyperbolic tangent (tanh) activation function. We also incorporated a dropout layer after each dense layer with a dropout rate of 0.3. Dropout randomly removes 30% of the neurons during training, preventing any one neuron from becoming too specialized and therefore preventing overfitting. Overfitting occurs when the model learns the training data too well and does not perform well on new data.</p><p>For our neural network model, we used the adam optimizer and mean squared error loss function for training, the sigmoid function for feature scaling, and cosine similarity for similarity metrics.</p><p>The model’s performance can be improved by experimenting with different hyperparameters such as the number of neurons in each layer, dropout rate, activation functions, loss functions, and evaluation metrics.</p><p><strong>Parameter Tuning</strong></p><p>After we’d built the structure of our model, we had to optimize and tune the hyperparameters. Hyperparameters refer to the parameters of a machine learning model that are not learned from the data but are instead set by the user. Fine-tuning these hyperparameters is crucial to optimizing our model’s accuracy and efficiency. We chose to look at the cosine similarity between the predicted song and playlist vectors and we kept this metric in mind as our goal to optimize as we tuned the model.</p><p>One of the key hyperparameters we fine-tuned was the number of hidden layers and the number of neurons in each layer, which control the capacity of the model to learn complex relationships about the data. We had to strike a balance between having enough layers to capture these complex representations in the song features data, and not having too many layers that might lead to overfitting. Overfitting is when the model is trained too well on the training data, so it doesn’t generalize well to new unseen data. After experimenting with different configurations, we settled on a two-layer model with 200 neurons in each layer. This configuration provided a good balance between model complexity and generalization performance, and we found that adding more layers did not significantly improve performance. Additionally, by keeping our model simpler, training was less computationally expensive.</p><p>Next we experimented with different activation functions for each layer, including ReLU, tanh, and sigmoid. ReLU performed the best for the first two layers, while tanh was used in the output layer. The choice of activation function can significantly impact the model’s ability to learn complex patterns in the data.</p><p>To avoid the problem of our model potentially overfitting to the training data, we introduced a dropout rate of 0.3. This parameter randomly drops out some neurons during training, forcing the model to learn more robust features of the dataset. A higher dropout rate can lead to more regularization but may also harm the model’s performance, and we found that 0.3 was ideal for the song prediction model.</p><p>As for our loss function and evaluation metric, we focused on the mean squared error for loss and cosine similarity between latent representations of song and playlist vectors. Cosine similarity measures the cosine of the angle between two vectors in high-dimensional space with higher scores corresponding to more similar songs. One of the benefits of using cosine similarity is that it is scale-invariant, meaning that it is not affected by the magnitude of the vectors being compared, which is particularly useful when working with high-dimensional data like our Spotify playlist data. Ultimately, cosine similarity gave us a framework for determining which songs were similar within our latent representations.</p><p>The last thing for us to decide is how many epochs to train the data on. In general, the number of epochs to train a model on is a tradeoff between computational resources and performance. Training a model for too many epochs can lead to overfitting and wasted computational resources, while training a model for too few epochs can result in poor performance. In our project, we found that training the model for around 50 epochs provided a good balance between computational resources and performance. Here is a look at our loss function and cosine similarity after the first 10 epochs of training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*6vJtK4U9M7T79ALL" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*WqWVOeSRW3sxRXfA" /></figure><p>In the end, we were able to get our model to around .86 testing accuracy for cosine similarity, with the chosen hyperparameters, which we were very happy with.</p><p><strong>Conclusion</strong></p><p>Our methods used very different methods, yet they both converged into fundamentally decent song recommendation models. Our KNN model was able to find a cluster of songs related to any given playlist and our deep learning model was able to learn the structure of playlists and use that to predict new songs.</p><p>Beyond the parameter tuning, we subjected both of our models to extensive testing by plugging many playlists supplied by us and our friends into each model. Our results, though anecdotal, were illustrative of the relative strengths of each model. They were both generally able to supply songs that fit the general “vibe” of a given playlist. However, the KNN model struggles when given playlists that have a large variety within their types of music. The Deep Learning model, on the other hand, is completely blind to genre information and cannot tell the difference between certain types of music. For example, it struggles to distinguish between Mariachi and Hip-Hop and also between Christian Rock and Alt Rock. Both models also only recommend songs released before 2021.</p><p>However, we are all very pleased with our results. While our models can be improved with a few simple tweaks, such as incorporating genre information more naturally and obtaining a more modern song output, these are not fundamental changes. Overall, we have built two modern and effective recommendation algorithms that can guide listeners to new and previously unknown songs.</p><p><strong>Data Sources</strong></p><ul><li><a href="https://www.kaggle.com/datasets/rodolfofigueroa/spotify-12m-songs">1.2 million song dataset</a></li><li><a href="https://www.aicrowd.com/challenges/spotify-million-playlist-dataset-challenge">Million Playlist Dataset Challenge</a></li></ul><p><strong>Github</strong></p><p><a href="https://github.com/ved-phadke/spotifyplaylistenhancer">https://github.com/ved-phadke/spotifyplaylistenhancer</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f8c0e7e878cf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Predicting Plane Ticket Prices]]></title>
            <link>https://medium.com/@ucladsu/predicting-plane-ticket-prices-ab1be41f8ad0?source=rss-e0043e83850a------2</link>
            <guid isPermaLink="false">https://medium.com/p/ab1be41f8ad0</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[flight]]></category>
            <category><![CDATA[exploratory-data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[UCLA Data Science Union]]></dc:creator>
            <pubDate>Wed, 08 Mar 2023 20:57:39 GMT</pubDate>
            <atom:updated>2023-03-08T20:57:39.325Z</atom:updated>
            <content:encoded><![CDATA[<p>Project Leads: Tony Lei, Hairan Liang</p><p>Project Members: Alyssa Liu, Kevin Hamakawa, Kaitlyn Li, Tiffany Le, Wyan Wan</p><p><strong>Introduction</strong></p><p>Through this project, we intend to provide the average passenger guidance on purchasing the most financially reasonable flight ticket based on timing, airlines, destination, and other relevant factors. Using a decision tree regressor, we were able to predict future single plane ticket prices trained with the Kaggle dataset <a href="https://www.kaggle.com/datasets/dilwong/flightprices">Flight Prices</a>, which contains information on one-way flights found on Expedia between 2022–04–16 and 2022–10–05.</p><p>In order to properly conduct our analysis/modeling, we made the data more manageable by filtering the dataset to flights departing from LAX. Furthermore, we cut the data roughly in half by splitting the flights up into two categories: one with flights that had one or more stops and the other with flights that were nonstop. Due to the simplicity of working with the nonstop flight data, we performed a majority of our modeling with this specific dataset.</p><p><strong>Description of Data Set / Data Cleaning</strong></p><p>Our dataset had 27 different columns containing variables including flight date, destination airport, total fare, arrival time, and distance.</p><p>Before starting our EDA and modeling, the data set required additional cleaning to make it more applicable to our overall goal. In particular, we established two new columns, “timeofday” and “timedifference,” that grouped the set’s existing time data (i.g. dates, hours, minutes, etc.) through the use of Python functions and regex patterns. The “timeofday” column returned either “morning”, “afternoon”, or “evening”, based on the time of day, while the “timedifference” column provided the time difference between the flight date and the date in which the price was listed. These new columns provided further insight on our data set that would be relevant to our findings and assist us in building a more comprehensive model.</p><p><strong>EDA</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/774/0*mx8FZtby_WSRVXw0" /></figure><p>We found that the day of the week the flight was scheduled on did have an impact on the average ticket prices. Flights on the second half of the week (Thursday, Friday, Saturday, and Sunday) had higher average prices than earlier in the week, which could mean there is a higher demand for flights later in the week since people might go on weekend trips. It is important to note that this is the case for our dataset, but our dataset may not represent general flight price patterns. Nonetheless, <a href="https://blog.google/products/travel/how-to-find-the-best-deal-on-your-next-flight/">Google’s data</a> also concurs that Monday, Tuesday, and Wednesday flights are generally cheaper than weekend flights.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*quS0yYrQV2lsgKbE" /></figure><p>Fares fluctuate over the months due to travel seasons. The peak in fare prices during July reflects the demand for traveling during summer months by students and parents with children. Considering this demographic makes up most of the US population, travel is in high season and thus, warrants a premium. The same should be expected during December — winter break — but due to the severe winter storms, airlines canceled more than 12,000 flights which contributed to an unusual dip in average fare during the winter season. Our data modeling was consequently impacted by this unprecedented situation, so we must proceed with caution when training our models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*r5ZRPyMzxSo7R4WT" /></figure><p>Considering our data set includes seven distinct airlines, we decided to create a box and whisker plot to observe individual ticket prices issued by each airline. We originally believed that the prices should be similar across airlines, but this graph disputes this idea. There is a large variation in fare pricing for the different airlines. Delta has the highest range in pricing while Frontier Airlines has the lowest range. American Airlines has significantly more extreme outliers, which indicates that prices are much higher than its median. These differences in variation between airlines may be due to the fact that certain airlines offer more luxury seats and upgrades or offer a greater variety of flights . Therefore, based on this observation — airlines having varying pricing schemes — we believe that the specific Airline companies would be an integral variable in building our model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Fd-sWes7KS5KfrSk" /></figure><p>The graph above displays the average fare of ticket prices grouped by regions and holidays. As our data set included international airports across the United States, such as San Francisco, Boston Logan, and John F. Kennedy, we decided to categorize them by their location to avoid overgeneralizing our averages. With that in mind, regions farthest from LA have higher fares due to the longer distance and duration of the flight across all holidays. When comparing holidays, Juneteenth and Patriot’s Day have the highest average fares, likely due to certain states allowing professionals and students the day off. Considering the fluctuations in average fare across holidays, we believed that it would be a valuable attribute for us to consider in our model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uFxrOFG2nVvoXSgC" /></figure><p>We were also interested in the ideal time to book a flight, so we graphed the days left for departure versus ticket price. From the graph above, we can see that the price of flight tickets is highest within 10 days before the flight departure date. Between 10–45 days prior to the flight departure date, the average price of the flights generally declines. However, flights with a departure date of 45 days or more in advance are priced higher.</p><p><strong>Description of Models</strong></p><p>We spent a lot of time selecting different regression models, such as linear regression, XGBoost, ridge regression, lasso regression, and random forest regression. Ultimately, after performing hyperparameter tuning using cross validation, we settled on using our random forest regression model, which performed with an R-Squared value of 0.93 and an RMSE of 55.89.</p><p><strong>Findings</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uc9R1LakephVWSts" /></figure><p>We graphed the feature importances of each column for our random forest regression model.</p><p>From the graph above, we found that segmentsDurationinSeconds, class_label, flightMonth, flightDay, date_difference, and seatsRemaining were the most important variables in determining total fare using our model.</p><p><strong>Conclusion/Limitations:</strong></p><p>Overall the most important factors in our model were segmentsDurationinSeconds, class_label, flightMonth, flightDay, date_difference, and seatsRemaining. Due to the substantial size of the dataset, we were not able to try out as many models as we would have liked. We could have experimented on a smaller subset of the data to reduce fitting time during the model selection and tuning process. Furthermore, we could have also looked at more starting airports other than LAX, and even analyzed flights that had more than one segment. The dataset itself also contained several limitations with the actual dates of the flights, as the weather across the U.S. during December of 2022 led to cancellation of a lot of flights. The data also only included flights from 2022, so we could have looked at other years.</p><p>To conclude, through cleaning and exploring our dataset, we were able to discover meaningful insights into key variables that could impact the price of a flight. We used this information to build our model, which we also had to optimize by trying out several different ones and tuning the hyperparameters. Our Random Forest Regressor model performed quite well, but there is definitely still room to grow and we hope to not only improve our model by testing it on other datasets but eventually deploy it through a web app so that people can save money and optimize their flight booking process.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ab1be41f8ad0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Analyzing Gaming Trends with Stream Titles and Viewer Statistics]]></title>
            <link>https://medium.com/@ucladsu/analyzing-gaming-trends-with-stream-titles-and-viewer-statistics-4a6bc17385e5?source=rss-e0043e83850a------2</link>
            <guid isPermaLink="false">https://medium.com/p/4a6bc17385e5</guid>
            <dc:creator><![CDATA[UCLA Data Science Union]]></dc:creator>
            <pubDate>Thu, 01 Dec 2022 03:40:14 GMT</pubDate>
            <atom:updated>2022-12-01T03:40:14.204Z</atom:updated>
            <content:encoded><![CDATA[<p>Project Leads<em>: Chelsea Chen, Madeline Kim </em>&amp; Project Members:<em> Grant Dufek, Paul Schumacher, Nicole Yamachika</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QkMflyeDEcXowT1J.jpg" /></figure><p><strong>Introduction</strong></p><p>With the rise of live streaming around the world, the platform Twitch has become increasingly relevant in today’s culture. On Twitch, streamers can earn money by recording themselves live as viewers interact through text chats. Much of the platform’s popularity lies in gaming streams, where streamers will film themselves playing video games. As different video games become more or less popular, Twitch content changes to reflect the emerging trends and patterns of the gaming community.</p><p>Using data on specific video game streams containing information on views, stream popularity, and stream titles, we examine the data to find the popularity trends for games and the impact of words in stream titles on views.</p><p><strong>Datasets and General Exploratory Data Analysis</strong></p><p>The first dataset we examined was pulled from <a href="https://www.kaggle.com/datasets/rankirsh/evolution-of-top-games-on-twitch">Kaggle</a> and contains data on the top video games streamed on Twitch from 2016 to 2021. Data on various games are categorized and sorted by the month and year that the data was collected, and includes the number of hours each game was watched and streamed on Twitch. In addition, it included the peak viewers and channels, which measured the maximum number of viewers/channels that watched or streamed the game in the specified month. It also showed the total number of streamers and the average number of viewers of the game for the month.</p><p>The dataset included data on viewership including peak viewership. Peak viewers ranged from 441 viewers to 3,123,208 viewers in this dataset. Average viewership ranged from 120 viewers to 479,209. Viewer retention is an incredibly important metric for livestream popularity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/403/0*ku30Rywv-jjXu5O1" /></figure><p>The remaining datasets that were used for the project were manually pulled from the <a href="https://dev.twitch.tv/docs/api/">Twitch API</a>. Using the package<a href="https://github.com/Freguglia/rTwitchAPI/blob/master/README.md"> rTwtichAPI</a> in R, information on the top streams of select individual video games were pulled from the Twitch API. The games were <em>Alien: Isolation, Chess, CyberPunk, Genshin Impact, League of Legends, Grand Theft Auto V, Valorant, Virtual Casino, World of Warcraft</em>, and <em>Just Chatting </em>(a game category for streamers livestreaming things other than games). The data was pulled manually for each game at least once a day for over a week, and then combined to create one large dataset for each game. The columns included the popularity ranks of each stream at the time of the data collection, information on the user and games (ID numbers, names, and user logins), as well as the stream titles, viewer count, start time, and maturity ratings (over and under 18). The thumbnail url and tag ids were included as well, and all of the data was collected on streams that were in the English language category.</p><p>For EDA, we looked at the ten games that a) had the most streamers, b) were most often streamed and c) had the biggest audience. By a considerable margin, the game Fortnite has the most significant total number of streamers over the period from 2016 to 2021 with 28,018,115 streaming accounts. This number becomes even more outstanding because streamers for Fortnite did not appear in the data set until the middle of 2017. Afterwards, the number of Fortnite streamers dramatically increased and peaked in December 2018 at 1,013,029 monthly streamers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/0*sCIGZ12fbgr8MPw8" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/418/0*CSPJRp25rRkrnL1J" /></figure><p><em>Top 10 games with the most </em><strong><em>streamers</em></strong><em> in total and over time</em></p><p>A similar picture appears while analyzing the top ten games regarding the number of total hours streamed. As in the previous graph, the most streamed game is Fortnite, followed by League of Legends. The third place in terms of streaming hours goes to Call of Duty: Warzone, even though this game is ranked only in 9th place in the number of streamers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/0*0hMPg_zmAGdOOdkM" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/418/0*ezNHQ4AL_zxlaZDP" /></figure><p><em>Top 10 most </em><strong><em>streamed</em></strong><em> games in total and over time</em></p><p>Surprisingly, Fortnite (3,98 bn. watchers) is not the most viewed game of the data set. While looking at the development of hours watched, we recognized the rise of the game Just Chatting from 37,737 hours in September 2018 to 3,544,601 in April 2021. This title goes to League of Legends with 6,34 bn. accounts following the streams.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/0*FUDeMdG89103Akr0" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/418/0*uwSVeqgZd2Hr8x3R" /></figure><p><em>Top 10 most </em><strong><em>watched</em></strong><em> games in total and over time</em></p><p>From the various game datasets collected, a select few were ultimately used for further exploration of individual game analysis.</p><p><strong>Advanced Exploratory Text Analysis — <em>Alien: Isolation</em></strong></p><p>The game <em>Alien: Isolation </em>data set contains 877 streams, whereas 871 of them have a title. Since there is no correlation between these titles, we dropped the untitled streams out of our data set. Of the remaining stream, 504 were mature, and 367 were not. All streams were displayed live and hosted in English.</p><p>Conducting an Advanced Exploratory Text Analysis, we divided the data into a train and test set, prepared the training corpus into one giant string, and split it by white space. We now ended up with 5799 total strings and 2835 unique strings, each representing a series of letters and/or symbols not separated by white spaces. The most common ones were | (192 appearances), Alien, (98), Isolation (94), — (73), to (66), the (65), with (49), a (47), and (44), Alien, (40). Analyzing the strings more in-depth, we specifically looked at strings containing a number, a hyphen connecting two words, special punctuation, only lowercase letters, only uppercase letters, the same character 3+ times in a row, and emojis. The results with a respective example can be found in the following table:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SMXFN5jqfYYiHIROxtxbvA.png" /></figure><p>To dive deeper into the content analyses, we apply Tokenization to extract the meaningful words before we apply word net-lemmatization to normalize each token (= reasonable expression) to its word stem. After using these functions, we end up with 1356 unique tokens, which vary in length (= letter count) as visualized in the following chart:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/386/0*FpKdWAj9LTdlPcra" /></figure><p>To understand which tokens were used the most in the streaming title, we removed 1528 tokens that were stop words, making up 28.38% of the tokens. This elimination of redundant tokens allowed us to analyze the most frequently used tokens as a single word (unigram), a pair (bigram), a triple (trigram), and as a quadruple (fourgram) are summarized in the following four charts:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/452/0*0QRgTl-mdbiVZjFY" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/0*7k-CkJqk_qAMD-dz" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/452/0*pMoCUPLcNsT52-V7" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/0*UnaKxg1Sv7S5ajwX" /></figure><p>Finally, we analyzed the number of sentences, tokens, charters and stop words per title on average. The results can be seen in the following table:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/0*XBrJtlllLkuZwlMt" /></figure><p>After the initial exploration of stream titles under <em>Alien: Isolation</em>, we then proceeded to do topic analysis.</p><p><strong>Topic Analysis — <em>Alien: Isolation</em></strong></p><p>We apply three topic analysis models to find patterns among the 871 titles. These models are Top2Vec, Latent Dirichlet Allocation (LDA), and Latent Semantic Analysis (LSA).</p><p>Starting with <strong>Top2Vec</strong> as an unsupervised learning method, we tried to cluster all the titles into general topics. Therefore, we put at first all the stream titles in a list and applied the Top2Vec model. It narrowed down all titles into three main topics/ categories, each consisting of respectively 339, 272, or 260 titles. The following table summarizes the characteristic words for each subgroup.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ylUu-rCwJ_F2otnGFNeIVA.png" /></figure><p>As we can see, it is hard to tell the difference between the topics since there is a lot of overlap of words between the topic groups because all the titles were chosen to describe the game Alien: isolation. In the following step, we use these characteristic words of the topic words to create a Topic Vector for each subgroup. Lastly, for each of the different Topic Vectors, we print the streaming titles, which in all their words, are closest to the respective topic vectors, as shown in the following graph.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mrgi3IyFbMJX1ijcNezsTg.png" /></figure><p>As a second topic analysis model, we chose <strong>Latent Dirichlet Allocation (LDA)</strong> to obtain relationships between multiple documents in a corpus and to find groups of titles with similar topics. After removing all punctuations, changing all uppercase to lowercase letters and removing all non-regular expressions and stop words, we further limit the selections of words to expressions that are represented in at least two titles and appear in a maximum of 95% of the titles. We then apply the <em>LatentDirichletAllocation</em> model to create four arrays of topics and words to find matching words characterizing each topic, as seen in the following table. Even though the topics are still similar to each other, the removal of stop words makes the results more differentiable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LWGgfy7r55-OazLLkZ79yg.png" /></figure><p>In the last step, we attach a topic number to every title in the original data set depending on matched words characterizing each topic.</p><p>Finally, we use<strong> Latent Semantic Analysis (LSA) </strong>to extract the relationship between different words in the titles. After removing stop words, we utilize the function TF-IDF Vectorization to create a matrix of titles and total words. In the following step, we decompose this matrix to find topics and respective matching words characterizing each topic. A pivotal difference to LDA is that we can define the number of topics before we run the model. We chose five topics and showed the top 7 words characterizing this topic in the following table. As pointed out before, it is difficult to differentiate between the concepts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wAV4NShKZ1jVJcbNy-xaCA.png" /></figure><p><strong>Topic Analysis — <em>Chess</em></strong></p><p>The individual chess data set contained data from 589 streams in the Chess category on Twitch over the span of 2 weeks. This data included stream titles along with viewership. A further investigation into topic analysis for stream titles in the category was undertaken. The first step was to process the titles by tokenizing the text along with removing stop words and punctuation. With the processed titles, a new dataframe could be created keeping track of how many times a token appeared in a stream title and the views of the stream which had the token in their stream title. Then, the top 15 tokens in terms of average views and total views were plotted.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BanMRSbFMkQXYsHw" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VBmoeAZjwYab55Ku" /></figure><p>From the plots above, putting the name of famous Grand Master Chess players often resulted in high average viewerships such as Hansen, Mamedyarov, and Naroditzsky. The token event or events also resulted in high average viewership either denoting a chat command or signifying a chess event, which tends to garner high viewership. Surprisingly, no chess openings or time controls (different chess formats) made the either the top 15 in highest average viewership or total viewership.</p><p><strong>Topic Analysis — <em>Genshin Impact</em></strong></p><p>The Genshin Impact Twitch dataset included 1299 rows of individual live streams, with each stream title, start date, streamer username, streamer ID, and language used in the stream. The stream dates in the dataset were mainly from Fall 2022 (around October 2022). Twitch livestreams count viewers live, so the rate on which users are attracted to a stream title and stream thumbnail are incredibly important for streamers to increase viewership. Using the Genshin Impact Twitch dataset, we performed topic analysis on the livestream titles for this gaming category using Top2Vec.</p><p>Top2Vec is a topic modeling algorithm that generates topics present in text. Using individual stream titles, we first cleaned each row of non-alphanumeric characters, including emojis, special characters and punctuation. Each title was coerced to all lowercase, and then Top2Vec was applied to each.</p><p>Top2Vec found two main topics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VWH6UEYBtO_eXjuW" /></figure><p>Since this dataset was pulled from recent fall 2022 Genshin Impact Twitch streams, the words under each topic that the Top2Vec algorithm detected reflect this (for example, “nilou” was found in both topics, and for context Nilou was a newly released playable character on October 14, 2022). Unsurprisingly, the top titles for both topics had the word “Genshin.” Interestingly, another common word from both topics was “discord” and “vtuber”, which are nonspecific to Genshin Impact or Twitch, but are larger subcultures of the streaming and gaming communities as a whole.</p><p>The top title for each topic was chosen based on stream rank from the Genshin Impact Twitch dataset and the score assigned by Top2Vec for its respective topic. The top title for topic 1 was rank 142 and read “Finishing genshin event, weeklies and abyss…”, and the top title for topic 2 was rank 29 and read “Having an Attitude of Gratitude. I’m grateful for my Waifus :D | Genshin Impact | Life Conversations | !discord”.</p><p><strong>Machine Learning</strong></p><p>Besides topic analysis, we also attempted to predict stream views based on the stream title words; however these did not produce any meaningful results. Instead, we explored the Kaggle dataset on top video games and modeled viewership over time.</p><p>The Kaggle dataset included monthly viewership and streamer statistics for over 200 categories, over the time span from January 2016 until December 2021. The statistics included: hours-watched, peak viewers, peak channels, streamers, average viewers, average channels, and average viewer ratio. A particularly notable statistic is hours-watched, as it denotes not only whether a category has a lot of viewers, but also viewer retention. A deeper look was then performed on the hours-watched statistic and an attempt was made to utilize various machine learning methods in order to predict it. The machine learning methods that would be utilized are the Decision Tree Regressor and Decision Tree Classifier.</p><p>The Decision Tree Regressor is utilized to build a regression model in a tree style, but can only be applied to continuous variables. The Decision Tree Classifier builds a classification model in a tree style, but utilizes discrete variables. Both of these models resulted in very low accuracy scores, indicating a weak relationship between the hours-watched and the various other variables.</p><p>From here, we decided to look at the twitch platform performance as a whole. By combining the data for the 200 categories for each month since January 2016, we were able to investigate the change in total hours watched on the platform. This resulted in the below figure representing viewership with respect to time for Twitch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*D0IfqielA_76RGiV" /></figure><p>From this figure, it appears that until the start of 2022, Twitch’s growth in Total Hours Watched could be modeled by an exponential function. Utilizing curve fitting methods, the equation: f(x) = a*e(bx+c)+d ; was fitted to the graph resulting in the plot below. With parameters: a = 5.529, b =0.026, c =17.881 d=1466036.642, this exponential function fit relatively well as the Mean Absolute Percentage Error was 11.22%. This can alternatively be expressed as a Mean Absolute Percentage Accuracy of 88.78%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5NioFSVNGi-4fSxx" /></figure><p><strong>Conclusion</strong></p><p>Before coming to some general conclusions, we must also address the various shortcomings of our datasets and approaches.</p><p>The Kaggle dataset has not been updated to collect data for the past year of 2022, so it is difficult to definitively conclude if Twitch viewership will continue to follow an exponential growth. For instance, this was a period of time where the competition of YouTube streaming had grown and therefore Twitch viewership could have fluctuated as well.</p><p>The Twitch API game datasets were also collected over a short period of time of less than two weeks in the month of October of 2022. The choice of which games to collect data on were subjectively chosen for “top games” on Twitch as well as other categories we were simply interested in. Only the top 100 streams could be pulled using the Twitch API so stream titles of smaller streamers were not analyzed as in depth as larger streamers with more views. The data we were able to pull from the Twitch API was also extremely limited. Ultimately only the stream title and viewership columns were useful to our topic analysis, but did not produce significant results in machine learning.</p><p>It is not surprising that the views on Twitch can not be accurately predicted based on stream titles alone, but there are still various common word trends in top stream titles for different game categories.</p><p>It would also be interesting to attempt our analysis again with more data pulled over a longer period of time to see if these trends uphold or change. And for future exploration, based on common word trends, it may be plausible to predict the game category a stream title falls under to help streamers automate the selection of game category they are streaming under on Twitch.</p><p>Finally, if you are interested in the code used throughout this project, feel free to look at our <a href="https://github.com/the-data-science-union/DSU-F2022-Twitch">Github</a> page.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4a6bc17385e5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Predicting MLB Playoff Teams From Their Yearly Statistics]]></title>
            <link>https://medium.com/@ucladsu/predicting-mlb-playoff-teams-from-their-yearly-statistics-2fc058f163af?source=rss-e0043e83850a------2</link>
            <guid isPermaLink="false">https://medium.com/p/2fc058f163af</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[predictive-analytics]]></category>
            <category><![CDATA[sports-analytics]]></category>
            <category><![CDATA[baseball]]></category>
            <dc:creator><![CDATA[UCLA Data Science Union]]></dc:creator>
            <pubDate>Fri, 30 Sep 2022 22:42:03 GMT</pubDate>
            <atom:updated>2022-10-06T01:51:10.018Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q_DVOAGc2J9u8Y_G72lWEw.jpeg" /></figure><h3>Introduction</h3><p>If you are a sports fan this day in age, you probably know that a significant amount of game preparation and in-game decisions are based upon data analytics. The first professional sport to start using modern data collection and analysis on a large scale to decide game plans, which players to use, and how to learn from the past was Major League Baseball (MLB). In the past 25 years, the way front offices, managers, and fans view baseball has drastically changed thanks to a widespread adoption of data analysis by the MLB.</p><p>With data collection and usage being so valuable to professional baseball in the past few decades, the amount of statistics being tracked has grown very large in a very short amount of time. And, with hitting, pitching, and defense all being essential to winning, how is it possible to determine which metrics are the most important?</p><p>Using data from the website Baseball Reference, a website which contains datasets covering a variety of MLB statistics throughout MLB history, we set out to determine which metrics are the most valuable and see if we could use them to predict if a given team will make the playoffs or not.</p><h3>Methodology</h3><p>To gather our data and conduct our analysis, we used the various tools and libraries of Python. We began by webscraping our data off of Baseball Reference using the requests and BeautifulSoup libraries and cleaned the data using pandas. We gathered team data from eight different datasets covering hitting, pitching, and fielding metrics over the 21 season period from 2000 to 2021 and recorded if each team from each year made the playoffs or not. There were hurdles to overcome such as columns being interpreted as the wrong datatype, the shortened 2020 season (there were 60 games played in 2020 instead of the normal 162), and different variables only being available in some but not all of the years we were analyzing. We determined that removing the 2020 season from each of our eight datasets would be the most effective way to deal with the scaled down season. Also, depending on the dataset, we either removed the variables that were not tracked every year or removed years that didn’t have all variables. Some datasets did not have any problems with missing variables.</p><p>After the datasets were tidy and organized, we moved on to conducting exploratory data analysis (EDA). We used the matplotlib and seaborn libraries to make visualizations that describe patterns we found and which variables seemed most correlated with playoff status (which is discussed in the following sections). We also made interactive dashboards using Tableau as a way to see patterns on a team-to-team basis and across the seasons.</p><p>We then moved on to analyzing our data using machine learning and the scikitlearn library. Three different types of models were used on each dataset to try and predict if a given team made the playoffs or not. Depending on which models were used, some datasets had to be standardized, or variables had to be removed to improve model performance. We recorded the accuracy scores of each model over many iterations of sampling and generating a model. We then went back to Baseball Reference and retrieved each dataset from the ongoing 2022 season. We adjusted necessary variables to represent a 162 game season and predicted which teams from the current season were to make the playoffs. The results of these predictions are mentioned in the following sections which discuss each of the eight datasets that were used and the results of their analysis.</p><h3>Hitting Data</h3><h4>Standard Hitting Metrics</h4><p>The first dataset to be discussed describes standard hitting. Standard hitting contains the raw and original data of the batters’ information on each team, as apposed to more advanced and complicated metrics. In recent years, advanced stats have become a greater part of the mainstream conversation, but it is still important to look at standard metrics.</p><p>The standard batting variable that is the most correlated to playoff status is LOB (runners left on base). Other metrics that have a moderate correlation with playoff status include OBP (on base percentage), SLG (slugging percentage), OPS (on base percentage plus slugging percentage) and OPS+.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VwbfKRNDM4dtx5sW" /></figure><p>There were also quite a few variables that were not significantly correlated with playoff status. From the barplot, we can see one such variable is K percentage (hitter strikeout rate). It shows that teams who do not make the playoffs have a slightly higher strikeout rate, however, the K percentage of non-playoff teams is too close to infer a difference. This could possibly be because pitchers with lower strikeout rates might be able to induce weak contact, which leads to outs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BEKgeNXW2M6w8FKW" /></figure><p>The above graph, shows the distribution of GDP (double plays grounded into) and OBP of teams in different years. As we can see, in general, teams that make the playoffs have higher values for OBP while it is relativley evenly distributed for GDP. Since less outs tend to lead to better team results, it makes sense on-base percentage is different for playoff status but it is surprising to see no difference for double plays.</p><p>For machine learning analysis, the Logistic Regression model provided the most accurate results out of the three models that were used. It consistently predicts around 12 teams be in the playoffs, with the teams with the strongest likelihoods being the Atlanta Braves, Houston Astros, Los Angeles Dodgers, Milwaukee Brewers, Minnesota Twins, New York Mets, New York Yankees, Philadelphia Phillies, St. Louis Cardinals, and Toronto Blue Jays. The prediction accuracy is about 75%-80% which is not bad, but still has potential to be much better.</p><h4>Advanced Hitting Metrics</h4><p>The domain that we are now going to focus on is advanced hitting analysis. The advanced hitting dataset has unique metrics that distinguish it from other domains of hitting. Some metrics focus on the direction of a batted ball — whether the ball is batted on the pull side of the batter (Pull %), push (Push %) side of the batter or the center field (Cent %). There are also metrics that focus on the trajectory of a batted ball — whether the batted ball is a fly ball (FB %), line drive (LD %), or a Ground Ball (GB %). Finally, there are also some unique metrics that focus on batters that covered more bases than they were expected to (XBT %).</p><p>On conducting some preliminary EDA on this dataset, we noticed that the metrics Win Probability Added (WPA) and Relative on-base Average (rOBA) impacted a team playoff’s status for that season the most. WPA is a measure of the change in probability of winning the game that a batter made with their performance. Additionally, rOBA is a measure of a batter’s offensive contributions in context to their batting’s run value. Since WPA is a metric that specifically measures chances of winning, it isn’t very surprising that it closely determines if a team makes it to the playoffs or not. And in fact, it did have a correlation value of over 0.53. Additionally, we uncovered some other interesting relationships as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/402/0*uuEziV93xK59IoVE" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/402/0*CmRXCXJCkq2dJzwU" /></figure><p>From the above visualization, we notice how out of all the directions a batter can hit a ball in, those that have higher home run percentages tend to hit the ball in their pull direction. Additionally, another interesting observation is how this dependence is stronger among teams that make the playoffs versus those that don’t make the playoffs that season. From the accompanying visualizations we can also see how there is no relationship (or a slight negative one in teams that make the playoffs) between balls batted to push direction and the home run percentage of a batter, and there is a negative correlation between balls batted to the push direction and home run percentage.</p><p>To put these findings to the test, we explored multiple machine learning models to create a predictor that can predict teams that make the playoffs based on these significant features, and other less significant ones in the dataset. After cleaning the data, and using a MinMaxScaler to scale the feature values, we experimented with Support Vector Machines, Logistic Regression algorithms, and Random Forest Classifiers to determine an ideal model. We found that the model with the best accuracy score was the Random Forest Classifier with a mean performance score of 83.67% after 100 iterations. We were also able to visualize how the accuracy scores and team predictions were distributed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/615/0*hDzJYfyo2IxgQLHi" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/0*LSGrewABcydO_igD" /></figure><p>The New York Mets, New York Yankees, and Los Angeles Dodgers were predicted the most number of times by our model to make the 2022 playoffs. Additionally, both Atlanta Braves and Houston Astros are teams that are fairly competitive but to a lesser extent, according to the model.</p><p>In conclusion, we found many interesting relationships between the advanced hitting metrics, specifically between the home run percentage, extra bases taken percentage, and playoff status of a team. We were also able to build a robust and sufficiently dependable model to predict the teams that would make the playoffs for the ongoing Major League Baseball season, and possibly future ones as well.</p><h4>Sabermetric Hitting Metrics</h4><p>The sabermetric hitting dataset measures hitting, but focuses on more advanced batting statistics in particular. Specifically, the sabermetric hitting dataset quantifies things such as how a player would have performed had they played in different parks. For example, the ‘lgOBP’ metric indicates the on-base percentage that an average player would have in same playing conditions of a given team. These metrics are useful for quantifying how well a team did given its circumstances, and indicate how more basic metrics do not always tell the complete story.</p><p>One variable that indicated a correlation with playoff status was runs scored per game (R/G). Intuitively, this makes sense — teams that create more runs are more likely to win, and therefore advance to the playoffs. Surprisingly, measures such as batting average on balls in play (BABIP) did not make a significant impact on whether teams made it to the playoffs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/490/0*bcR2BHHjWfYU2ajs" /></figure><p>Above, we can see an example plot of the 2000–2021 sabermetric batting dataset, in which adjusted batting wins (BtWins) is plotted against runs per game (R/G), and plot points are sized according to team isolated power (ISO). The points are additionally colored by whether the given team made the playoffs that year. We can see that the orange teams (teams that made the playoffs) are clustered in the top-right of the graph, indicating that they on average had higher BtWins and R/G over their season, and that these plot points are larger, indicating that their batters had a higher ISO.</p><p>When using this dataset with machine learning models to predict whether teams would make the playoffs, there were mixed results. This can be understood from the visualization below, which shows the accuracy of 1000 random forest models trained on different splits of the sabermetric batting dataset. The plot shows that the accuracy tended to stay between 75% and 80% accuracy, with no models making it above 90%. This likely indicates two things — one is that the model needs to be further tuned by adjusting the hyperparameters to maximize its performance, and the other is that the sabermetric hitting dataset on its own is not indicative enough of whether a team will make the playoffs. At a heuristic level, the teams predicted for the 2022 season seem accurate (teams that are performing well in the current season seem likely to make the playoffs).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/0*C-Sjc7drCKQQyev2" /></figure><p>Overall, it seems as though predicting which teams will make the MLB playoffs given their current season stats is a completely achievable goal, however, more metrics spanning all aspects of baseball would need to be included. Additionally, using past data to predict which teams make the playoffs is likely imperfect, as adjustments to things such as the flight of the baseball have impacted which stats are important to maximize in order to create a winning MLB team.</p><h3>Pitching Data</h3><h4>Advanced Pitching Metrics</h4><p>Now we will dive into the picthing datasets by beginning with advanced pitching. The advanced pitching dataset measures key pitching statistics, but compared to the other pitching datasets, this dataset examines pitching domains such as pitching ratios, batted balls, batting averages against the pitching teams, and win probabilities. This is different from statistics aggregated over all of games played, which we will see in the relief and starting pitching datasets to come. Some of these statistics include Strike Out Percentages (SO%) and Base-Out Runs Saved (RE24). These statistics represent calculations derived from the observations made in the games that have occurred during the season.</p><p>Some of the variables most correlated to a team’s playoff status were the RE24 and Championship Win Probability added by Pitcher (cWPA) statistics, with correlations of 0.58 and 0.62 respectively. These correlations are reasonable to encounter, as these statistics are calculated to predict a team’s win probability based on the other pitching statistics, and playoff-contending teams are the ones that are most likely to put up these numbers, and therefore, have higher win probabilities.</p><p>We were surprised that home run percentage (HR%) didn’t have as negative a correlation as we thought it would (R = -0.2), so we decided to take a closer look. When viewing the distribution of home run percentages allowed by pitching teams across the 20 seasons, for the most part, teams that conceded less home runs throughout the season were more likely to make the playoffs. However, there were not any significant differences between the the centers of the playoff-team HR% distribution and the non-playoff team HR% distribution. Below is a visualization of the distributions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/838/0*_RSH_hipSWwc1Mg0" /></figure><p>We used a Random Forest machine learning model to predict 2022 playoff teams based on the past twenty seasons of data. The random forest classifier functions by constructing trees that make branching decisions based off of each feature. It will try to pick a point in the variable to split the boundary, and then make a decision based on this split. Each tree is based off of random splits of the features, leading to a “forest” of trees that are uncorrelated, and thus independent of each other. This model was constructed from the variables below, chosen for their relative lack of correlation with other variables in the dataset — quantified by variance inflation scores.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/342/0*mzJVBeWCZ826eZZk" /></figure><p>This model achieved an overall accuracy of 86% for the 1000 iterations that we ran it for. Below are the proportions of iterations in which the random forest predicted a specific team to make the playoffs — the random forest predicted 12 different teams to make the playoffs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*T2KboDFbTk7RZF-S" /></figure><p>The model seems to correspond with many pundits’ predictions, with the model sharing all but one of the teams that garnered at least an 80% chance of making the playoffs according to the live prediction trackers of <a href="https://projects.fivethirtyeight.com/2022-mlb-predictions/">FiveThirtyEight</a>. However, the data for this article was webscraped in the middle of the MLB regular season (early July 2022), so these results are very different from the prediction trackers at time of this article (late August 2022). Indeed, the model only included the Seattle Mariners as a playoff team in 11% of simulations, however at time of writing, FiveThirtyEight predicts that the Mariners have a 86% chance of making the playoffs.</p><p>The variables in the advanced pitching dataset were suitable as predictor variables to predict MLB teams’ playoff chances, due to the success we see from the random forest classifier.</p><h4>Starting Pitching Metrics</h4><p>The next dataset used for analysis measures starting pitching data. In contrast to the other pitching datasets, the starting pitching dataset focuses on data obtained from the starting pitcher. This includes some unique metrics such as Quality Start Percentage (QS%), Bequeathed Runners (BQR), and Pitches per Game Started (Pit/GS).</p><p>The variables that demonstrated the most correlation to playoff status were Quality Start (QS) and Bequeathed Runners Score (BQS). QS measures the amount of times a starting pitcher who pitched for at least 6 innings allowed 3 or fewer earned runs. Teams that made playoffs had an average of approximately 83 Quality Starts compared to an average of approximately 74 Quality Starts for non-playoff teams. It is interesting to note that non-playoff teams had the greater max in this category (110) than playoff teams (108), which suggests that teams that do qualify for playoffs are more likely to have better consistency regarding Quality Starts. Another variable that showed correlation to playoff status was Bequeathed Runners Scores (BQS), which measures the number of runners on base when the starting pitcher left the game who subsequently scored. Playoff teams had a BQS of about 67, while non-playoff teams had a BQS of about 77.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*bJmIyzxb_IpPAnxc" /></figure><p>This histogram of BQS frequency between playoff and non-playoff teams illustrates the correlation between BQS and playoff status; the KDE curve for the non-playoff teams is more left-skewed than the KDE curve for playoff-qualified teams. It can also be observed that teams that failed to qualify for the playoffs tend to have a greater amount of large outliers than playoff teams, which suggests that teams that have poor games in which they produce a large BQS are less likely to make playoffs.</p><p>In predicting whether each team in the MLB will qualify for the playoffs or not, one of the machine learning models we developed was the Decision Tree model. Over 1000 iterations, the Decision Tree model produced predictions with an accuracy of 88.47% with a standard deviation of 2.40%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/0*dbTMAjJugUyJTGVq" /></figure><p>This bar graph shows the proportion of playoff appearances predicted by the Decision Tree model for each team. We can observe that traditional powerhouse teams such as the New York Yankees and the Los Angeles Dodgers were amongst the several teams that were predicted to be extremely likely to qualify for the playoffs. This corresponds to the current standings of these teams in the 2022 season, from which we can conclude that the variables that were used from the starting pitching dataset from 2000 to 2021 are, to a great extent, accurate predictors of playoff status.</p><h4>Relief Pitching Metrics</h4><p>The last pitching dataset we used pertains to relief pitching statistics. The relief pitching dataset looks at statistics in the context of relief pitchers as opposed to starting pitchers and general pitching statitstics. Some unique relief pitching metrics include Saves, Holds, Leverage Situations, and more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/490/1*eQ7aygPXLSzTTvdb14WOMA.png" /></figure><p>The variables that seemed to be the most correlated with playoff status were BHD (games entered while behind), AHD (games entered when ahead), and LGR (losses in relief). This does make sense as games entered when ahead/behind is closely associated with whether a team won the game or not. It was surprising to see that the more well-known metrics like SV% (save percentage) and Holds did not seem to have strong correlations with playoff status.</p><p>The model that performed the best in predicting playoff teams was the logistic regression model. Over 50 iterations, the model produced an average accuracy score of 87.7%. The model predicted 12 teams to make the playoffs in 2022, with the Atlanta Braves, Houston Astros, Los Angeles Dodgers, Milwaukee Brewers, Minnesota Twins, New York Mets, and New York Yankees being the teams that were predicted to make the playoffs during all 50 iterations. These predictions seem very accurate because at the time of the predictions (late July 2022), all 12 teams were at least within a few games of a playoff spot.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/1*-KwHpnmGPqMyUktAv0VAAQ.png" /></figure><p>In addition, the other models used for machine learning analysis produced similar teams for the 2022 predictions. This indicates that relief pitching metrics reveal a great deal about a given team’s playoff chances.</p><h3>Fielding Data</h3><h4>Standard Fielding Metrics</h4><p>For the first of the fielding data sets we will look at team standard fielding. While the outfield dataset looks at outfielder metrics, standard fielding covers all the fielders including outfielders, infielders, pitchers, and catchers. In addition, the team standard fielding dataset includes defensive efficiency and excludes variables such as total zone fielding runs above average (Rtz) and total zone outfield arm runs above average (Rof).</p><p>To find the correlations between the different variables all at once, we used a seaborn heatmap. From the heatmap, we found that defensive efficiency and innings played were the variables most correlated to playoff status. While we expected defensive efficiency to be more strongly correlated with playoff status, we did not expect innings played to be as strongly correlated as we didn’t expect the length of games to relate to the team’s chances of making the playoffs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/390/0*qyRssbagTfpTze8W" /></figure><p>One of the machine learning models that was used for analysis was K-Nearest Neighbors, which had an average accuracy of 73%. K-Nearest Neighbors measures the distance between data points, and assumes that those that are closer together are similar, so with the data of teams that made the playoffs, the algorithm assumed that teams with similar stats would also make the playoffs. In the predictions, the Cincinnati Reds showed up the most in the predictions the model made. For the most part, the predictions seemed to be only somewhat accurate and the model had 18 different teams show up in various predictions of which teams would make the playoffs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/606/0*WKGdkK9R-HqV9DyB" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*hRmagtT3GwDFjv8H" /></figure><p>Overall, the data showed that in general, a higher defensive efficiency means a team is more likely to make the playoffs, and that the variables used for K-Nearest Neighbors modeling are mostly suitable as they usually give an accuracy score around 0.7.</p><h4>Outfield Fielding Metrics</h4><p>The outfield dataset provides information on outfielder defense, including defensive chances, fielding percentage, outfield double plays turned, errors and more. One variable that was correlated with playoff status was putouts. Teams that made it to the playoffs generally had a lower amount of putouts on average.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/413/0*zCE9HhvzXVPrz5BJ" /></figure><p>One variable that did not seem to be indicative of playoff status is average error. Some teams that did not make it to the playoffs had higher average errors than teams that did make playoffs, but there were plenty that had fewer average errors as well.</p><p>For machine learning analysis, the decision tree model had the highest accuracy scores when it came to making playoff predictions. Interestingly, the top five teams predicted to make the playoffs were similar, with mild variation across all models used (including logistic regression and K-nearest-neighbors). The decision tree model predicted the New York Yankees, New York Mets, Toronto Blue Jays, Tampa Bay Rays, and the Houston Astros as the top five most frequent teams to make the playoffs, with the Yankees having the highest frequency (94 out of 100 iterations).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/406/0*kRX9ubivkd_BkF2o" /></figure><h3>Conclusion</h3><p>Through the analysis of hitting, pitching, and fielding metrics, it can be concluded that all three aspects of baseball are very telling of a team’s playoff chances. However, there is a lot that can be done to improve the results of the predictive models. Several different aspects of the team’s performance from hitting, pitching and fielding would need to be taken into consideration simultaneously in order to maximize prediction accuracy. Furthermore, to ensure a high accuracy rate, variables that have high correlation with one another should be eliminated. Thus, the target should be a small amount of variables across all datasets that are the most correlated with playoff status. We succeeded in creating a soild foundation for the predictive models, but expanding and refining the variables used for the models would drastically improve their accuracy. In addition, our primary goal was to make basic models as opposed to going to great extents to make high accuracy models. Therefore, trying out more models and adjusting the data and parameters more would definitely improve the accuracy of the models as well.</p><h3>Data Sources</h3><ul><li>“Major League Baseball Team Standard Batting.” Baseball Reference, <a href="https://www.baseball-reference.com/leagues/majors/2022-standard-batting.shtml">https://www.baseball-reference.com/leagues/majors/2022-standard-batting.shtml</a>. Used the 2000–2022 versions of this data table.</li><li>“Major League Baseball Team Advanced Batting.” Baseball Reference, <a href="https://www.baseball-reference.com/leagues/majors/2022-advanced-batting.shtml">https://www.baseball-reference.com/leagues/majors/2022-advanced-batting.shtml</a>. Used the 2000–2022 versions of this data table</li><li>“Major League Baseball Team Sabermetric Batting.” Baseball Reference, <a href="https://www.baseball-reference.com/leagues/majors/2022-sabermetric-batting.shtml">https://www.baseball-reference.com/leagues/majors/2022-sabermetric-batting.shtml</a>. Used the 2000–2022 versions of this data table.</li><li>“Major League Baseball Team Advanced Pitching.” Baseball Reference, <a href="https://www.baseball-reference.com/leagues/majors/2022-advanced-pitching.shtml">https://www.baseball-reference.com/leagues/majors/2022-advanced-pitching.shtml</a>. Used the 2000–2022 versions of this data table.</li><li>“Major League Baseball Team Starting Pitching.” Baseball Reference, <a href="https://www.baseball-reference.com/leagues/majors/2022-starter-pitching.shtml">https://www.baseball-reference.com/leagues/majors/2022-starter-pitching.shtml</a>. Used the 2000–2022 versions of this data table.</li><li>“Major League Baseball Team Relief Pitching.” Baseball Reference, <a href="https://www.baseball-reference.com/leagues/majors/2022-reliever-pitching.shtml">https://www.baseball-reference.com/leagues/majors/2022-reliever-pitching.shtml</a>. Used the 2000–2022 versions of this data table.</li><li>“Major League Baseball Team Standard Fielding.” Baseball Reference, <a href="https://www.baseball-reference.com/leagues/majors/2022-standard-fielding.shtml.">https://www.baseball-reference.com/leagues/majors/2022-standard-fielding.shtml.</a> Used the 2000–2022 versions of this data table.</li><li>“Major League Baseball Team Outfield Fielding.” Baseball Reference, <a href="https://www.baseball-reference.com/leagues/majors/2022-specialpos_of-fielding.shtml">https://www.baseball-reference.com/</a>leagues/majors/2022-specialpos_of-fielding<a href="https://www.baseball-reference.com/leagues/majors/2022-specialpos_of-fielding.shtml">.shtml</a>. Used the 2000–2022 versions of this data table.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2fc058f163af" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring K-pop and BTS Song Popularity with Spotify Audio Features & Song Lyrics]]></title>
            <link>https://medium.com/@ucladsu/exploring-k-pop-and-bts-song-popularity-with-spotify-audio-features-song-lyrics-40284399948f?source=rss-e0043e83850a------2</link>
            <guid isPermaLink="false">https://medium.com/p/40284399948f</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[kpop]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[bts]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[UCLA Data Science Union]]></dc:creator>
            <pubDate>Wed, 01 Jun 2022 19:02:54 GMT</pubDate>
            <atom:updated>2022-06-01T19:02:54.887Z</atom:updated>
            <content:encoded><![CDATA[<h4>Project Leads: Emily Gong, Tiffany Le &amp; Project Members: Audrey Tey, Chelsea Chen, Julia Martin, Justin Kaufman, Sarah Kosic, Sheridan Ohlinger</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N5Hx342oHq8ZANBKt6G4LA.jpeg" /><figcaption><a href="https://pitchfork.com/news/bts-perform-butter-at-2022-grammys-watch/">Photo Credit</a></figcaption></figure><h3><strong>Introduction</strong></h3><p>This project is for all the K-pop fans, ARMY’s, and music lovers (with taste). We will be using specific features of K-pop songs, with some focus on BTS songs, to predict each song’s popularity through machine learning and natural language processing.</p><p>We were first inspired by the <a href="https://www.kaggle.com/datasets/mogura20/bts-147-songs-audio-features-spotify-api">BTS 147 Songs Audio Features (Spotify)</a> dataset from Kaggle, but to conduct our analysis, we referred to the following two datasets: the <a href="https://www.kaggle.com/datasets/ericwan1/kpop-artists-and-full-spotify-discography">Kpop Artists and Full Spotify Discography</a> and <a href="https://www.kaggle.com/datasets/kailic/bts-lyrics?resource=download">BTS Lyrics</a> dataset (more details in later sections).</p><p>Before proceeding, we extracted the “Popularity” scores Spotify generated for each song we observed through the <a href="https://developer.spotify.com/documentation/web-api/">Spotify API</a> and appended it to the datasets mentioned above. This way, we had a fixed metric we could use to determine the accuracy of our results produced by our models. In addition, we extracted the corresponding audio features and popularity scores based on the songs IDs listed in the “BTS Lyrics” data set.</p><h3><strong>Machine Learning: Predicting Popularity of K-pop Songs</strong></h3><p>By Audrey, Chelsea, Julia</p><blockquote>The machine learning models utilized for this project, including Bagging, Random forests (single tree and pruned tree), Boosting, Ridge Regression and Lasso Regression, were largely focused on reporting the popularity scores predicted from observing the audio features Spotify provided.</blockquote><h4><strong><em>Description of Dataset</em></strong></h4><p>The original dataset we used was downloaded from <a href="https://www.kaggle.com/datasets/ericwan1/kpop-artists-and-full-spotify-discography?select=273_kArtists.csv">Kaggle</a> and has 16,329 K-pop songs by 273 different solo and group artists released from the 1990s to 2021. We narrowed our selection to the 100 most popular K-pop artists, which left us with 8,528 songs.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d99fbbea799bccdfb9cd3aef7a9ee934/href">https://medium.com/media/d99fbbea799bccdfb9cd3aef7a9ee934/href</a></iframe><p>The dataset includes the artist, song, and album name, as well as the Spotify track ID. From the Spotify track IDs, we were able to use the Spotify Web API to pull the Spotify popularity score for each song. Spotify <strong>popularity scores</strong> on a scale from 1 to 100 (1 being the least popular, 100 the most popular) based on an <a href="https://www.loudlab.org/blog/spotify-popularity-leverage-algorithm/">algorithm</a> that takes into account streams, shares, likes, adds to playlists, and how recent all of those actions took place.</p><p>In addition, there are 13 variables that describe aspects of the songs: danceability, energy, key, loudness, mode, speechiness, acousticness, instrulmentalness, liveness, valence, tempo, duration, and time signature. Spotify provides more information <a href="https://developer.spotify.com/documentation/web-api/reference/#/operations/get-several-audio-features">here</a> about what each of these variables mean and their units. We decided to focus on these 13 variables for our modeling.</p><p>The 8,528 songs of the dataset were randomly split into three different datasets: “train” (5970 songs), “validation” (1705 songs), and “test” (853 songs). As the name implies, the “train” dataset is for training the models we wish to use. The “validation” dataset is to check the prediction error of the trained models and fine-tune performance. The “test” dataset is to test the final accuracy of each model.</p><h4><strong><em>Description of Models</em></strong></h4><p>As our goal was to predict popularity from a list of other variables, we focused mainly on methods for regression analysis. We selected the following <strong>supervised machine learning</strong> models to be trained on our dataset:</p><ul><li>Bagging</li><li>Random forests, including a single tree and a pruned tree</li><li>Boosting</li><li>Ridge regression</li><li>Lasso regression</li></ul><p><strong>Bagging, random forests and boosting</strong> required the tuning of parameters to improve the model. For bagging and random forests, we created 4 models for each method with 500, 1000, 1500, and 2000 trees, although we disregarded the models with 2000 trees because the small decrease in RMSE compared to the models with fewer trees was not worth the overfitting present in the model with 2000 trees.</p><p><strong>Ridge and lasso regression</strong> did not require the tuning of parameters as they are linear model regularizations, where a normal linear model is first trained before applying regularization. Regularization is training a model by adding a penalty in the loss function, so that coefficients would not take on extreme values.</p><p>While some models are better than others, we decided to evaluate the performance of all of them side-by-side and compile the results to better understand how the models were predicting popularity.</p><h4><strong><em>Predictions and Accuracy of Models</em></strong></h4><p>We decided to use <strong>root mean squared error (RMSE)</strong> to evaluate the performance of our models, and to compare the models with one another. RMSE is the square root of the average of all squared differences between the actual value of the response variable (popularity score, in this case) and the value predicted by the regression model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/0*DQL25WPYbSQVod4j" /><figcaption><a href="https://towardsdatascience.com/what-does-rmse-really-mean-806b65f2e48e">Photo Credit</a></figcaption></figure><p>While mean squared error (MSE) is one of the most popular performance metrics used, we decided to use RMSE as it can serve as a good estimator for the standard deviation of the distribution of errors. As such, it could give us insight into how far off our predictions from each model were. Furthermore, RMSE does not penalize smaller errors like MSE does.</p><p>Below is a table listing the machine learning methods used and their respective RMSEs as calculated from the validation and testing datasets. A bar graph of the testing RMSEs are also plotted below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/299153e0eea1dc48a5b93ca2a541dc61/href">https://medium.com/media/299153e0eea1dc48a5b93ca2a541dc61/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZmSmhBmL37i_NhnN" /></figure><p>We can see that there is <strong><em>not</em></strong> a big difference in the performance of all models, with the best performing model being random forest with a RMSE of 13.917, and the worst performing models being the single and pruned trees with a RMSE of 15.140. This is expected because a single or pruned tree is a building block of the random forest ML method but is not good enough to serve as a model for prediction or regression on its own.</p><p>The random forest model performed the best on the testing dataset. Diving further into that model, we wanted to view the distribution of errors, or the differences between the actual popularity scores and the scores predicted by the model. The bar chart below shows that distribution, with counts of songs grouped based on their errors (<em>d</em>, difference between actual and predicted popularity scores).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Wh5HlxYq3TZ0KDh5" /></figure><p>The plot shows a right-skewed Normal distribution, with the most number of songs having a popularity score difference of 5 to 10. 52 out of 853 songs (or roughly 6% of songs) were extremely accurately predicted, and had predicted scores within +/- 1 of their actual scores. There were also predictions that did not fare well, with differences of more than 30.</p><h4><strong>Comparing Model Predicted Popularity and Actual Popularity Scores</strong></h4><p>The following graphs depict scatterplots of predicted popularity scores, from the various models we explored, against the actual popularity scores of the songs from the “test” dataset. The blue line plotted on top of the scatterplot is the line of best fit or regression line, with the grey shaded region indicating the 95% confidence interval of the range the regression line would be drawn. The red line is the line of equality, the y = x line that we ideally want all of our points to fall along if our predictions were completely accurate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XdbNhYnV7xxn4n0IwLSRKw.png" /></figure><p>From these graphs, we can see that none of our models predicted above a popularity score of 60 even though actual popularity scores reached up to 78.</p><p>Despite the regression lines falling closely along the equality line, this does not necessarily indicate good predictions. We can clearly see variation of dispersed predicted vs actual scores across the graphs.</p><p>However, as previously indicated with the lowest RMSE score, on the “Random Forest Predicted vs. Actual Popularity” we can see slightly more points gathered around the equality line compared to the other graphs. Bagging with 1000 trees also showed similar trends.</p><h4><strong><em>Insights &amp; Case Studies</em></strong></h4><p>To explore the “test” dataset further, we compared the predicted popularity scores across all the models run with the actual score and calculated the RMSE.</p><p>The table below shows a list of interesting cases that we picked out from the dataset. The last column on the right explains why the case study was picked out. The first three songs had the lowest RMSE, meaning that they were the most accurately predicted songs out of the testing dataset of 853 songs. However, we were not able to find relevant reasons to account for this.</p><p>The fourth song had the highest RMSE, meaning that its popularity score was the least accurately predicted, and it is also classified as an unexpectedly popular song. Unexpectedly popular songs are songs that had the highest actual popularity scores, but also fairly high RMSEs, and were not predicted to be popular.</p><p>On the flipside, the last song is categorized as an unexpectedly unpopular song as it had the lowest actual popularity score and a fairly high RMSE, and was predicted to be popular.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*99gJN-FKCAX7bKPX" /></figure><p>Curiously, the two songs with the highest RMSEs were “0X1=LOVESONG” and “LO$ER=LO♡ER,” both by the artist TXT (TOMORROW X TOGETHER). These two songs had an actual popularity score of 75 and 78 respectively, and are amongst the highest scores in the “test” dataset; however the predicted popularity scores from our models underscored them in the 30s range.</p><p>These two songs were the title tracks of TXT’s last two comebacks in 2021, with “0X1=LOVESONG” released in May 2021 and “LO$ER=LO♡ER” released in August 2021; hence they are part of the more recent songs included in our dataset. It is therefore not surprising that they have high Spotify popularity scores since these songs were released more recently, combined with the fact that TXT has a fairly large, dedicated fanbase.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/599/0*-vIbx_tuFgD3XxIr" /><figcaption><a href="https://thestandom.com/article/tomorrow-x-together-win-10th-roty-at-seoul-music-awards/">Photo Credit</a></figcaption></figure><p>Besides being released in the same year, some interesting similarities between these two songs are that they are part of the genre of Pop-Rock, and played off the characteristics of Rock to invoke the theme of teen angst. Although TXT has been popular since their debut, these two songs were incredibly popular upon their release because of their focus on teen angst, garnering them a larger fanbase and greater exposure during the time periods of the songs release dates.</p><p>Therefore, across their discography that dates back to 2019, both of these songs are among TXT’s most streamed songs, despite these songs being released around a year ago.</p><p>Our models can not account for many of these outside factors to what makes a song popular, highlighting the shortcomings of attempting to predict song popularity based solely on spotify song characteristics.</p><h4><strong><em>Limitations</em></strong></h4><p>Another limitation of our case studies is that all of the case study songs came from the test dataset because that was the data we used to evaluate our model. Therefore, there may be songs from the training dataset that would show more extreme RMSEs and make for more interesting and standout case studies.</p><p>In regards to the actual popularity scores, Spotify calculates these scores with an algorithm that takes into account several factors like streams, likes, adds to playlists, and more. However, Spotify favors songs that are currently or recently played a lot. Songs that were played a lot in the past may have lower popularity scores than songs played a lot currently, even if they were both played around the same number of times during their respective eras. This is a limitation of our model because the actual popularity variable may not be very accurate and may be an underestimate of how popular the song really is.</p><p>Lastly, the dataset we used for this project originally had K-pop songs from 273 artists and had songs from bands and artists that have not come out with new music in several years. We decided to refine this dataset but handpicked 100 artists that are generally more well-known and have at least 200,000 monthly listeners in order to arrive at a dataset that represents more current K-pop music to better align with the original goal of the project, predicting the popularity of BTS songs. Therefore, the handpicked selection of artists is a limitation of this project in that we started with a selection of songs from artists that are already considered popular, so in the future it would be interesting to make a model for the entire dataset.</p><h3>Exploring BTS song lyrics with NLP: Sentiment Analysis, KNN Classification &amp; <strong>Type-Token Ratio</strong></h3><p>By Justin, Sarah, Sheridan</p><blockquote>We explored Natural Language Processing topics to evaluate how repetitive a song is and conduct sentiment analysis on BTS song lyrics. This way, we could develop insights into why certain BTS songs may be more popular than others based on texts in addition to the song audio features we explored earlier.</blockquote><h4><strong><em>Description of Dataset</em></strong></h4><p>The original dataset was downloaded from <a href="https://www.kaggle.com/datasets/kailic/bts-lyrics?resource=download">Kaggle</a> and was collated as a casual project using data pertaining to 19 Korean and English albums from Genius and Big Hit.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/02e1823dae34fab286c047eb4ce3497d/href">https://medium.com/media/02e1823dae34fab286c047eb4ce3497d/href</a></iframe><p>With 230 songs in total, we mainly focused on the lyrics column, which contains the text corpora of each song, all translated into English. To connect the lyrics with popularity, we also extracted the corresponding data from the Spotify API. Another issue we encountered was having different popularity scores for the same song. For such incidents, we decided to only keep a copy with the highest score and proceed with our analysis.</p><h4><strong><em>Sentiment Analysis</em></strong></h4><p>By Justin &amp; Sarah</p><p>We decided to conduct sentiment analysis on each song to determine if there was a link between a song’s sentiment and its popularity. Essentially, sentiment analysis is identifying the “attitude” of text; more specifically, it is the process of categorizing a piece of text as negative, neutral, or positive for our purposes. To carry this out, we used a model called VADER (Valence Aware Dictionary for Sentiment Reasoning), which uses a dictionary from which it calculates the intensity or strength of emotion of text based on lexical features. Utilizing this, we were able to calculate sentiment scores for each BTS song.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/0*OKvFjdNrabctPKKl" /></figure><p>From this, we were able to create a dataframe that contained each BTS song, its corresponding positive, neutral, and negative sentiment score, as well as its popularity. This would be utilized to feed into a model that would attempt to predict a BTS song’s popularity based off these scores.</p><p>Using <a href="https://www.geeksforgeeks.org/nlp-how-tokenizing-text-sentence-words-works/">word tokenization </a>and ridding of <a href="https://nlp.stanford.edu/IR-book/html/htmledition/dropping-common-terms-stop-words-1.html">stop words</a> for each BTS song’s lyrics, we were able to gather the top 10 most frequent words that appeared in each song as well as their corresponding frequencies. From here, along with the sentiment scores of each song, we produced word clouds of the most negative and positively scored songs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2bokY-ttzHb30uNfMyqhCw.png" /><figcaption><em>Word Cloud of Most Negative Words (left) &amp; Most Positive Words (right)</em></figcaption></figure><p>It can be seen that the most negatively scored songs have very negative words like rage, malice, boom, and the most positively scored songs have very positive words like love, baby, dream.</p><h4>K-Nearest-Neighbors Classification</h4><p>With our sentiment analysis for each song, we wanted to explore if these scores had any predictive power with respect to song popularity. Specifically, we wondered if the sentiment analysis could classify a song correctly as either high or low popularity. To do this, we first partitioned our numeric popularity data into high and low categories using Kernel Density Estimation. This technique allows us to simulate a smooth curve in the shape of our data that we can use to form our high and low classes based on minimum and maximum values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*e9SUm8XAblknrBHB" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/486/0*XFA57tGnts4QFKrc" /></figure><p>Once we had high and low popularity classes, we employed a K-Nearest-Neighbors classifier on the negative, neutral, and positive sentiment analysis scores to predict popularity class. Using the elbow method, we plotted error vs k to find our optimal K value of 10.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/0*A6_zZDMCJw5K__my" /></figure><p>We got a resulting accuracy of <strong>62%</strong> on our test data, which is slightly better than random guessing. We suspect that the small number of variables and small distribution of scores may have played a role in limiting the accuracy of our model. Further, there may just not be a strong correlation between popularity and sentiment that we can leverage for making predictions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/403/1*tdi9MVZ_MBVBPy1IxKJAvg.png" /><figcaption>Confusion Matrix with y_test and y_predict, where 0 means high popularity and 1 means low popularity</figcaption></figure><p>Overall, although we did not get great predictive results, our sentiment analysis seemed to work well with our intuition of positive and negative lyrics. While a great introduction to the world of NLP, there are many areas in which we could improve in the future. VADER was built and optimized for tweets and social media, so would likely not have the best results when extrapolated to song lyrics. By figuring out ways to better modify and apply current sentiment analysis tools to our task, hopefully our sentiment analysis scores would become more accurate and be able to make more accurate predictions.</p><h4>Delving into Repetitiveness of Kpop Songs with Type-Token Ratio &amp; Searching its Correlation with Popularity</h4><p>By Sheridan</p><h4><strong>Data Processing:</strong></h4><p>Our variables of interest were Spotify’s generated popularity score of each song and each song’s type-token ratio. The type-token ratio (TTR) is the number of types — the amount of unique utterances — divided by the token — the total number of utterances in a song — for each song. Specifically, tokens are identified as any individual string of text, separated by a space on either side. These utterances, or tokens, can include actual words and your typical “la la la” and “na na na” sounds often heard in BTS’s songs. Therefore, the type-token ratio is a metric that gives some insight as to how repetitive a song is. If a song has a high number of tokens (individual linguistic utterances) and a low number of types (unique linguistic utterances), it has a low type-token ratio, indicating how the same utterances are repeated multiple times. As listeners, songs with high repetition are often seen as “catchy,” so we wanted to see if a linguistic measure of “catchyness”, or repetitiveness, would correlate with a song’s popularity.</p><p>The data for type and token values was gathered using a linguistic software known as AntConc. Original Korean song lyrics gathered from azlyrics.com were individually converted to tokenized text files, through individual copy and pasting into google docs, and downloading as plain text files with the correct unicode format (UTF-8). A tokenized text file refers to one in which individual utterances are separated by a space on each side, with an utterance being a word, or a sound you heard in songs like “la” or “na.” Then, the files were uploaded into AntConc to provide me with each song’s individual type and token count. Using AntConc’s Concordance Tool, we also extracted other metrics including how often the song’s title was repeated and how many times the most frequent token was repeated to understand how different forms of repetition can affect popularity.</p><p>After inputting these values into an Excel file, we uploaded everything into R in order to produce our scatter plots. Songs that are considered intros, outros, or were highly instrumental with total token values of less than 100 were removed, as they are not representative of a typical BTS song and thus would affect our findings. Additionally, the ‘skits’ that are often included in BTS’s albums were excluded since they lacked the melodic component songs have which defeats the purpose of looking at “catchyness” or repetition, as they are more of a dialogue than a song.</p><h4><strong>Findings</strong></h4><p>We used a simple scatterplot for this approach in order to get the most interpretable results in understanding how lyrics contribute to a song’s popularity. There was a clear negative trend with a small P-value attached to slope, but there was quite a bit of variation in the plot resulting in an R-squared of only around .22.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s68f03krpB5UlMxImMiOyg.png" /></figure><p>The visualization shows popularity score against type-token ratio, with a clear negative correlation between popularity and type-token ratio. This implies that a song with low repetition, i.e., one with a high type-token ratio, is less popular than one with high repetition, or a low type-token ratio. Additionally, individual points were color coded based on their high frequency ratio, a rate that divides each song’s most frequent token by its total number of tokens. For example, BTS’s Dynamite repeats the string of tokens “na na na” around 90 times, making it the most frequent token in the song which has 500 token total. This gives it a relatively high “high frequency ratio” compared to other songs, around .18. The high this ratio, the lighter the purple color is on the plot, and we see an overall lighter color for the top left section of the graph, where highly repetitive and popular songs are found. Thus, a bit of extra insight can tell us that having one particular lyric repeated throughout a song (or a high frequency ratio), not just an overall high level of repetition (or a high type-token ratio), can play a role in determining popularity as well.</p><p>Some explanations for the high variance include:</p><ul><li>Though the software we used was able to understand Korean lyrics, BTS fans aren’t always able to! Therefore, it may be hard to correlate popularity with songs sung in Korean when fans may not be too sure of what is being said when they are listening.</li><li>A song’s catchiness would be ideally influenced by both its lyrics and instrumentals, so the type-token ratio metric calculated may only capture half of the catchy aspect we were searching for. If all of these songs were purely instrumental (no lyrics at all), we assume that their popularity scores would drastically differ, so it would be difficult to model popularity solely based on linguistic variables.</li></ul><h4><strong>Advantages &amp; Limitations of the Approach</strong></h4><p>This model is much more simple than others used in the project since it is meant to be highly interpretable. However, linguistic variables can become quite complex and uninterpretable when transformations are applied to them due to the nature of human-generated language (in comparison to computer-generated language). Thus, this model is meant to focus on analyzing and interpreting this specific set of linguistic data and isn’t meant to be highly accurate or applicable to all sets of song lyrics.</p><p>While we believe that the models were a good fit, we would have a more holistic idea of how catchyness contributes to song popularity if we could observe how the combination of both ‘linguistic’ and ‘instrumental’ variables (bpm or danceability score) influences popularity scores.</p><p>The errors encountered while conducting this analysis mostly stemmed from having included different languages in each text file. While the AntConc software can work well in multiple languages, it cannot process multiple at the same time. The software can easily determine sentences and phrase-wide linguistic features if it was only looking at Korean text, but because BTS lyrics include both English and Korean words, the software is unable to compute complex linguistic analysis like identification of relative or complement clauses. Consequently, this limitation can make tokenizing and concordance slightly different from its true counterpart — as identifying strings of tokens when multiple languages are present in a text file, and searching for specific strings of tokens when multiple languages are present in a text file both prove difficult. Finally, the website the lyrics were gathered from weren’t always completely tokenized. Thus, we had to tokenize the lyrics manually even though we were not fluent in Korean. Therefore, the tokenization of Korean is likely to be slightly incorrect even though we did our best.</p><h3>Conclusion</h3><p>We performed analysis on both audio features and lyric texts in this project, which are essentially the two main components of every song. While we were hoping to determine what specific factors strongly contribute to a K-pop song’s popularity, we were unable to come to a perfect conclusion by the end of the project due to certain limitations.</p><p>Most prominently, we had a working understanding of the meanings and calculations behind Spotify’s popularity and audio feature scores to base our analysis on. Therefore, our limited knowledge of these metrics affected our interpretation of our results since our exploration was heavily reliant on these metrics.</p><p>Likewise, our lyrics dataset had certain issues we tried to circumvent when conducting our sentiment analysis. First, many Korean songs were translated into English. While it may have helped ease our sentiment analysis, the modification leads to a huge loss in the nuances of the songs, such as the rhymes, the catchiness, and the cultural contexts of certain lyrics. Also, our lyrics dataset represented BTS’s entire discography, including several repackaged albums that had repeated songs. Spotify had assigned different popularity scores to these recurring songs, so we had chosen to conduct analysis on the version that had the highest score. By omitting these differences, we may have overlooked additional insights on how Spotify determined a songs’ popularity. In addition, the lyrics data set also had discrepancies in the titles of the songs it presented which made it difficult to merge with other datasets.</p><p>In conclusion, while we can tentatively suggest that lyrics and a collection of audio features do influence a K-pop song’s popularity score, our results may not be as useful in determining the popularity scores of other songs outside our dataset.</p><p>If you would like to explore more about what we did in the project, please feel free to visit our <a href="https://github.com/the-data-science-union/DSU-S2022-BTS">Github page</a>. Thank you very much for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=40284399948f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>