<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Marton Nagy on Medium]]></title>
        <description><![CDATA[Stories by Marton Nagy on Medium]]></description>
        <link>https://medium.com/@nagymarton000?source=rss-310b89481867------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*VoqTZSGV0LaUfVbJ</url>
            <title>Stories by Marton Nagy on Medium</title>
            <link>https://medium.com/@nagymarton000?source=rss-310b89481867------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 17:39:29 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nagymarton000/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Ryanair’s online reputation is suffering. But does this affect the share price?]]></title>
            <link>https://medium.com/@nagymarton000/ryanairs-online-reputation-is-suffering-but-does-this-affect-the-share-price-bf0316a2ada4?source=rss-310b89481867------2</link>
            <guid isPermaLink="false">https://medium.com/p/bf0316a2ada4</guid>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[streamlit]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[sentiment-analysis]]></category>
            <category><![CDATA[ryanair]]></category>
            <dc:creator><![CDATA[Marton Nagy]]></dc:creator>
            <pubDate>Thu, 19 Dec 2024 20:54:19 GMT</pubDate>
            <atom:updated>2024-12-25T21:06:13.318Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4vKXS7aYR3dzFCbx" /><figcaption>Photo by <a href="https://unsplash.com/@portuguesegravity?utm_source=medium&amp;utm_medium=referral">Portuguese Gravity</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><em>This article has been compiled as part of my final assignment for the Web Scraping with Python course at Central European University. The source code is hosted on </em><a href="https://github.com/marton-nagy-marton/WebScraping_HA3"><em>GitHub</em></a><em>.</em></p><p>It has been shown in multiple scientific articles (see e.g. <a href="https://doi.org/10.1016/j.knosys.2014.04.022">Li et al., 2014</a> or <a href="https://doi.org/10.1016/j.eswa.2015.07.052">Nguyen et al., 2015</a>) that share prices of certain stocks may be well predicted using the online sentiments associated with the company behind the stock. However, most of these articles use either social media posts or news articles to track the public opinion on certain companies. There is, however, a third source which may a be good indicator of the public image of a company: online rating and review aggregation sites.</p><p>To test this idea, I scraped all English-language reviews of Ryanair from TrustPilot, and analysed their association with Ryanair’s share price (RYAAY). I also included some control variables on Ryanair’s performance scraped directly from Ryanair’s website, such as number of passangers per monthly and monthly load factor data.</p><p><em>Keywords: web scraping, Python, TOR, API, sentiment analysis, Streamlit, Ryanair, TrustPilot</em></p><h3>Part 1: Scraping the data</h3><p>To get all the necessary data, we need to get information from three distinct sources: TrustPilot’s site for the reviews, Yahoo Finance for the share price, and Ryanair’s site for the control variables. This chapter will guide you through the process of getting the data.</p><h4>Part 1.1: Scraping reviews from TrustPilot</h4><p>To scrape data from <a href="https://www.trustpilot.com/review/www.ryanair.com">Ryanair’s TrustPilot page</a>, I mimic the API call my browser makes every time it loads a new page. I extracted the exact call from the Network tab of Chrome, then I wrote a function in Python that scrapes one page and returns the reviews as dictionary. One interesting thing I had to do was rotating my user agent header to avoid getting banned after only scraping a few pages.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f713d4c54d4b6f6e227b145bf6fbea83/href">https://medium.com/media/f713d4c54d4b6f6e227b145bf6fbea83/href</a></iframe><p>However, if we were to scrape the site by only calling this function for every page, we would still get blocked after a certain point (in my experience, after around a 100 pages). To bypass this issue, I use the TOR-network to get a new session and IP after every 50 pages scraped. To set this up, I first had to include some additional info in my torrc file (of course, for everything to work, you would need to replace the data with your locally appropriate inputs).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e4797c7fc9a2e57d60cc528fde137ba4/href">https://medium.com/media/e4797c7fc9a2e57d60cc528fde137ba4/href</a></iframe><p>With this set up, I defined two functions for renewing my TOR-IP and for creating a new TOR-session (of course, you have to have your proper environmental variables in place for this to run). Note, that in order for these functions to work, you need to have your local TOR-browser open and connected to the TOR-network.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2e0a18322b377c7746c638ead4d2b1db/href">https://medium.com/media/2e0a18322b377c7746c638ead4d2b1db/href</a></iframe><p>Now everything is set up to scrape the actual website. I accomplish this with a simple while loop that runs until it either gets an error or an HTTP response other than 200. When I executed the code, it stopped after it had got an HTTP 404. We can be happy about this, as this means that it scraped all the pages, until there was none left. All in all, it scraped 14,736 reviews.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ee72388252c41f2fc503555eb6c3227e/href">https://medium.com/media/ee72388252c41f2fc503555eb6c3227e/href</a></iframe><p>Now we need to parse the reviews into a tidy DataFrame. I accomplish this by setting up a parsing function, and mapping it to the scraped raw reviews. After that, I also do some type conversions, and I create some new columns from the dates. Also, note that I scaled the rating of each review to a 0–1 scale for easy comparability with sentiment scores (calculated later).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/eac970a0c98fb27d326c57c895545991/href">https://medium.com/media/eac970a0c98fb27d326c57c895545991/href</a></iframe><h4>Part 1.2: Getting the stock price data from the GoldHand package</h4><p>Getting historical price data for the RYAAY ticker was relatively simple, as I could simply use the <a href="https://github.com/misrori/goldhand/tree/main">GoldHand package</a> to seamlessly scrape data from the Yahoo Finance API. With the resulting DataFrame, I again do some type conversions and create new variables from the date.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/89a99c9d46bf79462596ceaea4582cde/href">https://medium.com/media/89a99c9d46bf79462596ceaea4582cde/href</a></iframe><h4>Part 1.3: Getting control variables from Ryanair’s site</h4><p><a href="https://investor.ryanair.com/traffic/">This site</a> presents monthly data for investors about Ryanair’s number of passengers (guests) and its load factor. I simply scrape the HTML code of this site. Scraping this is rather easy, as we can simply find all HTML tables, and read them directly into Pandas. Then we can clean up the DataFrame using Pandas. The clean-up process is a bit tedious, as the tables are not quite well structured. Also, I had to pay special attention to assign dates for the rows, as the website presents data for financial year months, which starts in April for Ryanair.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8ab9f0a6fd2fd94b61fb058e9f243384/href">https://medium.com/media/8ab9f0a6fd2fd94b61fb058e9f243384/href</a></iframe><h3>Part 2: Data manipulation and exploratory analysis</h3><p>It turned out, that even though I have scraped reviews until as far as 2009, years before 2018 usually had very small number of reviews. So, I dropped all observations before 2018 from every data table.</p><p>Before we can join the three datasources, we still need to calculate sentiment scores for all of the reviews. I do this with the nltk.sentiment.SentimentIntensityAnalyzer function, using the VADER dictionary. I scale the sentiments from -1 to +1 to a 0–1 scale.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/78ee12ee6ca3677917b94599073cfafa/href">https://medium.com/media/78ee12ee6ca3677917b94599073cfafa/href</a></iframe><p>Now we can join the three data tables. For this, I calculate monthly average ratings and monthly average sentiment scores for the reviews, and montly last closing prices for the stock data. Note, that later I will also present some results for data aggregated by week. The calculations for that are analogous to the monthly calculations, thus not presented here. One key difference however was that I could not include the control variables, as they are on a monthly level.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2873d096a9dcda281e2e030262c5b2f4/href">https://medium.com/media/2873d096a9dcda281e2e030262c5b2f4/href</a></iframe><p>Next, I create some dummies for certain external shock factors. Also, I impute the load factor with zeros (as during the lockdown period, there were few missing values) and drop observations where we have any missing data. I also create a quarter variable.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8c371da5a10b5f2d1d17c08e2c57793d/href">https://medium.com/media/8c371da5a10b5f2d1d17c08e2c57793d/href</a></iframe><p>For the exploratory analyis, I plotted all time series in my monthly dataset. As they were far from being stationary, I took the log difference for RYAAY and simple differences for the other variables. As I noticed a strong seasonality in the number of guests, I also calculated a seasonal difference for that variable with a 12-month offset. All of these time series plots are later presented in Part 4. As these differenced variables were more or less stationary (apart from some variance clustering), we can move on to calculating some regression estimates.</p><h3>Part 3: Analysis</h3><p>I estimate 6 models in total for the model, in increasing complexity order:</p><ul><li>Model (1) uses rating and sentiment data only, in a linear spline set-up with a knot at zero;</li><li>Model (2) adds to this the quarter dummies and the external shock dummies;</li><li>Model (3) also includes the monthly number of guests, interacted with the quarter;</li><li>Model (4) uses the seasonal difference of guests instead;</li><li>Model (5) includes the monthly number of guests and the load factor, both interacted with the quarters;</li><li>and Model (6) uses the seasonal difference of guests instead, but also includes load factor interacted with quarter.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*fGeJUxKOStJsSBPW0_hIwQ.png" /><figcaption>Note: FD means first difference</figcaption></figure><p>Interestingly, most of our models agree that there is a relationship between the log return of RYAAY and monthly average ratings and sentiments. For the interpretation, remember that both of these variables were scaled to a 0–1 scale before. Based on the below models, we can say the following:</p><ul><li>when the change in average ratings is negative, there is usally no significant relationship with price,</li><li>for positive changes, a 0.01 unit change in average ratings is associated, on average, with a 1.5–2% increase in the montly returns,</li><li>for negative changes in sentiments, a a 0.01 unit change is associated, on average, with a 0.5–1% increase in the montly returns,</li><li>for positive changes in sentiments, a a 0.01 unit change is associated, on average, with a 1–1.5% decrease in the montly returns.</li></ul><p>As a sort of robustness check, we can also look at weekly level data — but bare in mind that we won’t have the control variables scraped from Ryanair’s website, so we can only estimate Model (1) and (2) from the above listed set-ups.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/707/1*0hBfX1iYYtpmFmcigLXA9w.png" /><figcaption>Note: FD means first difference</figcaption></figure><p>What we can see here, is that neither of the variables, which were significant in the monthly models, show up as significant here. So, our results turned out to be not quite robust, and we should not really use them for making decisions on whether to buy or sell Ryanair stock.</p><h3>Part 4: Visualization dashboard</h3><p>To help understanding the results, below you can find a Streamlit dashboard built on the compiled database. The dashboard has the following functionalites:</p><ul><li>on the sidebar, you can select whether you want to see time series or regression result plots and whether you are interested in monthly or weekly data;</li><li>if you select time series, than you can select as many variables as you want to see their plots in a grid layout;</li><li>if you select regression results, you can choose whether you want to see log return or level price predictions;</li><li>then you can select all the models that you are interested in, which will be plotted on the same chart, against actual log returns or level prices.</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fryaay-stock-vs-trustpilot-reviews.streamlit.app%2F%3Fembed%3Dtrue&amp;display_name=Streamlit&amp;url=https%3A%2F%2Fryaay-stock-vs-trustpilot-reviews.streamlit.app&amp;image=https%3A%2F%2Fstorage.googleapis.com%2Fs4a-prod-share-preview%2Fdefault%2Fst_app_screenshot_image%2F51c93d7b-14b5-4a5e-ab0c-2bc6e13bad82%2FHome_Page.png&amp;type=text%2Fhtml&amp;schema=streamlit" width="560" height="630" frameborder="0" scrolling="no"><a href="https://medium.com/media/9810f2d05b9213bd95a62ee25a234686/href">https://medium.com/media/9810f2d05b9213bd95a62ee25a234686/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bf0316a2ada4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The reputation of WizzAir is at an all-time low — but why?]]></title>
            <link>https://medium.com/@nagymarton000/the-reputation-of-wizzair-is-at-an-all-time-low-but-why-18298b92fa05?source=rss-310b89481867------2</link>
            <guid isPermaLink="false">https://medium.com/p/18298b92fa05</guid>
            <category><![CDATA[airlines]]></category>
            <category><![CDATA[aviation]]></category>
            <category><![CDATA[trustpilot-reviews]]></category>
            <category><![CDATA[wizzair]]></category>
            <category><![CDATA[sentiment-analysis]]></category>
            <dc:creator><![CDATA[Marton Nagy]]></dc:creator>
            <pubDate>Fri, 06 Dec 2024 12:19:57 GMT</pubDate>
            <atom:updated>2024-12-06T12:21:20.770Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cod3g3pRlTrfHUd9" /><figcaption>Photo by <a href="https://unsplash.com/@dawidtkocz?utm_source=medium&amp;utm_medium=referral">Dawid Tkocz</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>The reputation of WizzAir is at an all-time low — but why?</h3><p><em>This article was written for my third Data Engineering 2 Home Assignment at Central European University.</em></p><p>A company’s public reputation is central piece in its success. But measuring reputation is tricky business. One possible option is to look at review aggregation sites, and calculate the sentiment of reviews for the company. Note that this might be a biased estimate of reputation if only dissatisfied costumers tend to leave reviews — however, it is still worth taking a look at.</p><p>In order to do so, I scraped all reviews on WizzAir from TrustPilot. Then, I run a sentiment analysis (both general and targeted) on reviews translated to English using AWS Comprehend. Lastly, I visualized the results to uncover insights about the evolution of WizzAir’s reputation between 2017 and 2024.</p><h4>Part 1: Scraping reviews from TrustPilot</h4><p>For the scraping, I employed an API based approach. I extracted the API call my browser makes to TrustPilot when loading a new page of reviews from the Network tab, and replicated it in Python. As pagination is simply a parameter of the API call, I could easily loop through the pages until I got a HTTP code other than 200.</p><pre>def get_reviews_page(session, page):<br>    #I rotate randomly my user agent header to avoid being banned.<br>    USER_AGENTS = [<br>        #Windows User Agents (other types ommitted from sample code)<br>        &#39;Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36&#39;,<br>        &#39;Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36&#39;,<br>    ]<br>    headers = {<br>        #[ ... ]<br>        &#39;referer&#39;: f&#39;https://www.trustpilot.com/review/www.wizzair.com?page={page}&#39;,<br>        #[ ... ]<br>        &#39;user-agent&#39;: random.choice(USER_AGENTS),<br>        #[ ... ]<br>    }<br>    #scraping the first page requires a slightly different syntax.<br>    if page == 1:<br>        params = {<br>            &#39;languages&#39;: &#39;all&#39;,<br>            &#39;sort&#39;: &#39;recency&#39;,<br>            &#39;businessUnit&#39;: &#39;www.wizzair.com&#39;,<br>        }<br>    else:<br>        params = {<br>            &#39;languages&#39;: &#39;all&#39;,<br>            &#39;page&#39;: str(page),<br>            &#39;sort&#39;: &#39;recency&#39;,<br>            &#39;businessUnit&#39;: &#39;www.wizzair.com&#39;,<br>        }<br>    try:   <br>        response = session.get(<br>            &#39;https://www.trustpilot.com/_next/data/businessunitprofile-consumersite-2.2560.0/review/www.wizzair.com.json&#39;,<br>            params=params,<br>            headers=headers,<br>        )<br>    except Exception as e:<br>        print(f&#39;Exception: {e}&#39;)<br>        return None<br>    #the function prints the HTTP code is something goes wrong<br>    if response.status_code != 200:<br>        print(f&#39;HTTP: {response.status_code}&#39;)<br>        return None<br>    return json.loads(response.text)[&#39;pageProps&#39;][&#39;reviews&#39;]</pre><p>This all sounds simple, but I encountered the problem, that the API blocked my IP after making a few hundred calls. To find a workaround for this, I employed two strategies:</p><ol><li>I rotate my request header user agent randomly to appear as different systems making the calls;</li><li>And I make the calls through the TOR network, renewing my session after every 50 page scraped.</li></ol><pre>def renew_tor_ip():<br>    with Controller.from_port(port=9052) as controller:<br>        controller.authenticate(&#39;&lt;your tor password would go here&gt;&#39;) #replace password to match your own<br>        controller.signal(Signal.NEWNYM)<br><br>def create_tor_session():<br>    session = requests.Session()<br>    session.proxies.update({<br>        &#39;http&#39;: &#39;socks5h://127.0.0.1:9050&#39;,<br>        &#39;https&#39;: &#39;socks5h://127.0.0.1:9050&#39;,<br>    })<br>    return session<br><br>run = True<br>i = 1<br>reviews_list = []<br>renew_tor_ip()<br>session = create_tor_session()<br>while run:<br>    if i % 10 == 0:<br>        print(f&#39;At page: {i}&#39;)<br>    if i % 50 == 0:<br>        renew_tor_ip()<br>        session = create_tor_session()<br>    t = get_reviews_page(session, i)<br>    if t is None:<br>        run = False<br>        print(f&#39;Stopped at page {i}&#39;)<br>    else:<br>        reviews_list.extend(t)<br>        i += 1</pre><p>This way, I could manage to scrape all reviews on WizzAir: 14,135 in total. Then I parsed the reviews in JSON to a Pandas DataFrame.</p><h4>Part 2: Using AWS Translate and Comprehend to process the reviews</h4><p>Even though I had all the reviews, it would have been quite costly to process all of them using AWS services. So, I decided to create a subsample with 200 randomly chosen reviews from each year between 2017 and 2024 with languages supported by AWS Translate.</p><p>Then, I used AWS Translate to translate all non-English reviews from my subsample, and I assigned the results to a new DataFrame column. The cost of this calculation will be presented at the end of this chapter.</p><pre>translate = boto3.client(&quot;translate&quot;,<br>                         aws_access_key_id=&quot;&quot;, #substitute with your actual keys<br>                         aws_secret_access_key=&quot;&quot;)<br>subsample[&#39;translated_text&#39;] = subsample.apply(lambda x: x[&#39;text&#39;] if x[&#39;language&#39;] == &#39;en&#39; else<br>                                               translate.translate_text(Text = x[&#39;text&#39;],<br>                                                                        SourceLanguageCode = x[&#39;language&#39;],<br>                                                                        TargetLanguageCode = &#39;en&#39;<br>                                                                       )[&#39;TranslatedText&#39;],<br>                                               axis = 1<br>                                              )</pre><p>Having all reviews in English, I could now turn towards sentiment analysis. I run a general and a targeted sentiment analysis on all of the reviews in my subsample using AWS Comprehend. As the results of Comprehend are quite complex JSON objects, I decided to store them in a dictionary and parse later.</p><pre>comprehend = boto3.client(service_name=&quot;comprehend&quot;,<br>                          region_name=&quot;eu-west-1&quot;,<br>                          aws_access_key_id=&quot;&quot;, #substitute with your actual keys<br>                          aws_secret_access_key=&quot;&quot;<br>                         )<br>results = []<br>i = 0<br>ids = list(subsample_translated[&#39;id&#39;])<br>for t in list(subsample_translated[&#39;translated_text&#39;]):<br>    r = {&#39;id&#39; : ids[i], &#39;input&#39; : t}<br>    try:<br>        r[&#39;gen_sentiment&#39;] = comprehend.detect_sentiment(Text=t, LanguageCode=&quot;en&quot;)<br>    except Exception as e:<br>        print(f&#39;Error at text {i}: {e}&#39;)<br>        r[&#39;gen_sentiment&#39;] = None<br>    try:<br>        r[&#39;targ_sentiment&#39;] = comprehend.detect_targeted_sentiment(Text=t, LanguageCode=&quot;en&quot;)<br>    except Exception as e:<br>        print(f&#39;Error at text {i}: {e}&#39;)<br>        r[&#39;targ_sentiment&#39;] = None<br>    results.append(r)<br>    i += 1</pre><p>Before moving on, let’s take a look at how much using these services costed. For Translate, the <a href="https://aws.amazon.com/translate/pricing/">pricing page</a> states that it costs $15 for every 1 million character translated. So, I summed up the length of all the non-English reviews and multiplied it by the cost per character. It turned out, that the translation costed appr. $5.73. Before doing the translation on the whole subsample, I also made some trial calls which costed $0.01 in total.</p><pre>subsample[&#39;translation_price&#39;] = subsample.apply(lambda x: 0 if x[&#39;language&#39;] == &#39;en&#39; else len(x[&#39;text&#39;])*(15/1_000_000), axis=1)<br>subsample.translation_price.sum()<br><br># &gt;&gt; 5.73468</pre><p>For Comprehend, <a href="https://aws.amazon.com/comprehend/pricing/">pricing</a> is $0.0001 per unit for both general and targeted sentiment analysis. One unit means 100 characters, but if we request an analysis for a text less than 3 units long, we still get charged for 3 units at minimum. So, I calculated the unit-length of each translated review (with special attention to the 3 unit minimum price), and concluded that both general and targeted sentiment analysis costed appr. $0.89 each.</p><pre>subsample_translated[&#39;units&#39;] = subsample_translated[&#39;translated_text&#39;].apply(lambda x: 3 if len(x)/100 &lt; 3 else len(x)/100)   <br>gen_sentiment_price = subsample_translated.units.sum()*0.0001<br>targ_sentiment_price = subsample_translated.units.sum()*0.0001<br>print(gen_sentiment_price)<br>print(targ_sentiment_price)<br><br># &gt;&gt; 0.8909870000000001<br># &gt;&gt; 0.8909870000000001</pre><p>The detailed cost breakdown is presented in a table at the end of the post.</p><h4>Part 3: Clustering targeted sentiments</h4><p>As Comprehend identified more than 4,000 unique sentiment targets in the reviews, I had to cluster these before making any analysis. I did an initial clustering with the DBSCAN algorithm.</p><pre>#extracting unique sentiment targets<br>ts_texts = set()<br>for r in parsed_sentiments:<br>    for t in r[&#39;targeted_sentiments&#39;]:<br>        ts_texts.add(t[&#39;text&#39;])<br>ts_texts = list(ts_texts)<br><br>#clustering with DBSCAN<br>vectorizer = TfidfVectorizer(stop_words=&#39;english&#39;)<br>X = vectorizer.fit_transform(ts_texts)<br><br>dbscan = DBSCAN(eps=0.3, min_samples=5, metric=&#39;cosine&#39;)<br>dbscan.fit(X)<br><br>labels = dbscan.labels_<br><br>clustered_sentiments = {}<br>for i, label in enumerate(labels):<br>    if label not in clustered_sentiments:<br>        clustered_sentiments[label] = []<br>    clustered_sentiments[label].append(ts_texts[i])</pre><p>This has then been manually refined to correctly group prices and to have meaningful category names. In the end, I ended up with the following categories: Price, Payment &amp; Refunds; Ticket; Airport; Time &amp; Delay; In-flight Experience; Company; Customer Service; Baggage; and Covid.</p><p>The next thing was to assign sentiment labels to the clusters for each review. I did this by averaging the sentiment scores for all targets that belonged to a certain cluster in the review, and I assigned as a sentiment label the highest scoring sentiment score. Note that Time &amp; Delay turned out to be a bad cluster as it only had neutral targeted sentiments in it, so I omitted it from later analysis.</p><pre>def find_key_by_value(dictionary, search_string):<br>    for key, value_list in dictionary.items():<br>        if search_string in value_list:<br>            return key<br>    return None<br><br>def determine_sentiment(scores):<br>    if scores[&#39;Positive&#39;][0] == 0:<br>        return None<br>    <br>    avg_sentiment = []<br>    avg_sentiment.append(scores[&#39;Positive&#39;][1] / scores[&#39;Positive&#39;][0])<br>    avg_sentiment.append(scores[&#39;Negative&#39;][1] / scores[&#39;Negative&#39;][0])<br>    avg_sentiment.append(scores[&#39;Neutral&#39;][1] / scores[&#39;Neutral&#39;][0])<br>    avg_sentiment.append(scores[&#39;Mixed&#39;][1] / scores[&#39;Mixed&#39;][0])<br>    max = 0<br>    max_index = 0<br>    i = 0<br>    for sent in avg_sentiment:<br>        if sent &gt; max:<br>            max = sent<br>            max_index = i<br>        i += 1<br>    return [&#39;POSITIVE&#39;, &#39;NEGATIVE&#39;, &#39;NEUTRAL&#39;, &#39;MIXED&#39;][max_index]<br><br>def avg_sentiment_by_tg(clusters, review_sentiment):<br>    helper = {<br>        &#39;price_payment_refunds&#39; : {<br>            &#39;Positive&#39; : [0,0],<br>            &#39;Negative&#39; : [0,0],<br>            &#39;Neutral&#39; : [0,0],<br>            &#39;Mixed&#39; : [0,0],<br>        },<br>        # ...<br>        # continues the same way for all other clusters<br><br>    }<br>    for target in review_sentiment[&#39;targeted_sentiments&#39;]:<br>        group = find_key_by_value(clusters, target[&#39;text&#39;])<br>        if group is not None:<br>            helper[group][&#39;Positive&#39;][0] += 1<br>            helper[group][&#39;Positive&#39;][1] += target[&#39;sentiment_score&#39;][&#39;Positive&#39;]<br>            helper[group][&#39;Negative&#39;][0] += 1<br>            helper[group][&#39;Negative&#39;][1] += target[&#39;sentiment_score&#39;][&#39;Negative&#39;]<br>            helper[group][&#39;Neutral&#39;][0] += 1<br>            helper[group][&#39;Neutral&#39;][1] += target[&#39;sentiment_score&#39;][&#39;Neutral&#39;]<br>            helper[group][&#39;Mixed&#39;][0] += 1<br>            helper[group][&#39;Mixed&#39;][1] += target[&#39;sentiment_score&#39;][&#39;Mixed&#39;]<br>    return {<br>        &#39;id&#39; : review_sentiment[&#39;id&#39;],<br>        &#39;general_sentiment&#39; : review_sentiment[&#39;general_sentiment&#39;],<br>        &#39;price_payment_refunds&#39; : determine_sentiment(helper[&#39;price_payment_refunds&#39;]),<br>        &#39;ticket&#39; : determine_sentiment(helper[&#39;ticket&#39;]),<br>        &#39;airport&#39; : determine_sentiment(helper[&#39;airport&#39;]),<br>        &#39;time_delay&#39; : determine_sentiment(helper[&#39;time_delay&#39;]),<br>        &#39;in_flight&#39; : determine_sentiment(helper[&#39;in_flight&#39;]),<br>        &#39;company&#39; : determine_sentiment(helper[&#39;company&#39;]),<br>        &#39;customer_service&#39; : determine_sentiment(helper[&#39;customer_service&#39;]),<br>        &#39;baggage&#39; : determine_sentiment(helper[&#39;baggage&#39;]),<br>        &#39;covid&#39; : determine_sentiment(helper[&#39;covid&#39;])<br>    }<br><br>sentiments_df_prep = []<br>for r in parsed_sentiments:<br>    sentiments_df_prep.append(avg_sentiment_by_tg(clustered_targets, r))<br><br>review_sentiments_df = pd.DataFrame(sentiments_df_prep)</pre><p>Lastly, I created a new DataFrame with all the initial review data, as well as general sentiment labels and targeted sentiment labels for the identified clusters.</p><h4>Part 4: Analysis and Visualization</h4><p>First, I wanted to visualize how the review sentiments evolved over time. To do this, I calculated the share of reviews having a certain general sentiment per year and plotted it on a 100% stacked bar chart. As we can see below, the share of positive reviews has always been quite low (note my remark about the possible bias of review aggregation sites in the beginning), but it has been steadily diminishing in the past few years.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XWdksH1SP-A9YdTbrLWG4w.png" /><figcaption>Source: own calculation</figcaption></figure><p>Second, I wanted to see the relationship between general sentiments and actual ratings. I translated sentiment labels to a sort of pseudo-rating with negative being 1, mixed 2, neutral 3 and positive 4 and 5. I plotted these on a heatmap with the actual ratings. The heatmap reveals that generally, strictly positive or strictly negative reviews tend have a corresponding rating of 1 or 4–5 (but there are a number of reviews which do not follow this pattern). Also, the heatmap shows that mixed and neutral sentiment categories do not really correspond to a certain actual rating band.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9CzxImQzCGJ4uNzsu7RzIQ.png" /><figcaption>Source: own calculation</figcaption></figure><p>Lastly, I wanted to see how the targeted sentiment for the identified clusters evolved over time. For this, I plotted the share of negative sentiment reviews in each cluster by year. Some interesting trends on the chart are that:</p><ul><li>The share of reviews with negative sentiment towards customer service show a slight negative trend;</li><li>The company itself is constantly targeted quite negatively;</li><li>And the negative reviews about tickets show a slight upward trend.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kHwd7Qn9iBU_8ZfSD7ZCFg.png" /><figcaption>Source: own calculation</figcaption></figure><p>A major limitation of this analysis is that it only contains a random subsample of all reviews, which may or may not be representative of all reviews. Also, the clustering of the sentiment targets could be heavily improved by manually assigning clusters to each target.</p><p>Nonetheless, this article sheds some light on the reputation of WizzAir and on which factors contribute the most to it.</p><h4>Appendix: Costs of using AWS</h4><p>The below table presents a detailed cost breakdown of using AWS Translate and Comprehend for this analysis.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f7674cf9a19ff9393f60583cb740f629/href">https://medium.com/media/f7674cf9a19ff9393f60583cb740f629/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=18298b92fa05" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>