<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Agnieszka Sitko on Medium]]></title>
        <description><![CDATA[Stories by Agnieszka Sitko on Medium]]></description>
        <link>https://medium.com/@agnieszka.sitko_69843?source=rss-2023fa5edace------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*qeRMy5G6zF4PGWeaRXGjiA.jpeg</url>
            <title>Stories by Agnieszka Sitko on Medium</title>
            <link>https://medium.com/@agnieszka.sitko_69843?source=rss-2023fa5edace------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 21:00:27 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@agnieszka.sitko_69843/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[AI is coming: a short story of emotion recognition starring Game of Thrones | Tooploox]]></title>
            <link>https://medium.com/@agnieszka.sitko_69843/ai-is-coming-a-short-story-of-emotion-recognition-starring-game-of-thrones-tooploox-aca0c51b8120?source=rss-2023fa5edace------2</link>
            <guid isPermaLink="false">https://medium.com/p/aca0c51b8120</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[emotions]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[game-of-thrones]]></category>
            <dc:creator><![CDATA[Agnieszka Sitko]]></dc:creator>
            <pubDate>Tue, 28 May 2019 11:25:09 GMT</pubDate>
            <atom:updated>2019-06-12T15:06:46.349Z</atom:updated>
            <content:encoded><![CDATA[<h3>AI is coming: a short story of emotion recognition starring Game of Thrones</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nX9ngPdVitquSNwb.jpg" /></figure><p>Game of Thrones Season 8 has just come to an end, and it was emotional! How emotional? Using face detection and emotion recognition we analyzed which scenes evoke certain reactions among fans of the series. In this post, we are presenting how to measure emotions of over 200 viewers spending no money on surveys. And how to close a project in a week!</p><h3>Dataset</h3><p>You have probably already noticed wide popularity of Game of Thrones. It has over 1.5 million rates on IMDB! What is also interesting is that videos of people reacting to the show have gained incredible popularity on the internet — there are even dedicated YouTube channels publishing fans’ reactions with clips reaching millions of views!</p><p>This fascinating phenomenon enabled our AI team to collect a dataset of 18086 frames with subjects reacting to various moments of the final season of Game of Thrones.</p><h3>Methodology</h3><p>Having the dataset, we began <em>data cleaning — </em>the beloved phase of all data scientists and machine learning engineers.</p><p>As one may expect, YouTube videos tend to be quite irregular in their structure. Typically, in a reaction video there are viewers sitting on a couch and a frame from the movie of interest as an overlapping layer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7zA__6Oh3O3pTqcQ" /><figcaption>A typical frame from a reaction video (although we’re not sitting on a couch ;))</figcaption></figure><p>What is even more challenging, viewers may have some additional comments they share while pausing the show. As they were of no interest to us, we decided to ignore those fragments and filter only the parts of YouTube videos which contain scenes of Game of Thrones.</p><p>The next steps were to cut out the overlapping frame and match the frame with the scene in the corresponding episode. In parallel, we combined two pretrained solutions: face-detection and emotion recognition models.</p><p>To sum this up, the established pipeline was as follows:</p><ol><li>GoT frame detection</li><li>Frame matching</li><li>Face detection</li><li>Emotion recognition</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aTFXxssSGI5R62iRzT-hbQ.png" /></figure><h3>GoT frame detection</h3><p>How difficult can it be to detect rectangles on an image? Well… it can be nearly impossible if there are no actual rectangles at all. Edges between a YouTube video’s background and overlaid GoT scenes are visible only due to the optical illusion called <a href="https://en.wikipedia.org/wiki/Illusory_contours">illusory contours </a>. In this task, we had to rely a little bit on the incredible vision of humans and make some annotations. Thankfully, it was enough to manually mark coordinates only on a single frame per YouTube video. It was also a great opportunity to get more familiar with the data.</p><p>Having the coordinates, we could use <a href="https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_imgproc/py_canny/py_canny.html">Canny Edge Detection</a> and <a href="https://docs.opencv.org/2.4/doc/tutorials/imgproc/imgtrans/hough_lines/hough_lines.html">Hough Line Transform</a> to analyze every video frame by frame and check if we’re dealing with a shape limited by horizontal and vertical lines or a rectangle as some may call it. This way, using classical Computer Vision algorithms, we were able to not only extract GoT scenes but also detect fragments of videos where GoT wasn’t displayed. Perfect!</p><h3>Frame matching</h3><p>Are you waiting for Deep Learning? Not yet! We used a <a href="http://www.hackerfactor.com/blog/index.php?/archives/529-Kind-of-Like-That.html"><em>difference hash </em></a>to encode each frame from Game of Thrones as a 64-bit number and map it to the time when it was displayed. We created a mapping like that for each episode separately.</p><p>Now, to perform the actual matching of a frame extracted from a YouTube video we encode it using the same <em>difference hash </em>function and search for the most similar one in the previously created dictionaries.</p><p>If you wonder how we define the similarity, it’s Hamming distance between bits represented as zeros and ones.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*B20LrF3CsPn7TswW" /><figcaption>Frame matching with a difference hash and Hamming distance</figcaption></figure><h3>Smoothing Time Series Data</h3><p>Algorithms (like people) are sometimes imperfect. First two steps produced good results but they were a bit noisy. Below you can see an example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/398/0*dy8PyL6I7Jff3Pmo" /></figure><p>On the x axis we plot the time in a YouTube video and on the y axis the time in an episode of Game of Thrones. As long as authors of YouTube videos don’t edit them in a crazy manner, we would expect the scenes from GoT to be present in the same order they appear in the show. Having said so, we knew we needed a straight line here and we didn’t want to affect correctly matched points. We managed to create such a method for smoothing our data by deleting anomalies. Just look at the results!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/398/0*keEK6zrzE-WzCvMZ" /></figure><h3>Face detection</h3><p>As we wanted to determine emotions expressed by each person in the video, we actually needed to … inspect their faces. We extracted 18086 frames with GoT viewers in them .It means that we had to automate the whole process of face detection — it was too daunting to do it manually. We used an already trained face detection model that allowed us to extract a rectangle with a face from each frame. We used a <a href="https://github.com/ageitgey/face_recognition">well-established Python library</a>, which made the whole process fast and seamless . We could choose between HOG features-based detector or Convolutional Neural Network. We had 8x 1080Ti GPUs available so we went for deep learning without any hesitation.</p><h3>Emotion recognition</h3><p>Once we detected faces of people watching the episode in the YouTube video, it was finally time to detect their emotions. We used the model already trained on popular FER 48×48 pixel grayscale images of faces annotated with one of seven emotions -anger, disgust, fear, happiness, sadness, surprise, neutral. We used <a href="https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge/data">dataset available on Kaggle</a> . It consists of <a href="https://github.com/abhijeet3922/FaceEmotion_ID">model publicly available on GitHub</a> that was accompanied by a thorough explanation so we had a lot of trust in the model itself .</p><p>Oh, and if you want to know how AI understands human emotions, you might be interested in by Ivona, our AI researcher.</p><h3>Results</h3><p>Almost done! The time has come to see what we’ve got. Analyzing the emotional profile of GoT viewers alone would have been great but we decided to go a step further. We found on GitHub including the exact times when a given character appears in the show. We couldn’t resist!</p><h3>Viewers aversion to Daenerys increased over time</h3><p>You may get mad from time to time but you must agree that slaughtering an entire city wouldn’t make you a very likable person. The figure below clearly shows an upward trend of aversion to Dany. If she only had Data Science she would know before it was too late…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jc6AzZiF6bUkpk9AucHMWg.png" /></figure><h3>Chances are the dragons made you happy</h3><p>Scary or not, for some reason the dragons often appeared in scenes that made people laugh. Was the dragon looking at John and Dany kissing funny or was it his sigh expressing pure disapproval? We don’t know, but dragons appear among the top characters causing happiness. The figures below present characters who were evoking a given emotion the most.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6l5G5qWz9PVofmCimdTekg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fOMRmrbs23PzZH81zdUthA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ocL-mduhPBZ-snt2M4myrA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BLNczU4Oy9XVNnTtmKmpvA.png" /></figure><h3>Going two-dimensional</h3><p>By adding a second dimension, we can see not only how often a character evoked a given emotion but also how “emotionally intense” he or she was in general in the whole season.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*it8jmlTqh0Y22xts2d7XbQ.png" /></figure><h3>Final remarks</h3><p>So, here we are — we hope you are convinced that it is possible to deliver a successful project in a week, having a lot of fun along the way. If you want to follow in our footsteps, consider our protips that we learned on the go:</p><ol><li>Use what is already available before working on custom solutions.</li><li>Better done than perfect — time boxing is the king.</li><li>Smart division of tasks and parallelization is a must when you want to deliver something fast, but you need the right team to benefit from that.</li><li>Be realistic and critical. We’re aware that there are many biases in our analysis. For example, people tend to be happier when watching movies with their friends.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rqOBYZ6Mff8tQd8tHEmyTg.png" /></figure><p>This post was written by joint efforts of Agnieszka Sitko, Adam Słucki &amp; Dominika Basaj.</p><p><strong>Wanna discuss emotion recognition or any other AI-related topic? </strong><a href="/service/mailto:business@tooploox.com"><strong>Let’s talk!</strong></a></p><p><em>Originally published at </em><a href="https://www.tooploox.com/blog/ai-is-coming-a-short-story-of-emotion-recognition-starring-game-of-thrones"><em>https://www.tooploox.com</em></a><em> on May 28, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aca0c51b8120" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>