<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[VinBlog - Medium]]></title>
        <description><![CDATA[Name says it all. - Medium]]></description>
        <link>https://medium.com/vinblog?source=rss----4f8d96eefc6e---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>VinBlog - Medium</title>
            <link>https://medium.com/vinblog?source=rss----4f8d96eefc6e---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 03:13:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/vinblog" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Gollum is a pretty vulnerable guy]]></title>
            <link>https://medium.com/vinblog/gollum-is-a-pretty-vulnerable-guy-4e4144a8b243?source=rss----4f8d96eefc6e---4</link>
            <guid isPermaLink="false">https://medium.com/p/4e4144a8b243</guid>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[lord-of-the-rings]]></category>
            <dc:creator><![CDATA[Vinith Misra]]></dc:creator>
            <pubDate>Wed, 18 Apr 2018 15:48:19 GMT</pubDate>
            <atom:updated>2018-04-18T03:17:43.212Z</atom:updated>
            <content:encoded><![CDATA[<p>(Originally published 12/23/2014. Thanks to <a href="http://www.stanford.edu/~rccooper">Roshni Cooper</a> for valuable feedback and suggestions.)</p><p>Watson’s <a href="http://www.ibm.com/smarterplanet/us/en/ibmwatson/developercloud/user-modeling.html">User Modeling</a> technology (try the <a href="http://watson-um-demo.mybluemix.net/">demo</a>!) is a slick piece of software that can estimate personality traits from blocks of text produced by an individual. Here, we (ab)use it to analyze and compare various LOTR characters’ personalities in Tolkien’s novels and Jackson’s films.</p><h3>Background Fluff (and a figure)</h3><p>The <a href="http://www.rottentomatoes.com/m/the_hobbit_the_battle_of_the_five_armies/">Battle of the Five Armies</a> is now in wide release, and Peter Jackson’s interpretation of Middle-Earth appear to be coming to end, for <a href="http://www.salon.com/2014/12/16/the_hobbit_the_battle_of_the_five_armies_peter_jacksons_long_goodbye_to_middle_earth/">better</a> or for <a href="http://www.wsj.com/articles/the-hobbit-review-peerless-fantasy-flawlessly-rendered-1418841781">worse</a>. (Actually, maybe <a href="http://www.newshub.co.nz/home/entertainment/2018/04/peter-jackson-may-produce-amazon-s-reported-1-billion-lord-of-the-rings-series.html">not</a>.) While many of the discussions about LOTR these days tend to focus on the plot elements, on the lore, or on Jackson’s love of CGI, I am more in seeing what he did with the <em>characters</em>.</p><p>Over the course of Jackson’s six films and Tolkien’s many writings, LOTR fans have <a href="http://churchm.ag/lord-of-the-rings-personality-types/">interpreted</a> the characters’ personalities in <a href="http://lotrproject.com/quiz/">various</a> <a href="http://www.matthewbarr.co.uk/lotr/">ways</a>. Additionally, <a href="http://gary.appenzeller.net/TheHobbitDiff.htm">some</a> very strong <a href="http://badassdigest.com/2013/01/09/why-jrr-tolkiens-son-hates-what-peter-jackson-has-done/">opinions</a> have been <a href="http://www.worldcrunch.com/culture-society/my-father-039-s-quot-eviscerated-quot-work-son-of-hobbit-scribe-j.r.r.-tolkien-finally-speaks-out/hobbit-silmarillion-lord-of-rings/c3s10299/#.VJZgnMABxX">formed</a> about the <a href="http://lotr.wikia.com/wiki/Tolkien_vs._Jackson:_Differences_Between_Story_and_Screenplay">changes</a> introduced by Jackson, Walsh, Boyens, del Toro, and Sinclair in the various live action LOTR films. While many of these analyses focus on plot and lore deviations by the filmmakers, there has also been some <a href="http://www.councilofelrond.com/moviebook/">attention</a> on changes to the underlying characters.</p><p>A common observation, for instance, is that film-Gandalf is significantly more <a href="http://www.lotrplaza.com/archives/index.php?Archive=First%20Age&amp;TID=611">likeable</a> whereas book-Gandalf is significantly more all-knowing.</p><p>In other cases the comparison is less clear. For instance, does book Aragorn have <a href="http://www.councilofelrond.com/moviebook/aragorn-2/">more</a> self-doubt, or <a href="http://www.lotrplaza.com/showthread.php?15079-Aragorn-in-the-movies">less</a>? Debate <a href="http://www.thetolkienforum.com/archive/index.php/t-16969.html">rages</a> in the forums. Just how alpha is Aragorn?</p><p>So what does the data have to say about this? And to be clear, there is no shortage of data: we have easy access to every line of dialog spoken by every character in every book and every film.</p><p>(For those who haven’t yet seen it, know that this isn’t the first time someone has taken a data-driven appraoch to LOTR. Emil Johannson’s <a href="http://lotrproject.com/">LOTRproject</a> is jaw-dropping for both its scope and its aesthetics.)</p><h4>The technology</h4><p>Enter IBM’s Watson <a href="http://www.ibm.com/smarterplanet/us/en/ibmwatson/developercloud/user-modeling.html">user modeling services</a>. After all, who better to ask about understanding people than Watson? Trained on large corpuses of text data and correlated with personality information, user modeling can estimate an individual’s personality traits based on a collection of text he/she wrote. You can actually play around with it <a href="http://watson-um-demo.mybluemix.net/">yourself</a>.</p><p>Be prepared for a surprise! It can often identify personality characteristics that even humans have trouble noticing.</p><p>To illustrate, I harvested 2 years of LOTR-related twitter data (<a href="https://blog.twitter.com/2014/building-a-complete-tweet-index">thanks</a> to Twitter’s recently completed public index!), and estimated the average personality spectrum of a person who tweets about LOTR.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XJpMZ3pk_ssL_tNx-mSrLw.png" /><figcaption>Here’s what Watson has to say about people who tweet about #LOTR (all numbers are percentiles against the general twitter population).</figcaption></figure><p>Underlined in red are the three most emphasized personality facets (imagination, authority-challenging, and intellect) and the most prominent need (self-expression). Imagination, intellect, and self-expression each have obvious links to the LOTR fandom, and require no explanation. It’s cool that the personality analytics machinery is able to pick this out.</p><p>The need to challenge authority may come as a surprise, but it is worth noting that there is a <a href="http://en.wikipedia.org/wiki/The_Lord_of_the_Rings#Impact_on_popular_culture">historical link</a> between LOTR and rebelliousness.</p><h3>Character spectrum</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dA62xxz_ATFCt2Lm9aNHwA.jpeg" /><figcaption>A difficult-to-read illustration of character traits emphasized in the films vs the books.</figcaption></figure><p>This first graphic compares characters to one another on a variety of traits. This analysis is based on the lines they speak in the LOTR trilogy and the first two Hobbit films (not counting the <a href="https://www.youtube.com/watch?v=Sl7w2Z0vGpA">Russian</a> version). Some of the less interesting traits and characters are filtered out, along with those characters who have insufficient data for a particular trait. (See the bottom of this post for more technical details.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-wC0Y5MzBDtl3rz_1MwuhQ.jpeg" /></figure><p>One can see in this blowup —unsurprisingly — that Bilbo and Gollum are diametric opposites in terms of self-consciousness.</p><p>More interestingly, Thorin isn’t too far off from Gollum. He may have better personal hygiene, but Watson confirms for us that he doesn’t exactly suffer from social anxiety.</p><h4>Some additional tidbits from the graphic:</h4><ul><li>Gollum is stupid, friendly, and — this one is interesting — surprisingly vulnerable.</li><li>Gimli and Aragorn sit at opposite ends of the happy/sad divide.</li><li>Despite his relatively sparse dialog, Legolas displays some extraordinarily strong elf-like characteristics: guarded, fearless, distrustful, calm, and stable.</li><li>As compared to Frodo, Sam tends to be slightly friendlier and slightly less depressed. He also has a stronger tendency towards cooperation.</li></ul><h3>Jackson vs. Tolkien</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kQG-xsn9U3oEldByrPSIjA.jpeg" /></figure><p>We can also compare Tolkien’s characters to their film versions. Traits in red have significantly increased emphasis in the films, and traits in blue have significantly increased emphasis in the books. Again, characters and traits with insufficient supporting data have been filtered out, as have some less interesting characters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/891/1*1pYvgcOPmEGHhg8qjJapmg.jpeg" /></figure><p>Readers have <a href="http://forums.theonering.com/viewtopic.php?t=87533">noted</a> in the past that book Gandalf is all confidence and power as compared to his film counterpart. Watson confirms that his character was indeed “nerfed” by Jackson: going from book to film, extraversion and assertiveness go down while cautiousness and neuroticism go up.</p><p>A few more observations:</p><ul><li>Book Denethor’s more regal qualities are diminished in the films, while his moodiness and anxiety is emphasized.</li><li>Legolas has an artistic, eloquent streak to him in the books. This is muted by Jackson, who prefers a competent and forthright Legolas.</li><li>Aragorn is indeed less alpha in the films than in the books.</li></ul><h3>Technical Info</h3><h4>Extracting character data</h4><p>It is not particularly difficult to extract character dialog from any of the films. Film scripts and transcripts are readily available online for both the <a href="http://www.ageofthering.com/atthemovies/">original trilogy</a> and the <a href="https://docs.google.com/document/d/1oxKorXTe8HNWnle-7qmPs3OEFrgPYSnor0bsfpoCT1U/edit?usp=sharing">first</a> and <a href="https://docs.google.com/document/d/1tehSv_j4NxmkGiBlkjPQH7tphfJzFL5v6lXQl08xz-U/edit?usp=sharing">second</a> Hobbit films.</p><p>As a byproduct of rigid screenplay structuring, you can pull dialog and identify speakers with pretty straightforward regular expressions.</p><p>Tolkien’s books are just as readily <a href="http://www.ae-lib.org.ua/texts-c/tolkien__the_lord_of_the_rings_1__en.htm">available</a>. However, novels aren’t as clearly structured as screenplays, and this makes for a more complex dialog extraction task.</p><p>Thankfully, Tolkien follows the discipline of having at most one character speaking in any given paragraph. He follows several other rules as well: in what appears to be more than 95% of cases, the speaker is explicitly identified with a phrase such as “Bombadil said” or “Bombadil whispered.”</p><p>It is very much possible to estimate the remaining 5% of cases by identifying the most recently-mentioned individual prior to a particular dialog. However, this process does introduce some misattributions. Erring on the side of precision over recall, we simply throw out these 5% of cases. This does reduce the sample size by a bit, but since we focus on characters with a significant amount of dialog, the effect is far less catastrophic than assigning lines from one character to another.</p><h4>Character Spectrum</h4><p>Given a collection of text produced by an individual, Watson User Modeling produces personality scores for each of a few dozen personality “facets.” These scores, however, are only meaningful in relation to the personality scores from other individuals.</p><p>For instance, the percentage-valued scores produced by the <a href="http://watson-um-demo.mybluemix.net/">demo</a> are percentiles against a background population of individuals. As such, instead of reporting the raw scores, I report the <em>ranking</em> of characters for each personality trait. This effectively normalizes both for distributional differences between traits (e.g. the depression facet appears to have much more of a spread than neuroticism), and general stylistic differences between dialog written for film vs novels (e.g. characters on screen tend to be more extraverted).</p><p>Only four characters are reported for each trait. This serves two purposes. First, there is oftentimes both a lack of evidence-for and a lack of evidence-against a particular trait for a particular character. The estimated score in such cases is less reliable. Secondly, and perhaps more importantly, the results are surprisingly hard to digest in their full form due simply to information overload. The rankings I report are in some sense the most interesting insights contained in Watson’s raw data.</p><p>In selecting the “interesting” results, there is a very real danger of multiple comparisons and the <a href="http://www.slate.com/articles/health_and_science/science/2013/07/statistics_and_psychology_multiple_comparisons_give_spurious_results.html">garden of forking paths</a>. The limited set of characters considered protects us a bit against this, but to be safe we include a fairly large number of results in the figures produced.</p><h4>Jackson vs. Tolkien</h4><p>In comparing the film to the novels, it is tempting to simply compare for each character the raw personality scores produced in each setting. This is not without danger, however. There are systemic differences in style of writing between screenplays and novels. As noted above, characters on film may tend to score higher in extraversion than those on the page.</p><p>Additionally, there exist fundamental differences in Tolkien’s style of writing as compared with that of Jackson et al. Many of these differences (e.g. the extraversion shift) should *not* be recorded as character changes, because audience members mentally compensate for a global shift of this sort.</p><p>The solution, as with the character rankings, is to measure personality facets as a ranking amongst characters. This has a normalizing effect: if Sam is the most cheerful character in the books and the second least cheerful character in the films, this indicates a shift in his cheerfulness as perceived by a reader/viewer.</p><p>I only report those personality traits whose ranking shifts significantly between page and screen for a given character. More specifically, let <em>C</em> be a character, let <em>x_C^T</em> be the character’s book rank for trait <em>T</em>, and let <em>y_C^T</em> be the character’s film rank for trait T. I report T as being more emphasized in the movies (red) if <em>y_C^T — x_C^T &gt; \theta</em>, and being more emphasized in the books (blue) if <em>y_C^T — x_C^T &lt; -\theta</em> , where <em>\theta</em> is a threshold parameter (set to 2 in this experiment).</p><p>As with the character spectrum, the traits that are reported are further filtered by noise considerations and interest.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4e4144a8b243" width="1" height="1" alt=""><hr><p><a href="https://medium.com/vinblog/gollum-is-a-pretty-vulnerable-guy-4e4144a8b243">Gollum is a pretty vulnerable guy</a> was originally published in <a href="https://medium.com/vinblog">VinBlog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>