<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Aindrila Basu on Medium]]></title>
        <description><![CDATA[Stories by Aindrila Basu on Medium]]></description>
        <link>https://medium.com/@aindrilabose?source=rss-176e81323619------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*XpeGhap4dxvFwq2M</url>
            <title>Stories by Aindrila Basu on Medium</title>
            <link>https://medium.com/@aindrilabose?source=rss-176e81323619------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 21:47:50 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@aindrilabose/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Chat GPT’s Performance on the Winograd Schema Challenge: Exploring the Nuances of Language…]]></title>
            <link>https://medium.com/@aindrilabose/chat-gpts-performance-on-the-winograd-schema-challenge-exploring-the-nuances-of-language-90d16b164d52?source=rss-176e81323619------2</link>
            <guid isPermaLink="false">https://medium.com/p/90d16b164d52</guid>
            <category><![CDATA[knowledge-representation]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[reasoning]]></category>
            <category><![CDATA[winograd-schema-challenge]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Aindrila Basu]]></dc:creator>
            <pubDate>Thu, 23 Mar 2023 11:11:33 GMT</pubDate>
            <atom:updated>2023-03-23T11:11:33.284Z</atom:updated>
            <content:encoded><![CDATA[<h3>Chat GPT’s Performance on the Winograd Schema Challenge: Exploring the Nuances of Language Comprehension</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*vBwb9-ZWxRbjwXJJKkfS2w.png" /></figure><h3>Introduction</h3><p>The interpretation and processing of human language has advanced significantly in recent years due to Natural Language Processing (NLP). One of the benchmarks for evaluating the performance of NLP models is the Winograd Schema Challenge, which tests a model’s ability to understand contextual nuances and resolve ambiguities in language. This post will evaluate the performance of ChatGPT on the Winograd Schema Challenge.</p><h3>What is the Winograd Schema Challenge?</h3><p>The Winograd Schema Challenge is a test of an intelligent system and an alternative to the Turing Test.</p><p>It is essentially a set of sentence pairs that differ by a single word known as the special word, where the meaning of the sentence depends on the choice of that word. The sentences have certain features:</p><ol><li>Two entities are mentioned in the sentence. The entities can be two males, two females, two objects or two groups of people.</li><li>There is a pronoun in the sentence, typically he/him/his, she/her/hers,<br>they/them or it. The pronoun is used to refer to one of the entities, but it is also suitable to mention the other entity.</li><li>The test requires understanding which entity the pronoun is referring to. The first answer option is usually the first entity and the second option is the second entity.</li><li>Finally, there is a special word in every sentence as well as the question. When this word is replaced by an alternate word, the sentence is still meaningful, but the response to the question changes.</li></ol><p>The goal of the challenge is to correctly answer which entity the pronoun is referring to when either the special or alternate word is used in the sentence.</p><p>The original sentence pair proposed by Terry Winograd was:</p><p>The city councilmen refused the demonstrators a permit because they<br>[feared/advocated] violence.</p><p>The pronoun <strong>they </strong>can refer to either the <strong>city councilmen</strong> or the <strong>demonstrators</strong>, based on whether we choose the word <strong>feared </strong>or <strong>advocated</strong>.</p><h3>ChatGPT’s Performance on the Winograd Schema Challenge</h3><p>ChatGPT is a large language model developed by OpenAI, based on the GPT (Generative Pre-trained Transformer) architecture. It is designed to generate human-like text in response to a given prompt or question. But does it possess human-like intelligence to comprehend the nuances of the language? Lets find out!</p><p>ChatGPT was evaluated on the first 20 questions from the <a href="https://cs.nyu.edu/~davise/papers/WSOld.html">Collection of Winograd Schemas</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/822/1*4IfePgea3DLZdjaGYswpYQ.jpeg" /><figcaption>Response generated by ChatGPT to the first two questions.</figcaption></figure><p>ChatGPT performed well in the first few questions, which were easy to comprehend. As the level of questions became more difficult, the performance of ChatGPT deteriorated. Although ChatGPT is capable of many intelligent tasks, including generating text on any topic, it is not able to completely comprehend the nuances of the English language.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/890/1*COzNT84FxCbgUV2CSVelwA.jpeg" /><figcaption>Incorrect response generated by ChatGPT</figcaption></figure><p>Lets consider the sentence:</p><p>Tom threw his schoolbag down to Ray after he reached the bottom of the stairs.</p><p>And the question:</p><p>Who reached the bottom of the stairs?</p><p>According to ChatGPT, ‘Tom reached the bottom of the stairs…’, which is incorrect. ChatGPT considers the first subject, ‘Tom’ to be the referent of the pronoun ‘he’, and is unable to comprehend that the schoolbag can only be thrown up from the bottom of the stairs.</p><p>In a situation where some background knowledge is required to answer questions correctly Chat GPT does not perform good enough and ended up only answering 10 out of 20 questions correctly, with a 50% accuracy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*XXZ-GIWf9FkS1-ausl0FCA.jpeg" /><figcaption>Incorrect response showing background knowledge is necessary to answer Winograd Schema questions correctly</figcaption></figure><p>Lets consider another sentence:</p><p>The large ball crashed right through the table because it was made of styrofoam.</p><p>To answer the question correctly, one needs to have prior knowledge of what styrofoam is and whether or not it is heavy or light. Since we, as humans, are aware that styrofoam is light, we could easily comprehend that ‘it’ refers to the ‘table’ and not the ‘ball’.</p><p>There is no doubt that ChatGPT can perform many intelligent tasks, like summarizing text, suggesting captions, or even writing a short story. But it is still far away from possessing true intelligence.</p><h3>Conclusion</h3><p>ChatGPT’s performance on the Winograd Schema Challenge with only 50% accuracy highlights the fact that generative language models still have a long way to go to possess the ability to understand contextual nuances and resolve ambiguities in sentences. As NLP models continue to improve, we can expect better understanding of the language and enhanced ability to perform intelligent tasks. However, it is also important to evaluate their performance to ensure that the models are unbiased and accurate.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=90d16b164d52" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>