<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Adam Tomkins on Medium]]></title>
        <description><![CDATA[Stories by Adam Tomkins on Medium]]></description>
        <link>https://medium.com/@artomkins?source=rss-64a0912c086c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*4cC6UZD2ChnN7GEQxeazmQ.jpeg</url>
            <title>Stories by Adam Tomkins on Medium</title>
            <link>https://medium.com/@artomkins?source=rss-64a0912c086c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 13:38:53 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@artomkins/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Truth-Preserving Summarization: Exploring Constrained Generative Model Approaches]]></title>
            <link>https://medium.com/@artomkins/truth-preserving-summarization-exploring-constrained-generative-model-approaches-b0007c2e0d79?source=rss-64a0912c086c------2</link>
            <guid isPermaLink="false">https://medium.com/p/b0007c2e0d79</guid>
            <category><![CDATA[linguistics]]></category>
            <category><![CDATA[text-summarization]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Adam Tomkins]]></dc:creator>
            <pubDate>Wed, 28 Dec 2022 00:03:48 GMT</pubDate>
            <atom:updated>2022-12-28T00:03:48.091Z</atom:updated>
            <content:encoded><![CDATA[<p>Generative text models are impressive in their ability to generate natural text, given prompts. Their skill comes from sampling underlying token (eg. word) probabilities given a list of previous input of tokens.</p><p>Generally, however, this isn’t the whole story about how we generate our own thoughts and text when we write. Although, if you ask someone out of the blue to explain phenomena they know *almost* nothing about, it can really seem like building a sentence one word at a time. We call this bloviating, and it has a role in communication. It’s just only a small, not particularly information-rich approach to communication.</p><p>While there is, and I hope always will be, some mystery to how we convert our internal thoughts, into external representations, merely picking the next most likely token misses the higher-level goal-oriented and constrained approach to communication.</p><p>In his essay <em>Logic and Conversation</em> (1975) and book <em>Studies in the Way of Words</em> (1989), Paul Grice outlined four key maxims of conversation — <em>quantity</em>, <em>quality</em>, <em>relation</em>, and <em>manner</em> which frame some high-level constraints over how language can be crafted to improve goal-oriented outcomes, or in other words, <em>co-operation</em>.</p><ol><li><strong>The maxim of quantity</strong>, where one tries to be as informative as one possibly can, and gives as much information as is needed, and no more.</li><li><strong>The maxim of quality</strong>, where one tries to be truthful, and does not give information that is false or that is not supported by evidence.</li><li><strong>The maxim of relation</strong>, where one tries to be relevant, and says things that are pertinent to the discussion.</li><li><strong>The maxim of manner</strong>, when one tries to be as clear, as brief, and as orderly as one can in what one says, and where one avoids obscurity and ambiguity.</li></ol><p>These maxims may provide a useful framing for how we can think about improving the generative model approach to natural language generation, with respect to the underlying value of <em>language as a tool for co-operation</em>.</p><p>Here, we will look at perhaps the hardest to determine maxim, the maxim of Quality, wherein one tries to be truthful. The issue of <em>model hallucination</em> is specifically problematic in domains where truth must be maintained, and the cost of misrepresentation is high, for instance, in the domain of <em>Medical Literature summarization, </em>wherein hallucination could factually incorrect statements and lead to unsubstantiated actions, and real world ramifications.</p><blockquote>Identifying these hallucinations post-hoc is incredibly difficult, due to the lack of explainability and introspection.</blockquote><p>This post, I will explore possible approaches to constraining generative models, to create more purpose-driven answers inspired by the Grician Maxims.</p><p><strong>Can we encode truth-preservation into the summarization step?</strong></p><p>A generative model builds its text <em>n</em> tokens at a time, through a beam search, attempting to find the sequence of tokens with highest probability of occurring, given some context. This leads us to ask, can custom search strategies align generative models with the Grician maxims?</p><p>The built in beam-search constraints operate at a token-choice level, this is useful to encouraging specific tokens and token sequences in the generated output. This has been used to great effect by the <a href="https://github.com/Hellisotherpeople/Constrained-Text-Generation-Studio">Constrained Text Generation Studio</a>, which states:</p><blockquote>At each generation, a language model is actually sampling from a probability distribution of its entire vocabulary (which is usually tokenized sub-words). Why don’t we just ban the tokens within the vocabulary which violate the chosen constraints before the sampling step?.</blockquote><p>This works well for many of the token-based requirements such as poetry requirements, meaning matching, vocabulary requirements etc.</p><p>However, truth-content cannot be meaningfully determined on a token by token basis, the truth-value of an utterance must be determined, at a minimum, on a sentence level. This is a fundamental limitation of beam search over small subsets of text tokens. We do not work at the level of cohesive meaning.</p><p>This leads us to consider an approach of post-generation pruning, similar to token pruning, on the level of truth statements. One approximation to truth we can use here, is entailment. We can ask, is the generated summary, or each of its truth-value constituent parts, entailed by the original summary.</p><h4><strong>Entailment as a measure of Truth Preservation</strong></h4><p>If it cannot derive the summary from the source, it may indicate that the generative model has<strong> added text to the core meaning, or removed a core fact. </strong>In the first case we have discovered a mechanism for detecting truth-altering additions, in the second, a way to flag issues in meaning preservation in a summary.</p><p>Alternatively, entailment could fail if the summary is garbled so much, that we can no longer derive the meaning of the summary, from the meaning of the source. Another useful thing to be able to flag.</p><p>While this may not be a perfect measure, we can intuit that if a summary cannot be reconciled with its source, it will likely be a bad summary on the whole.</p><p><strong>Entailment </strong>allows us to ask the question, does a premise, entail a hypothesis? In our case we can reframe our source text as the premise, the known truth, and the summary as a hypothesis.</p><p>An clear example of how this process would work has been seen in the case of descriptive summarisation:</p><blockquote>Source: “Across this set of documents, Huntington’s Disease is the most prevalent disease”</blockquote><blockquote>Summary Text: “Huntington’s Disease is the most prevalent disease <strong>in America</strong><em>”</em></blockquote><p>Here we can see the generative model hallucinated an extra part of the summary which is not in the source text, misrepresenting the underlying data. This is precisely the kind of summary which we can use entailment on.</p><p>For instance, using the Flan-T5 model we can ask, does this source entail the summary:</p><blockquote><strong>Premise</strong>: In this literature, Huntington’s Disease is the most prevalent disease. <br><strong>Hypothesis</strong>: Huntington’s Disease is the most prevalent disease in America. <br><strong>Does the premise entail the hypothesis?</strong> explain your answer</blockquote><blockquote>&gt;&gt;&gt; It is not known if Huntington’s Disease is the most prevalent disease in America. So the answer <strong>is it is not possible to tell</strong>.</blockquote><p>This shows that, given a sufficiently good way to measure entailment, we can learn a lot about how to police and flag issues with generative summarization, now, we need only generalise it to each of the sentences in a summary, to get an score of truth preservation, and a filtered, improved summary.</p><p>A potential naive algorithm for truth-preserving summarization may be:</p><ol><li>Create a generative summary of a piece of text.</li><li>Process this summary into truth-value containing chunks (Sentences for simplicity).</li><li>For each sentence, ask, is this summarised sentence entailed by the original text, if not, <em>remove it</em>.</li><li>(Optional) Update the rejected sentences, and re-test.</li></ol><p>Further to this, the entailment measure of a summarization quality allows us to ask the reverse question:</p><ol><li>What sentences of the original text are no longer entailed by the summarization</li></ol><p>This will give us some notion <em>of coverage-by entailment</em> as a proxy measure of what content has been used <em>successfully</em> in the summarization. Knowing what is captured, and what is removed, is a key strength of the Extractive Method of sumamrization, recapturing this ability is a useful way of building generative capabilities into larger systems.</p><h3>Conclusions</h3><p>We have seen that generative models are incredibly useful, but incredibly hard to police and constrain. Taking inspiration from the work of Paul Grice, we have seen how we could use entailment as a proxy for truth-preservation in constraining, evaluating and improving generative model output, recapturing a link of summary coverage provided by extractive methods.</p><p>All we need is a sufficiently robust method of entailment, which may turn out to be <a href="https://medium.com/@artomkins/pick-your-suffering-on-the-fragility-of-generative-models-in-summarisation-821cad11b8a0">a deal less certain than it sounds.</a></p><h3>Reading List</h3><ol><li><a href="https://arxiv.org/abs/2206.05395">Why is constrained neural language generation particularly challenging?</a> (2022)</li><li><a href="https://arxiv.org/abs/2005.00558">POINTER: Constrained Progressive Text Generation via Insertion-based Generative Pre-training</a> (2020)</li><li><a href="https://arxiv.org/abs/2012.15416">Directed Beam Search: Plug-and-Play Lexically Constrained Language Generation</a></li></ol><h3>Interesting Code</h3><ol><li><a href="https://github.com/Hellisotherpeople/Constrained-Text-Generation-Studio">Constrained-Text-Generation-Studio</a></li><li><a href="https://huggingface.co/blog/constrained-beam-search">Constrained Beam Search Introduction</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b0007c2e0d79" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pick your Suffering: On the Fragility of Generative Models in Summarization]]></title>
            <link>https://medium.com/@artomkins/pick-your-suffering-on-the-fragility-of-generative-models-in-summarisation-821cad11b8a0?source=rss-64a0912c086c------2</link>
            <guid isPermaLink="false">https://medium.com/p/821cad11b8a0</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[text-summarization]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <dc:creator><![CDATA[Adam Tomkins]]></dc:creator>
            <pubDate>Tue, 27 Dec 2022 15:18:15 GMT</pubDate>
            <atom:updated>2022-12-27T15:39:56.929Z</atom:updated>
            <content:encoded><![CDATA[<p>Back when I was choosing between remaining in academia or continuing my research in a corporate environment, I read a timely book that had one piece of advice that stuck out to me. <em>Pick your suffering.</em></p><p>That is when choosing between two options, it’s natural to think, what is the best bits of both options, and compare them. In contrast, <em>picking your suffering</em> dictates that it may be useful to compare the worst parts of both solutions, and <em>pick the least worst</em>.</p><p>In life, you may want to be less conservative and defensive when picking your challenges, however, when it comes to developing solutions and deploying software, the battles you pick will define the success of your product.</p><blockquote>Today’s battle is Text Summarization in the Life Sciences.</blockquote><p>When choosing between generative models for summarization, and extractive, we may want to take the same advice, <em>Pick your suffering</em>.</p><p>When a generative model works at its best, it can be <em>incredible</em>. More readable, and more intuitive than extractive summarization by far.</p><p>However, if we look at the <em>suffering</em> we are choosing when we go <em>generative over extractive</em>, we see a very different picture, and depending on your use case, this may be the straw that breaks the camel’s back.</p><p>In a previous proof-of-concept, I built a system for <em>“truth-constrained summarization”</em>, as an attempt to bring Grices <em>Maxim of Quality</em> to the generative world.</p><blockquote><strong>The maxim of quality</strong> is where one tries to be truthful, and does not give information that is false or that is not supported by evidence.</blockquote><p>In our “truth-constrained summarization” attempt, we use a combination of generative models, and entailment, to ensure that the generative model isn’t creating text that cannot be substantiated by the document we are trying to summarise.</p><p>This is only required because we know that generative models do hallucinate and add text that was not in the original. While the approach was promising, the underlying assumption is that the entailment model, in itself a generative model, is reliable, whereas generative summarization is not.</p><p>This was a fair assumption for a proof of concept, but the ramifications are quite important.</p><p>Firstly we can look at the reason we made this assumption. The model in question, the T5-Flan, has a prompt set-up that is not prone to the random generation of text, it is constrained to three outputs “yes”, “no” and it “is not possible to tell”. This is perfect as it means we no longer require natural language understanding to understand our model output.</p><p>By choosing a prompt-training structure that removes random generation and interpretation, we can stop worrying about the meaning of the generated text, and instead ask the more familiar question, Is the output reliably correct? Can we trust it to do the heavy lifting of truth constraints in text generation?</p><p>The answer, unsurprisingly, is no. The same fragility persists, even within the well-defined, prompt constrained space that we are using.</p><p>Here we show that entailment models are fallible and can give <strong>contradicting output</strong> <strong>for the same input</strong>, depending on the final form of the prompt.</p><p>Take this example prompt (you can try it out <a href="https://huggingface.co/spaces/osanseviero/i-like-flan">here</a>):</p><blockquote><strong>Premise:</strong> <em>The man has a cat. The cat has a hat. It is red.<br></em><strong>Hypothesis:</strong> <em>The cat is red.<br></em>Does the premise entail the hypothesis? <em>Explain your answer</em></blockquote><blockquote><em>&gt;&gt;&gt; </em>It is <strong>not possible to tell</strong> that the cat is red from the information given. The hat on the cat is red. So the answer is it is not possible to tell.</blockquote><p>Here we see that the answer is correct. The <strong><em>It</em> </strong>from<strong> </strong><em>“</em><strong><em>It</em></strong><em> is red” </em>is correctly resolved to mean, the hat. This is a triumph of reasoning from natural text, and a lot of nuance goes into creating the correct answer. The only downside here is that to use this output in a larger system, we must then <em>interpret </em>this answer to be a yes, or no, or it’s not possible to tell.</p><p>Luckily, we can alter the prompt to get this constrained output, by not asking for an explanation of the reasoning:</p><blockquote><strong>Premise:</strong> <em>The man has a cat. The cat has a hat. It is red.<br></em><strong>Hypothesis:</strong> <em>The cat is red.<br></em>Does the premise entail the hypothesis?</blockquote><blockquote><em>&gt;&gt;&gt; yes</em></blockquote><p><em>Wait, yes?</em></p><p>Here we’ve got our red flag. First, it’s wrong. But we can forgive that, every model is wrong sometimes. But what is more problematic, is that the model is unreliable. That sophisticated reasoning that is displayed above is not a reliable measure of what is happening under the hood.</p><blockquote>Intuitively, adding an explanation, should not in any circumstance, change your answer, only elaborate on it. But that is not what we see.</blockquote><p>This is incredibly problematic, as the simple prompt, which returns either <em>yes</em>, <em>no</em>, or <em>it is not possible to tell, </em>is precisely the constrained output we want to use when we use entailment within a larger system, such as summarization.</p><p>This is troubling and underlines a few things to be wary of with current generative models. In their raw form, they are <strong>incredibly fragile</strong> and fail in unexpected, and are hard to guard against ways. While we could find other prompts that circumvent this issue, can we trust that it doesn’t have its own unreliable failure modes?</p><p>This inability to easily find and correct failure modes in generative models may be the perfect illustration of <em>“Choose your suffering”</em>. If your use case depends on the accuracy of your text, and the reliability of your systems, choosing the path of policing a hallucinating unpredictable generative model, may be choosing a rabbit hole of suffering.</p><blockquote>Policing the output of a generative model, requires the exact natural language understanding capabilities, that the generative model was supposed to solve.</blockquote><h3>Does extractive model summarization always preserve meaning?</h3><p>On the flip side, in a discussion I had recently about summarization and generative models, the following was mentioned as a reasonable argument for choosing extractive over generative models in truth-dependant domains.</p><blockquote>We cannot afford for a summary to misrepresent the underlying text, so we extract from the core text instead of generating new text for a summary.</blockquote><p>Generally, this is a fair thing to say, generative models are known to hallucinate, and policing them is hard. However, it is worth thinking about the truth of this sentiment. Are extractive models immune to text misrepresentation? <em>Absolutely not</em>. Let us use the example above, and see some potential failings of a naive extractive approach:</p><p><em>The man has a cat. The cat has a hat. It is red.</em></p><p>An extractive method, which picks out the top <em>n</em> sentences, may suggest that the cat having a hat is less important that the colour of the hat, and as such provide the following summarization:</p><p><em>The man has a cat. </em><strong><em>It</em></strong><em> is red.</em></p><p>We can see here, that due to a failure of co-reference resolution, we have completely changed the meaning of the sentence.</p><p>While this is a small edge case, and a trivial example, it is at least instructive. For one, tackling this issue is considerably more well-defined than fact-checking the generative case. For instance, to reduce this edge case, we could:</p><ol><li>Fall back to co-reference resolution pre-extraction, as a way to minimise this occurrence.</li><li>Use entailment models to check for any introduced incongruities, (reintroducing the main problem of fragility.)</li></ol><p>Here, the suffering we have picked, is much more reliably dealt with, with an established set of tools and methods, without re-introducing the very issues we’re trying to solve.</p><p>In the end, the product may be less flashy, but more reliable, explainable and maintainable. Choose the suffering you can best handle, and deliver the product your can best iterate on.</p><h3>Conclusion</h3><p>While neither the generative nor extractive approach is perfect, in this case, we can see that chasing down the edge cases of extractive summarisation, is a much more constrained problem than chasing down the edge cases of generative models.</p><p>In the end, it depends on your final use case, and improvements to both approaches are continuously being developed. Whatever you chose, one thing is certain, it’s going to be an exciting year ahead in AI and NLP.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=821cad11b8a0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Finding the The Unknown Unknowns in AI]]></title>
            <link>https://medium.com/@artomkins/finding-the-the-unknown-unknowns-in-ai-61a40ea9819f?source=rss-64a0912c086c------2</link>
            <guid isPermaLink="false">https://medium.com/p/61a40ea9819f</guid>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ontology]]></category>
            <category><![CDATA[explainability]]></category>
            <dc:creator><![CDATA[Adam Tomkins]]></dc:creator>
            <pubDate>Fri, 12 Aug 2022 13:31:31 GMT</pubDate>
            <atom:updated>2022-08-12T13:31:31.676Z</atom:updated>
            <content:encoded><![CDATA[<p>Imagine someone who cannot say, I don’t know. How would you find out what they actually do know if they can never that they don’t actually know?</p><p>This is the problem we have with relying on large language model embedding for our many of our AI classifications. The core issue here is, that the underlying encoding is a dense vector representation, there is no representation for <strong>explicit knowledge </strong>or <strong>Ignorance</strong>, every vector looks the same.</p><p>In effect, when presented with something that it cannot sensibly represent, it must represent it as <em>something</em>, and we never know if or when this gung-ho forced representation is happening. It reminds me a lot of one of my favorite quotes from the Wheel of Time.</p><blockquote>“You can never know everything, and part of what you know is always wrong. Perhaps even the most important part. A portion of wisdom lies in knowing that. <strong>A portion of courage lies in going on anyway.</strong>”</blockquote><blockquote>Robert Jordan</blockquote><p>In general, soldiering on through, in the face of ignorance has given large language models great results, however it leaves them with a significant blind spot, and as developers, it doesn’t give us the tools to know when a representation plagued by a flawed representation.</p><p>This has some huge implications when we look at fast moving language, and domain specific understanding. We can nicely demonstrate this with a topical classification example from the medical domain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f_itBagWC7uSrFgy5Xdjog.png" /><figcaption>Unknown terms muddy representations, effecting down-stream tasks.</figcaption></figure><p>Here we can see that when given a zero-shot text classification problem, not understanding the crucial term BA.4 allows the model to do a poor job of classification. When given the meaning for BA.4, through taxonomic promotion, we get an unambiguous, and correct result.</p><p>The problem with Large Language models, is we have no introspective behavior to ask if the representation of a concept is robust, or forged through ignorance. In essence, the task of this blog is to ask, can we reverse engineer what the <strong>unknown unknowns</strong> of a language model are?</p><blockquote>There are known knowns. These are things we know that we know. There are known unknowns. That is to say, there are things that we know we don’t know. But there are also unknown unknowns. There are things we don’t know we don’t know.</blockquote><blockquote><a href="https://www.brainyquote.com/authors/donald-rumsfeld-quotes">Donald Rumsfeld</a></blockquote><p>There are many approaches that we could use to introspect these models to detect if a representation is flawed. In order for the process to be scalable it needs to be fast and efficient. Given a good model, we can use this to identify, and augment the language we are using to classify, before we pass it through the encoder, sidestepping the encoder-blind spot problem.</p><p><strong>Approach: Catching Good Representations as Outliers</strong></p><p>The intuition we will take, is that when asked to make a representation of an unknown term, the model will give vague, or random representations. And that their may be shared representational features between unknown concepts, in essence, a marker for the unknown embedded in the representations, similar to how humans add “err and erms” to their speech when they become uncertain.</p><p>To test this, idea, we look at a 2D projection of words, using a general sentence embedding model. We look at the Top words for the Brown corpus, Words from medical terms from the DOID ontology and words made from random noise.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hm5HoBAnKgC-13_eHmenHg.png" /><figcaption>2D Projections of Normal Words, Medical Terms, and Random characters</figcaption></figure><p>These are generally easily separated, as expected. This makes sense as many medical terms, are standard English, for example embedded within the brown corpus as the Medical Terms: Responsive, Isolated and Temporal. On the flip-side, many medical terms here are entirely entwined within the random representations, such as Transudative, Meesmann and Motile. In these cases we can assume that the semantically devoid and the medically obscure are shrowded in the same ignorance.</p><p>Its from this intuition what we aim to build a system to recognize, when the representation is indistinguishable from random noise in the embedding space.</p><p>The instinctual quick and efficient approach would be to train a classifier to classify raw embedding, if they are based on known or unknown concepts. However, if we already had the data of unknown and known terms, we wouldn’t need to build this classifier.</p><p>To get around this problem, we are going to use <em>Outlier detection</em>, that way we only need one training set, not a labeled dataset. While we cannot easily enumerate a dataset we are sure their are strong internal representations of, we absolutely can generate a dataset of complete random strings, and embed them, with high certainty the model has never seen these before.</p><blockquote>mPrd, dgB, acari, uwdQM, MM.x, Mh8U6, kbvG, y23k, BA.4, jT0N4, FBAk, 64Zpi, PKwcR, HCL-V,</blockquote><p>The theory here, is that if we can learn to classify stings that are outliers from Random Noise, we have found meaningful representations. Any tokens that are indistinguishable from random noise, we know we may need to augment with ontological knowledge, to avoid muddied representations.</p><p>To really drive this point home, lets take another look at that list of random characters above. Are any of them meaningful? As it turns out, I snuck in a few meaninful examples to prove my point. <a href="https://disease-ontology.org/?id=DOID:0060032">HCL-V,</a> acari and BA.4 are all meaningful medical concepts, some established, but niche, and some novel and evolving. In this case neither you, nor hopefully our language models, could tell them from noise.</p><p>One key aspect here, is that since we are detecting noise, we assume:</p><ol><li>Noise is clustered and somewhat systematic, otherwise outlier detection will not work.</li><li>The Noise is well crafted to match the distribution of sub-word units in natural language.</li><li>We are doing word-level encodings, as multi-word tokens bring in meaning from individual words to muddy single-word tokens.</li></ol><p><strong>Results</strong></p><p>Given a large Medical Ontology we can ask, what items in this ontology can the language model not represent well. Below we get a list of terms from the ontology that, according to our classifier, are close enough to noise as to be indistinguishable as an outlier.</p><blockquote>SM-AHNMD : disease <a href="https://disease-ontology.org/?id=DOID:0060032">driver<br>HCL-V </a>: a autoimmune disease of musculoskeletal system<br><a href="http://purl.obolibrary.org/obo/DOID_934">ileus</a> : a viral infectious disease<br>Tics : a primary ciliary dyskinesia<br><a href="https://disease-ontology.org/?id=DOID:0050737">Apodemus</a> : a autosomal recessive disease<br>Ehrlichia : a secondary hyperparathyroidism<br>Sauria : an autosomal dominant disease<br>Poodae : an autosomal dominant disease<br>Gadoidei : an autosomal dominant disease<br>Otophysi : an autosomal dominant disease<br>Evosea : an autosomal dominant disease<br>Opisthokonta : an autosomal dominant disease<br>Ixodida : Charcot-Marie-Tooth disease type 2</blockquote><p>As we can see, there are many words here that the ML algorithm considers noise, especially with respect to autosomal dominant diseases.</p><p><strong>Comparing Models</strong></p><p>Different sentence embedding models will have different capabilities in creating meaningful representations for fringe concepts. While the general bert models trained on natural language may fail to capture some instances of medical words, models such as sci-bert are trained to focus on more scientific text.</p><p>As we are model independent, we can test to see how the blind spots across models line up. We can plot model agreement and disagreement, a visual confusion matrix, plotting the outlier scores for each word.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TrW4uCNJFRfXVXRXYm84Dg.png" /><figcaption>Here we plot the outlier scores for both the Sci-bert and general Bert encodings. Purple represent model agreement on noises, and blue model agreement on meaningful.</figcaption></figure><p>Here we show that there is a reasonable disconnect between the models. For shared agreement of useless representations we have some great words I’ve never seen before: <em>Teebi, Ankylosing,</em> and <em>Kipper-Feil</em> and <em>Epiphyseal,</em> with some strangely out of place words, such as the borderline case <em>Atrophy </em>and the number 3.</p><p>For words we have agreement of meaning, we have some reasonable candidates such as <em>Pregnancy</em> and <em>Enamel</em> and <em>Dental</em> with some surprising candidates such as Poliposis and Amelogenisis. These latter examples hint at either an underlying problem with this method, or much more widely read models than previously thought.</p><p><strong>Take Away</strong></p><p>If you are serving any kind of dense-vector representation for classification, or semantic search, your representations will have blind spots that you cannot unravel, this will provide a <em>hard barrier to performance improvements </em>until you manage to</p><ol><li>Identify your blind spots</li><li>Augment your representations with external knowledge</li></ol><p>While some of these blind spots will be solved through fine-tuning, you can never get away from the underlying issue. And in a fast moving industry, can you really afford to fine-tune every model everytime a new drug, treatment, regulation or research code is adopted?</p><p>The bottom line here is that language is flexible. Language models are not. Our approach provides a potential, but not fool proof approach for the identification of the unknown unknowns, which is key to catching, and solving poor representations in dense vector models, before they happen.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=61a40ea9819f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Shared Misunderstanding: Where representations in Large Language Models go wrong.]]></title>
            <link>https://medium.com/understanding-natural-language/shared-misunderstanding-where-representations-in-large-language-models-go-wrong-11b1c7b1d14d?source=rss-64a0912c086c------2</link>
            <guid isPermaLink="false">https://medium.com/p/11b1c7b1d14d</guid>
            <dc:creator><![CDATA[Adam Tomkins]]></dc:creator>
            <pubDate>Wed, 03 Aug 2022 22:11:50 GMT</pubDate>
            <atom:updated>2022-08-05T17:28:15.421Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Welcome to a set of blogs looking at the intersection of language, knowledge representation and AI. In this post, we’re going to take a deep dive into some of the concepts of Prototype Theory, and ask if the advances in AI and NLP can provide any new tools to investigate these cognitive phenomena.</blockquote><p>Imagine a “piece of furniture”. What do you see? A “chair”, a “table”, a collection of “different pieces of furniture”? Or a “half-table, half chair, all impractical monstrosity”, which kind of resembles a chair if you squint, a table if you cross your eyes?</p><p>Most likely you’ll have imagined some kind of furniture. For me, it’s a chair. In my mind, a chair is the prototypical piece of furniture. Your prototype could be different.</p><p>Now, let’s imagine that chair. It’s easy if you try. It’s instantly recognizable. A table too; and a stand-lamp. They’re all easy and natural to imagine.</p><p>So, what is the difference here? What sets apart “a piece of furniture” from a “chair” and a “table”.</p><p>Chairs and tables seem more fundamental, more “real” than furniture. Cognitive Scientist and psychologist <a href="https://en.wikipedia.org/wiki/Eleanor_Rosch">Eleanor Rosche</a> calls these more fundamental objects “<a href="https://en.wikipedia.org/wiki/Prototype_theory"><em>Basic level categories</em></a><em>”, </em>a<em> </em>key concept in Prototype Theory, which we will dive further into in another post.</p><p>Basic Level Categories are real, they are defined mostly in terms of sensory-motor <a href="https://en.wikipedia.org/wiki/Affordance">affordances</a>, or how we interact with physical objects. Everything else, such as abstractions and hierarchies, are a feat of the imagination. These abstractions are not inherent in any object but are created for our convenience. There is no real thing “furniture”, there are only the basic level categories of chairs and tables, etc.</p><p>This is why it is easy to picture a chair, but when picturing a piece of furniture, we default to picturing different types or <em>instances </em>of furniture.</p><p>The way we categorize the world around us - and the way we build hierarchies of the things in the world - is core to how we interact with, internally represent and externally communicate about the world. The hierarchies we come up with make communication and reasoning about the world much more efficient.</p><p><strong>The Role of Hierarchies in Understanding</strong></p><p>These shared hierarchies of the world allow us to communicate with incredible efficiency, allowing the shared hierarchy to do the heavy lifting in a conversation.</p><blockquote>Do you have any furniture at home?</blockquote><p>Answering this question depends on the shared understanding of what constitutes furniture. In this case, we’re talking about a hierarchy which maps the abstract concept <em>furniture</em> to all of the types of furniture below it. Without this shared hierarchy, in a linguistic <a href="https://en.wikipedia.org/wiki/Flatland"><em>flat-land</em></a><em>,</em> we would have to enumerate all the things we believe to be furniture, to get the exact answer we expect. If you’ve ever been to the <a href="https://www.bauhaus.de/en/sammlung/highlights/205_moebel/">Bauhaus Furniture museum</a>, you would know how tedious this will become.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*aDaSfEk8HD9CRGfUiQCu7g.png" /><figcaption><em>Do you have a chair, OR a table, OR a couch OR a welsh dresser OR a </em><a href="https://en.wikipedia.org/wiki/List_of_furniture_types"><em>…</em></a><em> at home?</em></figcaption></figure><p>As you can imagine communicating without hierarchies is very precise, but it is incredibly inefficient. Hierarchies allow us to <em>template</em> language to gain both precision, and efficiency. Even if our shared hierarchies are not perfectly aligned, the benefit of efficiency is worth the cost of absolute precision. I, for instance, would disagree with Wikipedia that a Hi-Fi counts as furniture.</p><p><strong>The Missing Text Problem</strong></p><p>The Missing Text Problem is the idea that in order to understand any piece of text, there is unwritten information that the reader and writer share, which cannot be inferred from the text alone, but is crucial in the full understanding of that text. Shared hierarchies provide insight into a partial solution to this problem.</p><p><strong>Experiment: Does AI approximate Human Level Hierarchical Understanding</strong></p><p>When reading about Basic Level Categories in “<em>Women, Fire, and Dangerous Things: What Categories Reveal About the Mind”</em>, I immediately thought about how the methods we use to demonstrate the intuition of the concept rely heavily on both imagination, and communication.</p><p>With the advent of AI, we can start to ask, do machines follow the same abstractive patterns that we believe humans follow? After all, the explosion of modern AI is built upon the promise of better representations, using neural networks to build powerful “abstractions” for improved language understanding.</p><p>So, the natural question to ask is what does an AI imagine when we ask it to imagine chairs, tables, and furniture? Does it represent furniture as a selection of the instances that make up furniture, or does it not represent furniture as an abstract concept at all?</p><p>Luckily, with Natural Language-driven image creation AIs such as <a href="https://huggingface.co/spaces/dalle-mini/dalle-mini">Dall-E</a> we can begin to put this to the test.</p><p>Let us start with the basics, to reassure ourselves that AI representations can indeed create credible imaginings from language alone.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/747/1*Lzu8mSbGzM2fBWA-lVE4VQ.png" /><figcaption>Chairs As Imagined by Dall-E</figcaption></figure><p>Here we see that when asked to imagine a chair, we get nine very credible chairs. (Not one chair that I would hesitate to sit on, although top-right is a little suspect). We can go further, as ask to make tables as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/759/1*uvwXLHKUZTNm1CbJ-9yCcA.png" /><figcaption>Tables as Imagined by Dall-E</figcaption></figure><p>We can see that the AI has a good grasp of tables, with some small interesting elaborations, such as the centre table, which creatively merges with the curtains, and a possible chair.</p><p>One could anticipate that when DaLL-E is asked to represent the general concept of furniture. If it follows the principle of abstraction and Basic-level Categories, we should see a selection of different kinds of furniture. But what we see is much more “Dali-esque”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/743/1*cM07EitPFym-XOapCJqTtw.png" /><figcaption>Furniture, as imagined by Dall-E</figcaption></figure><p>What we see is that the internal representation of furniture is being interpreted in a way that results in weird and wonderful hybrids. Some, such as the bottom right, are fair examples of individual pieces of furniture, recognisable to us. However, in the bottom center, we see a frankensteinian chair-table combination, which breaks with the human approach to abstract visualisation; instead it attempts to combine all the definitive features of things considered furniture into a nightmarishly impractical joke.</p><p>What does this tell us about the internal representations used in these models?</p><p>Here, the representation of furniture has aspects of all the things it has seen that are furniture, creating an “average” set of furniture features. Good features for sure; intelligent and recognisable features, such as handles, surfaces, legs and draws. The amalgamation of these features maps quite nicely onto the metaphor of family resemblances spoken about in Prototype Theory, however that is where the similarities end. Internally, the representation of the word “furniture” is an average of all the furniture, not a collection of prototypical furniture. The machinery used to build these representations has failed to internally create any meaningful hierarchies.</p><p>This is innate in how embeddings work. Everything is embedded into the same space. There are no real abstractions being made. This representation is useful, but it is not understanding. It can help us build great classifiers, but it doesn’t really help us understand how humans think.</p><p>To me, this motivates the question of what we need to do next, to improve knowledge representation in AI models, hierarchies must play a role.</p><p><strong>A Shared Misunderstanding Between AI</strong></p><p>We mentioned that to understand text well, there is a wealth of missing information that we as humans have internally and generally share. We rely on shared understanding for efficient communication, and at least one part of that understanding is ontological hierarchies. From this understanding, we can reasonably ask: Do these AI models have any shared understanding? Can they communicate ideas and be understood between themselves? Here we complete that circle.</p><p>In case furniture isn’t a riveting example to you, we can see how the same system butchers abstractions, this time with animals.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/775/1*1I9sTIReHFZNvc6YvxV2zA.png" /><figcaption>Animals as imagined by Dall-E</figcaption></figure><p>These grotesque approximations of animals again show how the models used by DaLL-E do not handle abstractions well. In fact, not a single one of these are recognisable as an animal. When I asked a biologist to name these creations I got: Scampir (my personal favourite), Dongang, Cog, a Tigeroo, and Shion, among others.</p><p>But, if you ask an AI trained in zero-shot classification to tell you what it sees, it will confidently assert that this is an animal; just not a specific one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pr2Rc6HmiFaq3VRi2PAf5Q.png" /><figcaption>Zero Shot Image classification.</figcaption></figure><p>On a technical note, this is fascinating. The AI making this decision (FLAVA) is a separate AI than the one that built the image (Dall-E), but they steadfastly agree, that this is absolutely an animal. Both AIs trained with different data, extracted universal properties, and have come to the same conclusions. In a sense there is a shared understanding that has been abstracted from the data: the core features of an animal.</p><p>One AI imagined a never before seen animal and the other AI also understands it to be an animal.</p><p>On one hand, this is a triumph of shared representation through different exposure. But on the other hand it is terribly, terribly wrong. What we see is an abomination, not an animal.</p><p>While a potentially silly example, it exposes an issue this these representations, and brings the story back to linguistics, the power of abstraction and base level categories.</p><blockquote>We’re asking about ontological types and we’re being answered with statistical correlations. This basic miscommunication is a red flag when it comes to trust in AI models.</blockquote><p>When we ask, “Is this an animal?”, we’re not asking, “Does it share enough features to resemble an animal?” We’re asking, “Does this belong to the super-class animal?”</p><p>Without learning to represent the hidden hierarchies in language, these systems cannot understand or answer the logical question we are actually asking.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=11b1c7b1d14d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/understanding-natural-language/shared-misunderstanding-where-representations-in-large-language-models-go-wrong-11b1c7b1d14d">Shared Misunderstanding: Where representations in Large Language Models go wrong.</a> was originally published in <a href="https://medium.com/understanding-natural-language">Understanding Natural Language</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>