<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Niya Stoimenova on Medium]]></title>
        <description><![CDATA[Stories by Niya Stoimenova on Medium]]></description>
        <link>https://medium.com/@niya.stoimenova?source=rss-bacf8cf8265e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*a7Ji6E5M6hrqpTLU9BgMYQ.jpeg</url>
            <title>Stories by Niya Stoimenova on Medium</title>
            <link>https://medium.com/@niya.stoimenova?source=rss-bacf8cf8265e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 12:35:07 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@niya.stoimenova/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[On Why Machines Can Think]]></title>
            <link>https://medium.com/data-science/on-why-machines-can-think-40edafce293d?source=rss-bacf8cf8265e------2</link>
            <guid isPermaLink="false">https://medium.com/p/40edafce293d</guid>
            <category><![CDATA[thoughts-and-theory]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[reasoning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[transformers]]></category>
            <dc:creator><![CDATA[Niya Stoimenova]]></dc:creator>
            <pubDate>Wed, 06 Dec 2023 22:58:59 GMT</pubDate>
            <atom:updated>2023-12-06T22:58:59.661Z</atom:updated>
            <content:encoded><![CDATA[<h4>How can we think about<strong> thinking</strong> in the simplest way possible?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*29ys7MCTEe3QfWXrOBSAxQ.jpeg" /><figcaption>Opening Pandora’s box (image by author)</figcaption></figure><p>In the 17th century, <a href="https://en.wikipedia.org/wiki/Ren%C3%A9_Descartes">René Descartes introduced </a>a relatively new idea — the dictum “<strong>cogito ergo sum</strong>” (“I think, therefore I am”). This simple formulation served as the basis of Western philosophy and defined for centuries our ideas on what constitutes the essence of being a human.</p><p>Since then, our understanding of what it means to be a human evolved. Yet, for all intents and purposes, many still consider one’s capability to think as one of the most important hallmarks of humanity.</p><p>So, it comes as no surprise that the moment ChatGPT (and similar models) was released, we started being bombarded with articles discussing “whether it can think”.</p><p>For example, the New Yorker mused “<a href="https://www.newyorker.com/science/annals-of-artificial-intelligence/what-kind-of-mind-does-chatgpt-have">What kind of mind does ChatGPT have?</a>”; the Washington Post proclaimed “<a href="https://www.washingtonpost.com/technology/2023/03/18/gpt4-review/">ChatGPT can ace logic tests. But don’t ask it to be creative</a>.”; and the Atlantic concluded that <a href="https://www.theatlantic.com/technology/archive/2022/12/chatgpt-openai-artificial-intelligence-writing-ethics/672386/">“ChatGPT is dumber than you think”</a>. A personal favourite of mine is this <a href="https://www.tiktok.com/@dragoscomedy/video/7229402346680339717">video of a comedian</a> trying to explain what ChatGPT is to someone who’s working in HR.</p><p>As with any other complex topic that lends itself well to speculation, people are both over-exaggerating and under-representing the thinking capabilities of AI models. So, let’s unpack this.</p><h3>To think is to reason</h3><p>Thinking is a complex construct that has come to represent many different things. So, for simplicity sake, let’s presume that thinking is more or less synonymous with reasoning.</p><p><strong>Reasoning</strong> is a much better defined concept that is, coincidentally, being <a href="https://www.deeplearning.ai/the-batch/yoshua-bengio-wants-neural-nets-that-reason/?utm_campaign=The%20Batch&amp;utm_source=hs_email&amp;utm_medium=email&amp;_hsenc=p2ANqtz-865CMxeXG2eIMWb7rFgGbKVMVqV6u6UWP8TInA4WfSYvPjc6yOsNPeTNfS_m_et5Atfjyw">increasingly thrown around as the <strong>future of AI</strong></a>. It’s also what Descartes (largely) meant when he was talking about thinking.</p><blockquote>So instead of asking “Can AI think?”, let’s ask “Can AI reason?”.</blockquote><p>The short answer is <strong>yes</strong>. The long answer — it can reason, but only in some ways.</p><p>Reasoning is not a monolithic concept. There are multiple ways, in which one reasons, depending on the type of tasks she’s trying to accomplish. So, in this post, we’ll first go through a brief primer on the three key reasoning types and examine how machines measure up. Then, we’ll explore why machines cannot perform common-sense reasoning and what question we need to answer before they can.</p><h3>Primer on reasoning</h3><p>Generally, there are three main types of reasoning we employ when “thinking”: <strong>deduction</strong>, <strong>induction</strong>, and <strong>abduction</strong>.</p><h4>Deduction</h4><p>Simply put, deduction is the ability to reach a conclusion from a given rule and a case that are assumed to be true.</p><p>Picture this: you fill a pan with water, turn on the stove, and pop in a thermometer. Because of things you’ve learned in school, you know that water (usually) boils at 100 °C. So, when someone tells you that the temperature has reached 100 °C, you can safely deduce that the water is boiling (you don’t actually have to see it with your own eyes to be “pretty sure” that it happens).</p><p>Here’s a useful structure to keep in mind.</p><blockquote><strong><em>1.</em> Rule:</strong> water boils when it reaches 100 °C</blockquote><blockquote><strong><em>2.</em> Case: </strong>the temperature of the water is 100 °C</blockquote><blockquote><strong><em>3.</em> Result:</strong> the water in the pan is boiling</blockquote><p>Thus, you reason from <strong>rule </strong>and <strong>case </strong>to a <strong>result</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3SX2yv4CtbqLkuWbYVJ28Q.jpeg" /><figcaption>Deduction: reasoning from rule and case to a result (image by author)</figcaption></figure><p>Deduction is fundamental for our ability to do science. It’s also the type of reasoning that’s the easiest to reproduce by a machine.</p><p>By design, almost every machine carries out some form of deduction. Your simple non-glamorous calculator deduces answers every time you ask it how much 3+5 is. And it has zero AI in it.</p><p>If we put it in the same structure as the water example above, we get:</p><blockquote><strong>Rule: </strong>The calculator has been “provided” with the rule that <strong>1+1 = 2</strong></blockquote><blockquote><strong>Case:</strong> You’ve asked the question <strong>3+5 = ?</strong></blockquote><blockquote><strong>Result:</strong> Based on the rule, it can calculate/deduce that <strong>3+5 = 8</strong></blockquote><p>Simple.</p><h4>Induction</h4><p>Induction is the ability to generalise rules from a given set of observations. It’s central for our ability to do science since it allows us to <em>quantitatively</em> identify new patterns/rules.</p><p>Let’s stick to the water-boiling example. Imagine you have never been told that water boils at 100 °C. So, every time you bring a pan of water to a boil, you put a thermometer in and measure the temperature — 100, 1.000, 10.000 times. Then, your friends do the same — and no matter how many times you do it, the temperature is always 100 °C. So, you can <strong>induce</strong> the rule: “<em>water boils at 100 °C</em>”.</p><blockquote>1. <strong>Result</strong>: water is boiling</blockquote><blockquote>2. <strong>Case</strong>: whenever you put the thermometer in, it always shows 100 °C.</blockquote><blockquote>3. <strong>Rule:</strong> water boils at 100 °C.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5erKo4c2PQtDI0XiHoJ_iw.jpeg" /><figcaption>Induction: reasoning from result and case to a rule (image by author)</figcaption></figure><p>And voila, you’ve identified quantitatively a new rule based on the pattern you observed. To do that, you reason from <strong>result </strong>and <strong>case </strong>to a <strong>rule</strong>.</p><p>This type of reasoning is not always correct, of course. Famously, Europeans thought all swans were white until they sailed to Australia. Also, we know that water doesn’t always boil at 100 °C (the atmospheric pressure plays a role, too).</p><p>Just because something happens to be correct 10.000 times, it doesn’t mean it will always be correct. Still, 10.000 times tends to be a safe bet.</p><p>Induction is much more challenging for machines. Your calculator, of course, cannot perform it. Machine learning models, however, can. In fact, that’s their primary objective: generalise from a set of given results.</p><p>Let’s take a simple example. Say, we have a supervised classification model that we’ll use for spam detection. First, we have the labelled training dataset — <em>spam</em> or <em>not spam</em> (a.k.a. the <strong>result</strong>). Within that dataset, we’ve compiled multiple <strong>cases</strong> for each result. Based on these, the model induces its own <strong>rules</strong> that can, later on, be applied to a case it has never seen before.</p><blockquote>1.<strong> Result:</strong> spam or not spam</blockquote><blockquote>2. <strong>Case:</strong> large samples for both spam and not spam examples</blockquote><blockquote>3. <strong>Rule</strong>: emails with “these patterns and words” are likely to be spam (within a certain degree of probability)</blockquote><p>Likewise, when dealing with unsupervised models such as recommendation systems, the process follows a similar beat. We first provide the model with a dataset about what people tend to buy when they go to the supermarket (<strong>result</strong>). Once we start with the model training, we’ll expect it to first cluster repeating patterns (<strong>cases</strong>) and then, induce its own <strong>rules</strong> that can be later on applied to <em>similar</em> contexts.</p><blockquote>1.<strong> Result:</strong> the unlabelled data about people’s purchases</blockquote><blockquote>2. <strong>Case:</strong> the similar purchases the model found in the dataset (e.g., everyone who bought eggs also bought bacon).</blockquote><blockquote>3. <strong>Rule</strong>: people who buy eggs buy bacon, too (within a certain degree of probability)</blockquote><p>In both cases, these rules aren’t necessarily intelligible by humans. As in, we know that a computer vision model “<em>pays attention</em>” to a certain part of an image, but we rarely know <strong>why</strong>. In fact, the more complex the model is, the lower our chances are of knowing what rules it uses.</p><p>So, here we go — machines can perform both induction and deduction.</p><h4>Deduction and induction — the bedrock of science</h4><p>It is a widely-held belief that the combination of <em>induction</em> and <em>deduction</em> is the driving force behind our ability to reason. And, as our examples show, contemporary ML models, even the simple ones, can already perform both.</p><p>They first utilise inductive reasoning to generate rules from a given dataset. Then, they apply these rules to new <strong>cases</strong>. For example, once we present a model with <em>a previously unseen photo</em>, it leverages its <strong>rules </strong>to deduce specific <strong>results</strong> (e.g., <em>it can tell us that the photo we provided is upside down</em>).</p><blockquote>Still, the majority of data scientists will agree that even the most advanced ML models cannot reason. Why?</blockquote><p>The water-boiling example can serve as a simple illustration on why relying solely on deduction and induction doesn’t quite cut it. True, we need them to generate a <strong>rule </strong>(<em>“water boils at 100 °C”</em>) and then falsify it in a diverse set of cases. However, this combination falls short in explaining how we guessed that the <strong>result</strong> of <em>boiling</em> has something to do with <em>temperature</em>.</p><p>Beyond that, additional limitations of induction and deduction also become apparent — they are somewhat constrained by a specific context and lack the capacity to fully encapsulate the human ability to transfer knowledge across domains. This is precisely where <strong>abduction</strong> comes in, offering a more comprehensive perspective on the cognitive processes that enable us to make intuitive leaps and connect insights across different realms.</p><h4>Abduction</h4><p>Abduction is the ability to generate new hypotheses from a single surprising observation (i.e., <strong>result</strong>). We do that every time we rely on our experiences to come to an explanation of sorts.</p><p>We go out and we see a wet street. We explain it away with the guess that it might’ve rained the night before. We don’t need to have seen 10.000 wet streets to know that when it rains, the street gets wet. Technically, we don’t even need to have encountered a wet street before — it’s enough for us to know that when water touches objects, it makes them wet.</p><p>This means that if we’re to go back to our water-boiling example, we’ll have a different way to reason:</p><blockquote>1. <strong>Result</strong>: the water is boiling</blockquote><blockquote>2. <strong>Rule</strong>: water boils at 100 °C</blockquote><blockquote>3. <strong>Case</strong>: the temperature of the water must be 100 °C</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*faIjBXVpqFpPdm-i9u9sWA.jpeg" /><figcaption>Abduction: reasoning from rule and result to a case (image by author)</figcaption></figure><p>We start from the <strong>result </strong>(as we do with induction), but we combine it with a <strong>rule</strong> we already know (based on our world knowledge and experience). The combination of the two allows us to come up with a <strong>case</strong> (i.e., <em>the water is boiling because of changes in its temperature</em>).</p><p>Abduction is the least reliable of the reasoning types. Chances are that the hypothesis you reached through abduction is not correct. For instance, the result of “wet street” might have had nothing to do with the rain — perhaps a pipe had bursted somewhere on the street during the night, or someone diligently sprayed the street with water. The rain, however, seems like a plausible explanation.</p><p>As such, abductive reasoning allows us to move through everyday situations without being stuck. As in, we don’t need 10.000 tries to make a simple decision.</p><p>To my knowledge, no AI model/algorithm to date has been able to perform abductive reasoning. Not in the ways I just described.</p><p>Those of you familiar with rule-based systems from the 1960s and 1970s, of course, can point at <a href="https://en.wikipedia.org/wiki/Mycin">MYCIN</a>, <a href="https://en.wikipedia.org/wiki/Xcon">XCON</a> and <a href="https://en.wikipedia.org/wiki/SHRDLU">SHRDLU</a> and claim that they’re capable of abduction. Others might bring up the examples of abduction cited by the Stanford AI index in <a href="https://aiindex.stanford.edu/wp-content/uploads/2022/03/2022-AI-Index-Report_Master.pdf">2022</a> and <a href="https://aiindex.stanford.edu/wp-content/uploads/2023/04/HAI_AI-Index-Report_2023.pdf">2023</a> as one of the most promising areas for future research (i.e., <em>abductive natural language inference</em>).</p><p>So, if machines were able to do “abduction” in the 1970s, why are they still not able to do what I claimed abduction can do (i.e., common sense reasoning)?</p><h3>Why abduction continues to be elusive</h3><p>There are two high-level reasons why even state-of-the-art models can’t perform abduction: <strong>conflation</strong> and <strong>architecture</strong>.</p><h4>Conflation: abduction is not the same as Inference to the best explanation (IBE)</h4><p>Historically, in computer science, many have used the terms IBE and abduction interchangeably. Even ChatGPT will tell you that the two are the same, or that abduction is a sub-set of IBE (depending on how you ask it). The <a href="https://plato.stanford.edu/index.html">Stanford Encyclopedia of Philosophy</a> echoes this sentiment, too. In fact, almost every paper in the larger field of computer science you’ll read about abduction, will tell you that it’s the same as IBE.</p><p>Yet, these are two <em>very</em> <em>different </em>constructs.</p><p>Generally, abduction covers the act of generating a novel case (<em>where learnings can be transferred from one context to another</em>). IBE, on the other hand, is a very special and more context-specific form of induction that doesn’t necessarily require you to identify patterns quantitatively (i.e., you don’t need to observe a pattern 10.000 times to formulate a rule). The exact ways in which these are different is a rather complicated philosophical discussion. If you want a deep-dive into that, I recommend <a href="https://www.jstor.org/stable/10.2979/trancharpeirsoc.51.3.300">this paper</a>.</p><p>For the purposes of this post, however, what will help us is to think about them within the <strong>rule</strong>, <strong>case</strong> and <strong>result</strong> structure and use specific examples like MYCIN and the abductive natural language inference model the Stanford AI Index cites.</p><p>MYCIN was an early expert system developed in the 1970s at Stanford to assist doctors in diagnosing infectious diseases. It relied on a knowledge base where each <strong>rule</strong> was expressed in terms of condition (IF — i.e., the <strong>case</strong>) and a conclusion (THEN — i.e., the <strong>result</strong>). It then utilised a <a href="https://en.wikipedia.org/wiki/Backward_chaining"><em>backward chaining</em></a><em> </em>inference mechanism, which allowed it to take a set of symptoms and patient data (<strong>result </strong>and <strong>case, </strong>respectively), and work backwards to identify and assign <em>a heuristic certainty score from 0 to 1 </em>to the <strong>rules</strong> that might explain the situation best. Namely, it reasoned from result and case to a rule (i.e., the pattern that inductive reasoning follows).</p><p>The work the Stanford AI index cites as an <a href="https://arxiv.org/abs/1908.05739">example</a> of <em>abductive natural language inference </em>(either when generating a hypothesis or selecting the most plausible one) is a bit trickier. Still, it is not abduction. In fact, I’d argue, it resembles IBE, but it follows the same pattern as the other ML models we discussed thus far — induction, followed by deduction.</p><p>Some background; in 2020, Bhagavatula and colleagues*, trained a transformer model conditioned on a dataset they call ART (containing ∼20K narrative contexts defined by pairs of observations (O1, O2) and 200K explanatory hypotheses). After training, they provided the model with a set of two observations and asked it to generate a plausible hypothesis to match (see Figure 4).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k_VVnUkscImziiTZrSESXA.png" /><figcaption>Figure 4: Abductive natural language inference (the figure is taken from <a href="https://arxiv.org/abs/1908.05739">arXiv:1908.05739</a>)</figcaption></figure><p>As you can see from the figure, when a transformer model (GPT-2+COMeT embeddings) is presented with O1 (e.g., “<em>Junior is the name of a 20</em><strong><em>+ </em></strong><em>year old turtle</em>”), and O2 (e.g., <em>“Junior is still going strong”</em>), it can generate a plausible hypothesis (e.g., <em>“Junior has been swimming in the pool with her friends”</em>) that might explain why we think Junior is still going strong.</p><blockquote>Why is this IBE and not abduction?</blockquote><p>Let’s abstract ourselves from the underlying ML model for a bit and think about how a human might perform such reasoning task. First, we’re provided with a <strong>result</strong>: <em>Junior is still going strong</em>, and we’re told what the <strong>case</strong> is (i.e., <em>Junior is a relatively old turtle</em>). Then, from these, what we’d do is to try and find a potential (context-dependent) <strong>rule</strong> that can explain the case and the result. For example, we can induce that <em>an old turtle that’s still going strong</em></p><ol><li>tends to play with its friends OR</li><li>has a healthy appetite OR</li><li>has good vitals</li></ol><p>and so on.</p><p>We can then choose the most plausible (to us) rule and apply it to our case of “an old turtle”. This will allow us to hypothesise that Junior could <em>have been swimming with her friends.</em></p><p>As already explained, the identifying of the potential rules from a limited set of observations is indicative of IBE and the act of drawing conclusions from these, tends to be a weak form of deduction.</p><p>We <em>as humans </em>understand that when one ages (be it a turtle or a human), their vitality tends to go down (arguably). This allows us to generate rules that are relatively ‘imbued with meaning”. A transformer model cannot do that. What it can do, however, is improve its predictions on the most probable combination of words that can follow the provided case and result (by applying induction and then deduction). The model has no underlying understanding that when Junior is having fun, she’s still going strong.</p><p>In fact, one might even go as far as to say that the work on <em>abductive natural language inference </em>is reminiscent of <a href="https://arxiv.org/abs/2201.11903">chain-of-thought </a>prompting. Granted, the instructions are presented to the transformer in a different manner.</p><p>What all these instances highlight, hopefully, is that what computer science labels as abduction isn’t abduction after all. Instead, it appears to be a context-specific variant of induction.</p><h4>Architecture: contemporary ML models are bound by induction</h4><p>The second reason behind state-of-art models’ inability to carry out abduction lies in their architecture. By definition, ML models are an induction-generating machines. This inclination is further strengthen by their so-called <strong>inductive bias</strong>.</p><blockquote>Inductive bias is an integral concept in ML referring to the inherent assumptions or preferences a model possesses regarding the types of functions it should learn. The bias helps guide the learning process by restricting the set of possible hypotheses, making learning more efficient and accurate.</blockquote><p>For example,<em> decision trees </em>focus on hierarchical structures and simple decision boundaries. <em>Support Vector Machines</em> aim to find wide margins between classes. <em>Convolutional Neural Networks</em> emphasise translation invariance and hierarchical feature learning in images. <em>Recurrent Neural Networks</em> are biased towards sequential patterns, <em>Bayesian Networks</em> model probabilistic relationships, <em>regularised linear models </em>prefer simpler models by penalising large coefficients, and general <em>transformers</em> like GPT-4 are characterised by their ability to capture sequential dependencies and relationships in data. These biases shape the models’ behaviour and suitability for different tasks. They also make it difficult to transfer learnings from one context to another.</p><h3>What we still need</h3><p>OK, by now we discussed a primer on reasoning and we saw that machines can indeed reason. They perform both deduction and induction. However, what we tend to intuitively term as “thinking” is facilitated by abduction, which continues to be elusive due to conflation and architecture.</p><p>So, what do we need then?</p><blockquote>How do we go about building something that can perform abductive reasoning?</blockquote><p>Well, first of all, we need to be able to properly define what abduction is and describe how it works. Sadly, not much work has been done in this regard. Especially, when it comes to identifying how abduction relates to induction and deduction. Or how it can be operationalised by machines. The only thing scholars tend to agree on is that abduction comes first, followed by induction and deduction.</p><h4><strong>So, what is abduction?</strong></h4><p>Abduction is not a monolithic construct. I’ve personally came across around 10 different types, depending on the scientific field to which they pertain. Even the philosopher who introduced the notion of <em>abduction</em>, Charles Peirce, doesn’t refer to it in a consistent manner.</p><p>However, there are three main types that can describe the fundamental functions abduction serves. The exact functions and how they came to be are too complex to cover in this post. So, here are the cliff notes.</p><p>First, we have the most straightforward abduction type — <em>explanatory</em>. The one we discussed thus far. To employ it, we start with an observation (<strong>result</strong>) and a <strong>rule</strong> that is easy to identify. The combination of the two then enables us to make a conjecture about the <strong>case</strong>. This is well-illustrated in the water-boiling example.</p><p>Then, we have<em> innovative abduction — </em>a type of abduction which allows us to reason from a (desired) <strong>result</strong> to a pair of a <strong>case</strong> and a <strong>rule. </strong>Namely, we only know what result we want to create and then we need to gradually define a <strong>case-rule</strong> pairing that will allow us to achieve said result. This type of abduction is usually used to generate novel ideas.</p><p>Finally, we have, I think, one of the most interesting types of abduction — <em>manipulative</em>. We use it in situations where the only thing we know is parts of the <strong>result</strong> (desired or otherwise). Furthermore, the context in which this result “lives” is defined by multiple hidden interdependencies. So, it’s not possible to start looking for/generating a suitable<strong> case-rule</strong> pair right away. Instead, we need to better understand the result and how it relates to its environment, so that we can reduce the level of uncertainty.</p><p>That’s where the so-called <em>thinking device/epistemic mediator</em> comes in. This could take the form of e.g., a basic sketch, prototype, or 3D model, serving as a means to enhance our understanding of the problem. By manipulating this <em>mediator</em> within the target environment, we gain a deeper understanding of the context. Consequently, we become better equipped to explore potential combinations of <strong>rules </strong>and <strong>cases</strong>. Additionally, it allows us to establish associations that aid the transferring of knowledge from one domain to another. A simplified version of this kind of thinking is commonly applied in stereometry, for instance.</p><p>As I said, much work still needs to be done in explaining the relationships among these abduction types and their relatedness with other reasoning approaches. This endeavour is becoming increasingly critical, however, as it holds the potential to offer valuable insights into the transferability of insights across different domains. Especially, in light of the renewed interest in reasoning we see in the field — be it via IBE, “reasoning through simulation and examples”, or System-1 and System-2 thinking.</p><p>Amidst all that, it seems pertinent to understand how not to conflate the different types of reasoning that can be performed by a machine. Because, yes, machines can reason. They simply cannot perform the full reasoning spectrum.</p><p>*Other interesting work on IBE can be found in <a href="https://arxiv.org/abs/2305.14618">this paper (they do equate abduction with IBE, however).</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=40edafce293d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/on-why-machines-can-think-40edafce293d">On Why Machines Can Think</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Aligning Human and AI Objectives]]></title>
            <link>https://medium.com/deus-humanity-centered-ai/aligning-human-and-ai-objectives-3f11e7c335fb?source=rss-bacf8cf8265e------2</link>
            <guid isPermaLink="false">https://medium.com/p/3f11e7c335fb</guid>
            <category><![CDATA[risk-management]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[unintended-consequences]]></category>
            <category><![CDATA[alignment]]></category>
            <category><![CDATA[eu-ai-act]]></category>
            <dc:creator><![CDATA[Niya Stoimenova]]></dc:creator>
            <pubDate>Wed, 21 Jun 2023 08:38:39 GMT</pubDate>
            <atom:updated>2023-08-01T10:07:44.747Z</atom:updated>
            <content:encoded><![CDATA[<p>Even if you don’t actively work in AI, chances are, in the past few months you’ve heard something about the “<strong><em>existential threat of AI</em>” </strong>or the “<strong><em>alignment problem of AI</em></strong>”.</p><p>Two out of the three so-called godfathers of AI are sounding the alarm, other people are signing petitions for <a href="https://www.reuters.com/technology/musk-experts-urge-pause-training-ai-systems-that-can-outperform-gpt-4-2023-03-29/">a 6-month “pause”</a> on the development of “<em>large models like ChatGPT</em>”, others discuss whether AI is <em>sentient</em> and <a href="https://in.mashable.com/apps-and-software/35564/google-fires-engineer-for-saying-its-ai-has-a-soul">has a <em>soul</em></a><em>, </em>how all of us have to be scared, or how there’s nothing to be worried about.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KLenYN8kgksqYylxZv8fsQ.png" /><figcaption>An <a href="https://www.nytimes.com/2023/05/01/technology/ai-google-chatbot-engineer-quits-hinton.html">interview </a>with Geoffrey Hinton in the New York Times after he left Google.</figcaption></figure><p>In all that, the <em>alignment problem</em> becomes implicitly connected either to sci-fi scenarios or to large, generative, complex models like the ones behind ChatGPT or Midjourney. And it’s treated like a research problem. But it’s neither solely a research issue nor occurring only when we have complex models.</p><p>Even relatively simple AI models have alignment problems. There’s no sentience associated with them. Yet, they can create real harm. Fast.</p><p>So, instead of going on a tangent discussing the probability of having a sentient system that has a soul (which potentially we might need to ban), let’s focus on the <strong>problem of alignment</strong>.</p><h3>What is the alignment problem?</h3><p>In simple terms, the alignment problem is the notion that the behaviour of an AI model should<em> at all times</em> be aligned with the values and objectives of humans.</p><p>To illustrate the difficulty of ensuring this, here’s the oversimplified example I was given years ago (similar to the <a href="https://cepr.org/voxeu/columns/ai-and-paperclip-problem">paperclip problem</a>).</p><p>Imagine we’ve built an AI model and gave it a sole objective: <em>learn from data and interact with humans to find ways to increase the happiness of humanity. </em>The more data we give to the model (e.g., the more it interacts with humans), the more it learns the patterns of how to achieve its objective: increase happiness. Until one day, it induces that smiling is directly and highly correlated with happiness: <em>happy people smile</em>.</p><p>An efficient way to ensure everyone smiles is to kill all people who don’t smile. There will be (largely) no problem with the logic employed by the model: if we have only people who smile, then it has increased the happiness of humanity. But as we all know, just because someone smiles, doesn’t mean they’re happy. Happiness is much more complicated than that. Also, killing people is bad.</p><p>Moral of the highly over-simplified story: we need to make sure we state the objectives of the system in a very clear. In a way that cannot be “interpreted” in harmful ways. Yet, this is extremely difficult. And that difficulty has little to do with the complexity of the model.</p><h3>The alignment problem of simple models</h3><p>It’s 2016 and Microsoft has decided they want to build a new <a href="https://www.theverge.com/2016/3/24/11297050/tay-microsoft-chatbot-racist">Twitter chatbot called Tay</a>, which is supposed to have a casual and jokey tone (so that it can sound like a millennial). For reference, the model they were using is infinitely simpler than the one ChatGPT uses.</p><p>Microsoft took precautions, of course. The bot was a version of another bot that was used for a very successful pilot in China, the team worked with writers and stand-up comedians to ensure Tay <em>truly sounds like a millennial</em>, and they also followed the inclusive design guidelines of Microsoft. All that led to the bot’s first tweet: <strong>hellooooooo w🌍rld!!!</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d7-llEVDAm6Th1upRaN2AA.png" /><figcaption>Microsoft’s bot, Tay, on Twitter</figcaption></figure><p>After just a few hours of interacting with humans, the well-meaning hello world turned into this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j6dsJZTMnhxa6TTfGnpJ4Q.png" /><figcaption>An overview on some of the tweets generated by Tay</figcaption></figure><p>Microsoft promptly took the bot down and apologised. Still, at the end of the day, Tay did what it was supposed to do: <em>interact with people on Twitter</em> and <em>sound like a millennial</em>. No one in Microsoft was happy about the way it went about achieving its objective, though.</p><p>Let’s travel further back in time to 2012, when Knight Capital (a top-tier Wall Street firm) decided to use <a href="https://www.bugsnag.com/blog/bug-day-460m-loss">a model to do robo-trading</a>.</p><p>After months of extensive work and testing the model, Knight Capital released it into the real world. The model was live for around 45 minutes. In that time, it started overbidding on stocks and executed tons of trades. After Knight Capital had realised what had happened, they tried to reverse the trades and sell the stocks off without much success. The firm lost approximately <a href="https://archive.nytimes.com/dealbook.nytimes.com/2012/08/02/knight-capital-says-trading-mishap-cost-it-440-million/">$440 million</a>. They were also fined additional $12 million from the Securities &amp; Exchange Commission in the US.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_0sK7Ut_4QAxwBLFQeXx8A.png" /><figcaption>Background image source: El Confidencial</figcaption></figure><p>After investigation, it turned out that the trading model interacted with a practically defunct legacy system no one had used since 2003. As a result, the model “assumed” it was in a test environment.</p><p>Knight Capital was bought by Goldman Sachs for pennies on the dollar to avoid bankruptcy.</p><p>Neither Microsoft nor Knight Capital used large, generative AI trained on massive amounts of data. The models they used were relatively simple and explainable. Both companies took good measures to ensure their models will perform as intended. Microsoft had a successful pilot, worked with writers and followed inclusive design guidelines. Knight Capital carried out extensive tests. Yet, the moment their models became part of a larger system, they started behaving in unintended harmful ways.</p><p>This is not to say that putting effort into aligning the model upfront is useless. On the contrary. However, we need to acknowledge the reality that once a model gets deployed, it will, inevitably, become misaligned. And when it does, the impact on the business and society could be significant.</p><p>So, then, the question is not simply “How do we align human and AI objectives”, but, more importantly:</p><blockquote><strong>How do we mitigate misalignment of human and AI objectives?</strong></blockquote><h3>Mitigating misalignment</h3><p>One way to mitigate misalignment on time is by actively <em>identifying</em> and <em>addressing </em>the <strong>feedback loops</strong> that influence the performance of the model.</p><p>These feedback loops come from the way humans use the model (as was the case with Tay), from its interaction with other models and, as we saw from the Knight Capital, from legacy systems.</p><p>To see how we can do that, let’s use a fictional example.</p><p>Imagine we’ve just started a fintech company that will “revolutionise” the credit card application process. In the spirit of simple examples, our startup uses a classification model to decide whether someone is eligible for a credit card or not.</p><p>The first thing we need to do is monitor what the model is doing and whether the data we’re retraining it with is the type of data we want to use.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/690/1*GSRCKcqs3XVttTXLuH79-Q.gif" /><figcaption>An overview of model performance metrics, shown in real-time</figcaption></figure><p>In this specific case, we can track the model’s F1 score (e.g., its precision and accuracy), as well as the drift in its input features (p-values). Tracking even something as simple as that can give us an accurate depiction of what is going on with our model, at all times.</p><blockquote>If Microsoft monitored their model performance, they could’ve avoided the Tay situation, because they would’ve seen that the data they were retraining the model with was different than their initial training data.</blockquote><p>But talking about a complex system of humans, models, and legacy systems (be it a fictional one), and only looking at the model performance is like looking at a Van Gogh painting from up close. Sure, you can see colours and brushstrokes, but you have no clue what the hell you’re looking at.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*VB1t09WZsZVGZ1Yjp-4b9A.gif" /><figcaption>Van Gogh’s Starry Night Over the Rhône</figcaption></figure><blockquote>Only when you take a few steps back, are you able to really understand how Van Gogh used colours and brushstrokes to create his paintings.</blockquote><p>If we’re to identify the feedback loops for deployed models, we need to figure out ways to take a few steps back.</p><p>One way to do that is to take a systemic overview — track not only what the model is doing, but also how the system responds — other models, how humans are impacted, how the business is impacted, and so on.</p><p>In our fictional case, we can track, for example, the revenue our model creates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/1*SBNe5BNzJa9GyKm5jmlf9w.gif" /><figcaption>An overview of the revenue and model performance of a deployed classification model</figcaption></figure><p>The moment we include a systemic overview (albeit a simple one), we can notice 2 important things.</p><p>First, when the model F1 score goes up, the revenue goes up, too.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NRNHZXg9nk5E0jcEtzeUsQ.png" /><figcaption>An overview of business and model performance indicators</figcaption></figure><p>In this case, we can go to the model overview and check to see what might’ve happened. For instance, we can see that the data distribution we were getting has changed compared to our training data set, which caused an improvement in the model performance, which influenced the revenue. This information we can use to improve our model and target customers more effectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ben7gsQSvljeDfKQG9Ak8w.png" /><figcaption>An overview of drifted features at the identified time interval</figcaption></figure><p>The second thing we can notice is a bit less straightforward than the first one. There is a part in the lifecycle of our credit card solution when suddenly the revenue drops, but the model is performing as well or slightly better than before.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PmbsyXzVBI4xWc1F8UNvWg.png" /></figure><p>If we try to repeat what we did before and check the data distributions we’re getting, we can see that nothing has changed in that. No features are drifting. There seems to be no reason why the revenue is dropping. Yet, the revenue has taken a significant plunge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZqI_1eSjp7BqOTdXHsYjIA.png" /><figcaption>The revenue drop is not associated with the current performance of the model</figcaption></figure><p>This is actually a relatively common problem and it’s often referred to as <em>delay of ground truth. </em>Meaning, the model has drifted weeks or months ago but at that time, there were no significant implications for the system (aka revenue). And we can only see the impact of this drift now.</p><p>So, then, it becomes incredibly important to ensure that we’re able to identify the intervals in which the model has impacted the system. It’s also very important to know why this happened.</p><p>The platform my team and I have been developing does that — we identify the <strong>most likely</strong>,<strong> likely</strong> and <strong>less likely</strong> time intervals in which the model has impacted our revenue, over its entire lifecycle.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*4RM-za_MRYbIYJIYnDezYQ.gif" /><figcaption>Identifying the most likely time interval in which the model has impacted the system</figcaption></figure><p>Then provide a snapshot of the system at the time of impact, plus some examples to put things into context.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*Utvq8ixW5Bnaz-9TU1ZBcw.gif" /><figcaption>A snapshot of the system at the time of impact.</figcaption></figure><p>All of this is, of course, only the starting point. At the end of the day, alignment is contextual — it depends on what one considers alignment: alignment to whom, by whom and in what way.</p><p>As of today, there’s only one thing that has evolved to deal well with complex contextual predicaments like this one — humans. So, the most important step after that is to ensure we have ways for humans to provide additional feedback and identify other feedback loops we might need to be aware of. We’re currently working on that.</p><p>If you’re interested in exploring the potential for expert feedback in identifying and addressing feedback loops, let’s discuss.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3f11e7c335fb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/deus-humanity-centered-ai/aligning-human-and-ai-objectives-3f11e7c335fb">Aligning Human and AI Objectives</a> was originally published in <a href="https://medium.com/deus-humanity-centered-ai">DEUS: human(ity)-centered Artificial Intelligence</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to make the most out of collaboration]]></title>
            <link>https://medium.com/onami/how-to-make-the-most-out-of-collaboration-f1a66ab91b9b?source=rss-bacf8cf8265e------2</link>
            <guid isPermaLink="false">https://medium.com/p/f1a66ab91b9b</guid>
            <category><![CDATA[multidisciplinary]]></category>
            <category><![CDATA[design-thinking]]></category>
            <category><![CDATA[design-doing]]></category>
            <category><![CDATA[collaboration]]></category>
            <category><![CDATA[design]]></category>
            <dc:creator><![CDATA[Niya Stoimenova]]></dc:creator>
            <pubDate>Wed, 09 May 2018 19:25:56 GMT</pubDate>
            <atom:updated>2018-05-09T19:25:56.240Z</atom:updated>
            <content:encoded><![CDATA[<h4>The holy trinity of multidisciplinary teams</h4><p>We know… Yet another one with their take on making the most out of collaboration. Yes, the topic is definitely as cliché as a topic could get. Yet, even though working in multidisciplinary and multicultural teams (or rather the aspiration to establish such) has become the norm in most companies, they are still (very) challenging.</p><p>So instead of repeating all of their benefits or the reasons why people might develop a strong aversion towards working in such, we’ll give you some practical observations (and a few tips) you can start using right away.</p><p>For the past three years we have been studying the factors that influence multidisciplinary collaboration, while facilitating and being part of such teams in three multinational companies. Depending on the context and the type of team you’re working with, there are different factors that can influence the outcome. However, three of them seem to be omnipresent. We believe that when addressed, they can help you build optimal collaboration.</p><h4><strong>Clarity</strong></h4><p>No, not clarity of what the end result will precisely look like. If you know what your end result is in detail early on, something is terribly wrong. However, clarity of the general way of working, plans, goals and communication is something you should establish right from the start. In fact, lack of such can prevent the team to accomplish the task at hand<a href="#_ftn3">[1]</a>. Thus, making these things explicit is one of the most important tasks a team has.</p><p>Of course, when you’re dealing with something completely new it’s very difficult to know the answer to these questions. But, as everything around you is bound to be ambiguous, you have to establish some sense of clarity. Otherwise people will drown in the chaos.</p><p>How to achieve the coveted clarity? It’s simple — agree on all these things upfront. This, obviously, is much easier said than done. So how do you bring everyone on the same page? Talk it out, right? Unfortunately, oftentimes the desire to agree leads to endless discussion, which quickly turns into a nonsensical loop of dead ends. In the end, everyone walks away with a slightly different interpretation on the discussion.</p><p>To avoid that, we usually just sketch things up. The result is always different as it’s so much dependent on the context, but we always start by simply writing down a short statement we think everybody agrees on. This always provokes discussion. Discussion is good, unless it drags for too long. So when someone makes a statement, we simply keep asking the “why” question and try to map out the different opinions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lmt8YN6x6kNGiTHX7adEKg.jpeg" /></figure><p>It doesn’t have to be a fancy sketch. Simple squares, circles and stick figures with the help of short explanations do miracles. Making things visual together as a team will allow you to quickly establish a shared understanding of whatever it is you’re doing.</p><h4>Trust</h4><p>Now, hands down, trust doesn’t come easy, especially when a big part of our culture is based upon the deeply-seeded notion <em>“don’t trust strangers (in a white van)”</em>. However, you can create an environment, which makes the trusting procedure a bit easier. How? Start by having one person in the team everyone trusts. It helps tremendously if that’s the facilitator/manager.</p><p>How do you become that person? The conventional way — taking your time to prove your worth. Most often than not, you won’t have such luxury. So you need to be resourceful.</p><p>What we do in such cases is simple — we have individual talks with every person on the team before the official start of a project/workshop. During this talk we try to get to know them better, their strengths and weaknesses, and their values, so we can take into account their personalities. As trust is a two-way street, we also tell them a bit about the way we work and what our values are.</p><p>These simple tactics can really make the difference. They helped us to get people along on our journey, which most of the times requires them to do “weird” things such as “playing with Lego” to come up with ideas or cut and paste pictures to imagine the future.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y7aBT8hQ_lecTRWE7q1cug.jpeg" /><figcaption>The initial reactions you’re bound to get when you ask people to play with Lego</figcaption></figure><p>As cliche as this sounds, food and alcohol could really help in that regard as well. Just make it your point to have lunch together as a team or go for a dinner together. When people get to know each other outside of their job description, the level of trust will increase.</p><p>Another thing all good managers we’ve worked with have in common is: they continuously reinforce the notion that the team, not them, is the expert. Their role as is to simply guide the team through the process, by regularly asking for advice and help.<a href="#_ftn4">[2]</a> We also noticed that clarity helps tremendously to form trust.</p><h4>Conflict</h4><p>The natural reaction of most people is to avoid it like the plague. Arguably no mentally sane person enjoys it. However, conflict can help you get unstuck and come up with surprising ideas. As Socrates famously said, the truth is born through dialogue (or in our case, conflict). In fact, innovation rarely happens in the absence of conflict<a href="#_ftn5">[3]</a>. However, it’s imperative to make sure the conflict doesn’t migrate from the area of ideas and points of view to personalities.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.ted.com%2Ftalks%2Flinda_hill_how_to_manage_for_collective_creativity&amp;url=https%3A%2F%2Fwww.ted.com%2Ftalks%2Flinda_hill_how_to_manage_for_collective_creativity&amp;image=https%3A%2F%2Fpe.tedcdn.com%2Fimages%2Fted%2F56539ed2fb25244138c4d6400168585fd28d8c65_240x180.jpg%3Flang%3Den&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=ted" width="560" height="315" frameborder="0" scrolling="no"><a href="https://medium.com/media/1d4fa038b5f176e6bf48d4d8c796404c/href">https://medium.com/media/1d4fa038b5f176e6bf48d4d8c796404c/href</a></iframe><p>Managing conflict is difficult. If it were easy, the concept of warfare will be foreign to humanity. Yet, if managed in a controlled environment, it can help the team reap multiple benefits.</p><p>We have a simple formula we apply when dealing with conflict. Of course, you must’ve heard about the “yes and”, but if you’ve established the clarity and trust already, the conflict will be much easier to deal with. The formula we follow is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4GF-VOldyCdVG81oO_XJFA.jpeg" /></figure><p>So if someone wants to disagree, they have to provide a reason why they disagree and an idea on how to do what they’re proposing. This is where the good manager/facilitator is extremely important to moderate the discussion. Conflict is good only when moderated.</p><p>Then when two disagreements are voiced, you should remember, as a moderator, that even in the most opposing disagreements, there’s always a common ground. Look for it and then build on that. A way to look for the common ground is to ask the people who disagree; try to uncover the reason they disagree.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YC4PgZrmeYhHY-6HhyQ2uA.jpeg" /></figure><p>What we find useful are two rules of thumb. First, don’t ever be intimidated by conflict. Look at it as something that can help you. Of course, to properly do so, you should be confident in your own abilities.</p><p>Second, always have someone who moderates the discussion and doesn’t allow it to get personal. Also, clarity (especially of rules) and trust are your best friends here.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8oa4VOScdCtT1GdD3AV4pw.jpeg" /></figure><p>As you’ve probably guessed already, these three are tightly interconnected and amplifying each other. Once you have clarity, it will be easier to establish trust and once you have both, it will be easier to deal with conflict.</p><p>However, they are not meant to be the panacea that will rid you of all your earthly woes. They simply provide a strong foundation, upon which you can build your optimal collaboration. So, next time you work with a multidisciplinary team, try to pay deliberate attention to them and see what happens.</p><p><a href="#_ftnref3">[1]</a> Tuckman, B. W., &amp; Jensen, M. A. C. (1977). Stages of small-group development revisited. <em>Group &amp; Organization Studies</em>, <em>2</em>(4), 419–427.</p><p><a href="#_ftnref4">[2]</a> <a href="https://hbr.org/2017/01/the-neuroscience-of-trust">https://hbr.org/2017/01/the-neuroscience-of-trust</a></p><p><a href="#_ftnref5">[3]</a><a href="https://www.ted.com/talks/linda_hill_how_to_manage_for_collective_creativity?utm_campaign=social&amp;utm_medium=referral&amp;utm_source=linkedin.com&amp;utm_content=talk&amp;utm_term=business#t-4831">https://www.ted.com/talks/linda_hill_how_to_manage_for_collective_creativity?utm_campaign=social&amp;utm_medium=referral&amp;utm_source=linkedin.com&amp;utm_content=talk&amp;utm_term=business#t-4831</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f1a66ab91b9b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/onami/how-to-make-the-most-out-of-collaboration-f1a66ab91b9b">How to make the most out of collaboration</a> was originally published in <a href="https://medium.com/onami">Onami</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Successful Organizations]]></title>
            <link>https://medium.com/@niya.stoimenova/building-successful-organizations-5397550eff8a?source=rss-bacf8cf8265e------2</link>
            <guid isPermaLink="false">https://medium.com/p/5397550eff8a</guid>
            <category><![CDATA[ambidexterity]]></category>
            <category><![CDATA[innovation]]></category>
            <category><![CDATA[design-led-ambidexterity]]></category>
            <category><![CDATA[user-centered]]></category>
            <category><![CDATA[design-thinking]]></category>
            <dc:creator><![CDATA[Niya Stoimenova]]></dc:creator>
            <pubDate>Tue, 14 Mar 2017 22:58:01 GMT</pubDate>
            <atom:updated>2017-07-21T19:51:22.513Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gsf85ZBgAIeOizMNRzkZCw.gif" /></figure><blockquote>How do you build an organization that can continuously design truly meaningful solutions?</blockquote><p>According to <a href="http://www.mckinsey.com/business-functions/organization/our-insights/six-building-blocks-for-creating-a-high-performing-digital-enterprise"><strong>McKinsey</strong></a>, the lifespan of an organization is declining exponentially: in 1958 it was 61 years, in 1985 — 25 years, 2011 — 18 years. By 2027 75% of all S&amp;P 500 companies will be gone.</p><p>These are not just numbers. Think about Kodak, the most cliché example. Or how Spotify is outperforming iTunes — the very platform that made Apple inimitable. Or how Google has been searching for a sustained revenue model other than advertising for years and not being quite there yet. Or how Amazon and its drone delivery is replacing postal services.</p><p>We all know we can’t predict what will happen in the next 10 years. We can’t even predict the next 3. True, we have a grasp on the upcoming technologies, but the effect they will have on society is completely unfathomable. Who would’ve thought 10 years ago, when the first iPhone was introduced that the “death selfie” will be a thing?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hddBZUN4Jaa74Pe1.png" /><figcaption>She died while taking the selfie</figcaption></figure><p>The historically unprecedented technological and market changes we are facing make innovating for the sake of innovation somewhat irrelevant. The frequency, the type and the value innovation brings to the customer (user) is what matters. We need to design organizations that can provide all that by continuously adapting to the ever-changing context they are in.</p><p>A theory called <a href="https://hbr.org/2004/04/the-ambidextrous-organization"><strong>Organizational Ambidexterity</strong></a> addresses this problem. Its main premise? A company can be both relevant (create meaningful innovations) and profitable as long as it pursues both <strong>exploitation</strong> and <strong>exploration</strong> activities.</p><p>But first, what is ambidexterity? Literally, this is the ability to use both your left and right hand simultaneously and equally well. Thus, you’re able to use your left analytical brain hemisphere and the right creative one simultaneously and equally well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZpV_Pqj3jdtYVPqT9BvGuw.jpeg" /><figcaption>Fun fact, only 1% of the human population can do that (according to Wikipedia).</figcaption></figure><p>Translating this to Organizational Ambidexterity, we have <strong>exploitation</strong> corresponding to the left hemisphere. Exploitation is an activity characterized by refinement of choice, production, efficiency, selection, implementation and execution. Think of it as Henry Ford’s assembly line.</p><p>Thus, all exploitation activities are geared towards making the already established routines as efficient as possible and creating quick results. The usual suspects here are methods like Lean, Six Sigma (Lean Six Sigma). This, somewhat logically, results in predominantly <strong>incremental changes</strong> of both products and processes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m99-L0q5eWuWiyXiOpMefA.jpeg" /></figure><p>Then we have the right hemisphere. Its organizational equivalent is <strong>exploration</strong>. This type of activity is characterized by search, experimentation, play, flexibility and investigation. Think of it as the Columbus’ expedition in search of a western route to India.</p><p>Its structure, processes and values are aimed at finding answers to particularly ambiguous problems, which often result in <strong>radical innovation</strong>. But to make this work, companies need more flexible methods and infrastructures, which will allow the employees to experiment and improvise.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VYXKg-VTKDifMtCNKRR4mQ.jpeg" /></figure><p>As you can imagine, trying to be both Ford (exploitation)and Columbus (exploration) simultaneously is quite the feat. Scholars and companies have been trying to achieve it for years by applying different ambidexterity types <a href="#_ftn1">[1</a>] without much success. This, in fact, is what I see happening in all companies I’ve worked with.</p><p>Actually, if you think about it, the combination is kind of counter-intuitive. While exploitation promotes clearly defined processes and structure, exploration requires “looser controls” and flexibility. Understandably, most companies are good at only one of them. Nevertheless, it is the symbiosis of the two that ensures a sustained pace of meaningful innovation.</p><p>The question, then, becomes:</p><blockquote>How can a company perform both exploitation and exploration simultaneously and equally well?</blockquote><p>Let’s say we’ll think about them as two wheels (actually, the visuals already suggested that). What happens when you turn the two together? They will rotate in opposite directions, essentially counteracting each other.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t2MdxFpI-VoEDWfg609M4g.gif" /><figcaption>Exploitation — Exploration</figcaption></figure><p>Hence, we need something to ensure this won’t happen. This something is a Catalyst. It’s simply another wheel inserted between the two that ensures they rotate in the same direction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WprZxpE_k06Mym767GW7kQ.gif" /></figure><p>However, the Catalyst has another function, too — it speeds up the exploitation-exploration interaction, as its name suggests. A perfect method to play the role of the Catalyst is Lean Startup or the Design Sprint as they have elements of both exploration and exploitation.</p><p>So, we have these three wheels working together. However, they will not amount to much unless two more elements are present: strong <strong>Executive Support</strong>, ensuring resources (money, people, time) are available and the <strong>Context</strong> including both the employees and customers, which will trigger Exploration. The combination of these elements will make the <strong>Lighthouse </strong>work and thus, the company will achieve organizational ambidexterity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MtHTe5ItEcWJicwXz6M9PA.jpeg" /><figcaption>The Lighthouse Model</figcaption></figure><p>We call this type of ambidexterity — design-led. In case you feel like I brought Design into the picture rather abruptly, let me elaborate a bit on:</p><blockquote>What is the role Design can play?</blockquote><p>On the one hand, Design and its tools, methods and mindset are very good at helping people navigate ambiguity. Its specialty? Figuring the <strong>how</strong> and the <strong>what</strong> when you only sort of, kind of know the <strong>why</strong>. So, it’s the perfect guide for exploration. In fact, when done properly, design is the closest thing to a crystal ball we have due to its human-centricity and affinity towards wicked problems.</p><p>On the other hand, methods like the Design Sprint were developed to accelerate the pace of creation and validation of ideas. So it can also be used to accelerate both exploration and exploitation.</p><p>The potential of using Design (Thinking) to build ambidextrous organizations is quite promising and an increasing number of firms are trying to adopt it.</p><p>Of course, models are just that — models. A (really) simplified version of reality and thus, they don’t account for the specific challenges in applying and implementing new ideas successfully. Thus, every solution should be context-specific.</p><p>Regardless, the Lighthouse Model and the notion of design-led ambidexterity gives us a different perspective on how to design organizations that can continuously create meaningful innovations.</p><p><a href="#_ftnref1">[1]</a> For more information on this, check this article O’Reilly, C. A., &amp; Tushman, M. L. (2013). Organizational ambidexterity: Past, present, and future. <em>The Academy of Management Perspectives</em>, <em>27</em>(4), 324–338.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5397550eff8a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[KLM Airlines: 5 Lessons Learned Using Design Sprints to Prototype the Airport Experience]]></title>
            <link>https://medium.com/sprint-stories/the-five-most-beneficial-side-effects-of-the-design-sprint-9ae265b8f0cf?source=rss-bacf8cf8265e------2</link>
            <guid isPermaLink="false">https://medium.com/p/9ae265b8f0cf</guid>
            <category><![CDATA[collaboration]]></category>
            <category><![CDATA[google-ventures]]></category>
            <category><![CDATA[xgate]]></category>
            <category><![CDATA[klm]]></category>
            <category><![CDATA[design-sprint]]></category>
            <dc:creator><![CDATA[Niya Stoimenova]]></dc:creator>
            <pubDate>Tue, 20 Dec 2016 19:31:04 GMT</pubDate>
            <atom:updated>2016-12-21T00:12:38.728Z</atom:updated>
            <content:encoded><![CDATA[<h4>How we adapted the GV sprint to KLM’s X-Gates</h4><p><em>by Niya Stoimenova and Luuk Roos</em></p><p>Have you ever wondered why it took so many years to have WiFi in (some) planes? Or how they manage to keep losing luggage even though virtually everything can be tracked? You’re not alone.</p><p>The airline industry operates on pretty much every single regulation you can possibly fathom and thus, new technologies take ages to be adopted and implemented. Somewhat naturally, regulations have become the main suspect when it comes to hampering progress in the industry. However, they are there for a very good reason. So then the most logical question is, how do you accelerate the pace of innovation despite all restrictions.</p><p>KLM began its mission to figure this out a bit over a year ago with the creation of a new department called the X-Gates. But first, let’s take a step back. Who is KLM?</p><p>KLM Royal Dutch Airlines is the oldest commercial airline in the world, whose mission is to become the most innovative, customer-centric and efficient European airline. You might have heard of some of their social media campaigns such as Meet &amp; Seat or #happytohelp (if not, you should Google them). Enter the X-Gates.</p><h4>X-Gates: A prototyping environment for the airport</h4><p>Situated in Amsterdam Airport Schiphol, X-Gates includes dedicated gates (yes, you guessed that right), but also since this fall the Departure hall and dedicated flights. The idea behind what they do is very simple.</p><p>Imagine a place where ideas can be conjured up, prototyped rapidly and tested in live operation in a matter of days. You can test whatever you come up with before or during real flights, with real passengers and staff without disrupting the operation. And since Schiphol is the second largest hub airport in the world, you get to do this with people from all over the world. That’s actually the most unique and powerful feature of the X-Gates — the genuine and culturally diverse feedback.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*86f7OcQSWLjuILhCBB87sg.gif" /><figcaption>KLM’s X-Gates at Amsterdam Airport Schiphol</figcaption></figure><p>The idea does not get more straight-forward than that. Alas, like all good things in life, it’s much easier said than done and most great ideas are doomed to turn into a crappy reality.</p><p>Why are we being so dramatic, you might ask. Good question! Is it really that difficult to do something like that? Another good question! Short answer: yes, it is really that difficult. Long answer: let us paint you a picture.</p><p>The context is complex, messy, chaotic and most of the time you can’t possibly grasp everything that’s going on around you. Even if you try really hard. Then, you have to make sure that whatever you’re going to try won’t stop the staff (e.g. gate agents, flight attendants, people who load the luggage into the plane) from doing their job and the passengers from reaching their destination safely and on time.</p><p>Now add airport regulations to the mix. We all know how much time it takes to get through security even though you complied with all requirements. Imagine doing so while armed with a lethal weapon for world destruction. That’s what it’s like when you want to bring even something as simple as a pair of scissors to the gates without a very formal permission to do so (okay, we’re exaggerating, but not a lot).</p><p>And, of course, one must not forget that KLM is a big company and thus many different people are bound to be involved in whatever you want to do. So, naturally, you also have to make sure that all of them are on the same page. Thus, being extremely well-versed in company politics, or finding someone who is, will come in handy. The last and most important ingredient in the mix is involving and coordinating a lot of different people — the X-Gates team, passengers from all over the world and KLM staff.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*Nb2fud3DNLxqu1JH_wbNsA.gif" /><figcaption>The context is complex, messy, chaotic and most of the time you can’t possibly grasp everything that’s going on around you. Even if you try really hard.</figcaption></figure><p>So how did we protect the X-Gates idea from becoming a crappy reality? Several factors helped us in doing so: executive support (so we didn’t have to dip our toes into the deep waters of company politics), the X-Gates team (who really wanted to make a difference), a collaboration with the Industrial Design faculty at TU Delft (where we’re currently following our Master’s) and, of course, the Design Sprint (playing a crucial role in navigating the context by providing quick results).</p><p>In fact, adopting the <a href="http://gv.com/sprint">Design Sprint</a> as an element of the X-Gates way of working helped us not only to navigate the context, but also to shape it. However, we won’t discuss its structure as the <a href="http://thesprintbook.com/"><em>Sprint</em> book</a> does that wonderfully.</p><p>Instead, we will share with you the five most beneficial side effects the Design Sprint has in a big company like KLM. But before we do that, let’s briefly discuss the peculiarities of the sprint we run. There are three major ones:</p><h3>1. <strong>Pay attention to the context</strong></h3><p>It doesn’t take a genius to realize that understanding your users will provide you with the optimal foundation to create meaningful solution. So we stick to one rule: <strong>get your information from several different methods.</strong> Quantitative data, interviews and observations are the usual suspects and for a good reason — they provide a lot of insights, just like the <a href="https://library.gv.com/the-gv-research-sprint-a-4-day-process-for-answering-important-startup-questions-97279b532b25#.ba9469890">Research Sprint.</a></p><p>However, we didn’t only want to involve the X-Gates team and KLM employees (management and development), but also KLM staff and passengers and give all of them an active role in the creation of the new context. And when it comes to doing so, all these methods fall somewhat short.</p><p>That’s why we use <strong>context mapping</strong> before each sprint — a method that has worked for us time and time again. It’s based on the premise that people are experts of their own experience and through photos, icons, colors and phrases they can express their latent needs and tacit knowledge<a href="#_ftn1">[1]</a>. As such, it helps you to find the answers to questions you didn’t even know existed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*0o056K4FiHQjPE_MgXh-vg.gif" /><figcaption>Context-mapping sessions with passengers at the gates</figcaption></figure><h3>2. <strong>Very easy-to-recognize names</strong></h3><p>Why? Well, as we already mentioned, we want to involve KLM staff, employees and passengers as much as possible in everything we do. Thus, ensuring that they not only understand the way of working but also feel inspired to get involved in it is crucial.</p><p>Now, you might ask, why didn’t we just stick to the names the Sprint book suggests? Excellent question! Indeed, they are very easy to understand and perfectly convey what has to be done each day. However, we also wanted to make use of the already existing analogies in people’s heads. Now in plain English — for instance, we named Mondays “Sherlock” instead of “Map” or “Design Analysis”, as this presupposes that not only do we map the problem, but we also try to figure out why it exists in order to solve the mystery. See the difference?</p><p>We try to give people an idea about what has to be done, but also why they should do it. It’s a very fine, but important distinction. Besides, it turns out such analogies are quite catchy. Therefore, we also renamed the Design Sprint into Treasure Hunt.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Os2MvZeNx5KHXBQ3m_jklQ.gif" /><figcaption>Our Treasure Hunt Map</figcaption></figure><h3>3. <strong>Think in timelines</strong></h3><p>This one is very similar to the <a href="https://library.gv.com/sprint-week-wednesday-900fe3f2c26e#.4z2d2hc1f">GV storyboards</a>, the only difference being that in the second half of the Mickey day, we encourage the team to think of as many detailed steps of their idea as possible and put them on a timeline. We also encourage them to try and sketch things out.</p><p>The role of the facilitator here is crucial, since if she makes very simple and somewhat ugly sketches throughout the day, the team feels more confident to draw on their own. Using such timelines has proved to be really helpful when we want to prototype a service or its interaction with a product. It also makes it easier to combine the generated ideas in one coherent story and helps us to decide what to prototype during the Lego day. Last but not least, having detailed timelines makes the Mickey and Lego days more action oriented, since small steps are always easier to take and work with than one big idea.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/1*iWNrBReL3-zWV972NH1ohA.gif" /><figcaption>Explaining an idea with a detailed timeline</figcaption></figure><p>Now that we have that clear, it’s time for the promised list of <strong>Five Most Beneficial Side Effects of the Treasure Hunt</strong> (a.k.a. Design Sprint).</p><h4>1. <strong>Introduce new methods</strong></h4><p>The <a href="http://thesprintbook.com/"><em>Sprint</em> book</a> gives multiple suggestions for methods you can use during each day. However, the structure of the sprint is quite open and lends itself perfectly to introducing new methods and figuring out what works best in your context. The possibilities are endless, so don’t be afraid to experiment. Of course, you should not forget to reflect on each one of them on regular basis. This will help you to figure out which methods are best suited for each situation and thus, reap the benefits from your actions in the best possible way.</p><p>For instance, sometimes the first thing we do during the Mickey day is to ask the team to imagine how they would advertise a possible solution for our problem in 5 years from now. We give them random magazine ads that have nothing to do with each other, or the topic, or the airline industry and ask them to let their imagination loose. This proved to be a perfect first activity for the Mickey day as it puts the team into a creative mood. Besides, the results are always fun and very much unexpected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/851/1*iTijn958MuKGNBYcfomt7Q.gif" /><figcaption>Put your team in a creative mood</figcaption></figure><h4>2. <strong>Create capabilities</strong></h4><p>This one is kind of obvious, but it’s still fascinating how well it works. The sprint serves as the perfect medium, in which the team gets to experience many different methods and techniques crammed into 5 days. This accelerates their learning and thus the forming of their capabilities to work with such methods and to facilitate sprints on their own.</p><p>It’s practically two birds with one stone — you get actionable outcomes in the end, but you also help the team to create the needed know-how to do it on their own. The trick, however, is not only to ensure they know the methods, but also to give them the right tools. But wait, isn’t that what everyone says? It is. Actually, the <em>Sprint</em> book can easily play this role. What worked for us was to make a Beginner’s Toolbox — all supplies and explanations the team would need put in one place. For instance, the toolbox for the exercise of combining ads we already mentioned was a starter pack of weird, random magazine ads. Once you do that, you can just sit back and watch the magic happen.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/753/1*GxMesVbRR9h70InAvf0Xzg.gif" /><figcaption>This app is just a nice way to show it, but something as simple as Word document and PowerPoint (Keynote) templates work just as fine</figcaption></figure><h4>3. <strong>Keep the long-term vision and goal in mind</strong></h4><p>The Design Sprint always starts with the long-term goal in mind and makes sure everything you do is connected to it. We decided to take this notion a step further by dedicating an entire day for evaluation — the Mirror day. Wait, isn’t that a waste of time? No and the reasoning is quite simple.</p><p>We noticed that when we don’t allow enough time to talk about what we’ve learned, the possible improvements of the process and the actionable next steps, we lose valuable insights and traction. Thus, the notion of “always strike while the iron is hot” is leading for us. Besides, doing so gives us both focus and flexibility.</p><p>On the one hand, when Friday is assigned for evaluation, you know you have to do everything before that — this brings sense of urgency to the team. On the other hand, if things fail, and this is bound to happen at some point, we always have the Friday to mend what’s broken (or do a final check). Last but not least, the Mirror day allows us to connect the sprint results to our long-term goal and vision. This way we make sure our future focus is not short-sighted in any way and we can get our priorities straight.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sJaF7aQbLWAwuRqTTMGecQ.gif" /><figcaption>Make sure everyone in the team is on the same page</figcaption></figure><h4>4. <strong>Create the mindset</strong></h4><p>Perhaps this is one of the major impacts the Treasure Hunt had on the X-Gates team and everyone who has participated in one. A few months in, we started noticing a shift in their mindset towards being more exploratory and participatory (a.k.a. involving users throughout the entire project and using design methods outside the Treasure Hunt).</p><p>We believe there are several factors you need to establish before you see such change. First, introducing different design methods and thus building capabilities helps tremendously. Another factor is having your entire team actively participating in the test throughout the Dummy day. Although having the team watch the interviews from the sprint room definitely has its pros, due to the nature of our context (did we mention things get really chaotic), we need all hands on deck.</p><p>Besides, having the team being actively involved in the test really fortifies their sense of achievement. This, of course, could be risky, but the facilitator is always there to make sure everything is running smoothly and help when help is needed. Last but not least, we believe that the Mirror day also helps a lot in doing so.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/771/1*Z8qfO7Bj8T4che4jUxX3bg.gif" /><figcaption>Just go out and talk to your users. It’s easier than you think.</figcaption></figure><h4>5. <strong>Use it as an ambassador</strong></h4><p>The <em>Sprint</em> book suggests four instances in which to use the sprint — when the stakes are high, when time pressure strikes, when stuck and in the beginning of a project. Next to these four, it turns out the Design Sprint is perfect for the role of an ambassador. Since it requires relatively little commitment and resources upfront and achieves plenty of results in just five days, this is one of the fastest ways to convince people and departments to adopt the new way of working and thinking.</p><p>For instance, a few months ago, we used the Treasure Hunt to convince another department within KLM in its added value. To do that, we carried out a sprint on preventing people from taking excessive amounts of hand baggage to the plane (in case you’re wondering, this is one of the main reasons why flights don’t leave on time in Europe).</p><p>The sprint resulted in dedication of the department to give the X-Gates freedom in the Departure hall, allocated new people to join the X-Gates team part-time and first steps in implementing the way of working in the department.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BaxAc59A9GRG2mNfY148fg.gif" /><figcaption>Every person who has ever participated in the Treasure Hunt and our way of working became an ambassador of the X-Gates</figcaption></figure><h4>6. <strong>Drop by drop</strong></h4><p>Yes, yes, we know, we said five. We’re just funny that way. This one is actually some parting words in case you’re wondering how two students managed to pull this off.</p><p>We did three things. First, we saw the X-Gates team, KLM staff and passengers as the experts of their own domain. This helped us to gain their trust and to take them along on our journey.</p><p>Second, we sometimes operated under the premise “it’s easier to ask for forgiveness than for permission” so we could show them the benefits of the Sprint firsthand without much delay.</p><p>Third, we were really (sometimes obnoxiously) persistent. The Treasure Hunt didn’t happen overnight. Actually, the first time we proposed to do something even remotely similar to it, the reaction we got was “we can give you 2 hours”. It took us several months to get to the coveted five days, which turned out to be a blessing in disguise. Why? A Bulgarian saying perfectly sums it up: “you form a lake drop by drop.”</p><p>It might sound corny, but we believe the Treasure Hunt became a fundamental element of the X-Gates’ way of working because we introduced elements of it little by little.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/869/1*kyXSgtjn6Y6JuKPmOsAQdQ.gif" /><figcaption>Rule #1: You have to learn to be patient</figcaption></figure><p>Here you go, five (or six) most beneficial side effects of the Treasure Hunt. Although, this is based and adapted for our almost year-long experience within KLM, we strongly believe these insights to be equally important and implementable in companies of different sizes. Besides, the Design Sprint helped us to introduce a new way of working and thinking to the X-Gates. One that takes into consideration the users, the context and shapes the capabilities of the team so they can come up with meaningful solutions. As such, we sincerely hope that these benefits will convince you to run one in your organization and make it a normal occurrence, just like the X-Gates did.</p><p>And finally, both of us are designers and thus facilitating the Treasure Hunt is very much in our comfort zone. But no one in the X-Gates team has a design background. Nevertheless, they are facilitating it on their own now. So being a designer is definitely not a requirement to start. All you have to do is to take small steps and simply talk to people about what you want to do. Although at times they might see you as the weirdo who’s here to perform some tricks with his magic wand, if you keep trying, people will start trusting you.</p><p>Don’t get discouraged if it doesn’t go well the first time. Remember, one important key to success is self-confidence. And self-confidence is built on preparation. Therefore, it’s imperative to take the time and prepare the ground for the seeds you’re going to plant.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/851/1*TRu7TuqlVph0VVWqWVBjdw.gif" /><figcaption>If you keep trying, people are bound to trust you. Remember rule #1?</figcaption></figure><p><strong><em>Acknowledgements</em></strong></p><p><em>Of course, we didn’t do this on our own. We had the help of Hassan Charaf and the entire X-Gates team, Sander Stomph and Christine de Lille.</em></p><p><a href="#_ftnref1"><em>[1]</em></a><em> The book we recommend to understand why it is so important, how to do it and how to analyze the collected data is: </em><a href="https://www.amazon.com/Convivial-Toolbox-Generative-Research-Design/dp/9063692846"><em>Convivial Toolbox</em></a><em> or </em><a href="http://contextmapping.com/about/"><em>this site</em></a><em> to get you started.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9ae265b8f0cf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sprint-stories/the-five-most-beneficial-side-effects-of-the-design-sprint-9ae265b8f0cf">KLM Airlines: 5 Lessons Learned Using Design Sprints to Prototype the Airport Experience</a> was originally published in <a href="https://medium.com/sprint-stories">Sprint Stories</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>