<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Charles Chi on Medium]]></title>
        <description><![CDATA[Stories by Charles Chi on Medium]]></description>
        <link>https://medium.com/@cchi?source=rss-9474a3ebe66f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*yTk-vcLIuC_3A8TGrvLWUg.png</url>
            <title>Stories by Charles Chi on Medium</title>
            <link>https://medium.com/@cchi?source=rss-9474a3ebe66f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 19:11:45 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@cchi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Frontier Is Shifting with AI]]></title>
            <link>https://medium.com/ai-assimilating-intelligence/the-frontier-is-moving-with-ai-c8aca647c9be?source=rss-9474a3ebe66f------2</link>
            <guid isPermaLink="false">https://medium.com/p/c8aca647c9be</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[economics]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[productivity]]></category>
            <dc:creator><![CDATA[Charles Chi]]></dc:creator>
            <pubDate>Sun, 13 Sep 2026 05:13:20 GMT</pubDate>
            <atom:updated>2026-09-13T08:08:47.284Z</atom:updated>
            <content:encoded><![CDATA[<h4>What a classic economics graph helps me understand about working with AI and Engineering</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GjIJVpch1jnxRCg1gXCMtQ.png" /><figcaption>An Abstract “Efficient Frontier”. The constrained inner curve opening into a larger field of possibilities.</figcaption></figure><p>I keep thinking about a graph from economics class back in the day.</p><p>It showed two things a country might want to produce, using a limited set of resources, and a curved boundary around what was possible. We could choose different points along the curve, but we could not move beyond it unless something fundamental changed.</p><p>Well, that something fundamental HAS changed. As a Machine Learning Engineer, I still have the same number of hours in a day. I still need to understand a problem, work through imperfect data, write and review code, run experiments, test assumptions, and figure out whether the result is actually useful or impactful. Those constraints did not disappear when AI tools became more capable.</p><p>But the amount and type of work I can realistically attempt has shifted. Tasks that used to take a meaningful portion of the day can sometimes be compressed into an hour. An unfamiliar codebase becomes easier to approach. Trying a second implementation becomes less expensive. Writing the first version of a test, query, or document no longer has to begin with a blank page.</p><p>AI did not remove the usual engineering trade-offs. It shifted the curve.</p><h3>The Production Possibilities Frontier</h3><p>The graph I keep thinking about from my previous economics class is called the <strong>Production Possibilities Frontier</strong>, or <strong>PPF</strong>. In the standard economics example, imagine an economy that can produce two goods, such as wheat and machinery. Its land, labor, capital, and technology are limited. If it directs more of those resources toward wheat, it will usually produce less machinery. If it wants more machinery, it has to give up some wheat production. Classic tradeoff with limited resources.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*3ozO0mR-aNpAOx1cw_X0_A.png" /><figcaption>A traditional production possibilities frontier with points inside, on, and outside the curve</figcaption></figure><p>A simplified Production Possibilities Frontier illustrated example. Points on the curve use the available resources efficiently, while points outside it are not currently attainable. The curve represents the best combinations of those two goods that the economy can currently produce. This is the <strong>“Efficient Frontier”</strong>.</p><p>While a point inside the curve is possible, it’s “inefficient”. Some resources are unused or poorly allocated. A point on the curve is “efficient”, utilizing max resources given constraints. Producing more of one good requires giving up some of the other (always about tradeoffs!). A point outside the curve is very attractive but currently out of reach, i.e. unattainable.</p><p>This is the standard interpretation of the PPF. <a href="https://openstax.org/books/principles-economics-2e/pages/2-2-the-production-possibilities-frontier-and-social-choices">OpenStax’s Principles of Economics</a> explains how the curve represents trade-offs under limited resources. Countries have different frontier profiles. (aside: trading allows two countries to focus more on what they’re good at, and both parties benefit… a topic for a different time). This curve also distinguishes productive points on the frontier from inefficient points inside it and unattainable points outside it.</p><p>That last category is the interesting one. If the economy develops better technology, trains its workers, or improves how production is organized, the boundary can move outward. Combinations that were impossible before can become possible. The economy still has constraints, but it can now produce more from them.</p><p>The PPF is a simplified model, of course. Software &amp; development work cannot be measured as neatly as tons of wheat or machinery/widgets. Still, it gives us a useful way to think about what is happening.</p><h3>Engineering Has a Frontier Too</h3><p>Engineering is full of trade-offs, as any good dev knows.</p><p>We can move quickly, but we may take on technical debt. We can investigate every edge case, but ship later. We can build something flexible enough for many situations, but add complexity for the people who only need the common case. We can spend another week improving model performance, but that week cannot also be spent improving monitoring or the product experience.</p><p>Translating software engineering into the PPF, for a simple version of the graph, imagine the x-axis represents the breadth of work we complete. This could include features, experiments, analyses, documentation, or different approaches we are able to try. The y-axis represents depth and quality: reliability, maintainability, careful evaluation, and how well we understand what we built.</p><p>Our time and attention are fixed. Moving farther on one axis often costs us something on the other.</p><p>Some engineering improvements help us move from inside the curve toward it. Better requirements reduce wasted work. Good tooling removes repetitive steps. Clear interfaces make systems easier to change. These practices help us use our existing capacity well.</p><p>Most of the time, that is what “productivity” work means. We become more efficient within the boundary we already have.</p><p>Occasionally, a new technology changes the boundary itself. Dramatically so like what we’re witnessing today.</p><h4>The Curve is Shifting</h4><p>Working with current AI tools, AI agents feels very different because they reduce the cost of attempting something.</p><p>In machine learning engineering, the work <em>around</em> a model is often larger than the model itself. There is code to inspect, data to validate, investigations to understand, pipelines to trace, experiments to compare, failures to debug, and results to explain. Each task carries a setup cost. Even a small idea may not be worth exploring if it requires hours just to reach the first useful signal.</p><p>AI lowers some of that setup cost. I can ask for an initial map of an unfamiliar part of a system. I can turn a rough thought into a small prototype and see where it breaks. I can compare two approaches before investing deeply in either one. I can draft tests, queries, or documentation while the context is still fresh. None of these outputs are automatically correct, but the first attempt arrives sooner.</p><p>That matters because engineering is partly a process of trying, observing, learning, and revising. When attempts are expensive, we make fewer of iterations. We may stay closer to familiar solutions. We may leave a useful question unexplored because answering it would take too long.</p><p>When the cost of an attempt falls, more paths become reasonable to explore.</p><p>This is where the frontier shifts. It is not only that the same ticket gets completed faster. The feasible set gets larger. But one still needs discernment, more important than ever. Within the same week, an engineer might be able to test an additional hypothesis, add the missing evaluation, improve the explanation, or explore an approach that previously would have been cut for time.</p><p>Work that sat just outside the old curve can move inside the new one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*l7cOxvXncBbIbppSV4T-NQ.png" /><figcaption>An engineering frontier shifting outward with AI</figcaption></figure><p>AI expands the feasible set, but it does not choose the right point for us.</p><h3>The Bottleneck Moves</h3><p>There is an obvious catch. Producing an answer <em>faster</em> does not make the answer <em>right</em>.</p><p>AI can generate plausible code that misses an edge case, if not deeply validated or carefully designed. It can explain a system confidently while misunderstanding an important detail. It can help create five possible implementations when the real need is to decide whether the feature should exist at all.</p><p>As execution becomes cheaper, the entire end-to-end cycle can get shorter. Within that smaller total, verification and testing could take a larger share. Some of the recovered time can go toward exploring ideas, bringing a prototype into production, and helping other people understand and adopt the result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7fXqR5-szfrjl86a4F0aIw.png" /><figcaption>A conceptual comparison of engineering attention before and with AI</figcaption></figure><p>AI can shorten the path to a first answer. Correctness, context, and judgment still belong to the engineer. We still need to be mindful of where we spend our time.</p><p>The engineer still has to understand the system well enough to recognize a bad assumption or sneaky oversight. We still need to evaluate model behavior, inspect failures, consider security and privacy, and decide what level of quality the situation requires. In many cases, reviewing generated work can cost more than writing a clean version from scratch.</p><p>There is also a temptation to fill the newly available space with more output. More code, more experiments, more documents, more features. But producing more things is not the same as producing more value. If AI helps us move outward on the graph, we still have to choose a useful point on the new curve.</p><p>This is the part I find most important. <strong>AI changes the economics of execution, but it does not make judgment abundant.</strong> It may even make judgment <em>more</em> valuable because there are now more plausible paths competing for attention.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tcTFv8RBIQzBUWpP" /><figcaption>Photo by <a href="https://unsplash.com/@imaginebuddy?utm_source=medium&amp;utm_medium=referral">Imagine Buddy</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>A Larger Set of Possibilities</h3><p>The<strong> Production Possibilities Frontier</strong> does not tell an economy what it should produce. It only shows what it <em>can</em> produce and the <em>trade-offs</em> between the available choices.</p><p>AI works similarly for ML/AI/Software engineering. It does not decide which problems matter, which compromises are acceptable, or when a result is trustworthy. It gives us a larger feasible region and leaves us with the responsibility of choosing within it.</p><p>That is why I find the old economics graph more useful than a simple promise of productivity. “Faster” and “more efficient” only describes doing the same work in less time. An outward shift describes something broader: attempting work that previously did not fit, combining speed and depth in new ways that technology gifts to us, and using the recovered time for better questions, thinking, and strategy.</p><p>The frontier is shifting. The interesting questions are how we use the space it creates and which new possibilities we choose to pursue.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c8aca647c9be" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai-assimilating-intelligence/the-frontier-is-moving-with-ai-c8aca647c9be">The Frontier Is Shifting with AI</a> was originally published in <a href="https://medium.com/ai-assimilating-intelligence">AI: Assimilating Intelligence</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Query, Key, Value in Transformers and LLMs]]></title>
            <link>https://medium.com/ai-assimilating-intelligence/understanding-query-key-value-in-transformers-c579b93054cc?source=rss-9474a3ebe66f------2</link>
            <guid isPermaLink="false">https://medium.com/p/c579b93054cc</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Charles Chi]]></dc:creator>
            <pubDate>Wed, 02 Oct 2024 07:10:07 GMT</pubDate>
            <atom:updated>2024-10-27T18:11:07.504Z</atom:updated>
            <content:encoded><![CDATA[<h4>[LUCID-LEARNERS] A DECODING AI SERIES</h4><h4>Building intuition through accessible, simplified examples to explore inner workings of the Attention Mechanism.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aCgbtUGUqcJ-gpOp9ji5Lw.png" /><figcaption>Students collaborating in a classroom as an analogy for Query-Key-Value in Transformers. Image generated by Author via ChatGPT.</figcaption></figure><p><a href="https://medium.com/ai-assimilating-intelligence/lucid-learners/home"><strong><em>[ Lucid Learners ]</em></strong></a><em> is a series where we provide intuitive explainers of AI concepts</em></p><h3>Introduction</h3><p>Transformers have revolutionized Natural Language Processing (NLP), and we’ve all heard about the standout feature: <strong>attention mechanism</strong>. A significant part of why the attention mechanism works so well is due to what we call the <strong>query-key-value </strong>components, which allows transformers to efficiently search through large amounts of information, extracting the most relevant parts (we’ll see an example in a minute!)</p><p>First, let’s break this down with an analogy. We’ll show how this mechanism helps transformers focus on what matters, like a student trying to understand their role in a class-wide project. Afterward, we’ll dive into a simplified example with code to further demonstrate how query, key, and value work in practice.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QGNAdGkUg2p9Rsl0" /><figcaption>Photo by <a href="https://unsplash.com/@taypaigey?utm_source=medium&amp;utm_medium=referral">Taylor Flowe</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>The Analogy: A Classroom Full of Students</h3><h4>The Setting…</h4><p>Imagine we’re in a classroom filled with students, each with their own textbooks and notes. You, one of the students, are curious about how well you understand your specific role in a class-wide project, and want to figure out how you fit into this larger effort. You represent the <strong>query</strong> in this scenario — you’re <strong>actively seeking context</strong> from the others to understand how you contribute.</p><p>You ask: <strong><em>“What do you know that can help me understand my role in this class-wide project?”</em></strong></p><p>Each of your fellow students begins flipping through their textbooks and notes (representing their <strong>keys</strong>) to see if they have relevant information that could help clarify your role. Some students have detailed notes, others may have brief summaries, and some may have nothing relevant at all! The students who find useful information eagerly raise their hands to provide their answers (their <strong>values</strong>).</p><p>Here’s how their responses might look:</p><ul><li><strong>Alex</strong> flips through their notes and says, <em>“I have detailed information on the project’s objectives, which can help you understand how your part fits into the bigger picture.”</em> This is highly relevant to your question, so you weigh this answer heavily.</li><li><strong>Betty</strong> checks her textbook and offers, <em>“I have a summary of the project goals, but it’s more general.”</em> While helpful, it might not be as directly applicable as Alex’s response.</li><li><strong>Clark</strong> looks through his notes but only has information about a different project, which isn’t relevant to your query, so his contribution isn’t useful in this context.</li></ul><p>You carefully weigh all the returned answers based on how <strong>relevant</strong> they are to your original question about your role in the project. <strong>Alex’s</strong> detailed explanation gets more weight because it directly addresses your query. <strong>Betty’s</strong> summary is still useful but carries less weight, while <strong>Clark’s</strong> response is largely ignored because it doesn’t help answer your question.</p><p>Let’s map this directly onto transformers. In a simplified example using <strong><em>“dog plays fetch”</em></strong><em>,</em> each token will play a similar role as the students:</p><ul><li><strong>Query (You):</strong> As the student asking for help, you represent a token trying to understand its role within the sequence. The query vector encodes what the token is asking — essentially, <em>“how do I relate to the other tokens around me?”</em> In the case of <strong>“plays,”</strong> it could be asking, <em>“Who is performing this action?”</em> or <em>“What action is being performed?”</em></li><li><strong>Keys (Students’ Textbooks and Notes):</strong> The textbooks and notes held by the other students represent the key vectors. Each token holds a key, which indicates how relevant its information is to the query. For example, the word <strong>“dog”</strong> has a key that signals to the model it is important in explaining who is performing the action in <strong>“plays.”</strong></li><li><strong>Values (Students’ Answers):</strong> The answers provided by the students are the value vectors. These values represent the actual information the transformer uses to update the query token’s understanding. Once <strong>“plays”</strong> identifies that <strong>“dog”</strong> is relevant, it retrieves the value for <strong>“dog”</strong> to understand that <strong>“dog”</strong> is the one performing the action.</li></ul><h4>“Self-Attention” = All Tokens Asking and Answering Simultaneously</h4><p>In transformers, every token in a sequence performs this process of asking questions (querying) and responding (providing keys and values) at the same time. This happens in parallel, meaning tokens are both seeking relevant context and offering their own context to others simultaneously (in a real classroom, this would likely be utter chaos!).</p><p>In the sentence <strong><em>“dog plays fetch”</em></strong><em>,</em> the token <strong>“plays”</strong> queries all the other tokens to gather context about what action is happening and who is involved. The token <strong>“dog”</strong> responds by providing information about the subject, while <strong>“fetch”</strong> provides context about the object of the action.</p><p>This <strong>self-attention</strong> process is at the core of what makes transformers so powerful. They allow every word (or token) to dynamically adjust its importance based on the surrounding context, leading to a more accurate and nuanced understanding as the model processes multiple layers of the network.</p><h3>A Closer Look at How the Query-Key-Value Components Work</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_hZJ5SV3ZDHf6_Pb" /><figcaption>Photo by <a href="https://unsplash.com/@zayyerrn?utm_source=medium&amp;utm_medium=referral">Ahmed Zayan</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Now that we’ve visualized the analogy of students in a classroom, let’s dive into how the query-key-value mechanism operates technically within a transformer. Each word (or token) in a sentence requires context from the other tokens to fully capture its meaning. The query-key-value mechanism enables the model to efficiently gather and focus on the most relevant information as it processes the input.</p><h4>Query, Key, and Value as Vectors</h4><p>As discussed in the classroom analogy, each token generates <strong>query</strong>, <strong>key</strong>, and <strong>value</strong> vectors. To reiterate, these vectors represent:</p><ul><li><strong>Query</strong>: What the token is seeking (what information it needs).</li><li><strong>Key</strong>: The information the token holds (how relevant it is to other tokens’ queries).</li><li><strong>Value</strong>: The actual content the token shares if its key is found relevant.</li></ul><p>These vectors are derived from the token’s <strong>embedding</strong> through learned transformations using weight matrices: <strong>Wq</strong> (for queries), <strong>Wk</strong> (for keys), and <strong>Wv</strong> (for values). During training, these matrices are adjusted via backpropagation to help the model learn which tokens should focus on others in various contexts.</p><h4>The Process of Searching, Matching, Updating</h4><p>Each token has its query, key, and value vectors, the transformer follows a <strong>search-match-update</strong> process to determine which tokens provide the most relevant context:</p><ol><li><strong>Search (Comparison):</strong> The query of each token searches across the sequence, comparing itself to the keys of all other tokens to measure relevance. This is done using the <strong>dot product</strong> between the query and key vectors, generating a <strong>similarity score</strong>.</li><li><strong>Match (Similarity Scores):</strong> These similarity scores indicate how closely related the tokens are, e.g. if <strong>TokenA’s</strong> query closely matches <strong>TokenB’s</strong> key, <strong>TokenB</strong> holds relevant information for <strong>TokenA</strong>.</li><li><strong>Update (Weighted Sum):</strong> The similarity scores are passed through a <strong>softmax function</strong> to normalize them into <strong>attention weights</strong> (probabilities between 0 and 1). The transformer then computes a <strong>weighted sum</strong> of the value vectors based on these attention weights. This updated context enriches the query token’s understanding, based on the most relevant tokens in the sequence.</li></ol><p>Let’s now further solidify understanding with a simplified code example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gfIHeh8k8MzRNgFx" /><figcaption>Photo by <a href="https://unsplash.com/@wasdrew?utm_source=medium&amp;utm_medium=referral">Andras Vas</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>Simple Vectors in Action: “dog plays fetch” example</h4><p>We’ll work with just three tokens: <strong>“dog,” “plays,” and “fetch,”</strong> each represented as 3-element vectors to show how this mechanism works.</p><p>To start, we create the vector representations (embeddings) for each token. These embeddings, as simplified vectors, represent how the tokens are encoded in the transformer model, capturing their “meaning” for further computations:</p><pre>import numpy as np<br><br>token_dog = np.array([1.0, -0.5, 0.2])    # Token: &quot;dog&quot;<br>token_plays = np.array([0.4, 0.1, 0.4])   # Token: &quot;plays&quot;<br>token_fetch = np.array([0.6, -0.8, 0.7])  # Token: &quot;fetch&quot;<br># note: real embeddings are way more dimensions, often 1000+</pre><p>Next, we create randomized weight matrices for the query, key, and value transformations. In a real transformer model, these weights are learned during training (and with way more layers). Here, they are randomized for illustrative purposes — the weight matrices are used to transform the token embeddings into their corresponding query, key, and value vectors.</p><pre>W_q = np.random.rand(3, 3)  # Query transformation<br>W_k = np.random.rand(3, 3)  # Key transformation<br>W_v = np.random.rand(3, 3)  # Value transformation<br><br># W_q, similar to W_k and W_v, just with different values<br># [[0.59687471 0.03090141 0.43619974]<br>#  [0.46871257 0.84796101 0.68756735]<br>#  [0.91708439 0.56656351 0.30848419]]</pre><p>Now, we compute the query, key, and value vectors for tokens. We’ll focus on <strong>“plays”</strong> as the querying token, seeking information from <strong>“dog”</strong> and <strong>“fetch”</strong>. The <em>query</em> vector tells us what <strong>“plays”</strong> is seeking to understand (e.g., who or what is playing). The <em>key</em> vectors for <strong>“dog”</strong> and <strong>“fetch”</strong> represent the information those tokens hold, and the <em>value</em> vectors represent the actual content those tokens will contribute if found relevant.</p><pre># &quot;plays&quot; is the querying token<br>query_plays = np.dot(token_plays, W_q)<br><br># compute key and value for &quot;dog&quot;<br>key_dog = np.dot(token_dog, W_k)<br>value_dog = np.dot(token_dog, W_v)<br><br># compute key and value for &quot;fetch&quot;<br>key_fetch = np.dot(token_fetch, W_k)<br>value_fetch = np.dot(token_fetch, W_v)<br><br># example values:<br># query_plays = [0.6524549  0.32378207 0.36663031]<br># key_dog     = [0.36174811 0.02760755 0.43975015]<br># value_dog   = [ 0.19861341  0.71442551 -0.07100919]<br># key_fetch   = [ 0.19983013 -0.10961549  0.1634138 ]<br># value_fetch = [0.21657409 0.4065721  0.12485315]</pre><p>To determine how relevant <strong>“dog”</strong> and <strong>“fetch”</strong> are to <strong>“plays,”</strong> we calculate the similarity scores by taking the <strong>dot product</strong> between the query from <strong>“plays”</strong> and the keys of the other tokens.</p><pre># calc sim score for &quot;plays&quot; query to &quot;dog&quot; key<br>similarity_plays_dog = np.dot(query_plays, key_dog)<br><br># calc sim score for &quot;plays&quot; query to &quot;fetch&quot; key<br>similarity_plays_fetch = np.dot(query_plays, key_fetch)<br><br>print(similarity_plays_dog)<br># 0.4061888862102592<br><br>print(similarity_plays_fetch)<br># 0.15480106867962512<br></pre><p>Next, we use the <strong>softmax function</strong> to convert these similarity scores into <strong>attention weights</strong>. These weights determine how much attention <strong>“plays”</strong> should pay to <strong>“dog”</strong> and <strong>“fetch.”</strong></p><pre>similarity_scores = np.array([similarity_plays_dog, similarity_plays_fetch])<br>attention_weights = np.exp(similarity_scores) / np.sum(np.exp(similarity_scores))<br><br>print(f&quot;Attention Weights for &#39;dog&#39; and &#39;fetch&#39; with respect to &#39;plays&#39;:\n{attention_weights}&quot;)<br># Attention Weights for &#39;dog&#39; and &#39;fetch&#39; with respect to &#39;plays&#39;:<br># [0.56251806 0.43748194]<br># Note: the weights will add up to 1.</pre><p>Using these attention weights, we compute the <strong>weighted sum</strong> of the <em>value</em> vectors from <strong>“dog”</strong> and <strong>“fetch”</strong> to update the context for <strong>“plays”. </strong>With new context, it is enriched by most relevant information from other tokens</p><pre># Compute the weighted sum of values for &quot;dog&quot; and &quot;fetch&quot;<br>weighted_value_dog = attention_weights[0] * value_dog<br># [ 0.11172363  0.40187725 -0.03994395]<br><br>weighted_value_fetch = attention_weights[1] * value_fetch<br># [0.09474725 0.17786795 0.054621  ]<br><br># Combine the weighted values to update &quot;plays&quot; context<br>updated_context_plays = weighted_value_dog + weighted_value_fetch<br>print(&quot;Updated Context for &#39;plays&#39;:&quot;, updated_context_plays)<br># Updated Context for &#39;plays&#39;: [0.20647088 0.5797452  0.01467704]</pre><p><em>Note: The outputs will differ if you’re running this on your own due to randomization. Feel free to play around with the different values and components!</em></p><p>The updated context vector for “plays” is a <strong>contextualized representation</strong>, enriched with relevant information from the other tokens (“dog” and “fetch”). This vector, along with the updated context vectors for all other tokens, is passed downstream to the next layer for further processing and refinement.</p><h4>Scaling Up: 3 tokens to Many More</h4><p>In our (overly) simplified example, we only worked with three tokens: <strong>“dog,” “plays,”</strong> and <strong>“fetch.”</strong> While this helped illustrate the core concepts, real-world transformer models handle much larger sequences — often hundreds or even thousands of tokens. Each token generates its own query, key, and value vectors, and the model calculates attention across all tokens simultaneously. This ability to efficiently scale and process large sequences is what makes transformers so powerful for NLP tasks.</p><h4>Layering On: Refining Context Over Time</h4><p>The <strong>search-match-update</strong> process doesn’t occur just once; it repeats across <em>multiple layers</em> in a transformer. Each layer recalculates the query, key, and value vectors based on updated token embeddings, refining the relationships between tokens with each pass. For example:</p><ul><li><strong>Layer 1</strong>: “Plays” might begin to understand who is performing the action (e.g., <strong>“dog”</strong>).</li><li><strong>Layer 2</strong>: It refines further, recognizing what the action involves (e.g., <strong>“fetch”</strong>).</li><li><strong>More layers</strong>: Capture more complex, long-range dependencies or subtleties in meaning across the sequence.</li></ul><p>At each layer, the model recalculates the weighted sum of values, gradually refining the context and relationships. By handling large attention matrices, transformers focus on the most relevant tokens at each step of the sequence.</p><h4>The Outcome: Context-Aware Embeddings</h4><p>We’ve made it! By the end of this multi-layered process, the transformer produces refined, <strong>context-aware embeddings</strong> for each token. These embeddings capture not only the individual meaning of the token but also its relationships to other tokens in the sequence. This results in a powerful, context-rich representation that enables transformers to excel at complex tasks.</p><h3>Conclusion</h3><p>Transformers have redefined what’s possible in NLP by using the query-key-value mechanism and the multi-layered self-attention process. From handling small, simple token relationships to scaling across massive sequences, transformers adapt dynamically, making them one of the most effective tools in modern NLP and AI.</p><p>For continued learning, you may want to check out similar explainers in our <a href="https://medium.com/ai-assimilating-intelligence/lucid-learners/home">Lucid Learners</a> series!</p><ul><li><a href="https://medium.com/ai-assimilating-intelligence/positional-encodings-in-llms-intuitively-explained-3b973044f664">Positional Encodings in LLMs Intuitively Explained</a></li><li><a href="https://medium.com/ai-assimilating-intelligence/cross-entropy-in-large-language-models-llms-4f1c842b5fca">Cross Entropy in Large Language Models (LLMs)</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c579b93054cc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai-assimilating-intelligence/understanding-query-key-value-in-transformers-c579b93054cc">Understanding Query, Key, Value in Transformers and LLMs</a> was originally published in <a href="https://medium.com/ai-assimilating-intelligence">AI: Assimilating Intelligence</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI and Empathy: Is It Real?]]></title>
            <link>https://medium.com/ai-assimilating-intelligence/ai-and-empathy-is-it-real-f74f0857e87b?source=rss-9474a3ebe66f------2</link>
            <guid isPermaLink="false">https://medium.com/p/f74f0857e87b</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[reflections]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[empathy]]></category>
            <category><![CDATA[journal]]></category>
            <dc:creator><![CDATA[Charles Chi]]></dc:creator>
            <pubDate>Sun, 21 Jul 2024 01:32:38 GMT</pubDate>
            <atom:updated>2024-07-23T19:44:20.695Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://medium.com/ai-assimilating-intelligence/cognitive-collider/home">[Cognitive Collider] Thought-Provoking Insights Series</a></h4><h4>Can AI Truly Understand and Connect with Human Emotions?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gb7o5aO4blllFlGNsv9gQg.png" /><figcaption>Is empathetic connection possible between AI and humans? Blending nature and technology. Image generated by Author via ChatGPT.</figcaption></figure><p><a href="https://medium.com/ai-assimilating-intelligence/cognitive-collider/home"><em>[Cognitive Collider]</em></a><em> is a series offering thought-provoking takes that inspire reflection and deeper understanding.</em></p><p><em>We’re trying a different take and creating a more reflective series that still touches upon AI, but aims to elevate our perspectives to look beyond just technology.</em></p><p>Lately, I’ve been pondering a question that seems to be growing more relevant as AI continues to advance: <em>Can AI truly exhibit empathy?</em></p><p>When it comes to the more complex and nuanced feelings that make up the human experience, empathy is such a fundamentally human trait, involving not just understanding someone else’s emotions but actually feeling them. So, how does that translate to artificial intelligence?</p><p>Right now, AI systems are pretty good at <strong><em>simulating</em></strong> empathy. Think about the chatbots and virtual assistants we interact with; they’re programmed to recognize emotional cues and respond in a way that <em>seems</em> caring or supportive. For example, if you tell an AI assistant that you’re feeling stressed, it might suggest some breathing exercises, offer comforting words, give supportive vibes — and it all might actually work! But here’s the thing: As someone who works on building these systems… and you peek behind the curtain on the actual mechanics, it really is all based on algorithms and data patterns — it may not be actually authentic and genuine, i.e. AI can be very good at <em>mimicking</em> the high quality datasets that foundational models are trained on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YztD3LNMs28z-EW_" /><figcaption>Photo by <a href="https://unsplash.com/@askkell?utm_source=medium&amp;utm_medium=referral">Andy Kelly</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>There’s a debate on this topic indeed. Some people believe that as AI technology continues to improve, it might develop a kind of <em>cognitive empathy</em> — after all, we humans, although complex, are also systems. Can the human mind be boiled down into algorithms and generalized? This also means AI could get better at recognizing and responding to our emotional states as we ramp up efficiency and compute power. We could be seen as a deterministic <em>biological</em> machine in some ways, following a set of programmed instructions and responding to stimuli in a predetermined way.</p><p>However, others argue that because AI lacks consciousness and subjective experience (and always will), any empathy it displays will <em>always</em> be artificial and just copycatting patterns. Sure, although it might assemble together old patterns in new and interesting ways, it’s still just dynamically repeating patterns discovered from the the training datasets. It’s like a really sophisticated memory and mimicry, but without the <em>true</em> emotional connection. And it’s about the incentive rewards that are assigned during the training process. Then, that begs the questions: a) if emotions can be modeled and codified and 2) defining what is that goalpost benchmark of acceptance to aim and ultimately surpass. In contrast, machines are the set of programmed instructions and humans are non-deterministic, unable to be truly modeled after.</p><p>Personally, I think the truth lies somewhere in between. While it’s unlikely that AI will ever be able to fully replicate the complex and nuanced emotions of the human experience, I do believe that it’s possible for them to recognize and respond to our emotions in a meaningful and useful way, which can improve our lives for the better.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EyYdpe8tSgSqwDUZ" /><figcaption>Photo by <a href="https://unsplash.com/@lhgerona?utm_source=medium&amp;utm_medium=referral">Lyman Hansel Gerona</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>I also think about the implications of trusting AI in roles that require empathy. On the positive side, empathetic AI could be a great tool in areas like mental health support or customer service. AI can be available 24/7, offer consistent support, never judge us, and never gets “tired”. There are ethical concerns too. Can we really trust a machine with our deepest feelings (and the companies behind the models)? And what happens if we start relying too much on AI for emotional support, potentially isolating ourselves from real human interactions?</p><p>Will there be some downward spiral in biases with the way the systems push towards “what works”? Just as social media algorithms create self-reinforcing feedback loops by continuously recommending content that propagates the users’ existing interactions feeding more of the same, AI systems might default to the solutions it knows works well and this cycle could potentially steer human thinking/interactions in a specific direction, limiting diverse perspectives and innovative problem-solving. Same perspectives will get recycled and thus get narrower and narrower.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9un-gnugzn27khga" /><figcaption>Photo by <a href="https://unsplash.com/@domingoalvarze?utm_source=medium&amp;utm_medium=referral">Domingo Alvarez E</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>While AI can convincingly simulate empathy (and getting better at it by the day), it might be <em>“good enough”</em> for the 80% of use-cases for most tasks. However, it remains an open question whether it can ever achieve true empathy and ever feel it in the way humans do, and leave us longing for something more. Perhaps there can be an “emotional module” or set of weights to code empathy that we introduce into the deep layers of the models (and might already be in the works?)</p><p>I believe it’s important to consider the role that empathy will play — we’re asking AI to help with our day-to-day problems! After all, it’s a fundamental part of what makes us human, and if we can imbue our machines with even a fraction of that understanding and compassion, I think it could lead to a better, more connected world for all of us… while also helping us understand mysteries of ourselves better.</p><p>This ongoing evolution of AI will keep pushing us to redefine and reconsider what empathy truly means, the role of AI in relationships — and how we balance the benefits of technology with the essential need for genuine human connection and trust, especially considering how AI integrates into society as it evolves. It’s a fascinating journey, and I’m eager to see where it leads us.</p><p><em>More on this later as tech unfolds and we reflect on this topic! For more, go ahead and visit our </em><a href="https://medium.com/ai-assimilating-intelligence/latest"><em>latest Assimilating Intelligence articles</em></a><em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fJes96hMWcWPfDIA" /><figcaption>Photo by <a href="https://unsplash.com/@drscythe?utm_source=medium&amp;utm_medium=referral">Dominik Scythe</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f74f0857e87b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai-assimilating-intelligence/ai-and-empathy-is-it-real-f74f0857e87b">AI and Empathy: Is It Real?</a> was originally published in <a href="https://medium.com/ai-assimilating-intelligence">AI: Assimilating Intelligence</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Classification Metrics through a Bookshelf Analogy]]></title>
            <link>https://medium.com/ai-assimilating-intelligence/understanding-classification-metrics-through-a-bookshelf-analogy-0f9bc15be212?source=rss-9474a3ebe66f------2</link>
            <guid isPermaLink="false">https://medium.com/p/0f9bc15be212</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[metrics]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[evaluation]]></category>
            <dc:creator><![CDATA[Charles Chi]]></dc:creator>
            <pubDate>Mon, 15 Jul 2024 22:48:06 GMT</pubDate>
            <atom:updated>2024-10-09T17:35:43.404Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://medium.com/ai-assimilating-intelligence/lucid-learners/home">[Lucid-Learners] A Decoding AI Series</a></h4><h4>A Beginner’s Guide to Precision, Recall, F1-Score, Accuracy, and ROC/AUC.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B3FZoCIqGKPm-5kHvyaBcg.png" /><figcaption>Let’s classify relevant and interesting books. Image generated by Author via ChatGPT</figcaption></figure><p><a href="https://medium.com/ai-assimilating-intelligence/lucid-learners/home"><strong><em>[ Lucid Learners ]</em></strong></a><em> is a series where we provide intuitive explainers of AI concepts</em></p><p>When it comes to evaluating the performance of machine learning models, particularly around classification tasks, several metrics are frequently used: <strong>precision, recall, F1-score, accuracy, and ROC/AUC</strong>. These metrics help us understand how well a model is performing. You probably already heard of these if you’re in DS or ML, and they are widely covered — this article is a quick refresher and can serve as a reference. If you’re learning about these concepts for the first time, this article will make them more intuitive and easier to remember through this analogy: Let’s imagine a scenario involving a wonderful task of selecting books for your many bookshelves from a large pile of books!</p><h3>The Bookshelf Analogy</h3><p>Imagine you are a book enthusiast who receives a large pile of books every month. You decide that you cannot possibly add every book to your collection, despite ample shelf space in your home, so it is prudent to figure out which books to place on your bookshelf. You want your bookshelf to contain only interesting and relevant books. Each month, you review the new pile of books and make decisions based on certain criteria — ultimately to choose if you want to keep or donate the books. This process can help us understand the key metrics in machine learning evaluation.</p><h4>Primer on Key Terms</h4><p>Before diving into specific metrics, let’s clarify some important terms (feel free to skip this section if you are already familiar with the building blocks).</p><ul><li><strong>True Positive (TP):</strong> Books that are interesting and you correctly put on the bookshelf. You’re delighted to find this book at a later date and happily read it with joy and excitement.</li><li><strong>False Positive (FP):</strong> Books that are not interesting but you mistakenly put on the bookshelf. You’ve “classified” it was relevant but turns out later when you go to read it, you shouldn’t have accepted it.</li><li><strong>True Negative (TN):</strong> Books that are not interesting and you correctly reject. Whew! Didn’t miss much here and nothing was wasted.</li><li><strong>False Negative (FN):</strong> Books that are interesting but you mistakenly reject. Oops, this was a missed opportunity — that book could’ve been something great.</li></ul><p>Understanding these terms is crucial as they form the basis of many evaluation metrics. Here’s a visual representation from datacamp:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*CajK9gVqCvbUTVm9-Tmoqw.png" /><figcaption>image credits: <a href="https://www.datacamp.com/tutorial/what-is-a-confusion-matrix-in-machine-learning">https://www.datacamp.com/tutorial/what-is-a-confusion-matrix-in-machine-learning</a></figcaption></figure><p>Another fun take, if we’re predicting unicorns vs horses:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r7RZdg-HPGwJprws7Nh3TA.jpeg" /><figcaption>image credits: <a href="https://x.com/allison_horst/status/1199732736006619137">https://x.com/allison_horst/status/1199732736006619137</a></figcaption></figure><p>Okay, back to the bookshelf analogy!</p><h4>1. Precision: The Quality of Your Selections</h4><p><strong>Definition:</strong> Precision is the ratio of correctly selected books (true positives) to the total number of books you selected (true positives + false positives).</p><pre>Precision = TP / (TP + FP)</pre><p>The denominator is the total for how much you classified as relevant, regardless of whether they are actually relevant or not. Example:</p><ul><li>You selected 10 books to place on your bookshelf.</li><li>Out of those 10 books, 8 are interesting (true positives) and 2 are not (false positives).</li><li>Precision = 8 / (8 + 2) = 0.8 or 80%.</li></ul><p>Thus, precision measures the quality of the books you choose to put on your bookshelf. If you select 10 books and 8 of them are, in fact, interesting and relevant, your precision is 80%. High precision means you are good at selecting only the books you truly enjoy.</p><h4>2. Recall: The Coverage of Your Selections</h4><p><strong>Definition:</strong> Recall (also known as Sensitivity or True Positive Rate), is the ratio of correctly selected books (true positives) to the total number of interesting books in the pile (true positives + false negatives).</p><pre>Recall = TP / (TP + FN)</pre><p>The denominator includes all the books that are actually relevant in the universe of consideration, assuming you had the time to read and verify all the books. In reality, this could be a harder metric to measure since it requires you to know which ones you actually missed.</p><ul><li>There are 20 interesting books in the pile.</li><li>You selected 8 of these interesting books.</li><li>Recall = 8 / (8 + 12) = 0.4 or 40%.</li></ul><p>Thus, Recall measures how many of the interesting books from the pile you managed to put on your bookshelf. If there were 20 interesting books in the pile and you selected 8 of them, your recall is 40%. High recall means you are good at picking up most of the interesting books.</p><p>With our example, this is what the confusion matrix looks like with the values filled in:</p><pre>                   |      Predicted      |<br>                   | Positive | Negative |<br>|------------------|----------|----------|<br>| Actual Positive  | TP = 8   | FN = 12  | 8/(8+12) = 0.4 recall<br>|        Negative  | FP = 2   | TN = 78  | <br>------------------------------------------<br>                      8/(8+2) <br>                      = 0.8<br>                     precision</pre><p>Where:</p><ul><li>Total books (N) = 100</li><li>Actually interesting books (positives) = 20</li><li>Actually uninteresting books (negatives) = 80</li><li>Precision = 0.8</li><li>Recall, or Sensitivity = 0.4</li><li>Incorrectly relevant (FP, aka Type I Error) = 2</li><li>Missed Opportunities (FN, aka Type II Error) = 12</li></ul><p>To recap, here is the matrix but now with added formulas, including some extra metrics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/754/1*xPt_YCOccCLBX6dRVUrqCw.png" /><figcaption>image credits: <a href="https://medium.com/@_SSP/confusion-matrix-f7ff01c5bbb6">https://medium.com/@_SSP/confusion-matrix-f7ff01c5bbb6</a></figcaption></figure><h4>Acknowledging the Tradeoff between Precision and Recall</h4><p>There’s often a tradeoff between precision and recall. Improving one typically results in a decline in the other. Why?? This tradeoff arises because of the nature of making predictions and the balance between being strict or lenient with your selections.</p><p><strong>Stricter Selection: High Precision, Low Recall.</strong> If you are very strict about which books you put on your bookshelf, you might end up selecting only those you are absolutely sure are interesting. This approach ensures that most of the books you choose are indeed interesting (high precision). However, this also means you might miss out on many interesting books because you are too selective (low recall). Example:</p><ul><li>You chose to add 5 books on your bookshelf, all of which are interesting.</li><li>Out of 20 interesting books in the pile, you were super selective and only picked out 5 you knew for sure were going to be good.</li><li>Precision is 100% (5 / 5)</li><li>Recall is only 25% (5 / 20)</li></ul><p><strong>Lenient Selection: High Recall, Low Precision. </strong>On the other hand, if you are lenient and willing to give more books a chance, you will likely pick up more interesting books from the pile (high recall). However, this approach also means you might end up with more books that are not interesting (low precision). You’re not being discerning enough in this case. Example:</p><ul><li>This time, you select 28 books from the pile.</li><li>18 books are relevant out of the potential 20 books from the pile.</li><li>But because you were more permissive, you also happened to add 10 books that are not actually interesting.</li><li>Recall now is 90% (18 / 20), you only missed two!</li><li>Precision drops to 64% (18 / 28), because of the inclusion of non-interesting books.</li></ul><p><strong>Balance.</strong> In practice, the optimal balance between precision and recall depends on the requirements of your specific task. You’ll need to understand the domain and which one is more important. Some scenarios might prioritize precision, while others might prioritize recall:</p><ul><li><strong>Medical Diagnosis:</strong> High recall is crucial to ensure no potential positive cases are missed, even if it means some false positives.</li><li><strong>Email Spam Filtering: </strong>High precision is important to ensure that legitimate emails are not incorrectly marked as spam, even if some spam emails get through.</li></ul><p>Understanding the tradeoff between precision and recall helps in making informed decisions about model thresholds and evaluation criteria, depending on the application and its consequences. With this, there is another metric that could help.</p><h4>3. F1-Score: Balancing Between Precision and Recall</h4><p><strong>Definition</strong>: The F1-score is the harmonic mean of precision and recall.</p><pre>F1-Score = 2 * (Precision * Recall) / (Precision + Recall)</pre><p>It provides a single metric that balances the trade-off between precision and recall. Example:</p><ul><li>Precision = 80%.</li><li>Recall = 40%.</li><li>F1-score = 2 * (0.8 * 0.4) / (0.8 + 0.4) = 0.533 or 53.3%.</li></ul><p>The F1-score helps you understand the balance between selecting interesting books and covering as many interesting books as possible into one statistic. If both precision and recall are high, you have a high F1-score. Often you use this if there is a class imbalance or if both precision &amp; recall are important for the task at hand.</p><p>In the example, the 53.3% F1 score is low because the recall has brought it down. So, if we can tolerate a lower precision but could increase recall asymmetrically higher, this could boost the F1-Score significantly (aka brings up the harmonic mean)</p><h4>4. Accuracy: The Overall Success Rate</h4><p><strong>Definition</strong>: Accuracy is the ratio of correctly selected books (true positives + true negatives) to the total number of books evaluated.</p><pre>Accuracy = (TP + TN) / (TP + TN + FP + FN)</pre><p>This metric factors in the total books you sifted through and classified as relevant or not relevant. Example:</p><ul><li>You have 100 books in the pile.</li><li>20 are interesting and relevant, and you correctly selected 8 of them.</li><li>You correctly did not select 70 uninteresting books.</li><li>Accuracy = (8 + 70) / 100 = 0.78 or 78%.</li></ul><p>Accuracy measures the overall success rate of your book selection process. It considers both the books you correctly chose to put on the bookshelf and those you correctly chose not to.</p><p><strong><em>Note</em></strong>: Accuracy can be misleading when dealing with class imbalance. For instance, if in reality we only had 5 relevant books out of 100, you could simply reject all books and still appear to have a high success rate. Say you correctly did not select 95 uninteresting books, i.e. if you reject all books, your accuracy would be: (0 + 95) / 100 = 0.95 or 95%. Despite a high accuracy, the we fail to identify any of the interesting books, which is critical for most applications.</p><h3>ROC/AUC: True Positive Rate vs. False Positive Rate</h3><p><strong>Concept</strong>: ROC (Receiver Operating Characteristic) and specifically, the AUC (Area Under the ROC Curve) is a metric used to evaluate the performance of a classifier determine how well the model can distinguish between different classes at different points of the classification.</p><p>The ROC curve plots the <strong>TPR, or true positive rate </strong>(recall) against the <strong>FPR, false positive rate</strong>, providing a graphical representation of a model’s performance across different thresholds.</p><p><strong>True Positive Rate (TPR)</strong>: we already went over above, also known as recall or sensitivity. Again, TPR is the ratio of correctly identified positive cases to the total actual positive cases (measures the model’s ability to correctly identify ALL relevant instances)</p><pre>TPR = TP / (TP + FN)<br><br># remember: FN, or False Negative, datapoints were &quot;supposed&quot; to be<br># predicted as positive, but we missed it (i.e., actually positive)<br># so, (TP + FN) gives us &quot;all possible positive cases&quot;</pre><p><strong>False Positive Rate (FPR)</strong>: FPR is the ratio of incorrectly identified positive cases to the total actual negative cases. It measures the model’s likelihood of falsely identifying negative instances as positive, i.e. actual negative cases that are incorrectly identified as positive. You can think of this as “the rate of false alarms”, how often you mistakenly put bad books on your bookshelf.</p><pre>FPR = FP / (FP + TN)<br><br># FP, or False Positive, is incorrectly identify as positive (i.e. actually negative)<br># TN, or True Negative, is correctly identified as negative.<br># so, (FP + TN) gives us &quot;all possible negative cases&quot;</pre><p>Continuing our book selector example:</p><ul><li>There are 100 books in the pile to sort through.</li><li>Of those, 20 are interesting (actual positive cases).</li><li>You correctly identified 8 interesting books (TP).</li><li>You missed 12 relevant books (FN).</li><li>You incorrectly identified 2 uninteresting books as interesting (FP)</li><li>You correctly rejected 78 uninteresting books (TN).</li></ul><p>Calculations:</p><ul><li><strong>TPR</strong> = 8 / (8 + 12) = 0.4 or 40%</li><li><strong>FPR</strong> = 2 / (2 + 78) = 0.025 or 2.5%</li></ul><h4>Enter the ROC!</h4><p>Understanding the ROC Curve: The ROC curve is a plot that helps visualize the performance of a classification model.</p><ul><li>X-axis: False Positive Rate (FPR)</li><li>Y-axis: True Positive Rate (TPR)</li></ul><p>In our bookshelf selector example, each point on the ROC curve represents a different threshold for deciding whether a book is interesting or not, ranging from the strictest to the most permissive criteria. Starting with your most strict mode, you will not allow many books through, minimizing false positives, but as you become more permissive, you might misclassify more uninteresting books as interesting, thereby increasing false positives.</p><p>While we saw what one single calculation assuming one decision threshold point, the ROC is formed by considering how strict vs lenient we are by playing out scenarios and seeing how our TPR and FPR changes, then plot them. By adjusting this threshold, we change the balance between TPR and FPR. For instance:</p><ul><li>If we are very strict and only put books on the shelf that we are almost certain are interesting, TPR might be low, but FPR will also be low.</li><li>If we are very lenient and put almost every book on the shelf, both TPR and FPR will be high.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*09oUQcRAguJLrGZw3aBjwg.png" /><figcaption>image credits: <a href="https://developers.google.com/machine-learning/crash-course/classification/roc-and-auc">https://developers.google.com/machine-learning/crash-course/classification/roc-and-auc</a></figcaption></figure><p>This tradeoff allows us to see which point in the curve we can tolerate between TPR and FPR. A model with good performance will have a ROC curve that rises steeply towards the top-left corner of the plot, indicating a high TPR (recall) and a low FPR. This means the model is effective at identifying interesting books while minimizing the number of uninteresting books it mistakenly selects.</p><h4>What about AUC (Area Under the Curve)?</h4><p>AUC measures the entire area underneath the ROC curve, representing the integral of the curve.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*OGFsuE7RflKv6dfnIWcUeQ.png" /><figcaption>image credits: <a href="https://developers.google.com/machine-learning/crash-course/classification/roc-and-auc">https://developers.google.com/machine-learning/crash-course/classification/roc-and-auc</a></figcaption></figure><p>AUC helps you understand how well your selection process can distinguish between interesting and uninteresting books. A high AUC means you are good at discerning the interesting books from the uninteresting ones.</p><p>AUC ranges from 0 to 1, where 1 means perfect distinction and 0.5 means no distinction (random guessing).</p><ul><li>AUC = 1: Perfect distinction between classes.</li><li>AUC = 0.5: No distinction, equivalent to random guessing.</li><li>AUC &lt; 0.5: Worse than random guessing, indicating potential issues with the model.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*NdwM5-sSicpfk_LvQ1FKeg.png" /><figcaption>image credits: <a href="https://vitalflux.com/roc-curve-auc-python-false-positive-true-positive-rate/">https://vitalflux.com/roc-curve-auc-python-false-positive-true-positive-rate/</a></figcaption></figure><p>Now, let’s say that you have a method (“Method 1” ) to determine interesting &amp; relevant books, but also enlist the help of a friend (“Method 2”, as another classifier or approach), so you two would have different methods and criteria to determine which books to keep on your bookshelf, therefore different performance and ROC Curves. If you could plot the TPR vs FPR over several thresholds, you might end up with a chart like this with two curves and you can compare how the TPR vs FPR behaves and determine which method is better!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/706/1*Og7RH5SgfNQx8fX2B7CaeA.png" /><figcaption>image credits: <a href="https://www.originlab.com/doc/Origin-Help/ROC-Curve">https://www.originlab.com/doc/Origin-Help/ROC-Curve</a></figcaption></figure><p>ROC/AUC is crucial because it provides a single metric that summarizes the performance of a model across all classification thresholds. This is especially useful in scenarios with class imbalance, where simply looking at accuracy might be misleading. In addition, AUC:</p><ul><li>Evaluates the trade-off between TPR and FPR, giving a comprehensive picture of model performance, to compare across classification models.</li><li>Helps in making informed decisions about model thresholds, which can be critical depending on the application’s requirements.</li><li>Ensures that your model performs well across various conditions, making it robust and reliable.</li></ul><p>By understanding and interpreting the ROC/AUC, you can make informed decisions about model thresholds, select the best model for your needs, and ensure that your model performs well across various conditions.</p><h3>Conclusion</h3><p>Using the bookshelf analogy simplifies the understanding of classification metrics like precision, recall, F1-score, accuracy, and ROC/AUC. These metrics each provide different insights:</p><ul><li>Precision measures the accuracy of your positive predictions.</li><li>Recall assesses your ability to identify all relevant instances.</li><li>F1-Score balances precision and recall into a single metric.</li><li>Accuracy reflects the overall success rate of your classification process but can be misleading with class imbalances.</li><li>ROC/AUC evaluates the trade-off between true positive rate and false positive rate, giving a comprehensive view of model performance.</li></ul><p>By understanding these metrics and their interrelationships, you can better evaluate and improve the performance of your machine learning models. This approach ensures you select the most appropriate model for your specific needs and conditions, leading to more effective and reliable results.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0f9bc15be212" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai-assimilating-intelligence/understanding-classification-metrics-through-a-bookshelf-analogy-0f9bc15be212">Understanding Classification Metrics through a Bookshelf Analogy</a> was originally published in <a href="https://medium.com/ai-assimilating-intelligence">AI: Assimilating Intelligence</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Positional Encodings in LLMs Intuitively Explained]]></title>
            <link>https://medium.com/ai-assimilating-intelligence/positional-encodings-in-llms-intuitively-explained-3b973044f664?source=rss-9474a3ebe66f------2</link>
            <guid isPermaLink="false">https://medium.com/p/3b973044f664</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Charles Chi]]></dc:creator>
            <pubDate>Fri, 07 Jun 2024 20:33:32 GMT</pubDate>
            <atom:updated>2024-06-08T06:51:17.300Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://medium.com/ai-assimilating-intelligence/lucid-learners/home">[Lucid-Learners] A Decoding AI Series</a></h4><h4>Understanding the mechanics and role of Positional Encodings in generative AI models</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j5KAvIWqIMIOHnd1WCvhsg.png" /><figcaption>sine waves merging into waves rock formations and landscape. Image generated by Author via ChatGPT</figcaption></figure><p><a href="https://medium.com/ai-assimilating-intelligence/lucid-learners/home"><strong><em>[ Lucid Learners ]</em></strong></a><em> is a series where we provide intuitive explainers of AI concepts</em></p><h3>Introduction</h3><h4>Positional Encodings in LLMs</h4><p>In large language models (LLMs), understanding the order of words is important for capturing accurate context in language processing. <strong>Positional encodings</strong>, or <strong>positional vectors</strong>, are key to this capability, as part of the Transformer architecture to work effectively. They help models keep track of the position of each word in a sequence, ensuring that the context and meaning are preserved.</p><p>If we have a sentence, the word “apple” near the front can have a fundamentally different meaning than if the same word “apple” appears later in the sentence. Positions allow models to understand ordering and context:</p><ul><li>“An apple fell from the tree.” (apple = the subject)</li><li>“Alice gave Bob an apple.” (apple = the object)</li></ul><p>We’ll dive into a tour of positional encoding component, and this article aims to provide a clear and intuitive explanation of positional vectors, demonstrating their importance and how they are implemented in LLMs.</p><h4>Why Positional Encodings Matter</h4><p>The order of words matters significantly for their meaning. Positional vectors <strong>represent the position</strong> of elements within a sequence. Unlike models that inherently process sequences in a step-by-step manner (and thus implicitly capture the order of the tokens), the Transformer architecture processes <strong>all tokens in a sequence at once, simultaneously</strong> (this allows it to gain a comprehensive view of all the tokens). The parallel processing is enabled by the self-attention mechanism, but it also means that the model has no inherent sense of the position of each token in the sequence — they are not aware of which words came earlier or later from the input! Positional encodings solve this problem by providing each token with a positional context.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xWTrvKcRff6jPYRG93Voow.png" /><figcaption>Transformer Model (Vaswani, et al. 2017) — here highlighted is the positional encoding component of the Transformer Architecture, which is the topic for this article.</figcaption></figure><h3>The Mechanics of Positional Encodings</h3><p>So, how do they actually work? Positional encodings are added to the input embeddings to inject information about the position of each token. This allows the Transformer to take the order of the tokens into account when processing the input sequence.</p><ol><li>First, each token in the input sequence is first converted into a dense vector using an embedding layer.</li><li>Positional encodings are generated using a deterministic function, involving sinusoidal functions. It’s very important to note that these encodings are of the <strong>same dimension as the input embeddings</strong>, ensuring that they can be directly added element-wise.</li><li>Then, positional encodings are added to the input embeddings element-wise. This means that for each token embedding, the corresponding positional encoding vector is added to it.</li><li>The combined embeddings, which now contain both the semantic information from the token embeddings and the positional information from the positional encodings, are fed downstream to the Transformer layers.</li></ol><h4>Why use sinusoidal (frequency) functions?</h4><p>Instead of using simple integers (1, 2, 3, ...) or scaled numbers (0 to 1), which lack smoothness and can’t capture relative positions well, sinusoidal functions are used to allow better comparability and generalization. These functions provide smooth, continuous encodings that generalize well to unseen positions. Different frequencies in these functions capture various positional granularities, enabling the model to understand fine-grained and broad positional patterns, making them an ideal choice for positional encodings.</p><p>To encode positional information, each component of the positional embedding changes based on both a) the word’s position and b) the element’s index within the word/token embedding’s dimension. This pattern is accomplished by using different frequencies.</p><p>Okay! Here are the functions that are used:</p><pre># For even-indexed dimensions<br># use the sine function because the formula 2i results in even numbers.<br>PE(pos, 2i) = sin(pos / (10000 ** (2 * i / d_model)))<br><br># For odd-indexed dimensions<br># use the cosine function because the formula 2i+1 results in odd numbers.<br>PE(pos, 2i+1) = cos(pos / (10000 ** (2 * i / d_model)))<br><br># Where:<br># pos = is the position of the token in the sequence.<br># i = is the dimension of the encoding (the element position in vector)<br># d_model = is the total dimension of the model</pre><p>For a model dimension (“d_model”) of 4, it means the encoding vector has 4 components, calculated as such:</p><ul><li>At index 0 (2*0): Apply a sine function for the position encoding.</li><li>At index 1 (2*0+1): Apply a cosine function for the position encoding.</li><li>At index 2 (2*1): Apply a sine function for the next dimension of the position encoding.</li><li>At index 3 (2*1+1): Apply a cosine function for the next dimension of the position encoding.</li></ul><p>Something that could be confusing that might warrant an explanation is that index 𝑖 refers to the “pair index” within the positional encoding vector, not the absolute index.</p><p>For the first pair (i=0):</p><ul><li>Index 0: 2 ∗ 0 = 0 (use sine)</li><li>Index 1: 2∗0 + 1 = 1 (use cosine)</li></ul><p>For the second pair (i=1):</p><ul><li>Index 2: 2 ∗ 1 = 2 (use sine)</li><li>Index 3: 2 ∗ 1 + 1 = 3 (cosine)</li></ul><p>This pairing continues for higher dimensions. For each dimension d of the positional encoding, the function alternates between sine and cosine.</p><p>Here’s an excellent resource and visualization from the article <a href="https://erdem.pl/2021/05/understanding-positional-encoding-in-transformers">Understanding Positional Encoding in Transformers</a> by Kemal Erdem. There’s an interactive tool you can play around with to see how <em>pos </em>and <em>i </em>affects the values for the positional encodings:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r0FLjwb8ntBKLkzM3GZp6w.png" /><figcaption>Source: Kemal Erdem, <a href="https://erdem.pl/2021/05/understanding-positional-encoding-in-transformers">https://erdem.pl/2021/05/understanding-positional-encoding-in-transformers</a></figcaption></figure><h4>What’s the 10000 all about?</h4><p>It’s simply a scaling factor, and helps in creating a wide range of frequencies. It was empirically chosen during the development of the Transformer model — it strikes a balance between having a large enough scaling factor to avoid rapid oscillations for small position values and small enough to ensure that the values do not become excessively large for higher positions.</p><h4>What’s up with the different treatment of even vs odd dimensions?</h4><p>The reason for using sine for even dimensions and cosine for odd dimensions in positional encodings is to create different patterns for each position. This helps the model tell the positions apart more easily. Sine and cosine functions are like two different shapes that fit together in a way that makes each position’s encoding unique. By alternating between these two functions, the model gets a mix of information that helps it understand where each word is in the sentence. This way, it can better understand the order and meaning of the words.</p><p>For a deeper understanding, here’s an excellent video that explains the position embeddings more in-depth.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FdichIcUZfOw%3Fstart%3D372%26feature%3Doembed%26start%3D372&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DdichIcUZfOw&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FdichIcUZfOw%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/80a24659df80150c11dc357f3316cbe7/href">https://medium.com/media/80a24659df80150c11dc357f3316cbe7/href</a></iframe><p>Now, with some of the concept explained and out of the way, let’s check out a simple example to solidify our understanding.</p><h4>A Simple Calculation and Walkthrough</h4><p>To illustrate how positional vectors work, let’s calculate positional encodings for a simple 3-word sentence “AI learns quickly” with a 5-dimensional model. This approach employs sinusoidal functions to generate positional encodings.</p><pre>import math<br><br>def calculate_positional_encoding(pos, dimension, d_model=5):<br>    if dimension % 2 == 0:<br>        return math.sin(pos / (10000 ** (2 * dimension / d_model)))<br>    else:<br>        return math.cos(pos / (10000 ** (2 * (dimension - 1) / d_model)))<br><br># Sentence: &quot;AI learns quickly&quot;<br>sentence = [&quot;AI&quot;, &quot;learns&quot;, &quot;quickly&quot;]<br>position_encodings = []<br>d_model = 5<br><br>for pos in range(1, len(sentence) + 1):<br>    encoding = [calculate_positional_encoding(pos, i) for i in range(1, d_model + 1)]<br>    position_encodings.append((sentence[pos - 1], encoding))<br><br>for word, encoding in position_encodings:<br>    print(f&#39;{word}: {encoding}&#39;)</pre><p>output:</p><pre>AI: [0.5403023058681398, 0.0006309573026154199, 0.9999998009464214, 3.981071705534864e-07, 0.9999999999999207]<br>learns: [-0.4161468365471424, 0.0012619143540422218, 0.9999992037857646, 7.962143411069097e-07, 0.999999999999683]<br>quickly: [-0.9899924966004454, 0.0018928709030918876, 0.9999982085182674, 1.1943215116602066e-06, 0.9999999999992868]</pre><p>These values show how each position in the sequence is uniquely encoded, ensuring that the model can distinguish between different positions and interpret the sequence correctly.</p><p>Let’s now expand the example sentence, and include an embedding vector and show how positional encodings are added to the embeddings. We’ll update our previous example and use numpy to calculate and keep the vectors for the ensuing addition operation.</p><p>Sentence: <strong>“AI models learn quickly and efficiently from data patterns.”</strong></p><pre>import numpy as np<br>import math<br><br>def generate_positional_encodings(sequence_length, d_model):<br>    positional_encodings = np.zeros((sequence_length, d_model))<br>    for pos in range(sequence_length):<br>        for i in range(d_model):<br>            if i % 2 == 0:<br>                positional_encodings[pos, i] = np.sin(pos / (10000 ** (2 * i / d_model)))<br>            else:<br>                positional_encodings[pos, i] = np.cos(pos / (10000 ** (2 * (i - 1) / d_model)))<br>    return positional_encodings<br><br># Example sentence: &quot;AI models learn quickly and efficiently from data patterns&quot;<br>sentence = [&quot;AI&quot;, &quot;models&quot;, &quot;learn&quot;, &quot;quickly&quot;, &quot;and&quot;, &quot;efficiently&quot;, &quot;from&quot;, &quot;data&quot;, &quot;patterns&quot;]<br>sequence_length = len(sentence)<br>d_model = 5<br><br># Generate positional encodings<br>positional_encodings = generate_positional_encodings(sequence_length, d_model)</pre><p>Let’s also “simulate” dense embeddings (in reality these will be actual embeddings with semantic meaning and linguistic features captured inside the values. But for this example, we’ll use random values).</p><pre>dense_embeddings = np.random.rand(sequence_length, d_model)</pre><p>Then, we can easily add positional encodings to dense embeddings:</p><pre>combined_embeddings = dense_embeddings + positional_encodings</pre><p>Get the outputs (with some formatting):</p><pre>print(f&quot;{&#39;Word&#39;:&lt;12} | {&#39;Dense Embedding&#39;:&lt;42} | {&#39;Positional Encoding&#39;:&lt;42} | {&#39;Combined Embedding&#39;:&lt;42}&quot;)<br>print(&quot;-&quot; * 130)<br>for i in range(sequence_length):<br>    dense_rounded = np.round(dense_embeddings[i], 4)<br>    pos_enc_rounded = np.round(positional_encodings[i], 4)<br>    comb_rounded = np.round(combined_embeddings[i], 4)<br>    print(f&quot;{sentence[i]:&lt;12} | {str(dense_rounded):&lt;42} | {str(pos_enc_rounded):&lt;42} | {str(comb_rounded):&lt;42}&quot;)</pre><p>Final output (note: due to the randomized init for dense embeddings, you’ll have different numbers)</p><pre>Word         | Dense Embedding                            | Positional Encoding                        | Combined Embedding<br>-----------------------------------------------------------------------------------------------------------------------------------<br>AI           | [0.9627 0.6713 0.7336 0.7419 0.0421]       | [0. 1. 0. 1. 0.]                           | [0.9627 1.6713 0.7336 1.7419 0.0421]<br>models       | [0.9168 0.9847 0.1892 0.4235 0.715 ]       | [0.8415 0.5403 0.0006 1.     0.    ]       | [1.7583 1.525  0.1899 1.4235 0.715 ]<br>learn        | [0.1363 0.1034 0.2683 0.5004 0.3414]       | [ 0.9093 -0.4161  0.0013  1.      0.    ]  | [ 1.0456 -0.3128  0.2695  1.5004  0.3414]<br>quickly      | [0.5439 0.6112 0.4675 0.831  0.9475]       | [ 0.1411 -0.99    0.0019  1.      0.    ]  | [ 0.6851 -0.3788  0.4693  1.831   0.9475]<br>and          | [0.7945 0.7641 0.0237 0.4837 0.9754]       | [-0.7568 -0.6536  0.0025  1.      0.    ]  | [0.0377 0.1104 0.0262 1.4837 0.9754]<br>efficiently  | [0.4657 0.1751 0.6779 0.105  0.9144]       | [-0.9589  0.2837  0.0032  1.      0.    ]  | [-0.4932  0.4588  0.6811  1.105   0.9144]<br>from         | [0.3328 0.6424 0.4164 0.5149 0.1143]       | [-0.2794  0.9602  0.0038  1.      0.    ]  | [0.0534 1.6025 0.4201 1.5149 0.1143]<br>data         | [0.2107 0.1013 0.3387 0.8952 0.9953]       | [0.657  0.7539 0.0044 1.     0.    ]       | [0.8677 0.8552 0.3431 1.8952 0.9953]<br>patterns     | [0.2825 0.0005 0.1314 0.4468 0.7118]       | [ 0.9894 -0.1455  0.005   1.      0.    ]  | [ 1.2719 -0.145   0.1364  1.4468  0.7118]</pre><p>Full code:</p><pre>import numpy as np<br>import math<br><br>def generate_positional_encodings(sequence_length, d_model):<br>    positional_encodings = np.zeros((sequence_length, d_model))<br>    for pos in range(sequence_length):<br>        for i in range(d_model):<br>            if i % 2 == 0:<br>                positional_encodings[pos, i] = np.sin(pos / (10000 ** (2 * i / d_model)))<br>            else:<br>                positional_encodings[pos, i] = np.cos(pos / (10000 ** (2 * (i - 1) / d_model)))<br>    return positional_encodings<br><br># Example sentence: &quot;AI models learn quickly and efficiently from data patterns&quot;<br>sentence = [&quot;AI&quot;, &quot;models&quot;, &quot;learn&quot;, &quot;quickly&quot;, &quot;and&quot;, &quot;efficiently&quot;, &quot;from&quot;, &quot;data&quot;, &quot;patterns&quot;]<br>sequence_length = len(sentence)<br>d_model = 5<br><br># Generate positional encodings<br>positional_encodings = generate_positional_encodings(sequence_length, d_model)<br><br># Simulate dense embeddings for each word<br>dense_embeddings = np.random.rand(sequence_length, d_model)<br><br># Add positional encodings to dense embeddings<br>combined_embeddings = dense_embeddings + positional_encodings<br><br>print(f&quot;{&#39;Word&#39;:&lt;12} | {&#39;Dense Embedding&#39;:&lt;42} | {&#39;Positional Encoding&#39;:&lt;42} | {&#39;Combined Embedding&#39;:&lt;42}&quot;)<br>print(&quot;-&quot; * 134)<br>for i in range(sequence_length):<br>    dense_rounded = np.round(dense_embeddings[i], 4)<br>    pos_enc_rounded = np.round(positional_encodings[i], 4)<br>    comb_rounded = np.round(combined_embeddings[i], 4)<br>    print(f&quot;{sentence[i]:&lt;12} | {str(dense_rounded):&lt;42} | {str(pos_enc_rounded):&lt;42} | {str(comb_rounded):&lt;42}&quot;)</pre><h3>Conclusion</h3><p>Positional vectors are essential for helping large language models (LLMs) understand the order of words in a sequence. By using functions to generate positional encodings, we create unique patterns that enable models to distinguish between different positions. This process, demonstrated through our example, ensures that each word and its position are represented, maintaining both meaning and context for language processing.</p><p>Positional encodings are a critical component for modern AI models that use the Transformer architecture, allowing them to handle sequential data effectively. Beyond just the mechanics, this addition enhances the sophistication of how AI models process and comprehend language, paving the way for their application in diverse and complex tasks.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3b973044f664" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai-assimilating-intelligence/positional-encodings-in-llms-intuitively-explained-3b973044f664">Positional Encodings in LLMs Intuitively Explained</a> was originally published in <a href="https://medium.com/ai-assimilating-intelligence">AI: Assimilating Intelligence</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Intuition on Log Probabilities in Language Models]]></title>
            <link>https://medium.com/ai-assimilating-intelligence/building-intuition-on-log-probabilities-in-language-models-8fd00f34c03c?source=rss-9474a3ebe66f------2</link>
            <guid isPermaLink="false">https://medium.com/p/8fd00f34c03c</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Charles Chi]]></dc:creator>
            <pubDate>Sun, 12 May 2024 05:29:29 GMT</pubDate>
            <atom:updated>2024-11-13T13:56:05.428Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://medium.com/ai-assimilating-intelligence/lucid-learners/home">[Lucid-Learners] A Decoding AI Series</a></h4><h4>Understanding Practical Usage and Harnessing Logprobs Effectively in Your AI Applications</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bt-FeqEB-1Ql8vH1QfpJKA.jpeg" /><figcaption>Studying Technical Concepts. Image generated by Author via ChatGPT</figcaption></figure><p><a href="https://medium.com/ai-assimilating-intelligence/lucid-learners/home"><strong><em>[ Lucid Learners ]</em></strong></a><em> is a series where we provide intuitive explainers of AI concepts</em></p><h3>Introduction</h3><h4>Log Probabilities in Language Models</h4><p>Log probabilities (“<em>logprobs</em>”) are a fundamental aspect of how language models like GPT (Generative Pre-trained Transformer) compute and handle the vast amount of token probabilities associated with natural language processing. Let’s explore why logprobs are important, how they function, and their practical implications through some examples.</p><h4><strong>Why Logprobs?</strong></h4><p>Working with direct probabilities (numbers between 0 and 1 that we all know and love), while intuitive for human mind, has some shortfalls. It can be tricky, especially when dealing with long token/word sequences. These probabilities can become incredibly small, leading to a couple main issues. Therefore, switching to log space has some benefits, including:</p><ul><li><strong>Simplification of Calculations</strong>: Multiplying many small numbers can be computationally intensive and prone to errors. Because logarithms convert these multiplications into additions, they streamline the process significantly. For example, the log of a product of probabilities, <strong><em>log⁡(𝑎×𝑏)</em></strong>, simplifies to <strong><em>log⁡(𝑎)+log⁡(𝑏)</em></strong>. This transformation is particularly advantageous when computing the probabilities of long sequences of tokens, since it simplifies handling the cumulative probabilities of strings of tokens, making the calculations both manageable and more efficient.</li><li><strong>Numerical Stability</strong>: Small probabilities are susceptible to <em>underflow</em>, a condition where values close to zero are rounded to zero, potentially leading to incorrect computations. By using logprobs, which transform these small numbers into a more manageable scale, helps avoid such scenarios. In NLP tasks, where we might be dealing with very small probabilities, utilizing logprobs helps retain tokens that contribute to the context/meaning of generated texts, even when their raw probabilities are low.</li></ul><p>These properties are helpful for tasks like scoring and ranking different model outputs, based on their likelihood.</p><h3>Let’s Dive In!</h3><h4>Building Intuition for Interpreting Log Probabilities</h4><p>Log probabilities (logprobs) of output tokens reveal how likely each token is to appear at a specific position in a sequence, given the preceding context. Think of a logprob as the <em>logarithm of the probability of a token</em>, expressed as <strong>log(p)</strong>, where p is the token’s probability of occurring after the previous tokens, such as in generative NLP tasks.</p><p>Here are some keys to remember for logprobs:</p><ul><li><strong>Range of Values</strong>: Logprobs can range from any negative number (negative infinity) up to 0.0, which indicates a certainty of 100%, or absolute confidence that the token will appear next.</li><li><strong>Higher Log Probabilities</strong>: indicates that the model considers the token very fitting in that context (a “less negative” number, closer to 0). This is useful to understand how confident the model is about its outputs, or to explore alternative responses that the model deemed feasible.</li><li><strong>Lower Log Probabilities</strong>: the smaller the probability, the more negative the log probability becomes. As the probability value approaches zero, the logprob heads towards negative infinity. There is no limit to how negative the logprob can get, as it’s unbounded on the lower end.</li><li><strong>Evaluating Token Choices</strong>: We can analyze the logprobs for alternative candidate tokens, and discern which tokens the model found likely or unlikely — this insight can be particularly useful for understanding and improving model behavior during development and testing (inspect it to understand what and how model is producing the responses).</li></ul><h4>Computing Joint Probabilities</h4><p>The joint probability of a sequence of tokens (i.e. a sentence) is the probability of observing that entire sequence. Using a simplified example, the probability of the sequence “<em>The dog ran</em>” is the probability that the first word is “<em>The</em>”, the second word is “<em>dog</em>”, and the third word is “<em>ran</em>”, in that exact order.</p><p>We can compute the joint probability of a sequence using log probabilities, as you might’ve guessed. Instead of multiplying raw probabilities (which might be impractically small), you can now sum the log probabilities of each individual token in the sequence (an advantage of working in log space).</p><pre>logprob(&quot;The dog ran&quot;) = log(P(&quot;The&quot;)) + log(P(&quot;dog&quot; ∣ &quot;The&quot;)) + log(P(&quot;ran&quot; ∣ &quot;The dog&quot;))</pre><p>Calculating joint probabilities for the sequence:</p><pre># Token probabilities<br>token1_prob = 0.8<br>token2_prob = 0.2<br>token3_prob = 0.5<br><br># Multiply together for the joint probabilities for the sequence<br>joint_prob = token1_prob * token2_prob * token3_prob<br>joint_prob = 0.8 * 0.2 * 0.5<br>joint_prob = 0.08</pre><p>Calculating logprobs for the same sequence:</p><pre># Token probabilities<br>token1_logprob = log(0.8) ≈ -0.223<br>token2_logprob = log(0.2) ≈ -1.609<br>token3_logprob = log(0.5) ≈ -0.693<br><br># Adding together for the joint probabilities for the sequence<br>joint_logprob = token1_logprob + token2_logprob + token3_logprob<br>joint_logprob = −0.223 + −1.609 + −0.693<br>joint_logprob = -2.525</pre><p>If we convert this joint_logprob back to joint_prob, it will be 0.08. See the following section: <strong>“Converting Log Probabilities to Linear Scale”</strong></p><h4><strong>Averaging Per-Token Logprobs</strong></h4><p>To normalize and compare sentences of different lengths, averaging the per-token logprobs can be effective and more fair. We would sum the log probabilities of each token as above, but then divide by the number of tokens in the sentence (we’re basically adjusting by token length). Through this “normalization”, it can be more useful to compare, since the outputs can be used to rank or score sentences.</p><p>For instance, we have two sequences of different lengths:</p><pre>Sentence A: &quot;The dog ran&quot; with log probabilities: [−0.2,−1.0,−0.5]<br>Sentence B: &quot;The cat&quot; with log probabilities: [−0.2,−0.3]</pre><p>To compare these using average per-token logprob:</p><pre>Average for Sentence A: (−0.2 −1.0 −0.5) / 3 = −0.5667<br>Average for Sentence B: (−0.2 −0.3) / 2 = −0.2</pre><p>Despite Sentence A having a lower joint logprob, Sentence B shows a higher likelihood per word, providing a different perspective.</p><h4>Converting Log Probabilities to Linear Scale</h4><p>Sometimes, it’s beneficial to convert logprobs back to a linear scale to make them easier to interpret, especially when presenting data to those unfamiliar with logarithmic representations. To do this, we use the exponential function (with <em>e</em>) to revert it back to its original probability format, which can then be expressed as a percentage.</p><pre>linear_probability = e ** {logprob}</pre><p>Converting logprobs to a linear scale allows easier comprehension and communication of probabilities to stakeholders, as percentages are a more intuitive measure for most people. These linear probabilities facilitates comparisons as well, particularly in visualizations or reports where stakeholders may review model outputs.</p><p>Suppose we have a log probability of <strong>−0.693</strong>. To find the corresponding probability, we calculate:</p><pre>probability = e ** (-0.693) ≈ 0.5</pre><p>Doing this in Python:</p><pre>import numpy as np<br><br># Example logprob value<br>logprob = -0.693<br><br># Convert logprob to linear scale and express as percentage<br>linear_prob = np.round(np.exp(logprob) * 100, 2)<br><br>print(f&quot;Linear Probability: {linear_prob}%&quot;)<br># output: Linear Probability: 50.01%</pre><p>This tells us that the token has a 50% chance of occurring, a much more intuitive figure than the log probability value of -0.693. Thus, the transformation allows us to interpret the model’s confidence in more familiar terms, helping in scenarios where clear communication of probability is essential.</p><h4>Practical Use Cases for Log Probabilities in Language Models</h4><p>By examining log probabilities, developers can gauge the model’s certainty in its predictions, to enhance confidence in model responses from LLMs. This is particularly useful in several scenarios:</p><ul><li><strong>Response Selection</strong>: In systems where multiple potential responses can be generated, log probabilities help in selecting the most likely response. For example, in a chatbot, comparing the average per-token logprob across different response options allows the system to choose responses that the model deems more plausible.</li><li><strong>Debugging and Model Training</strong>: Developers can use logprobs to understand how certain or uncertain a model is about its predictions. Observing unexpectedly high or low logprob values for certain responses can signal issues that might require further investigation or adjustment in the training data or model parameters.</li><li><strong>Dynamic Interaction Adjustment</strong>: In interactive applications like digital assistants, log probabilities can help the system dynamically adjust its interactions based on the confidence level of its responses. If a model is less confident, it might request more information or offer multiple suggestions instead of a single definitive answer.</li><li><strong>Classification Tasks and Human-in-the-Loop</strong>: In classification tasks where accuracy is crucial, log probabilities can be used to determine the level of confidence in the model’s classifications. For responses where the model shows high confidence (high logprob values), the system can automatically process decisions without human oversight; while classifications with lower confidence (low logprob values), the system can flag these items for human review and approval. This approach ensures a balance between efficiency and accuracy, leveraging human expertise where the model’s certainty is not sufficient.</li></ul><h3>Conclusion</h3><p>Log probabilities are essential tools for enhancing the reliability and functionality of large language models (LLMs), found in the core mechanics of foundational model training, as well as interpretability use-cases. Through inspection of these “certainty” measurements, we can facilitate better decision-making in response generation and development of our AI systems.</p><p>Thus, understanding and utilizing logprobs not only improves the computational efficacy of language models, but also know they can be leveraged in evaluating and comparing generated text outputs. It simplifies the handling of probabilities, allowing models to efficiently manage long sequences of tokens, and helps explain generated model outputs.</p><p>Equipped with this knowledge on this quick <a href="https://medium.com/ai-assimilating-intelligence/lucid-learners/home">Lucid Learner</a> walkthrough, we encourage you to apply these insights to advance the development of LLM applications in your work!</p><p>For related reading, check out our article: <a href="https://medium.com/ai-assimilating-intelligence/cross-entropy-in-large-language-models-llms-4f1c842b5fca"><strong>Cross Entropy in Large Language Models (LLMs)</strong></a> to explore how these probabilities are calculated and further utilized.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8fd00f34c03c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai-assimilating-intelligence/building-intuition-on-log-probabilities-in-language-models-8fd00f34c03c">Building Intuition on Log Probabilities in Language Models</a> was originally published in <a href="https://medium.com/ai-assimilating-intelligence">AI: Assimilating Intelligence</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Overfitting and Underfitting during Model Training]]></title>
            <link>https://medium.com/ai-assimilating-intelligence/overfitting-and-underfitting-in-model-training-e0b14a89bd49?source=rss-9474a3ebe66f------2</link>
            <guid isPermaLink="false">https://medium.com/p/e0b14a89bd49</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[beginner]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[mls]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Charles Chi]]></dc:creator>
            <pubDate>Sun, 24 Mar 2024 04:37:49 GMT</pubDate>
            <atom:updated>2024-03-25T04:06:58.172Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://medium.com/ai-assimilating-intelligence/lucid-learners/home">[Lucid-Learners] A Decoding AI Series</a></h4><h4>Gain a quick understanding and ensure your machine learning models learn just right.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mVJy44hEA62EC17CICn6xg.jpeg" /><figcaption>ML Model Training is like Studying. Underfitting vs Overfitting differences.</figcaption></figure><p>This quick article is part of our <strong><em>[ Lucid Learners ],</em></strong><em> a series where we provide Bite-Sized AI &amp; ML Insights.</em></p><p>Today’s topic is one of the core concepts during model training, made accessible for beginners just starting out in ML/AI.</p><h3>Introduction</h3><p>In machine learning, training a model is like preparing for a test to learn the subject at hand. But what happens when your study habits are a bit off? You might find yourself either cramming to memorize the test questions and answers without grasping the underlying concepts; vs. not studying enough to understand the core basic principles and underlying concepts.</p><p>These pitfalls are known as overfitting and underfitting. Knowing these concepts and how to detect &amp; mitigate them is crucial for developing models that perform well on the data they were trained, as well as generalizing to the new, unseen data during inference.</p><h4>Overfitting: Too Much Memorization, Not Enough Learning</h4><p>Overfitting occurs when a model learns the details and noise in the training data to the point where it negatively impacts its performance on new data. It’s like the student is cramming hard, memorizing the answers to specific questions so well that you get thrown off by slightly different questions on the actual test. The model’s intricate understanding of the training data prevents it from generalizing to new data, leading to poor performance.</p><h4>Underfitting: Not Learning Enough</h4><p>Conversely, underfitting is when a model cannot capture the underlying trend of the data. It’s similar to a student who hasn’t studied enough (e.g. napping through or skipping study sessions, needs more study time, or distracted on the wrong topics) and therefore can’t even grasp the basic questions of a test. An underfitted model is overly simplistic — it doesn’t learn enough from the training data to make any meaningful predictions.</p><h3>Detection of Overfitting &amp; Underfitting</h3><h4>Quick word on Training vs. Test Data</h4><p>The splitting of data plays a pivotal role in helping to detect these issues. Essentially, we divide up our available dataset into two main parts: the training data and the test data.</p><ul><li>Training data is what we use to teach our model, to let it learn and understand the patterns, trends, and relationships within our data. Think of it as the textbook from which a student studies.</li><li>Test data is like the final exam, used to evaluate how well our model has learned and can apply its knowledge to new, unseen data.</li></ul><p>This separation is important because it gives us a fair assessment of the model’s ability to generalize, which is the ultimate test of its efficacy. By maintaining a clear boundary between these two sets (ensure no leakage though! like shared questions for the practice questions vs. actual test questions), we ensure that our model is tested and validated on a clean, unbiased dataset that it hasn’t encountered before, offering us insight into how it might perform in the real world.</p><h4>How to Spot?</h4><p>Detecting overfitting and underfitting involves keeping an eye on <em>performance metrics</em> for both the training set and a separate validation set.</p><ul><li>If a model that performs exceptionally well on its training data but poorly on the validation set is likely overfitting.</li><li>If a model shows poor performance on both sets, it’s probably underfitting.</li></ul><p>Visual tools like <em>learning curves</em>, which plot the model’s performance over time/samples/iterations, can also be helpful. Diverging performance curves suggest overfitting, while curves that stay close together but with high error indicate underfitting.</p><p>Here are some excellent visuals:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*zyfDrOE-qf382R_8_Cj8Pg.png" /><figcaption>Image Credits: <a href="https://www.linkedin.com/pulse/overfitting-underfitting-machine-learning-ml-concepts-com/">https://www.linkedin.com/pulse/overfitting-underfitting-machine-learning-ml-concepts-com/</a>. “Error” = predictions vs actual difference. Therefore, “training error” =&gt; training data and “test error” =&gt; test data</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C4S2klKiYL4eDE3RUIBk0g.jpeg" /><figcaption>Image Credits: <a href="https://towardsdatascience.com/overfitting-and-underfitting-principles-ea8964d9c45c">https://towardsdatascience.com/overfitting-and-underfitting-principles-ea8964d9c45c</a>. For more in-depth learning! This is another way to view Error vs Complexity.</figcaption></figure><h3>Mitigation Strategies</h3><p>When you actually start training an ML model, these can serve as guidelines to revisit. First gain a rough understanding and intuition <em>why </em>each of these techniques might help combat overfitting/underfitting.</p><h4>For Overfitting</h4><ul><li><strong>Simplify the Model</strong>: Starting with a simpler model can prevent overfitting by limiting the capacity to memorize the training data. Gradually increase complexity as necessary, but always question if each added element of complexity is justifiable.</li><li><strong>Reducing the Number of Features</strong>: Related to above. Involves identifying and removing features, which can help in simplifying the model. Fewer features mean less complexity, making the model less prone to overfitting. Perform feature selection or simply analyze the impact of each feature on the model’s performance can guide which features to include/discard.</li><li><strong>Regularization Techniques</strong>: Implement L1 and L2 regularization to add a penalty on the magnitude of model coefficients, discouraging overly complex models that are likely to overfit.</li><li><strong>Pruning (for decision trees)</strong>: Pruning reduces the size of decision trees by cutting off branches that have little importance, thus simplifying the model and reducing the risk of overfitting.</li><li><strong>Dropout (for neural networks)</strong>: Dropout randomly disables a fraction of neurons during training, which helps prevent the network from becoming overly dependent on any specific set of features, i.e. helps generalize.</li></ul><h4>For Underfitting</h4><ul><li><strong>Increase Model Complexity</strong>: Sometimes, a more complex model is necessary to capture the nuances of your data. Adding more layers to a neural network or using a more sophisticated algorithm can provide the model with the capacity to learn more complex patterns.</li><li><strong>Feature Engineering</strong>: Improving model performance can often be achieved by creating new features from your existing data. The right features are not always the raw data but can be transformations of that data, which better highlight the relationships between variables. You can add more features as well.</li><li><strong>Reduce Regularization</strong>: If you’ve implemented regularization, reducing its strength might allow your model to learn more from the training data, thus overcoming underfitting.</li><li><strong>Increase Training Time</strong>: For models that haven’t fully converged, simply allowing more time for training can help. This means running more epochs or iterations, giving the model additional opportunities to learn from the data. Related: tweaking learning rate may help as well.</li><li><strong>Adding More Data</strong>: If feasible, incorporating more training data can help the model to identify and learn the underlying structure of the dataset. More data provides a broader representation of the problem space, potentially improving the model’s accuracy and generalization.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VKbiS8P_8WN7UboBzTWtYg.jpeg" /><figcaption>Image Credits: <a href="https://towardsdatascience.com/overfitting-and-underfitting-principles-ea8964d9c45c">https://towardsdatascience.com/overfitting-and-underfitting-principles-ea8964d9c45c</a>. A table with ways to combat underfitting and overfitting.</figcaption></figure><h3>Conclusion</h3><p><strong>Why It’s Important</strong>: Understanding and managing overfitting and underfitting is crucial for developing machine learning models that are accurate and reliable on new, unseen data. It’s a balancing act between complexity and simplicity, learning, and generalizing.</p><p><strong>Finding the Balance</strong>: The key to successful machine learning models lies in finding the balance between overfitting and underfitting. This involves careful experimentation, validation, and tuning of your model inputs/parameters. Model training is an iterative, and potentially a never-ending process. Continuous learning, adaptation, and experimentation are part of the journey to creating models that not only perform well on known data but can also generalize to new, unseen data effectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*ZzI_NPj9qZ9-aoHBV2YIxg.jpeg" /><figcaption>Image Credits: <a href="https://aiml.com/what-is-underfitting/">https://aiml.com/what-is-underfitting/</a>. Here’s a summary of different types of ML problems with some useful visuals.</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e0b14a89bd49" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai-assimilating-intelligence/overfitting-and-underfitting-in-model-training-e0b14a89bd49">Overfitting and Underfitting during Model Training</a> was originally published in <a href="https://medium.com/ai-assimilating-intelligence">AI: Assimilating Intelligence</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Exploration vs. Exploitation Tradeoff: Navigating Life’s Choices]]></title>
            <link>https://medium.com/ai-assimilating-intelligence/the-exploration-vs-exploitation-tradeoff-navigating-lifes-choices-52925e540c63?source=rss-9474a3ebe66f------2</link>
            <guid isPermaLink="false">https://medium.com/p/52925e540c63</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[personal-growth]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[self-improvement]]></category>
            <dc:creator><![CDATA[Charles Chi]]></dc:creator>
            <pubDate>Mon, 18 Mar 2024 03:11:08 GMT</pubDate>
            <atom:updated>2024-03-19T04:19:38.807Z</atom:updated>
            <content:encoded><![CDATA[<h4>Balancing between familiarity and adventure, applying this concept in real life for achievement, fulfillment, and growth.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TimEWm24XpCbTV6xFKywfw.png" /></figure><h3>Exploration vs. Exploitation</h3><p>There’s a principle in the data science and machine learning that also has profound relevance to our everyday lives:<strong> the exploration vs. exploitation tradeoff</strong>. Essentially, this concept describes the tension between venturing into unknown territories (exploration) and optimizing outcomes within familiar grounds (exploitation).</p><p>Imagine <strong><em>exploration</em></strong> as setting sail to discover new continents — inherently higher risk, such as embarking on ML novel research projects never done before at your company. Meanwhile, <strong><em>exploitation</em></strong> is cultivating the lands you already possess—going with what you know works, such as enhancing a proven predictive model you already have in production.</p><h4>In Machine Learning</h4><p>Traditionally, exploration vs exploitation tradeoff in ML contexts are mentioned in reinforcement learning (RL) and the model training process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A3lcdoeusXJqU-0G84wdmQ.png" /><figcaption>Infographic Credit: [ <a href="https://www.linkedin.com/in/graceschi/">Grace Chi</a> ]</figcaption></figure><p><strong>Exploration</strong> in RL involves the algorithm trying out various actions to discover new strategies or behaviors that could lead to better long-term rewards. This might mean taking unexpected paths or making non-intuitive decisions to gather valuable information about the environment. On the other hand, <strong>exploitation</strong> leverages the knowledge the algorithm has already acquired to make the best possible decisions based on existing data, focusing on maximizing immediate rewards. Balancing exploration and exploitation during the learning process is important: too much exploration can lead to inefficiency and slow learning, while excessive exploitation might cause the model to miss out on potentially optimal strategies. This tradeoff ensures the development of robust, effective RL models that can adeptly navigate their environments.</p><p>During model training process, optimization often requires toggling between exploration and exploitation as well. <strong>Exploration</strong> here involves experimenting with various algorithmic &amp; modeling approaches, trying different tools, or developing feature sets &amp; feature engineering to uncover potential improvements in model performance. This could mean testing new types of neural network architectures or exploring different data preprocessing techniques to enhance accuracy, which may or may not work out in the end. <strong>Exploitation</strong>, on the other hand, focuses on refining and tuning the prior identified, best-performing models: leveraging the most promising configurations and techniques discovered in the past to solidify gains and optimize performance via small adjustments. Striking the right balance ensures that the algorithm not only achieves high performance but also remains adaptable and innovative.</p><p>I could take it yet a step further: on a broader scale and in general, managing ML projects within a company entails a similar choices between exploration and exploitation. <strong>Exploration</strong> might include investigating <em>new</em> ML applications, data sources, or technologies that could offer competitive advantages or solve unaddressed problems. It represents the strategic pursuit of innovation, essential for staying ahead in rapidly evolving areas. <strong>Exploitation</strong> involves maximizing the value from <em>existing</em> ML projects and investments. This includes scaling successful models to new verticals, improving model efficiency, ensuring models are robust and reliable, incrementally improving our existing already operationalized models in production. Thus, effective ML project management requires navigating this dynamic: constantly evaluating when to push the boundaries and <em>take risks</em> with new initiatives, in contrast to <em>capitalizing</em> on proven successes to drive tangible business outcomes.</p><h4>Applied To Our Lives</h4><p>A relatable mini scenario many of us encounter also illustrates this point: When dining out with friends at a restaurant you’ve been before, do you want to experiment with a seasonal dish that could either be a delightful surprise or a disappointment; or do you play it safe by ordering a familiar dish you’re confident you’ll enjoy? Of course, it certainly goes beyond this example to more consequential scenarios when making tough decisions.</p><p>This principle mirrors the choices we face in our own lives. By examining and understanding the Exploitation vs. Exploration concept in-depth, we can unlock tactics for personal development and decision-making that transcend the choice of risk vs. comfort, urging us toward a more nuanced understanding of growth.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VuheScGGNmkZ4nYL" /><figcaption>Photo by <a href="https://unsplash.com/@hautier?utm_source=medium&amp;utm_medium=referral">Christophe Hautier</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>A Delicate Dance between Mastery and Discovery</h3><h4>Exploitation: Doubling Down on Strengths and Familiarity</h4><p>First, let’s discuss <strong>exploitation</strong> — what’s familiar, predictable, and often considered safe. Beyond AI and technical projects, in real life, exploitation can be closely tied to our daily routines and habits, such as our morning rituals or the structured flow of our workweek. This approach could be associated with “doubling down on your strengths” and also remaining within our “comfort zones”.</p><p>Exploitation offers stability, something we anchor on; it means playing to our strengths, and mastering them. We develop and utilize our existing knowledge/skills/experience to yield <em>reliable outcomes</em>. This strategy can lead to quick wins and the immediate satisfaction of achieving goals. And, if <em>consistently sustained</em> in the long game, this path can <em>result massive accomplishments</em>. The path is more familiar and known.</p><p>However, it’s crucial to acknowledge that over-reliance on what’s familiar can also be a double-edged sword. While it secures predictable results, it may simultaneously restrict our growth and the discovery of new opportunities. If unchecked, it could morph into <em>stagnation</em> — and can consequently be used as an excuse to continue on the same familiar well-traveled path, one refusing to deviate because “it works”.</p><p>This is when moving past your comfort zone will benefit — here’s a common example you or your friends might be familiar with: Staying at a comfortable yet unfulfilling job due to fear of change, because it provides stable pay and benefits. But, as we surmise, this has hidden costs and highlights the paradox of comfort: it can both <em>sustain and limit </em>us at the same time. Taking a leap to another role/company or new career path could uncover latent potential, leading to unexpected joy and fulfillment. We must recognize when exploitation, despite its benefits, becomes a hindrance to personal and professional development.</p><h4>Exploration: Venturing, Discovering, Pioneering</h4><p>Switching gears now to <strong>exploration — </strong>it inherently embraces the risks and challenges of delving into uncharted territories. This involves pushing boundaries, diving into unknowns and discomfort, often requiring more effort and courage. It’s just like stepping off the well-trodden trail, to instead pioneer a path into the dense unexplored jungle with a machete. Progress is slow, navigation is essential, and success depends on trusting one’s intuition.</p><p>Exploration is not just about seeking novelty for its own sake; it’s a <em>choice to trust and invest in our potential</em> to adapt, innovate, grow. This venture into new domains embodies the essence of strategic thinking and the pioneering spirit, pushing us beyond our current limits. The benefit of exploration lies in its potential to offer profound personal breakthroughs and deeper fulfillment, challenging us to see beyond our known capabilities and understanding, while keeping perspectives fresh.</p><p>However, it’s important to acknowledge that the path of exploration comes with its inherent risks and uncertainties. It can sometimes bring us to dead ends or on circuitous detours, requiring (potentially significant) investment of time and resources without the guarantee of immediate or tangible payoffs. These ventures into the unknown may not always yield the expected results, presenting real possibilities of setbacks or failures along the way. But it’s crucial to recognize that exploration, by its nature, values the journey itself, fostering a resilience by navigating through uncertainty and learning from every outcome, successful or not.</p><p>Despite the potential for failure, exploration’s true value lies in its ability to possibly lead to exponential growth and innovation by questioning the status quo, finding better ways of doing things. We could open ourselves up to novel experiences and knowledge, discovering hidden strengths, and igniting new desires. This pursuit of the unknown propels us forward, encouraging a continuous cycle of innovation and transformation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*n0kUXiyJTSFvDg23" /><figcaption>Photo by <a href="https://unsplash.com/@jjying?utm_source=medium&amp;utm_medium=referral">JJ Ying</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Integrating Exploitation AND Exploration</h3><p>As with many things in life, the two are not mutually exclusive and it doesn’t have to be such a binary decision. The dynamic between exploration and exploitation doesn’t have to mean we must only pick one mode. Life’s complexity allows — and indeed, benefits from — a more integrated and balanced approach. Recognizing the signs and that we can shift between these modes enables a more nuanced navigation through life’s challenges and opportunities.</p><p>Balancing these approaches requires us to <strong><em>oscillate between comfort and curiosity</em></strong>, leveraging what we know while being open to the unknown. This is a fluid, ongoing process that adapts as our circumstances and goals evolve. The key to mastering this balance lies in our ability to gauge when to lean into the familiarity of exploitation and when to embrace the uncertainties of exploration.</p><h4>Strategic Allocation</h4><p>A practical way to achieve this balance is through thoughtful allocation of our time, energy, attention. Borrowed from the finance/budgeting world: apply the <strong>70–20–10 guidelines</strong> (please do adjust percentages to what works for you in your situation). As an example: dedicate 70% of your efforts to refining existing skills, strengths, knowledge; 20% to exploring related, tangent areas that stretch your capabilities; and 10% to stepping into entirely new endeavors. Of course, the prerequisite step is to audit your time and attention, track where the efforts are going in the first place. Point is: taking an aware, structured approach ensures a solid foundation while still encouraging growth and innovation. Then, adjust the knobs depending on the situation like a diversified portfolio. If the season calls for it, ramp up on new riskier endeavors once resources are stable and available.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/986/1*85UbOqgLY-_R7gymZeMF9Q.png" /><figcaption>Infographic Credit: [ <a href="https://www.linkedin.com/in/graceschi/">Grace Chi</a> ]</figcaption></figure><p>To illustrate, a machine learning engineer, can apply the “70–20–10 rule” to their professional development to improve strengths and to counteract stagnation, foster growth. They dedicate 70% of their efforts to enhancing their expertise in current job responsibilities (following tutorials, reading blogs/articles, honing engineering skills). To expand their knowledge horizon, 20% of their time is invested in studying the latest academic papers or textbooks in AI, providing a deeper understanding of new theories, foundational math/stats, their practical applications, while perhaps dipping into the realm of cloud infrastructure operations. The remaining 10% is reserved for experimental projects, particularly in the emerging Generative AI field, allowing her to explore innovative applications and stay at the forefront of AI advancements. This strategic blend of deepening existing skills that align with core strengths, acquiring new knowledge, and pioneering with side projects keeps her career in AI consistent, fulfilling, and progressive.</p><h4>Intentional Exploration</h4><p>Exploration, especially in the context of professional and personal growth, demands <em>intentionality and awareness </em>rather than being a mere escape or an aimless endeavor. For engineers and technical professionals, this means embarking on exploration with clear goals and intention, whether for learning/discovery or solving specific problems — all should be anchored towards an applied end goal. Choosing to explore should not be used as a diversion but as a practical decision to enhance skills, broaden understanding, or innovate within your field. <strong>Mindful exploration</strong> is driven by a purpose — taking calculated steps (risks) towards achieving defined objectives, maximizing the chances that taking the exploration path begets a fruitful outcome.</p><h4>Self-awareness and Reflection</h4><p>Self-awareness and reflection are crucial in this dance. Pay attention to feelings of stagnation or restlessness as indicators that it might be time to “explore” new territories or learn new skills — attune to the energy if it calls for it. Conversely, when you find yourself in a season/state of flow, “exploit” and capitalize on this momentum to solidify your expertise and achieve mastery in your current endeavors. Knowing oneself and practicing timely shifts between exploration and exploitation can lead to a more fulfilling journey. You adjust the levers and are in control.</p><p>Understanding our predisposition towards either exploration or exploitation can serve as a compass in the quest for balance. We might naturally default to one mode of operating. Self-reflection enables us to recognize the signs, when we might be too entrenched in the comfort of the known (fear of change) or, conversely, when we’re overextending into the realm of the new at the expense of our well-being or distracting us from a truth we need to face.</p><h3>Conclusion: The Path to Growth</h3><p>As humans, we <em>paradoxically seek stability and novelty</em> at the same time. Setting personal goals with an equilibrium in mind allows us to handle life’s uncertainties with confidence, embracing change while maintaining a sense of stability. Navigating between exploration and exploitation requires a keen sense of self, an openness to adapt, and the courage to step into the unknown when necessary. By fostering an environment where both modes can coexist and inform one another, we set the stage for continuous learning, personal growth, and ultimately, a richer life experience.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3y6LH9jTBKxk-gPU6KS6vw.png" /><figcaption>Infographic Credit: [ <a href="https://www.linkedin.com/in/graceschi/">Grace Chi</a> ]</figcaption></figure><p>Exploitation represents growth through incremental improvements within the realm of the known, where leveraging existing skills and knowledge can lead to steady, consistent progress and achievements. Exploration, though riskier, opens the door to potentially exponential growth by venturing into uncharted territories and embracing the uncertainty that comes with it. Both modes are essential for comprehensive growth — exploitation solidifies our foundation, making us more efficient and confident in our abilities, while exploration pushes our boundaries, challenging us to adapt, innovate, and discover new potentials. By skillfully and consciously wielding both instruments, we can ensure a balanced path to one’s development, optimizing our growth potential in all aspects of our lives.</p><p>Thus, the exploration vs. exploitation tradeoff, a vital concept in the realm of artificial intelligence projects, parallels to the complexities of real life situations. By applying the principles of this tradeoff to our lives, we can cultivate a deeper, yet more adaptive approach to personal and professional growth. In the end, the art of choice isn’t just about deciding between the safety of the known and the promise of the unknown. It’s about recognizing that in the interplay of these forces, lies the potential for a life lived fully and consciously.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=52925e540c63" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai-assimilating-intelligence/the-exploration-vs-exploitation-tradeoff-navigating-lifes-choices-52925e540c63">The Exploration vs. Exploitation Tradeoff: Navigating Life’s Choices</a> was originally published in <a href="https://medium.com/ai-assimilating-intelligence">AI: Assimilating Intelligence</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build vs Buy: The #1 Question to Ask.]]></title>
            <link>https://medium.com/ai-assimilating-intelligence/build-vs-buy-the-1-question-to-ask-5219cb2387db?source=rss-9474a3ebe66f------2</link>
            <guid isPermaLink="false">https://medium.com/p/5219cb2387db</guid>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[decision-making]]></category>
            <dc:creator><![CDATA[Charles Chi]]></dc:creator>
            <pubDate>Sun, 03 Mar 2024 01:45:38 GMT</pubDate>
            <atom:updated>2024-06-21T18:39:22.679Z</atom:updated>
            <content:encoded><![CDATA[<h4>Decisive considerations when navigating tradeoffs during your assessment phase</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jAAS2mFq2WsucQLOvXfj6w.png" /></figure><h3>Introduction</h3><p>The decision to build in-house solutions or buy off-the-shelf products is a common point of debate. I’ve lead and participated in my fair share of these tech decisions, then seen solutions play out for better or worse. Drawing from my personal experiences, my goal here is to shed light on the factors that could guide this critical decision, while providing some reflections, learnings, and tips.</p><p>There’s a whole bunch of great guides and articles out there that cover this exact topic. Check out the links in image captions below to access solid sources for an introduction, various aspects/questions to consider, and practical tips.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/785/1*CSRdnwdJ5OAfc_Kx229FeA.png" /><figcaption>Image Credit, Source: <a href="https://blog.mergify.com/build-or-buy-software-the-question-to-ask-yourself/">https://blog.mergify.com/build-or-buy-software-the-question-to-ask-yourself/</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UGP1vQ2-wC4K1yY4hrcuuw.png" /><figcaption>Image Credit, Source: <a href="https://www.chameleon.io/blog/build-vs-buy">https://www.chameleon.io/blog/build-vs-buy</a></figcaption></figure><h3>Expanded Perspectives on The Build vs. Buy Decision</h3><p>I want to touch upon the more intriguing aspects when making this pivotal decision for your particular project. I’ve noticed throughout my career that these often emerge as common underlying themes —always super critical and sometimes overlooked.</p><h4>Beyond Financial Considerations: The Hidden Costs</h4><p>The initial step of the build vs. buy decision often focuses on the apparent, upfront costs, tempting decision-makers to make straightforward comparisons based on pure $ amount, as this is the easiest thing to consider and is the most real. However, we must know this: the choice is not purely financial; the true impact of this decision extends far beyond financial metrics, exerting a “tax” on your team and adjacent groups the reality is more complex.</p><p>The reality encompasses <strong>hidden costs </strong>like maintenance, setup, and integration, which significantly slow your team’s velocity. I call these “hidden taxes” on the team, that manifest as complexity — while tucked away, never truly disappears. It would just be shuffled around. The hidden costs can disrupt flow and increase your team’s surface area of risk and responsibilities.</p><p>Another counterpart is <strong>opportunity costs</strong>: if going down a build route, realize what will your team be giving up and <em>cannot</em> work on instead. Resources/time are finite, and must be deployed to work on a short list of priorities. By doing A, you won’t have capacity to do B.</p><p>If taking the buying route, it has its own set of challenges. <strong>Integration costs</strong> are “non-zero”, involving some form of investment and dev work, requiring time and resources. Be as clear as possible on the integration costs, and make sure it’s worth the investment.</p><p>Acknowledging and quantifying these hidden costs is crucial. Therefore, it’s extremely important to do your best and quantify costs, translate them into real dollars (or time) that your team has to cover to the leadership.</p><h4>Control vs. Leverage Tradeoff</h4><p>This tradeoff is a concept that <a href="https://medium.com/u/06f77cd28371">Ro Kommalapati</a> and I discuss all the time. There’s undeniable attraction to purchasing solutions: the promise of speed and leverage. Yet, this route may not always cater to every use case your team faces. The control and customization offered by building in-house can be crucial for certain applications for both vertical and horizontal scales, despite the longer time to value.</p><p>By knowing the limitations of “off-the-shelf” options and communicating with the vendors your requirements will give you a better understanding of the “<em>activation energy</em>” (the potential developer time or end user ramp-up time) involved to get a purchased solution working. There will likely be some investigation, development, and testing/validation to get the solution usable. Just by going the “buy” route, it doesn’t mean it will magically be adopted and integrated overnight. However, the leverage enables your team to adopt the solution swiftly, accelerating pace and shortening time-to-market. A small investment can yield significant results when executed effectively.</p><p>The control vs. leverage tradeoff is like choosing between transit via a high-speed train vs. a car. Buying solutions is like being on train tracks: fast and direct but confined to a predetermined path and a set of destinations, with limited range of flexibility (you’ll sometimes need to walk the last mile). In contrast, building custom solutions is like driving a car: it offers more control, nimbleness/agility, with flexibility to take detours or scenic routes, but at a potentially slower pace.</p><h4>The #1 Question: Strategic Imperative</h4><p>While considering hidden cost, opportunity costs, and control vs leverage tradeoffs are super important, it’s not the most paramount dimension. In my countless tech decision-making exercises, I’ve come to recognize a pivotal question that often doesn’t receive the spotlight it deserves, yet it fundamentally shapes the trajectory of technology within the org.</p><p>Here’s the #1 question to ask during the solution when you’re at a crossroads: <strong>Which choice aligns with your strategic objectives and core competencies? </strong>This question cuts to the core of the debate.</p><p>Building a solution may offer greater control and align closely with your organization’s core competencies. Building solutions that are outside core competencies might not actually be necessary or worth it — there are more important aspects of your product that deserves the attention/resources. If a solution definitely falls within your core competencies: building it in-house not only makes sense but is often necessary. This approach ensures that critical capabilities are tailored precisely to your needs, fully controlled, and can evolve as your strategy does. It’s about investing in areas that directly contribute to your<strong> competitive edge</strong> and are integral to your product or service offerings.</p><p>Buying can provide quick access to specialized capabilities and expedite implementation, which might fit into business objectives to deliver solutions fast. This approach can be especially appealing when time-to-market is a pressing concern, to beat out the competition or gain first mover advantage. For functions/systems that are peripheral or support your main operations — those outside your core competencies — subscribing to an existing solution can be the smarter choice. These areas, while necessary, do not differentiate your business in the market. Outsourcing them to specialized providers can free up valuable resources, allowing you to focus on strengthening and expanding your core competencies.</p><p>In essence, the build vs. buy decision is not just about cost or convenience — it’s about <strong>strategic alignment with your core competencies</strong>. Know the needs of the business and focus on building solutions that enhance and expand your core competencies. Consequently, choose to buy those solutions that <em>support</em> them. Thus, you can ensure that every decision contributes towards strengthening your competitive position and long-term strategic success.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*M71BS9N8lTFJnQuq" /><figcaption>Photo by <a href="https://unsplash.com/@ronnyrondonph?utm_source=medium&amp;utm_medium=referral">Ronny Rondon</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>Assessing the Tradeoffs: A Practical Approach</h4><p>With that being said, establishing a <strong>clear set of assessment criteria is vital</strong>. Start with understanding the exact problem you’re trying to solve, avoiding the trap of chasing the latest technology for its own sake. Get the goal written down and get alignment with stakeholders, defining success criteria, while staying rooted in practicality. It will always depend on your specific project use-case.</p><p>Last tip: You’ll also need to<strong> allocate proper time</strong> to canvas/research providers, design/estimate your own build, and develop proof-of-value before committing further either way. Yes, while there may be time constraints to support business (potentially high pressure to make a decision quickly), but these decisions really cannot and should not be premature, otherwise you will regret it later. You’ll be making a rushed suboptimal decision with lack of real clarity and full understanding of the problem — not ideal.</p><p>Thus, consider the following dimensions in your decision-making process:</p><ul><li><strong>Hidden Costs: </strong>What are the hidden taxes with upfront, overhead, integration, maintenance costs? Interruption and distraction costs are as real as the financial sticker price. For opportunity costs: what are other priorities might you be neglecting by choosing one path over the other?</li><li><strong>Control vs. Leverage:</strong> Evaluate the balance between having control over custom-built solutions versus leveraging the speed and ease of off-the-shelf products. Building in-house provides customization and control vital for your strategic initiatives, whereas buying can offer quick scalability and access to advanced features without less time.</li><li><strong>Strategic Alignment and Core Competencies:</strong> The key consideration! Determine if the solution aligns with your organization’s strategic objectives and enhances your core competencies. Focus on building in-house when the solution is strategic to your business and falls within your core capabilities. Consider buying when the solution supports non-core activities or when speed and efficiency are paramount.</li></ul><h4>Concluding Thoughts: Takeaway Wisdom for Tech Decisions</h4><p>The build vs. buy debate is more than a financial deliberation. By understanding the hidden costs, considering the strategic alignment, and carefully assessing all trade-offs, you can make a decision that not only addresses immediate needs but also positions your organization for long-term success. In navigating this complex solution boundary, root your decisions in practicality, with a keen eye on the broader strategic objectives of your organization.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5219cb2387db" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai-assimilating-intelligence/build-vs-buy-the-1-question-to-ask-5219cb2387db">Build vs Buy: The #1 Question to Ask.</a> was originally published in <a href="https://medium.com/ai-assimilating-intelligence">AI: Assimilating Intelligence</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Elevating ML Productivity: The Compounding Returns of Developer Ergonomics]]></title>
            <link>https://medium.com/ai-assimilating-intelligence/elevating-ml-productivity-the-compounding-returns-of-developer-ergonomics-ab6aa5897119?source=rss-9474a3ebe66f------2</link>
            <guid isPermaLink="false">https://medium.com/p/ab6aa5897119</guid>
            <category><![CDATA[investment]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[organizational-culture]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Charles Chi]]></dc:creator>
            <pubDate>Sun, 25 Feb 2024 03:47:01 GMT</pubDate>
            <atom:updated>2024-02-25T07:55:28.030Z</atom:updated>
            <content:encoded><![CDATA[<h4>How investing in developer ergonomics can enhance ROI and engineer efficiency for your ML workflows.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t4DaKrFAQQHoF_4iciyp1g.png" /></figure><p>In the realm of machine learning and artificial intelligence, an often overlooked concept is this term we here at <strong><em>AI: Assimilating Intelligence</em></strong> call <strong>“developer ergonomics”</strong>. When it comes to engineering and ML project management, this idea is just as important as the latest and greatest algorithms and technologies. We’ll talk about what this term actually means, and draw parallels to strategic allocation of capital in the investment world. Prioritizing developer ergonomics is not just a matter of convenience but a <em>strategic investment</em> that can yield <em>significant dividends</em> over time.</p><h3>Developer Ergonomics: The Strategic Play</h3><h4>What is Developer Ergonomics?</h4><p>Essentially, developer ergonomics revolves around optimizing the development environment, providing productive frameworks and tools to <em>enhance productivity</em> and <em>reduce friction</em> in deploying ML models, from idea inception to production (MLOps). But it shouldn’t stop there, and must be applied on a continual basis for model release and monitoring (e.g. CI/CD, observability). This allows machine learning engineers (MLEs) and Data Scientists (DS) to focus on innovation rather than being bogged down by cumbersome release and maintenance processes. Think of it as ensuring your best players are not only in the game but also playing in their best form.</p><p>Take inspiration from the DevOps realm and integrate the applicable parts to the ML space. Great AI/ML Engineering orgs have their operations tidy and streamlined. And it’s not a culture that can be instilled overnight — it needs to be fostered over time with <em>consistent engineering discipline </em>to achieve that highly sought <em>technical excellence</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6DXvwlndXvjboQhtQf4CQw.jpeg" /><figcaption>Image credits: “What is MLOps — Everything You Must Know to Get Started”. [<a href="https://towardsdatascience.com/what-is-mlops-everything-you-must-know-to-get-started-523f2d0b8bd8">article link</a>]</figcaption></figure><p>Sometimes, it’s totally difficult to justify to stakeholders and decision makers: investing large upfront time and resources in developer ergonomics can seem counterintuitive at first (“slow down to speed up” as my great friend <a href="https://medium.com/u/114262e93499">Spierer</a> often reminds me). But, of course, easier said than done — teams could frequently receive pushback, especially in high-pressure organizations:</p><ul><li>“How much effort to invest now? It seems like a fairly expensive investment, especially because of that looming deadline.”</li><li>“When is the right time? Is it even necessary at this moment?”</li><li>“It sounds like over-engineering, is it even worth it?”</li><li>“We’ll just do it later” (then never get to it, move on and that tech debt unsurprisingly piles up…)</li></ul><p>You know the drill. However, we can and should reframe this problem. This post aims to equip readers with a higher perspective to allow us not to just focus on the short-term returns, because that’s the easier thing to do.</p><p>Today, let’s draw some similarities that mirror the <em>compounding effect</em> seen in financial investments in markets — an idea we can take back to our colleagues/orgs to better justify these strategic investment efforts. That dollar today is going to be worth much more in the future, and make time play on our side, not against us. The goal is to rethink our philosophy so these longer investment plays are easier to justify in a new mental framework.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TKRer4nrojgGQokx" /><figcaption>Photo by <a href="https://unsplash.com/@austindistel?utm_source=medium&amp;utm_medium=referral">Austin Distel</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>Investing in Developer Resources: $ Capital == Dev Hours</h4><p>Drawing a parallel between capital investment and developer hours reveals the finite nature of our resources. Just as a wise investor allocates capital to various assets to maximize returns, a forward-thinking ML team invests in ergonomic tools and practices to maximize developer output and innovation (as guiding principles). The compounding benefits of this investment may not be immediately apparent, but become <em>undeniable over time — </em>enhancing productivity, morale, and ultimately: the bottom line.</p><p>When resources are finite, teams must be more selective with how they want to allocate their capital. You will stand at a crossroads:</p><ul><li>Do you want to invest in the incremental linear returns, like fixed income asset classes? (ship out the small feature, repeatedly do manual tasks, maintain one-offs, copy redundant code)</li><li>Or, do you prefer to take a bit of gamble on exponential growth? (invest in MLOps, streamline workflows, do one-click/automated deploys, enhance observability).</li></ul><p>Sure — inherently, there’s a bit of risk when considering longer time horizons with unknown, harder-to-quantify benefits, but the payoffs could be potentially huge. And, luckily, in the latter case, the risk is not as large as it may seem, and the payoffs are nearly guaranteed if done the right way. You just have to find the <em>courage to forego that easier short-term payoff</em>, stick to a methodology to frame/communicate investments, and advocate for a company culture that supports/encourages these types of investment plays.</p><h4>The Impact of Developer Ergonomics on Morale and Productivity</h4><p>Improving developer ergonomics directly correlates to improved morale and productivity among MLEs. It goes without saying: by minimizing tedious manual tasks, we enable our teams to engage more deeply with creative and innovative aspects of their work. This not only accelerates the dev / model release cycle but also fosters a culture of innovation and continuous improvement. Increasing experiment speed, shortening time for feedback is insanely critical for ML/AI teams. We need to take care of and look out for our most highest leverage most valuable resources.</p><p>Exploring the psychological concept of “flow”, where individuals achieve an optimal state of engagement and creativity, ergonomic environments contribute to higher productivity and innovation, and less context switching. Imagine if the company’s valuable MLEs are caught up managing deployments that have tons of hiccups, require constant babysitting, have a lack of observability, and need to do manual troubleshooting — that slowdown is a <em>huge tax burden</em> and will slow returns to a crawl. The debt may not be immediately apparent, but costs will manifest in nasty ways — you can only cover it for so long before it becomes unmanageable. A shift to a more ergonomic workflow over time will increase productivity and can fundamentally change how teams approach problem-solving and innovation in my daily work, freeing up for the harder and intensive tasks. Make the returns passive, and returns will compound.</p><h4>Scaling ML Workflows: From Manual Processes to Automation</h4><p>We all have to start somewhere, yeah? We can’t just jump to “automate all the things” right away. Firstly, starting with manual processes allows teams to fully understand the intricacies and patterns within their workflows. This is critical when deciding which processes to automate and when, and we don’t want over-optimize too early, as teams still need to balance out product and business needs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9xwgAzZUa_5_qmafhWEqHA.png" /><figcaption>From the famous xkcd webcomic: <strong>Is It Worth the Time? </strong><a href="https://xkcd.com/1205/">https://xkcd.com/1205/</a></figcaption></figure><p>Automation, while powerful, should be applied judiciously to ensure it amplifies efficiency without sacrificing quality or flexibility. It’s just like navigating a complex financial market. Investors carefully select their moments to buy or sell, ML teams must discern the precise timing and methods for automating their processes. We first need to have a good grasp and identify which processes are ripe for automation and ensuring that the transition enhances efficiency without compromising the quality or adaptability of the output.</p><p>Here’s an applied framework we use in our planning and weekly cadence when considering which processes need improved ergonomics:</p><ol><li><strong>Assessment</strong>: Begin by evaluating your current workflows for bottlenecks, repetitive tasks, and areas where human error is most prevalent. You need to document and itemize the opportunities; keep an on-going long-term investments list.</li><li><strong>Prioritization</strong>: Not all processes benefit equally from automation. Prioritize those that are high in volume, repetitive, and prone to error. Where are the bottlenecks, the biggest pain points? What are the small papercuts and often disrupt flow? Those could be solved with a quick task and address right away with high ROI. (see section below on prioritization, more tips and details here!)</li><li><strong>Selection</strong>: Choose automation tools and technologies that are scalable, maintainable, and cost-effective, ensuring they integrate seamlessly with existing workflows. This requires a spike effort: investigation, research, and trial phase.</li><li><strong>Implementation</strong>: Start small, with pilot projects to test the effectiveness of automation. Use these pilots to refine your approach before scaling up and investing more capital, doubling down the efforts to tangent processes.</li><li><strong>Evaluation</strong>: Continuously monitor the impact of automation on workflow efficiency, quality of output, and team morale. Adjust as necessary to optimize the benefits. We want to stay agile here. Ideally, you can quantify “dev hours saved”, which could translate to real dollar costs. It’s important to make these costs real and relatable.</li></ol><p>Teams could run this playbook on a sprint, monthly, or quarterly basis. Whichever suits teams best, and it all depends on a per-case basis — the important thing is to allocate that strategic time to consider opportunities for improving dev ergonomics. Of course, teams must factor in project deadlines and deliverables that support the business, to achieve the right balance. Just make sure you don’t go too far off to one extreme.</p><h4>Diversifying Investments in ML Workflows</h4><p>In the realm of finance, diversification stands as a cornerstone strategy for risk management and ensuring a robust portfolio. This principle holds equally true in the development and scaling of machine learning projects, where the balance between projects promising immediate returns and those aimed at long-term benefits is vital. A balanced approach not only <em>hedges against risk</em> but also <em>cultivates a steady stream of innovation and efficiency enhancements</em> across the board.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*t6KTkRmWRdjSn6Qf" /><figcaption>Photo by <a href="https://unsplash.com/@markuswinkler?utm_source=medium&amp;utm_medium=referral">Markus Winkler</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Here are some strategies for project selection and prioritization, framing ML projects as financial investments:</p><ol><li><strong>Risk and Return Analysis</strong>: Just as financial portfolios are carefully constructed based on the risk and return profile of each asset, ML projects should be evaluated on their potential impact (return) normalized by the complexity and uncertainty they introduce (risk). See <a href="https://www.investopedia.com/terms/s/sharperatio.asp">Sharpe Ratio</a>. This exercise helps in classifying projects into “<em>high risk, high reward</em>” and “<em>low risk, incremental reward</em>” buckets, enabling a balanced project portfolio.</li><li><strong>Correlation Consideration</strong>: In finance, assets are diversified to reduce risk exposure through non-correlated investments. Similarly, in ML workflows, projects should be diversified to include a mix of initiatives that are not closely related. This ensures that if one project encounters hurdles/blockers, others can still proceed unaffected, maintaining overall momentum, productivity, happiness.</li><li><strong>Taking Inventory</strong>: Regularly review and assess the current project pipeline, much like taking inventory of a financial portfolio and re-balancing. This involves evaluating ongoing projects for their progress, alignment with strategic goals, and contribution to the overall balance of the portfolio. Are you on the right track? Has risk profiles changed now that we have more experience and may have discovered new findings?</li></ol><p>Teams can approach the selection and prioritization of ML projects with a portfolio mindset, weighing both immediate benefits and the potential for long-term value creation, striking the right balance. This diversified investment approach in ML workflows not only mitigates risk but also ensures a dynamic and innovative development environment, poised for sustained growth and efficiency, with happy developer ergonomics and morale!</p><p>A critical tip here is: in order to <em>more effectively communicate the hidden costs</em>, teams must be able to <em>speak the language of its stakeholders</em>. We achieve this by staying disciplined and creative with how to quantify returns; then translate investment costs into something real that the decision-makers can fully feel and understand. Putting in the effort to understand your audience and meet them where they are goes a long, long way.</p><h3>Conclusion</h3><p>Investing in developer ergonomics is parallel to smartly managing a portfolio of investments. It requires a balance of short-term gains and long-term strategic planning, with unwavering and consistent discipline. For teams looking to supercharge their ML workflows, prioritizing ergonomics is not just an operational decision but a critical and strategic one that can significantly influence the team morale, trajectory of innovation, productivity, and effectiveness.</p><p>By fostering an environment that prioritizes the well-being and efficiency of our model developers, we not only enhance our near-term output but also establish the foundation for continued innovation and compounding success. We must invest in better developer ergonomics — provide supportive environment and operational tools with the same strategic mindset we apply to our financial portfolios. The most successful teams will be those that innovate not only in their technical endeavors but also in how they empower and enable their greatest asset — their developers.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ab6aa5897119" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai-assimilating-intelligence/elevating-ml-productivity-the-compounding-returns-of-developer-ergonomics-ab6aa5897119">Elevating ML Productivity: The Compounding Returns of Developer Ergonomics</a> was originally published in <a href="https://medium.com/ai-assimilating-intelligence">AI: Assimilating Intelligence</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>