<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Zoiner Tejada on Medium]]></title>
        <description><![CDATA[Stories by Zoiner Tejada on Medium]]></description>
        <link>https://medium.com/@ZoinerTejada?source=rss-748c8e764b06------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*qZdERtpftk6DfrUi.png</url>
            <title>Stories by Zoiner Tejada on Medium</title>
            <link>https://medium.com/@ZoinerTejada?source=rss-748c8e764b06------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 03:34:26 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ZoinerTejada/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[AI: Context Stuffing and RAG Patterns in Prompt Engineering]]></title>
            <link>https://medium.com/@ZoinerTejada/ai-context-stuffing-pattern-in-prompt-engineering-fb26d0378c64?source=rss-748c8e764b06------2</link>
            <guid isPermaLink="false">https://medium.com/p/fb26d0378c64</guid>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[gpt-4]]></category>
            <dc:creator><![CDATA[Zoiner Tejada]]></dc:creator>
            <pubDate>Mon, 08 May 2023 12:15:42 GMT</pubDate>
            <atom:updated>2023-05-10T12:09:59.528Z</atom:updated>
            <content:encoded><![CDATA[<p>A core pattern that unlocks powerful capabilities from large language models.</p><p>[This article is a part of the <a href="https://medium.com/@ZoinerTejada/exploding-head-ai-intro-15b1712886a4">🤯AI series</a>]</p><p>In a different post, <a href="https://medium.com/@ZoinerTejada/ai-memory-recall-in-large-language-models-597a0f44fd35">Memory &amp; Recall in Large Language Models</a>, we introduced the concept of context stuffing, which as the name implies gives the LLM more context (usually knowledge that is more recent than the data upon which it was trained).</p><p>Context stuffing is a useful pattern in certain scenarios, but it comes with limitations, namely an upper limit on the amount of text that can be provided as context. There are two factors to this limit, one is the hard upper limit on the number of tokens that can be sent to the model, and the other is a practical one- the cost of processing the tokens. The issue with cost is fairly obvious (more tokens increases cost), so we’ll delve into the Max Tokens issue here.</p><p><strong>49.152 pages</strong></p><p>The largest version of GPT 4 is called “gpt-4–32k” and according to the documentation it supports a maximum of 32,768 tokens.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/773/1*59JMCm9W0_vlNKhYf3moCQ.png" /><figcaption>OpenAI Documentation for GPT 4 (https://platform.openai.com/docs/models/gpt-4)</figcaption></figure><p>OK, so what does this limit mean in practical terms. Let’s do some simple approximations to get a perspective of just how much text this supports.</p><p>According to the OpenAI documentation, a <a href="https://platform.openai.com/docs/introduction/tokens">rough rule of thumb</a> is that 1 token is about 0.75 words of English text.</p><ul><li>So, this means a maximum of about 24,576 words.</li><li>Next, assume atypical page of text has about 500 words.</li><li>Dividing 24,576 words by 500 words/page gets us 49.152 pages.</li></ul><p>So that’s the upper limit. If you can squeeze all your context into 49.152 pages of data great. But as you start to combine lots of text to “catch up” your model’s knowledge, as we explain with the <a href="https://medium.com/@ZoinerTejada/ai-memory-recall-in-large-language-models-597a0f44fd35">50 First Dates Metaphor</a>, with any form of conversational history (e.g., any previous interactions and results you need to provide the model to keep a dialogue going) and you will likely find you quickly exhaust this limit.</p><p>Importantly, you don’t get to use all of these 49.152 pages for your context, as the Max Tokens also has to include the number of tokens used by the generated completion result.</p><p><strong>Context Stuffing is an Optimization Problem</strong></p><p>This gets us to an interesting realization. If you can’t put everything in the context, how do you choose what to include in the context?</p><p>This is where we like to think of context stuffing as an optimization problem. From all of the possible context, and you somehow search thru that candidate data and provide as context only the most relevant text?</p><p>This is the essence of the Retrieval Augmented Search (or RAG) pattern.</p><p><strong>Retrieval Augmented Search</strong></p><p>With Retrieval Augmented Search, the core idea being a loose interpretation of<a href="https://arxiv.org/abs/2005.11401"> this research paper</a>, you use some external mechanism to select the text to include in context.</p><p>So, our prompt structure can be something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/405/1*fsO1nqjIxdfdDRhBWeMjGg.png" /><figcaption>Example prompt structure</figcaption></figure><p>And our question, might be something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/1*afh1U8WEPytzbrsT5d1q8g.png" /><figcaption>Example user question</figcaption></figure><p>But because the underlying data used to train the model dates back to 2021, it won’t have an up-to-date answer for this. For the context we need to find some text, say a recent news article, that provides the most up to date context. We can do something like full text search for terms matching the question. Typically, this is done with vector search (aka an embeddings database) using the vectorized form of the query to search a database of vectorized documents, and the text document behind the vector that most strongly matches is retrieved and used as context.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/423/1*nGZDUUjq3p_2D-JFwwZATg.png" /><figcaption>Illustration of processing pipeline using RAG (source: author/Solliance)</figcaption></figure><p>Clearly, this helps make significantly better use of whatever number of tokens are useable by the model.</p><h3>Crazy🤯.</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fb26d0378c64" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI: Memory & Recall in Large Language Models]]></title>
            <link>https://medium.com/@ZoinerTejada/ai-memory-recall-in-large-language-models-597a0f44fd35?source=rss-748c8e764b06------2</link>
            <guid isPermaLink="false">https://medium.com/p/597a0f44fd35</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[gpt-4]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[openai]]></category>
            <dc:creator><![CDATA[Zoiner Tejada]]></dc:creator>
            <pubDate>Thu, 04 May 2023 14:37:08 GMT</pubDate>
            <atom:updated>2023-05-04T14:37:08.718Z</atom:updated>
            <content:encoded><![CDATA[<p>Large language models are like “50 First Dates”</p><p>[This article is a part of the <a href="https://medium.com/@ZoinerTejada/exploding-head-ai-intro-15b1712886a4">🤯AI series</a>]</p><p><strong>Always forgetting, up to a point</strong></p><p>I like to describe large language models as characters. In fact, I like to think of them as that character Lucy that Drew Barrymore plays in the movie <a href="https://www.imdb.com/title/tt0343660/">“50 First Dates”</a>. In that movie, Adam Sandler’s character, Henry Roth, finds in Lucy the girl of his dreams and discovers she suffers from a severe case of short term memory loss. In fact, after a certain point in her life she does not make anymore long term memories. Henry Roth, who marries her, sweetly creates a video to remind her of their life together to date. Every morning when she wakes up not knowing who this person is, they watch the video together so she can get caught up on the current events that she forgot when she went to sleep.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/275/1*KSDtpmpaCeWQXQ_Od8Glmg.png" /><figcaption>Movie poster from 50 First Dates (<a href="https://www.imdb.com/title/tt0343660/">https://www.imdb.com/title/tt0343660/</a>)</figcaption></figure><p><strong>Large language models are like Lucy.</strong></p><p>Irrespective of the number of billions of parameters that the large language models have, these models have two forms of “memory”: parametric memory and nonparametric memory.</p><p>The parametric memory models have are effectively the memories encoded in the model weights during training. Their “knowledge” is only as current as the most recent data the model was trained against.</p><p>With GPT-4 available in <a href="https://azure.microsoft.com/en-us/products/cognitive-services/openai-service/">Azure OpenAI Service</a> (and from OpenAI), this cut-off on long-term memory is well documented:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/835/1*j-73hHCeMNuZhIcbTqsd0g.png" /></figure><p>So according to the documentation, the model was trained with data up until September 2021. This means the model doesn’t know who the current president is, the current price of MSFT stock or the any current global events. Just like for Henry Roth, this can present a problem!</p><p>Comically, the solution to how we catch a model up on recent events is almost exactly like what Henry Roth did for Lucy- before the day starts show her a summary of what she needs to know for the day.</p><p>With large language models, the becomes a prompt engineering technique called context stuffing, because along with your query to the model you provide the text that provides any necessary context for the model to be “current”. This is the transient non-parametric memory that you inform. You have to show this context to the model every time you prompt it.</p><p>Here is an example of how you might structure such a prompt to understand news about cryptocurrencies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/452/1*V6zCDnWbk-9dVbt-BoioOw.png" /><figcaption>Source: author / Solliance</figcaption></figure><p>In the above example, I gave GPT 3.5 some context that provided a paragraph from a recent news article that provided up to date information. Without this context the model results would have stated that there were two such addresses, which was true according to the model’s training data from back in September 2021.</p><p>Obviously, how exactly how you identify what context to stuff from the user query is its own challenge and a topic I will cover in a future post.</p><p><strong>What about fine-tuning?</strong></p><p>This is also a topic we will explore in more depth in a future post. The short answer is that currently these models either cannot be fine-tuned or for the versions of GPT that can be fine-tuned, you end up using older versions that sacrifice the InstructGPT capabilities that make them so powerful with carefully crafted prompts. In my experience, simply put, they perform worse- a lot worse.</p><h3>Kind of 🤯 right?</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=597a0f44fd35" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI: Deep Reinforcement Stack]]></title>
            <link>https://medium.com/@ZoinerTejada/ai-deep-reinforcement-stack-eca561360c39?source=rss-748c8e764b06------2</link>
            <guid isPermaLink="false">https://medium.com/p/eca561360c39</guid>
            <category><![CDATA[azure-machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <dc:creator><![CDATA[Zoiner Tejada]]></dc:creator>
            <pubDate>Mon, 20 Mar 2023 01:19:06 GMT</pubDate>
            <atom:updated>2023-03-20T01:19:06.671Z</atom:updated>
            <content:encoded><![CDATA[<p>Taking a peek at the DRL toolbox.</p><p>[This article is a part of the <a href="https://medium.com/@ZoinerTejada/exploding-head-ai-intro-15b1712886a4">🤯AI series</a>]</p><p>The literature on Deep Reinforcement Learning is still heavily academic, littered with Greek symbols (aka Math) and if you are just getting started it can be a little overwhelming how you go from this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/486/1*2aOE0-gV1ua_fZeAc_uhFw.png" /><figcaption>Example objective function being optimized in Deep Reinforcement Learning</figcaption></figure><p>to being able to run this:</p><pre>    def optimize_model(self):<br>        T = len(self.rewards)<br>        discounts = np.logspace(0, T, num=T, base=self.gamma, endpoint=False)<br>        returns = np.array([np.sum(discounts[:T-t] * self.rewards[t:]) for t in range(T)])<br><br>        discounts = torch.FloatTensor(discounts).unsqueeze(1)<br>        returns = torch.FloatTensor(returns).unsqueeze(1)<br>        self.logpas = torch.cat(self.logpas)<br><br>        policy_loss = -(discounts * returns * self.logpas).mean()<br>        self.policy_optimizer.zero_grad()<br>        policy_loss.backward()<br>        self.policy_optimizer.step()</pre><p>Source: <a href="https://github.com/mimoralea/gdrl/blob/master/notebooks/chapter_11/chapter-11.ipynb">Implementation of REINFORCE by Miguel Morales</a></p><p>Since a picture can be worth a thousand words, here is the stack I tend to use most often.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/769/1*BbD9kZoWmO7oV0llZi3wjQ.png" /></figure><p>Want to kick the tires with this stack? See my <a href="https://medium.com/@ZoinerTejada/ai-deep-reinforcement-learning-d317a4310d19">previous article</a> in this series.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eca561360c39" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI: Deep Reinforcement Learning]]></title>
            <link>https://medium.com/@ZoinerTejada/ai-deep-reinforcement-learning-d317a4310d19?source=rss-748c8e764b06------2</link>
            <guid isPermaLink="false">https://medium.com/p/d317a4310d19</guid>
            <category><![CDATA[azure-machine-learning]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Zoiner Tejada]]></dc:creator>
            <pubDate>Fri, 17 Mar 2023 01:57:44 GMT</pubDate>
            <atom:updated>2023-03-17T02:10:08.340Z</atom:updated>
            <content:encoded><![CDATA[<p>A rewarding journey.</p><p>[This article is a part of the <a href="https://medium.com/@ZoinerTejada/exploding-head-ai-intro-15b1712886a4">🤯AI series</a>]</p><p>Let’s cut to the quick. What is Reinforcement Learning and what makes it Deep?</p><p>Reinforcement Learning is concerned with solving sequential decision-making problems. Deep Learning techniques are used with Reinforcement Learning because the deep neural networks of deep learning are excellent at approximating functions, and reinforcement learning is pretty much all about approximating an objective function (standby for more on that). The other reason deep learning mind-blowing in its support of reinforcement learning is that with deep learning, the neural network model does the feature learning as a part of the training process (an expert human doesn’t need to specify the features manually, as is typically needed in traditional machine learning).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/1*zNs0amv-vfs3_5he5FurZA.png" /><figcaption>Comparison of feature learning between Machine LEarning and Deep Learning</figcaption></figure><p>We have an objective (or more literally an objective function, but we’ll come back to that) when solving these problems. We take actions and get feedback from the world about how close we are to achieving the objective. Reaching the goal involves taking many actions in sequence, each action changing the world around us. We observe these changes in the world as well as the feedback we receive before deciding on the next action to take in response.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/755/1*KYIEOuJIfrg0900R3zldLw.png" /></figure><p>More formally, we have an Agent that observes the state of the Environment, uses a policy that evaluates the current state to select an action. This action is taken against the Environment, which responds be transitioning to the next state and possibly providing some reward.</p><p>I like to think of these agents as glorified cookie monsters, who want to do everything they can to eat the most cookies they can- that is their objective. The cookies, those are their reward.</p><p>Some example problems that can be framed this way:</p><ul><li>Playing video games</li><li>Robotics, Autonomous Systems, Driving a Car, Balancing a Pole</li><li>Algorithmic Trading</li></ul><p>Deep Reinforcement Learning is a topic almost as wide as it is deep. Before we dive it, let’s cover some interesting results as motivation.</p><p>Can you write a program to master the Atari game Breakout?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/180/1*mByFG_u74gZRCCkNQfsmjw.png" /><figcaption>Screenshot from Breakout</figcaption></figure><p>Mnih et al in 2013 published the first deep learning model to successfully learn control policies directly from high dimensional sensory input using reinforcement learning.</p><p>The method applied to play <em>seven</em> Atari 2600 games in the Arcade Learning Environment <em>without </em>any adjustment to the latter’s architecture. The results outperformed all previous approaches on six games, and surpassed expert human players on three games.</p><p>🤯 2013 Paper: “<a href="https://arxiv.org/pdf/1312.5602.pdf">Playing Atari with Deep Reinforcement Learning</a>”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RuKZgtLx3Abiv4tQGAemzA.png" /><figcaption>Screenshots from the 2013 paper <em>Playing Atari with Deep Reinforcement Learning</em></figcaption></figure><p>By 2016, most Atari 2600 games were mastered, except for one: Montezuma’s Revenge. The trouble with this game was its brutality- pretty much everything killed you, quickly and you only got a reward for completing a level (which was complex on its own). An agent trying to get by on trial and error gets almost no reward signals.</p><p>The insight was to enhance the agent with two new signals that gave it intrinsic motivation to explore: <em>novelty</em> (have I seen this before) and <em>surprise</em> (predict what you expect to see, compare to what is seen).</p><p>🤯 2016 Paper: “<a href="https://arxiv.org/pdf/1606.01868.pdf">Unifying Count-Based Exploration and Intrinsic Motivation</a>”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pGC3gVNnB9W3GgLtZIHhTg.png" /><figcaption>Illiustration from the 2016 paper Unifying Count-Based Exploration and Intrinsic Motivation</figcaption></figure><p>Deep Reinforcement Learning was used so flexibly to master Atari games over 6 years ago, and the pace of innovation has only accelerated.</p><p><strong>“HELLO WORLD” Deep Reinforcement Learning Style</strong></p><p>The Hello World scenario of DRL is CartPole. Thousands of papers have been written using it.</p><p>Basically you can visualize the scenario like this: you have a cart that moves on track, and a pole on the cart. Keep the pole upright for as long as you can. Oh and you can only do so my forcefully nudging the cart left or right.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/644/1*74U9aF1abcpPPQlVQGGxRw.png" /><figcaption>Screenshot of CartPole in action</figcaption></figure><p>Let’s break this down a little further.</p><ul><li><strong>Objective: </strong><br> Keep the pole upright for 500-time steps</li><li><strong>State: </strong><br> [Cart Position, Cart Velocity, Pole Angle, Pole Angular Velocity]</li><li><strong>Action:</strong><br> 0 to move cart fixed distance left OR <br> 1 to move cart fixed distance right</li><li><strong>Reward:</strong><br> +1 for each time step pole remains upright</li><li><strong>Termination:</strong><br> When poll falls over ( &gt; 12° from vertical) OR<br> Cart moves off screen OR <br> Max time step of 500 reached</li></ul><p>Keep this scenario in mind as we proceed (and as you proceed in learning about DRL- it is a surprisingly useful scenario to ground your understanding with).</p><p><strong>How do RL Agents Learn?</strong></p><p>The agent <em>learns</em> to pick good actions by interacting with the environment in a process of trial and error The agent uses the reward signals it receives to <em>reinforce</em> good actions</p><p>What are these signals?</p><ul><li>The signals exchanged are (state, action, reward) often written as (s, a, r)</li><li>Each (s, a, r) tuple for a <em>single time step </em>is called an <em>experience</em></li><li>The time horizon from start to finish is called an <em>episode</em></li><li>The sequence of experiences within an episode is called a <em>trajectory</em></li><li>An agent typically needs many episodes to learn a good policy</li></ul><p><strong>What can an Agent Learn?</strong></p><p>An agent can learn one of these primary functions:</p><ul><li>A policy</li><li>A value function</li><li>A model of the environment</li></ul><p>It can also learn <strong>both</strong> a policy and a value function.</p><p><strong>What is a Policy?</strong></p><p>The sum of the rewards an agent receives is called the <em>objective. </em>The agent’s goal is to maximize the objective. It does this by selecting good actions. The function that an agent uses to decide on an action is called a <em>policy.</em></p><p>The policy is the core item that the agent learns, though not always directly</p><p>A policy π maps a state s to an action a:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/120/1*tDP9xY18JqrweEOIKMGlig.png" /></figure><p>A policy can be stochastic (random) such that it may probabilistically output different actions for the same state.</p><p>In this case it is expressed as an action sampled (indicated by ~) from a policy:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/88/1*jsOY5-Q0eKPCYLQQjzyvOA.png" /></figure><p>Let’s go back to CartPole to apply what we just learned.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/405/1*yBAIQZu-iBUNjQoUKFVa9w.png" /></figure><p>Example policy for CartPole:<br> π(s) = always move left</p><p>Analysis: Good to correct the initial lean, bad in the long run as it will cause the pole to fall on the right.</p><p><strong>How is the reward used?</strong></p><p>Assume we have this trajectory of experiences (from first to last) from an episode:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/411/1*q5wu5HtS0rEmeQ2FRnHvyA.png" /></figure><p>The <em>return</em> (the <em>total of </em>rewards the agent sees) is defined as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/435/1*YdE0eGa6ojai8uQTJ8Jvpw.png" /></figure><p>Which can be compactly re-written as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/178/1*BPY-p-5c4R4NJk6gfLqI9g.png" /></figure><p>Let’s go back to CartPole to apply what we just learned.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/240/1*AEb39nJt54PA4sgK99X-HQ.png" /></figure><p>Recall the Reward for CartPole:<br>+1 for each time step pole remains upright<br>The undiscounted return of the episode (assuming it ends at s2):</p><p>R(τ) = 1+1+1=3</p><p><strong>Discounting Rewards over time</strong></p><p>In the previous definition of return, you may have wondered what the γ (gamma) term was all about.</p><p>Typically, you don’t take the full value of the reward at each time step, but rather a sum of the <em>discounted</em> rewards over all time steps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/180/1*x6a1pp4bS3eYmLZhnvE0gg.png" /></figure><p>The γ term basically means that the rewards that come earlier in the trajectory can be weighted more heavily (counted more completely) than those rewards earned later, which can be used to encourage the learning behavior we want. For example, we can:</p><ul><li>focus only on the current reward (set γ to 0.0) or</li><li>focus more on recent rewards (set γ <em>closer</em> to 0.0) or</li><li>focus less on recency by letting rewards that occur much later have an impact (set γ <em>closer</em> to 1.0) or</li><li>treat all rewards with the same importance (set γ to 1.0)</li></ul><p>If you have a background in a different field, like digital marketing, you might think this is a credit assigment problem. It is! The calculation of the Return is a way to assign the right “credit” to actions that ultimately led to the desired outcome, and not assign credit to those that did not help or worse.</p><p><strong>What is a Value Function?</strong></p><p>A value function takes a state <strong><em>or</em></strong> a state-action pair <br> and estimates the expected return of the trajectory:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/210/1*uUoDdlvEXiE05uE3uiAr1Q.png" /></figure><p>or</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/239/1*QmbbUJxUgkpiTHlvh5LdBQ.png" /></figure><p>Value functions help an agent understand how good the states and available actions are in terms of the expected future return. The state-value function (V) you can think of as the average value of being in that state, irrespective of the action taken. The action-value function (Q), you can think of as providing the value of taking a given in action from that state.</p><p>Let’s go back to CartPole to apply what we just learned.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/1*qSr2tLgmrR3YBKXtvmTI4Q.png" /></figure><p>Recall the State for CartPole:<br> [Cart Position, Cart Velocity, Pole Angle, Pole Angular Velocity]</p><p>Let’s look at some example states and hypothetical values that could result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/1*Zu9_-1UGBJ5eBOnKBrSP7g.png" /></figure><p>Because the pole is straight up.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/284/1*er5iWC8_0R65mwHrPa9_xQ.png" /></figure><p>Because the pole is going to fall over past 12 degrees from upright (90 degrees).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/463/1*NpsMYtGbLLYoKAtuXyIA7Q.png" /></figure><p>Because this will help the pole get back upright.</p><p><strong>What is an Environment Model?</strong></p><p>The agent can learn the <em>transition function </em>implicitly used by the environment. If the agent can learn this function, it can “predict” the next state to which the environment will transition, enabling it to plan good actions without interacting with the environment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/136/1*Cw_sTkDA8tlzW0aeMnpdLA.png" /></figure><p>Think of it like learning the rules of chess. Now that you know the rules, can predict the next state for an action.</p><p>This is referred to as Model-Based RL. In many cases the environment model is <em>impossible </em>to learn, so other approaches are taken to learn a Policy without having this model. This is referred to as Model-Free RL.</p><p>😅phew! That was a lot of theory. Let’s get our hands dirty.</p><p><strong>Deep Reinforcement Learning in Azure</strong></p><p>Without further ado, let’s see how we can run some state-of-the-art deep reinforcement learning algorithms to master CartPole, the Atari game Breakout and attempt a Robotics example training a Cheetah to move.</p><p>As usual, we’ll run these examples in Azure Machine Learning. As we are just getting started in DRL, rather than write all of our agents from scratch, we’ll show using the industry standard implementations from <a href="https://github.com/DLR-RM/stable-baselines3">Stable Baselines 3</a>, which provides a set of reliable implementations of reinforcement learning algorithms in PyTorch, and we’ll use those algorithms from the <a href="https://github.com/DLR-RM/rl-baselines3-zoo">RL Baselines 3 Zoo</a> which provides scripts for training, evaluating agents, tuning hyperparameters, plotting results and recording videos of agents in action.</p><p>Grab a copy of <a href="https://github.com/solliancenet/advanced-computing-workshop/blob/main/Lab%202/01_rl_baselines_zoo_labs.ipynb">this notebook</a> and import it into your Azure Machine Learning environment.</p><p>You won’t need too powerful a compute instance for this, I used a `STANDARD_DS12_V2`.</p><p>Once you loaded the notebook into your environment and spun up your compute instance, step thru the network executing each cell.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FyQvb7jowxWHErBsTnchcQ.png" /></figure><p><strong>Check your understanding!</strong></p><p>After you’ve experimented with notebook, see if you can answer these questions.</p><p><strong>Training CartPole-v1</strong></p><ul><li>What hyperparameters were loaded? From where?</li><li>How many environments were created to train your agent?</li><li>What “policy” was used?</li><li>What was the average episode reward of your last episode? How long did the episode run for? Why are they the same?</li></ul><p><strong>Evaluating Your Trained Agent</strong></p><ul><li>What was the average reward achieved by your agent? How long did the episode run for to get that reward? Why are they the same?</li><li>How did this compare to the pre-trained, fully-tuned agent?</li><li>After training your agent just a little more, what reward were you able to achieve?</li></ul><p><strong>Pre-Trained Breakout and Half Cheetah</strong></p><ul><li>If you watch the breakout gameplay, where are the reward signals coming from? What techniques did the agent learn?</li><li>If you watch the half cheetah, something is clearly wrong…any thoughts on what you could do to improve it?</li></ul><p>Pretty 🤯 stuff, right?!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d317a4310d19" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI: Embeddings]]></title>
            <link>https://medium.com/@ZoinerTejada/ai-embeddings-3fb51516bc8b?source=rss-748c8e764b06------2</link>
            <guid isPermaLink="false">https://medium.com/p/3fb51516bc8b</guid>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[azure-machine-learning]]></category>
            <category><![CDATA[gpt]]></category>
            <dc:creator><![CDATA[Zoiner Tejada]]></dc:creator>
            <pubDate>Thu, 16 Mar 2023 01:07:40 GMT</pubDate>
            <atom:updated>2023-03-17T02:09:29.031Z</atom:updated>
            <content:encoded><![CDATA[<p>Turn text into coordinates.</p><p>[This article is a part of the <a href="https://medium.com/@ZoinerTejada/exploding-head-ai-intro-15b1712886a4">🤯AI series</a>]</p><p><strong>What are embeddings?</strong></p><p>An embedding is an information dense representation of the semantic meaning of a piece of text.</p><p>It simply a vector of floating-point numbers, such that the distance between two embeddings in the vector space is correlated with semantic similarity between two inputs in the original format.</p><p><strong>The power of embeddings</strong></p><p>Embeddings basically convert text to coordinates that maintain relative distances between text that has similar meaning.</p><p>Applications include:</p><ul><li>Semantic similarity</li><li>Clustering</li><li>Full-text search</li><li>Classification</li><li>Recommendations</li></ul><p><strong>Embeddings in Azure OpenAI</strong></p><p>You can use the very powerful models from Azure OpenAI to acquire the embeddings for a given piece of text. In Python, you access this API using the openai library.</p><p>Here’s an example call:</p><pre>response = openai.Embedding.create(<br>  engine=&quot;gpt3-similarity-babbage-001&quot;,<br>  input=&quot;A neutron star is the collapsed core of a massive supergiant star.&quot;<br>)</pre><p>What you get back is a vector of numbers, whose shape depends on which model you selected for creating the embeddings. In the case of the above code (where we selected babbage-001 ) the returned vector has 2,048 dimensions:</p><pre>len(response.data[0].embedding)</pre><pre>2048</pre><p>Taking a peak at this vector, we get something like this for the above text and model:</p><pre>[0.012890788726508617,<br> 0.01312282308936119,<br> -0.006028592120856047,<br> -0.000760019407607615,<br> 0.022240908816456795,<br> 0.018734613433480263,<br> -0.01254703477025032,<br> -0.005061783362179995,<br> 0.009289962239563465,<br> -0.016311144456267357,<br> -0.011825150810182095,<br>...<br>]</pre><p>Think of this vector as a highly nuanced coordinate system, where each dimension encodes some kind of meaning such that if you encoded another piece of text about neutron stars it would be closer to the original text than say a piece of text about movie stars. 🤯</p><p>Want to try this out? Grab a copy of <a href="https://github.com/solliancenet/advanced-computing-workshop/blob/main/Lab%201/azure_open_ai_labs.ipynb">this notebook</a> and import it into your Azure Machine Learning environment.</p><p>The only additional thing you will need is to deploy your own endpoint and then retrieve your OpenAI API Key. Here are the standard instructions to do that: <a href="https://learn.microsoft.com/en-us/azure/cognitive-services/openai/quickstart?pivots=rest-api#prerequisites">Get started generating text using Azure OpenAI Service</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3fb51516bc8b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI: Foundation Models]]></title>
            <link>https://medium.com/@ZoinerTejada/ai-foundation-models-55345ffb36d9?source=rss-748c8e764b06------2</link>
            <guid isPermaLink="false">https://medium.com/p/55345ffb36d9</guid>
            <category><![CDATA[foundation-models]]></category>
            <category><![CDATA[gpt]]></category>
            <category><![CDATA[azure-machine-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Zoiner Tejada]]></dc:creator>
            <pubDate>Thu, 16 Mar 2023 00:31:19 GMT</pubDate>
            <atom:updated>2023-03-17T02:09:01.107Z</atom:updated>
            <content:encoded><![CDATA[<p>Foundation Models are Mind Blowing</p><p>[This article is a part of the <a href="https://medium.com/@ZoinerTejada/exploding-head-ai-intro-15b1712886a4">🤯AI series</a>]</p><p>Foundation models like GPT-3 and Codex are models trained on broad data at scale so they can be adapted to a wide range of downstream tasks. We will cover how the Azure OpenAI Service enables you to use language models for text and code generation, reasoning, inferencing and comprehension and how you can customize these powerful models with few shot learning using your labeled data.</p><p>The Stanford Institute for Human-Centered Artificial Intelligence’s (HAI) <a href="https://crfm.stanford.edu/">Center for Research on Foundation Models</a> (CRFM) coined the term foundation model. This term refers to “any model that is trained on broad data (generally using self-supervision at scale) that can be adapted (e.g., fine-tuned) to a wide range of downstream tasks”. At their core, these models are based on deep neural networks and self-supervised learning.<br>Foundation models are “designed to be adapted (e.g., finetuned) to various downstream cognitive tasks.</p><p>We will focus on Foundational Language Models. The recent mind-blowing capability of these foundation models is due in no small part to the industry’s ability to build and train models with a massive number of parameters, just look at this hockey stick growth in the number of parameters available in recent years:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*98AeKo-rg0cvs5AA-8g21Q.png" /></figure><p><strong>What’s in the box with Azure OpenAI Language Models?</strong></p><p>GPT-3 base series: understand and generate natural language</p><p>Codex series: understand and generate code from natural language</p><p>Embeddings series: understand and use embeddings</p><p>One way to think about the model options in Azure Open AI is to view them on a spectrum of model “power”, where speed and cost increase from A to D:</p><ul><li>Ada (least powerful, fastest, least expensive)</li><li>Babbage</li><li>Curie</li><li>Cushman</li><li>Davinci (most powerful, slowest, most expensive)</li></ul><p><strong>The power of completions</strong></p><p>OpenAI’s models can do everything from generating original stories to performing complex text analysis, “completing” text based on a “prompt”.</p><p>The completions endpoint provides a text-in, text-out interface</p><ul><li>Prompt: Your input text</li><li>Completion: Generate text that attempts to address your prompt</li></ul><p>Applications:</p><ul><li>Classification</li><li>Generation</li><li>Conversation</li><li>Transformation</li><li>Summarization</li></ul><p>The following screenshots shows an example using the Open AI Studio Playground with the Tldr; prompt that requests a summary of paragraph.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s11KGyoLKbjNdvLOdLZ_iQ.png" /></figure><p>The responses the foundation model returns can be surprising and mind-blowing. For example, if I try a little Monty Python humor:</p><p>Prompt: <em>What is the unladen airspeed of a European swallow?</em></p><p>Response: <em>The unladen airspeed of a European swallow is about 11 meters per second, or 24 miles per hour.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1mGnxj4lkWj1Yw2vaIJHYg.png" /></figure><h3>Wow.🤯</h3><p>Want to try this for yourself? Follow the simple steps in this document to access the Open AI Studio environment:</p><p><a href="https://learn.microsoft.com/en-us/azure/cognitive-services/openai/quickstart?pivots=programming-language-studio">https://learn.microsoft.com/en-us/azure/cognitive-services/openai/quickstart?pivots=programming-language-studio</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=55345ffb36d9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ EXPLODING HEAD AI]]></title>
            <link>https://medium.com/@ZoinerTejada/exploding-head-ai-intro-15b1712886a4?source=rss-748c8e764b06------2</link>
            <guid isPermaLink="false">https://medium.com/p/15b1712886a4</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[diffusers]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[gpt-4]]></category>
            <dc:creator><![CDATA[Zoiner Tejada]]></dc:creator>
            <pubDate>Wed, 15 Mar 2023 23:38:59 GMT</pubDate>
            <atom:updated>2023-05-10T12:11:11.772Z</atom:updated>
            <content:encoded><![CDATA[<p>A new blog series for this AI moment.</p><p>AI is having quite the moment, and it&#39;s a great time to be building mind blowing solutions that use AI. This is a blog series intended to explore the implementation side of advanced AI on topics that are, frankly, mind blowing. We also felt a nod to <a href="https://medium.com/u/b1574f0c6c5e">Hugging Face</a> was in order, since they are the original AI company to name themselves with an emoji.</p><p>The aim of this blog series is to provide applied techniques with code samples and/or step-by-step instructions so you can touch and feel these powerful techniques in action, and hopefully get inspired to apply them to your own solutions.</p><p>For the most part our tech stack will consists of <a href="https://azure.microsoft.com/en-us/products/machine-learning/">Microsoft Azure Machine Learning</a>, <a href="https://code.visualstudio.com/">Visual Studio Code</a>, Jupyter Notebooks and <a href="https://pytorch.org/">PyTorch</a>, a whole lot of GitHub and the occasional 3rd party API when expedient.</p><p>The topics we aim to cover run the gamut of mind-blowing topics, including:</p><ul><li>Deep Reinforcement Learning and Inverse Reinforcement Learning</li><li>Natural Language Processing with Deep Learning<br>- Embeddings</li><li>Generative AI<br>- Prompt Engineering<br>- Large Language Models (e.g., GPT 3x/4x)<br>- Diffusers (e.g., DALL-E, Stable Diffusion)<br>- Code Generation (e.g., GPT 3x)</li><li>Crossover topics that combine several mind-blowing techniques<br>- Diffusers applied to Reinforcement Learning<br>- Embeddings as the latent vector space for prompting image generation</li></ul><p><strong>Articles in this series</strong></p><p><a href="https://medium.com/@ZoinerTejada/ai-foundation-models-55345ffb36d9">Foundation Models are Mind Blowing</a></p><p><a href="https://medium.com/@ZoinerTejada/ai-embeddings-3fb51516bc8b">Embeddings are Coordinates</a></p><p>Patterns in Prompt Engineering:</p><ul><li><a href="https://medium.com/@ZoinerTejada/ai-memory-recall-in-large-language-models-597a0f44fd35">Memory &amp; Recall in Large Language Models</a></li><li><a href="https://medium.com/@ZoinerTejada/ai-context-stuffing-pattern-in-prompt-engineering-fb26d0378c64">Context Stuffing and RAG Patterns</a></li></ul><p><a href="https://medium.com/@ZoinerTejada/ai-deep-reinforcement-learning-d317a4310d19">Deep Reinforcement Learning</a></p><p><a href="https://medium.com/@ZoinerTejada/ai-deep-reinforcement-stack-eca561360c39">Deep Reinforcement Learning Tool Stack</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=15b1712886a4" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>