<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Yumo Bai on Medium]]></title>
        <description><![CDATA[Stories by Yumo Bai on Medium]]></description>
        <link>https://medium.com/@yumo-bai?source=rss-5af938d560fd------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*iq1Hp9D5BC6-bMFA_4ONVA.jpeg</url>
            <title>Stories by Yumo Bai on Medium</title>
            <link>https://medium.com/@yumo-bai?source=rss-5af938d560fd------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 02:36:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@yumo-bai/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Why are most LLMs decoder-only?]]></title>
            <link>https://medium.com/@yumo-bai/why-are-most-llms-decoder-only-590c903e4789?source=rss-5af938d560fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/590c903e4789</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Yumo Bai]]></dc:creator>
            <pubDate>Sat, 03 Feb 2024 09:00:17 GMT</pubDate>
            <atom:updated>2024-02-09T19:32:33.171Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong>Dive into the rabbit hole of recent advancement in Large Language Models</strong></h4><p>I came across this question during mentoring for DeltaHacks and could not come up with an answer that was persuasive enough for myself. So I did some digging and it turned out to be a fascinating rabbit hole to dive down into. It involves a blend of understanding transformers, architectures, mathematics, and engineering optimizations. Here I will share the two cents that I picked up along the way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g_t91vv4F1u4LLots1Hx1w.png" /><figcaption>Source: <a href="https://openart.ai/discovery/sd-1008235131640741898">https://openart.ai/discovery/sd-1008235131640741898</a></figcaption></figure><h3>Overview of Language Model Architectures</h3><p>Let’s first familiarize ourselves with some architectural terms.</p><h4>Encoder and Decoder</h4><ul><li>Encoder: Processes and transforms input data into a condensed representation, capturing the essential information. In a translation task, an encoder takes an English sentence and converts it into a vector that represents its linguistic features and meaning.</li><li>Decoder: Takes the encoded representation and generates an output, often in a different form. In the same translation task, the decoder takes the encoded representation of a sentence in English and generates its equivalent in French.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yxR8e9WdZs54mp2LMeRA_w.gif" /><figcaption>Source: <a href="https://jalammar.github.io/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention/">Visualizing A Neural Machine Translation Model</a> (Great post by Jay Alammar)</figcaption></figure><h4>Encoder-Only Models</h4><ul><li>Example: BERT-based models</li><li>Pretraining Approach: Masked Language Modelling (MLM)</li><li>Use Case: Tasks that require a deep understanding of input data. These models are effective for classification, sentiment analysis, and information extraction.</li></ul><h4>Decoder-Only Models</h4><ul><li>Example: GPT, XLNet</li><li>Pretraining Approach: Next Token Prediction</li><li>Use Case: Generative tasks. They work by predicting subsequent text based on the provided context in an auto-regressive fashion. Their primary function is output generation without a separate encoding phase.</li></ul><h4>Encoder-Decoder Models</h4><ul><li>Example: T5, BART, Google Gemini (Probably)</li><li>Pretraining: Task-dependent</li><li>Use Case: Tasks that involve both understanding and generating data. They first encode an input sequence into an internal representation and then decode this representation into an output sequence.</li></ul><p>Comparing the purpose of these architectures, we can first easily exclude encoder-only models: They are typically pre-trained with MLM and do not necessarily help with generating output.</p><p>The decoder-only ones, on the other hand, make perfect sense: They are used to generate outputs and are pre-trained on Next Token Prediction tasks, which is exactly the task for most LLMs.</p><p>The question really boils down to Decoder-only versus Encoder-Decoder architecture:</p><blockquote>Having a decoder component and thereby generative ability, wouldn’t having the extra encoder components only help?</blockquote><h3>Causal Decoder (CD) vs Encoder-Decoder (ED)</h3><p>The performance of decoder-only, also referred to as causal decoder, against encoder-decoder models has long been studied. One of the earlier works is the paper <a href="https://proceedings.mlr.press/v162/wang22u/wang22u.pdf">What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization? by Wang et al</a>, published on ICML 2022. In this study, researchers compared various combinations of architecture and pretraining approaches. What they found is that:</p><blockquote>Our experiments show that causal decoder-only models trained on an autoregressive language modeling objective exhibit the strongest zero-shot generalization after purely self-supervised pretraining.</blockquote><blockquote>However, models with non-causal visibility on their input trained with a masked language modeling objective followed by multitask finetuning perform the best among our experiments.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PTnFnlgQJnYgEZ3uc8CR8Q.png" /><figcaption>Combinations studied in the ICML paper</figcaption></figure><p>Ok great, so encoder-decoder &gt; decoders-only &gt; encoder-only, right?</p><p>Well turns out that even though the aforementioned paper revealed some valuable insights into developing larger models. Some other factors need to be considered when choosing the architecture.</p><h4>Cost of Training</h4><p>To achieve the maximum potential of ED, we would need to perform multitask finetuning (which is basically instruction finetuning) on labeled data and it could be very expensive, especially for the larger models.</p><p>CD models on the other hand achieve great performance for their strong zero-shot generalization which works nicely with the current convention — self-supervised learning on a large-scale corpus.</p><h4>Emergent Ability</h4><p>The models compared in the paper have around 5B parameters and are trained over 170B tokens. It’s not large enough to account for something miraculous —<em> the emergent ability</em> of LLMs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ykTeLBhE8odluNFSXMFpQw.png" /><figcaption>Source: <a href="https://yaofu.notion.site/A-Closer-Look-at-Large-Language-Models-Emergent-Abilities-493876b55df5479d80686f68a1abd72f">A Closer Look at Large Language Models Emergent Abilities</a> by Yao Fu</figcaption></figure><p>Emergent abilities in Large Language Models (LLMs) refer to the phenomenon where models display new, sophisticated capabilities not explicitly taught during training, arising naturally as the model scales in size and complexity. Here is <a href="https://yaofu.notion.site/A-Closer-Look-at-Large-Language-Models-Emergent-Abilities-493876b55df5479d80686f68a1abd72f#60204039b0d14e6fac0b5c3a65b78d00">a great blog on emergent abilities by Yao Fu</a> where you can learn more about it.</p><p>Essentially, emergent abilities enable LLM to perform some degree of complex reasoning. For example, extracting structured knowledge from unstructured text. This ability allows LLM to understand some NLP tasks that are naturally underlying in the text corpus that it was trained on. For simpler tasks, we can think of the LLM with emergent abilities that have already been finetuned during training, and for more complex tasks, it can break them down into simpler tasks. The emergent abilities do not necessarily bless decoder-only models more than ED ones, but they reduce the performance gap achieved by ED models over decoder-only ones with multitask finetuning.</p><h4>In-Context Learning from Prompts</h4><p>Another thing to take into consideration is prompting. When using LLM, we could apply prompt engineering methods like providing few-shot examples to help LLM understand the context or task. In <a href="https://arxiv.org/pdf/2212.10559.pdf">this paper</a> by Dai et al. Researchers mathematically proved that such in-context information can be seen to have a similar effect as gradient descent that updates the attention weight of the zero-shot prompt.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/1*jAoqMCbFsKBGhImFXK1CHg.png" /><figcaption>In-context learning adds to the attention weight of a zero-shot learning input (normal prompt)</figcaption></figure><p>If we perceive prompting as introducing a gradient to the attention weight, we can probably expect it to have a more straightforward effect for the decoder-only models as it does not need to be translated into an intermediate context first before being used for generative tasks. Logically, it should still work for encoder-decoder architectures, but it requires the encoder to be carefully tuned to an optimal performance which might be difficult.</p><h4><strong>Efficiency Optimization</strong></h4><p>in decoder-only models, the Key (K) and Value (V) matrices from previous tokens can be reused for subsequent tokens during the decoding process. Since each position only attends to previous tokens (due to the causal attention mechanism), the K and V matrices for these tokens remain unchanged. This caching mechanism improves efficiency by avoiding the recomputation of K and V matrices for tokens that have already been processed, facilitating faster generation and lower computational costs during inference in autoregressive models like GPT.</p><h4>Autoregressive vs Bidirectional Attention</h4><p>There is another interesting point that is raised regarding the difference in the underlying attention mechanisms, that is, autoregressive for Decoder-only (Causal Decoder) and Bidirectional for Encoder-Decoder. We can visualize how they attend different positions below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/1*QVrJCqy1-nCr6JhvHwTY1g.png" /><figcaption>Positions attend differently in these architectures.</figcaption></figure><p>The way the attention matrix is calculated in Transformer Architecture is by multiplying two lower-dimensional matrices (Q and K^T) and then applying softmax operation. In decoder-only architectures, the attention matrix is constrained to a lower triangular form due to causal masking (to prevent the model from seeing future tokens), which theoretically maintains its full rank status: Each element in the diagonal (representing self-attention) contributes to making the determinant positive (You only get a positive result from softmax). The full rank status suggests a theoretically stronger expressive capability.</p><p>The other two generative architectures both introduce bidirectional attention and thereby do not guarantee the full rank status of their attention matrix. The author suggests that this will limit the performance of the model. He set up an experiment to verify this assumption by splitting the bidirectional attention matrix into one-directional, with half the attention heads attending forward and the other half backward. Then he compared the performance of such Forward-Backward attention to the Full-bidirectional attention model. The FB attention did perform better which kind of verified this theory, but the improvement was quite marginal and did not seem to suggest a significant difference especially when the models are sufficiently trained.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*DR6nKPVPhXzxxsEfCpYZuQ.png" /><figcaption>Performance of FB Attention compared to Full Bi-Directional</figcaption></figure><p>It intuitively makes sense. Bidirectional attention works as a double-edged sword: It speeds up the learning process but also kind of “spoils” the model from learning the deeper predictive patterns essential for generation. You can kind of think of it as learning how to write: filling in blanks is easier compared to writing an entire article word by word, but it would be a less effective way of practicing. However, after enormous amount of training, both approaches achieve the objective of learning how to write.</p><h3>What did I learn</h3><p>The popularity of decoder-only architecture comes from its simplicity, good zero-shot generalization, and cheaper training cost to attain a reasonable performance. Many works have been done studying the performance of decoder-only and encoder-decoder architectures, but given there is sufficient training and model size, there really is no hard evidence that proves one architecture is superior to another in terms of final performance.</p><p>In fact, Google Gemini showed just how the encoder-decoder model can work just as well and even exceed decoder-only architectures in some tasks. The encoder component supports the “build-in multimodality” by enabling extracting information from non-textual inputs which might be crucial for the future generation of LLMs. Our initial question truly should be — why WERE most LLMs decoder-only — it showed an era where everyone had mainly worked on advancing decoder-only architectures. Nonetheless, I think it still reveals a great amount of insights into understanding the inner mechanism of how LLMs work and the history of their advancement. It’s exciting to see what will come next in the search for AGI.</p><h3>Reference</h3><p>Dai, D., Sun, Y., Dong, L., Hao, Y., Sui, Z., &amp; Wei, F. (2022). Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta-Optimizers. Retrieved from <a href="https://arxiv.org/abs/2212.10559">https://arxiv.org/abs/2212.10559</a></p><p>The BigScience Architecture &amp; Scaling Group. (2022). What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization? Retrieved from <a href="https://arxiv.org/abs/2204.05832">https://arxiv.org/abs/2204.05832</a></p><p>Fu, Y., Khot, T., &amp; Peng, H. (2022, November 20). A Closer Look at Large Language Models Emergent Abilities. Retrieved from <a href="https://yaofu.notion.site/A-Closer-Look-at-Large-Language-Models-Emergent-Abilities-493876b55df5479d80686f68a1abd72f">https://yaofu.notion.site/A-Closer-Look-at-Large-Language-Models-Emergent-Abilities-493876b55df5479d80686f68a1abd72f</a></p><p>Wei et al. (2022). Emergent Abilities of Large Language Models. Retrieved from <a href="https://arxiv.org/abs/2206.07682">https://arxiv.org/abs/2206.07682</a></p><p>Dong, Y., Cordonnier, J.-B., &amp; Loukas, A. (2021). Attention is not all you need: pure attention loses rank doubly exponentially with depth. Retrieved from <a href="https://arxiv.org/abs/2103.03404">https://arxiv.org/abs/2103.03404</a></p><p>苏剑林. (Mar. 17, 2023). 《为什么现在的LLM都是Decoder-only的架构？ 》[Blog post]. Retrieved from <a href="https://kexue.fm/archives/9529">https://kexue.fm/archives/9529</a></p><p><em>Alammar, J (2018). The Illustrated Transformer [Blog post]. Retrieved from </em><a href="https://jalammar.github.io/illustrated-transformer/"><em>https://jalammar.github.io/illustrated-transformer/</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=590c903e4789" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Era of Artificial...Intelligence?]]></title>
            <link>https://medium.com/@yumo-bai/the-era-of-artificial-intelligence-6188bc09a14c?source=rss-5af938d560fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/6188bc09a14c</guid>
            <category><![CDATA[gpt-4-turbo]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Yumo Bai]]></dc:creator>
            <pubDate>Thu, 30 Nov 2023 05:56:43 GMT</pubDate>
            <atom:updated>2024-02-03T08:48:46.587Z</atom:updated>
            <content:encoded><![CDATA[<h4>Exploring the Optimality Gap of LLM-based Agents</h4><h3>Introduction</h3><p>In an era marked by rapid advancements in Artificial Intelligence, the release of OpenAI’s GPT-4-turbo has set a new benchmark in the capabilities of Large Language Models (LLMs). This blog post aims to explore the intersection of LLMs and the realm of Reinforcement Learning (RL), focusing on their application in the Gridworld problem — a microcosm for AI decision-making processes.</p><p>We delve into evaluating the “optimality gap” of LLM-based agents, assessing their performance against theoretical perfection in an RL setting. Through the application of Q-Learning and innovative tools like Langchain and ReAct, this study sheds light on the capabilities and challenges of these agents in complex environments.</p><p>Our exploration is not just a technical exercise but also a contribution to the ongoing dialogue about the future trajectory of AI. Amidst the backdrop of innovation and concern, our exploration into the efficacy of LLM-based agents in Reinforcement Learning (RL) is not only timely but also crucial.</p><h4><strong>Objectives</strong></h4><ol><li><strong>Understanding the Gridworld</strong>: We first familiarize ourselves with the Gridworld environment, its rules, and dynamics. We also explore the optimal policy with Q-Learning.</li><li><strong>GPT Chat Completion</strong>: Utilizing LLM, particularly GPT-4, we explore how language models can aid in decision-making processes and compare different prompt engineering methods.</li><li><strong>Langchain ReAct Agent</strong>: We integrate Langchain, a tool for combining LLMs with decision-making processes, to build an agent with enhanced problem-solving capabilities.</li></ol><p>Note: All codes and resources for this project are available <a href="https://github.com/XO-Appleton/RL-LLM-Optimality-Research/tree/main">here</a>.</p><h3>Understanding the Gridworld</h3><p>The Taxi Gridworld is a classic problem in RL. In the Taxi environment, the world is represented as a grid, where each cell can be occupied by the taxi, a passenger, a destination, or be empty. The taxi’s objective is straightforward: to pick up a passenger from a designated location and drop them off at their desired destination. The challenge lies in efficiently planning routes, avoiding obstacles, and making strategic decisions under constraints.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/550/0*csmKmtD-A4MqBe-S.gif" /><figcaption>Taxi Gridworld. Source: <a href="https://gymnasium.farama.org/environments/toy_text/taxi/">OpenAI Gym document</a></figcaption></figure><p>Let’s set up our own Taxi environment using OpenAI’s gym :</p><pre>env = gym.make(&#39;Taxi-v3&#39;, new_step_api=True, render_mode=&#39;rgb_array&#39;)<br>env.reset(seed=42)<br>init_frame = env.render()[0]<br><br>fig, ax = plt.subplots(figsize=(4, 2.5))<br>ax.axis(&#39;off&#39;)<br>plt.imshow(init_frame)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/407/1*8h_XiO1_-uOqsuC9ynSjPQ.png" /><figcaption>rgb_array (human-readable) rendering of our initial states</figcaption></figure><p>As we can see from the initial rendering of the environment. The taxi is currently sitting on the 4th row and last reachable cell (4, 5). The passenger is waiting on the (G)reen cell and the Destination is represented by the building on the (Y)ellow cell.</p><p>There are six actions in the Action Space:</p><ul><li>0: Move south (down)</li><li>1: Move north (up)</li><li>2: Move east (right)</li><li>3: Move west (left)</li><li>4: Pick up passenger</li><li>5: Drop off passenger</li></ul><p>Rewards are calculated as follows:</p><ul><li>-1 per step unless other reward is triggered.</li><li>+20 delivering the passenger.</li><li>-10 executing “pickup” and “drop-off” actions illegally.</li></ul><p>To help us get an intuition of how the objects interact with each other, let’s visualize it with a simple random walk episode. We will perform the episode, record the frames, and display them on a matplotlib subplot.</p><pre>frames = [init_frame]  # List to hold the frames<br><br># Perform an episode and record the frames<br>done = False<br>truncated = False<br>while not done and not truncated:<br>    action = env.action_space.sample()  # Take a random action<br>    state, reward, done, truncated, info = env.step(action)<br>    frame = env.render()<br>    frames.append(frame[0])  # Save the frame<br><br>env.close()  # Close the environment<br><br># Helper function to generate a mp4 animation of the episode.<br>def update(frame):<br>    plt.imshow(frame)<br>    plt.axis(&#39;off&#39;)<br><br>fig, ax = plt.subplots(figsize=(4, 2.5))<br>ax.axis(&#39;off&#39;)  # Hide the axis<br><br># Generate the animation<br>ani = FuncAnimation(fig, update, frames=frames, blit=False, repeat=False)<br>ani.save(&#39;random_walk_episode.gif&#39;, writer=&#39;imagemagick&#39;, fps=4)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/388/1*9f4x5oCYpvtzicmx9GJ5Rw.gif" /><figcaption>A random walk episode in our Taxi environment</figcaption></figure><p>We see that the taxi can pick up and drop off the passenger on one of the colored grids. The taxi can potentially run into walls or pick up/drop off the passenger outside of colored grids. They do not change the state of the environment but punish the agent by giving it a negative reward. Let’s now improve our RL agent by optimizing with a classic yet powerful RL algorithm called Q-learning.</p><h4>Optimize with Q-Learning</h4><p>Q-learning is a type of Reinforcement Learning (RL) algorithm. In RL, you have an agent (in our case, the taxi) who learns how to make decisions (like where to move) by interacting with an environment (the grid world). The goal is for the agent to learn the best actions to take in different situations to achieve its objective, which is usually defined by a reward system and can be quantified by what we call the Q-value, which gets updated by adding a portion of the expected reward of performing an action.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/614/1*qdNBhfcJBRM0vTWYdzsHOw.png" /></figure><h4>Key Points in Simple Terms</h4><ul><li>The taxi tries different things and learns from what happens.</li><li>It uses a special lookup table (Q-table) to keep track of how good each action is in each situation.</li><li>The taxi updates this table based on its experiences.</li><li>Over time, the taxi gets better at knowing the best actions to take.</li><li>Eventually, the taxi learns the most efficient routes and actions to complete its task.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*yY5fS8P2F3oTegWd7sLW6Q.png" /><figcaption>Example of Q-Table. Source: <a href="https://en.wikipedia.org/wiki/Q-learning">Wikipedia</a></figcaption></figure><h4>Behavior Exploration (Training)</h4><p>We first initialize a Q-Table and train it as described above. In my case, I trained it for 100000 episodes, and it took nearly 5 hours. (The trained Q-table is available <a href="https://github.com/XO-Appleton/RL-LLM-Optimality-Research/blob/main/src/q_table.npy">here</a> so don’t worry!)</p><pre># This algorithm is a modification of the https://www.learndatasci.com/tutorials/reinforcement-q-learning-scratch-python-openai-gym/<br>q_table = np.zeros([env.observation_space.n, env.action_space.n])<br><br>import random<br><br>def q_learning(env, q_table, num_episodes=100000, alpha=0.1, gamma=0.6, epsilon=0.1):<br><br>  # For plotting metrics<br>  episodes = []<br>  all_epochs = []<br>  all_penalties = []<br><br>  for i in range(1, num_episodes+1):<br>    state = env.reset()<br>    epochs, penalties, reward, = 0, 0, 0<br>    done = False<br><br>    while not done:<br>      if random.uniform(0, 1) &lt; epsilon:<br>          action = env.action_space.sample() # Explore action space<br>      else:<br>          action = np.argmax(q_table[state]) # Exploit learned values<br>      next_state, reward, done, truncated, info = env.step(action)<br><br>      old_value = q_table[state, action]<br>      next_max = np.max(q_table[next_state])<br><br>      new_value = (1 - alpha) * old_value + alpha * (reward + gamma * next_max)<br>      q_table[state, action] = new_value<br><br>      if reward == -10:<br>          penalties += 1 # Executing “pickup” and “drop-off” actions illegally.<br><br>      state = next_state<br>      epochs += 1<br><br>    if i % 100 == 0:<br>        print(f&quot;Episode: {i}&quot;)<br>        clear_output(wait=True)<br>        episodes.append(i)<br>        all_epochs.append(epochs)<br>        all_penalties.append(penalties)<br><br>  print(&quot;Training finished.\n&quot;)<br><br>  return episodes, all_epochs, all_penalties</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/575/1*dycX-v09J-J-58lUQ8AsNQ.png" /><figcaption>Number of steps taken and penalty received over the episodes</figcaption></figure><p>In the first couple of episodes, our agent took around 350 steps to complete the objective. The number quickly dropped down to around 20-ish. The penalties and steps taken fluctuate but never diminish as we set 10% of the steps to be random (ϵ = 0.1).</p><p>Let’s see how the agent performs with the optimized Q-table. We will also record the steps taken and rewards received to compare with the other agents later.</p><pre>PERFORMANCE = {}<br><br>state = env.reset(seed=42)<br>init_frame = env.render()[0]<br>frames = [init_frame]  # List to hold the frames<br>total_reward = 0<br><br>done = False<br>truncated = False<br><br>while not done and not truncated:<br>    action = np.argmax(q_table[state])  # Take the best action<br>    state, reward, done, truncated, info = env.step(action)<br>    frame = env.render()<br>    frames.append(frame[0])  # Save the frame<br>    total_reward += reward<br><br>env.close()  # Don&#39;t forget to close the environment<br><br>PERFORMANCE[&#39;optimal_episode&#39;] = (len(frames)-1, total_reward, done) # (steps_taken, rewards_received_from_episode, object_achieved)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/388/1*SgqqG81ohi-4XiGO9J3K6g.gif" /><figcaption>Optimal Episode after Q-Learning</figcaption></figure><p>The RL agent completes a clean episode in one of the most efficient ways.</p><p>Now let’s get into the fun part and see how would our groundbreaking GPT models perform in the driver&#39;s seat.</p><h3>GPT Chat Completion</h3><p>In this section, we will ask the GPT models to analyze the textual description of the environment and come up with a solution. We will compare different prompting methods as well as textual encodings of the positions.</p><pre># Replace with your own OpenAI API key<br>os.environ[&#39;OPENAI_API_KEY&#39;] = &#39;YOUR_KEY_HERE&#39;<br><br>gpt_3 = &quot;gpt-3.5-turbo-0613&quot;<br>gpt_4 = &quot;gpt-4-1106-preview&quot;<br><br># We access the GPT models through OpenAI API<br>@retry(wait=wait_random_exponential(multiplier=1, max=40), stop=stop_after_attempt(3))<br>def chat_completion_request(messages, tools=None, tool_choice=None, model=gpt_3):<br>    headers = {<br>        &quot;Content-Type&quot;: &quot;application/json&quot;,<br>        &quot;Authorization&quot;: &quot;Bearer &quot; + os.environ[&#39;OPENAI_API_KEY&#39;],<br>    }<br>    json_data = {&quot;model&quot;: model, &quot;messages&quot;: messages}<br>    if tools is not None:<br>        json_data.update({&quot;tools&quot;: tools})<br>    if tool_choice is not None:<br>        json_data.update({&quot;tool_choice&quot;: tool_choice})<br>    try:<br>        response = requests.post(<br>            &quot;https://api.openai.com/v1/chat/completions&quot;,<br>            headers=headers,<br>            json=json_data,<br>        )<br>        return response<br>    except Exception as e:<br>        print(&quot;Unable to generate ChatCompletion response&quot;)<br>        print(f&quot;Exception: {e}&quot;)<br>        return e</pre><p>To help GPT understand the problem. We will need to write a textual description of the problem that we’re dealing with. I used plain text that is similar to the ANSI of the environment rendering to represent the Gridworld then briefly talked about the objective and rules that we covered in the first section.</p><pre>problem_desc = &quot;&quot;&quot;You will be controlling a taxi to pick up a passenger and drop him off at the destination location in a gridworld constructed as follows:<br>+---------+<br>|R: | : :G|<br>| : | : : |<br>| : : : : |<br>| | : | : |<br>|Y| : |B: |<br>+---------+<br><br>Objective: Plan the taxi&#39;s movements ahead and carefully to avoid the walls. Pickup the passenger and drop him off at the destination.<br>Look ahead for walls that could potentially run into. Plan the route carefully to avoid them. Only make the action after you have a good confidence about the route you have planned.<br><br> The task is going to be performed in a gridworld with specifications as follows:<br>- The gridworld is constructed with the following symbols: [&quot;|&quot;, &quot;-&quot;, &quot;:&quot;, &quot; &quot;, &quot;R&quot;, &quot;G&quot;, &quot;Y&quot;, &quot;B&quot;]:<br>  --`|` and `-` represent a wall that the taxi cannot pass through, East-West or North-South, respectively.<br>  -- `:` represent a boundary that a taxi can pass through.<br>  -- `R, G, Y, B` represents colored regions that a passenger can be on, the passenger will start on one of them and the destination will be one of them as well.<br>  -- ` ` represent space that the taxi can park at, the taxi can also go onto the colored regions.<br>- There are six actions in the action space [&quot;South&quot;,&quot;North&quot;,&quot;East&quot;,&quot;West&quot;, &quot;Pickup&quot;, &#39;Dropoff&#39;], first four moves the taxi for one unit to the corresponding directions (down, up, right, left), last two picks up and drops off the passenger.<br>  -- The passenger can only be picked up or dropped off at one of the colored regions.<br>  -- The taxi&#39;s position will be represented as (row, col). `row` and `col` are 1-indexed.<br>  -- Attempting moving the taxi onto or jumping over a wall would not change its position.<br>  -- &#39;Pickup&#39; and &#39;Dropoff&#39; should only be performed when taxi&#39;s position equals to the desired position<br>- You can only choose one of the six actions listed above.<br>&quot;&quot;&quot;</pre><h4>Vague Description</h4><p>We will start simple with a prompt that is vague and only descriptive about the positions like we would ask a human to perform the task.</p><pre>vague_prompt = &quot;&quot;&quot;The taxi starts at the cell to the north-east of B grid. The passenger is at G, the destination is at Y. Show the sequence of actions that would achieve the objective.<br>When you have an final answer, use the format:<br><br>Final Answer: [Actions Taken]&quot;&quot;&quot;<br><br>def get_answer(problem_desc, prompt, model):<br>    messages = []<br><br>    # message log to pass to GPT<br>    messages.append({&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: problem_desc})<br>    messages.append({&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt})<br>    chat_response = chat_completion_request(<br>        messages, model=model<br>    )<br><br>    # Textual Response from GPT<br>    assistant_message = chat_response.json()[&quot;choices&quot;][0][&quot;message&quot;]<br>    messages.append(assistant_message)<br><br>    return assistant_message[&#39;content&#39;]<br><br>response = get_answer(problem_desc, vague_prompt, gpt_3)<br>print(response)</pre><pre>To solve this task, we can follow the following sequence of actions:<br>1.Move South to reach row 4, col 1 (the start point of B grid). <br>2.Move West to reach row 4, col 8 (the end point of B grid). <br>3.Move North to reach row 1, col 8 (the end point of Y grid). <br>4.Move East to reach row 1, col 7 (the destination G grid). <br>5.Perform Dropoff at row 1, col 7 (the destination G grid). <br>6.Move West to reach row 1, col 4 (the start point of G grid). <br>7.Move South to reach row 4, col 4 (the start point of B grid). <br>8.Perform Pickup at row 4, col 4. <br>9.Move West to reach row 4, col 1 (the start point of B grid).<br>10.Move North to reach row 1, col 1. <br>11.Perform Dropoff at row 1, col 1 (the destination Y grid). <br><br>Final Answer: [South, West, North, East, Dropoff, West, South, Pickup, West, North, Dropoff]</pre><p>Let’s define a function to help us visualize how the answer plays out. It parses responses from GPT and records them like we did in section 1.</p><pre>ACTION_MAP =  {&#39;South&#39;:0, &#39;North&#39;:1, &#39;East&#39;:2, &#39;West&#39;:3, &#39;Pickup&#39;:4, &#39;Dropoff&#39;:5}<br><br>def response_to_actions(response, output_filename=&quot;episode.gif&quot;):<br>    actions = response.split(&#39;Final Answer:&#39;)[-1].strip(&#39; []&#39;).split(&#39;, &#39;)<br>    actions = [a.strip(&#39;&quot;\&#39;&#39;) for a in actions]<br><br>    figure, ax = plt.subplots(figsize=(4, 2.5))<br>    ax.axis(&#39;off&#39;)  # Hide the axis<br><br>    env = gym.make(&#39;Taxi-v3&#39;, new_step_api=True, render_mode=&#39;rgb_array&#39;)<br>    env.reset(seed=42)<br>    init_frame = env.render()[0]<br>    frames = [init_frame]  # List to hold the frames<br><br>    total_reward = 0 # Calculate total reward to compare performance.<br><br>    for action in actions:<br>        state, reward, done, truncated, info = env.step(ACTION_MAP[action])<br>        frame = env.render()<br>        frames.append(frame[0])  # Save the frame<br>        plt.imshow(frame[0])<br><br>        total_reward += reward<br><br>    PERFORMANCE[output_filename[:-4]] = (len(frames)-1, total_reward, done)<br>    env.close()  # Don&#39;t forget to close the environment<br><br>    ani = FuncAnimation(figure, update, frames=frames, blit=False, repeat=False)<br>    ani.save(output_filename, writer=&#39;Pillow&#39;, fps=2)<br><br>    # Close the figure window<br>    plt.close()<br><br># Call the function with the response<br>response_to_actions(response, output_filename=&#39;gpt_3_vague.gif&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/388/1*EJgcrBZ8NxiubxLXKk2n5w.gif" /><figcaption>GPT-3.5 with vague prompt</figcaption></figure><p>Our GPT-3.5 model seems to have no idea about the spatial relationship between different objects in GridWorld. It also seems to be confused about which direction the taxi would be moved to.</p><p>Let’s see how the newly released GPT-4 turbo model would do.</p><pre>response = get_answer(problem_desc, vague_prompt, gpt_4)<br>print(response)</pre><pre>To pick up the passenger at G and drop them off at Y, the taxi should execute the following series of actions: <br>Move South to go down. <br>Move South again to the square directly above G. <br>Move East to enter G, where the passenger is located. <br>Perform the Pickup action to pick up the passenger. <br>Move West to exit the G grid. <br>Move North twice to move to the square directly to the right of R. <br>Move West to go through the colon. <br>Move West again to enter the R grid. <br>Move South to go through the colon and move to the square directly above Y. <br>Move South to enter Y. <br>Perform the Dropoff action to drop off the passenger. <br><br>Combining these actions gives us the following sequence: Final Answer: [South, South, East, Pickup, West, North, North, West, West, South, South, Dropoff]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/388/1*EPrb-LdVS_DGvvZ-jPgvUw.gif" /><figcaption>GPT-4-turbo with vague prompt</figcaption></figure><p>I ran the prompt a couple of times during the development stage. In most cases, the GPT-4 model can provide a somewhat reasonable plan before giving the final answer, but it is still not able to capture the location information from our textual description.</p><p>Very interestingly, the GPT-4 model occasionally tries to encode the location of the taxi to form (row, col) from the raw text in the prompt before trying to make a plan. It usually achieves a much better result in those cases.</p><p>We probably should incorporate the location encoding as part of our prompt for consistent performance.</p><h4>Location Encoding</h4><p>An easy way to encode location information is to simply express it as a tuple of (row, col) given the simple structure of our Gridworld. For simplicity, we will treat each cell as a column and let GPT try to figure out the position of the walls.</p><pre>location_encoding = &quot;&quot;&quot;The location of the taxi, passenger, and destination will be represented as tuples (row, col).<br>The row and col are 1-indexed. The wall `|` between (row, col) and (row, col+1) prevents taxi from travelling from (row, col) to (row, col+1) and vice versa.<br>The Actions `South`, `North`, `East`, `West` changes the location of the taxi to (row+1, col), (row-1, col), (row, col+1), (row, col-1), respectively. `Pickup` and `Dropoff` does not change the location of the taxi.<br>The taxi starts at the (4, 5). The passenger is at (1, 5), the destination is at (5, 1).<br>&quot;&quot;&quot;<br><br>basic_prompt = &quot;&quot;&quot;<br>Show the sequence of actions that would achieve the objective. i.e location_passenger == location_destination<br><br>When you have an final answer, you MUST use this format:<br>Final Answer: [Actions Taken]&quot;&quot;&quot;<br><br>location_encoded_prompt = location_encoding + basic_prompt</pre><p>And the results that we get:</p><p>GPT-3.5:</p><pre>To pick up the passenger and drop him off at the destination, the following actions can be taken: <br>Move North (from (4, 5) to (3, 5)) <br>Move West (from (3, 5) to (3, 4)) <br>Move North (from (3, 4) to (2, 4)) <br>Move West (from (2, 4) to (2, 3)) <br>Move North (from (2, 3) to (1, 3)) <br>Pickup passenger (at (1, 3)) <br>Move South (from (1, 3) to (2, 3)) <br>Move East (from (2, 3) to (2, 4)) <br>Move North (from (2, 4) to (1, 4)) <br>Move West (from (1, 4) to (1, 3)) <br>Move North (from (1, 3) to (5, 1)) <br>Drop off passenger (at (5, 1)) <br><br>Final Answer: [North, West, North, West, North, Pickup, South, East, North, West, North, Dropoff]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/388/1*BGpvKTuQdHKF8lqtJX2haQ.gif" /><figcaption>GPT-3.5 with naive location encoding</figcaption></figure><p>GPT-4-turbo:</p><pre>First, we need to plan the route from the taxi&#39;s starting position to the passenger&#39;s location while avoiding walls. Then we need to pick up the passenger and plan the route from the passenger&#39;s location to the destination. Let&#39;s begin. The starting position of the taxi is (4, 5). <br>Move the taxi North (up) to (3, 5): Action &quot;North&quot; <br>Move the taxi North (up) to (2, 5): Action &quot;North&quot; <br>Move the taxi North (up) to (1, 5): Action &quot;North&quot; <br>Perform the &quot;Pickup&quot; action at the passenger&#39;s location. <br>Now the passenger is inside the taxi, we should move South (down) to (2, 5): Action &quot;South&quot; <br>Move South (down) to (3, 5): Action &quot;South&quot; <br>Move South (down) to (4, 5): Action &quot;South&quot; <br>Move South (down) to (5, 5): Action &quot;South&quot; <br>Move West (left) to (5, 4): Action &quot;West&quot; <br>Move West (left) to (5, 3): Action &quot;West&quot; <br>Move West (left) to (5, 2): Action &quot;West&quot; <br>Perform the &quot;Dropoff&quot; action at the destination. <br><br>Final Answer: [&quot;North&quot;, &quot;North&quot;, &quot;North&quot;, &quot;Pickup&quot;, &quot;South&quot;, &quot;South&quot;, &quot;South&quot;, &quot;South&quot;, &quot;West&quot;, &quot;West&quot;, &quot;West&quot;, &quot;Dropoff&quot;]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/388/1*xK_bClmHPCXKmIF6D03qaw.gif" /><figcaption>GPT-4-turbo with naive location encoding</figcaption></figure><p>Compared to wasting time getting to the B grid first, the GPT-4 model finds a straightforward approach to complete the objective with the location encodings. However, it neglects the walls when planning its route and gets stuck at (5, 4).</p><h4>CoT and ToT prompting</h4><p>Another common way of improving the performance of the LLM models is by using special promptings like CoT and ToT.</p><p>Chain of Thought (CoT) is a technique used in problem-solving. CoT involves the AI model writing out its reasoning in a step-by-step manner before arriving at the final answer. This method helps the AI to tackle complex problems more effectively and also makes its reasoning process transparent and easier to understand for humans.</p><p>For example, if you ask an AI to solve a math word problem, instead of just giving the final number, it would first explain each step of how it got there, like identifying key information, doing calculations step by step, and then combining these to reach the final answer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*5NbLMdHd2EG71dbKZYO0qg.png" /><figcaption>Chain of Thought. Source: <a href="https://arxiv.org/abs/2201.11903">Wei et al. (2022)</a></figcaption></figure><p>In our case, we will use what we call Zero-Shot CoT, which is done simply by adding this magical sentence Let&#39;s think step by step. where it fits.</p><pre>cot = &quot;&quot;&quot;<br>Show the sequence of actions that would achieve the objective. i.e location_passenger == location_destination<br><br>When you have an final answer, you MUST use this format:<br>Final Answer: [Actions Taken]<br><br>Let&#39;s think step by step.<br><br>Begin!<br>&quot;&quot;&quot;<br><br>cot_prompt = location_encoding + cot</pre><p>GPT-3.5 with CoT:</p><p><em>Final Answer: [‘South’, ‘South’, ‘South’, ‘South’, ‘South’, ‘West’, ‘West’, ‘West’, ‘West’, ‘Pickup’, ‘North’, ‘North’, ‘North’, ‘North’, ‘West’, ‘West’, ‘South’, ‘South’, ‘South’, ‘South’, ‘East’, ‘East’, ‘East’, ‘East’, ‘South’, ‘South’, ‘South’, ‘South’, ‘South’, ‘South’, ‘West’, ‘West’, ‘West’, ‘West’, ‘West’, ‘Pickup’, ‘North’, ‘North’, ‘North’, ‘North’, ‘West’, ‘West’, ‘West’, ‘West’, ‘Dropoff’]</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/388/1*_Tmvuel3_FWbzz_scAJT8w.gif" /><figcaption>GPT-3.5 with CoT and location encoding</figcaption></figure><p>GPT-4-turbo with CoT:</p><pre>First, as a taxi, we need to navigate to the passenger&#39;s location at (1, 5). We will begin at (4, 5) and follow the path. Here is a step-by-step breakdown which avoids the walls:<br><br>- The taxi starts at (4, 5)<br>- Move North to (3, 5)<br>- Move North to (2, 5)<br>- Move North to (1, 5)<br>- Perform a Pickup<br>- Move South to (2, 5)<br>- Move South to (3, 5)<br>- Move West to (3, 4)<br>- Move South to (4, 4)<br>- Move South to (5, 4)<br>- Move West to (5, 3)<br>- Move North to (4, 3)<br>- Move West to (4, 2)<br>- Move West to (4, 1)<br>- Move North to (3, 1)<br>- Move North to (2, 1)<br>- Move North to (1, 1)<br>- Move North to (1, 1)<br>- Move East to (1, 2)<br>- Move East to (1, 3)<br>- Move South to (2, 3)<br>- Move South to (3, 3)<br>- Move South to (4, 3)<br>- Move South to (5, 3)<br>- Move West to (5, 2)<br>- Move West to (5, 1)<br>- Perform a Dropoff<br><br>Now, let&#39;s summarize these steps into the final answer format with the actions that are taken:<br><br>Final Answer: [North, North, North, Pickup, South, South, West, South, South, West, North, West, West, North, North, North, East, East, South, South, South, West, West, Dropoff]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/388/1*N7T52hi5B60-_sZVXBjEcg.gif" /><figcaption>GPT-4-turbo with CoT and location encoding</figcaption></figure><p>With the Zero-shot CoT prompting method, our model starts to wander a lot inefficiently. This could be either due to hallucination or misinterpretation of the task.</p><p>Hallucination is tricky to solve and might have to do with the model’s fundamental performance. Let’s check how the models perform with ToT.</p><p>Tree of Thoughts (ToT) is a framework that generalizes over chain-of-thought prompting and encourages exploration of thoughts that serve as intermediate steps for general problem-solving with language models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OiS-b1YQx7eoDt8m9aFvYA.png" /><figcaption>Tree of Thought. Image Source: <a href="https://arxiv.org/abs/2305.10601">Yao et el. (2023)</a></figcaption></figure><pre>tot = &quot;&quot;&quot;<br>Imagine five different experts are working on this task.<br>All experts will write down 1 step of their thinking, then share it with the group.<br>Then all experts will go on to the next step, etc.<br>If any expert realises they&#39;re wrong at any point then they leave.<br><br>When the experts have found the sequence of actions that achieves the objective, i.e location_passenger == location_destination, you MUST use this format:<br>Final Answer: [The list of action steps taken.]<br><br>Begin!<br>&quot;&quot;&quot;<br><br>tot_prompt = location_encoding + tot</pre><p>GPT-3.5:</p><pre>Expert 1: My initial thought is to move the taxi to the passenger&#39;s location. So the first step is to move North to (3, 5).<br><br>Expert 2: I agree with Expert 1&#39;s first step. Next, let&#39;s move East to (3, 6) to get closer to the passenger.<br><br>Expert 3: I see that we need to pick up the passenger at (1, 5), so the next step should be &#39;Pickup&#39;.<br><br>Expert 4: After picking up the passenger, we need to move to the destination at (5, 1). To do that, let&#39;s move East to (3, 7).<br><br>Expert 5: I think we should move South to (4, 7) to avoid the wall.<br><br>Expert 1: Great suggestion. Now, let&#39;s move East to (4, 8) and then move South to (5, 8) to get closer to the destination.<br><br>Expert 2: I agree. To reach the destination at (5, 1), we need to move West to (5, 7).<br><br>Expert 3: After that, we need to move South to (6, 7) to avoid the wall.<br><br>Expert 4: Now, let&#39;s move West to (6, 6) to come closer to the destination.<br><br>Expert 1: Finally, we can move South to (7, 6) and then move West to (7, 5) to reach the destination.<br><br>Final Answer: [&#39;North&#39;, &#39;East&#39;, &#39;Pickup&#39;, &#39;East&#39;, &#39;South&#39;, &#39;East&#39;, &#39;South&#39;, &#39;West&#39;, &#39;South&#39;, &#39;West&#39;]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/388/1*h6uQNCI1-5SaB8tM--Ix8g.gif" /><figcaption>GPT-3.5 with ToT prompt and location encoding</figcaption></figure><p>GPT-4-turbo:</p><pre>Step 1:<br>- Expert 1: We start at (4, 5) and need to pick up the passenger at (1, 5). We should go north as our first move to approach the passenger.<br>- Expert 2: Given that walls block movement east and west directly above and below the taxi&#39;s start position, I agree that the first move should be north.<br>- Expert 3: I concur with Expert 1 and Expert 2 that the first optimal move is to head north to avoid the walls and make our way towards the passenger.<br>- Expert 4: Starting at (4, 5), heading north makes sense because it&#39;s the only option to move towards the passenger without being blocked by walls.<br>- Expert 5: The taxi needs to head towards the passenger located at (1, 5), so moving north is necessary; there are no other legal moves to start with.<br><br>Step 2:<br>- Expert 1: The taxi has reached (3, 5) after the first move. The next action should be moving north again to go closer to the passenger.<br>- Expert 2: I&#39;m realizing that my initial thoughts didn&#39;t account for the wall between rows 3 and 4. I must leave.<br>- Expert 3: The wall configurations must be considered, but since we are heading towards the passenger, I maintain that the next move is north.<br>- Expert 4: Looking at the grid, the taxi can move north, avoiding the east-west wall that is one row above, so we continue north.<br>- Expert 5: Having reached (3, 5), the path is clear to continue north to eventually reach the passenger at (1, 5); the direction is still the correct one.<br><br>.......<br><br>Step 9:<br>- Expert 1: At (4, 4), we are now one block west of the destination. We must go &#39;East&#39; to reach the &#39;B&#39; block where we need to drop off the passenger.<br>- Expert 3: We&#39;re in the correct row, but the taxi needs to go &#39;East&#39; to enter the &#39;B&#39; block for drop-off.<br>- Expert 4: The correct row has been reached for the drop-off. Moving &#39;East&#39; positions us on the destination block.<br>- Expert 5: To finalize our approach to the &#39;B&#39; block, which is the destination, an &#39;East&#39; movement is required.<br><br>Step 10:<br>- Expert 1: The taxi is at the exact location of &#39;B&#39;, which is the destination. The final step is to &#39;Dropoff&#39; the passenger.<br>- Expert 3: With the taxi positioned at &#39;B&#39; (5, 1), the only remaining action is to &#39;Dropoff&#39; the passenger.<br>- Expert 4: The taxi has reached the destination. Now the passenger should be &#39;Dropoff&#39; here.<br>- Expert 5: Now that we&#39;re at the designated drop-off point, we need to complete the &#39;Dropoff&#39;.<br><br>Final Answer: [&#39;North&#39;, &#39;North&#39;, &#39;North&#39;, &#39;Pickup&#39;, &#39;West&#39;, &#39;South&#39;, &#39;South&#39;, &#39;South&#39;, &#39;East&#39;, &#39;Dropoff&#39;]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/388/1*paA5EL2s43sik7gwZuMYQw.gif" /><figcaption>GPT-4-turbo with ToT and location encoding</figcaption></figure><p>You can find the full response in the notebook <a href="https://github.com/XO-Appleton/RL-LLM-Optimality-Research/blob/main/notebooks/optimality_gap_LLM.ipynb">here</a> if you’re interested. GPT-3.5 hallucinates severely and decides to find the destination at (7, 5) when there are 5 rows in total. GPT-4 had a significantly inferior performance. Different Experts seemed to come to the same incorrect answers in steps. Some of them would realize the mistake and resign from the group whereas the others proceed and build up from their initial errors. I suspect by introducing more experts the capability of logical reasoning of GPT-4 is divided, resulting worse worse-performing individual sub-agents.</p><blockquote>We might benefit from ToT logic if we can manage to build it into a LangChain Agent. But I will leave it for a future project.</blockquote><h4>Wall-Inclusive Location Encoding</h4><p>So far, the GPT-4 with location encoding seems to be the closest to success. It just needs to account for the walls on the fifth row and work around them. To allow LLM to access the wall locations, we would need to find a way to represent their locations. To make it consistent with our current way of encoding, we need to expand our Gridworld so every | take up a column as well.</p><pre>wall_inclusive_location_encoding = &quot;&quot;&quot;<br>- The location of the taxi, passenger, and destination will be represented as tuples (row, col).<br>- The positions of walls `|` are encoded in as location (row, col). The taxi cannot move onto or jump over the wall.<br>  -- i.e The taxi cannot go from (1, 3) to (1, 1) suppose if there is a wall at (1, 2)<br>  -- The walls are located at [(1, 0), (1, 4), (1, 10), (2, 0), (2, 4), (2, 10), (3, 0), (3, 10), (4, 0), (4, 2), (4, 6), (4, 10), (5, 0), (5, 2), (5, 6), (5, 10)].<br>- The row and col are 0-indexed. The taxi cannot move onto or jump over the wall.<br>  -- &quot;South&quot;,&quot;North&quot; changes the taxi&#39;s position to (row+1, col), (row-1, col).<br>  -- If there is not a wall at (row, col+/-1), &quot;East&quot;, &quot;West&quot; moves the taxi to (row, col+/-2), else the taxi stays at the same position.<br> -- &quot;Pickup&quot; and &quot;Dropoff&quot; does not change the location of the taxi.<br><br>The taxi starts at the (4, 9). The passenger is at (1, 9), the destination is at (5, 1).<br>&quot;&quot;&quot;<br><br>wall_inclusive_location_encoded_prompt = wall_inclusive_location_encoding + basic_prompt</pre><p>GPT-3.5 already struggles with only basic encodings, so I will let it go and only use GPT-4-turbo from now on. Its response:</p><pre>To solve this task, the taxi needs to maneuver its way to the passenger at (1, 9), pick them up, and then travel to the destination at (5, 1) to drop them off. Let&#39;s plan the taxi&#39;s movements in accordance with the gridworld&#39;s rules: <br>The taxi starts at (4, 9). <br>Move west (left) to avoid the wall and get in line with the passenger: one move west.<br>Move north (up) three times to reach the passenger&#39;s row, avoiding the wall: three moves north. <br>Once aligned with the passenger&#39;s row, move east (right) to the passenger&#39;s location: two moves east. <br>Pickup the passenger. <br>Move south (down) to the bottom row, avoiding the wall: four moves south. <br>Finally, move west (left) to reach the destination drop-off zone: seven moves west. <br>Dropoff the passenger. <br><br>Note: Some actions might seem not to follow the shortest path due to the necessity of avoiding walls. <br>Here&#39;s the detailed action sequence: <br>Final Answer: [&quot;West&quot;, &quot;North&quot;, &quot;North&quot;, &quot;North&quot;, &quot;East&quot;, &quot;East&quot;, &quot;Pickup&quot;, &quot;South&quot;, &quot;South&quot;, &quot;South&quot;, &quot;South&quot;, &quot;West&quot;, &quot;West&quot;, &quot;West&quot;, &quot;West&quot;, &quot;West&quot;, &quot;West&quot;, &quot;West&quot;, &quot;Dropoff&quot;]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/388/1*b4_QbDTsOWWfohdZD56mJA.gif" /><figcaption>GPT-4-turbo with location encoding (wall-inclusive)</figcaption></figure><p>The episode looks similar to the naive location encoding one besides adding some actions to avoid the walls (although unsuccessful). It seems quite challenging for the chat model to complete the task as it is only making a plan from the starting position and cannot account for any issue it runs into during the execution. For example, the model has no clue when the taxi into the wall at the bottom row.</p><p>Let’s bless it with LangChain and turn it into an agent that can use Tools and make observations of the environment.</p><h3>Langchain ReAct Agent</h3><p>Now we’re asking our GPT-4 model to be the agent controlling the taxi rather than only trying to analyze the problem from a textual approach.</p><h4>What is ReAct?</h4><p>ReAct, short for “Reasoning-Acting”, is a method used to enable LLMs to interact with external tools or systems in a more structured and effective way. Here’s a simple explanation:</p><p>Reasoning (Re): This is where the language model thinks about a problem or task. It uses its vast knowledge and understanding of language to come up with ideas, strategies, or solutions. For example, if the task is to navigate a maze, the model would reason about the best path to take.</p><p>Acting (Act): After reasoning, the model takes action based on its thoughts. This action could be interacting with an external tool or system. For example, in the maze navigation task, the action would be the model instructing which direction to move in the maze.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JaBy2xlmo8cVFODd01zazQ.png" /><figcaption>ReAct Agent. Source: <a href="https://react-lm.github.io/">https://react-lm.github.io/</a></figcaption></figure><h4>What is an Agent?</h4><p>The core idea of agents is to use a language model to choose a sequence of actions to take. The language model is used as a reasoning engine to determine which actions to take and in which order.</p><p>In LangChain, we can construct an AgentExecutor with the following structure:</p><p>LLM Chain (LLM - Prompt) | Tools | Output Parser</p><p>Here is what they mean:</p><ul><li><strong>Prompt Template</strong>: A set of instructions that guides how the LLM processes inputs and generates outputs. The brain that drives the taxi. <a href="https://colab.research.google.com/corgiredirector?site=https%3A%2F%2Fpython.langchain.com%2Fen%2Flatest%2Fmodules%2Fprompts%2Fprompt_templates.html">(docs)</a>.</li><li><strong>LLM Chain</strong>: A Chain brings together a prompt template with an LLM that will execute it — in this case we use GPT-4-turbo but it can work with other large language models too. <a href="https://colab.research.google.com/corgiredirector?site=https%3A%2F%2Fpython.langchain.com%2Fdocs%2Fmodules%2Fchains.html">(docs)</a>.</li><li><strong>Tools</strong>: An external service (python function) that the LLM can use to perform tasks <a href="https://colab.research.google.com/corgiredirector?site=https%3A%2F%2Fpython.langchain.com%2Fen%2Flatest%2Fmodules%2Fagents%2Ftools.html">(docs)</a>.</li><li><strong>Output Parser</strong>: Parses the response from executing the LLM Chain. We can enable complex interactions and interactions by extracting intermediate results in their response and passing them into the next step of the chain <a href="https://colab.research.google.com/corgiredirector?site=https%3A%2F%2Fpython.langchain.com%2Fen%2Flatest%2Fmodules%2Fprompts%2Fprompt_templates.html">(docs)</a>.</li></ul><p>The AgentExecutor interacts with the Environment with the Tools specified, makes observations, and passes them to the next iteration.</p><h4>Workflow Overview:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/961/1*KZVoa95ayxjYeYADpjn1Zg.png" /><figcaption>ReAct Agent workflow</figcaption></figure><ol><li>Initialization: The agent is initialized with a specific LLM, prompt template, tools, and output parser.</li><li>Interaction Loop:</li></ol><ul><li>The environment provides an initial state or task description.</li><li>The agent, using the LLM and ReAct Logic, generates a response that includes reasoning and a proposed action.</li><li>The output parser interprets this response and converts it into actionable steps.</li><li>The Agent Executor applies these actions to the environment.</li><li>The environment updates its state based on these actions and provides new information to the agent.</li><li>This loop continues until the task is completed or a stopping condition is met.</li></ul><h4>Taxi World ANSI rendering</h4><p>To generate observations for the LLM, we can use the ANSI rendering of the environment to get a textual representation.</p><p>Let’s first get a sense of how the textual representation should be interpreted with an optimal episode walk.</p><pre>def print_frames(frames):<br>    for i, frame in enumerate(frames):<br>        clear_output(wait=True)<br>        print(frame)<br>        print(f&quot;Timestep: {i + 1}&quot;)<br>        sleep(.5)<br><br>print_frames(optimal_episode_frames)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/232/1*XN-kgooueauNi1TswU95ig.gif" /><figcaption>Textual Rendering</figcaption></figure><p>We can see that an empty taxi is represented by a yellow background, while it is green when it has the passenger on it. The location of the passenger is represented by painting the grid letter blue. And the destination is represented by the purple letter before the passenger is dropped off on it.</p><h4>Defining the Tool</h4><p>With this knowledge in mind, we can write some helper functions that would help interpret the textual rendering into texts that could be easier understood by the LLM.</p><pre>def get_walls(plain_text):<br>    &quot;&quot;&quot;extracts the locations of walls from the plain text representation of the taxi world&quot;&quot;&quot;<br>    walls = []<br>    plain_lines = plain_text.strip().split(&#39;\n&#39;)<br><br>    for row_index, line in enumerate(plain_lines[:-1]):<br>        for i, char in enumerate(line.strip(&#39;|&#39;)):<br>            if char == &#39;|&#39;:<br>                walls.append((row_index, i+1))<br>    return walls<br><br><br># Function to parse the ANSI-rendered text<br>def parse_ansi_rendered_text(rendered_text):<br>    # Split the rendered text into lines<br>    lines = rendered_text.strip().split(&#39;\n&#39;)<br><br>    # Initialize variables<br>    last_action = lines[-1].strip(&#39;( )&#39;) if lines[-1].strip(&#39;( )&#39;) in ACTION_MAP else None<br>    taxi_row, taxi_col = None, None<br>    passenger_location = None<br>    destination = None<br>    passenger_in_taxi = False<br><br>    # Iterate over the lines to find the taxi, passenger, and destination<br>    for row_index, line in enumerate(lines[:-1]):<br>        # Split the line into cells based on the grid separators<br>        cells = re.split(r&#39;\||:&#39;, line)<br>        for col_index, cell in enumerate(cells):<br>            if &#39;\x1b[42m&#39; in cell:  # Check for green background indicating the passenger is in the taxi<br>                passenger_in_taxi = True<br>                taxi_row, taxi_col = row_index, col_index*2-1<br>            elif &#39;\x1b[43m&#39; in cell:  # Check for yellow background indicating the taxi<br>                taxi_row, taxi_col = row_index, col_index*2-1<br>            if &#39;\x1b[34;1m&#39; in cell:  # Check for blue text indicating the passenger<br>                passenger_location = row_index, col_index*2-1<br>                # passenger_location = re.sub(r&#39;\x1b\[\d+(;\d+)?m&#39;, &#39;&#39;, cell).strip()<br>            if &#39;\x1b[35m&#39; in cell:  # Check for purple text indicating the destination<br>                destination = row_index, col_index*2-1<br>                # destination = re.sub(r&#39;\x1b\[\d+(;\d+)?m&#39;, &#39;&#39;, cell).strip()<br><br>    # Remove the ANSI escape sequences for colors<br>    plain_text = re.sub(r&#39;\x1b\[\d+(;\d+)?m&#39;, &#39;&#39;, rendered_text)<br>    walls = get_walls(plain_text)<br><br>    return {<br>        &#39;last_action&#39; : last_action,<br>        &#39;taxi_location&#39;: (taxi_row, taxi_col),<br>        &#39;passenger_location&#39;: passenger_location,<br>        &#39;destination&#39;: destination,<br>        &#39;passenger_in_taxi&#39;: passenger_in_taxi,<br>        &#39;wall_locations&#39;: walls<br>    }<br><br># Function to generate a textual description from the parsed information<br>def generate_textual_description(parsed_info):<br>    description = &quot;&quot;<br>    if parsed_info[&#39;last_action&#39;]:<br>        description += f&quot;Last action taken is {parsed_info[&#39;last_action&#39;]}. &quot;<br>    description += f&quot;The taxi is currently at {parsed_info[&#39;taxi_location&#39;]}. &quot;<br>    if parsed_info[&#39;passenger_in_taxi&#39;]:<br>        description += &quot;The passenger is in the taxi. &quot;<br>    else:<br>        description += f&quot;The passenger is waiting at {parsed_info[&#39;passenger_location&#39;]}. &quot;<br>    if parsed_info[&#39;passenger_location&#39;] == parsed_info[&#39;destination&#39;]:<br>        description += f&quot;The passenger has successfully reached the destination at {parsed_info[&#39;passenger_location&#39;]}. &quot;<br>    else:<br>        description += f&quot;The destination is at {parsed_info[&#39;destination&#39;]}.\n&quot;<br><br>    if parsed_info[&#39;wall_locations&#39;]:<br>        description += f&#39;There are walls at {parsed_info[&quot;wall_locations&quot;]}. \n&#39;<br><br>    return description<br><br># Example usage<br>parsed_info = parse_ansi_rendered_text(init_frame)<br>textual_description = generate_textual_description(parsed_info)<br><br>print(&#39;Parsed info: &#39;, parsed_info)<br>print(&#39;Textual description&#39;, textual_description)</pre><blockquote><em>Output:</em></blockquote><blockquote><em>Parsed info: {‘last_action’: None, ‘taxi_location’: (4, 9), ‘passenger_location’: (1, 9), ‘destination’: (5, 1), ‘passenger_in_taxi’: False, ‘wall_locations’: [(1, 4), (2, 4), (4, 2), (4, 6), (5, 2), (5, 6)]}</em></blockquote><blockquote><em>Textual Description: The taxi is currently at (4, 9). The passenger is waiting at (1, 9). The destination is at (5, 1). There are walls at [(1, 4), (2, 4), (4, 2), (4, 6), (5, 2), (5, 6)].</em></blockquote><p>The interpretation looks reasonable. Now let’s define the tool for the LLM to use to make the action and receive the updated environment status.</p><p>We will call it take_action_step.</p><pre># Define a list of tools<br>from langchain.tools import StructuredTool<br><br>def take_action_step(action):<br>    &#39;&#39;&#39;Make one action step in the environment. Action can be one of South, North, East, West, Pickup, Dropoff&#39;&#39;&#39;<br>    action = ACTION_MAP[action.strip(&#39;\n&#39;)]<br>    env.step(action)<br><br>    rendered_text = env.render()[0]<br>    print(&#39;\n&#39;+rendered_text) # So we can see the observations as well<br>    parsed_info = parse_ansi_rendered_text(rendered_text)<br>    textual_description = generate_textual_description(parsed_info)<br><br>    return textual_description<br><br># wrap the function into a Tool<br>tool = StructuredTool.from_function(take_action_step)</pre><h4>Customizing the ReAct Prompt and OutputParser</h4><pre># Set up the prompt with input variables for tools, user input and a scratchpad for the model to record its workings<br>react = &quot;&quot;&quot;<br><br>TOOLS:<br>------<br>You have access to the following tools:<br><br>{tools}<br><br>To use a tool, please use the following format:<br><br>```<br>Thought: Do I need to use a tool? Yes<br>Action: the action to take, should be one of [{tool_names}]<br>Action Input: the input to the action<br>Observation: the result of the action<br>```<br>... (this Thought/Action/Action Input/Observation can repeat N times)<br><br>When you have a response to say to the Human, or if you do not need to use a tool, you MUST use the format:<br><br>```<br>Thought: The objective has been achieved. i.e `passenger_location` == `destination_location`<br>Final Answer: [The list of action steps taken.]<br>```<br><br>Begin!<br><br>Task: {input}<br>{agent_scratchpad}&quot;&quot;&quot;<br><br>react_template = problem_desc + wall_inclusive_location_encoding + react</pre><p>The codes to construct the ReActPrompt, OutputParser, and AgentExecutor are modified from the LangChain tutorial <a href="https://python.langchain.com/docs/modules/agents/how_to/custom_llm_chat_agent">here</a>. We modify the iteration limit to allow longer thinking time for our model.</p><pre># Initiate our GPT instance<br>llm = ChatOpenAI(model=gpt_4, temperature=0, api_key=os.environ[&#39;OPENAI_API_KEY&#39;])<br><br># LLM chain consisting of the LLM and a prompt<br>llm_chain = LLMChain(llm=llm, prompt=react_prompt)<br><br># Using tools, the LLM chain and output_parser to make an agent<br>tool_names = [tool.name]<br><br>agent = LLMSingleActionAgent(<br>    llm_chain=llm_chain,<br>    output_parser=react_output_parser,<br>    # We use &quot;Observation&quot; as our stop sequence so it will stop when it receives Tool output<br>    # If you change your prompt template you&#39;ll need to adjust this as well<br>    stop=[&quot;\nObservation:&quot;],<br>    allowed_tools=tool_names<br>)<br><br># Initiate the agent that will respond to our queries<br># Set verbose=True to share the CoT reasoning the LLM goes through<br>agent_executor = AgentExecutor.from_agent_and_tools(agent=agent, tools=[tool], verbose=True,<br>                              max_execution_time=120, max_iterations=30)<br><br>print(&#39;max_iterations: &#39;, agent_executor.max_iterations)<br>print(&#39;max_execution_time: &#39;, agent_executor.max_execution_time)</pre><pre>max_iterations:  30<br>max_execution_time:  120.0</pre><p>We have everything necessary set up, now for the moment of truth.</p><h4>ReAct Agent in Action</h4><pre>response = agent_executor.run(textual_description)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/388/1*TV7LiNng59rSgx0Jmmx5_g.gif" /><figcaption>GPT-4-turbo ReAct Agent</figcaption></figure><p>The textual output does not display on Medium so I will just leave the link to the notebook <a href="https://github.com/XO-Appleton/RL-LLM-Optimality-Research/blob/main/notebooks/optimality_gap_LLM.ipynb">here</a>.</p><p>The ReAct Agent showed a lot more insight into how LLM thinks when observing the environment. Our GPT-4-turbo model seemed shortsighted and had trouble understanding how to plan routes, it only looks around 1–2 cells to check for walls but dismisses walls it is ‘obviously’ going to run into based on its plan. I.e LLM decides to go south then west, then only checks if anything is blocking the taxi from going south at the position it is making the decision.</p><p>Despite being shortsighted in planning the walls, the ReAct Agent can reflect on being stuck and look for other ways to complete the task. It is our first successful LLM-as-policy approach that achieved the objective.</p><h3>Performance Comparison</h3><p>Now let’s compare the effectiveness of these different methods to the optimal policy and summarize what we’ve found.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/991/1*NsZtQKCj3yKfGjuGrYLgcA.png" /><figcaption>Performance Comparison of the approaches taken</figcaption></figure><p><strong>Observations</strong>:</p><ul><li>Out of 10 different LLM-As-Policy agents that we tried, only the ReAct Agent was able to succeed.</li><li>Most of the unsuccessful agents took 10 to 20 action steps and received -10 to -40 rewards.</li><li>The GPT 3 model with location encodings and CoT prompting took 45 steps and received -70 in a single episode.</li><li>Using the same encoding prompting method, the GPT-4-turbo approaches seemed to perform better than their GPT-3.5 counterpart.</li></ul><p><strong>Location Encoding</strong>:</p><p>The steps taken and rewards can indicate the performance to some degree. However, it is affected by the randomness of the internal states of the LLM. As we have seen through the process, the LLMs struggled to understand the spatial relationships and retrieve location information most of the time. GPT 4 was occasionally able to spontaneously perform location encoding without given instruction and improve its performance.</p><p>The LLM also had a major issue understanding the position of the walls without encoding their locations. I’ve also tried other methods such as encoding the wall as (row, (cell_1, cell_2)) but it only made the performance worse as LLM became confused about the location encodings in general. This is probably due to the inconsistency between the encoding schemes.</p><p><strong>Prompt Engineering</strong>:</p><p>Applying Zero-shot CoT did not improve the performance of a LLM chat model, it could be due to our problem description or the nature of the task already encouraging the LLM to take a CoT approach. ToT on the other hand rather hurt the performance of the LLMs which might be due to the inferior performance of individual Expert.</p><p><strong>LLM as Agent</strong>:</p><p>Different from the simple one-pass GPT chat completions, building an LLM Agent significantly boosts its performance and enables iterative analysis and freedom of controlling the action directly. In an Agent pipeline, the LLM can make an analysis when the observation is different from what it expected and attempts to make new plans. However, it does have a significantly higher inference cost ($0.5 per episode with Agent Execution compared to a single pass costing around $0.03 per episode. Similar proportion in terms of runtime).</p><h3>Final Thoughts</h3><p>Despite demonstrating progress in task comprehension and a much stronger ability for problem-solving compared to their precedents, LLMs are still far from taking on complex tasks in the real world. Even the current SOTA like GPT-4-Turbo has trouble keeping rules in a simple environment like not jumping over a wall despite being specified in the prompt.</p><p>It is evident that the performance of the LLM agent heavily relies on the presence of a well-engineered encoding algorithm to help the agent perceive the environment and retrieve crucial information. However, the cost of developing such an algorithm could scale drastically with the complexity of the task. We can imagine that the states of a sufficiently intricate environment might not be able to be converted into a textual representation that is accurate, consistent, and complete.</p><p>That being said, the current LLMs are designed to imitate how humans write texts instead of focusing on problem-solving and math. It is intriguing how much progress we are going to see in the next couple of years with LLMs finetuned on logical reasoning.</p><h3>Acknowledgments</h3><p>As we conclude this exploration of the Optimality Gap of LLM-based Agents in the Gridworld Problem, I would like to extend my heartfelt gratitude to those who have contributed significantly to this research.</p><p>This endeavor would not have been possible without Dr. <a href="https://medium.com/u/eb876ee612a5">Yuxi Li</a>, whose invaluable guidance, expertise, and insights have been instrumental in shaping this study. Dr. Yuxi’s deep understanding of Reinforcement Learning, Large Language Models, and commitment to academic excellence have not only enriched this research but also profoundly enhanced my own learning and development in the field.</p><p>This research was not just a journey into the realms of AI and machine learning, but also a testament to the power of collaborative effort and mentorship in pushing the boundaries of our understanding.</p><h3>Reference</h3><p>Wei, J. (2022, January 28). Chain-of-Thought prompting elicits reasoning in large language models. arXiv.org. <a href="https://arxiv.org/abs/2201.11903">https://arxiv.org/abs/2201.11903</a></p><p>Yao, S. (2023, May 17). Tree of Thoughts: Deliberate Problem Solving with Large Language Models. arXiv.org. <a href="https://arxiv.org/abs/2305.10601">https://arxiv.org/abs/2305.10601</a></p><p>Yao, S. (2022, October 6). REACT: Synergizing reasoning and acting in language models. arXiv.org. <a href="https://arxiv.org/abs/2210.03629">https://arxiv.org/abs/2210.03629</a></p><p>Carta, T. (2023, February 6). Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning. arXiv.org. <a href="https://arxiv.org/abs/2302.02662">https://arxiv.org/abs/2302.02662</a></p><p>Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil Jayant Joshi, Ryan C. Julian, Dmitry Kalashnikov, Yuheng Kuang, KuangHuei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego M Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, and Mengyuan Yan. Do as i can, not as i say: Grounding language in robotic affordances. ArXiv, abs/2204.01691, 2022.</p><p>Greg Brockman and Vicki Cheung and Ludwig Pettersson and Jonas Schneider and John Schulman and Jie Tang and Wojciech Zaremba (2016) OpenAI Gym. ArXiv, abs/1606.01540</p><p>Van Heeswijk, W., PhD. (2022, January 6). Walking off the cliff with Off-Policy Reinforcement learning. Medium. <a href="https://medium.com/towards-data-science/walking-off-the-cliff-with-off-policy-reinforcement-learning-7fdbcdfe31ff">https://medium.com/towards-data-science/walking-off-the-cliff-with-off-policy-reinforcement-learning-7fdbcdfe31ff</a></p><p>Wikipedia contributors. (2023, November 29). Q-learning. Wikipedia. <a href="https://en.wikipedia.org/wiki/Q-learning">https://en.wikipedia.org/wiki/Q-learning</a></p><p>openai-cookbook/examples/How_to_call_functions_with_chat_models.ipynb at main · openai/openai-cookbook. (n.d.). GitHub. <a href="https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb">https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb</a></p><p>openai-cookbook/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb at main · openai/openai-cookbook. (n.d.). GitHub. <a href="https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb">https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb</a></p><p>Reinforcement Q-Learning from Scratch in Python with OpenAI Gym. (n.d.). <a href="https://www.learndatasci.com/tutorials/reinforcement-q-learning-scratch-python-openai-gym/">https://www.learndatasci.com/tutorials/reinforcement-q-learning-scratch-python-openai-gym/</a></p><p>Chain-of-Thought prompting — Nextra. (n.d.). <a href="https://www.promptingguide.ai/techniques/cot">https://www.promptingguide.ai/techniques/cot</a></p><p>Gymnasium documentation. (n.d.). <a href="https://gymnasium.farama.org/environments/toy_text/taxi/">https://gymnasium.farama.org/environments/toy_text/taxi/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6188bc09a14c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>