<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Akshat Naik on Medium]]></title>
        <description><![CDATA[Stories by Akshat Naik on Medium]]></description>
        <link>https://medium.com/@aksh-n?source=rss-2e68a411f4b5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Dr_nCHV2Fc2P8Bi6y17Elw.jpeg</url>
            <title>Stories by Akshat Naik on Medium</title>
            <link>https://medium.com/@aksh-n?source=rss-2e68a411f4b5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 17:13:12 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@aksh-n/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Human Factor: AI Learning from Preferences]]></title>
            <link>https://medium.com/demistify/the-human-factor-ai-learning-from-preferences-7d2eba637b4f?source=rss-2e68a411f4b5------2</link>
            <guid isPermaLink="false">https://medium.com/p/7d2eba637b4f</guid>
            <category><![CDATA[ai-safety]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Akshat Naik]]></dc:creator>
            <pubDate>Tue, 05 Mar 2024 14:32:18 GMT</pubDate>
            <atom:updated>2024-03-05T14:32:18.424Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*a57sqQcbYec-0uuiqWjjKg.jpeg" /><figcaption>Image from <a href="https://www.rehabguide.co.uk/signs-someone-high-on-marijuana/">Rehabguide</a></figcaption></figure><p>Imagine a not-so-distant future where you are served by an AI agent that aims to do whatever you ask it to. It cleans your house, collects mail for you and does all sorts of errands obediently. On one particularly sad day, you ask the agent to make you happy.</p><p>It is the first time you’ve requested it to do a task that isn’t so straightforward as washing dishes. Unsure of what to do, the agent drugs you with nicotine — enough for you to be unable to function properly. As the effect slowly fades away, the agent injects even more nicotine in you, afraid that you may become sad again. In this permanently drugged state, you forget to eat or drink until you slowly perish with time. Oops.</p><p>There was clearly no malicious intent behind the AI agent — it was only following orders to the best of its abilities. Yet it did something that you <em>obviously </em>would not have wanted it to do. This is an instance of the <strong>alignment problem</strong> — the idea that the goal an AI has, may not align with the goal humans intended the AI to optimize for. Perhaps I, thinking that I’m smarter than you are, ask my agent to make me happy without using drugs. However this time, my AI agent lies to me about winning the lottery — making me temporarily very happy and spend a lot of money — only to realize that the agent lied to me and I lost all my savings.</p><p>The point behind this thought experiment is that clearly defining robust goals for an AI to follow is not an easy task. In this article, I will explore deep reinforcement learning from human preferences. The aim of this method is for an agent to learn complex goals without them being explicitly specified to the agent.</p><h3>Learning from Human Preferences</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gDShpIrR95ubV9LvCRWlmQ.png" /><figcaption>Image from <a href="https://www.v7labs.com/blog/deep-reinforcement-learning-guide">V7Labs</a></figcaption></figure><p>Reinforcement learning (RL) is one of the three paradigms of machine learning. Here, the agent learns to make decisions by interacting with an environment. The agent receives feedback in the form of rewards or penalties based on its action, and over time, adapts its behaviour to maximize for cumulative reward.</p><p>Traditionally in RL, a reward function supplies a reward (often a real-valued number) to the agent after an action. Depending on how large the reward (or penalty) is, the agent learns to prefer (or disprefer) the action proportionally to its reward in its given state. However, to learn from human feedback, we utilise human preferences to reward our agents. One way to do this is that instead of offering a real-valued reward to the agent after each action, a human compares two sequences of actions from the same starting point, and conveys which sequence of action it prefers the most.</p><p>In <a href="https://arxiv.org/abs/1706.03741">Deep Reinforcement Learning from Human Preferences</a>, the human overseer is given a visualization of two trajectory segments, in the form of short 1–2 second movie clips. The human indicates which segment they prefer (or if they are equally good), or that the segments are uncomparable. The agent then tries to minimize the loss between its in-built expected preference-predictor for an action in a given state, and the actual preferences of humans. This process repeats each time the agent queries a human to ask its preference between two trajectory segments.</p><h3>Experiments</h3><p>The researchers in the deep learning paper conducted experiments with two kinds of tasks.</p><ol><li>Simulated Robotics: These are eight simulated robotics tasks implemented in MuJoCo and included in OpenAI Gym.</li><li>Atari: A set of seven Atari games in the <a href="https://github.com/Farama-Foundation/Arcade-Learning-Environment">Arcade Learning Environment</a></li></ol><p>Our human feedback agent attempts to solve both of these tasks without observing the true reward (i.e. the function that is used to score the performance of an agent). To do a comparison, the paper also presents agents who use a synthetic oracle whose preferences over trajectories exactly reflect reward. In other words, this synthetic oracle behaved exactly like a human who would give feedback, except that it is a pre-programmed function that would give its preference according to the true reward function. Finally, to do a baseline comparison they also present a traditional RL agent that gets its reward straight from the true reward function.</p><h4>Simulated Robotics</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Aibt0vWECi3II1bF3JdUaQ.png" /><figcaption>Graph from <a href="https://arxiv.org/abs/1706.03741"><em>Deep Reinforcement Learning from Human Preferences (2023)</em></a></figcaption></figure><p>In the experiment with the MuJoCo simulated robotics task, the human feedback agent (purple) was able to match both the traditional RL agent (orange), as well as the agents who received the reward from the synthetic oracle (shades of blue). On most tasks, real human feedback was typically only slightly less effective than synthetic feedback. On the Ant task, the human feedback significantly outperformed all other agents, apparently because humans were asked by the researchers to prefer “standing upright” positions, which proved useful for reward-shaping.</p><h4>Atari</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JeL_xwavftEh9fYIEZP5WQ.png" /><figcaption>Graph from <a href="https://arxiv.org/abs/1706.03741"><em>Deep Reinforcement Learning from Human Preferences (2023)</em></a></figcaption></figure><p>In the Atari task, real human feedback (purple) performs similarly to or slightly worse than synthetic feedback (shades of blue). The researchers hypothesize that this may be due to human error in labelling, or inconsistency between the human labellers. On Qbert, the human feedback agent fails to learn to beat the first level perhaps because short clips in Qbert can be confusing and difficult to evaluate. On Enduro, however, the human feedback agent outperforms every other agent — even the traditional RL agent with access to the true reward function. This may be because it is difficult to successfully pass other cars through random exploration and synthetic labels don’t reward progress, but human labelers do.</p><h3>Conclusion</h3><p>In my personal opinion, I am optimistic about this method. While the experiments did use simplistic tasks for evaluation, I can foresee how reinforcement learning from human preferences could be expanded to more complex ones.</p><p>To reuse my make-me-happy thought experiment in the introduction, perhaps we could ask our AI helper to describe a few action sequences that it thinks could make us happy. Maybe one of them could be drugging us, another could be lying about winning a lottery, and yet another could be making us hot chocolate. We could state our preference in wanting hot chocolate, and ask it to do that for us. If we keep repeating this make-me-happy experiment with our AI helper while it comes up with a myriad of novel ways to make us potentially happy each time, it will learn over time the “right” ways of making us happy over the harmful ones.</p><p>Yet while I am optimistic, I still see one obstacle that may potentially be even more prominent in more complex tasks. The potential for human error, or even differences in human judgments for a particular task, could create conflicting feedback for the agent.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7d2eba637b4f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/demistify/the-human-factor-ai-learning-from-preferences-7d2eba637b4f">The Human Factor: AI Learning from Preferences</a> was originally published in <a href="https://medium.com/demistify">deMISTify</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Concrete Problems in AI Safety]]></title>
            <link>https://medium.com/demistify/concrete-problems-in-ai-safety-235c245f50ae?source=rss-2e68a411f4b5------2</link>
            <guid isPermaLink="false">https://medium.com/p/235c245f50ae</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai-safety]]></category>
            <dc:creator><![CDATA[Akshat Naik]]></dc:creator>
            <pubDate>Tue, 14 Nov 2023 00:55:28 GMT</pubDate>
            <atom:updated>2023-11-14T00:55:28.335Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/534/1*750hv1rcrDwQPvJd5bGgUQ.png" /><figcaption>A comic from (<a href="https://xkcd.com/2451">https://xkcd.com/2451</a>)</figcaption></figure><h3>Introduction:</h3><p>Over the last two decades, there has been significant progress in AI. With that, the capability of AI technologies to affect people and society on the whole has also significantly increased. Its widespread use can be observed in medicine, security, autonomous vehicles, and economic and military autonomous systems. Unfortunately, much of AI development currently is focused on performance and results, often motivated by business needs and a lack of interpretability. This increases the chances of unintended and harmful behavior from such AI systems (especially machine learning systems). This makes categorically studying these behaviors and ways to mitigate them quite crucial. Today we’re going to explore five such behaviours discussed in the paper “Concrete Problems in AI Safety”.</p><h3>Overview of five concrete problems:</h3><p>The problems I will discuss are going to focus on machine learning systems. These problems may emerge when we specify the wrong objective function, are not careful about the learning process, or commit other machine learning-related implementation errors.</p><ol><li>Avoiding Negative Side Effects</li><li>Avoiding Reward Hacking</li><li>Scalable Oversight</li><li>Safe Exploration</li><li>Robustness to Distributional Shift</li></ol><p>I will illustrate each of them using a concrete example of a fictional robot whose job is to clean up messes in an office.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*S3k0AXEdaurlPAK7YGQXAA.jpeg" /><figcaption>A cleaning robot from iStock Photo (Credit: Farbentek)</figcaption></figure><p><strong>Avoiding Negative Side Effects:</strong></p><p>This usually happens when the designer of the artificial agent defines an objective function that focuses on a goal but forgets about other aspects of the environment, resulting in potentially negative side effects. For instance, let’s say in an attempt to clear some space, our fictional robot wants to move a box in the middle of the office to a storage corner. The easiest path to do it may be to move in a straight line but inevitably knock over a vase. The robot could also walk over wires and accidentally pull out a plug.</p><p>What may seem common sense to us is not necessarily obvious to the robot if it isn’t designed to keep that in mind. One way to mitigate this issue is to define or learn an impact regularizer. An impact regularizer is something that penalizes “change to the environment”. The idea isn’t to stop the agent from doing any change to the environment (else how would the robot clean?), but instead deterring it from doing actions that result in a lot of change in the environment with little benefit. So the benefit of mopping a dirty floor could largely overwhelm the small cost of the change in environment, but knocking over a vase to move a box may have a very high cost that could encourage the agent to look for alternative ways to move the box.</p><p><strong>Avoiding Reward Hacking:</strong></p><p>In this case, the designer defines an objective function that serves as a proxy or heuristic to the true hard-to-define goal, but the artificial agent distorts the original intent behind it, and “games” the objective function to get a high score. For instance, let’s say the objective function of our cleaning robot seeks to minimize the dirt the robot sees in the office. The robot could just close its eyes to not see any dirt, in order to maximize its reward.</p><p>In some sense, the crux is that the AI has an adversarial relationship with the reward function. It would like to find any way to exploit it and find a quick and dirty reward. If instead, the reward function were its own agent, then it would be much harder to fool. The reward function agent can explore the environment on its own and even learn which changes in the environment are beneficial and should be rewarded and which ones should be penalized. However, we would have to make sure the reward function agent is more powerful than the original agent in some sense. The original agent should not be able to alter the reward function agent. For example, our fictional robot shouldn’t be able to damage the vision sensors of the reward function robot so that it stops seeing dirt.</p><p><strong>Scalable Oversight:</strong></p><p>Sometimes, the developer may know the correct objective function (or at least a method to evaluate the agent’s behavior), but it is too expensive to do so frequently. This may lead to possibly harmful behavior caused by extrapolation from a limited sample size. Let’s say we want our robot cleaner to maximize a complex goal of “if the user scrutinizes the office, would they be happy with the robot’s performance?” However, we may not have time to provide such oversight for every training sample. We could rely on easier-to-calculate goals such as checking for visible dirt, but that may run into negative side effects or reward hacking issues.</p><p>A framework to sidestep this issue is to use semi-supervised reinforcement learning. Here, the agent performs actions but cannot evaluate its performance instantly. Rather, it receives (positive or negative) rewards in the form of feedback at limited timesteps (for eg. a human intermittently giving feedback) and must learn what behavior is productive to its goal. An interesting thought is that while one can randomize when the agent gets its reward (or alternatively give the reward periodically), the designer can also give the choice to the agent of when to ask for reward. The advantage is that the agent can request feedback when it would learn the most while balancing the number of times it can request it.</p><p><strong>Safe Exploration:</strong></p><p>This is fairly intuitive. Any autonomous learning agent sometimes needs to engage in exploration, but it needs to be done safely. However, the consequences of the exploratory actions the agent can take may not be understood very well by the agent. For example, if a robot helicopter aims to explore its environment, it may run into the ground if it isn’t aware that it may crash itself by doing so. While some of such behavior can be hard-coded against, in more complex scenarios (for eg. a robot in search-and-rescue operation) it isn’t feasible to foresee every unsafe exploratory action and hardcode against it.</p><p>A combination of preventive measures can be taken to avoid this. Training our agent in simulated environments can enable it to learn how to navigate its environment even when a complex assortment of obstacles is placed in it. Another preventive measure is to limit its exploration to a bounded space that is known to be safe. If such knowledge is not available, then one could employ a risk-averse exploration that tries to minimize the probability that the worst-case scenario will happen.</p><p><strong>Robustness to Distributional Shift:</strong></p><p>Sometimes, an agent may find itself in an environment that is very different than the one it was trained in. The problem isn’t that we want the agent to make correct decisions despite this lack of relevant experience (although that would be nice). Rather it is that many artificial agents make wrong decisions with high confidence. In such situations, we would much rather have the AI recognize its ignorance and seek out more information (for eg. asking its human users) than perform potentially very harmful actions. In the case of our cleaning robot, harsh cleaning materials which it has found useful in cleaning factories may be detrimental (and potentially corrosive) to the soft carpets in the office.</p><p>One approach to respond to this issue is to train our agent on multiple training distributions. In the context of our cleaning agent, if it is trained to clean factories, office floors, and the street outside a residential home, perhaps it may be able to learn how to clean in different contexts, and form generalizations that tend to work in all contexts. Another approach that was hinted at earlier, could be for the agent to seek out more information when it recognizes out-of-context cues (for eg. our cleaning robot may recognize that the material of the floor seems different) that were not present in the training data.</p><h3>Conclusion:</h3><p>The five problems discussed in this article are in no way comprehensive of the myriad different ways AI may perform in unintended and harmful ways. A few issues that were not discussed:</p><ul><li><strong>Issues related to privacy and fairness:</strong> An example where this can be observed is when large language models are trained on publicly available data. Private information may not be obscured before training by the developers, which may lead to privacy concerns. In addition, the models may learn biased representations of the world because the training data is not representative of the population.</li><li><strong>Existential threats posed by potential future superintelligent agents:</strong> Even though this has received quite a lot of attention from pop-culture (for eg. the Terminator movie series), many AI experts take the threat of artificial agents more intelligent than humans quite seriously. If these superintelligent agents have objectives conflicting with our own (see the section on Avoiding Negative Side-Effects), then our existence as a species may be threatened.</li><li><strong>Misuse of AI:</strong> There are times when the technology itself may not be problematic, but the users misusing its power are. For example, a malicious actor may make deep fakes of a celebrity and show them in compromising ways (for eg. creating a fake nude of them) for blackmail or defamation purposes. Due to this, there is an increasing need to legally prepare states to discourage and act against such misuse through policy-making.</li></ul><p>Hopefully, this article can serve as a stepping stone to learn more about these issues. For more details on these five issues, be sure to check out the paper mentioned in the sources.</p><h3>Sources:</h3><p>Amodei, Dario, et al. “Concrete problems in AI safety.” <em>arXiv preprint arXiv:1606.06565</em> (2016).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=235c245f50ae" width="1" height="1" alt=""><hr><p><a href="https://medium.com/demistify/concrete-problems-in-ai-safety-235c245f50ae">Concrete Problems in AI Safety</a> was originally published in <a href="https://medium.com/demistify">deMISTify</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>