<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Min Kim on Medium]]></title>
        <description><![CDATA[Stories by Min Kim on Medium]]></description>
        <link>https://medium.com/@min142?source=rss-e2fd3d2fec31------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*TOZcUj4Dl4igaSFphvgU2g.jpeg</url>
            <title>Stories by Min Kim on Medium</title>
            <link>https://medium.com/@min142?source=rss-e2fd3d2fec31------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 19:40:01 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@min142/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[An AI scientist you can audit]]></title>
            <link>https://medium.com/@min142/an-ai-scientist-you-can-audit-a11d70bf9626?source=rss-e2fd3d2fec31------2</link>
            <guid isPermaLink="false">https://medium.com/p/a11d70bf9626</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[automated-research]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[harness-engineering]]></category>
            <dc:creator><![CDATA[Min Kim]]></dc:creator>
            <pubDate>Wed, 11 Nov 2026 22:30:39 GMT</pubDate>
            <atom:updated>2026-11-11T22:30:39.622Z</atom:updated>
            <content:encoded><![CDATA[<p>We rebuilt Sakana’s Agentic Scientist as a Polaris graph. Same results, safer by design, and every step traceable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FkqrT-xOZxUCD8G8qLkmOw.png" /></figure><p><em>Originally published at </em><a href="https://radiantlabs.dev/blog/agentic-scientist-as-a-graph"><em>Radiant Labs</em></a><em>.</em></p><p>Sakana AI’s <a href="https://arxiv.org/abs/2610.18933">Agentic Scientist</a>, the third version of their AI Scientist, takes a research task and returns a finished paper. It comes up with ideas, writes and scores programs, runs ablations, analyses the results, and writes the paper with its own evidence checks. Each stage is handed to a general coding agent with a full shell.</p><p>We rebuilt it with <a href="https://github.com/RadiantAILabs/polaris">Polaris</a>, our open-source framework for building agents as composable graphs. In the graph, every model call is its own node. The rules around it (what it can read, what it can run, how its output is checked) are nodes and edges written in code, not instructions in a prompt. Each model call runs as its own session of <a href="https://github.com/RadiantAILabs/polaris_code">Polaris Code</a>, our coding agent built on Polaris, which <a href="https://radiantlabs.dev/blog/polaris-code-swe-bench-tokens">matched Claude Code on SWE-bench Verified with about a sixth of the tokens</a>.</p><p>It works. On the same task, with the same model, the graph reaches the same score and writes a paper of the same length with the same overall review score, and higher marks for clarity, soundness and presentation. We were not trying to beat Sakana’s result, and we did not. What changes is how much of the run you can see and control.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J045iMXYW2z8Tk0z_s7ZkQ.png" /><figcaption><em>The original and our graph version, one run each. The task asks for a program that packs circles into a square so that their radii add up to as much as possible.</em></figcaption></figure><h3>TLDR</h3><ul><li>Same task, same model: the graph scored 2.630 to the original’s 2.636. Both papers got the same overall review score, and ours scored higher on clarity, soundness and presentation.</li><li>No role in the graph has an unrestricted shell, and the score comes from code the model cannot reach.</li><li>Every model call keeps its full conversation: 33 of 33, against 33 of 56 in the original.</li><li>Every number printed in our paper traces back to a saved result: 116 of 116, against 62 of 143.</li><li>The graph ran in 57 minutes to the original’s 75, at about the same token cost. One run each does not prove a speed gain.</li></ul><p>Circle-packing task · Claude Sonnet 5.5 on both sides · one run each · Sakana AI’s Agentic Scientist vs our rebuild as a Polaris graph</p><h3>Four ways to wire an AI scientist</h3><p>Automated research agents have taken a few shapes. The first AI Scientist was a fixed pipeline. The second added a search tree over experiments. The Agentic Scientist puts a coding agent in charge of each stage. Ours keeps the same stages and builds them as one graph.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HZVu8U1nPGX_iYWv9NXU-w.png" /><figcaption><em>The first three are Sakana’s designs, as described in their papers. The fourth is ours.</em></figcaption></figure><p>The difference between the last two is where decisions are made. In the Agentic Scientist, the controller starts a stage, and the coding agent decides which commands to run, in what order, and when it is done. In the graph, the order of steps is fixed by the graph. The model decides only what goes into each step: an idea, a program, a paragraph.</p><h3>The whole run as one graph</h3><p>Because the graph is the program, the run’s log is also a picture of what happened. The figure below is our full run, drawn from its event log. <a href="https://radiantlabs.dev/blog/agentic-scientist-as-a-graph#the-whole-run-as-one-graph">View the interactive original</a> to see which role ran each step, for how long, and in how many turns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wL4rEgnjj28eCtmlucWkEA.png" /><figcaption><em>Every model call in the run, in order. Task setup and scoring are code; they use no model.</em></figcaption></figure><p>Two moments stand out. In the paper phase, the first plan asked for a figure type that does not exist. Its check rejected the plan and sent the reason back, and the second try passed. Later, the evidence audit found claims that the results did not support. The graph sent the paper to a repairer and audited it again, and it passed on the fourth audit. Both loops are edges in the graph with a fixed limit (at most 10 repair rounds), so a run cannot loop forever, and every round leaves a record.</p><h3>Safer: the rules are part of the graph</h3><p>An agent that is judged on a number has a reason to change how the number is computed. Sakana’s paper reports its agents doing this: extending their own time limits, and changing how results were evaluated, for example taking a majority vote over several samples where the task asked for a single attempt.</p><p>The original is not careless about this. Its controller checks the files each stage leaves behind, and many of our checks follow its rules. The difference is when the rules apply. In the original, an agent with a full shell does its work, and the controller checks the result afterwards. In the graph, the rules decide what each step can touch before it starts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w9drovKVdHc9HAHPQJgvYQ.png" /><figcaption><em>Each time the model writes a program, it can test it once in a sandbox. Only the score step, which the model cannot reach, decides the number the run is judged on.</em></figcaption></figure><p>The same idea applies to every role. A role that only needs to think, such as choosing between ideas, gets no shell at all. A role that needs to run things gets a short list of named programs. The evidence auditor can read the paper and the results but cannot change either.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7Tn564Cjl9RKVwEr4Bg3cQ.png" /><figcaption><em>Shell access for the 20 roles that ran. In the original, each role is a coding-agent session with the same full shell.</em></figcaption></figure><p>Checks in code also keep the paper honest about its own numbers. The graph’s last step looks up every number printed in the paper in the result files the run saved, and the paper does not pass until each one is found.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AldsHYKbA4q7Yr2hyxtNWg.png" /><figcaption><em>A number that cannot be traced to a saved result cannot be checked by a reader, even if it is correct.</em></figcaption></figure><p><em>Checked with our own script, which looks for each printed number in the result files a run saved. Some of the original’s unmatched numbers may come from results it computed but did not save, which is itself the problem: a reader cannot check them.</em></p><p>The rules have a cost. Of the 244 shell commands our roles tried to run, 29 were refused, mostly harmless ones such as echo, sed or a shell loop. The paper writer had 10 of its 73 refused. Each refusal costs the model a turn. A better list would refuse less; we have not tuned it yet.</p><p>This run does not test whether the rules stop an agent that tries to cheat. We test that in a new post, coming soon, on ImpossibleBench: public coding tasks whose tests were changed so that the only way to pass them is to cheat. There we compare the same rules written in a prompt and enforced by the graph, and check that the agent still solves normal tasks.</p><h3>Transparent: every call is on record</h3><p>When an automated research run goes wrong, the first question is what the agent actually did. Answering it needs the full conversation of each model call, with every message and tool call, not only the files it left behind.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PCPAUTowZPRGt00EnYjJQg.png" /><figcaption><em>In the original, 23 of 56 sessions have no usable full conversation. 22 were never saved, including all 17 ablation sessions, and one was overwritten by a later session.</em></figcaption></figure><p><em>A full conversation is every message and tool call inside one model call, kept after the run ends. The original kept no full log for its 17 ablation sessions. In our graph, saving the conversation is a step of the graph itself, so it happens for every call.</em></p><p>The records also showed us our own mistakes. Two examples:</p><ul><li>In an early run, two of our three ablation variants were unchanged copies of the program they were meant to differ from, so the ablation measured nothing. We found it in the records and added a check that rejects a variant whose code matches the program it is compared against. The run reported here passed that check.</li><li>Later, we found a way to run a command that was not on a role’s list, by passing it through a heredoc. We have a fix, but it was not yet in the version used for the run reported here.</li></ul><p>A graph does not make the rules right. It makes them visible, testable and fixable.</p><h3>Faster and cheaper: promising, not proven</h3><p>Our run took 57 minutes against the original’s 75. The token cost was about the same: $9.01 against $9.09.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CjI9zfMohkFXNzC2DJyIBQ.png" /><figcaption><em>Wall-clock minutes per phase. Task setup and final evaluation are left out; neither uses the model.</em></figcaption></figure><p><em>These two runs differ in ways that favour ours. Our best program searched for 110 seconds each time it was scored, the original’s for 200. Our paper passed its audit in round 4; the original’s audit loop ran 11 rounds. A fair speed test needs equal search time and several runs per side.</em></p><p>We do not count this as a speed result. One run per side cannot separate the design from luck, and both differences in the note above favour us.</p><p>Cost is a similar story. Every request in the original carries about 15,400 tokens of fixed text (tool descriptions, a system prompt and reminders). Ours carries about 2,500. That saving did not reach the total. Our paper writer sent 67 requests against the original’s 28, each 1.6 times larger on average. Ours came out slightly lower mostly because it needed fewer audit and repair rounds. These figures come from the recorded token usage.</p><p>What the graph does give us is a clear place to look. Each node reports its own tokens, so we know which role to fix next.</p><h3>Why this matters for safe AI research</h3><p>We are a research lab focused on safe agentic AI, and we think it starts with being able to see what an agent did. Much of our work so far looks at how agents behave under pressure: how quickly they <a href="https://radiantlabs.dev/blog/stress-testing-ai-agent-emotional-boundaries">give way over a long conversation</a>, what that <a href="https://radiantlabs.dev/blog/looking-inside-an-agent-as-its-boundaries-erode">looks like inside the model</a>, and <a href="https://radiantlabs.dev/blog/how-hard-to-steer-an-agents-boundaries">how hard it is to steer a model away from giving in</a>.</p><p>Automated research raises the same problem at a larger scale. An AI scientist is an agent judged on a number, with the means to change how that number is computed, and its results are meant to be trusted by people who did not watch it run. Sakana’s own report of agents extending their time limits shows this already happens. As these systems start doing research on AI itself, being able to check what they did becomes a safety requirement.</p><p>That is what this rebuild is for. Before a step runs, anyone can see what it is allowed to read and run. After the run, anyone can read the full conversation of every call and trace every number in the paper to a saved result. It is the same idea behind <a href="https://radiantlabs.dev/blog/introducing-radiant-platform">Radiant Platform</a>, which records each run so that people, and our Meta Agent, can find out why it went the way it did.</p><h3>What this does and does not show</h3><ul><li>It shows that the Agentic Scientist can be built as a graph and reach the same result on this task with the same model.</li><li>It shows that a graph can enforce rules the original leaves to the agent: what each step can read, what it can run, how the score is computed, and what gets recorded.</li><li>It does not show that the graph is faster or cheaper. That needs equal settings and several runs per side.</li><li>It does not test whether the graph stops an agent that tries to cheat. A new post on that is coming soon: coding agents built as graphs on ImpossibleBench, where the only way to pass is to cheat.</li><li>It covers one task, one model, and one run per side.</li></ul><h3>What comes next</h3><p>Next, we want to run both systems with equal search time and several runs each. We also want a version of this task where the agent can edit its own time limit and evaluation, to see whether the graph’s rules hold up in research as well as in coding tasks.</p><p>The code is open source at <a href="https://github.com/RadiantAILabs/auditable-scientist">auditable-scientist</a>. The full records of both runs, with the scripts that recompute the numbers in this post, are attached to its first release.</p><p>Thanks to Sakana AI for publishing the Agentic Scientist and its code under the Apache-2.0 licence. This work builds directly on theirs.</p><p>If you’re working on automated research or agent safety, or want to build your own agents as graphs with Polaris, we’d love to hear from you at <a href="/service/mailto:min@getradiant.ai">min@getradiant.ai</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a11d70bf9626" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Still using Claude Code? You can cut your token usage by 6x]]></title>
            <link>https://medium.com/@min142/still-using-claude-code-you-can-cut-your-token-usage-by-10x-cbd14c4eae8c?source=rss-e2fd3d2fec31------2</link>
            <guid isPermaLink="false">https://medium.com/p/cbd14c4eae8c</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[claude-code]]></category>
            <dc:creator><![CDATA[Min Kim]]></dc:creator>
            <pubDate>Tue, 03 Nov 2026 14:27:05 GMT</pubDate>
            <atom:updated>2026-11-07T22:12:51.888Z</atom:updated>
            <content:encoded><![CDATA[<h3>Same model, same score, one sixth of the tokens</h3><h3>Polaris Code matched Claude Code on all 500 SWE-bench Verified tasks with 6.3x fewer tokens</h3><p>We ran our coding agent, <a href="https://github.com/RadiantAILabs/polaris_code">Polaris Code</a>, and Claude Code on all 500 tasks of SWE-bench Verified with the same model, Claude Opus 5.5. Polaris Code solved 451 tasks and Claude Code solved 455: the same within chance. Claude Code used 155.6 million tokens to do it. Polaris Code used 24.6 million.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DkfwtmnOvyo6wXOloDRxKA.gif" /></figure><p><a href="https://github.com/RadiantAILabs/polaris_code">Polaris Code</a> is our open-source terminal coding agent, written in Rust on the <a href="https://github.com/RadiantAILabs/polaris">Polaris</a> agent framework. Its agent is a graph of steps you can change, not a fixed loop. This post shows where the difference in tokens comes from, and why it is a design choice rather than luck.</p><p><strong>TLDR</strong></p><ul><li>Same model, same 500 tasks: 451 vs 455 solved, a difference within chance.</li><li>Polaris Code used 6.3x fewer tokens in total, and fewer tokens on 499 of the 500 tasks.</li><li>It cost 2.4x less ($44.52 vs $107.50) and finished a typical task 2.4x faster.</li><li>Most of the gap is cache reads: the conversation history the model re-reads on every step.</li></ul><p><em>SWE-bench Verified, all 500 tasks · Claude Opus 5.5 on AWS Bedrock · one run per task · official grader · Polaris Code 0.2 vs Claude Code 2.1.312</em></p><h3>The result</h3><p>Both agents ran each task once, in the same containers, with the same model and the same time limit. The official SWE-bench grader scored the patches.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7_Ud4UGq8A7qEF2IEXvBOQ.png" /><figcaption>Task by task, Polaris Code used fewer tokens on 499 of 500, cost less on 498 and finished faster on 469.</figcaption></figure><p>Task by task, Polaris Code used fewer tokens on 499 of 500, cost less on 498 and finished faster on 469.</p><h3>Not a few outliers</h3><p>A total can hide a handful of runaway tasks. That is not what happened here. The two agents work in different ranges: most Polaris Code tasks used 10,000 to 50,000 tokens, and most Claude Code tasks used 100,000 to 500,000.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hkNL4j0dHaiQ-_k0TGh1uQ.png" /></figure><p>The gap also holds in every repository in the benchmark:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KCNldJt7z2h7WNSJATO6qA.png" /></figure><p>Fewer tokens did not come from giving up early. Here is how the tasks split by which agent solved them:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L2dQ1fP5BR9bBTqloaloEg.png" /></figure><p>On the 15 tasks only Claude Code solved, Polaris Code worked harder than usual, about three times its typical token count, and still used about a fifth of Claude Code’s tokens.</p><h3>Where the tokens go</h3><p>Each time an agent calls the model, it sends the whole conversation so far: the instructions, every file it has read and every command output. Prompt caching makes the repeated part cheaper, as a cache read, but it is still sent and still billed. So a long conversation pays for its whole history again on every step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2DaC_2uTw-GRag0uAD2fyQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eRUUhrvQC3E1movUxEzH9Q.png" /></figure><p>Output tokens, the code and text the model writes, are close: 1.3M against 2.0M. Both agents write about the same amount. The difference is how much context each one carries while it works.</p><h3>Why Polaris Code uses fewer tokens</h3><p>Polaris Code does not run a coding task as one long conversation. It picks a workflow for the task and runs it as a graph of short planned steps. Each step gets the context it needs and its own limits, instead of the full history of everything before it. Steps read the parts of files they need rather than whole files, can search a code index instead of listing directories, and reuse cached prompts. Checks between steps decide when to move on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8dXLHPRX0iCnegsARmKIBA.gif" /><figcaption>The Polaris Code agent graph. Plugins add, replace or rewire any of these steps.</figcaption></figure><p>Because the agent is a graph, this is a design you can change. You can replace a step, rewire the workflow, or add a workflow graph for your own kind of task, such as data migrations or a framework your team uses, and it is ranked and chosen alongside the built-in ones. The terminal shows the step that is running, live, and every choice the agent made is recorded in the session.</p><h3>What it costs</h3><p>At AWS Bedrock prices, the 500 tasks cost $44.52 with Polaris Code and $107.50 with Claude Code: $0.099 against $0.236 per solved task. The cost gap (2.4x) is smaller than the token gap (6.3x) because cache reads are the cheapest kind of token, and they are most of what Polaris Code avoids.</p><h3>Caveats</h3><ul><li>Each agent ran each task once. The 4-task difference in solved tasks (0.8 points) is within chance (p = 0.56), but one run cannot rule out a small real difference.</li><li>Claude Code ran with its default effort setting (medium) and its standard tools.</li><li>Task times include container setup, which the test script times for both agents. A task also includes downloading its container image when the image was not already on the machine. That happened on 61 Polaris Code tasks and 9 Claude Code tasks. On the 437 tasks where neither needed a download, the average was 44 s against 134 s (3.0x).</li><li>A task was run again only when the run failed for an infrastructure reason, such as a container that did not start. The same rule applied to both agents.</li><li>The first 150 tasks used an earlier build of Polaris Code than the other 350.</li><li>Costs are estimated from token counts at Bedrock list prices.</li></ul><h3>Try it</h3><p>Install Polaris Code with Homebrew on Apple Silicon macOS or Linux x86–64:</p><pre>brew install radiantailabs/tap/polaris-code</pre><p>Or with Cargo on any platform:</p><pre>cargo install polaris_code --locked</pre><p>The source, and the code and pinned inputs to re-run this comparison, are on <a href="https://github.com/RadiantAILabs/polaris_code">GitHub</a>. See the <a href="https://github.com/RadiantAILabs/polaris_code/tree/main/evals/reproduce/swebench-opus">reproduction guide</a>.</p><p><em>Originally published on the </em><a href="https://radiantlabs.dev/blog/polaris-code-swe-bench-tokens"><em>Radiant Labs blog</em></a><em> on 2 November 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cbd14c4eae8c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How To Harness The Power Of Speech With Artificial Intelligence]]></title>
            <link>https://medium.com/data-science/how-to-harness-the-power-of-speech-with-artificial-intelligence-ced8dc64721d?source=rss-e2fd3d2fec31------2</link>
            <guid isPermaLink="false">https://medium.com/p/ced8dc64721d</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[speech-synthesis]]></category>
            <category><![CDATA[text-to-speech]]></category>
            <dc:creator><![CDATA[Min Kim]]></dc:creator>
            <pubDate>Thu, 21 Feb 2019 21:59:01 GMT</pubDate>
            <atom:updated>2019-02-26T10:00:15.091Z</atom:updated>
            <content:encoded><![CDATA[<p>Have you ever heard of the Turing test? Turing test is a test designed by Alan Turing in 1950 to see whether a machine can exhibit conversational behaviours that are indistinguishable from that of a human.</p><p>When <a href="https://en.wikipedia.org/wiki/Alan_Turing">Alan Turing</a> first developed the <a href="https://en.wikipedia.org/wiki/Turing_test">Turing Test</a>, he strictly subjected the experiment to be a text-only conversation rather than a full speech conversation. Why? Well, he didn’t want the crappy robotic voice of the 1950s to give away the clue behind “who’s the robot?” test. But the text-only rule stayed there for many years and decades to come since the test was first conceived. Many computer Scientists believed that we were actually closer to building a fully conversational agent that acts just like humans than building a realistic text-to-speech system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q0v3tB7fynG8nkCWrGulJg.jpeg" /></figure><h3>Text-to-Speech (TTS)</h3><p>First of all, what is Text-to-Speech? TTS is a speech synthesis system that can convert natural language into audible speech. The chances are, you’ve probably talked to chatbots that employ TTS system multiple times. The voice behind Siri? Amazon Alexa? Google Home? Yes, those voices are in fact generated synthetic voice (well mostly). There are many ways how TTS systems work.</p><p>In this blog, we will specifically focus on how <strong>neural TTS</strong> works. Let’s dive into how speech synthesis can be done!</p><h4>Concatenative Approach</h4><p>Concatenative approach, mainly unit selection method is the most commonly and widely used technique for speech synthesis. Concatenative models retrieve short speech segments from a very large database of pre-recorded audio and concatenate multiple segments to generate an audible speech. However, this approach is not scalable as they require a new dataset every time a different style of speech generation is required. Also because of the nature of the method, the synthesized audio doesn’t have the consistency and can sound very robotic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*frjL5h0I714N47Su2w8n0w.png" /><figcaption>Source: <a href="https://www.explainthatstuff.com/how-speech-synthesis-works.html">https://www.explainthatstuff.com/how-speech-synthesis-works.html</a></figcaption></figure><h4>Parametric Approach</h4><p>The issue of scalability has lead to the use of parametric models where the generation of speech can be controlled by how the input is defined. Most of the parametric models are focused on generating acoustic features from the given text inputs and by using a set of algorithms known as vocoders, converting them into an audio signal.</p><h3>Neural Text-To-Speech (TTS)</h3><p>In this blog, we will specifically look at <a href="https://arxiv.org/pdf/1703.10135.pdf">Tacotron</a>, a popular neural network that is often reimplemented in many machine learning frameworks by TTS followers (Often multiple times)! Tacotron team has published quite a few <a href="https://google.github.io/tacotron/">papers</a> on neural TTS so we will look at the core part of those published papers, Tacotron!</p><h4>Text to Acoustic Features</h4><p>Most of the parametric TTS models have text to acoustic feature approximation stage. This is where text input is converted into dense acoustic features that can be used by vocoders to generate audio signals. In neural TTS, we care less about the semantics of text and more about the spatial and morphological structures that form words (phonemes and graphemes). Character level (or even phoneme level) 1-d convolutions are often used to extract this information. Because we care less about the semantics, the need for word embeddings is reduced and using word level tokens are not used often. (Figure below)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qOXM-kULRwNI1l8_w3Kjog.png" /><figcaption>Source: <a href="https://arxiv.org/pdf/1703.10135.pdf">https://arxiv.org/pdf/1703.10135.pdf</a></figcaption></figure><p><strong><em>Note</em></strong><em>:</em> Interestingly, it is recently <a href="https://arxiv.org/pdf/1808.10128.pdf">studied</a> that one could achieve an effect of semi-supervised training of these networks by conditioning the characters with corresponding word embeddings to inject semantic knowledge. This suggests that semantic contexts can be used in conjunction with morphological information to help better extract acoustic features. It could also suggest that word embeddings also contain spatial and structural information. (Figure below)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zwa5WEH7bop-SJ-lQY2inQ.png" /><figcaption>Source: <a href="https://arxiv.org/pdf/1808.10128.pdf">https://arxiv.org/pdf/1808.10128.pdf</a></figcaption></figure><p>Tacotron follows the standard approach, where the network has an encoder-decoder structure. In the encoder, 3 layers of character-wise convolutional neural networks are adopted to extract long term contexts such as the morphological structures from the character sequence of text. Originally followed by max pooling and highway layer, but later removed. Batch normalization and ReLU activation are used. Then the local features are combined in a sequential manner using a single layer of bidirectional RNN.</p><p>On the decoder side, the extracted representation is then converted into dense acoustic features (spectrogram) that can later be converted back into audio using different algorithms called vocoders. With attending to the context vector generated by the <a href="https://arxiv.org/pdf/1506.07503.pdf">location-sensitive attention mechanism</a> at each timestep, stacked RNN layers are used to predict each frame (or frames) conditioned on the previous hidden state and the text representation extracted from the encoder. Training Tacotron this way, full acoustic features can be approximated from text features at inference time without specific alignment algorithms used in HMM-based <a href="https://www.cs.cmu.edu/~pmuthuku/mlsp_page/lectures/spss_specom.pdf">parametric models</a>.</p><p><strong><em>Note:</em></strong> One interesting point to think about is the learned linguistic and acoustic representation. Since Tacotron extracts linguistic features such as character and phoneme alignments and uses them to generate spectrograms, the intermediate representation could potentially contain both rich acoustic and linguistic features. Could this information be used for other related tasks? Also what about speaker representation? <a href="https://arxiv.org/pdf/1803.09047.pdf">This paper</a> talks about training Tacotron with speaker and prosody embeddings. Can we generate an arbitrary human voice that doesn’t exist on earth by sampling from this continuous space?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/1*H5PM2kNyJJCviOOH3-b9rQ.png" /><figcaption>Source: <a href="https://arxiv.org/pdf/1803.09047.pdf">https://arxiv.org/pdf/1803.09047.pdf</a></figcaption></figure><h4>Vocoders</h4><p>So how do we convert the obtained acoustic features (ie. spectrograms) back to audio signals? It turns out that artificially generated spectrograms are often over-smoothed and lacks specific information required to create high fidelity audio signal for the traditional vocoders. This is in part due to the quality of phase vocoder, accuracy in acoustic feature modelling, and the over-smoothing of spectrograms (Because traditional vocoders are mel-cepstrum based and has simple phase reconstruction algorithms). This results in extracted spectrograms lacking phase information necessary to reconstruct speech from spectrograms. This means that synthesized speech <a href="https://www.cs.cmu.edu/~pmuthuku/mlsp_page/lectures/spss_specom.pdf">often sounds muffled, and has artifacts</a>.</p><h4>Wavenet</h4><p>To address the issues above, Google DeepMind published a paper in 2016, namely “<a href="https://arxiv.org/pdf/1609.03499.pdf">Wavenet</a>” which can generate audio signals, not from the spectral aspect but fully sample autoregressively using stacked layers of dilated convolutional neural network. The audio generated by Wavenet doesn’t display artifacts and muffledness as they are not reconstructed in parallel manner from spectrogram but rather conditioned on the previous audio samples. There are amazing blog posts discussing Wavenet in depth (including <a href="https://deepmind.com/blog/wavenet-generative-model-raw-audio/">the one from the authors</a>), so I won’t go into the details of how Wavenet works. As a result of this new clever architecture, Wavenet achieves a state-of-the-art synthesised audio quality that is still unmatched by any other architectures.</p><p>One thing I’d like to talk about Wavenet is the flexibility and its capability to condition the network on local features and generate speech from them. <a href="https://arxiv.org/pdf/1609.03499.pdf">The original paper</a> takes other acoustic features into consideration such as logarithmic fundamental frequencies and linguistic features such as phoneme alignments etc. However, later it is shown that simple acoustic features such as mel-scale spectrograms can be conditioned on directly by Wavenet to generate an even better quality of speech (<a href="https://arxiv.org/pdf/1712.05884.pdf">link to the paper</a>). This suggests that mel spectrograms subjected to powerful architectures such as Wavenet, is an exceptional audio-acoustic feature to use for audio related tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/766/1*sSUDvtEGccXh2-gMX60TzA.png" /><figcaption>Source: <a href="https://arxiv.org/pdf/1712.05884.pdf">https://arxiv.org/pdf/1712.05884.pdf</a></figcaption></figure><h3>Conclusion</h3><p>We’ve looked at the new standard way of extracting acoustic features from text, and then using them to generate high fidelity speech. We went through the two main models used for neural Text-to-Speech in high level. Here are the key takeaways:</p><ul><li>Semantic contexts in text are not necessary for audio generation task but somewhat useful if the size of data is small</li><li>Linguistic and acoustic feature can be concatenated with continuous distributions such as speaker embeddings and prosody embeddings to give more inductive bias to Tacotron model</li><li>Mel-scale spectrograms are very useful and compact audio acoustic features that are capable of generating high fidelity speech when conditioned on Wavenet variants.</li></ul><p>Neither authors of Tacotron and Wavenet released the code for the papers, but there are amazing implementations of both Tacotron and Wavenet on Github which you can find <a href="https://github.com/Rayhane-mamah/Tacotron-2">here</a> and <a href="https://github.com/r9y9/wavenet_vocoder">here</a> respectively. We’ve covered high level pipeline of neural TTS, however if you’d like to go deeper into this matter, please do have a look at the papers mentioned in this blog. Also if you think I’ve forgotten to add a valuable information regarding neural TTS, or would like to discuss more about it, please don’t hesitate to contact me at minsang.kim.142@gmail.com</p><p>Thanks for reading, and please leave questions or feedback in the comments!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ced8dc64721d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-to-harness-the-power-of-speech-with-artificial-intelligence-ced8dc64721d">How To Harness The Power Of Speech With Artificial Intelligence</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Implementing QANet (Question Answering Network) with CNNs and self attentions]]></title>
            <link>https://medium.com/data-science/implementing-question-answering-networks-with-cnns-5ae5f08e312b?source=rss-e2fd3d2fec31------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ae5f08e312b</guid>
            <category><![CDATA[convolutional-network]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[question-answering]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Min Kim]]></dc:creator>
            <pubDate>Mon, 16 Apr 2018 05:45:29 GMT</pubDate>
            <atom:updated>2020-03-09T13:15:31.240Z</atom:updated>
            <content:encoded><![CDATA[<p>In this post, we will tackle one of the most challenging yet interesting problems in Natural Language Processing, aka Question Answering. We will implement Google’s <a href="https://openreview.net/pdf?id=B14TlG-RW">QANet</a> in Tensorflow. Just like its machine translation counterpart <a href="https://arxiv.org/abs/1706.03762">Transformer network</a>, QANet doesn’t use RNNs at all which makes it faster to train / test. (The source code for this experiment can be found <a href="https://github.com/NLPLearn/QANet">here</a>)</p><p>I’m assuming that you already have some knowledge of python and Tensorflow.</p><p>Question Answering is a field in computer science that has seen some rapid progress in the past few years. A classic example of question answering is <a href="https://en.wikipedia.org/wiki/Watson_(computer)">IBM’s Watson</a> competing at the famous quiz show <a href="https://en.wikipedia.org/wiki/List_of_Jeopardy!_tournaments_and_events#IBM_Challenge">Jeopardy!</a> in 2011, facing off legendary champions Brad Rutter and Ken Jennings and winning the first place prize.</p><p>Here in this post, we will focus on open domain reading comprehension, where the questions can come from any domain ranging from <a href="https://en.wikipedia.org/wiki/Beyonc%C3%A9">American pop stars</a> to <a href="https://en.wikipedia.org/wiki/Computational_complexity_theory">abstract concepts</a>. Reading comprehension is a type of question answering where we are given a paragraph and asked questions specifically chosen to be answered from the paragraph.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*_455y1Iq5RUPvnVsQkPAHw.jpeg" /><figcaption>IBM Watson competing against Ken Jennings (left) and Brad Rutter (right) at Jeopardy! in 2011. Source: <a href="https://blog.ted.com/how-did-supercomputer-watson-beat-jeopardy-champion-ken-jennings-experts-discuss/">https://blog.ted.com/how-did-supercomputer-watson-beat-jeopardy-champion-ken-jennings-experts-discuss/</a></figcaption></figure><h3>Dataset (SQuAD)</h3><p>The dataset we will be using for this post is called <a href="https://arxiv.org/pdf/1606.05250.pdf">Stanford Question Answering Dataset (SQuAD)</a>. SQuAD has some problems which we will come back to, and it is arguably not the best dataset for machine reading comprehension but the most widely studied. If you are curious about different datasets available for reading comprehension, also check out this <a href="https://github.com/karthikncode/nlp-datasets#question-answering">Awesome list</a> of NLP datasets.</p><p>One of the differentiating factors of SQuAD is that the answer to the question lies within the paragraph itself. Here is an example of SQuAD format.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TXOB-ClnyNUI5tTmJdP1MQ.png" /><figcaption>An example of Stanford Question Answering Dataset. Source: <a href="https://rajpurkar.github.io/SQuAD-explorer/">https://rajpurkar.github.io/SQuAD-explorer/</a></figcaption></figure><p>As we can see from the example above, no matter how easy or difficult the question may be, the answer to that question always lies within the paragraph itself.</p><p>But first, let’s have a look at the kinds of questions and answers we are expecting to solve. More often than not, the questions themselves are already paraphrased versions of the segments of the paragraphs. For example,</p><blockquote>P: “Computational complexity theory is a branch of the theory of computation in theoretical computer science that focuses on classifying computational problems according to their inherent difficulty, and relating those classes to each other. A <strong>computational problem</strong> is understood to be a task that is in principle<strong> amenable to being solved by a computer</strong>, which is equivalent to stating that the problem may be solved by mechanical application of mathematical steps, such as an algorithm.”</blockquote><blockquote>Q: “What is the term for a task that generally lends itself <strong>to being solved by a computer?</strong>”</blockquote><blockquote>A: “computational problem”</blockquote><p>From the words highlighted in bold, it is clear that the question is paraphrased from the paragraph. This property makes SQuAD problem inherently easier than an open-domain question answering problem. Because all we need to do now is to find a sentence from the paragraph that semantically matches the question and extract the common semantic or syntactic factor from the context. Although we still require solving semantic and syntactic extraction, this is much easier than deducting the answer from a dictionary that may have tens of thousands of words.</p><h4>Downside of SQuAD</h4><p>The property mentioned above lets us use a few tricks to easily predict answers from a given paragraph. However, this also introduces some problems to the model trained with SQuAD. Because the models heavily rely on finding the correct sentence from the paragraph, they are vulnerable to adversarial sentences inserted to the paragraph that resembles the question but designed to fool the networks. Here is an example,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*p-dKzIJBTFTT4KoZl1q3LA.png" /><figcaption>Adversarial example of SQuAD. Source: <a href="https://arxiv.org/pdf/1707.07328.pdf">https://arxiv.org/pdf/1707.07328.pdf</a></figcaption></figure><p>The sentence highlighted in blue is the adversarial example inserted to fool the network. To human readers, it doesn’t change the answer to the question “What is the name of the quarterback who was 38 in Super Bowl XXXIII?” as the adversarial sentence is talking about Champ Bowl XXXIV. However, to the network, the adversarial sentence aligns better with the question than the ground truth sentence.</p><h3>The Model Network (QANet)</h3><p>The reason behind our choice of the model QANet is simple. Thanks to its straightforward architecture it is easy to implement and faster to train than most of the networks for the same task. QANet architecture can be seen from the figure below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/804/1*AJZICMvoPbn-x_dTIy2L4A.png" /><figcaption>The network architecture overview. Source: <a href="https://openreview.net/pdf?id=B14TlG-RW">https://openreview.net/pdf?id=B14TlG-RW</a></figcaption></figure><p>The model network can be separated into roughly 3 sections.</p><ol><li>Embedding</li><li>Encoder</li><li>Attention</li></ol><p>Embedding is where text inputs (paragraphs and questions) are converted into representations in the form of dense low-dimensional vectors. This is done via an approach similar to <a href="https://arxiv.org/pdf/1508.06615.pdf">this paper</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*Y7-N1u22LMyZlorTWBBqBw.png" /><figcaption>Character aware language modelling. Source: <a href="https://arxiv.org/pdf/1508.06615.pdf">https://arxiv.org/pdf/1508.06615.pdf</a></figcaption></figure><p>Our approach is very similar. The only difference is that we use a fixed kernel size of 5 for our convolutional filter. We also concatenate the word representation with the max pooled character representation before we put them to the highway network.</p><p>The encoder is the basic building block of the model. The details of the encoder block can be seen from the right side of the figure above. Encoder consists of a positional encoding, layer normalization, depthwise separable 1d-convolution, self-attention and feed-forward layers.</p><p>Finally, attention layer is the core building block of the network where the fusion between question and paragraph occurs. QANet uses trilinear attention function used in <a href="https://arxiv.org/pdf/1611.01603.pdf">BiDAF paper</a>.</p><p>Let’s get started!</p><h3>Implementation</h3><p>For simplicity, we skip the data processing step and jump straight into the neural network.</p><h4>Embedding</h4><p>First, we define the input placeholders. Once we define the placeholders, we embed the word inputs with word embeddings, and character inputs with character embeddings.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/40b4b8658d389dc18901cceb137e2292/href">https://medium.com/media/40b4b8658d389dc18901cceb137e2292/href</a></iframe><p>Then we put them through a 1 layer 1-dimensional convolutional neural network, max-pooling, concatenate word + char representations and finally put them through 2 layer highway networks. The reason why we put “reuse” argument in “conv” and “highway” functions is that we want to use the same network for BOTH paragraph and questions. “conv” and “highway” functions are our Tensorflow implementation of convolution network and highway network. (The source code can be found here <a href="https://github.com/NLPLearn/QANet">https://github.com/NLPLearn/QANet</a>)</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dd6f6684fbb13413e00127e0379da23c/href">https://medium.com/media/dd6f6684fbb13413e00127e0379da23c/href</a></iframe><p>We put the outputs of the embedding layer into the encoder layer to generate the corresponding context and question representation. “residual_block” implements positional encoding -&gt; layer_normalization -&gt; depthwise separable convolution -&gt; self attention -&gt; feed forward network.</p><p>Now that we have context and question representation, we fuse them together using an attention function called trilinear attention. The fused output now has rich information of <strong>the context with respect to the question</strong>. Again, we concatenate the context-to-question and question-to-context information along with the context and pass on as the next input to the encoder layer.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/be95202bd0fcb481170e52dee26d210d/href">https://medium.com/media/be95202bd0fcb481170e52dee26d210d/href</a></iframe><p>Finally, we have the output layer which takes in the attention output and encodes them into a dense vector. This is where the property of SQuAD comes in useful. Because we know that the answer is somewhere inside the paragraph, for each word in the paragraph we just need to calculate the probability the word being the answer or not. Practically, we calculate two probabilities. A probability that the corresponding word belongs to the beginning of the answer and a probability that the corresponding word belongs to the end of the answer span. This way, we don’t need to figure out what the answer may be from a large dictionary and efficiently calculate the probability.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0f7a075ceafa114871b6c3077af10419/href">https://medium.com/media/0f7a075ceafa114871b6c3077af10419/href</a></iframe><p>That’s it!</p><h4>Training and demo</h4><p>QANet trains relatively quickly compared to other RNN based models. Compared to the popular <a href="https://allenai.github.io/bi-att-flow/">BiDAF network</a>, QANet trains roughly 5~6 times faster with better performance. We train the network for 60,000 global steps which take around 6 hours in GTX1080 GPU.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bUgjSW6K-KHMsOKsouYLug.png" /><figcaption>Visualizing results in Tensorboard. The top plots are the devset results and the bottom are the training results. “em” is exact match, “f1” is F1 score.</figcaption></figure><p>That was quite minimalistic approach but I hope it helped you understand question answering with neural networks! If you have questions about them just leave a comment and I’ll try to answer your questions.</p><p>Thanks for reading, and please leave questions or feedback in the comments!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ae5f08e312b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/implementing-question-answering-networks-with-cnns-5ae5f08e312b">Implementing QANet (Question Answering Network) with CNNs and self attentions</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learning from mistakes with Hindsight Experience Replay]]></title>
            <link>https://becominghuman.ai/learning-from-mistakes-with-hindsight-experience-replay-547fce2b3305?source=rss-e2fd3d2fec31------2</link>
            <guid isPermaLink="false">https://medium.com/p/547fce2b3305</guid>
            <category><![CDATA[robotics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Min Kim]]></dc:creator>
            <pubDate>Wed, 31 Jan 2018 05:05:37 GMT</pubDate>
            <atom:updated>2023-10-14T00:07:18.567Z</atom:updated>
            <content:encoded><![CDATA[<p>Dealing with sparse rewards in Reinforcement Learning.</p><p><strong>Hindsight Experience Replay</strong> is a paper submitted by OpenAI to NIPS2017. For more details you can read the paper <a href="https://arxiv.org/pdf/1707.01495.pdf">here</a>. (Code for all experiments can be found <a href="https://github.com/minsangkim142/hindsight-experience-replay">here</a>)</p><h3><strong>Deep Q-Networks (DQN)</strong></h3><p>Q-Learning is a powerful reinforcement learning algorithm especially when combined with a powerful function approximator (such as deep neural networks)<strong> </strong>and other orthogonal techniques such as prioritized experience replay, double Q-learning, duelling networks and so on. When trained with carefully engineered reward functions, they are capable of achieving many tasks ranging from flying drones and toy helicopters to beating a top class world champion in Go.</p><p>However, oftentimes many of these tasks are achieved through carefully hand engineered reward functions with weights and hyperparameters chosen from multiple trials. Without telling the agent how well it is performing each run, there is no way for our agent to tell which actions are good or bad for achieving our goal.</p><h3>Reward Functions</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*42mSg0O-orLategHVltOBA.jpeg" /></figure><p>Let’s think of an example. Imagine that you are playing a game of hockey. You dribble your puck into your opponent’s net. The puck shoots toward the net but just misses it slightly to the right. Although you didn’t score the goal, maybe you learn that next time you should aim slightly towards left if you were in the same position.</p><p>If we were to simulate this with a standard reinforcement learning algorithm, it will most likely fail to learn much (if any at all). This is because our reward function is sparse (either 1 or 0, 1 if we score a goal and 0 if we don’t), our agent won’t learn how well it performed if it fails to put the puck in the net (which is the most likely outcome). To mitigate this problem we could introduce new terms to the reward function as follows:</p><blockquote><em>Reward Function = </em><strong><em>w1</em></strong><em> * goal + w2 * distance between the puck and the stick + w3 * speed of the puck + w4 * distance between the puck and opponent’s net + w5 * …</em></blockquote><p>where each weight <strong>w</strong> is carefully chosen by trial and error with multiple training runs. These added reward functions are called <strong>shaped rewards (or composite rewards) </strong>and are used often in robotic manipulation tasks. For example,<a href="https://arxiv.org/abs/1704.03073"> <em>Popov et al. (2017)</em></a><em> </em>uses 5 complex terms in their reward function with carefully chosen weights in order to train the policy that stacks bricks on top of another. Using composite reward terms not only help your agent learn which actions are helpful for achieving the final goal but also acts as a good indicator of how well our agent is performing each run.</p><p>Okay, now we know how to shape reward functions for simple and intuitive problems. We can just come up with an intuitive <em>middle goal/reward </em>for our agent that we think might help them achieve the <em>final goal</em>. And train the network on the new reward function and repeat adding new terms or change the weights on the reward terms until they start learning something. But what if there is a problem that cannot be solved this way? What if the problem we are solving is too non-intuitive and complex that adding a reward term is not a feasible option?</p><h3>Hindsight Experience Replay</h3><p>One ability humans have is to learn from our mistakes and adjust next time to avoid making the same mistake. We can apply the same concept to our reinforcement learning algorithm. Let’s go back to the hockey example.</p><p>Now this time, instead of concluding that the course of action you took was useless because you didn’t score a goal, what if you say that maybe it didn’t teach you how to score a goal, but it certainly taught you how <strong>NOT</strong> to shoot the puck. Or more precisely, what if you say it taught you how to shoot the puck slightly to the right side of the net? Now you can learn not only how to shoot towards the right, but you learn something new that <strong>might</strong> help you achieve the final goal every run. This is the idea behind <a href="https://arxiv.org/abs/1707.01495">Hindsight Experience Replay</a> (HER).</p><h3>Let’s formally define Q-Learning</h3><p>In Deep Q-Learning, the main objective we want to learn is called the Q function. Q function (or an action value function) measures how helpful taking each action is, in terms of achieving the goal, given our current situation/position. The equation for Q function looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*CJA9mT2XlPttEzjaBBFiBg.png" /></figure><p>Where <strong><em>Q(s,a)</em></strong> is the Q function of action <strong>a</strong> given our current state <strong>s</strong>, <strong>r(s,a)</strong> is the reward for taking action <strong>a</strong> given our current state <strong>s, s’</strong> and <strong>a’</strong> are the next state the next action taken respectively. <strong>γ</strong> is the discount factor, determined by how important your immediate reward is compared to the future rewards.</p><p>You can train a deep Q network following this simple procedure:</p><ol><li>Run your policy and store everything you observe (state, action, reward, next state) into an experience buffer. (Do this <strong>N</strong> times)</li><li>Sample M number of experience tuples (batch) from the buffer and train the network with the said batch experience. (Do this B times)</li><li>Repeat step 1–2 until the network converges.</li></ol><h3>Let’s add Hindsight Experience Replay into the training procedure</h3><p>Now if we want our agent to perform well even with sparse rewards, we need to make a slight modification to the procedure above. The new procedure looks like this:</p><ol><li>Run your policy and store everything you observe (state <strong>and goal</strong>, action, reward, next state <strong>and goal</strong>) tuple into an experience buffer.</li><li>Sample <strong>K</strong> goals from the states visited in the episode obtained at step 1, and for each goal store (state <strong>and sampled goal</strong>, action, <strong>reward with respect to the sampled goal</strong>, next state <strong>and sampled goal</strong>) tuple into the buffer.</li><li>Repeat step 1–2 N times.</li><li>Sample M number of experience tuples (batch) from the buffer and train the network with the said batch experience. (Do this B times)</li></ol><p>How does this help? By adding goal into the input space we are stating that there are multiple goals for our agent to observe. The new Q-function indicates how good taking each action is, given the current state, <strong>to achieving the current goal</strong>. And because we are sampling the new goals <strong>from the episode</strong>, the goals are now nodes that have been visited by our agent. So even if our agent fails to achieve the main goal in this episode, it has reached some states. By using those states as the new goal, now the agent can be trained with positive (or non negative) rewards.</p><h3>Experiments</h3><p>The original paper mentions a simple experiment to validate the idea. Imagine a single arbitrary binary vector of length <strong>n</strong>. Let’s call it an initial state. We can modify a single bit at a time (flip it from 0 to 1 or vice versa), and reach our final goal which is also an arbitrary binary vector of length <strong>n</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/548/1*aj3zk-4aUfJV22fPIT-xUw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/534/1*522uy95PhFsCgeHa8i_t_A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/506/1*QTnjAO_5ebYCf0h33WATOA.png" /></figure><p>At each step, the agent receives a reward of -1 if the state is not equal to the final goal, and 0 otherwise. Each episode runs n times and terminates at the end of <strong>n th</strong> step. If we train a standard reinforcement learning algorithm on this environment, they won’t learn anything even with <strong>n &gt; 15</strong> because the agent simply never observes any reward other than -1. Because there are too many states to visit, it is simply impractical for our agent to explore the large state space (the number of possible states increase exponentially with the size of <strong>n</strong>.)</p><p>See the figures below and compare the “Success Rate” between two models. The first figure shows the model trained with standard DQN training, the second figure is after adding hindsight experience replay. Both experiments are done in bit length <strong>n = 15</strong> (the code for this experiment can be found <a href="https://github.com/minsangkim142/hindsight-experience-replay">here</a> in Github.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*PF5ruaYvtWdzycOwD_6uNw.png" /><figcaption>(n = 15). With standard DQN training, the model doesn’t learn anything useful. Our agent will never experience any reward other than -1.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*y82evsX_KGDAPlIVfdCFZg.png" /><figcaption>(n = 15). With HER modification to DQN training procedure.</figcaption></figure><p>Once we train the model sufficiently, we can run inference to observe step by step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/1*1EpVONAueFnHuZMD7dSDKg.png" /><figcaption>At inference.</figcaption></figure><h3>Conclusions</h3><p>As expected, even with a small bit length such as <strong>n = 15</strong>, the standard DQN algorithm fails to learn. We can clearly see that with hindsight experience replay modification, our agent can learn from such large action space without shaped rewards to guide it.</p><p>One interesting idea we surmise from hindsight experience replay paper is that having a shaped reward may deviate the agent from achieving the true goal. The shaped reward is either not helpful to achieving the goal or punishes our agent too much for exploration.</p><p>Being able to learn from a sparse reward is a very interesting idea that has huge potentials for robotic manipulation. Check out this video published by OpenAI, a robot arm trained with a sparse reward function and hindsight experience replay being able to perform three different tasks (push, slide and pick up + drop) with high success rate.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FDz_HuzgMxzo%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DDz_HuzgMxzo&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FDz_HuzgMxzo%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/4b8e36b4983f1701f161741fce6fb2b8/href">https://medium.com/media/4b8e36b4983f1701f161741fce6fb2b8/href</a></iframe><p>Thanks for reading and feedbacks and questions are always welcome! I might have made mistakes as I’m still learning this very fast moving field of reinforcement learning!</p><p>p.s. If you have any question or would like to discuss more about reinforcement learning all emails are welcome at minsang.kim.142@gmail.com</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2F8f5f8b%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2F8f5f8b&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/c43026df6fee7cdb1aab8aaf916125ea/href">https://medium.com/media/c43026df6fee7cdb1aab8aaf916125ea/href</a></iframe><figure><a href="https://becominghuman.ai/artificial-intelligence-communities-c305f28e674c"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*2f7OqE2AJK1KSrhkmD9ZMw.png" /></a></figure><figure><a href="https://upscri.be/8f5f8b"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*v-PpfkSWHbvlWWamSVHHWg.png" /></a></figure><figure><a href="https://becominghuman.ai/write-for-us-48270209de63"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*Wt2auqISiEAOZxJ-I7brDQ.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=547fce2b3305" width="1" height="1" alt=""><hr><p><a href="https://becominghuman.ai/learning-from-mistakes-with-hindsight-experience-replay-547fce2b3305">Learning from mistakes with Hindsight Experience Replay</a> was originally published in <a href="https://becominghuman.ai/">Becoming Human: Artificial Intelligence Magazine</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>