<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Rajat Maheshwari on Medium]]></title>
        <description><![CDATA[Stories by Rajat Maheshwari on Medium]]></description>
        <link>https://medium.com/@rajatmaheshwari?source=rss-a330064dfe03------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*x5J6zG7oi25BgqqH.</url>
            <title>Stories by Rajat Maheshwari on Medium</title>
            <link>https://medium.com/@rajatmaheshwari?source=rss-a330064dfe03------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 19:09:32 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rajatmaheshwari/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Learning from Human Preferences]]></title>
            <link>https://medium.com/@rajatmaheshwari/learning-from-human-preferences-d50f9d81ca1c?source=rss-a330064dfe03------2</link>
            <guid isPermaLink="false">https://medium.com/p/d50f9d81ca1c</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <dc:creator><![CDATA[Rajat Maheshwari]]></dc:creator>
            <pubDate>Thu, 02 Nov 2017 14:41:19 GMT</pubDate>
            <atom:updated>2017-11-02T14:41:19.236Z</atom:updated>
            <content:encoded><![CDATA[<p>In this data driven world, where everything will be automated in the coming future, AI models and specially how we train them play an important role. Reinforcement Learning is one of the major techniques in Deep Learning which is used to create good AI agents. Reinforcement learning (RL) is an area of machine learning inspired by behaviorist psychology, concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward.</p><p>Till now, we have been using complex reward functions defined by humans to perform sophisticated tasks. Designing these reward functions is generally a difficult task and sometimes they result in undesirable and even dangerous behavior or bad outcomes. What if we can remove the need for humans to define complex goal functions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/1*czgyQoX86o_sRn68ynEyjg.jpeg" /></figure><p>We can use human feedback to judge and train the AI model. In this methodology, AI agent starts by acting randomly in the environment. Periodically, two video clips of its behavior are given to a human, and the human decides which of the two clips is closest to fulfilling its goal.The AI gradually builds a model of the goal of the task by finding the reward function that best explains the human’s judgments. It then uses RL to learn how to achieve that goal. As its behavior improves, it continues to ask for human feedback, and further refines its understanding of the goal. This approach will be very much useful for seemingly simple task which is simple to judge but challenging to specify mathematically.</p><p>There are many cases where human feedback could let us specify a specific goal more intuitively and quickly than is possible by manually hand-crafting the objective.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d50f9d81ca1c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Basics of Swarm Intelligence.]]></title>
            <link>https://medium.com/@rajatmaheshwari/basics-of-swarm-intelligence-4f166fa8302d?source=rss-a330064dfe03------2</link>
            <guid isPermaLink="false">https://medium.com/p/4f166fa8302d</guid>
            <category><![CDATA[swarm-intelligence]]></category>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[travelling-salesman]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ants]]></category>
            <dc:creator><![CDATA[Rajat Maheshwari]]></dc:creator>
            <pubDate>Tue, 25 Jul 2017 19:44:41 GMT</pubDate>
            <atom:updated>2017-07-25T19:44:41.224Z</atom:updated>
            <content:encoded><![CDATA[<h3>It’s all about ants,bees &amp; birds (but in computer’s sense)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*A06MUYwNLjLxqJcVcLBIKA.jpeg" /><figcaption>Credits-Techrepublic.com</figcaption></figure><p>Wikipedia says- <strong>Swarm intelligence</strong> (<strong>SI</strong>) is the <a href="https://en.wikipedia.org/wiki/Collective_behavior">collective behavior</a> of <a href="https://en.wikipedia.org/wiki/Decentralization">decentralized</a>, <a href="https://en.wikipedia.org/wiki/Self-organization">self-organized</a> systems, natural or artificial. The concept is employed in work on <a href="https://en.wikipedia.org/wiki/Artificial_intelligence">artificial intelligence</a>. The expression was introduced by <a href="https://en.wikipedia.org/wiki/Gerardo_Beni">Gerardo Beni</a> and Jing Wang in 1989, in the context of cellular robotic systems.<a href="https://en.wikipedia.org/wiki/Swarm_intelligence#cite_note-1">[1]</a>.</p><p>But I think <strong>it’s just learning from animals, how they find their food</strong>. So that we can also apply those techniques to find our food (or solution of a problem).</p><p>I’ll discuss one very common technique- <strong>Ant colony optimisation</strong> (ACO), there’s one other good technique namely, <strong>Particle Swarm Optimization</strong> (PSO), which I will discuss some other time.</p><blockquote><strong>#ANT COLONY OPTIMIZATION</strong></blockquote><p><strong>Basic Idea- </strong>Have you ever seen that ants usually walk in a disciplined manner, in a line or something like that and how they always find a shortest (maybe not always) path between their nest and food source. So the ACO’s idea is pretty much similar. When searching for food, ants <strong>initially explore</strong> the area surrounding their nest in a <strong>random</strong> manner. Actually they also leave a <strong>magic portion</strong> &amp; Ants can also smell that magic portion. When ants choose their way, they tend to choose, in probability, paths marked with strong smell. As soon as an ant finds a food source, it carries some of it back to nest. During the return trip, the quantity of magic portion ant leaves depends on the quality and quantity of food. So tasty and lots of food made them leave more portion. And this portion will guide the other ants. And that magic portion is <strong>Pheromone.</strong></p><p>Now imagine, two and more ants find that good food source, and now each of them returning back to their home, those who follows a shorter path will be able to make more trips from food source and nest, and hence the quantity of pheromone on these paths will also be greater than other paths and in some time more ants will follow the same path and leave more pheromone. After some time all ants will follow the same path.</p><p>Now, let’s break down the process in computer’s sense and see how a computer algorithm can solve such problem.</p><p>Prerequisites before solving a problem using ACO (If you’re not able to understand these, read the remaining part and then revisit this section) -</p><ul><li>Problem solution can be obtained from a finite set<strong> C</strong> of <strong>solution components</strong>- i.e you should be able to divide the problem’s solution into smaller solutions, and then we’ll combine some of these smaller solutions to get our main solution.</li><li>We need to define a<strong> pheromone model, </strong>this will be a probability function which will help us decide which solution component to pick depending on the pheromone values of these components.</li><li>And an <strong>update function</strong> which will have the pheromone evaporation and deposition quantity as a mathematical function or some heuristics.</li></ul><p><strong>Problem- </strong>We’ll work on a real life problem <a href="https://en.wikipedia.org/wiki/Travelling_salesman_problem"><strong>Travelling Salesman Problem (TSP)</strong> </a>, you’ll have some cities and the cost of travelling from each city to every other city, your aim is to find the minimum cost such that the salesman can visit every city and returned back to the one from which he started.</p><ol><li><strong>Converting into Graph- </strong>First, we’ll convert this problem, so that we can apply ACO on it. Let just assume each city is a node of a graph, and we have weighted (cost) edges from each node to every other node. Our aim is to find a closed path in graph G such that weight of all edges in our path is minimum.</li><li><strong>Constructing solution components- </strong>As the final solution is a sum of different edge weights, we can divide our solution into components and can treat each edge as a solution component.</li><li><strong>Some initializations- </strong>We will start by giving our ants some random starting points(in this case nodes) and we also need to set some initial pheromone values to each of the solution components (or edges).</li><li><strong>Defining a pheromone model-</strong></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/490/0*lXkPpQMKlGWFixg4.png" /></figure><p>e(i,j)- edge between nodes i &amp; j<br> tau(i,j)- pheromone value of edge between nodes i &amp; j<br> T- memory of an ant (list of already visited nodes/cities)<br> p (e(i,j)) — probability of an ant to pick edge between nodes i &amp; j</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/403/0*RLONYvAYbPk1IzGB.png" /></figure><p>this is a more general form of the probability function, here we also take into account the attractiveness of a solution component (ci), but we don’t need this for our problem, so just leave it for now. You can refer the paper I’ve attached in the footnotes to know more about this.</p><p><strong>5. Defining an update Function-</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/268/0*tGCD49_h_V_GC9ZU.png" /></figure><p>We’ll use this for <strong>evaporation of pheromone</strong>, it’s simply just decreasing the pheromone at edge (i,j) by rho factor. Now, one would ask why we even need to evaporate pheromone, we could just deposit pheromone on components which give us good solution because <strong>Pheromone evaporation is needed to avoid too rapid convergence of the algorithm towards a sub-optimal region</strong>, think about it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/191/0*uOhKOmQ3xBAhoCY5.png" /></figure><p>After creating a solution by an ant, let solution components set of this solution be<strong> s</strong>, we’ll calculate the target value of our set s or you could say the cost in this case. If a set <strong>s</strong> gives us bad value of f(s) i,e, more cost, the change in tau(i,j) will be less and vice versa in other case. So all of this will favour solution components who are part of good solutions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/357/0*SapJss6ps5g0u35R.png" /></figure><p>This again , is a general form of update function, left part is for evaporation (same, no changes), interesting thing is right part which is for deposition. Here one can use many techniques like AS update, IB update, BS update rule, Rank based AS, MAX-MIN and other kind of rules. Just for your curiosity I’ll describe IB update rule, for others you can refer the paper (link in footnotes). In IB update (or iteration best update) we deposit pheromones only on the solution components which are part of the best solution in an iteration. After all the ants in an iteration have created solutions, we pick the best one and update pheromones only on those solution components which are part of this solution (i.e why iteration best).</p><blockquote><strong>Pseudo Code-</strong></blockquote><p>Initialise Pheromone values and other constants</p><pre>1 Initialise pheromone values and other constants</pre><pre>2 <strong>While</strong> (max_iterations) <strong>do</strong><br>       <br>   2.1 assign random starting positions to ants<br>       <br>   2.2 <strong>For</strong> each ant <strong>do</strong><br>            build tour using pheromone model and roulette based       <br>            selection (see code for reference)<br>       <strong>End</strong><br>      <br>   2.3 Calculate cost of each tour</pre><pre>   2.4 Update solution components using evaporation and deposit  <br>       function</pre><pre>   2.5 If you shorter tour is found update <strong>best</strong></pre><pre><strong>End</strong></pre><pre><strong>return best</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/1*szzu-FbmxhpO_u16eSmdSg.jpeg" /><figcaption><a href="https://github.com/tommytrash00/ACO_for_tsp">https://github.com/tommytrash00/ACO_for_tsp</a></figcaption></figure><p>I’ve pasted one solution I got using this technique on 20 cities, for 50 or 100 cities check my <a href="https://github.com/tommytrash00/ACO_for_tsp"><strong>Github</strong></a> link.</p><p>For reference, and other techniques, use this review paper-<a href="https://bitchuckie.files.wordpress.com/2017/07/antcolonyopt.pdf"><strong>antcolonyopt</strong></a><br>For <strong>code</strong> check my <a href="https://github.com/tommytrash00/ACO_for_tsp"><strong>Github</strong></a></p><p>Hope you liked this basic tutorial, I tried not to get into the depths of the topic but at the same time, good enough information so that you can work on some real problem.<br>This is my first such blog, I’m planning to add more blogs/tutorials related to artificial intelligence or other related topics or anything I found interesting.</p><p><strong>P.S.-</strong> You can also use <a href="https://github.com/tommytrash00/TSP-using-simulated-annealing"><strong>this</strong></a> technique to achieve the same, there’s also a <a href="https://www.youtube.com/watch?v=3BgLwl41aco"><strong>video</strong></a><strong>.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4f166fa8302d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>