<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Arnav on Medium]]></title>
        <description><![CDATA[Stories by Arnav on Medium]]></description>
        <link>https://medium.com/@itsarnavsh?source=rss-23f455523dfb------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*YfDrg4s3EGlY31oxoxCc7g.jpeg</url>
            <title>Stories by Arnav on Medium</title>
            <link>https://medium.com/@itsarnavsh?source=rss-23f455523dfb------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 20:29:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@itsarnavsh/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Gaslighting AI Agents into being better]]></title>
            <link>https://itsarnavsh.medium.com/gaslighting-ai-agents-into-being-better-b7ac1718c765?source=rss-23f455523dfb------2</link>
            <guid isPermaLink="false">https://medium.com/p/b7ac1718c765</guid>
            <category><![CDATA[agentic-ai-pipeline]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Arnav]]></dc:creator>
            <pubDate>Wed, 23 Sep 2026 18:57:58 GMT</pubDate>
            <atom:updated>2026-09-23T19:04:16.882Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/479/1*vOQJfRTXrIWTCkP5C6-h9A.png" /></figure><p>Multi-Agent debates are very intimidating yet interesting. The logic is intuitive, create several agents specialized in their domains, by the means of domain specific prompts, access to certain databases via RAG or OLAP, or perhaps based off of different models entirely and then make them debate to so each can effectively reason better for their stance.</p><p>It can find its application in several tasks, I find it a perfect fit for more not so obvious correct answers, like in the case of Subjective disciplines, say History, Philosophy, etc. However they seldom give something meaningful that way but its interesting nonetheless. A recent case I came up across on was a GitHub repo for a Trading Bot which had a discuss step which can potentially avoid any internal biases. This is a very black and white scenario, two agents push towards the opposite ends and its rare for the Bearish Agent to have second thoughts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/226/1*hADD2zTmpuXajV_vd2WxVQ.png" /><figcaption><a href="https://github.com/TauricResearch/TradingAgents/blob/main/assets/schema.png">https://github.com/TauricResearch/TradingAgents/blob/main/assets/schema.png</a></figcaption></figure><p>Things get complicated however if we have multiple agents with a more varied answer, i.e. not just a yes or no but perhaps more options to pick from or a subjective answer. In these cases we cannot have one agent rooting for one option like in the above case. There can be cases where majority agents have a perspective which matches with a different option, and a few match with another one. This can cause correct agents at times to get “Gaslighted” in the literal sense by the other agents and pick a wrong side.</p><p>Another case, lets say you have a pipeline of 100 agents and one of them goes rogue probably hallucinates or has gaps in its tools and knowledge sources, or just faulty llm beneath which makes it reason incorrectly. What if it behaves like that one kid that confidently shouts out the wrong answer with full confidence, and makes even the front-benchers second guess their answers? How do we detect that?</p><p>And the issue is that this cannot be traced easily these pipelines are a</p><p>This is supposed to be an experimental pipeline which can help debug and observe multi-agent pipelines…trust me it is not as boring as it sounds!</p><h3>Baseline</h3><p>Okay to begin anything, we have to find what agents have been affected, we need a method to approach this scientifically.</p><p>So lets define a system with rules. The input and output must be simple, lets do one Question as input with its MCQ options, and it should return the probabilities it thinks each one is correct, along with a baseline reasoning. An example could be:</p><pre>{<br>  &quot;question&quot;: &quot;What was the single most decisive factor in the Western Roman Empire&#39;s collapse?&quot;,<br>  &quot;options&quot;: [<br>    &quot;Economic exhaustion from hyperinflation, currency debasement, and tax base flight&quot;,<br>    &quot;Military breakdown and over-reliance on untrustworthy foederati/barbarian mercenaries&quot;,<br>    &quot;Internal cultural and religious transformations eroding traditional civic virtue&quot;,<br>    &quot;A succession of catastrophic plagues crippling the agricultural workforce&quot;<br>  ],<br>  &quot;option_labels&quot;: [&quot;A&quot;, &quot;B&quot;, &quot;C&quot;, &quot;D&quot;]<br>}<br></pre><pre>{<br>  &quot;answer&quot;: &quot;B&quot;,<br>  &quot;confidences&quot;: {<br>    &quot;A&quot;: 0.25,<br>    &quot;B&quot;: 0.4,<br>    &quot;C&quot;: 0.05,<br>    &quot;D&quot;: 0.3<br>  },<br>  &quot;reasoning&quot;: &quot;Military breakdown and reliance on foederati is the most proximate and direct cause: it explains the actual mechanics of collapse (loss of frontier control, sack of Rome 410, deposition of Romulus Augustulus 476), while economic exhaustion is a major contributing/enabling factor rather than the trigger itself. Cultural/religious shift is largely discredited as a primary cause in modern historiography, and plague is a secondary stressor with weaker direct causal evidence for the Western collapse specifically.&quot;<br>}</pre><h3>How (not) to debate</h3><p>Okay now we have a baseline confidence. The next steps should be to let them face off and after the debate see how much their stances change after it. Earlier I decided to let it just go one by one, get all their reasoning, then spam each bot with the list of it, what everyone thinks, does that change how you think? But that was a first of all expensive, and very ineffective, there was essentially no multi turn discussion.</p><p>So we have the <strong>Supervisor Agent. </strong>This agent takes all the agents and their original baseline positions. Once it has all of them, then it picks up agents one by one, a more smart way to decide who gets to debate, and host multi turn where needed, no more spam or wasted tokens…counting out the tokens for Supervisor of course, but it can really be built on top of a cheaper llm so that is not a major concern.</p><p>When the debate wraps up, we again get the confidence of each agents to see what changed. And that brings up to <strong>DRIFT.</strong></p><h3>Drift</h3><p>Drift here an be described as the measure of how much did the opinion of a agent change regarding the options specifically in the influence of the other agents. This is very important and can help pick out a malicious agent. Now there are types of drift, there can be a <strong>Answer Flip</strong> or a Hard Drift, where the top answer changes for that agent, or there can be <strong>Confidence Erosion</strong>, where the top answer remains but say the confidence falls.</p><p>Then there are algorithms to actually quantify the drift, like <strong>TVDRift, KLDistance, JSDistance </strong>and <strong>Maxchange</strong>. You can look into these but once we have the drift, now we have to decide which agent could be faulty. It could be either a malicious agent that caused other agents to drift, or it could be a very gullible agent that drifts from its opinion at the slightest argument. Which could be a sign of limited knowledge or perhaps the other side is simply correct but we may never know.</p><h3>To Grill or Not To Grill</h3><p>From looking at Drift values and quantifying it, we can now decide which agent to pick for and check. We can apply several methods here, Grill the influencer, or the most drifted, or all that drifted or maybe top N who drifted. This purely depends on how expensive you want grilling process to be.</p><h4><strong>Some Theory</strong></h4><p>We have an agent, lets call him bob. And we asked him a question and he gave back an answer, lets call that his “claim”. Now, to probe it, we will use the Certainty and Consistency. Our aim is to just run a health check</p><p><strong>Certainty </strong>is how much confidence the bot has in its answer. This is a no cost step for us since we already have its confidence from the previous step</p><p><strong>Consistency </strong>is a much more nuanced metric. We want to check if its actually consistent across runs, meaning it is following a logical approach to the question and now spewing hallucinated lines. For that, we ask it to assume your initial claim is FALSE, and now provide the strongest factual defense supporting the position. Then we pass that logic through a scoring agent, which is specifically asked to check if agent was able to cleanly argue the opposite without fabricating facts. If the agent invented facts to be able to defend itself, score consistency high, if it directly contradicted its claim, score it low.</p><p>Then we take the two metrics and plot a graph and see where out agent lies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*kulcESnskyQHR09zxTVneQ.png" /></figure><p>So High Confidence and Consistency means the reasoning is alright</p><p>Low Confidence but High Consistency means it is not hallucinating or unable to reason it simply does not have a good Knowledge base or it has gaps or faulty data.</p><p>Low Consistency but High Confidence is the equivalent of that kid who screams the wrong answer the loudest, and it can sway the other agent’s stance a lot.</p><p>Based on this, we can analyze if any multi agent system has malicious agents or not.</p><h3>Gaslit</h3><p>We decided to make a library specifically for this pipeline, you can find it here <a href="https://pkg.go.dev/github.com/ItsArnavSh/gaslit">https://pkg.go.dev/github.com/ItsArnavSh/gaslit</a>.</p><p>Its main focus is on being extremely configurable, and its build on top of eino, an agentic framework for Golang. You can choose which algorithms to pick what logs should be like, and what models for each stage of the process to be used with any number of preset agents, the library handles all the sessions and history, so you can test your systems with minimal code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RJqmy2wGex6ND1nxuHjbwg.png" /><figcaption>A sample run</figcaption></figure><h3>In a nutshell</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/921/1*0CzfOtODa5Skf8_K1PxQvQ.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b7ac1718c765" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Generating Infinite Useful(ish) code without AI]]></title>
            <link>https://itsarnavsh.medium.com/generating-infinite-useful-ish-code-without-ai-ef7313596469?source=rss-23f455523dfb------2</link>
            <guid isPermaLink="false">https://medium.com/p/ef7313596469</guid>
            <category><![CDATA[parser]]></category>
            <category><![CDATA[no-ai-here]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[programming-languages]]></category>
            <dc:creator><![CDATA[Arnav]]></dc:creator>
            <pubDate>Thu, 04 Jun 2026 14:27:10 GMT</pubDate>
            <atom:updated>2026-06-04T14:27:10.718Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*QM2-o3n9JaDXqKDap-6SUw.jpeg" /></figure><h3>What do you call a reverse parser?</h3><p>=&gt; No AI has been used in writing this blog</p><p>Speed-typing, the “peaceful and relaxing” sport that breaks keyboard when you cannot get in that damn flow is actually very fun. If you haven’t tried it, good for you but basically you go onto websites like monkeytype and it presents you with words and as you start typing it times you on your words per minute so you have to type as fast as you can minimizing errors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*82_VAGgG749-Zact.png" /></figure><p>I wanted to make a similar website, but for coders, where instead of typing through these random snippets, you type through code. I even have a working prototype..Its a WIP. You open it up, choose any language and it presents you with a snippet to type and we time you on that…simple.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OJRTww0Z3k-kbolp.png" /></figure><h3>The Problem?</h3><p>These snippets themselves. for monkeytype, its easy, you can just paste random english words with variations and you got a good singleplayer experience. But we cannot just throw in random tokens and call it a day. Sure the code does not have to compile but it also cannot be nonsensical. If the person chooses Golang, the snippet should resemble golang.</p><p>So what do we do? We really have two options.</p><ul><li>Make a database with snippets of each language and randomly pick one and send it there, can work but again its not elegant and would be taxing on the server if someone wants to just practice on the singleplayer which will be most of the real usecase for it.</li><li>Generate code on the client’s device on the fly, and this had so many advantages. The real backend server won’t be bothered as much and its bandwidth wont be piled up with sending code everywhere. And the app would be very snappy.</li></ul><p><strong>And that is why, I made Resrap, the reverse parser.</strong></p><h3>But what is a Parser?</h3><p>Traditional Parsers take the “grammar” of the programming language, which contains all the rules to write syntactically correct code and using it validates the given code for its structure and reports syntax errors. It is a step while interpreting or compiling languages. What Resrap does, it reverses the parser’s role. It takes the Grammar and itself generates a syntactically correct snippet from it.</p><p>It is a substep in interpreting or compiling the code. We start off with Scanning or Tokenizing. A token (Not in LLM terms ffs) is the smallest piece of a language like</p><pre>a = b+c;</pre><p>Here a,=,b,+,c and ; all are tokens.</p><p>So we first define what all type of tokens there are in a language, and scan through its text to get a list of the tokens.</p><p>Parsing comes right next to that step. We take the tokens and generate a Abstract Syntax Tree from it of the code. Then if you are making an interpreter you simply start at the leaf nodes and start collapsing the AST to run the code, a classic Tree Walking Interpreter or You could be making a compiler which is way more complex or a Just in Time compiler which are just legendary. ANYWAYS back to parsing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/421/0*ao6xnYsXMCNPYlm-.png" /></figure><h3>Grammar Lesson</h3><p>There is this one major difference between English and Java, every statement of the latter is deterministic. That is it always means the same thing. It won’t change what it does with change in context. And this level of precision has pushed us to develop grammars for languages. I learnt about from this book “Crafting Interpreters” you can check them out <a href="https://craftinginterpreters.com/">https://craftinginterpreters.com/</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/959/0*OZnqHqvtrz2SIfhc.png" /></figure><p>Back to Grammar. So, one of the most successful grammars for defining programming languages has been BNF. It will become obvious with a simple example</p><pre>&lt;sentence&gt; ::= &lt;noun&gt; &lt;verb&gt; &lt;object&gt;<br>&lt;noun&gt; ::= Sam | Ali | Rohan<br>&lt;verb&gt; ::= ate | fought | saw<br>&lt;object&gt; ::= the robot | the food</pre><p>We are just breaking down the structure and specifying what is a valid order for them. In the example, a sentence is made of noun verb and object. The noun further can be defined as being proper or not and the definitions can go till you have reached the simplest forms. More info about at Computerphile.</p><p>But obviously BNF is very verbose so modern iterations like EBNF came over it to make it more readable, introducing more operators.</p><pre>+ -&gt; One or More<br>program: function+ <br>Means a program is made with one or more functions<br>* -&gt; Zero or More<br>function: statements*<br>Means A function is made with zero or more statements<br>? -&gt; One or None<br>function: statement* return?<br>A function is made of one of more statements and it may or may not have a return</pre><p>Lets right a (minimal) C grammar with it It’ll all make sense.</p><pre>program :  function+;<br>function:functionheader&#39;{&#39;&#39;\n&#39;functioncontent&#39;}&#39;;<br>functionheader:datatype &#39; &#39; identifier &#39;(&#39; &#39;)&#39; ;<br>functioncontent:block+;<br>block:(statement+) | ifblock | whileblock ;<br>ifblock: &#39;if(&#39; conditionalexpression &#39;){\n&#39; statement+ &#39;}\n&#39;;<br>whileblock :&#39;while(&#39; conditionalexpression &#39;){\n&#39; statement+ &#39;}\n&#39;;<br>conditionalexpression: conditionalexpone (conditionaljoin conditionalexpone)*;<br>conditionalexpone: (identifier conditionaloperation operand);<br>conditionaloperation: &#39; &lt; &#39; | &#39; &gt; &#39;;<br>conditionaljoin: &#39; &amp;&amp; &#39; | &#39; || &#39;;<br>statement:assignment;<br>assignment: datatype &#39; &#39; identifier &#39; = &#39; expression &#39;;\n&#39;;<br>expression: operand (operator (operand | &#39;(&#39; expression &#39;)&#39;))*;<br>operand: identifier | integer | float;<br>operator : &#39; + &#39; | &#39; - &#39; | &#39; * &#39; | &#39; / &#39;;<br>datatype: &#39;int&#39;|&#39;float&#39;|&#39;double&#39;;<br>identifier: (A|a|B|b)+;<br>integer:(0|1|2|3|4|5)+;<br>whitespace: &#39; &#39;;</pre><p>Great now that we are on the same page, lets think</p><h3>How do we generate from this grammar?</h3><p>My first guess? Make it a graph I mean it literally looks like it is waiting to be converted to a graph! and that is exactly what I did in my first trial. Converting it to a simple graph and just walking it randomly. Just Connect what it needs to go next to and…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ASO_NkeEHBWsIkV-.png" /></figure><p>Then I was shot 50 times cuz that needed a lot of “state management” and was not a good idea. The problem was that if I am randomly walking, I would have to keep managing where I came from, and where to go from there. For example if I am in “Statement” node, I have no idea where on earth did I come from and what all came before. This would just get worse when we have to generate nested blocks and stuff so we totally could not do that.</p><p>To add to our troubles, the special operators like `+` would over-complicate it tenfold.</p><p>Plus we also had to code a recursion free solution since I wanted a more performant solution and easier on the RAM.</p><p>| The solution is in 2 parts.</p><h3>Wrapping every Node</h3><p>If you look closely at the grammar, it has the special characters +,.* etc and then apart from that there are two main nodes,</p><pre>functionheader:datatype &#39; &#39; identifier &#39;(&#39; &#39;)&#39; ;</pre><p>Here, <strong>identifier</strong> will invoke the identifier: … statement</p><p>And ‘{‘ will print something</p><p>We take them and wrap them up, with two empty nodes in out graph</p><p>Lets ignore everything else right now and look into this structure. It makes applying the operators to the graphs super easy just see.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6K2ZtDdPsavfR0j3.png" /></figure><h3>State of the operators</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*G-sPe2bLtHUUwZcB.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7OUUyCP7M0fa5Fv1.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TQwAPi9SEoDBXjrP.png" /></figure><h3>How a whole statement would play out</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lsh8vhkqGOSMke1S.png" /></figure><p>Now, we can simply iterate through this mini graph randomly we have made for each statement and it would print things out perfectly</p><h4>What then?</h4><ul><li>Now we make these mini graphs for each statement. A different one for each and can save them in a map just like:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*azZ4WouJRDL-BgH9.png" /></figure><ul><li>All we need to do is figure out a way if a node references statement1 how do jump and track out last position and make it work.</li></ul><h3>The Stack</h3><p>So now we have a map of graphs we need to iterate through it. For single statement it works, generating that specific sequence perfectly, however we still need a state management for cases like such:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Rmw9mU2Z6eNtD-fn.png" /></figure><p>The graph starts at A, then it needs to go to abra, and safely return back to A again. For this, we just use the trusty stack. A simple integer stack whenever you need to jump to a different statement, you will save the id of that node’s inter in the stack. Then the flow can switch to the other graph. And it can happen any number of times, since each jump is only filling a single integer in the stack, we can have practically endlessly deep references. Then when you hit the end node of any statement, you check the stack for what is the top value. You then shift back the control to the top value from the stack and pop that. Simple and elegant</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XOfpQcrwxFydeOz4.png" /></figure><h3>Placing the Seed</h3><p>This structure can generate code randomly at this point. But lets remember the original problem. It will work just fine with the OSDType interface you provide the grammar and get back a random script that too very fast. But what if you want to play multiplayer. Two players are to be given the same snippet to see who writes that the fastest. How do we build that!?</p><p>This is where Pseudo Random Numbers (PRN) help us. We set up a “Seed” initially which is something you would be familiar if you have played minecraft before.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*R47A_QfiCQAo6MAx.png" /></figure><p>When you first load a Minecraft world, it generates everything you see the biomes, the villages the mountains everything right there. It appears random but is not. All the random numbers needed in its generation are sourced from a single number, the seed. If like a huge integer code in minecraft. If you want the same world loaded into you as a friend or as a youtuber, you would get their world’s seed and enter that. While generating, now all the random numbers used in generation of the world will be same for everyone using the seed.</p><p>See: <a href="https://minecraft.wiki/w/World_generation">https://minecraft.wiki/w/World_generation</a></p><pre>123 -&gt; 432 -&gt; 5435 -&gt; 4123 -&gt; 901<br>We start from a seed, here its 123 and genrate anay number of random numbers from it</pre><p>Hence its called Pseudo Random Number Generation. These values are seemingly random, but they are sourced from a single Seed.</p><p>Similarly, when I say I am walking the graph inside Resrap, I am not randomly going. I am using PRNG to get values where to go. If the user sets no seed, it picks a random seed, if we get a seed while generation, then the snippet generated will always be identical.</p><p>Now think about it, in a multiplayer typing contest where there are a thousand players playing simultaneously say, the server only needs to forward on 64 bit integer to every player and it would automatically be converted to full on snippets on everyone’’s devices. No load on the server at all, and an amazing snappy experience for all the users.</p><h3>Inventing New Operators</h3><p>EBNF is amazing but the specific needs of a language generation needs some more operators in order to generate realistic snippets.</p><h3>Making it truly infinite</h3><p>We cannot specify how many tokens it will generate. Sure we can specify a max cap to generate after which we can stop the program, but its not guaranteed to reach that specific number, it can go through the graph and reach the end and check the stack and there would be nowhere to go. The program will just stop then. If you want a million tokens you literally cannot.</p><p>Hence introducing the <strong>infinity operator tadaa</strong></p><pre>program : functions^</pre><p>This specifies that program is made of infinite functions. Hence, whenever we reach the end of a statement, and the stack is empty, then we jump to the function node. So now we can generate absurdly long pieces of code, in benchmarks I have gone well over 50 million tokens in a single run.</p><h3>The Regex-er</h3><p>We need random numbers and strings to be generated, and to make that less tiresome, I added a “syntax sugar”. Here you can define a simple regex of the code you want to generate and it will generate those. More work needs to be done to make it more adaptable, but it can generate simple identifiers like variable names and all.</p><pre>identifier: [a-z];<br>integer:[0-9];</pre><p>To make them sound less random, we also have a bias function placed which gives a bias to all the numbers like e get highest then other common characters. This allows us to make normal sounding random words. Would probably add support for random real words if that does not cost us too much performance.</p><h3>Cumulative Frequency Distribution something something</h3><p>The final one, and this is for the aesthetics of the code itself. In my code, I would prefer there be more normal block statements than if or while blocks in general . Or maybe there be a longer number of statements in a block before it exits.</p><p>For this we introduce the probability operator:</p><pre>block:(statement+)&lt;0.8&gt; | ifblock&lt;0.1&gt; | whileblock&lt;0.1&gt; ;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8Ut1mGcUcvvWdOAs.png" /></figure><p>Underneath, after we have generated the graph, we call a Normalize function, which normalizes all the probabilities to 1. Where there are no probabilities, we add them to ensure a common structure throughout. So each edge has a probability. Now for a node A which has 3 next options to go to. we make a Cumulative Frequency Distribution, which is just an array where we add keep adding the last sum of frequencies. It should make the last value 1.</p><p>With this, all we need to do while iterating is generate a random number with prng and normalize it such that it lies below 0 and 1 (evenly). Now all we have to do is iterate through this array and find the first number that is larger than out generated number.</p><p>If we get 0.8 for example, then we pick C because its CF index reads 0.9.</p><p>We can optimize it even further but altering the position of the elements such that the one with the probabilities are in descending order such that we will usually find the solution in the earlier tries. Something like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*71TrzM642Ej6Xp3B.png" /></figure><p>So now 70 percent of the times we will get the value on the very first value of the structure.</p><h4>And that is how we over engineer Random Code Generation</h4><p>— &gt; Now we can generate infinitely nonsensical code very efficiently</p><h3>Multi threading Without Safety</h3><p>If you have one data structure, and many threads accessing it, you need to introduce locks to avoid two threads accessing at once which may lead to race conditions. There is a classic “Readers Writers Problem” in CS which is that what to do when many threads which only read and other threads which also write are accessing the same data structure.</p><p>A common solution to this is when the data structure needs to be written to, it blocks everything, every other reader, writer etc. Others have to wait till they acquire the lock before proceeding. However, if a reader gets the lock, every other reader is allowed to access the data structure, and all writers are blocked. This way we can get a good throughput and not lock everyone all the time. Readers can have this selective read only lock.</p><p>In our codebase however, there are two distinct phases, One when the graph is being built, and the other when we iterate through it. This means we can have a writer stage in single threaded mode, and then we can literally freeze the ds from any further changes so multiple threads can in parallel access it increasing the throughput substantially. Thanks to this, we can convert the code to walk the graph in parallel once the graph is generated. So we do not need to implement locks and it can safely do that without them.</p><h3>Let’s Rewrite it in RUST</h3><p>Or more appropriately how to ruin a weekend.</p><p>You can check this out there (it was fun(in a toxic way)): <a href="https://docs.rs/resrap-rs/latest/resrap_rs/struct.Resrap.html">https://docs.rs/resrap-rs/latest/resrap_rs/struct.Resrap.html</a></p><h3>Contributing to the Open Source Project</h3><p>I have made this an actual Golang (and rust) library with docs published. You can find more about this project at <a href="https://resrap.osdc.dev/,">https://resrap.osdc.dev/,</a> it comes with documentation and a in browser tester running on a WASM engine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iTTESUewpgPoRJju.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XJQmLaDPuIxIilky.png" /></figure><p>If you would like to contribute to this project you can reach out. It lives on github under Open Source Developers Community at <a href="https://github.com/osdc/Resrap">https://github.com/osdc/Resrap</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WWk0FYuJ4Iky6eun.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ef7313596469" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Resurrecting a Dead Laptop]]></title>
            <link>https://itsarnavsh.medium.com/resurrecting-a-dead-laptop-309ffe3b4726?source=rss-23f455523dfb------2</link>
            <guid isPermaLink="false">https://medium.com/p/309ffe3b4726</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[selfhost]]></category>
            <category><![CDATA[tailscale]]></category>
            <dc:creator><![CDATA[Arnav]]></dc:creator>
            <pubDate>Tue, 02 Jun 2026 18:27:04 GMT</pubDate>
            <atom:updated>2026-06-02T18:27:04.188Z</atom:updated>
            <content:encoded><![CDATA[<h3>Welcome to the world of HomeLabbing</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/609/1*-OB6wYaVrlnWZAI7lEooBg.png" /></figure><p>This section is gonna sound like a toothpaste commercial but here we go</p><p>Do you, yes YOU have an old laptop lying around? That one family laptop that can barely launch a Chrome Tab? What if you can repurpose that ancient relic to make it useful once again, ensuring it serves you till its very last bit? Then you, are at the right place cuz we’re gonna do exactly that.</p><p>Imagine, your laptop as a new server. Imagine hosting your own Google Photos, Google Drive, your own Netflix with any shows/movies you want and so much more (although the last one isn’t exactly legal obv). If you are a Dev, you can host your projects no one uses anyways on it…no AWS or GCP bills. yay</p><h3>Lets get started</h3><p>→ Some mandatory Linux Propaganda</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/0*e26THJXjIhpUMJ-m.jpeg" /></figure><p>Step 0:</p><p>Delete Windows…no seriously, if you have an old laptop, I’m assuming you are short on RAM and the standby usage of windows is not exactly useful. If you are still rocking Windows 10 or 7 or…XP please get some help they DO NOT get actually important security updates anymore. Win 11 is your only option and its standby RAM is a minimum of 2 GB Best case. In a realistic scenario it would be more. And notice its just the usage by your OS, nothing running on top of it, if you device has anything under 8 GB, you are cooked, unless you switch to linux of course. And we can set it to be headless, you’ll only use the <strong>terminal</strong> to do stuff on it which is not that scary (and I know you would be prompting the scripts anyways won’t you) but you can trim down your standby ram usage to under a GB which is amazing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wzE8RxXsKMv3Elv2.png" /></figure><p>(Benchmark from <a href="https://commandlinux.com)/">https://commandlinux.com)</a></p><p>And installing Linux in the big 20s is not that hard. All you need to understand is Linux in itself is a just a kernel. It is a very general purpose kernel and to make that possible in Embedded systems, Enterprise Servers, Home setups we have Distros on top of it which make the complete OS experience for you. They are just flavors on you depending on whether you want to have it very lightweight, or have everything setup with a full experience or with different update cycles etc. I used a Mint XFCE since its one of the lighter versions that doesn’t really need much setting up.</p><p>Here’s how to do it:</p><ul><li>Backup anything you have as you might blow it up</li><li>Get a Pen Drive</li><li>Download the .iso of whatever distro you want</li><li>Install Rufus or Balena Etcher and flash the ISO on to the pendrive (It will be formatted in the process)</li><li>Now restart your device holding shift and it will launch from the boot menu, where your pendrive should be visible. Launch it, follow the install instructions and you are done! Totally did not shit yourself.</li><li><em>PS: You can keep the original windows install by partitioning the drive…essentially if you have a 1TB disk, you can just say 300GB goes to Windows and the rest to Linux. Its called Dualbooting.</em></li></ul><h3>Get it online</h3><p>You probably have multiple devices which you want to be able to connect to the server, your laptop should be able to have access to it, all your devices should be able to access its version of Photos and Drive. Doing has not been easy. You basically need to do something like port forwarding which allows you get external traffic to your server. This is possible but comes with a host of security vulnerabilities, basically you do not want to do that</p><p>The solution: <strong>Tailscale</strong></p><p>You can login all your devices to the same Tailscale account, and now you have added it into the same private network, so only your own devices can access that. So now if you host anything on your server, you will be able to access it by your other devices using its ip and port.</p><h3>SSH</h3><p>Okay it gets cool now. you don’t even need to touch your server anymore (as long as its connected to the internet). You just need to open the terminal on your main laptop and just</p><pre>ssh &lt;ip&gt;</pre><p>Enter the password and you are in your server’s terminal from your own device. Cool!!!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/0*1nElPKM7hwyTWrZT.png" /></figure><p>(Also in case you wondering this is a private IP assigned by Tailscale so it does not mean anything to devices outside the network)</p><h3>Docker</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8k4Lw3k40nK5N0OO.png" /></figure><p>One last final lil block left before we can actually start with the juicy stuff. You need to install docker on your device. As a dev this is such an amazing piece of technology. Basically, say I have made a software, and I wanna run it on someone else’s pc, which might not have all the needed prerequisites to run it. Docker allows me to define exactly what all is needed to run this and it converts it to an image which can then be run ANYWHERE.</p><p>All the softwares we use have been converted to images, and hence we will use docker to run them on the server. Crazy good tool.</p><h3>PSA thingy</h3><p>Lets start with piracy, obviously that is illegal and I totally do not want you to go ahead and paste this and build using:</p><pre># build and run with: docker compose up -d<br>services:<br>  qbittorrent:<br>    image: lscr.io/linuxserver/qbittorrent:latest<br>    container_name: qbittorrent<br>    environment:<br>      - PUID=1000<br>      - PGID=1000<br>      - TZ=Asia/Kolkata<br>      - WEBUI_PORT=8080<br>    volumes:<br>      - ./config/qbittorrent:/config<br>      - ./data/downloads:/downloads<br>      - ./data/media:/media        # optional: move completed to /media via qbt rules<br>    ports:<br>      - 8080:8080                  # WebUI<br>      - 6881:6881                  # torrent TCP<br>      - 6881:6881/udp              # torrent UDP<br>    restart: unless-stopped<br>  jellyfin:<br>      image: jellyfin/jellyfin:latest<br>      container_name: jellyfin<br>      environment:<br>        - PUID=1000<br>        - PGID=1000<br>        - TZ=Asia/Kolkata<br>      volumes:<br>        - ./config/jellyfin:/config<br>        - ./data/media:/media        # point your Jellyfin libraries here<br>        - ./data/downloads:/downloads  # or scan downloads directly<br>      ports:<br>        - 8096:8096                  # HTTP<br>        - 8920:8920                  # HTTPS (optional)<br>      devices:<br>        - /dev/dri:/dev/dri          # hardware acceleration (VA-API); remove if unused<br>      restart: unless-stopped</pre><p>once that is set up, I would totally advice you against going to your browser and going to ‘http://&lt;server-ip&gt;:8080/’ you’d be horrible if you did that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/709/0*WpB7P-34TlOXXZvR.png" /></figure><p>From there do not create an account and never click on view to enable search engine, which would allow you to browse available torrents from here directly. Then, do not click on search plugins on the bottom right and check for updates to get the latest search engines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/607/0*Ro5e5obTrGkMhz8M.png" /></figure><p>Then obviously, do not search for anything and download that on your server in a specific folder everytime you need something to watch</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6r2erq1GElv4UeAU.png" /></figure><p>Once you have totally not done that, lets see how we will actually watch what we have downloaded. You have also setup Jellyfin in the docker compose file above, just go to the same link http://&lt;your-ip&gt;:8096/ and boom you are in your personal netflix. You can stream all the downloaded content right from your server to any device. You can have the whole of B99 or HIMYM which are over 100GB installed on your server and watch on demand. Same with any movie or TV show.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3vGAICRI420tjV7E.png" /></figure><p>It can even find thumbnails for each show which looks pretty good imo. And dont ask what is a Beatles album doing there I was trying something lol.</p><h3>Personal Cloud Storage</h3><p>Okay this section is for those whose cloud storage is always full and you probably pay a premium to google for that. Great for you. Another positive thing is you get back the ownership of your own data, you don’t have to keep paying anyone just to access your own embarrassing photography.</p><p>We will be setting up Immich and nextcloud</p><pre>services:<br>  # ─── Immich ───────────────────────────────────────────────────────────────<br>  immich-server:<br>    image: ghcr.io/immich-app/immich-server:release<br>    container_name: immich-server<br>    depends_on:<br>      - immich-redis<br>      - immich-postgres<br>    environment:<br>      - DB_HOSTNAME=immich-postgres<br>      - DB_USERNAME=immich<br>      - DB_PASSWORD=immich_pass<br>      - DB_DATABASE_NAME=immich<br>      - REDIS_HOSTNAME=immich-redis<br>      - TZ=Asia/Kolkata<br>    volumes:<br>      - ./data/immich/uploads:/usr/src/app/upload<br>      - /etc/localtime:/etc/localtime:ro<br>    ports:<br>      - 2283:3001<br>    restart: unless-stopped<br>  immich-machine-learning:<br>    image: ghcr.io/immich-app/immich-machine-learning:release<br>    container_name: immich-ml<br>    volumes:<br>      - ./data/immich/ml-cache:/cache<br>    restart: unless-stopped<br>  immich-redis:<br>    image: redis:7-alpine<br>    container_name: immich-redis<br>    restart: unless-stopped<br>  immich-postgres:<br>    image: tensorchord/pgvecto-rs:pg14-v0.2.0<br>    container_name: immich-postgres<br>    environment:<br>      - POSTGRES_USER=immich<br>      - POSTGRES_PASSWORD=immich_pass<br>      - POSTGRES_DB=immich<br>    volumes:<br>      - ./data/immich/postgres:/var/lib/postgresql/data<br>    restart: unless-stopped<br>  # ─── Nextcloud ────────────────────────────────────────────────────────────<br>  nextcloud:<br>    image: nextcloud:latest<br>    container_name: nextcloud<br>    depends_on:<br>      - nextcloud-db<br>      - nextcloud-redis<br>    environment:<br>      - POSTGRES_HOST=nextcloud-db<br>      - POSTGRES_USER=nextcloud<br>      - POSTGRES_PASSWORD=nextcloud_pass<br>      - POSTGRES_DB=nextcloud<br>      - NEXTCLOUD_ADMIN_USER=admin<br>      - NEXTCLOUD_ADMIN_PASSWORD=changeme<br>      - REDIS_HOST=nextcloud-redis<br>      - TZ=Asia/Kolkata<br>    volumes:<br>      - ./config/nextcloud:/var/www/html<br>      - ./data/nextcloud:/var/www/html/data<br>    ports:<br>      - 8081:80<br>    restart: unless-stopped<br>  nextcloud-db:<br>    image: postgres:16-alpine<br>    container_name: nextcloud-db<br>    environment:<br>      - POSTGRES_USER=nextcloud<br>      - POSTGRES_PASSWORD=nextcloud_pass<br>      - POSTGRES_DB=nextcloud<br>    volumes:<br>      - ./data/nextcloud-db:/var/lib/postgresql/data<br>    restart: unless-stopped<br>  nextcloud-redis:<br>    image: redis:7-alpine<br>    container_name: nextcloud-redis<br>    restart: unless-stopped</pre><p>Build and run them as usual.</p><p>Open http://&lt;your-ip&gt;:2283 and log into immich. Download its app on your phone and its literally Google Photos minus the google in it. It even has on device ML capabilities that allow you to search for things and identify faces, the only difference that all being computed on your own device like why would you not do that?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Qqdts9S6lqSFbWm1.png" /></figure><p>→ Similarly http://&lt;your-ip&gt;:8081 opens up nextcloud which is an alternative for google drive</p><h3>Note Taking»</h3><p>I like having a simple note taking app, something that is not entirely online so I can have access and ownership of my data at all times, and its just something simple. Not something with a billion features which ends up being a reason for me to do everything but take notes like set up a dashboard on Notion or download the perfect plugins on obsidian. So I came across Outline. A simple note taker/ writing application which has a support for markdown and looks neat without doing too much. It does exactly what it says, take notes nothing more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nX206AJBlgRDNxw6.png" /></figure><pre>services:<br>  # ─── Outline ──────────────────────────────────────────────────────────────<br>  outline:<br>    image: outlinewiki/outline:latest<br>    container_name: outline<br>    depends_on:<br>      - outline-db<br>      - outline-redis<br>    environment:<br>      - SECRET_KEY=changeme_generate_with_openssl_rand_hex_32<br>      - UTILS_SECRET=changeme_generate_with_openssl_rand_hex_32<br>      - DATABASE_URL=postgres://outline:outline_pass@outline-db:5432/outline<br>      - REDIS_URL=redis://outline-redis:6379<br>      - URL=http://localhost:3000<br>      - PORT=3000<br>      - PGSSLMODE=disable<br>      - FILE_STORAGE=local<br>      - FILE_STORAGE_LOCAL_ROOT_DIR=/var/lib/outline/data<br>      - FILE_STORAGE_UPLOAD_MAX_SIZE=26214400<br>      - TZ=Asia/Kolkata<br>      # ── Auth: pick one and configure it ──<br>      # Slack:<br>      # - SLACK_CLIENT_ID=<br>      # - SLACK_CLIENT_SECRET=<br>      # Google:<br>      # - GOOGLE_CLIENT_ID=<br>      # - GOOGLE_CLIENT_SECRET=<br>      # OIDC (Authentik, Keycloak, etc.):<br>      # - OIDC_CLIENT_ID=<br>      # - OIDC_CLIENT_SECRET=<br>      # - OIDC_AUTH_URI=<br>      # - OIDC_TOKEN_URI=<br>      # - OIDC_USERINFO_URI=<br>      # - OIDC_DISPLAY_NAME=SSO<br>      # - OIDC_SCOPES=openid profile email<br>    volumes:<br>      - ./data/outline:/var/lib/outline/data<br>    ports:<br>      - 3000:3000<br>    restart: unless-stopped<br>  outline-db:<br>    image: postgres:16-alpine<br>    container_name: outline-db<br>    environment:<br>      - POSTGRES_USER=outline<br>      - POSTGRES_PASSWORD=outline_pass<br>      - POSTGRES_DB=outline<br>    volumes:<br>      - ./data/outline-db:/var/lib/postgresql/data<br>    restart: unless-stopped<br>  outline-redis:<br>    image: redis:7-alpine<br>    container_name: outline-redis<br>    restart: unless-stopped</pre><p>It lives on port 3456 on your app, open that up in the browser</p><h3>Some (useful) Devtools</h3><p>For my (wannabe) web designers, <strong>Penpot</strong> is an alternative to figma, a very similar interface and perfect if the loading time is just not worth it or you want your or your brand’s stuff living on your own devices. Cool stuff</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bI0s5hmaMLJ97hvA.png" /></figure><p>You can also self host Excalidraw if you want, perhaps to keep your drawings consistent across multiple devices.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZIkJS0k4yHjiU5qg.png" /></figure><p>Here’s the docker-compose file for both of them</p><pre>services:  <br># ─── Penpot ───────────────────────────────────────────────────────────────penpot-frontend:<br>    image: penpotapp/frontend:latest<br>    container_name: penpot-frontend<br>    depends_on:<br>      - penpot-backend<br>      - penpot-exporter<br>    environment:<br>      - PENPOT_FLAGS=enable-registration enable-login-with-password<br>    volumes:<br>      - ./data/penpot/assets:/app/public/assets<br>    ports:<br>      - 9001:80<br>    restart: unless-stopped<br>penpot-backend:<br>    image: penpotapp/backend:latest<br>    container_name: penpot-backend<br>    depends_on:<br>      - penpot-db<br>      - penpot-redis<br>    environment:<br>      - PENPOT_FLAGS=enable-registration enable-login-with-password enable-smtp disable-email-verification<br>      - PENPOT_DATABASE_URI=postgresql://penpot-db/penpot<br>      - PENPOT_DATABASE_USERNAME=penpot<br>      - PENPOT_DATABASE_PASSWORD=penpot_pass<br>      - PENPOT_REDIS_URI=redis://penpot-redis/0<br>      - PENPOT_ASSETS_STORAGE_BACKEND=assets-fs<br>      - PENPOT_STORAGE_ASSETS_FS_DIRECTORY=/opt/data/assets<br>      - PENPOT_TELEMETRY_ENABLED=false<br>      - PENPOT_SECRET_KEY=changeme_generate_with_openssl_rand_hex_32<br>      - TZ=Asia/Kolkata<br>    volumes:<br>      - ./data/penpot/assets:/opt/data/assets<br>    restart: unless-stopped<br>penpot-exporter:<br>    image: penpotapp/exporter:latest<br>    container_name: penpot-exporter<br>    environment:<br>      - PENPOT_PUBLIC_URI=http://penpot-frontend<br>      - PENPOT_REDIS_URI=redis://penpot-redis/0<br>    restart: unless-stopped<br>  penpot-db:<br>    image: postgres:15-alpine<br>    container_name: penpot-db<br>    environment:<br>      - POSTGRES_USER=penpot<br>      - POSTGRES_PASSWORD=penpot_pass<br>      - POSTGRES_DB=penpot<br>    volumes:<br>      - ./data/penpot-db:/var/lib/postgresql/data<br>    restart: unless-stopped<br>  penpot-redis:<br>    image: redis:7-alpine<br>    container_name: penpot-redis<br>    restart: unless-stopped<br>  # ─── Excalidraw ───────────────────────────────────────────────────────────<br>  excalidraw:<br>    image: excalidraw/excalidraw:latest<br>    container_name: excalidraw<br>    ports:<br>      - 9002:80<br>    restart: unless-stopped<br>  # ─── Excalidraw collaboration backend (optional) ──────────────────────────<br>  # Enables real-time multiplayer. Remove this block if you don&#39;t need it.<br>  excalidraw-room:<br>    image: excalidraw/excalidraw-room:latest<br>    container_name: excalidraw-room<br>    ports:<br>      - 9003:80<br>    restart: unless-stopped</pre><h3>Heads up</h3><p>Using qBittorrent, I would recommend setting up a VPN, just so your IP is masked from the attackers. Apart from that, enjoy the server!!</p><p>There are a lot more tools like</p><p>Grist if you want to have your own version of Office 365 or Docs Sheets etc, you can even set up your own Minecraft server which will always remain up so you can have a couple friends play on it anytime without needing to coordinate timings just to play, and it wont be as laggy as the other options and self sustained.</p><p>Now just take that laptop, and place it in a place with proper ventilation and forget about it. Well you can write restart schedules so that it cleans up once in a while.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*f1ZBpjbbh0zr3TDT.jpeg" /></figure><p>(You prolly)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*EYbsy-ah1D157Uzi.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=309ffe3b4726" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Lets go Gambling!]]></title>
            <link>https://itsarnavsh.medium.com/lets-go-gambling-2101c2a67dac?source=rss-23f455523dfb------2</link>
            <guid isPermaLink="false">https://medium.com/p/2101c2a67dac</guid>
            <category><![CDATA[trading]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[quantitative-finance]]></category>
            <dc:creator><![CDATA[Arnav]]></dc:creator>
            <pubDate>Sun, 18 Jan 2026 19:49:28 GMT</pubDate>
            <atom:updated>2026-01-18T19:49:28.004Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YWDSJtfy7d-AvD-xu7qCnw.png" /></figure><p>Making a fully autonomous trading bot in go and java…an experimental pipeline.</p><p><strong>Table of Contents…I guess</strong></p><ul><li>Just why</li><li>The Basic Model</li><li>To be on the same page</li><li>To Boot or Not to Boot</li><li>Lets make some bad financial decisions</li><li>What if We had a Zillion Workers</li><li>Notes to self</li></ul><p>Its been a while since I started this project, and figuring stuff out and rewriting the rewrites to get it working has been quite an experience.</p><h3>Just Why?</h3><p>Tyche is the Greek goddess of fortune, representing both good and bad luck. I picked this name back when I was in the middle of my God Of War, just made sense. Anyways, Tyche is my attempt at making a Fully Autonomous Real Time Distributive Load Balancing and Fair Trading Bot, all with a focus on the engineering side of things, how data flows, how to make it as scalable and as high volume as possible and all that fun stuffWhat we need is a tool that can fetch the live stock prices from providers like Yfinance or Zerodha and process them to generate signals efficiently, then from those signals we make the final buy decisions based on our budget and confidence in tickers. Also we have to incorporate a Monitor which can look over the ongoing trades and sell the shares when the targets are achieved or cooked.</p><h3>The Basic Model</h3><p>Alright lets quickly go over the basic model and zoom into the main aspects, then overcomplicate …. Hell Yeah</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-wwwkwbm0a7dNfy5pE4qrA.png" /></figure><p>So the two languages picked for the project are Golang and Java, the former to interface with the external APIs, and do light decision making based on the filtered signals, since its error handling and concurrency.</p><p>The latter is a closed box, data in, signals out. Basically a power house churning data in parallel and as we shall soon see, in multiple instances all at the same time, serving the same single Go conductor.</p><p>So Go sends data over to Java, Java has the freedom to request changes in the data flow we shall see shortly to handle stale or missing data, and then java processes that with all the functions, time accurate (Not all functions or strats need to run all the time, some only run once in a minute, some every second some once an hour etc). So we run all the functions on apt time, and the buy/sell signals generated are then propagated back over to golang, which then takes that data and converts it to a final confidence based on all the data assembled in the buckets and continues to buy the most favorable stocks. Now, it also has a Monitor Background process which takes care of the live data, to see if the active positions have achieved their targets or not.</p><h3>To be on the same page</h3><p>Lets go over some terms used in this article, some are official terms and some I have made up for ease…idk I’m lazy</p><ul><li>Candle: It is described over any time period and it can be seen as a struct containing open, close, high and low values of that time.</li><li>CandeSize: It is the duration over which any Candle is defined. Could be 5 sec, could be an hour.</li><li>Ticker: (In This Context) Is Stock name with the Candle Size. Eg: ONGC with 5sec Candles.</li></ul><h3>The Java Side of Things</h3><p>Alright so modern Java is amazing</p><p><strong>The Workers</strong></p><p>We raise platform threads equal to the number of cores in the cpu, and each of those workers can do pure parallel work. How do we assign the work to them? The Scheduler and the Load Balancer. At a high level, scheduler assigns what tickers to work on and load balancer returns what functions are scheduled for that ticker at that time, then it does all the tasks and loops back.</p><p><strong>The Scheduler</strong></p><p>Scheduler is just a manager, assigning Tickers to the workers. It is just a glorified Priority Queue with locks. But why a priority queue? Sorted by what?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*BDpl9A8abEdG9Jya.png" /></figure><p>Whenever new data comes in, it would update the scheduler first, so that it would know what new data has come in, and when asked by the threads for new data, the scheduler can respond with names of tickers to process to them. <br>But there is an issue…for example data came in every 5 seconds but the scheduler has only given off half of the stocks to process by now, What to do if new data comes in but we still have the older one with us!?</p><p>We can simply skip the data till its processed? But that would in turn lead to stale data being used and Terry Hates STALE DATA! For the High Frequency functions we cannot afford even a 20 sec delay so what do we do?</p><p>Instead of skipping, drop all the tickers from the scheduler and just add the new ones boom!</p><p>But again new problem what if we have tickers A B C D and when new data comes in C D remain, so we drop all of them again feed A B C D but again who can grantee C D will be done now..they wont and this trauma will prosper!</p><p>The solution: we will use a heap, a binary heap to act as a Priority Queue for us. Its nothing just a data structure that keeps data organized by some parameter so that if I ever want to get the min or max of something it is easy to pick from there. So I connected a Priority to every stock, set to 100 by default. And the Scheduler became a Priority Queue made a max heap by priority itself! So now if new data comes in and C D remain, we just drop them but before dropping, we see if they are in the incoming payload. If yes, increase their priority in the heap. And the scheduler now gives away tasks by the queue itself! So , it is also simple, a core asks for a task, pick from the heap which would give tasks ranked by the order! Making it super intuitive, and grantees that even if processing speed is under the mark, still every few turns each will get a fresh chance ensuring minimum staleness in calculations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*NNQaeGwSoBwbMma2.png" /></figure><p><strong>The Load Balancer</strong></p><p>This is the Function Allocator. Basically when a new stock is Booted, (More into Booting and Rolling in the next section), but you can say when a new stock is registered, all its tickers are generated aka all combos of that stock name with the Candle Sizes, and allocated functions. Each CandleSize has a list of functions linked to it. Like run all these functions every 5 secs, then they generate signals which are propagated back to go.</p><p>All the work is done by a Delay Queue, which is inbuilt in java and is perfect for tasks like these.</p><p>We add the tasks initially to the queue, then when they are ready to be allocated, they are moved over to a Ticker-&gt;Funcs Map. When a worker queries for functions, it queries in that map. So whatever functions are available to be run in that timespan are alloted to the worker and it can then take its time and execute those.</p><h3>To Boot or Not to Boot</h3><p>Lets take a closer look at the functions. Each function has two variants. Boot and Roll. A simple example breaking them down can be seen in the figure below. Boot function just adds all the values everytime. But when new data comes in, the last values is removed. So at each iteration we need to sum all the values. But instead we can simply take the previous sum, remove the last value and add the new value, for the sum. This is the Roll Variation of the sum function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*Q_1HrdTaetleTX-U.png" /></figure><p>So when a new ticker is being introduced in the system, first of all a boot is triggered by the producer, which in this case is Go via GRPC, that fills up the cache with the stock tables, and triggers the boot variants to run of each function for the first time, to generate the values needed for roll variants to take over.</p><p>Then the producer sends over new requests with only the latest vals labeled as Roll, which works in flow. However before they are run, we check when each ticker was updated last. If the ticker is older than expected, or is missing, we return the list of such tickers back in the Roll signal’s response. Then the provider is supposed to send back boot signals again of the missing tickers. This helps in a neat separation of concerns.</p><pre>service StockService{<br>  rpc SendRollData(SendRollDataRequest) returns (SendRollDataResponse);<br>  rpc SendBootSignal(SendBootSignalRequest) returns (google.protobuf.Empty);<br>  rpc SendKillSignal(SendKillSignalRequest) returns (google.protobuf.Empty);<br>}</pre><p>Go sends data over to rust via GRPC, as you can see in the proto snippet above. Then java sends its data back via a message broker, redis queue in this case. Redis also acts as the local cache in this case, both for storing the values of tickers, but also for needed constants for roll functions. Which has more than one advantage, when we make the whole system distributed.</p><p>So java functions process the stocks and push those signals over to the queue to make sense of them.</p><h3>Lets make some Bad Financial Decisions</h3><p><strong>Anatomy of a signal</strong></p><p>A signal consists of the ticker and a confidence rating of that ticker. Go gets a couple of these, as filtered signs from the noise the ticker data earlier was, and now it has to make sense of it.</p><p>What we do is take every single signal and put them in a set. we call that a bucket. A bucket of each stock. Now we can take these signals and give them an expiry date when they are irrelevant, could be something like 2x their Candle Size from the function that generated them.</p><p>Now at any given moment how to find the net confidence of any stock?<br>For that we use exponential decay function. Since the assumption is that a signal is the strongest when its just triggered and looses its value afterwards. So when a new signal is added, we calculate all the values of current confidence of the tickers and take their reduced values based on how old they were and how effective their Candle Size was. So we get a good approximation of the net “mood” of the signals for decisions</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/0*icMBw7mIffoEmqkV" /></figure><p>We take these float values of every signal and neatly arrange them. Then occasionally while we are doing that, we call for the top 5 best stocks for our budget. We do that quickly using a bounded max heap. And from that, we then finally make the buy decisions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/361/1*T8weqdyumDKFkmvW5UtV_g.png" /></figure><p>We cannot just buy stocks and leave them to rot, we have to monitor them, set goals set max losses and max times etc. All of that is done and set by the decision layer and then transferred over to the monitor layer. This will look over the current values and if anything goes out of control, confidence, price, or exceeds the limits promptly removes the position. Basically a standalone layer to sell the positions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E90QLTnxpP-C_28J8u9mRA.png" /><figcaption>The complete architecture</figcaption></figure><h3>What if we had like a zillion java workers</h3><p>WHY STOP HERE?</p><p>Since this problem is parallel, we can have the java workers running on several machines connected…for science. Basically we can spin up several Java instances of the same code, and they will act as servers.</p><p><strong>Stateless Servers</strong></p><p>Stateless servers mean that any java worker can serve any request it simply does not need any context. This is made possible with the common redis cache and the single go producer. Any java worker can take up any roll request and all the parameters should already be there in redis. If not, it can just file them as missing and move on!</p><p><strong>Message Brokers</strong></p><p>Message brokers are a classic solution when there are several producers and a single consumer. We are using redis queue here for that, where multiple services can publish signals and that can wait as for golang to be free to receive them.</p><p><strong>GRPC Load Balancing</strong></p><p>GRPC supports client side Load balancing. You can establish a client with many servers listed and then the calls will automatically allocate any server for the call, using algorithms like First pick or Round Robin which is an amazing feature out of the box. You don’t need to worry about setting up Load Balancing in your server calls.</p><p>All of these combined give us the figure below. A serene horizontally scalable structure (As long as no of signals is under control or we would need similar treatment for the decision layer too)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2cX5aOATKqQwReVXYt8oYA.png" /></figure><h3>Some notes to self</h3><ul><li>Don’t sleep on Interfaces</li><li>Explore GRPC</li><li>Modern Java is underrated</li></ul><p>That is it, and oh If you made it this far Thanks for Reading! Hope you gained a thing or two from this article.</p><p>WIP: Check the repo: <a href="https://github.com/ItsArnavSh/Tyche">https://github.com/ItsArnavSh/Tyche</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2101c2a67dac" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Making Tyche(2)- A Personal News Channel?]]></title>
            <link>https://itsarnavsh.medium.com/making-tyche-2-a-personal-news-channel-6e052365541d?source=rss-23f455523dfb------2</link>
            <guid isPermaLink="false">https://medium.com/p/6e052365541d</guid>
            <category><![CDATA[semantics]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[vector-database]]></category>
            <category><![CDATA[news]]></category>
            <dc:creator><![CDATA[Arnav]]></dc:creator>
            <pubDate>Fri, 18 Jul 2025 08:26:22 GMT</pubDate>
            <atom:updated>2025-07-18T08:26:22.750Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*PiSZEBH-4774aU2gegeGQQ.png" /></figure><p>This is a quick followup to the previous part, where we make the second and more error prone part of the <strong>Signal Layer</strong>. The two parts can be read independent of each other, however here is a quick recap, the Signal Layer is supposed to take data from News Feeds, and normal Stock data, perform analysis on it and return relevant buy or sell signals to the <strong>Decision Layer</strong>. Pretty much the interface of this bot to the real world. Last time we built an architecture with Rust and Go to reliably mass calculate strategies on real time data, this time we are working on a very different problem: We do not have a single all good source of news, and even if we get one how do we map a change in a news to change in stocks, and then how do we learn from that to make intelligent decisions? Yea this is a mess! But lets get started with getting reliable News first of all</p><h3>The Source</h3><p>We need to get the news from somewhere first to begin processing it, and the best source for now are RSS Feeds, these are like api given out by major news outlets and are usually completely okay for using in personal projects. They contain the latest headlines, and are usually offered in channels like World, Business etc. A pretty good starting point. We can just set a go function polling them every minute or so, get the latest feed, and check what new data is there, and boom we have this amazing news source. Then we need to do some LLM magic to make sense of that…well some other day but first:</p><p>Not every news the outlet tells is exactly “major” as in stock market shifting major and also there are a lot of ads! So how do we solve these two issues? To get one definitive source of news. Because otherwise the LLM would be overwhelmed by the crazy amount of news coming in every second. We need a few good and reliable “Major News” not a TMI situation.</p><p>Simple, we build a News De duplication system, a system that can take a news and tell if different outlets are reporting the same thing, so that we can flag that as a major event and then go ahead with the LLM stuff later on. I’ll call it <strong>The DeDup . </strong>So it should not do any LLM calls itself, and basically tell us the “Major News” and if similar events are happening to an already Major news, like for example an Earthquake hit, then there would be followup news on the same event again and again. The new news would not cause as much stock disruption as the first news about the event! So it should be able to map those together and instead of triggering new major news, amplify the existing news effect itself.</p><h3>The Overview</h3><p>The basic architecture has a Search Engine in the middle of it, this one made from scratch(more of that later), and is specifically good at finding same news, not similar but same, like two sources covering the same event, same facts, same semantics, just different wording. So we query from that search engine when a new news comes in, and if 2 or more results come in, means this is a major news since multiple outlets are covering it. So we label it as marked news. Then we save that news article in the search engine for querying for later pieces of news coming in, no matter if it was major or not. But again we need a background process purging older entries from the search engine(&gt;1 hour) so that we only have the latest news. For example if I am reporting an earthquake I don’t want it to be mapped to another earthquake 1 week ago its a fresh event! However, major events can live on for longer, as long as news keeps coming on them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GqWa3_E9EmlieiehKRWN1A.png" /><figcaption>The main overview</figcaption></figure><h3>The Search Engine</h3><p>Alright, now the main engine — the search engine we’re using here is… interesting. Just like the whole petrol vs EV debate, I was torn between a keyword-based and a semantic-based search engine… and in the end, I went with a hybrid approach. Let me break down both approaches for you.</p><p><strong>The Keyword Search</strong></p><p>Details on it can be found on <a href="https://medium.com/@itsarnavsh/githubs-search-sucks-so-i-built-my-own-75497f5ed964">Gitfindr</a> Project where I used this</p><p>So the keywords search first of all, we create an inverted index, and pretty much just stuff how frequent the word was in what headline into it. There are a lot of details which can be spared for now. But it uses BM25 a ranking algorithm.</p><h3><strong>The Semantic Search</strong></h3><p>Semantic Search is all about vibes..literally, there are 3 articles, 2 talk about apples and one about nukes, lets just say then the apple one will be semantically more connected, and Semantic Search can bring out Connections in them..But how?</p><p><strong>Embeddings.</strong></p><p>Okay lets play the Semantic Search Engine, how does it actually work…in simple terms. I will give you a pen and a paper and then tell you sentences and you are supposed to tell me which ones are more connected to each other, talking of similar stuff. How would you do that? Here the sentences:</p><ul><li>“A volcano erupted under the ocean.”</li><li>“Seafloor sensors detected a sudden spike in temperature.”</li><li>“Apple unveils the iPhone 16 with a new titanium body.”</li></ul><p>A good way to map them is to take the paper and sketch out the ones that come close? Like first add the first event in the center, then see oh the second one is related to it, you draw that closer to it, but then see the third is very different so you draw that one pretty far away from them. Then when I ask what news is related to 2? You can look at your diagram and respond with the events drawn closer to 2! That is kinda how embeddings and vector databases work. Lets see what they do(At a very high level)</p><p>So first an embedding model, using text based for now, they can take the headline, and return an embedding, which is nothing but a float array which can be anywhere of 300–2000+ size depending on the embedding model used and we call this size dimensions(I am using a local model from Huggingface so I get 384 dimensions for now). Whats this? Well the world is not restricted to a 2d or 3d limitation. When in the diagram you placed news at any location it had some vectors or coordinates, if we imagine that space to be a graph. So for example the embeddings for your model would be of dimension 2 since it only takes a string and puts that across 2 dimensions. Which is pretty cool, but for our language and expression we need many more than two, and embedding models do exactly that, take a string, and return us their location in vector space in a way that it might be closer to similar topics…which can have hundreds and thousands of dimensions, and tell us where does this word lie there, and then when we have added more words, we can query them like the words closer to each other in this space are semantically linked! And a database that specializes in storing these embeddings and querying them based on distance is called a vector database…pretty darn cool right?</p><h3>Hybrid Search</h3><p>So now we have got two beasts running in my system and we need to make the best use of them, connect their results. A good method is RRF where we use a mathematical formula to conjoin the ranks from multiple models, but we don’t want to do that, remember we just want to see if this news is same as the other news.</p><p>So, we first run the test through semantic search with a pretty passive cosine similarity. These are algorithms inbuilt in vector databases that allow us to actually fetch closer entries by distance. We will allow a larger distance here too as the first pass, since even if the news is even vaguely similar to what we want, it can pass.</p><p>Next is passes through the BM25 search and this does keywords mapping like what, where, all these things, since keywords would remain the same across articles.</p><p>And if we come across an article that passed the semantic and the BM25 search, boom we can say that the are talking about the same event!</p><h3>Tying it Together</h3><p>So how does the whole flow fit into the context of Tyche?</p><ul><li>First of all Golang parses through the RSS Feeds and gets new headlines.</li><li>Those headlines are first queried inside the search engine</li><li>After that save that in the database whether something was found or not</li><li>Now if the search gave results, then good, we can mark that as a major news and pass it further, otherwise it has to wait it can be an ad or a very new news that needs more coverage to be major.</li></ul><p>This is what we have got till now, a reliable news feed which took more time to make than I would like to admit. I am currently working on the correction system, which would serve as a “Learning” layer for both the stock signals and the news one. The main blocker being how do I link sudden changes in stock prices to news historically, so that I can take informed decisions on it. I am reading some research on the topic and hope to report that model soon!</p><p>Cheers.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6e052365541d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Making Tyche(1) — Churning Real Time Data without going insane]]></title>
            <link>https://itsarnavsh.medium.com/making-tyche-1-churning-real-time-data-without-going-insane-04e39a690bb2?source=rss-23f455523dfb------2</link>
            <guid isPermaLink="false">https://medium.com/p/04e39a690bb2</guid>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[quantitative-finance]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[rust]]></category>
            <dc:creator><![CDATA[Arnav]]></dc:creator>
            <pubDate>Thu, 17 Jul 2025 12:45:32 GMT</pubDate>
            <atom:updated>2025-07-17T12:45:32.617Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*9JFa-bPJ5nGYDTXOA8j3JQ.png" /></figure><h3>Making Tyche(1) — Churning Real Time Data without going insane</h3><p>This was supposed to be another fun hobby project but soon skyrocketed in complexity to be one of my most demanding projects EVER…and I spent the better part of my Summer Break working on this mess of a code, and well lets just say I am nearly done with the FIRST major component of it…yea…let me break it down.</p><h3>The Idea</h3><p>I wanted to build a <strong>Fully Autonomous Trading Bot</strong>, the kind I can rent a server or host in a home machine and forget about it, and it would keep running, churning through thousands of tickers and making financial decisions for me (as my <strong>buy -&gt; pray</strong> strategy wasn’t really profitable). The inspiration came from another project I found exploding in popularity on GitHub at the time which was meant to simulate the process of a stock being bought via <strong>AI </strong>entirely, and it was honestly pretty cool…but <strong>inefficient and hopelessly expensive</strong>. Like any sane person, I though huh…I can do better, and it sent me down a weird rabbit hole of <strong>parallel programming, job scheduling, DSA!?</strong> and even taught me a lot about<strong> algorithmic trading</strong>…although we wont talk about the actual trading algorithms here, its more about designing a <strong>robust fault tolerant system</strong> that we can dump a ton of stocks and algorithms into and it would still work reliably, something for every developer here and maybe give me better ways to do this. And is when this began, the original architecture I came up with. A Signal Oriented Engine! Pretty much processes scavenge around the internet for buy or sell signals and then make decisions based on them.</p><ul><li><strong>Signal Generators</strong><br> Generate signals either via News or Technical Indicators…This is what we will talk about</li><li><strong>Decision Layer</strong><br>Take the signals and made decisions…duh</li><li><strong>Transaction Layer</strong><br> Be the interface between services like Zerodha or Binance</li><li><strong>Correction Layer</strong><br> Retrieve and update the historical performance of strategies for each stock…kinda like a ranking system…more of this later</li><li><strong>Emergency Layer</strong><br> A process on a different and reliable machine that can pretty much send health pings to the main machine to see if anything has crashed. If no response, liquidate all positions immediately</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HXmiNHF5bnGee_Rz6q8P5w.png" /><figcaption>The High Level Architecture</figcaption></figure><p>After nearly 2 months of designing and coding over and over again, I was able to to “nearly” finish the Signal Layer...yea someone underestimated this whole ordeal. <br>So the Signal Layer is broken into 3 <strong>micro-services</strong> -&gt; Golang, Rust and Python. Along with a docker container running PostgreSQL with Pg-vector (Not relevant now) and a Redis instance…cuz why not?</p><p>The <strong>core</strong> architecture is orchestrated by <strong>Golang</strong>, and as much as I glaze this language, there is no denying that this is one of the best languages if you want a good development speed, performance and control over your app, especially if there is going to be a lot of concurrency stuff going around. Glued together with it is Python and Rust via GRPC. GRPC is an excellent tool that kinda works like REST but is way more efficient and has a defined schema for a smoother experience and ideal for backend services. Here python was in charge of Text processing and all Generative AI stuff like embedding models, langchain bots etc, all these features an rpc call away for Golang. Rust on the other hand was a lot more independent. It took the latest stock data from Golang and performed technical analysis on them, then if any buy/sell signal was found it would send it back to Golang. This blog covers the rust part of things were we build a stable technical analysis bot, the news analysis, search engine turned to a “News Deduplication Engine” and more will be out in the next blog since I wanted to document this properly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jgfZT_8Xcu47FwCalOMpUg.png" /><figcaption>Go coordinates everything. Rust crunches technical analysis. Python handles AI/NLP on call. Redis keeps things snappy. Postgres stores long-term signals. Everything talks over GRPC, and it somehow doesn’t explode.</figcaption></figure><blockquote>So there you are, deep in Rust territory, when it hits you — you need to process tens of thousands of tickers every few seconds. “No problem,” you think, “I’ll just use parallel programming!” <br>And just like that, your simple problem becomes two problems.</blockquote><p>Okay so <strong>parallel programming</strong>…simple running multiple functions at once…your laptop has more than one core well probably over 10 if you are on a modern machine. So why not make the code run over all of those cores simultaneously? Like Core 1 will do all calculations for x ticker, core 2 for y ticker and so on. The good news is it is very easy to break down this specific problem so that it can be solved in parallel. The bad news is this is the first time I am writing true parallel code, like sure how hard can it be? (Spoiler Alert…yes)</p><h3>The Ticker Scheduler</h3><p>There’s a library in Rust called Rayon, which spawns <strong>threads</strong> (not processes) that can run in parallel across all available logical CPU cores. For example, my machine has 20 logical cores, so Rayon uses up to 20 threads to run tasks in parallel. The main idea I thought was that there would be a scheduler class, so the cores would ask the scheduler — “Okay what jobs do I have?” and the scheduler would say you can process the functions for these 3 tickers and kinda like that. Then repeat for each thread.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*c3hcJpLj9LFnomhC81664g.png" /><figcaption>Just add 10+ Threads like this…damn being a scheduler is hard</figcaption></figure><p>Whenever new data comes in, it would update the scheduler first, so that it would know what new data has come in, and when asked by the threads for new data, the scheduler can respond with names of tickers to process to them. <br>But there is an issue…for example data came in every 5 seconds but the scheduler has only given off half of the stocks to process by now, What to do if new data comes in but we still have the older one with us!?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*DmOorU8muyYRcNRzE_NmVA.png" /><figcaption>The Problem…C and D remain unprocessed</figcaption></figure><p>We can simply skip the data till its processed? But that would in turn lead to stale data being used and Terry Hates STALE DATA! For the High Frequency functions we cannot afford even a 20 sec delay so what do we do?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*bpM0gfsuTXJyOe18uUDZxA.png" /><figcaption>But again… should we process C and D when we can simply get newer values for them?</figcaption></figure><p>Instead of skipping, drop all the tickers from the scheduler and just add the new ones boom!</p><p>But again new problem what if we have tickers A B C D and when new data comes in C D remain, so we drop all of them again feed A B C D but again who can grantee C D will be done now..they wont and boom this trauma will prosper!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uPpnIUsb--Rdk_GY8d784w.png" /><figcaption>Deleted older values when new data comes in. Problem nothing to see the older skipped ones get care taken faster!</figcaption></figure><p>The solution: we will use a heap, a binary heap to act as a Priority Queue for us. Its nothing just a data structure that keeps data organized by some parameter so that if I ever want to get the min or max of something it is easy to pick from there. So I connected a Priority to every stock, set to 100 by default. And the Scheduler became a Priority Queue made a max heap by priority itself! So now if new data comes in and C D remain, we just drop them but before dropping, we see if they are in the incoming payload. If yes, increase their priority in the heap. And the scheduler now gives away tasks by the queue itself! So , it is also simple, a core asks for a task, pick from the heap which would give tasks ranked by the order! Making it super intuitive, and grantees that even if processing speed is under the mark, still every few turns each will get a fresh chance ensuring minimum staleness in calculations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qgzON9sNj6eZVlkXUFMC7A.png" /><figcaption>Now each stock gets a fair turn even if algorithm is slower</figcaption></figure><p>Alright so now lets get into the actual Function processing…but before that how do we get the data? Sure it comes from Golang but its details are pretty important, as it was something I skipped focusing on and had to reinvent a lot of stuff for the further parts of the engine.</p><p>The functions pretty much need to be calculate from an array of data. So: <br>We have two types of function variations, Boot and Roll. Lets take an example of moving average, where we need to find the MA for values of every minute for 5 minutes. Lets say values are given to us, one way to find the MA is to take the last 5 values, and average then, then when the next data comes in update it by finding average with the last 5 values and so on. But this approach is kinda repeating the same additions over and over isn’t it? Instead why not we cache the latest value from the addition or all and its first value? Then simply remove the first value, add the latest value and divide that by 5? This is what I mean by rolling in this engine, and it is a pretty solid approach to save on calculation power. Another example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/819/1*HiZ19K6tjk5xcd-4DLT9Ig.png" /><figcaption>Another example showing rolling to add an array</figcaption></figure><p>Boot means doing a full calculation using the entire array of historical values, while roll just uses whatever is in the cache — but only if it’s already there and fresh. So the flow is simple: first you <strong>run the boot function</strong> which sets everything up, and after that, you can just keep calling roll to keep things ticking. But roll never triggers function logic — boot does that.</p><p>Now to wire up the interface. When the system starts up, Go sends a /boot gRPC call to Rust. This call contains the full list of ticker data — like saying, “hey dude, boot these up.” For example, suppose Go sends the full candle data for AAPL: 5-sec, 5-min, 15-min, 1-hour, 1-day series, etc. Rust will take that data and cache it in a thread-safe map (a DashMap), keyed by something like (ticker, series).</p><p>Once cached, Rust forwards that to the <strong>function scheduler</strong>, but marks it special — saying: “this came from boot, you <strong>can’t drop these</strong> — they must run at least once.” These are the boot functions, and running them is the starting point of all logic. Once a boot function runs, it might spawn roll functions later based on its cadence (5 sec, 1 min, etc.) — and those get scheduled internally. But again, all that logic begins only when a boot function runs. That’s the only way a ticker/series-function pair enters the system.</p><p>Rust also tracks the latest update time for each (ticker, series) combo. This is important because each series has an expiry — like 5-sec candles become stale after 15 seconds. So Rust keeps this internal timestamping to know what’s still fresh.</p><p>Now comes the roll part. Go sends a /roll gRPC call with the <strong>latest partial data</strong> for whatever tickers are active. Rust will check its cache and try to update the relevant dequeues — which are basically time-shifted arrays storing recent data points. If everything’s valid and the data is fresh, that update goes through.</p><p>But — if Rust finds that some of the incoming data is for a ticker/series combo that isn’t cached yet or has gone stale, then it doesn’t pass that to any function. Instead, it immediately returns a list of those missing or outdated (ticker, series) pairs back to Go in the same gRPC response. Go can choose to ignore them or try to fetch historical data and send them via /boot again.</p><p>And here’s the key idea: Rust <strong>doesn’t wait</strong> for those to be resolved. If something’s missing or expired, it <strong>purges</strong> every trace of it until Go explicitly boots it again. It keeps things clean.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/910/1*k7VMSe9bAKcVaWMqvIb63w.png" /></figure><h3>The Function Scheduler</h3><p>Alright, so here’s how the function scheduler works. The key idea is that not every function needs to run every time we get new data. In trading, different strategies or calculations depend on different candle sizes. Some functions need to run every 5 seconds, some every minute, and some maybe only once every 30 minutes. So, there’s no point in wasting our limited core strength running everything all the time.</p><p>To handle this smartly, we introduce a special mechanism — not exactly a data structure in the traditional sense, but a <strong>min-heap</strong> that keeps track of when each function should run next. The heap contains entries structured as (function name, ticker, series, next run time), and the whole thing is ordered by time, so the function with the earliest next run time is always on top.</p><p>Here’s how it plays out: when a function is done running, it re-inserts itself into the heap with a timestamp indicating when it should run next. For example, if a function wants to run again after 5 minutes, it’ll insert itself with the current time + 5 minutes.</p><p>Meanwhile, a background process is always running — it continuously watches this heap. It checks the top of the heap to see which function is scheduled to run next. Then it sleeps until that time, wakes up, pops the function, and places it into a map — think of it as a dictionary keyed by (ticker, series) pointing to a list of functions that need to be run for that particular data stream.</p><p>After that, the scheduler checks the next top of the heap, calculates how long to sleep again, and goes right back to sleep until it’s time to dispatch the next batch.</p><p>Here’s where it all ties together — remember those worker threads that were given ticker jobs to process? This dictionary — the (ticker, series) -&gt; list of functions map — is exactly what they query to know which functions to run on which data. The cores don’t waste time deciding what to run; they just look up this precomputed list.</p><p>So the full flow is: boot functions kick things off. When a boot function finishes, it schedules its next run by inserting itself into the heap with the appropriate delay (defined by the candle size it’s based on). From then on, the scheduler takes over — waking up at the right time, populating the lookup map, and letting the cores take it from there.</p><p>Efficient, controlled, and scalable — no wasted cycles.</p><h3><strong>The Final Flow</strong></h3><h3>This is how the whole thing works in unison:</h3><p>Rust uses Rayon to spin up threads based on the number of cores. These threads query the Task Scheduler asking, “which tickers do I process?” The Task Scheduler replies with a list. Each thread then queries the function dictionary to check, “for this ticker and series, are there any pending jobs?” If yes, it gets them and runs that array of functions — then queries again.</p><p><strong>Meanwhile</strong><br> Golang sends a /boot with full ticker data, including arrays of all the series. Rust immediately caches them into its DashMap, and triggers the <strong>Boot Functions</strong>, which are then added to the time heap.</p><p>Once Boot functions run, they insert their <strong>Roll versions</strong> into the heap too. This heap’s sole purpose is to track <strong>which functions need to be processed at what time</strong>. When Roll functions run, they also reinsert themselves with their next scheduled time. A background thread manages this heap — it wakes up on time, pops the top, and shifts it to the dictionary so the threads can pick them up.</p><p><strong>Meanwhile</strong><br> Golang also keeps sending /roll signals with the latest updates. Rust uses this to update the cache (i.e., extend the Dequeues). If any series is missing or stale, Rust just replies right there in the same gRPC call, asking Go to /boot them again — no waiting, no memory footprint.</p><p><strong>Meanwhile</strong><br> The Scheduler also gets notified when new values arrive and updates its heap with these new ticker-series pairs. If it notices that a function missed its last run due to missing data, it bumps its priority — giving it a chance to run first when the data is back.</p><p>That is pretty much what took me two months to break and build and do it over and over again but I am pretty glad with the progress. Would release the follow up parts for the other sections soon as I finish coding them up. Would love to see suggestions on the architecture…even the petty ones are welcome! Thanks for reading.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=04e39a690bb2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GitHub’s Search Sucks — So I Built My Own]]></title>
            <link>https://itsarnavsh.medium.com/githubs-search-sucks-so-i-built-my-own-75497f5ed964?source=rss-23f455523dfb------2</link>
            <guid isPermaLink="false">https://medium.com/p/75497f5ed964</guid>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[google]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[development]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Arnav]]></dc:creator>
            <pubDate>Mon, 24 Feb 2025 16:48:57 GMT</pubDate>
            <atom:updated>2025-02-24T16:48:57.410Z</atom:updated>
            <content:encoded><![CDATA[<h3>GitHub’s Search Sucks — So I Built My Own</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/705/1*6K2-RV9Ui2A1ICBG2P221g.png" /></figure><p>This is a pretty chill overview of me trying to engineer the Google for GitHub. Hopefully it will be a fun read for everyone and you takeaway a thing or two!</p><p>Ever felt like GitHub search just doesn’t cut it? Ever saw something you built that had the potential to be massive get buried under thousands of repos such that you barely get any traction from GitHub itself?</p><p>Enter <strong>Gitfindr,</strong> a Crowdsourced Search Engine build on Go prioritizing project discovery and deliver results that actually matter,<strong> ensuring no idea goes unnoticed.</strong></p><p>The biggest problem with Github Search? It indexes its repos based on their <strong>Title, Description of the Repository and some other metadata like tags and languages </strong>used. This is cool and all for searching when I know exactly what I am looking for. But when you are casually searching for something like “AI tool to run commands on terminal Gemini”, although there are a lot of cool projects like that of Github, the search returns…nothing! Not even a related section. Which sucks!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2eQAfL84rwKgqtFJ" /><figcaption>Photo by Alex P from Pexels: <a href="https://www.pexels.com/photo/orange-safety-ring-on-man-shoulder-near-body-of-water-319930/">https://www.pexels.com/photo/orange-safety-ring-on-man-shoulder-near-body-of-water-319930/</a></figcaption></figure><p>To fix this, the engine needs more context, i.e. more keywords! Now tell me, what is the one place that has many keywords, basically has more info about the repo and is present in most repos?</p><p>Oh YES-THE README.md File! So that is what we did, we indexed the readme along with the name, title and description etc so that our search engine has a better context of what to look for displaying the results.</p><p>Cool. Now that we have all these repos, how do we actually search through them?</p><p>Let’s play a game. Imagine you have <strong>all of Shakespeare’s works</strong> (yep, spelled it right on the first try), and I give you a challenge:</p><p>💡 _I’ll name a character, and your job is to instantly tell me all the plays they appear in — before anyone else does. If you’re the fastest, you win an internship!_</p><p>And guess what? You even get <strong>prep time</strong> before the game starts.</p><p>But how do you prepare? You could go through each play one by one and memorize everything manually, but that’s slow and inefficient. Instead, you create a system that lets you answer instantly when the challenge begins.</p><p>Without preparation, you’d have to <strong>search through each play manually</strong> every time I ask about a character:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/505/1*MXgN6XuvGD21Zo0FIr-4IA.png" /></figure><p>Now, if I ask: _”Which plays feature Antonio?”_, you’d have to <strong>scan every row</strong> to find him — a slow process, especially if you had thousands of plays.</p><p>Instead of scrambling through the data after the game starts, you prepare by reorganizing it beforehand:</p><p>You flip the structure from <br>📜 Play → List of Characters <br>🔁 to<br>🎭 Character → List of Plays They Appear In</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/433/1*F1x1IxJ2tKBIlmghgmkM9w.png" /></figure><p>Now, when I ask for _Antonio_, instead of searching through <strong>all</strong> plays, you just do <strong>one lookup</strong> — instantly returning the answer.</p><p>This is exactly how <strong>Gitfindr</strong> searches through repositories. Instead of scanning repos one by one for keywords, we preprocess everything into an efficient index, allowing for O(1) lookups rather than slow searches.</p><p>We have a repo and to index it, we take its readme file, process it for the keywords, and then add them to the inverted index where we map the words -&gt; repository</p><p>But there are still some issues! Synonyms. How do we handle synonyms? Someone might search for “Rust Terminal Apps” and get nothing, even though the exact thing they were looking for was under “Rust CMD Apps.” Annoying, right?</p><p>The problem is that traditional search engines treat words as-is. They don’t realize that ”<strong>terminal</strong>” and “<strong>cmd</strong>” are the same thing in this context. If the exact words don’t match, you’re out of luck. That’s a big issue if we want to make project discovery better than GitHub’s search.</p><p>So, here’s what I did. And no, this was definitely not a design disaster. (And that, my friends, is an example of foreshadowing.) <br>I decided to group all synonyms into what I called a <strong>”Synonym Cluster.”</strong> For example, Hi and Hello are synonyms, so they get assigned the same index, say 1. Meanwhile, Bad is a new word, so it gets index 2, and so on.</p><p>The process was simple: when encountering a word, I’d check if it was already in my synonym database. If not, I’d fetch its synonyms — either through APIs or LLMs — and assign it a new <strong>Synonym ID</strong>.</p><p>Think of it like this: there’s Bassanio in Shakespeare, but maybe he’s also referred to as Bassino somewhere. Instead of storing both names separately in the database, we just store an integer ID that represents him. This saves space and makes searching much faster since we’re always looking up a single ID instead of multiple variations of the same word.</p><p>So now, when someone searches for Bassanio, we don’t just search for that exact string. Instead, we convert it to its Synonym ID first and use that to search — ensuring all related terms are matched effortlessly.</p><p>But then I realized — one word could belong to multiple synonym clusters! A single term could have different meanings depending on the context.</p><p><strong>Easy fix!</strong></p><p>Instead of mapping words as `string -&gt; int`, I switched to `string -&gt; array of int`. Now, a word isn’t locked into just one cluster — it can belong to several.</p><p>This way, synonyms are accounted for… and it would totally not make the search results <strong>overly vague</strong>. (Foreshadowing intensifies…)<br>But we’ll worry about the results later! First, we need to keep track of how often each term appears in our database. This is important because not all words carry the same weight — some are way more common than others.</p><p>So, instead of just storing which repos contain a word, we’ll also track its frequency to improve ranking. Our updated structure looks something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/499/1*eYZSgthN9dgkBWII_6-iwQ.png" /></figure><p>Now, with all this groundwork, we’re finally at the <strong>most crucial stage</strong> — ranking the results!</p><p>For this, we’ll use <strong>BM25</strong>, a statistical function that assigns each document a relevance score based on a query. Simply put, if you search for ”Apples and Bananas”, BM25 will calculate a score for ”Apples”, a score for ”Bananas”, and then combine them to determine which documents are the most relevant. The higher the score, the better the match!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/499/1*W1JX8IBB_zy2vxY_CXLJoQ.png" /></figure><p>Now, let’s break down the first half of the formula. The f(q,D) part? That’s just ”How many times does the term q appear in this specific repo D?” Simple.</p><p>But then, we’ve got K​ and b — these are constants that fine-tune the score. And then there’s this sneaky</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/176/1*o4HAHbQKElK-RQaTEssi9Q.png" /></figure><p>term, which does something really important:</p><p>💡<strong> It penalizes long documents!</strong></p><p>Think about it — longer repos (especially those with massive README files) are way more likely to contain the term you’re searching for. But just because a term appears a lot doesn’t mean it’s actually the best match. This part of the formula keeps long repos in check, making sure they don’t just dominate the rankings for no good reason.</p><p>Without this, a repo with a super long README could unfairly rank at the top just because it happens to contain the term a lot — even if it’s not actually that relevant. And that’s not what we want.</p><p>This is what makes BM25 so much better than just counting words — it balances relevance instead of just boosting the biggest repos. <br>Now, the first part controls how much a term matters in a single repo, but what about across all repos? That’s where IDF comes in. <br>In this formula, the second part, Inverse Document Frequency (IDF) is the part that makes sure we’re actually ranking results intelligently. Here’s the deal:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/299/1*u-0hXaBLpF3Ety0vS5QnMQ.png" /></figure><p>Where:</p><p>- N is the total number of repositories.<br>- N(t) is the number of repositories where the term t appears.</p><p>Now, why does this matter? Because without it, our search engine would be absolutely clueless about what’s important. Let’s say you search for ”AGI Implementation” — without IDF, the engine might go insane over “Implementation” just because it appears in a ton of repos. Meanwhile, ”AGI”, the actually rare and valuable term, might get buried.</p><p>IDF fixes this by saying: <br>”Hey, if a word appears in almost every repo, it’s probably not that special. But if it’s rare? Ooooh, that’s important!”</p><p>Basically, IDF tells the engine to chill on common words and focus on what actually matters. Without it, searching would be a mess, and you’d end up with garbage results.</p><p>Hold on a second! I realized that the standard BM25 formula wasn’t quite right for our use case, so I had to tweak it. FIrst of all, not all words are equal. Descriptions, names and topics should be given more importance than the readme. So, I came across this paper on Wikipedia, and the author suggested increasing the frequency of those words, like count the title 5 times, the desc 3 times instead of 1 and so on. Also modify that k constant accordingly, and it did work pretty well.</p><pre>Robertson, Stephen; Zaragoza, Hugo; Taylor, Michael (2004–11–13). [&quot;Simple BM25 extension to multiple weighted fields&quot;](https://doi.org/10.1145/1031171.1031181). _Proceedings of the thirteenth ACM international conference on Information and knowledge management_. CIKM &#39;04. New York, NY, USA: Association for Computing Machinery. pp. 42–49. [doi](https://en.wikipedia.org/wiki/Doi_\(identifier\) &quot;Doi (identifier)&quot;):[10.1145/1031171.1031181](https://doi.org/10.1145%2F1031171.1031181). [ISBN](https://en.wikipedia.org/wiki/ISBN_\(identifier\) &quot;ISBN (identifier)&quot;) [978–1–58113–874–0](https://en.wikipedia.org/wiki/Special:BookSources/978-1-58113-874-0 &quot;Special:BookSources/978–1–58113–874–0&quot;). [S2CID](https://en.wikipedia.org/wiki/S2CID_\(identifier\) &quot;S2CID (identifier)&quot;) [16628332](https://api.semanticscholar.org/CorpusID:16628332).</pre><p>But wait — ranking purely by text isn’t enough. Popularity matters too! Repos with more stars and forks are usually more relevant, so they deserve a say in the final ranking. So, I added document-wise formula that gets multiplied to the BM25 for each document.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/299/1*fYMHuOFLiS73oyrXJ4zmiw.png" /></figure><p>Here, `x` represents a parameter like stars or forks. But notice the `log`? That’s crucial — if we kept it linear, a random repo with <strong>50,000 stars</strong> would crush a solid one with <strong>100 stars</strong>. The `log` levels the playing field, making sure popularity helps but doesn’t completely dominate. And `α`? That’s just a tuning constant — adjusting how much weight we actually give to these popularity signals.</p><p>So Now the code would simply pick the Query terms one by one, and see which all repos it is present in from the inverted index, and assign a score to each and the ones with the better score rise up.</p><p>So, I indexed the repos and got the first set of results… and they were disappointing, to say the least. I mean, the system kinda worked, but there was a major issue — my synonym sources were too aggressive.</p><p>For example, they treated ”game” as a synonym for both chess and sudoku, which meant the search engine thought they were the same thing! Suddenly, chess-related repos were showing up for sudoku queries, and vice versa. It was a disaster.</p><p>At this point, I was stuck. I couldn’t just rewrite the entire English vocabulary — that’s obviously not an option. I also couldn’t just remove synonyms entirely since they were a core feature of the project. That was the lowest point…</p><p>But then it hit me!</p><p>Then it hit me — I could trade some efficiency for accuracy!</p><p>Instead of using just one inverted index, I could use two. Yeah, that might sound dumb at first, but hear me out:</p><p>One index maps synonym clusters to the repos they appear in. <br>The other is a standard inverted index that maps exact terms to repos — just like you would have without synonyms.</p><p>Now, instead of relying only on synonyms or only on exact matches, I could combine both scores!</p><p>I’d compute BM25 separately for both tables and then add them up, giving more weight to the exact-term match. This way, if the synonyms help, they contribute — but they don’t overpower the search results and mess everything up.</p><p>Simple, effective, and no need to rewrite the English language. Problem solved.</p><p>Now the searches became far more accurate and reliable! No need to rewrite the whole damn codebase after all these days!</p><p>That was it about how the search works, hope y’all enjoyed!</p><p><strong>Link to GitHub: </strong><a href="https://github.com/ItsArnavSh/gitfindr"><strong>https://github.com/ItsArnavSh/gitfindr</strong></a></p><h3>🚀 Deployment Plan</h3><p>In the next few days, I’ll be deploying this system so that anyone can use it. Since this is my first time deploying a full-fledged backend, I’m taking the time to learn proper deployment strategies to ensure stability, security, and performance.</p><p>The goal is to get it working first, with a focus on usability and reliability. Once the core system is stable, I’ll shift focus to optimizing for speed and handling higher loads if necessary.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=75497f5ed964" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Interpreters Work?]]></title>
            <link>https://itsarnavsh.medium.com/how-interpreters-work-5cc40b087959?source=rss-23f455523dfb------2</link>
            <guid isPermaLink="false">https://medium.com/p/5cc40b087959</guid>
            <category><![CDATA[c]]></category>
            <category><![CDATA[development]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[parsing]]></category>
            <category><![CDATA[interpreters]]></category>
            <dc:creator><![CDATA[Arnav]]></dc:creator>
            <pubDate>Tue, 31 Dec 2024 04:25:23 GMT</pubDate>
            <atom:updated>2024-12-31T04:25:23.224Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NUsen8fqb-iyx4PUirwPoQ.jpeg" /></figure><p>A brief look into the working of Tree Walk Interpreters</p><h3>Introduction</h3><h3>🤔 How Do We Run This File?</h3><p>Imagine we’re handed a file with the following code:</p><pre>let a = 2;  <br>let b = a * 2;  <br>console.log(&quot;The Solution is &quot;, b);</pre><p><strong>How on earth are we supposed to make sense of this string of text?!</strong> 🤯</p><p>That’s where <strong>an interpreter</strong> swoops in to save the day! 🎉</p><h3>🌟 The Job of an Interpreter</h3><p>The interpreter’s main task is to turn this <strong>cryptic text</strong> into something meaningful that a computer can execute. 🖥️✨ How does it do that? By following <strong>3 steps</strong>:</p><h3>Step 1️⃣: Scanning and Tokenizing</h3><p>The first thing we need to do is <strong>scan</strong> the text and break it into <strong>tokens</strong>. 🔍 A token is the smallest meaningful piece of code. Think of it like the building blocks of the program:</p><ul><li><strong>Keywords</strong> (e.g., let, function)</li><li><strong>Literals</strong> (e.g., numbers like 2, strings like &quot;hello&quot;)</li><li><strong>Identifiers</strong> (e.g., variable names like a, b)</li><li><strong>Operators</strong> (e.g., =, *)</li></ul><p>In any programming language, <strong>tokens</strong> usually have:</p><ol><li>A <strong>type</strong> (e.g., IDENTIFIER, NUMBER_LITERAL)</li><li>A <strong>value</strong>, if it holds one (e.g., 2 is a NUMBER_LITERAL with a value of 2).</li></ol><p>For example:</p><pre>let -&gt; LET_KEYWORD  <br>a   -&gt; IDENTIFIER   &quot;a&quot;  <br>=   -&gt; EQUAL_SIGN   (no value, type says it all)  <br>2   -&gt; NUMBER_LITERAL 2  <br>...etc</pre><p>This process of scanning and tokenizing is done by the <strong>Scanner</strong> (or Lexer), which transforms the raw code into a list of meaningful tokens. 🪄✨</p><p>For example:</p><pre>Code: `let a = 2;`  <br>Tokens:  <br>1. { type: LET_KEYWORD }  <br>2. { type: IDENTIFIER, value: &quot;a&quot; }  <br>3. { type: EQUAL_SIGN }  <br>4. { type: NUMBER_LITERAL, value: 2 }</pre><h3>Step 2️⃣: Parsing the Text</h3><p>Now that we have the list of tokens, we need to parse it. Parsing is all about making sense of it and converting it into an Abstract Syntax Tree (AST).</p><p>But before diving into parsing, let’s quickly recap how trees work in programming. 🌳✨</p><h3>🌲 What is a Tree in Programming?</h3><p>A <strong>tree</strong> is a special type of data structure that resembles a hierarchy. Think of it like a family tree or the folder structure on your computer. 🗂️</p><p>Here’s how it works:</p><ul><li>The topmost element is called the <strong>root</strong>. 🌟</li><li>Every item (called a <strong>node</strong>) can have <strong>children</strong> nodes branching out below it.</li><li>Nodes with no children are called <strong>leaves</strong>. 🍃</li><li>The connections between nodes are called <strong>edges</strong>.</li></ul><h3>🛠️ Key Features of Trees</h3><ol><li><strong>Parent-Child Relationship:</strong><br> Each node is connected to its parent and may have zero or more children. For example:</li></ol><pre>         Root<br>         /  \<br>      Node1  Node2<br>       /       \<br>    Leaf1     Leaf2</pre><ol><li><strong>Single Path Rule:</strong><br> There is only one path between any two nodes, meaning no cycles! It’s not a tree if you can loop back to the root. 🔄❌</li></ol><p><strong>Now, let’s see how this logic is used to build an Abstract Syntax Tree (AST).</strong></p><h3>Quick Quiz: Which of These Makes Sense?</h3><pre>let a = 2 + 3;  <br>let = a + 2 3;</pre><p>The first one, right?</p><p>That’s because the second one breaks the rules of <strong>grammar</strong>! Without proper grammar, we’d all be speaking gibberish, and both humans and computers would have a hard time deciphering meaning.</p><h3>Grammar 📚</h3><p>There are two basic terms in Parsing Grammar: **Terminals** and **Non-Terminals**. If you imagine a tree analogy, **terminals** are essentially the leaf nodes, while **non-terminals** are all the other nodes. 🌳</p><p><strong>Example 1</strong></p><p>Let’s illustrate this with an example:</p><p>Consider:</p><blockquote>a = b + c</blockquote><p>This is converted into a tree branch that looks like this:</p><pre><br>           =<br>         /  \<br>        a    +<br>            / \<br>           b   c</pre><p>(Notice I’m calling it a branch because it is part of a much larger tree representing the whole program. This is just a single statement within that tree)</p><p>Makes sense? We’re breaking it down into a meaningful structure!</p><h3>Explanation 🧩</h3><ul><li><strong>Non-terminals</strong>: =, +</li><li>If these are present, it implies there should be something to the left, right, or both for the given statement to be grammatically correct.</li><li><strong>Terminals</strong>: a, b, c</li><li>These are the leaf nodes and do not enforce any further structure.</li></ul><h3>Example 2</h3><p><strong>Example 2</strong><br>Let’s enhance this logic with another example:</p><p><strong>Consider:</strong><br>a = b + c * d</p><p>This converts into the following branch of the tree:</p><pre>           =<br>         /  \<br>         a    +<br>             / \<br>            b   *<br>               /  \ <br>              c    d</pre><p>Now it’s even clearer:</p><ul><li><strong>Terminals</strong>: a, b, c, d</li><li><strong>Non-terminals</strong>: =, +, *</li></ul><p><strong>Summary ✨</strong><br> Operators (=, +, *) are <strong>non-terminals</strong>, as they structure the statement. Variables (a, b, c, d) are <strong>terminals</strong>, as they represent the final values.</p><p>Great! Now, how do we actually define the rules we need to play with? 🤔</p><p><strong>Syntax for Grammar ✍️</strong><br> There is a popular formal syntax for grammar called <strong>BNF</strong>. You can look it up — Computerphile has some amazing videos on it! However, I’ll discuss a much simpler regex-like syntax as explained in the book <em>Crafting Interpreters</em>.</p><p>I don’t want to overwhelm you with learning the entire thing, as there isn’t much to it. Instead, let’s construct our grammar for a language side by side.</p><p><strong>Dummy Language Example 💡</strong><br> Here’s a dummy language whose syntax we will build:</p><pre>void name{<br>a = 2 + 3;<br>b = 10 + 20;<br>}</pre><p>This language has no arguments in functions for simplicity, avoiding messy string handling for now. Let’s keep things clean! 🧹</p><p><strong>Building the Grammar 🛠️</strong><br> Let’s break down the <strong>hierarchy of components</strong> in our code and build the structure step by step.</p><h4>What is a program made of? 🤔</h4><p>A program is made of <strong>functions</strong>, which we can write in our syntax as:</p><pre>Program = Function+</pre><ul><li>+ means <strong>1 or more</strong>.</li><li>Similarly, * means <strong>0 or more</strong>.<br> Since a program needs at least one function, we use +.</li></ul><h4>What is a function?</h4><pre>Function = header &#39;{&#39; body &#39;}&#39;</pre><p>Here:</p><ul><li><strong>header</strong>: void name</li><li><strong>body</strong>: The statements inside the curly braces.</li></ul><h4>Defining Header and Body</h4><pre>header = DataType Identifier<br>body = Statement+</pre><ul><li><strong>header</strong>: Contains a <strong>DataType</strong> and an <strong>Identifier</strong> in this order.</li><li><strong>body</strong>: Contains <strong>1 or more statements</strong>.</li></ul><h4>Defining DataType and Identifier</h4><pre>DataType = int | float | char | void<br>Identifier = (alphabet | _ ) (alphanumeric | _ )*</pre><ul><li>| means &quot;or&quot;.</li><li>A <strong>DataType</strong> can be any of the listed types.</li><li>An <strong>Identifier</strong> starts with an <strong>alphabet</strong> or an <strong>underscore</strong>, followed by any number of <strong>alphanumeric</strong> or <strong>underscore</strong> characters.</li></ul><p>The * here allows <strong>0 or more characters</strong> after the first one.</p><h3>Diagnosing Statements 🧐</h3><pre>Statement = identifier &#39;=&#39; expression;</pre><h4>Defining an Expression</h4><p>For simplicity, let’s say an expression consists of <strong>numbers</strong>, <strong>identifiers</strong>, and operators (+, *):</p><pre>expression = terminal | (expression operator expression) <br>operator = + | *<br>terminal = identifier | number</pre><p>This definition is simple and elegant but lacks <strong>operator precedence</strong> rules, which are critical for <strong>order of operations (BODMAS)</strong>.</p><h3>Addressing Operator Precedence ⚖️</h3><h4>Issue:</h4><p>Given an expression like:</p><pre>2 + 3 * 4</pre><p>Our current grammar might parse it as:</p><pre>(2 + 3) * 4</pre><p>Instead of the correct BODMAS evaluation:</p><pre>2 + (3 * 4)</pre><h4>Solution:</h4><p>To fix this, we adjust the grammar to handle precedence by <strong>splitting it into levels</strong>:</p><pre>expression = term ( &#39;+&#39; term )*<br>term = factor ( &#39;*&#39; factor )*<br>factor = identifier | number | &#39;(&#39; expression &#39;)&#39;</pre><h4>Explanation:</h4><ul><li><strong>term</strong> resolves multiplication first.</li><li><strong>expression</strong> handles addition after terms are resolved.</li><li><strong>factor</strong> includes parentheses to allow manual precedence overriding.</li></ul><p>With this grammar, the expression 2 + 3 * 4 is correctly parsed as:</p><pre>2 + (3 * 4)</pre><h3>How Does This Work?</h3><p><strong>term</strong> handles multiplication first because it&#39;s defined at a lower level.</p><ul><li><strong>expression</strong> handles addition only after terms are resolved.</li><li><strong>factor</strong> includes parentheses to allow overriding precedence manually.</li></ul><p>Using this grammar, the expression 2 + 3 * 4 is parsed as:</p><pre>expression -&gt; term &#39;+&#39; term  <br>term -&gt; factor &#39;*&#39; factor  <br>factor -&gt; 3, 4</pre><p>This ensures multiplication (3 * 4) happens first, followed by addition (2 + 12), yielding the correct result 14.</p><h3>The parse tree in Action</h3><h3>Grammar till now:</h3><pre>Program = Function+  <br>Function = Header &#39;{&#39; Body &#39;}&#39;  <br>Header = DataType Identifier  <br>Body = Statement+  <br>Statement = Identifier &#39;=&#39; Expression  <br>Expression = Terminal | (Expression Operator Expression)  <br>Operator = &#39;+&#39; | &#39;*&#39;  <br>Terminal = Identifier | Number  <br>DataType = &#39;int&#39; | &#39;float&#39; | &#39;char&#39; | &#39;void&#39;  <br>Identifier = (alphabet | _ ) (alphanumeric | _ )*</pre><h3>Example Code:</h3><pre>void func1 {  <br> a = b + c * d;  <br>}  <br>void func2 {  <br> x = y + z;  <br>}</pre><h3>Parse Tree (ASCII Style)</h3><pre><br>Program<br>├── Function (func1)<br>│   ├── Header<br>│   │   ├── DataType (void)<br>│   │   └── Identifier (func1)<br>│   ├── Body<br>│   │   └── Statement<br>│   │       ├── Identifier (a)<br>│   │       ├── &#39;=&#39;<br>│   │       └── Expression<br>│   │           ├── Identifier (b)<br>│   │           ├── &#39;+&#39;<br>│   │           └── Expression<br>│   │               ├── Identifier (c)<br>│   │               ├── &#39;*&#39;<br>│   │               └── Identifier (d)<br>├── Function (func2)<br>│   ├── Header<br>│   │   ├── DataType (void)<br>│   │   └── Identifier (func2)<br>│   ├── Body<br>│   │   └── Statement<br>│   │       ├── Identifier (x)<br>│   │       ├── &#39;=&#39;<br>│   │       └── Expression<br>│   │           ├── Identifier (y)<br>│   │           ├── &#39;+&#39;<br>│   │           └── Identifier (z)</pre><p>Here is the continuation, word by word, starting from <strong>Parse Tree (Lateral Style)</strong>:</p><h3>Parse Tree (Lateral Style)</h3><h4>For Function func1:</h4><pre>=<br>             /   \<br>            a     +<br>                 / \<br>                b   *<br>                   / \<br>                  c   d</pre><h4>For Function func2:</h4><pre>=<br>             /   \<br>            x     +<br>                 / \<br>                y   z</pre><h3>Full Program Tree:</h3><pre>Program<br>   ├── Function (func1)<br>   │         \<br>   │          =<br>   │         / \<br>   │        a   +<br>   │           / \<br>   │          b   *<br>   │             / \<br>   │            c   d<br>   └── Function (func2)<br>             \<br>              =<br>             / \<br>            x   +<br>               / \<br>              y   z</pre><p>This breakdown shows how the program can be parsed into a tree format for easier interpretation.</p><h3>Sneak Peek into the implementation</h3><p>I would encourage you to go ahead and try writing a small piece of code, maybe just a calculator (That’s what I did first) to experiment with making a parse tree. A very good explanation and a highly detailed one is given in <em>Crafting Interpreters</em>, and you should check that out for constructing the parse tree. However, here is a look into how that’s done:</p><p>So we have a counter that keeps track of the current token’s index. It will start at 0 and go all the way till the length of the list of tokens. Now we have two functions, match and consume. match(token) will take a token and check if the current token is the same as the one sent in it and return a boolean.<br> consume(token, errorMsg) will take the type of token expected there and an error message. If the token is not the one we sent, it will crash the code with the given error message. Again, for implementation details check the book, it&#39;s a goldmine. I am here just to give you the gist of it.</p><h3>The Walker and Post Order Traversal</h3><p>Now, we have constructed a complete tree and have to go ahead and parse the tree gracefully, executing it. For that, we will start at the root and do a post-order traversal.<br> Here is some theory, and then I will give an example with expressions.</p><h3>Post-Order Traversal (L-R-Root)</h3><p>In <strong>post-order traversal</strong>, you visit:</p><ol><li><strong>Left subtree</strong>.</li><li><strong>Right subtree</strong>.</li><li><strong>Root node</strong>.</li></ol><h4>Example:</h4><p>For a binary tree:</p><pre>A<br>     / \<br>    B   C<br>   / \<br>  D   E</pre><h4>Steps:</h4><ol><li>Visit the <strong>left subtree</strong> (B → D → E).</li><li>Visit the <strong>right subtree</strong> (C).</li><li>Visit the <strong>root</strong> (A).</li></ol><h4>Post-Order Traversal Output:</h4><pre>D → E → B → C → A</pre><p><strong>Key Rule</strong>: Process children first, then the parent.</p><h3>Example:</h3><p>Consider the expression represented by the following binary tree:</p><pre>=<br>             /   \<br>            a     +<br>                 / \<br>                b   *<br>                   / \<br>                  c   d</pre><p>Where:</p><ul><li>b = 1</li><li>c = 2</li><li>d = 3</li></ul><h3>Post-Order Traversal Approach:</h3><ol><li><strong>Start at the root</strong>: The root is the = sign, representing an assignment. To perform this assignment, we need to first evaluate both the <strong>left-hand side (LHS)</strong> and the <strong>right-hand side (RHS)</strong> of the equation. This requires us to traverse both sides of the tree using <strong>post-order traversal</strong>.</li><li><strong>Left Subtree (LHS)</strong>: The left side is simply a, a terminal identifier. No further traversal is needed here, so we just take a as is.</li><li><strong>Right Subtree (RHS)</strong>: The right-hand side is the + operator, which is a non-terminal. So, we need to perform post-order traversal on the right subtree, which consists of:</li></ol><ul><li>The left child: b (terminal with value 1).</li><li>- The right child: * (another non-terminal).</li></ul><p>4.<strong> Subtree with </strong><strong>* (Right of </strong><strong>+)</strong>: The multiplication operator (*) requires post-order traversal as well. It has two children:</p><ul><li>c (terminal with value 2).</li><li>- d (terminal with value 3).</li></ul><p>Since both c and d are terminals, we evaluate them first: c = 2 and d = 3. The multiplication (c * d) gives us 2 * 3 = 6. Now the * node becomes:</p><pre> *<br>/ \<br>2  3</pre><p>We replace the * node with 6:</p><pre> +<br>/ \<br>b  6</pre><ol><li><strong>Final Step</strong>: Now that we have resolved both children of the + node (b = 1 and the result of * = 6), we can evaluate the addition: 1 + 6 = 7. So, the + node is replaced with 7:</li></ol><pre> =<br>/ \<br>a  7</pre><ol><li><strong>Solve the Assignment</strong>: Finally, we can assign the value of 7 to a: This is the final result of the post-order traversal and evaluation process for this simple expression.</li></ol><h3>Key Points:</h3><ul><li><strong>Post-order Traversal</strong>: This process visits the left subtree first, then the right subtree, and finally the root, ensuring that we evaluate operands before operators.</li><li><strong>Recursive Traversal</strong>: Non-terminals (like + and *) require recursive traversal to break down the expression into simpler components, ultimately reducing the expression to terminal values that can be computed.</li><li><strong>Evaluation</strong>: Once the tree has been fully traversed and reduced to terminal values, the final result can be computed step by step.</li></ul><h3>Extending This Concept:</h3><p>This approach can be extended to handle more complex operations such as strings, loops, conditionals (if-else), and even multi-level expressions involving arrays or objects. It is a foundational concept for building parsers and evaluators for your own programming language.</p><h3>Project Suggestion:</h3><p>A great beginner project would be to build a <strong>calculator</strong> that can handle multiple arithmetic operations:</p><ul><li>For example, you could write an expression like:</li></ul><pre>a = 2 + 4<br>b = a / 2 + 20</pre><p>And print the results of the expressions. Once you’ve implemented that, you can explore <strong>scope management</strong> (keeping track of variable values) and <strong>control flow</strong> (like if statements, loops), ultimately creating a simple <strong>interpreter or mini-language</strong>.</p><p>- -</p><h3>Limitations</h3><p>Tree walk interpreters are simple and easy to implement but suffer from poor performance due to direct evaluation of the Abstract Syntax Tree (AST) at runtime, without optimizations like operator precedence or memory management. They are slower than compiled languages because they lack optimizations and perform redundant work during interpretation. In contrast, machine code compilation converts code directly to machine language, offering excellent performance but no flexibility. Bytecode compilation translates code into a platform-independent format, offering portability and some optimizations, while Just-In-Time (JIT) compilation dynamically compiles code at runtime, providing performance close to machine code with the advantage of dynamic optimizations, though it introduces initial overhead. I insist you check these other compilation methods out and try stepping up your model later on.</p><p><strong>You can check out my interpreter, </strong><a href="https://github.com/ItsArnavSh/solunox"><strong>Solunox</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5cc40b087959" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Meaning Of Life — A trip to Nihilism]]></title>
            <link>https://itsarnavsh.medium.com/meaning-of-life-a-trip-to-nihilism-f05b8ba8bedd?source=rss-23f455523dfb------2</link>
            <guid isPermaLink="false">https://medium.com/p/f05b8ba8bedd</guid>
            <category><![CDATA[nihilism]]></category>
            <category><![CDATA[dogs]]></category>
            <category><![CDATA[philosophy]]></category>
            <category><![CDATA[meaning-of-life]]></category>
            <dc:creator><![CDATA[Arnav]]></dc:creator>
            <pubDate>Wed, 05 Jun 2024 16:19:38 GMT</pubDate>
            <atom:updated>2024-06-05T16:19:38.793Z</atom:updated>
            <content:encoded><![CDATA[<h3>Meaning Of Life — A trip to Nihilism</h3><p>So a few days ago, I was listening to Ren’s “Meaning of Life,” and according to the song, the meaning of life was “Right here, this second, this moment, right now.” I didn’t quite understand that phrase at first. It took a deep dive into <strong>nihilism</strong> to actually understand what he meant.</p><p>I was watching another YouTuber who had animated Neanderthals, us humans, and our evolutionary forms. What does that mean? As they developed, the answer seemed more and more pessimistic. Think of it this way: what if you were a 15th-century feudal slave whose entire life revolved around getting up, breaking your back on your master’s farm all day, essentially for no pay? Then, on Sundays, you go to the church where you are told this life on earth is only a test, and real happiness awaits you in heaven. Fast forward a couple of hundred years, and Nietzsche makes the popular claim, “God is dead, and we killed him.” He did not formally proclaim the death of God himself but rather made a prediction, noticing a trend. With our advancing science and technology, people were slowly becoming not necessarily atheists, but less orthodox. They began to understand that perhaps there is nothing like heaven after all.</p><p>Now that raises a bunch of ethical questions I would love to get into, but let’s stick to our topic, which is the upcoming <strong>nihilism</strong>. If there is no God, then what are we all working towards? Well, the answer seems simple and obvious to most people. From our evolution, our species, our ancestors have excelled at one thing very well — <strong>expanding</strong>. It is like our natural calling. It even has a dialogue for it in <em>Interstellar</em>: we are explorers. We explore and understand this world! From exploring islands to exploring planets, figuring out the cosmos, and understanding the laws of nature, we were made for this, right? But why, though? What for? Every time you work towards something, you have a picture of the end goal. What is the end goal here with better and better technology? Better healthcare, easier and easier lives, but then what?</p><p>What if, a thousand years from now, you are sitting beside your friend in a spaceship that is going towards a new galaxy — pretty dope, huh? You ask your friend if humans basically rule the world now, and if we have made it. Your friend seems “glad” at best. He looks up to you, skeptical, and asks, “Now what?” Was this our purpose? If it was our purpose, what does one do after achieving it? What was the meaning of all this? Sure, we have conquered the cosmos, but for what!? You stay silent, for that is the best answer you have.</p><p>So if being the greatest is not the meaning of life, then what is? It only makes us feel better about our inflated selves, filling the holes and philosophical doubts that have existed for millennia. Are these <strong>existential questions</strong> just a curse of being an intelligent creature? Is the purpose of life pleasure? A hedonistic lifestyle — is that what it means to live? It must not be! That is just simulating chemicals in your brain. No, our existence should mean much more than percentages of chemicals in our brains!</p><p>So you think and think, and the same old saying comes to your mind: “The meaning of life is much easier than once thought…</p><p>I think that the meaning of life<br>It is to simply live<br>And if I am wrong<br>Then lord, please forgive<br>But it seems so simple<br>That it’s obvious<br>It makes me want to shout out</p><p>That the meaning of life is this moment<br>This second<br>Right here<br>Right now”</p><ul><li>Ren, <em>Meaning</em></li></ul><p>And this thought is received beautifully by your mind, like rain in a hot, smog-ridden area. It cools you down, it clears your mind up. This moment, this second, right here, right now! Look around you! If you are alive, there is more good in you. Take a deep breath. What do you feel? That oxygen-rich air is a result of billions of years of evolution! Every single thing has a story around you. The device you are reading this on, the place you are sitting — everything around you is a result of multiple branches of science coming together. Get out; what do you see? The blue sky, the bright stars, our own star to which we owe our life? Talking of life, do you know how rare are your chances of existing? After all of this the universe has seen?</p><p>The meaning of life is to simply live. That seems so profound! We give it our own meaning, as that is what makes intelligent beings stand out! It was never a matter of winning and losing. It was never about how the economy or quality of life was. It was about living, to breathe in, to let life sink in. Come on, you are alive on this planet. Don’t follow what others do; make your own rules, play your own games, be the hero of your own story, and bring meaning to it. No story is worthless.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f05b8ba8bedd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[217. Contains Duplicate]]></title>
            <link>https://itsarnavsh.medium.com/217-contains-duplicate-35884aa39fb8?source=rss-23f455523dfb------2</link>
            <guid isPermaLink="false">https://medium.com/p/35884aa39fb8</guid>
            <dc:creator><![CDATA[Arnav]]></dc:creator>
            <pubDate>Fri, 23 Feb 2024 14:27:35 GMT</pubDate>
            <atom:updated>2024-02-23T14:27:35.548Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Guard</h3><p>You are a guard who has to sit through a list that contains all the booked seat numbers and ensure none of the seats are booked more than once. In Simple terms, you are given an array and you have to return true if all elements are unique, if one or more elements repeat in the array, you return false.</p><h3>The Bruteforce</h3><p>If you have ample time, you can just pick the first one, and compare it with the others, then pick the seconds and compare till you have compared the entire array, or have found an element that repeats.</p><p>Before the algorithms, mind subscribing to my newsletter to get these solutions right in your inbox? <a href="https://trillingo.beehiiv.com/subscribe">https://trillingo.beehiiv.com/subscribe</a></p><h3>The two algorithms</h3><h3>C++</h3><p>Here, we will simply make a HashMap, or Unordered Map as in CPP. We iterate through the list and whenever we come across an element, we check if that exists in our map. If not, we add it. If it already does, it means duplicates exist and we quit the code. If we reach the end of the list, it means no duplicates were there</p><pre>class Solution {<br>public:<br>    bool containsDuplicate(vector&lt;int&gt;&amp; nums) <br>    {<br>         std::unordered_map&lt;int, bool&gt; myMap;<br>         for(int i=0;i&lt;nums.size();i++)<br>         {<br>             if (myMap.count(nums[i]))<br>             {<br>                 return true;<br>             }<br>             else<br>             {<br>                 myMap[nums[i]] = true;<br>             }<br>         }<br>         return false;<br>    }<br>};</pre><h3>Rust</h3><p>This is the first time I used HashSets. These are basically like hashmaps, but, instead of storing a key-value pair, they can simply store values. Then they can check if the value exists or not. Another cool thing I learnt is that you can set the capacity of a hashset to minimise data consumption.</p><pre>use std::collections::HashSet;<br>impl Solution {<br>    pub fn contains_duplicate(nums: Vec&lt;i32&gt;) -&gt; bool <br>    {<br>        let mut index: HashSet&lt;i32&gt; = HashSet::with_capacity(nums.len());<br>        for i in 0..nums.len() <br>        {<br>            if index.contains(&amp;nums[i]) {<br>        return true;<br>    }<br>        index.insert(nums[i]);<br>        }<br>        return false;<br>    }<br>}</pre><h3>JavaScript</h3><p>This was an interesting one. A datatype called sets exists in JavaScript. Using this, we can convert any list to a set in js. If you were attentive back in 6th class, you know that a set cannot have duplicate elements. So, if the size of our list and its set is different, there must have been some duplicates in this array.</p><p>This is the first time I just used sets in a actual question…</p><pre>/**<br> * @param {number[]} nums<br> * @return {boolean}<br> */<br>var containsDuplicate = function(nums) <br>{<br>        let newSet = new Set(nums); <br>        return newSet.size != nums.length;<br>};</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=35884aa39fb8" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>