<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Inside Out: Through the Looking Glass - Medium]]></title>
        <description><![CDATA[This is dedicated to the process - Medium]]></description>
        <link>https://medium.com/inside-out-through-the-looking-glass?source=rss----78af52dc2fed---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Inside Out: Through the Looking Glass - Medium</title>
            <link>https://medium.com/inside-out-through-the-looking-glass?source=rss----78af52dc2fed---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 09:39:29 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/inside-out-through-the-looking-glass" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Images of the World]]></title>
            <link>https://medium.com/inside-out-through-the-looking-glass/images-of-the-world-1e3c74fc6443?source=rss----78af52dc2fed---4</link>
            <guid isPermaLink="false">https://medium.com/p/1e3c74fc6443</guid>
            <category><![CDATA[meaning]]></category>
            <category><![CDATA[religion]]></category>
            <category><![CDATA[alan-watts]]></category>
            <category><![CDATA[myths]]></category>
            <category><![CDATA[universe]]></category>
            <dc:creator><![CDATA[RAM ESHWAR KAUNDINYA]]></dc:creator>
            <pubDate>Sun, 12 May 2024 19:57:13 GMT</pubDate>
            <atom:updated>2024-05-12T20:22:34.294Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TnN0WyRU1MJEINeonJZHGg.jpeg" /><figcaption><a href="https://commons.wikimedia.org/wiki/File:Black_hole_-_Messier_87_crop_max_res.jpg#/media/File:Black_hole_-_Messier_87_crop_max_res.jpg">https://commons.wikimedia.org/wiki/File:Black_hole_-_Messier_87_crop_max_res.jpg#/media/File:Black_hole_-_Messier_87_crop_max_res.jpg</a></figcaption></figure><p>The Western world has long been living under two myths of the universe. These two myths have been the world as a construct — something created — and the fully automatic model.</p><p>By myth here I mean an image of the universe through which we understand it.</p><p>The image of the world as a construct comes from the ideas in Genesis of man created from dust. It comes with the image of God as a king — the ultimate ruler who sets forth the laws or logos which define the order of the universe.</p><p>The fully automatic model is a response to this kingly construct model which gets rid of the lawmaker but keeps the idea of the world as created. It says it does not matter whether the universe has a creator or not because this plays no role in the scientific enterprise which seeks to explain and predict. So we now have a model of the world which sees everything as a bunch of moving parts working together to form a machine. Why was this particular machine created? Well it’s just a statistical chance — a fluke of time and statistical regularity where we happened upon the universe we are in now.</p><p>What is the issue with such a model of the world? Well, for one the fully automatic model, if the logic is followed to its conclusion, implies that even thought must be a statistical happenstance. Rationality and our particular explanations and predictions about the universe are just statistical byproducts. So this means rationality and science are no more valid than any other statistical outcome — they can be no more true than any other statistical probability.</p><p>This leaves the individual in a precarious situation — the individual feels like they are nothing but unintelligent billiard balls coming together in a sea of unintelligent billiard balls. The universe is essentially stupid in this model, simply following the gears and laws which are laid out for it. Who am I? What is my relation to this world? Just a bunch of balls hitting other balls in space and time.</p><p>But the conclusion of this argument is that this perception is just another myth of the universe. One brought about by bad assumptions! In other words, it’s not even a logically consistent myth.</p><p>We live in a time in Western culture where the individual has never been more important. But, the key question of who am I and what is my relation to the universe is one which is left empty and void of meaning because of the myth of the universe we are operating under.</p><p>The image of the world as created had its place and effectiveness in early human history but it seems too crude to return to with the knowledge and wisdom acquired by humans over the years. Furthermore it does little to answer the two crucial questions of who I am and how I can relate to the world.</p><p>Now I treat those as two separate questions, but the reality is they are perhaps the same question asked in two different ways.</p><p>Let’s take a thought experiment. Say we have a single ball in empty space. There is no way to tell by looking at this ball if it is moving or stationary. This is because there is nothing to compare the ball to and therefore we cannot know whether the ball is moving. Now lets add a second ball. With this ball we can now distinguish motion. One ball is moving relative to the other. But which ball is moving and which is stationary? One cannot say. You could say either ball 1 is moving towards ball 2 or ball 2 is moving towards ball 1 and both would be equally correct. Similarly you could say ball 1 and ball 2 are both moving towards each other and this could be correct. What’s even more is you could say neither ball 1 nor ball 2 are moving but rather space itself is contracting bringing the two balls together. So with two balls we can establish motion but we cannot say which is moving with respect to what! Now if we add a third ball in we can introduce an element of majority vote. If it appears ball 3 is moving towards balls 1 and 2 while balls 1 and 2 are maintaining equal distance from one another we can now say by majority vote balls 1 and 2 are stationary and ball 3 is moving. But even that could be flipped — we could say balls 1 and 2 are moving together towards ball 3!</p><p>So you see from this simple thought experiment, if we were one of these balls, we only are able to define ourselves in relation to the other balls. In other words relationship is a fundamental quality of the subjective experience of the universe!</p><p>So what we define as ourselves then is also only describable in terms of relationship! In this way the question who am I and what is my relation to the universe are one and the same thing! You are only defined in relation to the universe. And the universe is only defined in relation to you!</p><p>As a wise rabbi once said, “If I am I because you are you, and you are you because I am I, then I am not I and you are not you”.</p><p>This therefore introduces another image of the universe — the universe as a drama. In this myth of the universe we are, each one of us, an actor on the world stage. As Shakespeare said, “All the world’s a stage, and all the men and women merely Players”. In this myth we do not see the universe as automatic or unintelligent. We instead see it as playing an infinitude of roles all of which are an expression of a single grand play. It’s all one big show!</p><p>In the relational picture of the world we also get another image — this is the image of growth. As two things become related to one another they begin to co-evolve, to grow into something which is greater than the sum of their parts. From this we begin to see organisms, or more accurately patterns, as emerging from the universe. We are then not precisely created and placed into the world, we are born out of it. We see then an image of the universe as an organism.</p><p>None of these pictures taken on their own are sufficient. We use images to describe what something is like in some way but not to say exactly what it is. In a sense this is the limitation of thought and language. But this need not bother us — we cannot precisely describe how we beat our own hearts but we do it all the same. We do not describe how we breathe but we do it all the same. And being able to describe it does not help us to do it better!</p><p>So we must learn to combine these myths of the universe. We must learn to synthesize them. So I can see myself as an individual who is so defined only in relation to everything else. I behave in certain mechanistic ways but only to play a part in the universe. And my mechanisms are a part of the natural growth of the universe. Yet they can be controlled they can be brought into order. But the order of things as they are is just as much order as the order of how you want things to be. That want and desire is also an order. We begin to see that only by combining these images of the universe can we move past the existential limits of any single image of the universe.</p><p>The world as a drama does not naturally lead us to examining the mechanisms of the parts of the drama. It can take us away from playing our roles with vigor. And it can limit us from defining our own roles!</p><p>The world as an organism can remove from us the desire to create the world in our own image. To learn to manipulate it for our needs. It can also remove from us the desire to act our part for our part is seen simply as an expression of the natural order not something to strive to play well! It simply happens.</p><p>The world as a construct can limit us from seeing ourselves as an integrated part of the universe and cause us to miss our fundamental relation to the universe.</p><p>And we must remember all the time that all images are like fingers pointing the way to the moon — we must not be caught up on the finger or we will miss all the heavenly glory.</p><p>Ideas Synthesized From:</p><p>Alan Watts, Bruce Lee, and Jiddu Krishnamurti</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FZKau4zDnYEI%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DZKau4zDnYEI&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FZKau4zDnYEI%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/216d5cd4bfa2038b8f4d99ad50e1e335/href">https://medium.com/media/216d5cd4bfa2038b8f4d99ad50e1e335/href</a></iframe><ul><li><a href="https://a.co/d/3kP5acW">Behold the Spirit: A Study in the Necessity of Mystical Religion</a></li><li><a href="https://a.co/d/8U687Ng">The Book: On the Taboo Against Knowing Who You Are</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1e3c74fc6443" width="1" height="1" alt=""><hr><p><a href="https://medium.com/inside-out-through-the-looking-glass/images-of-the-world-1e3c74fc6443">Images of the World</a> was originally published in <a href="https://medium.com/inside-out-through-the-looking-glass">Inside Out: Through the Looking Glass</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Death of the Intellectual]]></title>
            <link>https://medium.com/inside-out-through-the-looking-glass/the-death-of-the-intellectual-1bd34e442208?source=rss----78af52dc2fed---4</link>
            <guid isPermaLink="false">https://medium.com/p/1bd34e442208</guid>
            <category><![CDATA[consciousness]]></category>
            <category><![CDATA[intellectual]]></category>
            <category><![CDATA[empty]]></category>
            <category><![CDATA[mind]]></category>
            <dc:creator><![CDATA[RAM ESHWAR KAUNDINYA]]></dc:creator>
            <pubDate>Mon, 29 Apr 2024 22:54:02 GMT</pubDate>
            <atom:updated>2024-04-29T22:54:02.494Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/570/1*O2C4ZIH6Kwu48PSyr7MN2w.jpeg" /></figure><p>To know the world. This fundamentally is the role of the intellectual. This is fundamentally the downfall of the intellectual.</p><p>The philosopher Jiddu Krishnamurti — the man who said he was no guru and so acquired many non-followers — once asked “Can you contain the ocean in your hand?” In other words, can you contain the infinite within the confines of thought?</p><p>The intellectual who believes his thoughts, his viewpoints, his interpretation of the world, begins to place a net on the world. While nets are meant to capture things within them, nets are meshed items. They contain holes. Those holes allow the structure of the net to form. In other words the net which captures the thoughts and theories of the intellectual necessarily contains EMPTINESS to cast its net. And in so casting the net, it can only contain what is contained within the net. But if the net were all encompassing, there would be no net. There would be no outside to know that there was a net at all.</p><p>The intellectual has this problem you see. He believes his words more than the reality of the world. He can no longer see the totality in his quest to capture a phenomena.</p><p>When one sees this one sees the urgency of the death of the intellectual. The intellectual who believes his thoughts more than the truth they point towards will never be a complete intellectual. The true intellectual must allow for the death of the intellectual.</p><p>It is in this process which one truly discovers what the intellectual was attempting to capture in his net.</p><p>As Bruce Lee said:</p><p>“To learn to die, to accept defeat, is to be liberated from it. So when tomorrow comes, we must free our ambitious minds, and learn the art of dying.”</p><p>The intellectual must die before he can once again become the intellectual he sought to be.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1bd34e442208" width="1" height="1" alt=""><hr><p><a href="https://medium.com/inside-out-through-the-looking-glass/the-death-of-the-intellectual-1bd34e442208">The Death of the Intellectual</a> was originally published in <a href="https://medium.com/inside-out-through-the-looking-glass">Inside Out: Through the Looking Glass</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Brief History of Natural Language Processing (NLP)]]></title>
            <link>https://medium.com/inside-out-through-the-looking-glass/a-brief-history-of-natural-language-processing-nlp-5b77c71cd374?source=rss----78af52dc2fed---4</link>
            <guid isPermaLink="false">https://medium.com/p/5b77c71cd374</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[natural-language-process]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[transformers]]></category>
            <dc:creator><![CDATA[RAM ESHWAR KAUNDINYA]]></dc:creator>
            <pubDate>Wed, 03 Apr 2024 13:13:31 GMT</pubDate>
            <atom:updated>2024-04-03T13:13:31.591Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>To Know Words By The Company They Keep</strong></p><p>Much of modern NLP is statistics driven understanding of language as opposed to a Chomsky-esque approach of understanding language as syntactic trees. For many years Google Translate relied on syntactic tree based translations but as with much tree based methods, it was rigid, inflexible, and not robust enough. When Word2Vec and Recurrent Neural Networks came on the scene, Google Translate switched to these statistic driven methods to great success leading all the way up to present day transformers. The ideas behind statistic driven NLP are captured well in the much cited quote by John Rupert Firth, “You shall know a word by the company it keeps” (<a href="https://cs.brown.edu/courses/csci2952d/readings/lecture1-firth.pdf">link</a>).</p><p><strong>Continuous Bag of Words (CBOW)</strong></p><p>The earliest form of NLP focused on capturing statistics of various words within matrices. Two matrices in particular — the Term Document Matrix (TDM) and the Co-occurrence Matrix (CoM) were used to capture statistics of words across documents. The term document refers to some separate file e.g. website A could be a document and website B could be another document. How you choose to split data into documents is up to you, but the key idea is that there are statistics of words within a document and statistics of words across documents. The TDM and CoM are combined together to produce the CBOW matrix.</p><p><strong>Term Document Matrix</strong></p><p>This matrix has dimensions terms x documents. Each value represents the frequency of a term within a document.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/951/1*nfT_AvOtdqOZjRy4dw-KeA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/534/1*DAjkISen4eJGyz9rrroxnw.jpeg" /><figcaption><a href="https://images.app.goo.gl/kVxBP6C4Y1aLBQZm9">https://images.app.goo.gl/kVxBP6C4Y1aLBQZm9</a></figcaption></figure><p>This can be combined with what’s called the Inverse-Document Frequency (IDF) which is a combination of Term-Frequency (TF) and Inverse Document Frequency (IDF):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SD94-yBuI-2-pUkVDABCGw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HOAba2UFL6-mvsCBgdvNIg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/769/1*KBnCEdGn64ORUGgt9sy90w.png" /></figure><p>All this yields the same dimensionality matrix of terms x document. What each entry in the TF-IDF (Term Frequency Inter Document Frequency) matrix tells us is given a specific term and a specific document, how likely are we to see this term? Or in other words how surprised should we be in seeing this term in some other document?</p><p><strong>Co-Occurrence Matrix</strong></p><p>This uses a concept called a sliding window. You pick a center word, then pick a sliding window context around that word. Say you had “My favorite color is yellow” with a sliding window size of 1. If we chose color as our center word, we would then get a full context window of “favorite is”. We exclude the center word. Then we count up the co-occurrence of each word with each word in its sliding window context. We add this to a co-occurrence matrix. The matrix will end up looking something like this where p(t_i, t_j) is a normalized probability of the frequency of two token co-occurrences divided by the total co-occurrence of all tokens with token j.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/282/1*v6JTOs3Xvf-SgE7M9WSNdw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/1*FWbuh037iCYa0YXvE-6tJg.png" /></figure><p>t_i and t_j above are tokens i and j. It’s important to note here that while Term Document Matrices are counting FREQUENCIES across DOCUMENTS, Co-Occurrence Matrices are collecting PROBABILITIES WITHIN a SINGLE DOCUMENT.</p><p>The size of this matrix is tokens x tokens.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/797/1*hGIIklpzdDKCH8TtfE_Gqw.png" /></figure><p>You can use a Co-Occurrence matrix to find the most-similar other tokens to a single token by taking the cosine similarity (dot product) between the probabilities contained in a single row (or column) and the CoM. The idea being, two words which are similar to one another will also have similar co-occurrences (to know words by the company they keep).</p><p>In practice because a terms x terms size matrix is huge, we would use a matrix factoring technique called Singular Value Decomposition to shrink the size of this matrix. This technique is called SVD; if you don’t truncate the final output will give you the exact same matrix represented as three matrices multiplied with one another. In practice, we would reduce the dimensionality of the output so that we can get a matrix which is very close to the original matrix but in much smaller dimensionality.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*FDHzuZqDHh99lRnPqciNyg.png" /><figcaption>You can have Sigma be a mxd instead of a mxn and have V be a dxd thereby reducing the dimensionality of your output to X is all real numbers in range m x d</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/179/1*MOnvDJk5CIWlqbETcdUqNQ.png" /></figure><p><strong>Continuous Bag of Words (CBoW)</strong></p><p>This is a way to combine the TDM (or TF-IDF) and CoM into a single matrix which captures both the statistics together in one place. That’s why it’s called a continuous bag of words since it’s essentially mixing up the two statistic matrices in some way which we think preserves meaningful information. You end up with a documents x dimensionality reduction matrix. This is a nice concise matrix which captures a lot of statistics without having any matrix which is token size (i.e. vocabulary size). Storing a matrix which contains a row or column for every token will be huge and so this matrix is a very useful compact representation of the two above matrices.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TNcOAA9C8PpcmLKcP5bKig.png" /></figure><p>Note — you can change the quality of these statistics by altering the sliding window size used in the CoM, changing gamma weighting (i.e. weighting words further from the center word in a sliding window context higher or lower), changing the amount you reduce dimensionality, whether you included spaces in the tokenized input, etc.</p><p><strong>Word2Vec</strong></p><p>Word2Vec is a technique developed by Tomas Mikolov at Google in 2013. The core idea is to find meaningful vector representations of words which can be used in language modeling. It uses an encoding matrix and a decoding matrix. You multiply your input word (or words if you have multiple words as context) by an encoder matrix. That encoder matrix is multiplied by a decoder matrix. You then take the softmax of the output of the decoder matrix (i.e. find the predicted value with the highest probability) and select that as the next predicted word. You can also use it to simply find the most similar words to the context you are putting into the system. How you choose to use the output of the decoder is your choice, but the key concept here is the idea of an encoder-decoder system.</p><p><strong>Notes:</strong></p><p>|w| means vocabulary size</p><p>1-Hot Encoded vector means a vector with all values 0 except for a single location</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/874/1*c1Sdlw8o7Bt-C2oKvZhmug.png" /></figure><p><strong>CBOW Variant</strong></p><p>CBOW Variant of Word2Vec’s objective is to predict a center token given your context tokens. This gives you a good model of high frequency tokens.</p><p>The general steps of CBOW variant of Word2Vec are:</p><ol><li>Create 1-hot encoded vectors for context</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/871/1*ruJcQ9mUDCxMaZSxr26ssQ.png" /></figure><p>2. Use 1-hot encoded vectors to select for embedding vectors from V relevant to context. E.g. &lt;1,0,0&gt; could be “Ram”, &lt;0,1,0&gt; could be “is”, and “&lt;0,0,1&gt; could be “awesome”; the embedding length x 1 vector in matrix V at row 2 associated with Ram would be the vector encoding of Ram.</p><p>3. Average embedding vectors to get</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/59/1*xjE0xXieDl_h8gz5OZCBtA.png" /></figure><p>(if input is multiple words); NOTE — this will later be the crucial area which transformers improve things</p><p>4. Generate score vector by multiplying</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/379/1*3HN0DYuMljm8ZJsYce25qQ.png" /></figure><p>5. Turn score vector into probabilities using softmax (or contrastive loss/negative sampling optimization of softmax);</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/1*misYt7o9tUNIy0ziAstZ8Q.png" /></figure><p>6. Use cross-entropy loss to measure how close yHat_i is to the actual center word</p><p>7. Optimize U and V using gradient descent</p><p>This is the training algorithm for Word2Vec where the training is self-supervised. In other words, it uses its own language data to predict what the center word of a context should be given the words around it (what we saw in the Co-Occurrence Matrix). The closer it gets to guessing the center word (or token) correctly, the better the model is. This whole process ensures learning of good vector embeddings (the encoder matrix V). Later we will see the decoder U can be abstracted into neural networks or made more complicated but this basic embedding matrix decoder matrix idea is used a lot.</p><p><strong>Side Note on Softmax</strong></p><p>The classic softmax equation is quite computationally expensive as it requires a summation of all terms within a row in the denominator for every element in the row. We do still want to keep the power of the denominator in the softmax which is that it compares each element of a row to every other element in the row. But we can cut back on the computation by using something called a logistic sigmoid function. This compares each element to $1 + e^{-itself}$. This gates the output to between 0 and 1 (maintaining a probability like we want) but gets rid of the massive summation in the denominator of softmax. To get back the comparison across all elements in a row softmax gave us, we can choose some samples which are “negative samples”. In other words choose center tokens which are the wrong token and add the probability of not that to our loss function. So loss function would be a combination of cross entropy of correct prediction and cross entropy of (1-wrong prediction).</p><p>Some math associated with that I won’t lay out here — key takeaway is that we can optimize the training efficiently by using a negative sampling or contrastive loss gradient based on logistic sigmoid function rather than raw cross-entropy using the softmax activation function.</p><p><strong>Skip-Gram Word2Vec Variant</strong></p><p>While Word2Vec was trained on predicting a center word based on neighboring words, Skip-Gram’s training objective is to predict neighboring words given a center word. The result is that skip-gram gives you a better model of low frequency tokens while the CBOW Word2Vec Variant gives you a good model for high frequency words.</p><p>The algorithm steps of Skip-Gram are:</p><ol><li>Represent token via encoding vector V as</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/88/1*bq1xzbf4Q4MMITCZ68z6dw.png" /></figure><p>2. Decode center to get a score vector:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/410/1*7vreq3R7SfV_0WiOP25IMQ.png" /></figure><p>3. Pass score vector through activation function (softmax or contrastive-loss) and transform into probabilities:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/364/1*IXBrzHN-8BEMrEoPNKlf6A.png" /></figure><p>4. Evaluate loss using cross entropy for all token targets</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/89/1*HJ1ZR5WvXCxAYRYzwoyV8g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/1*h9M9CvOPQkwbXxzLbScGHQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yt9FK3bq7fK61XbzgL9jnw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/305/1*U8Lmcy95lo3HOutdM8oAlg.png" /><figcaption>The context of token i for a given sentence with a sliding window context of size k.</figcaption></figure><p><strong>Sub-Words/FastText Algorithm</strong></p><p>The idea of a word is loosely defined. Some languages don’t contain spaces between words unlike English (e.g. Japanese). As we saw with the CBoW, keeping matrices with vocabulary size dimensions is huge and untenable. Also we have an issue where if we see a word which is out of vocabulary, we don’t have a good way of handling that. We default to an out of vocabulary embedding row. It quickly becomes infeasible to create embeddings for every possible word as the calculations and matrix size needed to store this is huge.</p><p>The key idea is that we can use subword phrases — similar to phonemes like “ca” or “de” — to cut down on vocabulary size. If you observe, the alphabet has a limited 26 letters in English. As you combine these letters into words and phrases, the number of possibilities rises exponentially. However, the closer you get to representing things as individual letters, the smaller your vocabulary size has to be. Also this helps solve the out of vocabulary problem where new words can still be recognized because we have existing sub-word phrases which are contained in that word.</p><p>The tradeoff is that it can be slower to train a subword model as tokenization (splitting input data into separate tokens) takes longer and inference time takes longer. We’ll see later transformers address the inference time issue by having the encoder portion of the network deal with subword encodings but the decoder portion of the network deal with whole word encodings. Another tradeoff is that training a model on subwords requires a deeper neural network than if your model assumed all inputs were words.</p><p><strong>N-Grams</strong></p><p>In order to split tokens into subword pairings, we use an idea called n-grams. Each word create a gram set $G_w$. We would select a range of n e.g. $n \in \set{3,4,5,6}$. We also prepend and append angled brackets to help the model distinguish between two words.</p><p>For example, if we had the word “where” with n-gram range defined above, we’d get:</p><pre>&quot;&lt;wh&quot;<br>&quot;whe&quot;<br>&quot;her&quot;<br>&quot;ere&quot;<br>&quot;re&gt;&quot;<br>&quot;&lt;whe&quot;<br>&quot;wher&quot;<br>&quot;here&quot;<br>&quot;ere&gt;&quot;<br>&quot;&lt;wher&quot;<br>&quot;where&quot;<br>&quot;here&gt;&quot;<br>&quot;&lt;where&quot;<br>&quot;where&gt;&quot;<br>&quot;&lt;where&gt;&quot;</pre><p>We can compare this to n-gram of her which would have:</p><pre>&quot;&lt;he&quot;<br>&quot;her&quot;<br>&quot;her&gt;&quot;<br>&quot;&lt;her&quot;<br>&quot;her&gt;&quot;<br>&quot;&lt;her&gt;&quot;</pre><p>We can see that having the angled brace helps distinguish the phrase her in where from her because the model can see her in the case of where is contained not at the beginning of the word but in the middle of the word.</p><p>The full words also get their own separate embedding so the full word “&lt;where&gt;” and “&lt;her&gt;” have their own embedding separate from a sub-word embedding. The angle brackets are maintained to help distinguish the full word embedding vs a sub-word which may otherwise be identical to a full word.</p><p>The idea of how to use sub-words is to do essentially the same thing as skip-gram variant of Word2Vec, but change the loss function slightly.</p><p><strong>Skip-Gram Loss Reviewed</strong></p><p>In the skip-gram variant of Word2Vec we had in general form a loss function which looked to minimize the log-likelihood of the outputted probability.</p><p>As a refresher, log-likelihood is an objective function which says, given a bunch of probabilities, give me the joint probabilities of all of these probabilities. More formally, if you have a bunch of observations {x_1,x_2,…x_n} and you want to write the joint probability of all of these observations, you’d say {p(x_1)*p(x_2)*…*p(x_n)}. We can write this succinctly as the product sum of all these probabilities:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/334/1*gdPTQyyrbe4xiuvy6yQlDA.png" /></figure><p>We can rewrite this joint probability by taking the log of the whole thing. The log product rule means we can turn it into a summation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S1MDM87Bn-iU_0ohtppe0w.png" /></figure><p>This naturally leads to our skip-gram/FastText objective function. In the CBOW Word2Vec we used a cross-entropy objective function to score how off we were from our correct prediction. If you work through the math, you can simplify the cross-entropy function to $-log(p(w_{center}|w_{context})$. In other words the loss function becomes a log-likelihood. This means that the general form of our skip-gram CBOW loss function is given a center token, what is the log loss probability of all its context tokens? In other words, what is the joint probability of all the context tokens given a center token.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*73bI2Yx6HXSreGKnfKAQlw.png" /></figure><p>C_t is the set of all tokens int he context window centered around w_t.</p><p>The noise-contrastive version of this general form loss function equation is the same general form, but instead of a single log of probability just for the positive sample, you have a second log probability function for the negative samples.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IsoPqETpVZPNeqnp-u-L2g.png" /></figure><p><strong>FastText</strong></p><p>The idea of FastText is to replace the probability inside the log with a custom scoring function. We can use our n-grams from earlier to create this scoring function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/1*k_Wnowag2utmv53OFhuT0w.png" /></figure><p>We have the vector representations of the n-grams defined as $z_g$ and the vector representation of the whole word as $v_j$. The idea is to represent our probability not as the logistic sigmoid activation of the result of the encoder-decoder architecture, but instead as the dot product between all the n-gram vector embeddings and the word embedding.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/189/1*cXGGaMJPwaatA-ClPR5VzQ.png" /></figure><p>This means that we’ve drastically simplified the training process as instead of having to multiply by a big decoder matrix for every word, we simply train directly on the encoder matrix such that encoder embeddings can self-supervise the training. So it can take less than 10 minutes to train on 1 billion words.</p><p>There are some other ways of forming subword pairs such as Byte-Pair Encoding. I won’t go into it here, but the basic idea is to collapse pairs of characters into symbols and recursively do this until there is no pair that’s repeated more than once.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/1*uf0lb9ZFXVFRmEZEr_bZwQ.png" /></figure><p>The tradeoff to remember is that pre-processing data for n-gram or Byte-Pair Encoding takes much longer although training time is greatly reduced.</p><p><strong>Attention</strong></p><p>The idea of attention revolves around finding ways to focus on the relative importance of various tokens in the input to the model. For example if you had the sentence “Dog bites man” that’s not a newspaper headline, but “Man bites dog” surely is! So we want the model to pay attention not just to the words in the context but to their order and relationships.</p><p>There was a long period of time where Recurrent Neural Networks (RNNs) were the state of the art at these types of tasks. RNN’s are neural networks with a loop in them. They look something like this in their simplest form:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/389/1*XH6LeKduPpN9fm7Nhva98w.png" /></figure><p>The issues with RNN’s are two fold — one is that they generally need to be quite deep to have good outputs That increases computation time for training and inference. The consequence of the deepness and the fact that there is a loop means that the models are susceptible to the vanishing and exploding gradient problem. This is a problem where the backpropagated gradient signal explodes or becomes 0 because the more gradient multiplication there is, a very small or very large gradient will get grow exponentially with more layers (or with increased t).</p><p>Another problem with these systems was that they would heavily weight the most recent contribution to the model as that is fed directly into the model and all the previous outputs are contained in one massive term which makes the contribution of early words lose their impact.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*orUhYZ53lkrm27EZWKRhOw.png" /></figure><p>So the key question became how do we make language models pay attention to the right things in their input? And can we find a better way than RNN’s?</p><p><strong>Transformers</strong></p><p>Let’s back up to what attention could mean on a very high level. It essentially is some weighting assigned to each word in your context. If you remember our simple Word2Vec model which took an average of all the input context word embeddings, instead of a simple average we want a weighted average.</p><p>For example for “dog bites man” we can assign the weight 0.4 to dog 0.3 to bites and 0.4 to man. Then when man and dog are flipped, the model maintains high attention on those words. This gets to a subtle point though. The famous paper “Attention is All You Need” that set off the transformer revolution is misleading. Attention is NOT all you need as a key part of that paper was the introduction of what is called positional embeddings. If we return to our “dog bites man” example, just knowing the weights of how much to pay attention to each word doesn’t distinguish “dog bites man” from “man bites dog”. But, if you also knew the position of man and dog, then you can clearly see that there’s a difference between those two sentences.</p><p>But for now, lets stick with the idea of attention. We can write attention formally as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/1*wni6PgoEb1yBA1TddYhEWw.png" /></figure><p>W_a is a attention weight matrix</p><p>V_g is the n-gram embedding matrix</p><p>Vbar_g is the n-gram weight matrix with attention weights applied</p><p>Psi is an activation function (e.g. softmax)</p><p>The basic idea is that we learn a weight matrix to apply to the input and then multiply that back against the input.</p><p>The problem with the above is that it will always weight two words in the same way regardless of where in the sentence they appear. The “Dog bites man” vs “Man bites dog” problem.</p><p>Before getting to positional embeddings, lets try modifying how we are applying the attention weights.</p><p>First set up a simple equation where we get the n-gram matrix to pay attention to itself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/811/1*4ekvgD1q1ntnxMyS0LLMig.png" /></figure><p>V_g*V_g^T here is a self-attention block, but because there’s no weight matrices, there will be no learning. So lets add in weight matrices for attention.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kg2gFkfrFn3JYxVRP8VMng.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/536/1*FlG17R3cCAyFTjsGnLEkPw.png" /><figcaption>Key and Query attention weight matrices</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/358/1*Gv7hpC283CrEe-F6dW2uvw.png" /><figcaption>(where n is some input size) is n-gram embedded input sequence (context)</figcaption></figure><p>Adding the weight matrices allows us to contextualize our input as it changes AND the attention we pay to the input.</p><p>While there’s no standard on what is the key or query, you can interpret as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/269/1*hAMGtGbjUxzP6_bghjgpyQ.png" /><figcaption>Query (contextualized input)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/272/1*VrQL8euws-ebOmucrW1e-g.png" /><figcaption>Key (attention values to use)</figcaption></figure><p>In other words, the key and query combine together to behave like a neural network whose weights change depending on the input it sees! This is a very cool idea which has a lot of general use cases outside of just transformers and NLP which is why you see transformers used everywhere and not just in NLP tasks.</p><p>The last piece of the self-attention weighted input puzzle is dividing by the square root of the weight matrix dimension size — this is just done to stabilize the gradient as the system learns. So the full self-attention equation looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K7icSUFQ9rqj6XR2YQqWug.png" /></figure><p>Finally, we can project the result of this into some hidden dimension by multiplying the unweighted input $V_g$ by another weight matrix. This is basically the same thing as passing the result of the self-attention block through a fully connected layer that keeps the input in a reduced or modified embedding space.</p><p>Our final equation will look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OeT0gYadOL5XbJAf2So98A.png" /></figure><p>The three terms you’ll commonly see for this equation are query, key, and value. We’ve gone over query and key above. The value is the term outside the \psi i.e. activation function. Here that is the V_g * W_v term.</p><p>We can write everything above in simplified algebra by setting values as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/758/1*x1RYopi2YczCjA6hH2XrTA.png" /></figure><p>The final dimension of</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/358/1*c9RU72qp2eePhEJ0MqMEDA.png" /></figure><p>Normally the W_v weight matrix is of dimension d which is the same dimension as the key and query matrices.</p><p>Above is what is known as a single attention <strong>head</strong>. We can combine multiple heads together and learn multiple key, query, value matrices.</p><p>Now we can begin describing the full transformer architecture as outlined in the paper “Attention is All You Need”. Here’s a diagram from the paper describing the full architecture:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/863/1*wEkiuBOm7BvojbdZT0wooA.png" /></figure><p>Here’s the encoder portion of the transformer blown up and diagrammed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ak_2PTvlgoluMkZvbsLwpA.png" /></figure><p>Here’s the decoder portion of the transformer blown up and diagrammed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kp7pZHNJNIYli-FDZkMTXw.png" /></figure><p>You’ll note a few key things in the diagrams:</p><p>1. The Key, Value, Query Matrices we talked about above split up into multiple heads. The original paper splits it up into 8 as diagrammed above. They are also concatenated together for faster processing.</p><p>2. There is this idea of a “residual connection” also called a “skip connection” or “highway connection”. These are the arrows in the original transformer diagram which take the original input and add it back to the output of an attention block. Or we see it between the output of an attention block→add and norm to the output of a feed forward layer. These connections basically make it so that when training, there is enough information preserved of the original input that the model can actually learn well at the beginning. As training progresses the residual connection doesn’t contribute much more to the learning. It is just there to help guide early learning of the system.</p><p>3. There is an idea called “positional encoding”. The idea here is that the relative position of a word is passed through something like a sine function. So if a word is in position 1 it has a different value added to it than if it’s at position 3. These values help the model learn the positions of words explicitly which helps with the “man bites dog” problem.</p><p>Here’s an example of what positional embedding might look like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*-RXIhTzlNyc1zqFL964eGw.png" /></figure><p>1. There is a split between the decoder and the encoder. This is an important note. Some LLMs will be decoder only, others will be encoder only, and others will be encoder-decoder models. For most seq-seq models like Chat-GPT or Llama today they are all decoder only models. But because they are used for sequence to sequence tasks (e.g. translating a sentence from one language to another), they can also be encoder-decoder models. The idea of an encoder and a decoder is just our original idea from Word2Vec blown up in complexity with a transformer architecture.</p><p>2. The decoder has this special layer called a masked layer. All this is is a way to train the model on a single input sequence all at once. For example, if your input sequence was “I like cheese”, the masking would separate this out into a 3x3 matrix that looks like:</p><p>| I | -infinity | -infinity |</p><p>| — — | — — | — — |</p><p>| I | like | -infinity |</p><p>| I | like | cheese |</p><p>This lets the model predict the word it expects after I, then after I like without being able to “cheat” and see the next word already. And it lets us train the model to predict every next word in an input sequence all at once in a single batch. We have an input in the decoder of multiple batches (batch size) that lets us do this same thing on a bunch of different sentences all at once. That speeds up training of the model.</p><p>3. There are two models of attention. Self-attention is when the input of the transformer block is used to multiply against the query, key, and value attention matrices. Cross-attention is when the query is from the decoder but the key and value matrices are from the encoder. In the decoder case we want the decoder to take its decoded attention weight matrix and use that to weight the encoder’s attention matrices. This allows the decoder to pay attention to the encoder outputs directly in its downstream task (e.g. language translation).</p><p><strong>3 Classifications of Transformers:</strong></p><p>I mentioned in point 4 above, that there are three types of transformer models. Here is a rule of thumb understanding of what they are and when they are used. You will often see on hugging face models are described in one of these three categories.</p><ol><li>Autoregressive</li></ol><p><strong>A stack of decoder blocks</strong>. Pretrained on casual language modeling. Typically used for text generation.</p><p>2. Autoencoding</p><p><strong>A stack of encoder blocks. </strong>Pretrained on correction of corrupted sentences. Typically used for token-level and sequence-level classification tasks.</p><p>3. Sequence to Sequence</p><p><strong>Full encoder-decoder architecture</strong>. Pretrained typically for desired down-stream task like translation, summarization, or question-answering.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5b77c71cd374" width="1" height="1" alt=""><hr><p><a href="https://medium.com/inside-out-through-the-looking-glass/a-brief-history-of-natural-language-processing-nlp-5b77c71cd374">A Brief History of Natural Language Processing (NLP)</a> was originally published in <a href="https://medium.com/inside-out-through-the-looking-glass">Inside Out: Through the Looking Glass</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Guide to Progress]]></title>
            <link>https://medium.com/inside-out-through-the-looking-glass/a-guide-to-progress-cc09897fdf98?source=rss----78af52dc2fed---4</link>
            <guid isPermaLink="false">https://medium.com/p/cc09897fdf98</guid>
            <category><![CDATA[enlightenment]]></category>
            <category><![CDATA[progess]]></category>
            <category><![CDATA[scientific-theories]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[theory]]></category>
            <dc:creator><![CDATA[RAM ESHWAR KAUNDINYA]]></dc:creator>
            <pubDate>Sun, 07 May 2023 19:20:41 GMT</pubDate>
            <atom:updated>2023-05-07T19:21:46.938Z</atom:updated>
            <content:encoded><![CDATA[<p>Today’s society is marred by a lack of civil discourse and a general aversion to progress. I believe a revived commitment to the Enlightenment ideas of progress, science, humanism, and rationality as Stephen Pinker summarizes them in “Enlightenment Now”, is necessary for the continued flourishing of American culture.</p><p>As a student of Artificial Intelligence, my focus is often on engineering systems which exhibit intelligent behavior. But it is equally focused on thinking about what qualifies such behavior. It rests on ideas of philosophy, science, mathematics, and computer science. AI’s role is to provide the infrastructure to build on theories of intelligence and so I’m fundamentally concerned with this idea of progress. Progress cannot be achieved without a commitment to testing new ideas and making forward steps. We need some pillars on how to achieve these forward steps.</p><p>As the philosophy of science tells us, the way to approach exploring a hypothesis is to try and prove it wrong. In other words, what are the ways in which we can say something is NOT so. So in analyzing how should we approach progress, let’s look at the ways in which we should not approach it.</p><p>Allen Newell, a prominent figure in the history of AI and cognitive science, laid out his ideas of unified theories of cognition in the William James Lectures at Harvard in 1987. His framing of risks to posing such ideas spoke poignantly about how we should debate ideas and approach progress.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V-EDOcr0RRhqnsfiAy4Bog.png" /><figcaption><a href="https://vimeo.com/groups/632757/videos/393683244">https://vimeo.com/groups/632757/videos/393683244</a></figcaption></figure><p>I will reframe these ideas here in my own words.</p><ol><li>Falsification rather than approximation</li></ol><p>We should not seek solely to falsify an idea or to prove why a forward step is wrong. Rather we should approach an idea as an approximation of what we wish to achieve. This way, we do not simply see-saw from one idea to another without making any incremental progress.</p><p>2. Historical momentum</p><p>The taboos and tepidity of past thinkers should not pigeon hole your ambition. To take a contemporary example, Elon Musk deciding to start a private space faring company goes against the traditional thinking of the space and aviation enterprise as a domain only available to large scale government projects. If this momentum always dictated the actions of the future, Space X would never have been born yet alone achieve the success it has. Or a game programming example. John Carmack, John Romero, and friends at Id Software famously created the game Wolfenstein 3d on a Intel comptuer designed for word processing and text displays. Wolfenstein was a game which featured one of the first examples of 3D computer graphics in a real-time video game environment on a PC system available to everybody. If the team at Id had only ever viewed the Intel computer as a text processing system, we’d never be shooting Nazi’s in Wolfenstein 3d!</p><p>3. Who are you?</p><p>The question of qualification should not discourage robust and valid thinking. The beauty of logic is that it holds true regardless of who says it. You must contend with the ideas themselves and not the vessel who brought it forward. Therefore, just because one is not Einstein does not preclude one from providing the key ideas to progress. A phrase which is commonly denoted to immigrants of America is, “We don’t care where you’re from, we care where you’re going”. This same spirit applies to the domain of ideas and the steps to progress.</p><p>4. Speaking in Platitudes</p><p>It is easy to make a broad statement about the nature of something without ever saying anything specific. Without specificity, we can never evaluate the strength of an idea. Lao Tzu’s famous saying that laws breed criminals is specific enough that we can then test whether a society with no laws is truly a safer place for the people in it. If Lao Tzu had said “It’s not people’s fault they’re criminals!”, this statement is much to broad and sentimental to make any meaningful measure of.</p><p>5. But where’s my coffee?</p><p>While pessimism absolutely has its place, it must be qualified with constructivism. In other words, it does no good to simply look at a theory as half formed. Rather we must understand the theory is a work in progress and find ways to continue to fill the cup with water. If we simply look at the half full glass and moan about how there’s not enough water, we never get the chance to add more to it.</p><p>6. Look what everybody else is doing!</p><p>There is sometimes wisdom in the masses — we know this from experiments that show when more people estimate how many jelly beans are in a jelly bean jar, the average of those numbers is usually more accurate than an individual guess. You could call this the law of large numbers. But, what distinguishes an expert from a practitioner is the ability to discern through reasoning by fundamentals, what is a step worth pursuing different from what others are doing today. If you think of this in terms of chess, if Bobby Fischer only ever made the moves others had or would do on a board, nobody would speak of him today. We speak of Fischer because of his propensity to do the unthinkable and turn it into a magnificent victory. In other words, see what everybody else sees and say what nobody else has said.</p><p>Newell summarizes these ideas in the words of famous neurophysiologist and cyberneticist Warren McCullough (fundamental to the foundation of neural networks) -“Don’t bite my finger, look where I’m pointing!”</p><blockquote>“Don’t bite my finger, look where I’m pointing!” ~ William McCullough</blockquote><p>This in essence is how not to create progress. This is what we must know to avoid when speaking with each other over contentious ideas today. And this is how we can begin to revive the ideals of humanism, reason, progress, and science in ways which can revitalize our society.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cc09897fdf98" width="1" height="1" alt=""><hr><p><a href="https://medium.com/inside-out-through-the-looking-glass/a-guide-to-progress-cc09897fdf98">A Guide to Progress</a> was originally published in <a href="https://medium.com/inside-out-through-the-looking-glass">Inside Out: Through the Looking Glass</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Computational Theorem, Three Men, And One Elephant]]></title>
            <link>https://medium.com/inside-out-through-the-looking-glass/a-computational-theorem-three-men-and-one-elephant-7687f6b7ada6?source=rss----78af52dc2fed---4</link>
            <guid isPermaLink="false">https://medium.com/p/7687f6b7ada6</guid>
            <category><![CDATA[knuth-morris-pratt]]></category>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[string-matching]]></category>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[pattern-matching]]></category>
            <dc:creator><![CDATA[RAM ESHWAR KAUNDINYA]]></dc:creator>
            <pubDate>Sun, 31 Jul 2022 02:42:44 GMT</pubDate>
            <atom:updated>2022-07-31T13:21:37.245Z</atom:updated>
            <content:encoded><![CDATA[<h4>The Knuth-Morris-Pratt Algorithm</h4><p><strong>A Theoretical Challenge:</strong></p><p>Three scientists — one developing a text editor another studying palindromes and the last studying a theory of computation — all converged upon the same topic to co-create one of the most famous algorithms known today.</p><p>Alongside the programmers creating computer programs, there existed many computer theorists. One such theorist was a man by the name of Stephen Cook. A professor at UC Berkeley, Cook was particularly interested in a theory of computation known as a Finite State Machine or Finite Automata.</p><p>The computers we know today, we probably know along these lines: there’s a processor that reads your code, interprets it, accesses some location of memory (RAM), and produces some output. The key components from this view are a processor and some memory operating on numbers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/1*oD6ivX8YfGbdgeUaFwm8zw.png" /></figure><p>But another way to think about computers is to consider them as machines which can only exist in one state at a time. This machine can transition from one state to another based on the input it gets. We call these machines finite state machines.</p><p>Stephen Cook was working on a theorem using a particular kind of finite state machine called a deterministic pushdown automata. This is a type of finite state machine which produces no output. While normally finite state machines have very limited memory, pushdown automata are a type of state machine with a special structure called a stack. This stack can store a certain amount of memory the machine can use. The type of automata Cook was working with was a deterministic pushdown automata meaning whichever state the machine is in, you know exactly which states it can transition to given a certain input.</p><p>Say your machine is in State A and it has two transitions to State B and State C. If your input is 1 and you are in state A, you will go to state B. But if your input is 0 then you know you will go to state C. The transitions tell you which state you can go to given an input — they are deterministic.</p><p>Cook’s theorem revolved around a two-way pushdown automata. If you think of the input to a pushdown automata as a tape, a regular pushdown automata can only read this tape in one direction. A two-way pushdown automata can read this tape in two directions — backward and forward.</p><p>You might be thinking a bit like Donald Knuth was at this point. You see, Knuth was keeping up to date with the theories of computation coming from this finite state machine school of thinking but he said, “I never felt it made me a better programmer” (Fridman).</p><p>So what can a computer programmer get from this abstract theory of computation? Sure, you can theoretically model a computer using this framework but what use is that when you’re trying to implement a program?</p><p>Well as Richard Feynman said, “It is a problem as to whether or not to worry about the philosophy behind ideas” (Feynman). Feynman lays out an imaginary scenario between a Mayan astronomer and a student. The Mayans could accurately predict eclipses to stunning accuracy. But there was little discussion as to what the moon was or that the moon went around the earth, just calculations. Suppose the student comes in and says “I have an idea! I have this theory about the moon going around the earth” and the astronomer responds, “Well how accurately can you predict eclipses?” If the student says, “I haven’t got that far yet” then the astronomer could reply well I can and throw out the theory entirely. But that theory allows us to do more than just predict the time of eclipses it allows us to calculate the movement of planets and the position of objects in the sky. It leads to an understanding of gravity in an entirely new way!</p><p>So theory can be important, but so far Knuth had yet to encounter this with finite state machines.</p><p>But he stumbled across a theorem of Cook’s using this two-way pushdown automata. Cook worked out a theorem stating that any language recognizable by a two-way deterministic pushdown automaton in any period of time could be recognized on a random access machine in O(n) units of time.</p><p>A simple way to think about this theorem is to say, all the inputs that our finite state machine can read define some language L. If our input is recognized by our machine in some way, then our machine has recognized the language in our input. Because our input here is a tape which can only move sequentially, if our machine recognizes any consecutive sequence of entries from that tape, it has recognized the language that tape is written in because that sequence exists in the language L our machine can interpret. Cook’s theorem states that in a machine that can move the input tape backward and forward, if that machine can recognize a consecutive sequential string in any period of time, a regular computer using memory and a processor can recognize it in O(n) time.</p><p>For those of you less familiar with computer science notation, O(n) is a way of saying that the most amount of time the computer will take to perform this task is a function f(x) = nx i.e. it is linear. In computers, linear time is very fast!</p><p><strong>Three Blind Men and One Elephant:</strong></p><p>Another scientist steeped in the finite state machine theory of computation named D. Chester came out with work which showed that a set of strings beginning with an even palindrome could be recognized by the type of automaton Cook described. Knuth comes across this theorem and realized this was something useful! But he couldn’t imagine how to get a computer to recognize a language in anything less than O(n²) time — exponentially more time than should be theoretically possible! Finally, this was a place where the finite state machine automaton theories could be used for something useful — a computer program!</p><p>Knuth wanted to find out how this algorithm was working in the automaton so that he could apply those principles to a processor/memory (RAM) based computer.</p><p>After plastering everything about automaton theory on his blackboard and trying to refine his algorithm, he finally put together the last piece of the puzzle. Little known to him, Jim Morris, another computer scientist at Carnegie Mellon University, had found a similar solution independently also inspired by automata theory.</p><p>Morris was working on a text editor for the CDC 6400 computer and had a problem with storing text in a buffer. Buffers are used as ways to quickly access text that has just been typed or to print things more efficiently to your computer screen. They are a sort of auxiliary store of memory. Morris wanted to be able to create text buffers without moving a text file pointer backward (a pointer is a computer programming object pointing to a location in memory). He devised an algorithm which runs in O(m + n) time (much better than the O(n²))! But the implementors of the text editor didn’t fully understand it as it was much more complex than the standard way of solving this problem and when Morris returned to the program later, he found his method completely changed.</p><p>Knuth meanwhile showed his algorithm to another computer scientist named Vaughan Pratt. Pratt, currently a professor at Stanford University, took Knuth’s work and modified it. Initially Knuth had a data structure whose running time was proportional to the alphabet size of the input to the computer (i.e. how many possible letters were in the alphabet). Pratt, took his data structure and modified it such that its running time was independent of alphabet size. He also showed this work to Morris who was delighted to see the proof that this method ran in O(n + m) time. Morris’ original work did not prove/explain this very well and the method Knuth discovered was the same as Morris!</p><p>So three scientists from different paths collided upon this algorithm. But what is this algorithm they discovered?</p><p><strong>A Naïve Approach:</strong></p><p>Say you are looking for a certain string of characters in a text. We will call the string you are searching for the pattern and the text, text. How would you find the pattern “baby” in the sentence “Babe Ruth, am I a baby”? The naïve way is to begin at the first word of the sentence text array, check if it’s the same letter as the first letter of our pattern. If it is, then increment our text and pattern pointer to the next position. Keep doing so. If we never hit a mismatch and hit the end of our pattern, then we have found our string. But, if we do hit a mismatch, then we have to go back to the next letter in text and start the process all over again.</p><p>In our toy example here, we can see how this is a problem as at first “bab” will match with “bab” in our text. But then we hit an e and have to restart our search from the next letter of text, the “a” in “Babe”. But we’ve already read up to the “e” in “Babe”, so why do we need to search through the “abe” again?</p><p>In this example, we don’t lose a lot of time running this algorithm, but consider the worst case where we have a pattern “aaaaaaab” and a text array “aaaaaaaaaaaaaaaaaaa”. Here we have to go through every single character in our text array and compare to every single character in our pattern all to never get a match. This naïve approach has a worst case run time of O(n * m) where n is the length of text and m is the length of our pattern. If m is very close to n in size, then the runtime of this algorithm could be O(n²).</p><p><strong>The Elephant:</strong></p><p>Now we know from Cook’s theorem and Chester’s work that we should theoretically be able to solve this problem in O(n) i.e. linear time. We want our solution to approach this algorithmic complexity.</p><p>The Knuth-Morris-Pratt algorithm builds around the key insight that if we are scanning through our text array and have found matches, when we hit a mismatch, we already know that the characters up to that point have matched in some way. This is especially useful in cases where the pattern we’re looking for has some repeating text (this is where Pratt and Chester’s work with palindromes ties in!).</p><p>Say we have the pattern “mamamia” and are looking in the text “cmamimamamamia” for a match. One way to imagine how the KMP algorithm works is to think of the pattern we are trying to match as a tape (remember how Cook’s theorem relied on a two directional tape?), moving to the right above the text we are searching, while we keep track of the location of text we are currently looking at.</p><p>First we will scan through our text array until we hit our first match between text and pattern, the second element “m”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/510/1*d8wQgtxJesLv--7GLw2Irw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/516/1*znDDYm7xclV6Wt77OPYvHw.png" /></figure><p>We can align our pattern string on top of our text starting from the second element. Then we continue comparing the two until we hit a mismatch at “i”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*udDaY3tWtoNV_auoDtUR2g.png" /></figure><p>Now, we see that there is no substring in pattern before the current letter of our pattern which matches letters in text before this position. So we move the whole pattern to the position after the current pointer location.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aYjkMauO7UtL4dwhwdZhtA.png" /></figure><p>Then we increment our pointer and keep searching. Again we keep going until we hit a mismatch here:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KmVP5yMjbQzDxhInwTWvpw.png" /></figure><p>So far we have seen some great improvement over our naïve algorithm by never having to move our text pointer backward. But, here we get to see the other great improvement of the KMP algorithm. By relying on the fact that we remember the matching prefix strings in our pattern to suffix strings (i.e. by remembering letters earlier in the word which are repeated later in the word), we know that we only have to move our pattern tape two positions to maintain a match with our text. So here rather than moving the whole pattern one element past the current text pointer location, we shift it two places to the right.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SUWX1TarP8VrtL9v0YSswQ.png" /></figure><p>We notice that the letters still match and therefore we continue to the next letter in text and compare. Now we can keep going up to the end of text and confirm that we did indeed find the pattern we were looking for!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gZqRKaNZ09jMxmqLkdVm7g.png" /></figure><p>It’s important to note that programmatically we don’t need to actually shift our pattern to the right, all we do is move a pointer to our pattern array to the right position. Before describing how we know what position to shift our pattern pointer to, lets first see if we can describe our current algorithm in a general way.</p><p><strong>The Beating Heart:</strong></p><p>The pattern match process has a few steps:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/562/1*radCLs5Gszr4FrkfRVnSOg.png" /></figure><p>We can describe this through some pseudocode as well. If k is an iterator through our text array of size n, and j is an iterator through our pattern array of size m, and we are indexing our array from 1-n/1-m:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/496/1*k6MOc7nHij-TLa118G0Ogg.png" /></figure><p>So we are saying, while our pattern pointer is less than the size of our pattern array and our text pointer is less than the size of our text array, while our pattern pointer is greater than 0 (not at the first letter of our pattern) and our current text letter does not equal our current pattern letter, first find the right position to move our pattern tape to. If we found a non-zero pattern position to move to and our pattern letter and text letters match, move on to the next text and pattern letter. Keep moving on to the next pattern and text letter. If we never hit a mismatch, then we have found a matching substring and we exit out of the while loop. If we hit a mismatch, then we figure out the next position to move our pattern tape to and then repeat the process.</p><p><strong>The Quick Witted Brain:</strong></p><p>Awesome! So this tells us how to find a matching substring within a text in just the way we were looking for. But there was one part of this which was unexplained — how do you find the amount to shift our pattern to the right? In our pseudocode here this was our:</p><p><strong>do </strong>j = next[j]</p><p>One way to interpret what’s happening with the next array is that at element j in our pattern, it tells us the number closes to j, lets say i, which contains a matching prefix string to the suffix string at j. So i is the largest index closest to j indicating the smallest amount needed to shift our pattern to the right. More literally it tells us the index to jump the pattern pointer back to in order to continue comparing pattern with our current text pointer position.</p><p>The first element of our next array will always be 0 (or -1 depending on how you choose to implement array indexing). This is because at the first position of our pattern there can be no matching prefix pattern (i.e. that letter cannot be a repeating pattern) because there are no letters before it.</p><p>But from then on we need some algorithm to calculate what our next indices should be i.e. our indices representing how much to shift our pattern string to the right.</p><p>What we can do is calculate an intermediary array f[m] (same size as our pattern array).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/614/1*fvfXAVTJ1nP85r38Rg4VIA.png" /></figure><p>What we’re looking at here is a table with three rows — j, the index of the pattern array; pattern[j], letter at that location in our pattern; and f[j] the value at index j of the intermediary f[j] number. So what do these intermediary f[] numbers mean? Well if you look closely, the first element is 0 as we expect (no repeating pattern exists before the first element). The second element is 1, also meaning that there is no matching substring, but this time we want to return to the index 1 (i.e. the beginning of the word) as we are not currently at the beginning of the pattern. Now it gets interesting. At index 4, we do have a repeating pattern, “ma” exists at indices 12 and 34. You notice here that anytime we have a repeating string, the value of f[j] is the value of f[j-1] + 1 (starting from the second matching letter of the substring). So indices 3, 4, and 5 have numbers going up from 1, 2, to 3. Then we hit a mismatch again and so we no longer have a repeating substring — so our f[j] index tells us to return to the first element of our pattern.</p><p>So f then, tells us what up to what index we have a matching prefix string to the current repeating substring.</p><p>To describe this programmatically, we are saying, if pattern[j] = pattern[f[j]] then f[j+1] = f[j] + 1. So if we have a matching prefix up to our current index j, and the letter at the suffix string j is the same as the letter at the letter at f[j] , then we can say that the position to move our pattern tape to for the next element in our pattern should be f[j] + 1. So in the case of our repeating ma pattern, index 3 has f[3] = 1, pattern[1] = m and pattern[3] = m, so f[4] then equals f[3] + 1 = 1 + 1 = 2.</p><p>If however, pattern[j] != pattern[f[j]] then we no longer have a repeating substring, but we can use the same pattern matching algorithm we talked about before to find another matching substring! We set the text we want to search through as our pattern and search our pattern against itself. So now we can figure out how to calculate next using our f[] — for any j &gt; 1:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/726/1*BYybYdz0tqn65PwSsftk-A.png" /><figcaption>(Knuth et. al. 328)</figcaption></figure><p>Remember we interpreted f[] before as “up to what index do we have a matching prefix string to the current repeating substring”. Well, next now tells us what index to move our pattern tape to in order to continue checking from an earlier repeat string in our pattern.</p><p>Applying our calculation to our “mamamia” toy example, we get:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/613/1*pgUomjUHO60OUVPeG5IQjQ.png" /></figure><p>The interesting points here are when pattern[j] = pattern[f[j]] at index 5 (the m at 5 is equal to the m at 3), we then use our next array value at index 3. This is because we already know the substring at our current text index k does not match the substring from 3–5, and we know that 3 is a repeat letter of 5, so we can just shift the pattern tape all the way to the first index and restart comparisons.</p><p>If our toy example was slightly more complex, say we had a pattern with repeating subsets 3 letters long, then we could see a case where we need to jump back to an earlier repeated string to compare with our current index text[k] before jumping all the way to the beginning of our pattern. Let’s take “XYZXYZXZ”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*Ppi6vIEt6Bx3WOZ1G_d6qg.png" /></figure><p>A mismatch at index 8 doesn’t mean that the second repeated XYZ won’t match with the current letter since the letter next tells us to jump to in this case is “Y” at index 5 which is different from “Z” at index 8. So the letter at text[k] could match with the letter at pattern[5] in this case.</p><p>In this way our next array lets us accurately check for repeated substrings within our pattern array without ever having to move our text pointer back!</p><p>With this, we are ready to write out pseudocode for the algorithm to build our next array.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/404/1*7NsfSSYvYk9zNBZdojzB1Q.png" /></figure><p>You’ll notice that the first few lines look pretty much exactly like our pattern matching algorithm we saw before. We simply swapped the and condition of our first while look so we compare pattern[j] against pattern[t] (right shift the pattern array against itself) and then set the next[j] value according to the rule we mentioned earlier.</p><p><strong>Was Cook Right?</strong></p><p>This next building algorithm will take O(m) time to run. You’ll notice that the inner while loop runs no more than the number of times of the outer while loop and will often run far less often than the outer while loop since we will have fewer repeating strings in our pattern than letters in our pattern. Therefore, we will run at most O(2m) times and in asymptotic analysis this becomes O(m).</p><p>Our original pattern matching algorithm has the same big oh analysis but is now dealing with n elements of our text array. So this will have a O(n) running time. Multiply these two together and we have a total runtime of this algorithm of O(n*m).</p><p>Memory wise, we only require extra space for our next index table which will be the same size as our pattern array. So we will need O(m) memory. For most patterns we search for this should be a small number as we likely look for small strings within large text.</p><p>There are a few additional ways to optimize this algorithm — you can make j = 1, the first index of your pattern array, a special case to cut down on the number of comparison checks that have to be done in cases where you have a large alphabet and won’t hit a matching letter with your pattern for a long time. You can also save on the continuous while check is j &gt; m or k &gt; n checks by adding sentinel values to the ends of pattern[], text[], and next[]. You can use the same ideas to speed up creating the next table. And even more interestingly, if your pattern is of manageable enough size m, and you want to make a special case for every single j in your pattern, this will run even more efficiently!</p><p><strong>Legacy:</strong></p><p>Today there is an interesting derivative off of the KMP algorithm known as the Boyer-Moore algorithm. This builds on the KMP idea but both Boyer and Moore independently noticed that there could be an even faster way to match pattern strings. If you skip more rapidly over portions of your text array that can’t possibly lead to a match, then you can look at lest of your text and come up with a match — as long as this is done precisely. Zvi Galil also came up with a real time algorithm developed around the idea of letting the text pointer move while the j pattern pointer is moving down. The Knuth-Morris-Pratt algorithm and its derivatives are used from search engines, to text editors, to IDE’s today.</p><p><strong>Concluding Thoughts:</strong></p><p>The Knuth-Morris-Pratt algorithm is an ingenious way of applying a computational theorem to computer programming so that a theoretically predicted outcome gets a real implementation. Knuth joked in an interview that he was originally writing the paper titled, “Automata Theory can be Useful”! And Knuth quite humbly notes the importance of each member in discovering this algorithm saying, “each one of us discovered a different part of the elephant”.</p><p>References</p><p>Feynman, Richard. “Richard Feynman on ‘Philosophies behind Ideas’ (or Worldviews behind Theories).” <em>YouTube</em>, 8 Jan. 2019, <a href="https://www.youtube.com/watch?v=jW7sDih_CpQ.">https://www.youtube.com/watch?v=jW7sDih_CpQ.</a></p><p>Fridman, Lex. “Donald Knuth: Programming, Algorithms, Hard Problems &amp; the Game of Life: Lex Fridman Podcast #219.” <em>YouTube</em>, Lex Fridman Podcast, 9 Sept. 2021, <a href="https://youtu.be/EE1R8FYUJm0?t=5056.">https://youtu.be/EE1R8FYUJm0?t=5056.</a></p><p>GeeksforGeeks. “KMP Algorithm for Pattern Searching.” <em>GeeksforGeeks</em>, 20 July 2022, <a href="https://www.geeksforgeeks.org/kmp-algorithm-for-pattern-searching/">https://www.geeksforgeeks.org/kmp-algorithm-for-pattern-searching/.</a></p><p>Knuth, Donald E., et al. “Fast Pattern Matching in Strings.” <em>SIAM Journal on Computing</em>, vol. 6, no. 2, 1977, pp. 323–350., <a href="https://doi.org/10.1137/0206024.">https://doi.org/10.1137/0206024.</a></p><p>Mihailescu, Clément. “Ace the Coding Interviews.” <em>AlgoExpert</em>, <a href="https://www.algoexpert.io/questions/knuth-morris-pratt-algorithm.">https://www.algoexpert.io/questions/knuth-morris-pratt-algorithm.</a></p><p>nesoacademy. “Finite State Machine (Finite Automata).” <em>YouTube</em>, YouTube, 23 Dec. 2016, <a href="https://www.youtube.com/watch?v=Qa6csfkK7_I.">https://www.youtube.com/watch?v=Qa6csfkK7_I.</a></p><p>Sedgewick, Robert, and Kevin Wayne. “Algorithms.” <em>Princeton</em>, Princeton University, 2 Dec. 2015, <a href="https://www.cs.princeton.edu/courses/archive/fall15/cos226/lectures/53SubstringSearch.pdf.">https://www.cs.princeton.edu/courses/archive/fall15/cos226/lectures/53SubstringSearch.pdf.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7687f6b7ada6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/inside-out-through-the-looking-glass/a-computational-theorem-three-men-and-one-elephant-7687f6b7ada6">A Computational Theorem, Three Men, And One Elephant</a> was originally published in <a href="https://medium.com/inside-out-through-the-looking-glass">Inside Out: Through the Looking Glass</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Thousand Brains Theory of Intelligence]]></title>
            <link>https://medium.com/inside-out-through-the-looking-glass/the-thousand-brains-theory-of-intelligence-d5161e8a6007?source=rss----78af52dc2fed---4</link>
            <guid isPermaLink="false">https://medium.com/p/d5161e8a6007</guid>
            <category><![CDATA[intelligence]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[neuroscience]]></category>
            <category><![CDATA[thousand-brain]]></category>
            <category><![CDATA[cortical-columns]]></category>
            <dc:creator><![CDATA[RAM ESHWAR KAUNDINYA]]></dc:creator>
            <pubDate>Mon, 27 Sep 2021 04:03:46 GMT</pubDate>
            <atom:updated>2021-09-27T04:16:51.731Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/226/1*WJUualsbWxSE61DC7UZNeg.jpeg" /></figure><p>All of us have questioned our minds before. What is intelligence? Why are we able to perceive the world? Can we understand the brain?</p><p>I studied Cognitive Science in college and while I learned a great deal about different aspects of intelligence, I didn’t feel as if I had a cohesive picture.</p><p>Evidently, I was not the only one who felt this way.</p><p>Francis Crick, Vernon Mountcastle, and Jeff Hawkins are all individuals who have similarly said a theoretical framework of the brain is necessary to understand intelligence.</p><p>Francis Crick, a Noble Prize winner famous for playing an important role in discovering the double helix structure of DNA, stated in his 1979 essay “Thinking About the Brain”, “what is conspicuously lacking [in understanding the human brain], is a broad framework of ideas in which to interpret results”.</p><p>Vernon Mountcastle, neuroscientist at John Hopkins University, in his essay published in a small book, <em>The Mindful Brain</em>, begins to lay out grounds for a theory of the neocortex. The neocortex makes up of 70% of your brain and is the part responsible for high level thought and intelligence. Language, music, science, literature — all these things emerge from the neocortex. Mountcastle lays out two groundbreaking ideas:</p><ol><li>The brain grew larger over evolutionary time and added new parts on top of old ones. The old parts control primitive behaviors while the new ones create more sophisticated ones.</li><li>All the various things we see as intelligence — language, high-level thought, sight, hearing — all work on the exact same principle.</li></ol><p>Jeff Hawkins in his book “A Thousand Brains” lays out his incredible theory of intelligence building off the concerns of Crick and the foundations of Mountcastle.</p><p>As an artificial intelligence game developer and cognitive scientist, I have a deep desire to understand and replicate intelligence. I have always had my concerns about artificial neural networks. For one the development of them was not based on natural science. These models were being developed largely as computational and mathematical optimizations for solving specific problems. While inspired by the brain, they were not copying how the brain worked, largely because we didn’t fully understand how the brain worked at the time. Secondly, their applications seemed limited in nature. And there is no sign that this model of intelligence would be able to generalize beyond single trained tasks.</p><p>Hierarchical layers of nodes arranged in a neural network, each layer extracting higher and higher level features, is a useful computational model, but there wasn’t conclusive evidence that this was how the brain functioned.</p><p>Jeff Hawkins lays out many of the problems with that theory of the brain. An idea coming from the hierarchical model is that there are regions of the brain, from the front of the skull back, which extract higher and higher order features the further back you go. A V1 region of vision sees basic features, V2, starts to combine them into edges or colors, and so on until you have an image of a full object. But the so called V1 V2 regions were actually some of the largest areas in the neocortex. There should be no reason that extracting small features require so many neurons.</p><p>Secondly, while the allegory of extracting more and more detailed features from vision makes some sense with vision, that model makes much less sense for hearing and touch. What increasingly detailed features are your ears processing? What higher order of detail does touch extract?</p><p>Thirdly, how are all the inputs from vision combined into one cohesive image? The photoreceptors in our retina have blind spots and are heavily concentrated in the center meaning the images they detect are more a mishmash of features than a cohesive image. The hierarchical network model does not explain how these distorted and missing sensations are combined into one uniform and complete image.</p><p>There are other concerns, including the firing of neurons for feature detection that don’t seem to correspond to any particular features whatsoever, but you can start to see how the hierarchical model of the brain doesn’t seem to hold up.</p><p>When you image the neocortex, you see that its made up of about 150,000 2.5 mm slices of neurons arranged in vertical layers called cortical columns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/147/1*dXB635RdqpV2CZGy34LgAA.png" /><figcaption>Drawn by Nobel Prize winner Santiago Ramon y Cajal, 1899</figcaption></figure><p>And these slices are roughly the same across the entire neocortex. These are called cortical columns. This means that whatever algorithm the neocortex is using to learn about the world and develop intelligence is the same throughout the entire neocortex.</p><blockquote>This means that whatever algorithm the neocortex is using to learn about the world and develop intelligence is the same throughout the entire neocortex.</blockquote><p>Hawkins begins with an insight he had as a graduate student in Berkeley. He noticed that if any object in his room changed in the smallest way, he would notice it. If his stapler was longer that it used to be, put on the windowsill instead of his desk, he would immediately notice. And he would notice without having to pay attention to notice these changes.</p><p>There was only one explanation he could think of — the brain was constantly making multiple simultaneous predictions about what it would see, hear, and feel. Every movement you make, your brain is making predictions. From high level concepts like what should be on his calendar to low level ones like what his coffee cup should feel like, all of these involved predictions based on a model of the world in his brain.</p><p>Firstly, he describes that your model of the world is developed through movement. The world around you is constantly changing and so the inputs to your brain are constantly changing. Secondly, your body is constantly moving throughout this world. Your brain is utilizing movement of itself through the world or the world around it to create a model.</p><p>Secondly he asks a deeper question, “How does the neocortex, which is composed of thousands of nearly identical cortical columns, learn a predictive model of the world through movement?” (36).</p><p>The answer it turns out, lies in another mystery.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I6OiAT2ZR2zK31moJUqmCA.jpeg" /><figcaption>From “A Thousand Brains” pg. 44</figcaption></figure><p>Scientists had long understood that neurons in the brain consisted of a cell body, axons, dendrites, and synapses. Synapses are the connections between neurons. There is a concentration of synapses around the cell body — if any of these synapses receive enough input, the neuron will then spike or activate. The mystery is that only 10% of a neuron’s synapses are in the proximal area. 90% are too far away to cause a spike. And nobody knew exactly what they did.</p><p>Scientists starting in 1990 began to discover that other than a neuron activation spike, there were also dendrite spikes. This occurred when a group of synapses, 20 or so, receive input at the same time. The signal travels along dendrites to the cell body, not enough to activate the neuron, but enough to raise the voltage.</p><p>Hawkins realized that these dendrite spikes had to be predictions.</p><p>Each dendrite spike means that the neuron recognized a particular pattern of activity in other neurons. It is now in a predictive state, primed to be fired like sprinters on a track.</p><p>So when you see your cup start to fall off the table, your neurons send dendrite spikes to neurons which expect to see it hit the ground. These primed neurons will be the only ones to fire if the cup hits the ground, but if it begins to float in the air, other neurons will fire in addition to the predictive ones. And now, that floating cup has given your brain an unexpected input it now pays attention to.</p><p>So there we have how the prediction happens, it is biologically built into each and every neuron.</p><p>But how does the neocortex predict the next input when we move?</p><p>Think about your car in the garage. As you walk up to open the side door, you will run your hand along its side. Your fingers need to know where the handle is. Even if you close your eyes, you will still be able to move your hand to the right location because you have a model of your car that stems from your touch and your sight. But in order to know where to place your hand, your brain needs to know where your hand is in relation to it and where the hand is in relation to the car. On top of that, your brain needs to know what the object you’re touching is in order to make the correct predictions.</p><p>Hawkin’s insight into this was that every cortical column must generate reference frames. It must have a model of where your hand is located on your body and it must generate a model of car enough to understand where your hand is in relation to the handle. Your brain needs a reference frame for your body in relation to itself and your body in relation to the object.</p><p>Hawkins realized that contrary to popular belief, most of the neocortex did not process sensory input. Rather, it generated reference frames and tracked locations.</p><p>Reference frames are crucial for the brain to create models of what it is trying to understand because:</p><ol><li>It allows the brain to understand the structure of something</li><li>It allows the brain to manipulate the whole object at once</li><li>It allows for planning and creation of movements.</li></ol><p>Now the question is, how does the brain create reference frames? The answer comes from a part of the brain we have had for a very long time.</p><p>An animal which can tell where it is has a significant evolutionary advantage over one which does not. Think about a worm that simply moves towards light. While it can certainly survive on this simple rule, an insect which knows where the light will be, where it is, and where to move to get to that light will have a significant advantage over a blind crawler.</p><p>Mammals developed two regions of the brain dedicated to these mapping mechanisms — the hippocampus and the entorhinal cortex.</p><p>John O’Keefe and his student Jonathon Dostrovsky in 1971 discovered that rats fire off a neuron in the hippocampus every time it was in a particular location in a particular environment. These cells were named “place cells” — they mark where you are on a map. Different place cells become active as you move to different locations. But the same one will activate for a single location.</p><p>In 2005, May-Britt Moser and Edvard Moser discover signals from neurons in the entorhinal cortex, next to the hippocampus. These cells were named grid cells. Grid cells fire at multiple locations in an environment and effectively create a grid of your environment. Instead of remembering specific places, they fire off in intervals as you move around a space creating coordinates like you would see on a map.</p><p>Your brain has stamped markers with place cells tied to locations and grid cells which tell you where that location is in reference to other grid cells. Your brain combines these two pieces of information to understand where it is and where it is in reference to other locations.</p><p>These grid cell place cell equivalents exist in the neocortex as well. Clever fMRI studies in 2017 confirmed their existence in the neocortex.</p><p>How does the neocortex store these connections to create a reference frame? It’s not too difficult really. One layer of neurons in a cortical column represents sensory input. This sensory input region is connected to a reference frame region of the cortical column. If you’re blindfolded and are dropped into your garage, when you open your eyes your sensory input identifies the car, shoes, roof, and a whole bunch of other sensory details which are immediately associated with a reference frame. They work together to identify immediately that you are in your garage.</p><p>One more necessary element to develop a complete reference frame is orientation. You need to know what direction a cups handle and what direction your body is facing in order to correctly move to the location needed to pick up the cup. There is an old brain equivalent to orientation cells known as head direction cells. It turns out cortical columns contain similar cells, more generally called orientation cells.</p><p>Orientation cells, place cells, and grid cells all work closely together to create a reference frame.</p><p>This reference frame allows you to create models of the world.</p><p>And the truly incredible thing is, model generation and reference frames is an intelligence blueprint that applies not just to physical objects in the world. It also applies to high level concepts like language and high level thinking.</p><p>Hawkins gives the example of a politician who wants to enact a law. The politician needs to know what will happen if he takes certain actions. They have a reference frame for where they currently are, what happens if they conduct a press conference or force a referendum, what reference frame that moves the law into, and has an understanding of what series of steps to take to successfully move through various reference frames in order to enact the law.</p><p>Language might seem like a task which can’t simply be broken down into reference frames, but there are two crucial aspects to language all linguists agree upon — nesting and recursive structure. Nesting is how sentences are made of phrases, which are made of words, which are made of letters. Recursion is the ability for sentences to consist of an unlimited array of phrases. So a sentence can contain a phrase which contains another phrase and so on. For example, “I went to the store” and “I, after graduating, went to the store” and “I, after graduating, looking over my shoulder, went to the store”. But these two aspects that seem unique to language are actually common in real world objects too. A cup can contain images both of which have separate representations in your mind. But the cup with the image on it is a recursive object.</p><p>Each cortical column can have a reference frame for an object which links to another reference frame. Sort of like how a wikipedia article can hyperlink to another wikipedia article. This feature is enough to satisfy the needs of reference frames to be able to generate nesting and recursion.</p><p>High level concepts are harder to learn because the reference frames are not obvious. History can be arranged with the reference frame of a timeline, or via a map, or via the people involved. Finding the appropriate reference frame for higher level concepts is the true challenge of learning them.</p><p>With this laid out, we can lay out three fundamental ideas of reference frames:</p><ol><li>Reference frames are present everywhere in the neocortex</li><li>Reference frames are used to model everything we know, not just physical objects</li><li>All knowledge is stored at locations relative to reference frames</li><li>Thinking is a form of movement</li></ol><p>With this laid out we have all the grounds to talk about the Thousand Brains Theory of Intelligence.</p><p>The theory rests on the foundation that everything we deem as knowledge is a mental model. These models are formed with the tool of prediction. These predictions are generated via reference frames and movement between reference frames.</p><p>Knowledge is stored via the connections of synapses between neurons. Your brain is a distributed knowledge system. Areas of touch, sight, and hearing can all identify sensory inputs of a car with the mental model of a car.</p><p>These mental models are not stored in a strictly hierarchical fashion. Hierarchy is likely still a component, but is not strictly speaking, necessary. Each cortical column has a complete representation of an object. But a single cortical column cannot learn a model of every object in the world. So multiple cortical columns learn models corresponding to various senses and reference frames of objects.</p><p>How do these models integrate with one another so that we recognize a single object and not millions of fractures of an object?</p><p>Well, they work together to vote on the object they are seeing. In the neocortex, cells in some layers send axons long distances. They can connect axons from one side of the brain to another e.g. connecting areas for the left and right hands. Only a few cells in each cortical column do the voting. Cortical columns send out signals of what it thinks it is seeing and receives signals from other columns saying what they think it is seeing. The most common signals win out until the whole neural network settles on a single answer.</p><p>This is also why our perception is so stable most of the time. Even if your hand is touching different parts of a cup and moving along its side, the voting signals remain the same and the neural network does not change. So your percept of the cup remains as a cup. This is also why seizure patients with the connections between left and right sides of their brains cut off act as if they have two brains. Each side has different thoughts and conclusions. This is because the voting mechanism is cut off between the two sides and each side reaches its own conflicting conclusion.</p><p>Unlike the hierarchical theory, there is no single object that is stored in a layer high up. Rather, there are a thousand models of an object distributed across cortical columns all voting to decide what object is being perceived.</p><p>This is where the Thousand Brains name comes from.</p><p>Hawkins absolutely brilliant theory opens the way for artificial general intelligence. Here finally is a theory of the brain, based on natural science, which fully describes its functionality in terms of a generalized algorithm present via the neocortex that is capable of learning anything it wants.</p><p>With reference frames and models, augmented by the tools of prediction and movement, we can learn anything we set our minds to. Including the brain itself.</p><p>Hawkins book is remarkable because for the longest time I subscribed to the idea that, if the brain were smart enough to understand how it worked, it would be too dumb to figure it out.</p><p>It turns out I was wrong.</p><p>And now in my brain, I have a reference frame for the brain itself.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d5161e8a6007" width="1" height="1" alt=""><hr><p><a href="https://medium.com/inside-out-through-the-looking-glass/the-thousand-brains-theory-of-intelligence-d5161e8a6007">The Thousand Brains Theory of Intelligence</a> was originally published in <a href="https://medium.com/inside-out-through-the-looking-glass">Inside Out: Through the Looking Glass</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5 Books of 2020]]></title>
            <link>https://medium.com/inside-out-through-the-looking-glass/5-books-of-2020-b9a60c3b6ffc?source=rss----78af52dc2fed---4</link>
            <guid isPermaLink="false">https://medium.com/p/b9a60c3b6ffc</guid>
            <category><![CDATA[2020]]></category>
            <category><![CDATA[new-year]]></category>
            <category><![CDATA[books]]></category>
            <category><![CDATA[reading]]></category>
            <dc:creator><![CDATA[RAM ESHWAR KAUNDINYA]]></dc:creator>
            <pubDate>Thu, 31 Dec 2020 02:56:18 GMT</pubDate>
            <atom:updated>2020-12-31T02:56:18.045Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G_YjmxCVYChnZ9oAvH2gsw.jpeg" /></figure><p>I love Bill Gates’ breakdowns of his 5 books of the year. And in the spirit of the end of the year and as tribute to some of the mind-changing books I’ve read this year, I wanted to share my own 5 books of 2020.</p><p>I have to disclaim that two of these I have not read end to end but the contents I’ve touched on had such a large impact on me this year that I could not leave them out — those two are “The China Study” and “Early Indians”.</p><h4>“The China Study” by T Colin Campbell PHD and Thomas M. Campbell MD</h4><p>This book centers around the largest biological study of nutrition of all time that occurred during the Nixon era and the opening of relations with China. It lays down the scientific evidence on the power of a whole food plant based diet. The big kicker for me was reading and seeing the evidence behind why dairy was “the most potent carcinogen known to mankind” (cancer causing agent). If nothing else, it will at least make you consider your food choices when choosing to eat an animal based product.</p><h4>“Utopia For Realists” by Rutger Bregman</h4><p>This book was an eye opener for me in laying out historical and social science ground for the problems of today as well as making a convincing case for the solutions of Open Borders, a Universal Basic Income, and a fifteen hour work week. While you may not agree with everything in this book, one of the central ideas addressed here is the discontent felt on multiple sides of the aisle, expressed here in what I’m paraphrasing Bregman as saying, “Given everything that they’re against, what are people actually for?”</p><h4>“Early Indians” by Tony Joseph</h4><p>Coming from a particular culture, background, and upbringing, there are often a lot of questions which are left unasked. A lot of things are accepted, unquestioned, and left unexamined. This leads to cultures of superiority, claims of exceptionalism, and all kinds of “us vs. them” mentalities. But it was extremely humbling and eye-opening to read Tony Joseph’s Early Indians and understand the newly unearthed definitive genetic evidence solidifying the history of early human migrations, in particular those of us who ended up in India. Tracing the four migrations ending in the last Aryan migration — now confirmed with genetic evidence — it put into a whole new light how I identify with that heritage. Summed up in his final conclusion of the book — “The best way we can define ourselves is as a multi-source civilization, not a single-source one, drawing its cultural impulses, its traditions and its practices from a variety of heredities and migration histories”.</p><h4>“The War on Normal People” by Andrew Yang</h4><p>This was a book written by Democratic Presidential Candidate Andrew Yang. While you may not lean on this side of the political spectrum at all, the ideas presented in this book go hand-in-hand with Bregman’s idea in his book Utopia For Realists of trying to lay out “What are you actually for?”. Yang makes sweeping social analysis of a lot of the key problems introduced in this new technological revolution termed by many experts as “The Sixth Industrial Revolution”. I found his analysis of the impacts of AI, GDP as an outdated sole measure of the health of a society, and his general points on changing the way in which we measure our progress and status as a society to be extremely enlightening.</p><h4>“Ten Lessons for a Post-Pandemic World” by Fareed Zakaria</h4><p>This book by Fareed Zakaria was my most recent finish and one which paints an extremely insightful picture of what the pandemic will mean for the world. Laying out key trends and breaking them down into 10 key takeaway lessons, Fareed is extremely good at laying out the key things we should focus on in the coming years. One of the big points that stuck with me from the book is the idea that the debate between good government and bad government is outdated and that we should rather be asking the question — what does GOOD government look like. As Jon Stewart said, the best government is the government that governs best. Another key insight he brings up is how we should look at the rise of China and outlines the fact that a key question needs to be asked from the US perspective — what is the role the US wishes China to see play in this liberal international order? Extremely provocative read that will guide you through navigating the trends we will see in this new world of 2021.</p><p>I hope you found some of these suggestions interesting, provocative, and insightful. If you find any of these to be interesting to you, I hope you pick the book up — or if these ideas lead you to a new place, I wish you well along that path as well. Wish you a very productive and insightful year ahead and a happy early New Year!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b9a60c3b6ffc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/inside-out-through-the-looking-glass/5-books-of-2020-b9a60c3b6ffc">5 Books of 2020</a> was originally published in <a href="https://medium.com/inside-out-through-the-looking-glass">Inside Out: Through the Looking Glass</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My Steps to Tackling a Challenge]]></title>
            <link>https://medium.com/inside-out-through-the-looking-glass/my-steps-to-tackling-a-challenge-3ed7df8af67c?source=rss----78af52dc2fed---4</link>
            <guid isPermaLink="false">https://medium.com/p/3ed7df8af67c</guid>
            <category><![CDATA[fulfillment]]></category>
            <category><![CDATA[meditation]]></category>
            <category><![CDATA[awareness]]></category>
            <category><![CDATA[happiness]]></category>
            <category><![CDATA[challenges-and-growth]]></category>
            <dc:creator><![CDATA[RAM ESHWAR KAUNDINYA]]></dc:creator>
            <pubDate>Mon, 23 Mar 2020 19:24:33 GMT</pubDate>
            <atom:updated>2020-03-23T19:24:33.766Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/468/0*oF5BFzFIIwg8cABQ.jpg" /><figcaption><a href="https://www.videvo.net/video/moon-over-sea-at-night/1675/">https://www.videvo.net/video/moon-over-sea-at-night/1675/</a></figcaption></figure><p>We have a tendency to always go from out to in. To think, the world has these issues and this is my reaction to them. That can be great but with this piece of writing I want to do the opposite. I want to go from IN to OUT.</p><p>When I think of a challenge that I’ve had in my life, I think of this. What is my <strong>reaction</strong>?</p><p>One of the first things to check is to ask myself, what beliefs show up for me when I think of this challenge? When I am placed in a situation where for example, I can choose whether to say something or keep quiet, I ask myself what beliefs am I affirming to myself by choosing what I do. Do I think that by keeping quiet I’m staying safe and guarding the small story of myself? If so, then that’s <strong>FEAR</strong>.</p><p>When I realize that fear is the factor behind some belief then I know that the thought and action that would come out of that belief would be false. It would not be the real me.</p><p>But when that realization presents itself, my reaction, once again <strong>REACTION</strong>, is not to overcome it. In other words, it is not to do something to change it or fight it or make it disappear. Fear is a light. Fear is a guide and fear is an opening. It is a manifestation that exists within us of something that can move us beyond where we currently see ourselves.</p><p>So, from this place of accepting, of loving, and of being, I then begin to ask myself is this fear truly me? Am I this fear? Or is this fear something which is just a belief that some egoic story of me has? Is this fear something that a <strong>STORY </strong>I have of myself believes is true. If I can begin to distance who I am from this belief I have, then I can move beyond it. Then I can ask myself, is this really true? And even more importantly, who would I be without this thought!</p><p>And this is when I enter a place of possibility. Of discovery and of creation. Here, is where the solution to whatever challenge I was presented with lies. The next step <strong>SHOWS ITSELF</strong> to you. It becomes <strong>AVAILABLE</strong> to you. You did not move anywhere to get this thing, you did not exert any effort to find a solution. No, the solution <strong>CAME</strong>. And it came because you expanded yourself and your story to be something which is now available to receive this answer.</p><p>If the sun wanted to attract the planets, it cannot do so if it stays small. It needs to gain gravity, it needs to expand in some way and transform itself.</p><p>With this awareness, with this movement of going from IN to OUT, we can then tackle any problem which comes our way.</p><p>One thing I learned from interviewing as a game developer was the steps for how to tackle a problem. I break it down to five key things:</p><p>1) What do I fundamentally know to be true?</p><p>2) What am I given in this situation?</p><p>3) What would I like to know in this situation?</p><p>4) How can I connect what I know and what I would like to know?</p><p>5) What perspective shift do I need to understand this situation?</p><p>And most important of all, when I’m tackling a challenge I know this. The results <strong>DO NOT MATTER</strong>. What happens as a result of the work you do does not matter in comparison to who you <strong>BECOME </strong>from this challenge. Who you become is beyond the situation. And the challenge is simply the situation.</p><p>And that awareness is what I want to present to you all today. So I ask you, who can you become, beyond your beliefs and stories, that is the you which is beyond the situation? Who can you become? Your time <strong>IS NOW</strong>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3ed7df8af67c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/inside-out-through-the-looking-glass/my-steps-to-tackling-a-challenge-3ed7df8af67c">My Steps to Tackling a Challenge</a> was originally published in <a href="https://medium.com/inside-out-through-the-looking-glass">Inside Out: Through the Looking Glass</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Recursion — The Basis of Language and ML]]></title>
            <link>https://medium.com/inside-out-through-the-looking-glass/recursion-the-basis-of-language-and-ml-c008c98cb388?source=rss----78af52dc2fed---4</link>
            <guid isPermaLink="false">https://medium.com/p/c008c98cb388</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[recursion]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[language-faculty]]></category>
            <category><![CDATA[backpropagation]]></category>
            <dc:creator><![CDATA[RAM ESHWAR KAUNDINYA]]></dc:creator>
            <pubDate>Wed, 27 Jun 2018 00:28:17 GMT</pubDate>
            <atom:updated>2018-06-27T00:28:17.501Z</atom:updated>
            <content:encoded><![CDATA[<p>Multi-layer perceptron neural networks are the classic example of a machine learning network. While the standards today have moved to Convolutional Neural Networks (CNN) and Long Short-Term Memory (LTSM) Networks, the principles of multi-layer perceptron neural networks determine the ideas and principles of machine learning. The workhorse behind these neural networks is something known as backpropagation. A very simple way of describing a neural network is that it is an input output system. Shrek said it well when he said, “Ogres are like onions… Onions have layers, ogres have layers”. Neural networks are like onions. You have a simple input layer into which you feed in the data, then multiple inner layers which process and analyze the data, and finally an output layer which gives you the classification or regression which you desire. Now if the output doesn’t match what you expect or want it to be, then the network needs to update itself to be both more accurate and confident in its accuracy. This is where backpropagation comes in.</p><p>Let us analyze a simple multi-layer perceptron network in which there is only one neuron per layer. This allows us to take a look at the math on a simplified level.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FN28ZA6lnje_zrAlo2JBXw.png" /><figcaption>Figure 1</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3bZExuaZn_CO-FsT6pSeyQ.png" /><figcaption>Figure 2</figcaption></figure><p>The whole point of backpropagation — and by extension machine learning at large — is to optimize the cost function via finding a minimum of the function. Finding a global minimum is hard and there is no guarantee that a multi-perceptron layer neural network will find a global minimum, but it can find a local minimum via gradients and backpropagation. Essentially you find the direction that you want the algorithm to move in order to achieve the greatest reduction of Cost to the function and step in that direction. The step is achieved by adjusting weights and biases accordingly. The degree of that a change in weights or biases causes is determined via calculations in the figures above. Here is a brief translation of each of the variables mentioned above: <br><strong>a(L)</strong> is the activation level of a neuron on layer L<br><strong>w(L)</strong> is the weight on a specific neuron on layer L<br><strong>b(L)</strong> is the bias of a neuron on layer L<br><strong>C0</strong> is the cost function, defined as the difference between the outputted value and the expected value<br><strong>σ(z(L))</strong> is the sigmoid function applied to z(L)<br><strong>z(L)</strong> is the addition of biases and weights as applied to a perceptron; what I will refer to hereon out as **perceptron value**; this is defined mathematically as <strong>w(L)a(L-1) + b(L)</strong><br><strong>σ</strong> — for reference the sigmoid function is defined as <strong>1/(1+e^-z)</strong>; the graphed function looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*ybM88m5Qbh0G64rhLCtlfA.png" /></figure><p>I find it significant to include the graph here because I want to highlight that a small change in z causes a small change in the overall firing of the neuron — the whole reason why a sigmoid function is at play here in the first place!</p><p>** Note: I use (L) as a parameter above but it is notated mathematically as a superscript, medium simply does not allow superscript formatting **</p><p>As you can see from Figure 1, z(L) determines a(L) which determines C0. We want to find what a change in w(L) will do to the Cost C0 in order to continue optimizing the cost function in the right direction. So we need to relate three equations which determine C0 since w(L) is only in the z(L) equation directly. Through the z(L), a(L), C0 relationship however, we see that w(L) ties to C0. So here we apply the chain rule:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6_pXuKyuCYEAafVATw4YxA.png" /><figcaption>Figure 3</figcaption></figure><p>This is essentially the equation behind the powerhouse backpropagation. Let’s break down the components a little into words. How much a change in weight w(L) affects a change in perceptron value z(L) depends solely on the activation of the previous neuron. How much change in perceptron value z(L) affects activation turns out to be σ’z(L). The derivative of the sigmoid when calculated turns out to be negative and small with a minimum of approximately -1/4. This means that this term has relatively little overall numerical effect on the change in Cost C0. Also we see that since a change in z(L) is affected directly by a change in activation of a neuron in the previous layer, this term is really also dependent primarily on the activation of the previous neuron. Finally, the change in activation of the current neuron with respect to the cost function changes significantly with a small change in activation of the current neuron. This means that the closer a neuron is to an output layer, the more a change in activation level to this affects the overall output.</p><p>These calculations are repeated across all training examples, summed and averaged, and inputted into the gradient vector (way to find the largest direction of decrease) of the network. <br>When this backpropagation equation is applied to a multiple-neuron per layer network, the calculations change only in that now you have to keep add up the effects of multiple weights and neurons to calculate z and you have to sum across all cost values in a layer to calculate the full cost. Basically, you have to add up a few more indices, but the equations and processes rely on the same principles.</p><blockquote>Key Takeaways: <br>- activation of the current neuron plays the biggest role in calculation of ∆w’s affect on the Cost C0<br>- change in w(L)’s effect on a perceptron’s activation depends primarily on the activation of the previous neuron<br>- change in b(L)’s effect on Cost C0 depends primarily on activation of the previous neuron</blockquote><p>Incidentally, the change in an activation of a previous neuron’s effect on the Cost C0 is also dependent on the change in weight w(L), however since activation cannot be directly changed, we are more interested in the effect of a change in w(L) to C0. The change in activation of a previous neuron’s effect on C0 does however give us a useful measure to see how much neurons from previous layers are effecting neurons in the current layer.</p><p>You can see from takeaways one and two that the value with the biggest role in the change of cost function for a change in weight is the activation of a neuron on the current layer. From takeaways two and three you see that the activation of the previous neuron plays the greatest role in the effects of changes in weights and biases on the next perceptron’s activation. This means that to appropriately fine tune a system to its optimal performance, the system must be able to recursively go through each neuron in the network and apply the same backpropagation optimization method to all neurons. The system must have the capability to be recursive. This is phenomenally important and brings to light an interesting parallel between neural networks and Chomsky’s work with language. Chomsky’s current theories with regards to the language ability of humans narrows down the key of human language generation to the ability for recursion (see <a href="http://science.sciencemag.org/content/298/5598/1569.full">“The Faculty of Language: What Is It, Who Has It, and How Did It Evolve?”</a>). It is interesting that both neural networks, a computational way to represent natural learning processes, and an innate built in faculty for language in humans have the same core concept of recursion in their design. As the key to neural networks is communication, it seems that this corroborates the theory that the key to communication is the ability to think recursively.</p><p>Sources:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fvideoseries%3Flist%3DPLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DtIeHLnjs5U8&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FtIeHLnjs5U8%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/185207354ecc5cbb8c696f9905f006b6/href">https://medium.com/media/185207354ecc5cbb8c696f9905f006b6/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c008c98cb388" width="1" height="1" alt=""><hr><p><a href="https://medium.com/inside-out-through-the-looking-glass/recursion-the-basis-of-language-and-ml-c008c98cb388">Recursion — The Basis of Language and ML</a> was originally published in <a href="https://medium.com/inside-out-through-the-looking-glass">Inside Out: Through the Looking Glass</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Principles]]></title>
            <link>https://medium.com/inside-out-through-the-looking-glass/the-principles-1afa589c37d8?source=rss----78af52dc2fed---4</link>
            <guid isPermaLink="false">https://medium.com/p/1afa589c37d8</guid>
            <category><![CDATA[art]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[blog]]></category>
            <category><![CDATA[music]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[RAM ESHWAR KAUNDINYA]]></dc:creator>
            <pubDate>Fri, 15 Jun 2018 03:10:39 GMT</pubDate>
            <atom:updated>2021-09-27T04:06:25.670Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>I will look into my eyes knowing this — dedication, awareness, and sincerity will be my pillars. Around this I will consistently develop the self-confidence to know that my system is structured around me. In this I will find a centered fluidity.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*JAJi2IN7RXwqUz59qsuZOw.gif" /><figcaption><a href="https://en.wikipedia.org/wiki/Fractal#/media/File:Fractal_tree.gif">https://en.wikipedia.org/wiki/Fractal#/media/File:Fractal_tree.gif</a></figcaption></figure><p><strong>The reasons for creating “Inside Out”:</strong></p><ol><li>I believe that the process is much more valuable than the product</li><li>I believe that in articulating the process and thinking behind it, I can better frame my own thinking behind my study thereby bringing myself up while I give something to others which they can use to inform their own thinking</li><li>I believe that in sharing the process and dedicating myself to its refinement I can bring about a state of constant innovation focused on the <strong>process of creation </strong>over the creation</li></ol><p>All in All, it solidifies the reality that I am not the PRODUCT, but everything in between</p><blockquote>~ This is dedicated to the process</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1afa589c37d8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/inside-out-through-the-looking-glass/the-principles-1afa589c37d8">The Principles</a> was originally published in <a href="https://medium.com/inside-out-through-the-looking-glass">Inside Out: Through the Looking Glass</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>