<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Karan Praharaj on Medium]]></title>
        <description><![CDATA[Stories by Karan Praharaj on Medium]]></description>
        <link>https://medium.com/@karanpraharaj?source=rss-2eee9862711a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*VFjWd8dYNeOEmNQVM5gCBg.jpeg</url>
            <title>Stories by Karan Praharaj on Medium</title>
            <link>https://medium.com/@karanpraharaj?source=rss-2eee9862711a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 14:14:57 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@karanpraharaj/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How Are Algorithms Biased?]]></title>
            <link>https://medium.com/data-science/how-are-algorithms-biased-8449406aaa83?source=rss-2eee9862711a------2</link>
            <guid isPermaLink="false">https://medium.com/p/8449406aaa83</guid>
            <category><![CDATA[ai-bias]]></category>
            <category><![CDATA[fairness]]></category>
            <category><![CDATA[racism]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Karan Praharaj]]></dc:creator>
            <pubDate>Sat, 27 Jun 2020 10:10:20 GMT</pubDate>
            <atom:updated>2020-06-29T08:01:24.181Z</atom:updated>
            <content:encoded><![CDATA[<h3>Bias In AI Algorithms</h3><h4>Algorithms do what they’re taught. Unfortunately, some are inadvertently taught prejudices and unethical biases by societal patterns hidden in the data.</h4><p>After the end of the Second World War, the Nuremberg trials laid bare the atrocities conducted in medical research by the Nazis. In the aftermath of the trials, the medical sciences established a set of rules — The Nuremberg Code — to control future experiments involving human subjects. The Nuremberg Code has influenced medical codes of ethics around the world, as has the exposure of experiments that had failed to follow it even three decades later, such as the infamous <a href="https://en.wikipedia.org/wiki/Tuskegee_syphilis_experiment">Tuskegee syphilis experiment</a>.</p><p>The direct negative impact of AI experiments and applications on users isn’t quite as inhumane as that of the Tuskegee and Nazi experimentations, but in the face of an overwhelming and growing body of evidence of algorithms being biased against certain demographic cohorts, it is important that a dialogue takes place sooner or later. AI systems can be biased based on who builds them, the way they are developed, and how they’re eventually deployed. This is known as algorithmic bias.</p><p>While the data sciences have not developed a Nuremberg Code of their own yet, the social implications of research in artificial intelligence are starting to be addressed in some curricula. But even as the debates are starting to sprout up, what is still lacking is a discipline-wide discussion to grapple with questions of how to tackle societal and historical inequities that are reinforced by AI algorithms.</p><p>We are flawed creatures. Every single decision we make involves a certain kind of bias. However, algorithms haven’t proven to be much better. Ideally, we would want our algorithms to make better-informed decisions devoid of biases so as to ensure better social justice, i.e., equal opportunities for individuals and groups (such as minorities) within society to access resources, have their voices heard, and be represented in society.</p><p>When these algorithms do the job of amplifying racial, social and gender inequality, instead of alleviating it; it becomes necessary to take stock of the ethical ramifications and potential malevolence of the technology.</p><p>This essay was motivated by two flashpoints : the racial inequality discussion that is now raging on worldwide, and Yann LeCun’s altercation with Timnit Gebru on Twitter which was caused due to a disagreement over a downsampled image of Barack Obama (left) that was depixelated to a picture of a white man (right) by a face upsampling machine learning (ML) model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OF-l_lIT53IhUOEKniyFQg.png" /><figcaption>A depixelated picture of Obama upsampled to a picture of a caucasian male. <em>(Courtesy : @hardmaru on Twitter)</em></figcaption></figure><p>The (rather explosive) argument was sparked by this tweet by LeCun where he says that the resulting face was that of a white man because of a bias in data that trained the algorithm. Gebru responded sharply that the harms of ML systems cannot be reduced to biased data.</p><h3></h3><p>ML systems are biased when data is biased.This face upsampling system makes everyone look white because the network was pretrained on FlickFaceHQ, which mainly contains white people pics.Train the *exact* same system on a dataset from Senegal, and everyone will look African. https://t.co/jKbPyWYu4N</p><p>In most baseline ML algorithms, the model fits better to the attributes that that occur most frequently across various data points. For example, if you were to design an AI recruiting tool to review the résumés of applicants for a software engineering position, you would first need to train it with a dataset of past candidates which contains details like “experience”, “qualifications”, “degree(s) held”, “past projects” etc. For every datapoint, the algorithm of the hiring tool would need a decision or a “label”, so as to “learn” how to make a decision for a given applicant by observing patterns in their résumé.</p><p>For an industry where the gender disparity in representation is large, it is reasonable to assume that a large majority of the data points will be male applicants. And this collective imbalance in the data ends up being interpreted by the algorithm as a useful pattern in the data rather than undesirable noise which is to be ignored. Consequently, it teaches itself that male candidates are more preferable than female candidates.</p><p>I wish that this was merely an imaginary, exaggerated example that I used to prove my point. <a href="https://in.reuters.com/article/amazon-com-jobs-automation/insight-amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idINKCN1MK0AH">It is not.</a></p><p>LeCun wasn’t wrong in his assessment because in the case of that specific model, training the model on a dataset that contains faces of black people (as opposed to one that contains mainly white faces) would not have given rise to an output as absurd as that. But the upside of the godfather of modern AI getting dragged into a spat (albeit unfairly) has meant that more researchers will now be aware of the implications of their research.</p><p>The misunderstanding clearly seems to emanate from the interpretation of the word “bias” — which in any discussion about the social impact of ML/AI seems to get crushed under the burden of its own weight.</p><p>As Sebastian Raschka puts it, “the term <strong>bias</strong> in ML is heavily overloaded”. It has multiple senses that can all be mistaken for each other.</p><p>(1) <strong>bias</strong> (as in mathematical <strong>bias</strong> unit) ​ (2) “Fairness” <strong>bias</strong> (also called societal <strong>bias</strong>) ​ (3) ML <strong>bias</strong> (also known as inductive <strong>bias</strong>, which is dependent on decisions taken to build the model.) ​ (4) <strong>bias</strong>-variance decomposition of a loss function ​ (5) Dataset <strong>bias</strong> (usually causing 2)</p><p>I imagine that a lot of gaps in communication could be covered by just being a little more precise when we use these terms.</p><p>On a lighter note, never mind Obama, the model even depixelized a <strong>dog’s face</strong> to a caucasian man’s. It sure loves the white man.</p><h3></h3><p></p><p>Learning algorithms have inductive biases going beyond the biases in data too, sure. But if the data has a little bias, it is amplified by these systems, thereby causing high biases to be learnt by the model. Simply put, creating a 100% non-biased dataset is practically impossible. Any dataset picked by humans is cherry-picked and non-exhaustive. Our social cognitive biases result in inadvertent cherry-picking of data. This biased data, when fed to a data-variant model (a model whose decisions are heavily influenced by the data it sees) encodes these societal, racial, gender, cultural and political biases and bakes them into the ML model.</p><p>These problems are exacerbated, once they are applied to products. A couple of years ago, Jacky Alciné <a href="https://twitter.com/jackyalcine/status/615329515909156865">pointed out</a> that the image recognition algorithms in <a href="https://www.wired.com/story/when-it-comes-to-gorillas-google-photos-remains-blind/">Google Photos were classifying his black friends as “gorillas.”</a> Google apologised for the blunder and assured to resolve the issue. However, instead of coming up with a proper solution, it simply blocked the algorithm from identifying gorillas at all.</p><p>It might seem surprising that a company of Google’s size was unable to come up with a solution to this. But this only goes to show that training an algorithm that is consistent and fair isn’t an easy proposition, not least when it is not trained and tested on a diverse set of categories that represent various demographic cohorts of the population proportionately.</p><h3></h3><p></p><p><a href="https://www.npr.org/2020/06/24/882683463/the-computer-got-it-wrong-how-facial-recognition-led-to-a-false-arrest-in-michig">Another disastrous episode</a> of facial recognition tech getting it terribly wrong came as recently as last week when a faulty facial recognition match led to a Michigan man’s arrest for a crime he did not commit. Recent studies by <a href="https://www.nytimes.com/2018/02/09/technology/facial-recognition-race-artificial-intelligence.html">M.I.T.</a> and the <a href="https://www.nytimes.com/2019/12/19/technology/facial-recognition-bias.html">National Institute of Standards and Technology</a>, or NIST, found that even though face recognition works well on white men, the results are not good enough for other demographics (the misidentification ratio can be more than 10 times worse), in part because of a lack of diversity in the images used to develop the underlying databases.</p><p>Problems of algorithmic bias are not limited to image/video tasks and they manifest themselves in language tasks too.</p><p><a href="https://web.stanford.edu/~mjkay/LifeOfLanguage.pdf">Language is always “situated”</a>, i.e., it depends on external references for its understanding and the receiver(s) must be in a position to resolve these references. This therefore means that the text used to train models carries latent information about the author and the situation, albeit to varying degrees.</p><p>Due to the situatedness of language, any language data set inevitably carries with it a demographic bias. For example, some speech to text transcription models tend to have higher error rates for African Americans, Arabs and South Asians as compared to Americans and Europeans. This is because the corpus that the speech recognition models are trained are dominated by utterances of people from western countries. This causes the system to be good at interpreting European and American accents but subpar at transcribing speech from other parts of the world.</p><p>Another example in this space is the gender biases in existing word embeddings (which are learned through a neural networks) that show females having a higher association with “less-cerebral” occupations while males tend to be associated with purportedly “more-cerebral” or higher paying occupations.</p><h3></h3><p></p><p>In the table below, we see the gender bias scores associated with various occupations in the <a href="https://www.tensorflow.org/hub/modules/google/universal-sentence-encoder/1">Universal Sentence Encoder</a> embedding model. The occupations with positive scores are female-biased occupations and ones with negative scores are male-biased occupations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/970/1*ek5IVC12A2X3-iA5KzI0Ww.png" /><figcaption><em>Occupations with the highest female-biased scores (left) and the highest male-biased scores (right) (Courtesy : </em><a href="https://developers.googleblog.com/2018/04/text-embedding-models-contain-bias.html">https://developers.googleblog.com/2018/04/text-embedding-models-contain-bias.html</a>)</figcaption></figure><p>While it is easy for ML Researchers to hold their hands up and absolve themselves of all responsibility, it is imperative for them to acknowledge that they — knowingly or otherwise — build the base layer of AI products for a lot of companies that are devoid of AI expertise. These companies, without the knowledge of fine-tuning and tweaking models, use pre-trained models, as they are, put out on the internet by ML researchers (like GloVe, BERT, ResNet, YOLO etc).</p><p>Deploying these models without explicitly recalibrating them to account for demographic differences is perilous and can lead to issues of exclusion and overgeneralisation of people along the way. The buck stops with the researchers who must own up responsibility for the other side of the coin.</p><p>It is also easy to blame the data and not the algorithm. (It reminds me of the Republican stance on the second amendment debate: “Guns don’t kill people, people kill people.”) Pinning the blame on just the data is irresponsible and akin to saying that the racist child isn’t racist because he was taught the racism by his racist father.</p><p>More than we need to improve the data, it is the algorithms that need to be made more robust, less sensitive and less prone to being biased by the data. This needs to be a responsibility for anyone who does research. In the meantime, de-bias the data.</p><p>The guiding question for deployment of algorithms in the real world should always be “would a false answer be worse than no answer?”</p><p><strong><em>You can visit my page </em></strong><a href="https://karanpraharaj.github.io/"><strong><em>here</em></strong></a><strong><em>. My Twitter handle is @</em></strong><a href="https://twitter.com/IntrepidIndian"><strong><em>IntrepidIndian</em></strong></a><strong><em>.</em></strong></p><h3>References</h3><p>1) <a href="https://medium.com/%5B">Facial Recognition Is Accurate, if You’re a White Guy</a><a href="https://www.nytimes.com/2018/02/09/technology/facial-recognition-race-artificial-intelligence.html">https://www.nytimes.com/2018/02/09/technology/facial-recognition-race-artificial-intelligence.html</a>) by Steve Lohr</p><p>2) Krishnapriya, KS., Vangara, K., King, M., Albiero, V., Bowyer, K. <a href="https://arxiv.org/pdf/1904.07325.pdf">Characterizing the Variability in Face Recognition Accuracy Relative to Race</a> in <em>The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2019.</em></p><p>3) <a href="https://web.stanford.edu/~mjkay/LifeOfLanguage.pdf">Life of Language</a> by Martin Kay, Stanford University</p><p>4) <a href="https://developers.googleblog.com/2018/04/text-embedding-models-contain-bias.html">Text Embedding Models Contain Bias. Here’s Why That Matters.</a> by Ben Packer, Yoni Halpern, Mario Guajardo-Céspedes &amp; Margaret Mitchell, Google AI</p><p>5) Bolukbasi, T., Chang, KW., Zou, J., Saligrama, V., Kalai, A. <a href="http://papers.nips.cc/paper/6228-man-is-to-computer-programmer-as-woman-is-to-homemaker-d">Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings</a> in <em>Advances in Neural Information Processing Systems 29, 2016.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8449406aaa83" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-are-algorithms-biased-8449406aaa83">How Are Algorithms Biased?</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[You Are Building Self-Driving Cars. For Free.]]></title>
            <link>https://medium.com/swlh/you-are-building-self-driving-cars-for-free-8d2f1a65d59e?source=rss-2eee9862711a------2</link>
            <guid isPermaLink="false">https://medium.com/p/8d2f1a65d59e</guid>
            <category><![CDATA[autonomous-cars]]></category>
            <category><![CDATA[driverless-cars]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[self-driving-cars]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Karan Praharaj]]></dc:creator>
            <pubDate>Sat, 27 Jun 2020 02:03:51 GMT</pubDate>
            <atom:updated>2020-06-27T02:03:51.561Z</atom:updated>
            <content:encoded><![CDATA[<p>Everytime you solve a reCaptcha image, like the one below, you are unwittingly cleaning the data which trains machine-learning algorithms deployed in self-driving cars.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/271/1*8dfXNA-J--ftjZXpSQ5iDw.png" /></figure><p>When you solve a captcha image, you’re now doing more than just proving that you’re not a robot. You are actively teaching a (future) self-driving car to see stuff and identify it.</p><p>This is similar to teaching a baby to identify stuff by showing them the same things over and over again until they learn from their mistakes and start getting their identifications right. This feedback loop of learning-testing-evaluating-relearning is at the heart of this.</p><p>To be able to self-drive, your AI needs to learn to process, assess and then act on its surroundings. One way to “learn” the “surroundings” is to see a shitload of images of objects that it can potentially encounter in an on-road environment, and learn to classify them correctly.</p><p>Self-driving car companies develop machine learning algorithms that automatically classify these images (one eg. is classifying whether a zebra crossing is present in the field of view or not).</p><p>These classifier algos are then fed or “trained” with tens of millions of pictures, until their accuracy crosses a certain minimum threshold. However, merely getting more predictions right than wrong doesn’t cut it for the self-driving problem. There are literal lives at stake.</p><p>Even 80% isn’t good enough. (Imagine misidentifying stuff 20% of the time while driving on road.) The pictures have to be sufficiently exhaustive of possible scenarios AND need the classifier to barely make any mistakes on them.</p><p>And so because it is crucial to stamp out almost ALL wrong predictions made by these algos, they need to be verified. This is where you — as someone who is much more reliable at identifying stuff than the algo is — come in to the picture. These soft classified images are presented to you in the form of captcha images of streets, traffic lights, pedestrian crossings, animals, footpaths, other vehicles etc.</p><p>When your captcha-selection agrees with the classifier’s prediction (“annotation”), the confidence in that prediction increases. When there is a mismatch, the confidence drops. If the confidence drops beyond a certain threshold, the prediction is replaced with the “correct” label.</p><p>This task of annotating and labeling billions of images that capture an infinite number of road-situations/-surroundings, is virtually impossible for any company with a limited workforce. So somebody very clever came along and devised this method to trick us all into working for them without paying us a single penny. Now all your successful captchas are used to fine-tune the data that trains the brains of these cars.</p><p>Next time you see an autonomous vehicle, you know that you made it what it is today.</p><p><em>This is a very simplistic explanation of self-driving originally posted as a </em><a href="https://twitter.com/IntrepidIndian/status/1268613825772716032?ref_src=twsrc%5Etfw%7Ctwcamp%5Etweetembed%7Ctwterm%5E1268613825772716032&amp;ref_url=https%3A%2F%2Fkaranpraharaj.github.io%2Fpost%2Fselfdriving-captcha%2F"><em>twitter-thread</em></a><em> on my account. The post doesn’t venture into how the actuations interact with the decision-systems or even how the decisions themselves are arrived at. Self-driving is a much, much harder problem than this short writeup might lead you to believe.</em></p><p><strong><em>You can visit my page </em></strong><a href="https://karanpraharaj.github.io/"><strong><em>here</em></strong></a><strong><em>. You can follow my twitter feed </em></strong><a href="https://twitter.com/IntrepidIndian"><strong><em>here</em></strong></a><strong><em>.</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8d2f1a65d59e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/you-are-building-self-driving-cars-for-free-8d2f1a65d59e">You Are Building Self-Driving Cars. For Free.</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Identification Of Lexico-Semantic Word Relations — A Beginner’s Guide]]></title>
            <link>https://medium.com/data-science/identification-of-lexico-semantic-word-relations-a-beginners-guide-427ed331e287?source=rss-2eee9862711a------2</link>
            <guid isPermaLink="false">https://medium.com/p/427ed331e287</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computational-linguistics]]></category>
            <category><![CDATA[word2vec]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Karan Praharaj]]></dc:creator>
            <pubDate>Sat, 27 Jun 2020 01:42:49 GMT</pubDate>
            <atom:updated>2020-06-28T13:39:14.897Z</atom:updated>
            <content:encoded><![CDATA[<h3>Identifying Lexico-Semantic Word Relations — A Beginner’s Guide</h3><h4>A dummy’s introduction to my current work — recognizing the nature of the semantic relation between a given pair of words.</h4><p>When I ask you to “please bear with me”, you prepare yourself with the prospect of having to put up with me. If you are told “this teddy bear is fluffy”, your brain conjures up the image of a soft, lovable, furry toy that toddlers take to bed. If you read about bears being endangered, you think of a polar bear somewhere in the arctic sea ice. And when they will tell you on the news next week that the coronavirus crisis has plunged us into the worst bear market of our generation, you will instinctively know that they are speaking about stocks.</p><p>Your knowledge of the English language, along with your ability to understand context-dependencies, lexical and syntactic structures, and linguistic nuances helps you differentiate between four senses of the same word. You take it for granted and barely think about it but it has taken you years to acquire this ability. You learn, directly or indirectly, from your personal experiences. You learn through making associations between contexts, information, behaviours, and responses. The cascade of neurocognitive reactions that are set off as you subconsciously trigger a set of neutrons to communicate or listen is nothing short of a wonder. All of this together with your genetic endowment makes language effortless for you.</p><p>On the other hand, understanding human language is a difficult problem for computers. Unlike you and me, computers do not have the privilege of language training the way we do. Even programming languages aren’t directly interpreted by them — they are first converted to low-level machine language. True <em>machine code</em> is merely a stream of raw, usually binary (1s and 0s), data. While humans acquire the ability to parse, process, infer and communicate, for the computer, any word picked out from a human language is unintelligible gibberish until it is adequately trained to understand the language.</p><p>This task of teaching and empowering machines to understand language just as we do, is called Natural Language Processing or NLP. NLP is a branch of artificial intelligence and it is an umbrella itself for many other subproblems. Daily examples of such problems are search, speech recognition, translation, summarization, question-answering etc. But all of this begs the question — if computers can understand nothing but 1s and 0s, how can they make sense of the complexities of human language?</p><h3>Word Vectors — Representing words in the form of numbers</h3><p>Consider a space where all words in the English language are populated based on their semantic character. This imaginary space is such that words sharing similar descriptions or concepts share similar spacial properties. For instance, the words “cat” and “dog” would be in close vicinity of each other because the idea of a cat is very similar to the idea of a dog. Both are quadrupedal, domestic species that make for cute pets. For words that are not similar in meaning but represent the same concept, the positions of the words relative to each other encapsulate the relationship. In the semantic space, the relative position of “king” to the position of “queen” would be similar to the relative positions between “man” and “woman” or “boy” and “girl”, because the defining concept that separates the words in all three cases is the same — gender.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YEqMgMakhNXIYgeX.png" /></figure><p>In the example semantic space below, you can see how the vectors for animals like lion, tiger, cheetah, and elephant are very close together. This is intuitive because they are often discussed in similar contexts; for example, these animals are big, wild and, potentially dangerous — indeed, the descriptive word “wild” maps quite closely to this group of animals.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/778/0*vsEpl6NOYGsgGMUa.png" /></figure><p>Since words in their purest form cannot be interpreted by computers, we dumb them down by mapping the concepts and ideas that are inherent to the words into a representative set of numbers for each word. These sets of numbers are generated or “learned” algebraically by “neural networks” (a type of algorithm) and are called “word vectors”. These word vectors bear the ability to capture information about semantic relationships and syntactic structures across collections of words. Approaches to generating word vectors build on Firth’s (1957) <em>distributional hypothesis</em> which states:</p><blockquote><em>“You shall know a word by the company it keeps.”</em></blockquote><p>Put differently, <strong>words that share similar contexts tend to have similar meanings</strong>.</p><p>Word vectors can have any number of dimensions, although the standard number is usually 50, 100 or 300. Each of these dimensions represents a meaning or an abstract concept, the degree of which depends upon the numeric weight of the word on that particular dimension. Here is an example to illustrate this. Consider a lexicon of just ten words (rather than millions) and imagine that our word vectors are three-dimensioned (rather than three hundred).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/973/0*_860pMq1L8Mdf9xj.png" /></figure><p>In the figure above, for better understanding, we are imagining that each dimension captures a clearly defined meaning as opposed to an abstract idea. For example, if you imagine that the third dimension represents the concept of “fluffiness”, then each word’s weight on that dimension represents how closely it relates to that concept. It makes perfect sense for the rabbit to have the highest fluffiness factor at 0.45. This is quite a large simplification of word vectors as the dimensions do not hold such clearly defined meanings in reality, but it is a useful and intuitive way to wrap your head around the concept of word vector dimensions. We will not delve into the mathematical details of how neural networks learn word embeddings, because that would involve a long detour into linear algebra. But now you do know the underlying idea that drives the mathematics.</p><h3>Lexical Relation Resolution</h3><p>My current research work is focused on a problem called lexical relation resolution. A lexical relation is a culturally recognized pattern of association that exists between lexical items (a word, a part of a word, or a chain of words) in a language. For example, the lexical relation between “open” and “close” is that of antonymy, whereas “close” and “shut” are connected by a synonymy relationship. Other asymmetric lexico-semantic relations include co-hyponymy (e.g. phone ←→ monitor), hypernymy (e.g. phone → speakerphone) or meronymy (e.g. phone → mouthpiece), etc</p><p>Recognizing the exact nature of the semantic relation holding between a given pair of words is crucial and forms the basis for all the other NLP applications (question-answering, summarization, speech recognition etc.) that I mentioned above.</p><p>Several methods have been proposed in the past to discriminate between multiple semantic relations that hold between a pair of words. But, this continues to remain a difficult task, especially when it comes to distinguishing between certain relations. (e.g synonymy and hyperonymy).</p><h3>Research Work — Patches, Attention, Cuboid</h3><p>To solve this problem, our work proposes to investigate the introduction of related words in the neighbourhood of a particular word and gauge the effect it has on the prediction accuracy of word relations. Our original hypothesis was that if each word is augmented by the word vectors of a fixed number of neighbouring words (or “patches”), improved performance might be attained.</p><p>Many similarity measures exist to account for the lexical semantic relation that links two words. In our case, we use the cosine similarity measure which has proven to be successful in the past for a variety of semantic relations. To put it in plain speak, cosine similarity is a metric used to determine how similar two entities are. We extend the cosine similarity to patches in a straightforward manner. The similarity between two patches is the set of one-to-one cosine similarity measures between all words in their respective patches.</p><p>The next step consists of transforming a patch into a learning input, because although we can draw and visualise patches in our head, the computer needs it in the form of concrete, numeric data to understand it. The 300-dimensional word vectors of each word are compared to the 300-dimensional word vectors of all the words in the patch and a single similarity score for each comparison. In effect, we take 600 values (300 from each of the two words) and simplify them into one with the help of cosine similarity. If we set the no. of neighbours to be 10, that would give us 10 x 10 comparisons, resulting in 100 similarity values. These values together form the “intra-patch similarity table”. Given below are examples of four different patches along with their intra-patch scores.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/648/0*TbYmBQ_VKLj5jMw2.png" /></figure><p>It was decided that in addition to preserving concept-centrality <em>within</em> patches, it also makes sense to preserve relation-centrality <em>between</em> patches. It is important to acknowledge that only certain words in the two patches may be central to the decision of whether two words are in a lexical semantic relation. If two patches share a set of close semantically related words that are central to both concepts, the decision process should intuitively be more reliable.</p><p>However, our initial findings showed that the direct introduction of neighbour words did not lead to improvements. We figured that this was mainly due to a loss of concept-centrality that took place as a result of the change in strategy. If word relations are to be assessed by juxtapositioning two patches instead of two words, the required focus on the original word may be diluted.</p><p>The next logical step was to somehow weigh the word vectors based on their centrality to the concept. To do this, we introduced an <strong>attention mechanism</strong> based on the PageRank algorithm (which is one of the algorithms used by Google for their web search. PageRank was developed to measure the importance of website pages). We use it to assign a weight of centrality to each of the word neighbours in the patch. The more a word is central in the patch, the higher the score it receives. These scores are then to be used as attention weights to the corresponding word vector representations of the neighbours. The objective of this mechanism is to improve the predictive ability of our system based on the importance score of each word vector in the patch. We found that when deployed in combination with the correct architecture, attention-adjusted patches bolstered our model and gave a significant boost to previous results. I would have to tread into extremely technical territory to explain the specifics of the architectures, so, for now, we will spare ourselves those details.</p><p>Indeed, average improvements can reach 10.6% for binary classification (to make a decision between two relations) and 8% for multi-class classification (decision between more than two relations) over non-patch baseline approaches. As things stand, we believe that we might get even better results if we construct a cuboid, where the word vectors of two words, instead of being collapsed to a single similarity value, are preserved to a greater extent by only compressing them from 600 dimensions to 300 dimensions. (We do this by taking the dot product between the two vectors. Dot product of two 300-dimensional vectors results in a new 300-dimensional vector). Our results with the cuboid have been promising and have shown an enhanced performance on our previous baselines. However, there are many more tests that the model needs to come through before it can be claimed as an outright upgrade over its predecessors. Our next step is to compare our results with models that attempt to solve the same problem. Whichever way it ends, I will be sure to update this blog on our progress.</p><p>As for NLP in general, there is no doubt whatsoever in saying that we are still decades, or at best, years, away from being anywhere close to designing an artificial intelligence that speaks and communicates like us. The amount of data needed today to train a computer well for the simplest of tasks is tremendous. We have neither reached the peak in terms of quality of data representations nor do we have enough computational power to scale models trained on current data representations beyond a particular extent.</p><p>On reflection though, it will never stop being crazy to me that with a little knowledge of mathematics, sufficient computational power and a decent familiarity with a programming language, you can teach a completely inanimate object to understand the language of our species. It is quite surreal when you think about it.​</p><p><em>This blog post is based on the work carried out with Nesrine Bannour and Houssam Akhmouch, under the supervision of Prof. Gaël Dias.</em></p><p><strong><em>You can visit my page </em></strong><a href="https://karanpraharaj.github.io/"><strong><em>here</em></strong></a><strong><em>. You can follow my twitter feed </em></strong><a href="https://twitter.com/IntrepidIndian"><strong><em>here</em></strong></a><strong><em>.</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=427ed331e287" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/identification-of-lexico-semantic-word-relations-a-beginners-guide-427ed331e287">Identification Of Lexico-Semantic Word Relations — A Beginner’s Guide</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Attention for Machine Translation]]></title>
            <link>https://medium.com/data-science/attention-for-machine-translation-4a95db38ee34?source=rss-2eee9862711a------2</link>
            <guid isPermaLink="false">https://medium.com/p/4a95db38ee34</guid>
            <category><![CDATA[machine-translation]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[attention-mechanism]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Karan Praharaj]]></dc:creator>
            <pubDate>Sat, 27 Jun 2020 01:14:19 GMT</pubDate>
            <atom:updated>2020-06-28T13:35:31.514Z</atom:updated>
            <content:encoded><![CDATA[<h4>A brief round-up of the attention mechanism for neural machine translation models.</h4><p>We are not cerebrally hardwired to process loads of information at once. However, what we <em>are</em> good at, is focusing on a part of the information we’re given to make sense of it. When asked to translate a sentence from one language to another, you process the sentence by picking up individual words as you go, skewering them together into phrases and then mentally assigning corresponding words/phrases in the target language for each part.</p><p>When a written word is presented to you for translation, it kicks off a series of neurocognitive processes in your brain. Your normal language system in the brain reads the word in your native language, but for translation, this usual tendency must be suppressed so the translated meaning can emerge. A specific region of the brain (known as the “caudate nucleus”) coordinates this activity, much like an orchestral conductor, to produce stunningly complex behaviours. Essentially, the syntactical, phonological, lexical and semantic aspects of the word or sequence of words are encompassed, assimilated and then contextualized to render the said sequence of words into its equivalent in the target language.</p><p>Translations — be it by human or by machine — are not objective in their character, which is to say that there is no <em>unique</em> translation for any given sentence. Human translation inherently carries the risk of introducing source-language words, grammar or syntax into the target language rendering. A translation by two humans of a relatively long sentence can rarely ever be the exact same. Despite the variance in the final result, though, what does not change is the broad process by which the translations were arrived at. For any target-language word or phrase that was rendered in the translation, the translator paid attention to some parts of the source sentence more than others.</p><p>This innate quality of humans had not been given to machine algorithms until 2015, when Bahdanau, Cho and Bengio introduced the “attention” mechanism. This mechanism was proposed to maximise translation performance by allowing a model to automatically (soft-)search for parts of a source sentence that are relevant to predicting a target word, without having to form these parts as a hard segment explicitly. I concede that this sentence may be hard to digest, but we will make sense of it by breaking it down and paying attention to parts of it one at a time. (Very meta, I know.) The impact of this 2015 paper was profound and it would go on to become the building block for several state-of-the-art models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/841/0*bPr_P2ZZcnt1Nhq3.png" /></figure><h4>Why do we need Attention?</h4><p>In conventional neural machine translation models, an encoder-decoder combination is used with an encoder and a decoder for each language, or a language-specific encoder applied to each sentence whose outputs are then compared. An encoder RNN reads and encodes a source sentence into a fixed-length vector. A decoder then spits out a translation based on the vector fed to it from the encoder. The whole encoder–decoder system, which consists of the encoder and the decoder for a language pair, is jointly trained to maximize the probability of a correct translation given a source sentence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FWjiRK7J0J7VNiWZ.png" /></figure><h4>The Bottleneck</h4><p>A problem with this architecture is its over-reliance on one fixed-length vector to encompass all the necessary information and be a good quality representation of the source sentence. This pressure on the fixed-vector vector to compress and capture all the information is a bottleneck and it makes it difficult for the encoder neural network to perform well on long sentences. It has been shown earlier that the performance of a basic encoder–decoder deteriorates rapidly as the length of an input sentence increases.</p><h4>Core Idea</h4><p>In order to address the issue, Bahdanau et al. introduced an extension to the encoder-decoder model which learns to align and translate jointly. The new architecture deploys a bidirectional-RNN as an encoder and a decoder that will be able to focus on all hidden states instead of just the final hidden state. What this modification does, is afford the decoder a flexibility in decision-making and therefore identify the parts of the source sentence that may be more relevant for the prediction of the next word in the target sentence. This is the intuition of the attention mechanism, and it leads us now to the mathematics that goes into making this happen.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/0*lJH9lr5jiMo6Ry9b.png" /></figure><p>Thus, this new approach facilitates the information to be spread across the sequence of annotations, which can be selectively retrieved by the decoder accordingly.</p><p>By the medium of language, we manage to communicate ideas over long ranges of space and time, but the creation of syntactic bonds between words in a sentence that may or may not be in close proximity to each other underpins expression of ideas in any language. This is where attention steps in and aids the mapping of syntaxes from the source language to the target language. To identify relationships of words with other words that maybe far away in the same sentence — all while ignoring other words that just do not have much influence on the word we’re trying to predict — that is what attention aims to do.</p><p><em>I would have liked to explain the </em><a href="https://karanpraharaj.github.io/post/attention/"><em>algebra of the attention mechanism</em></a><em>, but I found Medium’s editor rather unconducive for mathematical writing. You can find the explanation at the </em><a href="https://karanpraharaj.github.io/post/attention/"><em>bottom of this page</em></a><em>.</em></p><p><strong><em>You can visit my page </em></strong><a href="https://karanpraharaj.github.io/"><strong><em>here</em></strong></a><strong><em>. You can follow my twitter feed </em></strong><a href="https://twitter.com/IntrepidIndian"><strong><em>here</em></strong></a><strong><em>.</em></strong></p><h3>References</h3><p>[1] Bahdanau, D., Cho, K. &amp; Bengio, Y. <a href="http://arxiv.org/abs/1409.0473">Neural machine translation by jointly learning to align and translate. </a>In <em>Proc. International Conference on Learning Representations</em> (2015)</p><p>[2] Cho, Kyunghyun, Aaron Courville, and Yoshua Bengio. <a href="http://arxiv.org/abs/1507.01053">Describing Multimedia Content using Attention-based Encoder–Decoder Networks.</a> (2015)</p><p>[3] Sutskever, I. Vinyals, O. &amp; Le. Q. V. <a href="https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf">Sequence to sequence learning with neural networks.</a>In <em>Proc. Advances in Neural Information Processing Systems.</em> (2014)</p><p>[4] Chris Olah’s blog post. <a href="https://distill.pub/2016/augmented-rnns/">“Attention and Augmented Recurrent Neural Networks</a>”</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4a95db38ee34" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/attention-for-machine-translation-4a95db38ee34">Attention for Machine Translation</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>