<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vishwani Gupta on Medium]]></title>
        <description><![CDATA[Stories by Vishwani Gupta on Medium]]></description>
        <link>https://medium.com/@vishwanigupta?source=rss-5a0d9771c890------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*rSyduOPkzPvmOU5CHePo6w.jpeg</url>
            <title>Stories by Vishwani Gupta on Medium</title>
            <link>https://medium.com/@vishwanigupta?source=rss-5a0d9771c890------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 04:53:33 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@vishwanigupta/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[More test scenarios for fake news detection.]]></title>
            <link>https://medium.com/@vishwanigupta/more-test-scenarios-for-fake-news-detection-201606693ab3?source=rss-5a0d9771c890------2</link>
            <guid isPermaLink="false">https://medium.com/p/201606693ab3</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[journalism]]></category>
            <category><![CDATA[fake-news]]></category>
            <category><![CDATA[misinformation]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Vishwani Gupta]]></dc:creator>
            <pubDate>Fri, 09 Mar 2018 21:43:09 GMT</pubDate>
            <atom:updated>2018-03-11T22:18:53.170Z</atom:updated>
            <content:encoded><![CDATA[<h4>Continued from <a href="https://medium.com/@vishwanigupta/combating-fake-news-with-the-help-of-machine-learning-b56537609564">Combating Fake NEWS with the help of Machine Learning.</a></h4><p>To garner more confidence in the <strong>algorithm</strong>, I ran it on a variety of cases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/648/1*6Tpc_NAwzchm5Jv2JLjYuQ.jpeg" /><figcaption><a href="http://home.bt.com/tech-gadgets/internet/why-does-fake-news-go-viral-11364225690826">Problem of fake news</a></figcaption></figure><blockquote>Note: In the following plots: X-axis represents the reliable sources and Y-axis represents the <strong>distance</strong> between reliable sources articles and query article. For the most related articles these distances should be lower and vice versa.</blockquote><blockquote>Also as we have seen in the previous post, if the value of <strong>distance</strong> is <strong><em>below 0.65</em></strong>, we can define a probability distribution of <em>trust</em> in the article. Whereas if the threshold is <strong><em>greater than 0.65</em></strong>, we can again define a probability distribution of <em>mistrust</em> in the article.</blockquote><h3>1. Right vs Left reports on the same news story</h3><p>Most of the news sources are biased. Some try to neutralize it whereas some make money out of it. Usually, such sources write a genuine story but with a infuriating or misleading headline.</p><p>One such example is<strong> Political mayhem news.</strong> These news sources though report genuine news in the news body (although opinionated), trick us in the news headline. Because of the infuriating headlines, such articles get shared on the social media in abundance. One such example is:</p><p><a href="https://politicalmayhem.news/breaking-senate-takes-first-step-to-punish-obama-for-his-role-in-russian-nuclear-bribery-case/">BREAKING: Senate Takes First Step To Punish Obama For His Role In Russian Nuclear Bribery Case - Political Mayhem</a></p><p>Although the title is very misleading, the content is genuine. The source claims that the article is shared 1.1K times on social media. As we all know, when we come across such articles on social media, we do not have time to go through the whole article but rely mostly on the headline. <strong>In this scenario, since the body is genuine but the title is not, I decided to label them individually.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/551/1*bVJEWLJlCCnq-xfKoT0WfA.png" /><figcaption><strong>Result: Genuine news but check the Title.</strong></figcaption></figure><p><strong><em>The above story is genuine and from the right wing media, we can also see that Fox news has many articles similar to this story.</em></strong></p><h3>2. When dicey sources publishes real news which is unorthodox.</h3><p>When we stumble on news articles which are not visible in mainstream media and are also <strong>unorthodox</strong>, we usually ignore them taking them to be fake or hoax. But sometimes these stories are genuine news. <strong><em>These stories though published by the main stream media are not in focus because Donald Trump makes breaking news every day!</em></strong></p><p><a href="http://yournewswire.com/pain-reliever-hormones-testicles/">Popular Pain Reliever &#39;Disrupts Male Hormones&#39; And &#39;Shrinks Testicles&#39;</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/551/1*CcR4c4HAPOI_YcAqXElLTw.png" /><figcaption><strong>Result: Genuine news and correct title</strong></figcaption></figure><p><strong><em>Such articles should not be assumed to be fake news. This problem can be avoided using this algorithm as it tags them as genuine news.</em></strong></p><h3>3. Articles from the new players in the industry</h3><p>When news players like Vice News (launched in December 2013), publishes some news article, we are not very sure of the genuineness of the article. For example,</p><p><a href="https://www.vice.com/en_us/article/9kz7a7/martin-shkreli-cried-before-getting-seven-years-in-prison">Martin Shkreli Cried Before Getting Seven Years in Prison</a></p><p>But when we run the algorithm for this article, we find that this is indeed true.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/551/1*p9q1P6gcXBnhpgqmLG9-mg.png" /><figcaption><strong>Result: Genuine news and correct title</strong></figcaption></figure><p><strong><em>Hence slowly this can enable user to build their trust in such news sources.</em></strong></p><h3>4. Very recent news story</h3><p>In case of a very recent news event, the algorithm tries to gather more proof as time passes. For example: <strong>On 8th March 2018, 21:12 PM CET, Trump turns the spotlight on violent video games,</strong> as reported by CBS and Reuters.<strong> </strong>But nothing was reported by most of the other reliable sources.</p><p>In such cases, when tested after some time, we can see that during the interval nearly all other sources reported similar stories.</p><p><a href="http://abcnews.go.com/Politics/trump-turns-spotlight-violent-video-games-wake-parkland/story?id=53593714">Trump turns spotlight on violent video games in wake of Parkland shootings</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/551/1*jazuz24zqEe0HTHeoZOw9g.png" /><figcaption><strong>Result: Genuine news and correct title but need more proof (8th March, 21:12 CET)</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/551/1*MI-AgNcM6ypzrBEgCgH1gg.png" /><figcaption><strong>Result: Genuine news and correct title (9th March, 0:17 CET )</strong></figcaption></figure><h3>5. Most hyped news</h3><p>One of the most famous fake news article was published by <strong><em>WTOE 5 News</em></strong><em>. </em>The article claimed that <strong>Pope Francis had broken with tradition and unequivocally endorsed Donald Trump for President of the United States</strong>. This turns out to be FAKE.</p><p>So when <strong><em>dailybeast</em></strong> published that <strong>Pope Francis said that Trump is not a Christian</strong>, it is very natural to be skeptical about the article.</p><p><a href="https://www.thedailybeast.com/what-pope-francis-really-said-about-trump-not-being-christian">What Pope Francis Really Said about Trump Not Being Christian</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/551/1*Y_dbbcHIYSqMbMvcMclCLw.png" /><figcaption><strong>Result: Genuine news and correct title</strong></figcaption></figure><p><strong><em>Again, using this algorithm we can avoid prior bias for news articles.</em></strong></p><h3>6. News stories from tech section</h3><p>The technology sector is also not exempted from the problem of fake news. Recently, there was an article which took real facts and concluded something ominous thus creating a hoax. They reported that <strong><em>concerned artificial intelligence researchers hurriedly abandoned an experimental chatbot program after they realized that the bots were inventing their own language.</em></strong></p><p><a href="http://bgr.com/2017/07/31/facebook-ai-shutdown-language/">Facebook engineers panic, pull plug on AI after bots develop their own language</a></p><p>In technical news articles, I rely on tech news sources such as:</p><blockquote><strong>techcrunch, verge, cnet, mashable, wired, thenextweb, engadget, techradar</strong></blockquote><p>Using these sources in my algorithm, I got the following result for the article:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/551/1*aDUUOEfn0cEDJov889vfCg.png" /><figcaption><strong>Result: Fake news and incorrect title</strong></figcaption></figure><p>Even with general news sources, the article was tagged as fake:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/551/1*IwTchozvEVcxot96mKwUfA.png" /><figcaption><strong>Result: Fake news and incorrect title</strong></figcaption></figure><h3>Conclusion:</h3><p>Using this algorithm, we can generate a score of similarity between query article and articles from reliable sources. We can protect ourselves from fake news articles published on social media through some novice news sources, yet not miss the genuine articles from the same sources. This is especially beneficial if a new news source comes into the picture.</p><p>Also we can calculate how frequently and in what ratio it publishes genuine news and deceptive news. Keeping this as a reliability score, we can use this as a reliable source. Hence it can break the monopoly of brands like CNN.</p><p>Facebook’s measures to fight fake news by prioritizing feeds just from the reliable sources has effected the business of new generation of digital media companies like Vice, Vox.</p><blockquote>“The news feed changes will have the most negative impact on publishers that rely primarily on Facebook for referral traffic and those companies that specialise in producing and distributing sponsored videos for Facebook,” says <a href="https://www.ft.com/content/482dc54a-1594-11e8-9376-4a6390addb44">Christopher Vollmer, global entertainment and media advisory leader at PwC</a></blockquote><p>Using such an approach, this can be avoided besides keeping the reader informed about the <strong><em>genuineness</em></strong> of the story.</p><blockquote>If you have some interesting test case which can be used as a test case, please add it in the comments!</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=201606693ab3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What I have learned during the last year living alone in Germany.]]></title>
            <link>https://medium.com/@vishwanigupta/what-i-have-learned-during-last-year-living-alone-in-germany-8c25de18835b?source=rss-5a0d9771c890------2</link>
            <guid isPermaLink="false">https://medium.com/p/8c25de18835b</guid>
            <category><![CDATA[germany]]></category>
            <category><![CDATA[women-in-tech]]></category>
            <category><![CDATA[ambition]]></category>
            <category><![CDATA[living-with-purpose]]></category>
            <dc:creator><![CDATA[Vishwani Gupta]]></dc:creator>
            <pubDate>Fri, 09 Mar 2018 15:16:42 GMT</pubDate>
            <atom:updated>2018-04-26T08:02:23.662Z</atom:updated>
            <content:encoded><![CDATA[<p>I come from a humble family where one is taught to work hard since it brings satisfaction and happiness.</p><p>Bachelors from a private university in India, made me realize that I have to do better. Hence I pursued masters course in Germany.</p><p>Here, not only did I find my calling but also learn how to live life:</p><ol><li>I stopped hiding my ambition to seem cool or make friends</li><li>I stopped caring if people judge me for working hard and avoiding parties.</li><li>I try even harder to solve a problem instead of asking for help after a little effort.</li><li>Stopped being paranoid about being an introvert(INTP). I try to be more vulnerable and share more.</li><li>I also try to challenge the opinions of established people.</li><li>Stopped feeling weird when I am the only girl in the room.</li><li>When stuck, try to find alternatives rather than giving up.</li><li>I don’t suppress my itch to do something great anymore just to avoid being labeled ambitious woman.</li><li>Read, read and read. I found my lost love again. Be it new research paper or a book, I make sure that I read something everyday.</li><li>There are so many awesome people in the society who are making this world a better place. I try to follow their steps.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8c25de18835b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Combating Fake NEWS with the help of Machine Learning.]]></title>
            <link>https://medium.com/@vishwanigupta/combating-fake-news-with-the-help-of-machine-learning-b56537609564?source=rss-5a0d9771c890------2</link>
            <guid isPermaLink="false">https://medium.com/p/b56537609564</guid>
            <category><![CDATA[news]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[fake-news]]></category>
            <category><![CDATA[misinformation]]></category>
            <dc:creator><![CDATA[Vishwani Gupta]]></dc:creator>
            <pubDate>Wed, 24 Jan 2018 22:05:07 GMT</pubDate>
            <atom:updated>2018-03-11T22:14:24.847Z</atom:updated>
            <content:encoded><![CDATA[<p>In the age of social media, everyone has witnessed the effect of fake news during USA Presidential election 2016, as also discussed in <a href="https://www.nytimes.com/2017/09/07/us/politics/russia-facebook-twitter-election.html">NYTimes</a>. Recently there have been many efforts to alleviate the effect of false stories. Many of these efforts use machine learning approaches to predict the probability of a story being false.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/1*KTwFmfVO2xHrHVXvjdsPhg.png" /><figcaption>Fake News is not only effecting people’s opinion, <a href="https://www.finance-monthly.com/2017/07/heres-why-fake-news-is-killing-the-economy/">killing economy </a>but also effecting democracy. There is an urgent need to combat it.</figcaption></figure><p>As a news junkie, I used to get annoyed by fake news articles shared on social media since the latter has become one of my main sources of getting news. To filter such articles from my news feed, I followed the same procedure as most of us would do, that is to fall back on our reliable sources and check if these sources have similar stories on the topic in the matter.</p><p>Nonetheless, searching for a similar article in the trusted sources is a slow and cumbersome process. To automate it, I tried to use <a href="https://en.wikipedia.org/wiki/Natural_language_processing">Natural Language Processing(NLP)</a> approaches and came up with a simple algorithm which will do all this hectic work for me. My intention was to give the link to the dubious story and the algorithm tells me whether it can be trusted or not.</p><p>I followed <strong><em>an unsupervised approach </em></strong>for detecting fake news. The main reasons I chose to use an unsupervised approach were as follows:</p><ol><li>One cannot predict the events in the real world scenario which may act as outliers in a supervised approach.</li><li>Actors in the news stories change very frequently. Therefore, training a neural network on a set of articles does not make much sense, if a new actor enters in the news.</li><li>Unsupervised approaches also have the benefit that one can easily generalize this problem to any language and region without training on a huge dataset.</li></ol><p>Regardless of Donald Trump’s claims, I trust the following sources as also cited by <a href="https://www.forbes.com/forbes/welcome/?toURL=https://www.forbes.com/sites/berlinschoolofcreativeleadership/2017/02/01/10-journalism-brands-where-you-will-find-real-facts-rather-than-alternative-facts/&amp;refURL=&amp;referrer=#509d7f2ce9b5">Forbes</a>,:</p><p>CNN, Washington Post, NYTimes, WSJ, CNBC, TIME, Al-Jazeera.</p><p>The algorithm find the 20 most related articles from each of these sources and generate a distance measure. Through this distance measure, it then classifies the query article as fake or genuine.</p><p>When one of my friends shared a fake story about ‘<em>US banks begin closing customer accounts caught using Bitcoin’,</em> I decided to test my approach.</p><p><a href="http://yournewswire.com/banks-close-accounts-bitcoin/">US Banks Begin Closing Customer Accounts Caught Using Bitcoin</a></p><p>My algorithm first scrapes the articles from these <strong><em>reliable news sources</em></strong> related to the Bitcoin and also makes use of the time period in which the article was published while scraping the articles. <strong><em>It then evaluates a similarity score/distance between those articles and the article in question using text mining</em></strong>.<strong><em> Text mining used in the algorithm makes sure that the idea behind the story and facts are understood by the algorithm.</em></strong></p><p><strong><em>As expected, the similarity scores were very low and the distances between the reliable source articles and query article were very high.</em></strong></p><blockquote>Note: In the following plots: X-axis represents the reliable sources and Y-axis represents the <strong>distance</strong> between reliable sources articles and query article. For the most related articles these distances should be lower and vice versa.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/551/1*2VDHEDZKzGz56j9_pUav7g.png" /><figcaption><strong>Result: Fake News.</strong> In this plot, the Y-axis is the distance between articles about Bitcoin on reliable sources and query article. The more the distance, the less is the similarity.</figcaption></figure><p>I used the box plots because it is a useful way to visualize the range and distribution of different articles’ distances from the query article. Details of a box plot can be found <a href="http://www.physics.csbsju.edu/stats/box2.html">here</a>.</p><p>At one point, the algorithm makes use of word embeddings as one of a text mining step to find distance/similarity between different articles. This can make NLP researchers wonder about the relatively new words which do not have a lot of examples to get good word vectors, since <a href="https://arxiv.org/abs/1402.3722">Word2vec</a> model which generates word vectors, needs a huge dataset to generate high quality word vectors. <strong>The latter is taken care by training the word vectors on an even smaller dataset using Kernel PCA Skip gram model (which was a generated as a part of my masters thesis). More details can be found in the following post:</strong></p><blockquote><a href="https://medium.com/@vishwanigupta/kpca-skip-gram-model-improving-word-embedding-a6a0cb7aad49">Kernel PCA Skip gram Model</a></blockquote><p>Intrigued by the fake article results, I also tested the algorithm for the <strong><em>genuine articles</em></strong> like:</p><p><a href="http://edition.cnn.com/2017/12/31/asia/kim-jong-un-new-year-address-nuclear/index.html">Kim Jong Un offers olive branch to South</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/551/1*ee1B_9-gqhFbC1YvbeHZow.png" /><figcaption><strong>Result: Genuine News. </strong>All the articles from these reliable sources have lower distance.</figcaption></figure><p>I found that most of the sources have reported the articles with same story and hence the distance between the query article and reliable source articles were very small.</p><blockquote>From these two examples, I saw that it is possible to set a <strong><em>threshold</em></strong> for the distance plots for the two cases when the query article is genuine and when it is fake.</blockquote><h3>Thresholding for fake articles:</h3><p>The threshold value was determined after testing a large set of articles (having both fake and genuine articles) using this algorithm. The set aslo included articles from different news sections such as politics, entertainment, sports, technology etc.</p><p>Based on the results from these tests, a threshold value of 0.65, was easy to comprehend when using Kernel PCA embeddings. If the value is <strong><em>below 0.65</em></strong>, we can define a probability distribution of <em>trust</em> in the article.</p><p>Whereas if the threshold is <strong><em>greater than 0.65</em></strong>, we can again define a probability distribution of <em>mistrust</em> in the article.</p><h3>Some special scenarios:</h3><p>I ran the algorithm on articles from various scenarios. For example: the articles, which might have false news article headline but true article body and vice versa.</p><p>For example:</p><p><a href="http://yournewswire.com/chelsea-clinton-church-satan-year/">Chelsea Clinton Wishes Church Of Satan &#39;Happy New Year&#39; On Twitter</a></p><p>Although the article title is genuine, the story of the article is false as they claim that Chelsea Clinton practices Satanism. Therefore the distances are above the threshold. Here are the results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/551/1*SGQ_X_-Pn60aQ7-2EAlhsA.png" /><figcaption><strong>Result: Fake News</strong>. The body of the article is having fake news.</figcaption></figure><p>Another example of a genuine news story and the results: As expected all the distances are below threshold.</p><p><a href="http://variety.com/2018/politics/news/steve-bannon-donald-trump-jr-russia-1202652030/">Steve Bannon Calls Donald Trump Jr.&#39;s Meeting With Russians &#39;Treasonous&#39; in New Book</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/551/1*7Hqw7np-hDJshDiY-1KDVw.png" /><figcaption><strong>Result: Genuine News</strong>. This also followed the same trend of threshold less than 0.65.</figcaption></figure><p>Now one might question that it is possible that the evaluated article is a <strong><em>very recent news </em></strong>and it might not be published else where. My algorithm takes care of this and will notify the user to test again in few minutes.</p><p>An example scenario is:</p><p><a href="http://ew.com/movies/2018/01/03/harvey-weinstein-paul-sorvino/">Paul Sorvino threatens Harvey Weinstein: &#39;I will kill that motherf---er&#39;</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/551/1*h9r78Kzl4Prmkgu5Ezl_nw.png" /><figcaption><strong>Result: Genuine News but needs more proof.</strong> This was tested when the article was just published. Here one can realize that only three sources have reported the similar article with distance very low or similarity high.</figcaption></figure><p>Here it can be seen that only three of the reliable sources reported on the same news. When tested after some time to accumulate more proof, one can see that during the interval CNN has also published a story related to the topic:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/551/1*b4UuGT3_pK0sYeErvy7lqQ.png" /><figcaption>More proof accumulated as some time passes.</figcaption></figure><h3>Future Work:</h3><p>Based on the thresholding and number of fake news articles it has published, I can assign a reliability score to each source and also to novice news sources. This will make the news sources to be more careful of what they publish.</p><h3>Facebook measures for fighting fake news:</h3><p>In the pursuit of making “meaningful social interactions”, Facebook has recently <a href="https://newsfeed.fb.com/?lang=en">announced</a> that the content directly from publishers won’t perform well unless people engage with it. This means that there will be less news related articles on the users’ Facebook wall. It is especially bad according to <a href="https://www.wsj.com/articles/what-facebooks-feed-changes-mean-for-the-news-1516617122">WSJ</a> as 45% of Americans get their news from <a href="http://quotes.wsj.com/FB">Facebook</a>.</p><h3>Helping the crusade against fake news:</h3><p>The algorithm in this article, offers a solution to this problem. There can be an initial check before any news post could appear on the Facebook wall. If the probability is below the threshold, it could appear on the wall. Furthermore, the number of reliable sources having the similar article should be weighed in the scoring.</p><p>The use of KPCA Embeddings allows me to generalize the algorithm on any language and region, even on languages for which one doesn&#39;t have a huge dataset.</p><h3>Conclusion:</h3><p>Using this algorithm, I can protect myself from fake news articles published on social media through some dubious sources, yet not missing the genuine articles from the same sources.</p><p>Once fake news publishers realize that this system can be used on a large scale and will filter out their fake news from people’s access (affecting negatively their business), they will be compelled to be more responsible with the content they share.</p><blockquote>Details of the algorithm will soon be published in a scientific paper.</blockquote><h3>If you are interested in how this can be a stepping stone:</h3><h4><a href="https://medium.com/@vishwanigupta/more-test-scenarios-for-fake-news-detection-201606693ab3">Here are some more cool examples</a></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b56537609564" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Improving word embedding using Kernel PCA Skip gram model]]></title>
            <link>https://medium.com/@vishwanigupta/kpca-skip-gram-model-improving-word-embedding-a6a0cb7aad49?source=rss-5a0d9771c890------2</link>
            <guid isPermaLink="false">https://medium.com/p/a6a0cb7aad49</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[word-embeddings]]></category>
            <category><![CDATA[word2vec]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Vishwani Gupta]]></dc:creator>
            <pubDate>Wed, 24 Jan 2018 15:48:49 GMT</pubDate>
            <atom:updated>2018-01-28T21:36:15.695Z</atom:updated>
            <content:encoded><![CDATA[<p>The Natural Language Processing problems are not trivial, making them “AI-hard problem” in the field of computer science. In order to understand a sentence, one should not only understand the words used in it but also the context in which those words are used since a word can have very different meaning when used in different contexts. This makes language understanding ambiguous. <br>The solution comes by focusing on the “atomic units” of a language, “words”. To get the precise results, there is a dire need of a numeric vector representation for the words. These word representations help in extracting relations between various human languages, text or speech data.</p><p>There are many representations proposed over time:</p><ol><li>One hot vectors</li><li>Skip gram model and CBOW model by <a href="https://arxiv.org/abs/1310.4546">Tomas Mikolov</a></li><li>fastText by <a href="https://fasttext.cc/">Facebook</a></li></ol><p>These approaches does not produce good word vectors when we have a relatively smaller data set since they depend on learning from the context in which the words are used and do not consider the morphological information embedded in the words. FastText has tried to use sub word information but don’t produce high quality word vectors with smaller datasets.</p><p>We propose an alternative approach which not only make use of the sub word information but also performs well on smaller datasets. This approach can be applied on both morphological and non-morphological languages. The main challenges we are trying to solve for generating word embeddings are:</p><ol><li>The need to train on a huge text corpus to extract relationship between words.</li><li>Word embeddings for out-of-vocabulary words using n-grams with significant increase in vocabulary size.</li><li>Slow rate of training even after application of optimization techniques such as softmax and negative sampling.</li></ol><p>Below, we first discuss nuts and bolts of the algorithm and then compare our model, Kernel PCA (KPCA) skip gram model with word2vec skip gram model trained using same parameters.</p><h3>Nuts and bolts of the approach:</h3><h4>Kernel Function:</h4><p>The kernel trick is a mathematical tool which can be applied to any algorithm which consists of a dot product between two vectors. Kernel trick states that to compute the dot products of vectors in the higher dimensional space, a kernel function can be used directly using the lower dimensional vectors. Therefore every dot product can be replaced by a kernel function. <a href="https://www.cs.utexas.edu/~mooney/cs391L/">A Kernel function</a>, can thus be defined as a function that takes as inputs vectors in the original space and returns the dot product of the vectors in the feature space. The feature space is usually a higher dimensional space where data can be linearly separable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/447/1*a1I5bTVfBE80uPrr0N0ypw.png" /><figcaption>Transformation from one space to another using a Kernel function φ <a href="https://www.cs.utexas.edu/~mooney/cs391L/">(Mooney<br>[2007])</a></figcaption></figure><p>In order for a function to be a valid Kernel function, it should fulfill the “Mercer’s theorem”. According to the Mercers’ theorem, every positive definite symmetric function can be seen as a kernel function. Here a positive definite symmetric functions correspond to a “positive definite symmetric Gram matrix” which can be understood as a matrix which has all positive eigenvalues. Therefore a function which fulfill these conditions can be refereed as kernel function.</p><p>We usually, prefer to work with<strong><em> Gaussian Kernel</em></strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/279/1*RGJrvhM64F6_AGr2O6WEwQ.png" /></figure><p>The Gaussian RBF kernel is very popular and makes a good default kernel as the feature space of this kernel has an infinite number of dimensions.</p><h4>Principle Component Analysis:</h4><p>Principal component analysis(PCA) is a very important tool in data analysis. It helps in reducing a complex data set to a lower dimensional space thus revealing the hidden, dynamics. It helps in getting the most meaningful basis to re-express any noisy data set.</p><h4><strong>Kernel PCA:</strong></h4><p>Kernel Principle Component Analysis is the nonlinear form of PCA, which allows to generalize standard PCA to nonlinear dimensionality reduction. In other words, it can be referred as a non-linear dimensionality reduction technique through the use of kernel function. <a href="https://link.springer.com/chapter/10.1007/BFb0020217">Schölkopf et al. [1997]</a> have shown that the generalization of PCA for non-linear data can be done via a kernel function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/397/1*K7B7TzzvLH-MJDL9fn22wQ.png" /><figcaption>Kernel PCA: A nonlinear kernel function κis used instead of the standard<br>dot product. It implicitly perform PCA in a possibly high-dimensional space F which is<br>nonlinearly related to input space. (Schölkopf et al. [1997])</figcaption></figure><h4>Skip gram model:</h4><p>Skip gram is a model proposed by <a href="https://arxiv.org/abs/1310.4546">Mikolov et al.</a>, which tries to maximize classification of a word based on the words in the same sentence. In other words, the model predicts the context words given their center word.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/478/1*0E7On2N1JsSLrlhLPWEuRw.png" /><figcaption>Skip-Gram Model <a href="https://arxiv.org/abs/1411.2738">(Rong [2014])</a></figcaption></figure><p>Here distance of words to the input word, represented by <strong>C,</strong> also comes in play. This distance is important since the more distant words are less related to the center word. So one can assign less weight to distant context words thus helps to attain better context words.</p><h3>Approach:</h3><ol><li>Pre-processing the dataset like Text8, 20News set, wiki dump.</li><li>Implementing Kernel PCA on words in a vocabulary.</li><li>Computing Kernel PCA of out-of-vocabulary words.</li><li>Training a word2vec model(Skip gram) with Kernel PCA Em-<br>beddings in Tensor flow.</li></ol><h3>Comparison:</h3><p>We have evaluated the performance of our Kernel PCA skip gram model by comparing it against the performance of the basic skip gram model trained using the same parameters. Since the model follows an unsupervised approach, evaluation of the model can only be done using “intrinsic evaluation tasks”, which assess how well the vectors capture the word meaning and the relationships between those words. This is achieved using evaluating the models on the following tasks:</p><ol><li>word similarity tasks, that include finding a word’s nearest neighbors.</li><li>word analogy tasks, that include calculating the semantic and the syntactic similarity between the words.</li></ol><h4>Word similarity tasks:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/1*hgBSL2dqphMp7mgWxAHLeQ.png" /><figcaption>The k =4-nearest neighbors are generated for vectors of 300 dimension on a<br>dataset 20NewsGroup of size 17 Million words (Non morphological Language)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/757/1*T6eUhDGOCAHKSAIMN4m5Nw.png" /><figcaption>The k =4-nearest neighbors for vectors of 300 dimension generated from training the models on the subset of the dataset, news.2013.de of size 52 Million words for the morphological rich language: German.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*oaUi6A1jUNpsjlV4k9UjtQ.png" /><figcaption>Two-dimensional PCA projection of the 300-dimensional skip gram vectors of countries and their capital cities trained on English language in 100,000 iterations on subset of dataset Text8. Countries(Red labels)and Captials(green labels)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/595/1*e_17WsX3Av9WyzOcc9bCig.png" /><figcaption>Two-dimensional PCA projection of the 300-dimensional Skip-gram vectors of countries and their capital cities trained on German language in 100,000 iterations on subset of dataset news.2013.de.shuffled.corpus. Countries(Red labels)and Captials(green labels)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/696/1*pSEUZ4wXpzei7o3eh8QWZw.png" /><figcaption>Comparison of Word Analogy accuracy of the two models trained for 1 Epoch using same parameters with 300-dimensional word vectors. NOTE: For vocabulary size of 118,000 for KPCA embeddings, we compute the KPCA for 20,000 words and the additional words(OOV) vectors</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/1*VKMGHuvKPRPcYM3r4pzZGg.png" /><figcaption>Word Analogy accuracy of both models through different epochs on Semantic-Syntactic data set</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/748/1*WJhZXzCtij7Q5Pibf0o8uA.png" /><figcaption>Comparison of architectures of the two models trained for 1 Epoch using same parameters with 300-dimensional word vectors on enWikidump of size 1 Billion words with Vocabulary size 30,000</figcaption></figure><h4>tsne plots:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xRpu5YpmZsza3qRjSXiq5A.png" /><figcaption>t-SNE visualization of <strong>KPCA Skip gram</strong> embeddings with Vocabulary size 118,000 words, trained in 100,000 iteration on Text8 dataset of size 17 Million words. If analyzed closely, we find clusters of nationality, nations, first names etc. Another important thing to note is that clusters of the nationality and the nations also lie close to each other.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8sT14GUMgkzDUqbktB276w.png" /><figcaption>t-SNE visualization of <strong>Skip gram</strong> model with vocabulary size 118,000 trained in 100,000 iteration on Text8 dataset. On analyzing this plot, we cannot find any prominent cluster. The bad performance of the model can be attributed to the fact that we are training the model on a relatively smaller database for 1 Epoch. The performance ought to improve when we train the model for more epoch.</figcaption></figure><p>From the above results, we can summarize that KPCA Skip gram Model performs far better than the basic skip gram model. This can be attributed to the fact that we already have trained the vectors for morphological similarities. The model just needs to learn the semantic and syntactic similarity. Since basic skip gram model learns morphological similarity implicitly so initializing the model with those helps in speed up the process of convergence. The performance is even better when the language is rich in morphemes such as German.</p><h3>Conclusion:</h3><p>This morphological information is encoded in the Kernel PCA embeddings which are obtained by applying Kernel PCA on some defined similarity function of the words. Therefore when we initialized the skip gram model by these morphological similarity words, we found that our model achieve fairly good accuracies in fewer iterations. We have also evaluated the embeddings by intrinsic methods of “word similarity” and “word analogy”. In every task, our model has performed better than the basic skip gram model. From the evaluation results, we understood that, a skip gram model can learn similarity between words based on context only when it has seen sufficient number of examples. This can be achieved by either training the skip gram model with a huge dataset or training it with a relatively smaller dataset but with numerous epochs. But when we fed the skip gram model with morphological embeddings, it already has a good initial point from where it can improve these embeddings. This works, especially well, when we are training on a morphologically rich language since semantic and morphological similarities converges in these languages therefore feeding the network with Kernel PCA embeddings, gives a boost to the network enabling it to learn meaningful word vectors even faster than the non morphological languages.</p><p>We also observe that it is possible to obtain fair quality word vectors using Kernel PCA skip gram model in just 1 epoch of training even on a smaller data set. Since skip gram model does not perform well when trained on a relatively smaller data set, our approach can be used in situations where we don’t have a huge data set. One such case arises when we have a very new data set such as NEWS articles. Because the current news events changes very often, it is difficult to get good word embeddings for new words which get associated with some context. For example: When Donald Trump won the presidential elections, the newly trained word embeddings for the word ‘president’ were still lie closer to ‘Obama’ and not close to ‘Trump’. This is because the skip gram model has not seen enough examples where word ‘president’ is used in context of ‘trump’. In such situations, our approach can do better as we can train on smaller datasets.</p><p>For more details, please visit:</p><p><a href="https://github.com/vishwgupta/MastersThesis-Improving-word-embeddings-using-KPCA">vishwgupta/MastersThesis-Improving-word-embeddings-using-KPCA</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a6a0cb7aad49" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>