<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Devansh Trivedi on Medium]]></title>
        <description><![CDATA[Stories by Devansh Trivedi on Medium]]></description>
        <link>https://medium.com/@D3V4N5H?source=rss-e53210527945------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*6US5znDCjdI_zBTLJD9SUg.jpeg</url>
            <title>Stories by Devansh Trivedi on Medium</title>
            <link>https://medium.com/@D3V4N5H?source=rss-e53210527945------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 15 Nov 2026 03:33:26 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@D3V4N5H/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[ML Day 42–48 : Week 7]]></title>
            <link>https://medium.com/@D3V4N5H/ml-day-42-48-week-7-f6c7f5382e62?source=rss-e53210527945------2</link>
            <guid isPermaLink="false">https://medium.com/p/f6c7f5382e62</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[sentiment-analysis]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Devansh Trivedi]]></dc:creator>
            <pubDate>Tue, 06 Nov 2018 19:37:13 GMT</pubDate>
            <atom:updated>2018-11-06T20:10:06.277Z</atom:updated>
            <content:encoded><![CDATA[<h3>ML Day 42–48 : Week 7</h3><h4>Sentiment Analysis 🙇🏻‍♂️🥴</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/0*kCCt18G0Im6hBY57" /></figure><p>Graph based Mining was fancy, but on hiatus for now, until I figure out how to import from CSV without that NULL error</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/617/1*-n3ra73F_3Z7XNWG-0AVyw.png" /><figcaption>From <a href="https://www.quora.com/How-does-sentiment-analysis-work-generally">Q</a>uora</figcaption></figure><p><a href="https://www.quora.com/How-does-sentiment-analysis-work-generally">How does sentiment analysis work, generally?</a></p><h3>TextBlob</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/590/0*pnW9sfgtR25S_EiG.png" /></figure><p><em>TextBlob</em> is a textual data processing library for diving into common Natural Language Processing (NLP) tasks such as part-of-speech tagging, noun phrase extraction, sentiment analysis, classification, translation, and more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1lEgBtyk_dhm3BY-8LmPVg.jpeg" /><figcaption>Error encountered during installation</figcaption></figure><p>This was funny because PUNKT is a Sentence Tokenizer</p><ol><li>Pankti in Hindi actually means a sentence (literally ‘line’)!</li><li>Punk as in ‘Daft Punk’ 🧛‍♂️</li><li>A friend’s name is Pankti and she was surprised to find out about this</li></ol><h4>SaaS (Sarcasm as a Sentiment):</h4><blockquote>“I am so happy that not one thing happened good today”</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nsTJyhpNMaPl3Ppl14OYmg.jpeg" /></figure><blockquote>The polarity ranges from : (-1 to 1)</blockquote><blockquote>-1 = Totally Sad</blockquote><blockquote>0 = Neutral</blockquote><blockquote>1 = Totally Happy</blockquote><p>Here, the result is 75% confident that it’s a happy statement. 🤦🏻‍♂️</p><h3>Part Of Speech (POS) Tagging</h3><p>Here are some Tag Abbreviations I found from the NLTK documentation:</p><h4>Basic:</h4><ul><li>N Noun</li><li>V Verb</li><li>ADJ Adjective</li><li>ADV Adverb</li><li>P Preposition</li><li>CON Conjunction</li><li>PRO Pronoun</li><li>INT Interjection</li></ul><h4>Advanced</h4><ul><li>CC coordinating conjunction</li><li>RB adverbs</li><li>IN preposition</li><li>NN noun</li><li>JJ adjective</li><li>VBP present tense verb</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XSIyXUdpWJzAOsR_GSvJIQ.png" /><figcaption>Tag = ‘VB’ and ‘NN’</figcaption></figure><p>I made up the sentence above to demonstrate that the same word can be a Verb and a Noun in the same sentence !</p><p>✅ Test NLTK Part Of Speech (POS) tagging</p><h3>Polarity — Positive vs. Negative</h3><p>Here are some words I tested</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/550/1*TKglfp2111PSGZk-q_HIPQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/958/1*qAIoFhTSVzdDe57QzAPHEA.png" /></figure><p>✅ Sentiment Analysis using TextBlob</p><h3>Inference</h3><blockquote>Positive = Happy</blockquote><blockquote>Negative = Sad</blockquote><p>The script didn’t execute the next day as in some cases MusixMatch didn’t return the lyrics (I’m assuming they didn’t have it yet). In which case it sends an empty list instead of dictionary. So that was breaking my script (even though it worked fine before, as it had lyrics for all top songs in india of that day’s charts). So I had to put a condition to make sure the script doesn’t fail.</p><p>✅ Fixed script breaking when <a href="https://medium.com/u/a64c8306bbf">Musixmatch</a> doesn’t have lyrics</p><p>I multiplied the scores with 100 to get percentage:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/679/1*jIRCDr2YImoF3M8gNoGeLw.png" /></figure><p>✅ Sentiment polarity to Percentage</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/668/1*5PproAOKvtM8LT9LaR_m4Q.png" /></figure><p>There were times when the polarity was 0, meaning ‘Neutral’. I think that information is not helpful… I’m not showing lines and songs that don’t yield a sentiment</p><p>✅ Only showing lines with non-zero sentiment</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/822/1*cAQbQcdb4vkiWC2Ypr6XSA.png" /></figure><p>Then I removed the negative sign from sad ones:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/978/1*c8e9fI7DqXM3D_B_l7libQ.png" /></figure><p>✅ Sad percentage value should be absolute, not negative</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/826/1*zoXp4dJTF5ylNVFm2yEoLg.png" /></figure><h3>Language Detection</h3><p>I was surprised by how accurately I could detect the language (using Google Translate).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0TG5l3AihbWhoe7ROjtAsg.png" /><figcaption>“hi” = Hindi</figcaption></figure><p>The song ‘Putt Jatt Da’ is actually in Punjabi, but I’ll let this one slide for now.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uZC0d1v89sbeacwH5uvf5g.png" /><figcaption>“es” = Español = Spanish</figcaption></figure><p>‘Taki Taki’ is Spanish. That’s right.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wX8SP8NH6N_Mkn4wXAqvkQ.png" /><figcaption>‘en’ = English</figcaption></figure><p>Believer is in English. So that is correct.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iZxi2QSapj1vKdUVtKw5VQ.png" /><figcaption>“gu” = Gujarati</figcaption></figure><p>I’m a Gujarati (demonym of Gujarat, India). I know Gujarati. And that up there, is not Gujarati. Proper Patola is a Punjabi song. Google Translate is still dodgy with Punjabi.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KLiW9NW5yZgSuldP3f2ghw.png" /></figure><p>This is my favourite example. ‘Kamariya’ is actually a Garba song (traditional dance of Gujarat). But this song, is Hindi film adaptation. So it has both: Gujarati and Hindi. So detecting it as either is good enough; I’m happy it detected it as Gujarati. I’m impressed that it detected Gujarati <strong>inside</strong> a Hindi song.</p><p>✅ Test Language Detection</p><h3>Comparing with Labels from API:</h3><p>I recalled that I am getting the language code from the API for each track already.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y7iu76fC6Kp0u6xL9GQyLw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1012/1*9j0xg446hdz7vK29FnDhCQ.png" /></figure><h3>Hybrid approach for Language Detection:</h3><p>I’m now verifying the same with the Language Detection and only Analyzing songs that are in English according to both methods.</p><p>✅ Restrict mining to english by detecting the language and verifying the detection with labeled data from API</p><p>I will now work on finding all the ways to compare environmental factors with lyrics in English language.</p><h3>Good News: Shout out to Andrew <a href="https://medium.com/u/27c3e4437cc4">Trask</a></h3><p>Andrew Trask, my <strong>role model</strong>, liked my tweet ! Hurray !</p><h3>Devansh Trivedi on Twitter</h3><p>100DaysOfMLCode #100DaysOfCode #100DaysOfX Day 46: ✅ Restrict mining to English by detecting the language and verifying the detection with labeled data from API https://t.co/Io3EQbMUi6 https://t.co/iEU6Dr7xBE https://t.co/1PpmSelZWr #PoweredByMusixMatch</p><p>Sir Trask is building <a href="https://openmined.org/">OpenMined</a> — a platform that merges cryptographic techniques such as homomorphic encryption and multi-party computation and blockchain technology to create the ability to train ML models with private user data. OpenMined will allow AI companies of the future to develop models, have them trained on user data without compromising user privacy, and incentivise users to train their model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AnBay4Q9y_o3U5MJFSNIlw.png" /></figure><p>He is a PhD student at Oxford University where he researches Deep Learning approaches with special emphasis on human language. Get to know him better in this video:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FqJ1rdVEcl5g%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DqJ1rdVEcl5g&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FqJ1rdVEcl5g%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/e4a42a806a72ccbcd03df13845552b8d/href">https://medium.com/media/e4a42a806a72ccbcd03df13845552b8d/href</a></iframe><h3>Sentiment Polarity Comparison:</h3><p>TextBlob contains two sentiment analysis implementations:</p><ol><li>PatternAnalyzer (based on the <a href="https://www.clips.uantwerpen.be/pattern">pattern</a> library)</li><li>NaiveBayesAnalyzer (an <a href="http://nltk.org/">NLTK</a> classifier trained on a movie reviews corpus)</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oqkl_URmHnXOmpeZat7JFw.png" /></figure><p>✅ Naive Bayes Tokenization and comparison with Pattern Analyzer</p><p>I would tolerate percentage difference, but in some cases the polarity is totally opposite. I will need to listen these songs, read lyrics and manually verify if the sentiments are consistent with the algorithms (which can be subjective).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i0jb47Ae-ENH0oeT7eulOQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/1*QVu6CuDaOdfXAf2BWE2ybw.png" /></figure><p>Types of combinations:</p><ol><li>NaiveBayes: +ve &amp; Pattern: +ve</li><li>NaiveBayes: -ve &amp; Pattern: -ve</li><li>NaiveBayes: +ve, Pattern: -ve</li><li>NaiveBayes: -ve, Pattern +ve</li></ol><h3>Using IBM Watson and BlueMix Console</h3><h4>Alchemy is now Watson Natural Language Understanding</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1018/1*YYmgJ6oVtMs7sWV0bhQH7A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*GwjzeFnmeaYTerrCKuTp9g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/401/1*NvalDIPQ8s7zn1IMB4hrQQ.png" /></figure><p>✅ Use @IBM Watson from BlueMix Console for Natural Language Understanding of ‘Entities and Keywords’</p><p>I will need more time to go through the documentation for taking full advantage of Watson’s services.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/327/1*5QlSFuvCwJQ90CMYOUF83w.png" /></figure><p>I did manage to get some keywords and entities from <a href="https://github.com/D3V4N5H/StarBoy/blob/master/starboy.csv">Starboy</a> song lyrics using the new API. However, there’s a lot left to learn and integrate with the current script, so IBM code is in a separate script for now.</p><h3>Code:</h3><ul><li><a href="https://github.com/D3V4N5H/StarBoy/blob/master/IBM%20NLU.py">D3V4N5H/StarBoy</a></li><li><a href="https://github.com/D3V4N5H/StarBoy/blob/master/Song%20Sentiment%20Analysis.py">D3V4N5H/StarBoy</a></li></ul><h3>More:</h3><p><a href="https://github.com/D3V4N5H/100DaysOfMLcode/blob/master/log.md">D3V4N5H/100DaysOfMLcode</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f6c7f5382e62" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ML Day 35–41 : Week 6]]></title>
            <link>https://medium.com/@D3V4N5H/ml-day-35-41-week-6-22b693b01de5?source=rss-e53210527945------2</link>
            <guid isPermaLink="false">https://medium.com/p/22b693b01de5</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[neo4j]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Devansh Trivedi]]></dc:creator>
            <pubDate>Mon, 29 Oct 2018 11:50:40 GMT</pubDate>
            <atom:updated>2018-10-29T13:05:28.787Z</atom:updated>
            <content:encoded><![CDATA[<h3>ML Day 35–41 : Week 6</h3><h4>New, Frontiers, Ahead, !</h4><h3>CSV (Cockroach-like Standard Vernacular)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/0*VhdDWJtZfJtcYtyF" /></figure><p>After my struggle to circumvent storage of data into files, the original query <em>requires</em> fetching data line-by-line from a <a href="https://tools.ietf.org/html/rfc4180">CSV</a> (Comma, Separated, Values) file</p><p>I compare it with cockroaches because it existed before me (like they existed before humans).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EpzIqA581f703_sDLv5Aig.png" /></figure><p><strong>This is epic</strong>. I’m making AI that will handle the data in a <em>way</em> that PRE-DATES the way my computer is made. Like using ANCIENT tools to build something modern.</p><blockquote>Come what may, the AI will do whatever it takes !</blockquote><blockquote>✅ Exported lyrics into a CSV</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/901/1*1hFk4aUQjMY0oJs18anS8g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5mTnXAFdc1UPk_4RzKz94A.png" /></figure><p>To quote my last week’s article,</p><blockquote>This is because of the way I’m fundamentally handling the lyrics from API</blockquote><p>Fortunately, I’m a Data Scientist. Computer Engineers are wizards. They practice magic for a living:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*r2M4HhHx-v3WHsXM.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/1*ryVSgZYp1C5XHeXaVYODEQ.png" /></figure><p>That’s not quite it. Let me use the other wand…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*N2F6lQ8x97hVILAp.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/478/1*tIC_1D_95NDXXmxnxKcsHA.png" /></figure><p>Wait, did you see those BLANK lines? We need to nitpick the non-empty lines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/240/0*P0n83lm1dqs3wiDP.gif" /></figure><blockquote>✅ CSV export is tip top, no blank lines now</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/478/1*smhpeZ2nuDdK-oiPSjQxVg.png" /></figure><p>OMG! Those ‘(Ha-ha-ha-ha)’ are still there! We don’t want our baby-AI to be confused. We know it doesn’t mean anything. So I’ll get those out of the way…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/478/1*6NV1KqTBBD2ELeYOpr9HJg.png" /></figure><blockquote>✅ Improved Lyrics cleaning using loops</blockquote><blockquote>Yep, now <em>that’s</em> what I’m talking about. Good job, Dev (Pun intended)</blockquote><p>But I had to manually copy it to the ‘import’ directory of the project’s graph to create nodes</p><h4>Before Removing Empty Lines</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2h5UfUhdhjo8hd4wUK811w.png" /></figure><h4>After Removing Empty lines</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mdadN86I57VGTku0R6APPA.png" /></figure><h3>Wherever you go, there you are !</h3><pre>save_path = &#39;/Users/d3v4n5h/Library/Application Support/Neo4j Desktop/Application/neo4jDatabases/database-25eef966-4d97-4421-9310-903f03421892/installation-3.4.6/import/&#39;</pre><blockquote>✅ Export Starboy lyrics CSV directly to the path of graph import directory</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/821/1*Xbk3tEYJfXDr4-i8VmW6fw.png" /></figure><h3>Using the Devil you know (CSV)</h3><p>Mind that CSV is <em>awesome</em>, it has never done anything wrong unto anyone (or any <em>thing</em>). I’m just thwarted because I don’t see a way to circumvent it’s NEED by modifying the CYPHER query.</p><blockquote>Later, I used this file to Create Nodes and mine Paradigmatic Associations using Jaccard Index.</blockquote><iframe src="https://cdn.embedly.com/widgets/media.html?url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DUsSmYu9npkU&amp;src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FUsSmYu9npkU&amp;type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/4afb5b2932dcd8887eab0ea885fc3b91/href">https://medium.com/media/4afb5b2932dcd8887eab0ea885fc3b91/href</a></iframe><blockquote>✅ Mine the lyrics imported from CSV<br>✅ Store the result in a Python dictionary</blockquote><h3>Voids in the Valves of the Venture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/1*QoKd8k4isGH3E7AWbT6lpQ.png" /></figure><p>Found ‘0’ values for words… My guess? 🤔 It’s the first words of some lines.<br>I checked the CSV file, and it didn’t have any ‘0’ in it !</p><p>I will investigate this matter if and when it turns out to be an issue. Don’t get the wrong idea, I’m not being sloppy giving the AI any leeway to give notorious recommendations.</p><h3>Scaling the Success</h3><p>Let’s do this for Top Tracks in india !</p><p>Approach:</p><pre>create_Nodes(song_Name) → <br>graph.create_Nodes(tx, song_Name) → <br>create_Nodes_Query_Fn(song_Name)</pre><p>Here, tx = Transaction</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NBhiHMgz9sGNCklRFy8PFg.png" /></figure><p>My code is right, but the Database throws Illegal Character error for songs that have SPACE in their name.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4EXtXL7Qmz520DRo15-GUw.png" /><figcaption>Google didn’t help</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*wq5K6ECND7GP-N0N7-zI5A.png" /><figcaption>Without Encoding</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/699/1*FwQ1A5yB-0Bv8HVSwCHUrg.png" /><figcaption>With Encoding</figcaption></figure><p>It turns out, Encoding didn’t help either, I just had to strip the SPACE characters out of the song name.</p><blockquote>✅ Fix file encoding issue</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IRPnxHSL6QmgVI5zVZS7OQ.png" /><figcaption>This TakiTaki.csv was “Taki Taki.csv” before</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_YjZavjhb3ApCgIIjD8OcA.png" /></figure><p>I removed the last empty line in the CSV manually and checked for other NULL values with Python. But the error <em>persists…</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/637/1*i8F_CuZYQMky2MPerFgCZA.png" /><figcaption>I found that the last line was empty and removed it</figcaption></figure><p>Starboy lyrics had null and still worked fine</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/709/1*ZrrKW9Zsi508NWPicTwUcw.png" /></figure><blockquote>And so now I suspect that “the Query is the Culprit” here</blockquote><h3>Code:</h3><ul><li><a href="https://github.com/D3V4N5H/StarBoy/commits/master/StarBoy.py">D3V4N5H/StarBoy</a></li></ul><h3>More:</h3><h3>Devansh&#39;s #100DaysOfMLcode</h3><p>The Making of Flubber Weebo for songs!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=22b693b01de5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ML Day 28–34 : Week 5]]></title>
            <link>https://medium.com/@D3V4N5H/ml-day-28-34-week-5-79f9dca5e092?source=rss-e53210527945------2</link>
            <guid isPermaLink="false">https://medium.com/p/79f9dca5e092</guid>
            <category><![CDATA[neo4j]]></category>
            <dc:creator><![CDATA[Devansh Trivedi]]></dc:creator>
            <pubDate>Mon, 22 Oct 2018 14:46:05 GMT</pubDate>
            <atom:updated>2018-10-22T14:59:50.209Z</atom:updated>
            <content:encoded><![CDATA[<h3>ML Day 28–34 : Week 5</h3><h4>Engineering Repercussions</h4><h3>Giving Camels some Comfort in the Crowd</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/0*o3ehiDX7byrX5UWs.png" /></figure><p>I underscored variable names for readability</p><blockquote>They were camelCase before</blockquote><blockquote>now_They_Are_Both !</blockquote><h3>Lyrics Gone MAD</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*TgqYxp-omhR25LiJ.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VPcnCzfVwEPICP27C1rK2g.png" /></figure><p>Array in a string?</p><p>‘[0] =&gt;’ ?</p><p>This all looks weird. This isn’t what I was promised when I signed up for the APIs. I’m shocked yet amazed cuz it kinda looks cool!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/245/1*Cpvfd71tD2WuL4k_muAlKQ.gif" /><figcaption>Another Gif from Taylor Swift’s “Blank Space”</figcaption></figure><h3>De-Duplication of Nodes</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*2jx21x31l5OqPy0JpPb8cg.gif" /><figcaption>From SpyKids</figcaption></figure><p>I implemented query and calls with a different approach to prevent duplicate nodes before Mining</p><blockquote>This “different approach” is using negative indexing and slicing to convert the lyrics into two separate list of words; yielding perfect word pairs</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3bZMaqhnW22PP3vXpVQO0Q.png" /></figure><p>Finally it works as intended, can’t wait to fetch and display keywords mined from each song now !</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aBhV-_WN2B_p5m_AbCx6JA.png" /><figcaption>Graph of Nodes from Chogada song Lyrics</figcaption></figure><h3>Ghosts of Innovation</h3><p>Importing weighted keywords from mined graph. The script is taking significant time to execute, which is kinda cool, I can take a walk, drink some water and get fresh air every time I test my script !</p><p>But Alas! I can’t “HAVE IT”. Trying different approaches to read Response code from the BoltStatementResult object. I really wish I could just iterate on it 😤</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/0*70LM8IOk7JPZ6O-x.jpg" /><figcaption>Ghosts of Girlfriends Past (2009)</figcaption></figure><p>Ok, I got it. But then the ALL of the weights were 1. This means my AI is very unbiased ☯️☯ and doesn’t understand the first thing about English language🤯</p><p>This is not Mined, this is MindLess 🤪</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KBw5VSwd225s-_VdTVxORA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HzKFn9CSGzjmwp_TlleJ8Q.png" /></figure><h4>Silver Lining: New Top Tracks!</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*6D6URRKmRMyCXBDo9WFfLQ.png" /></figure><p>SpiderMan Sunflower by Post Malone and Swae Lee is here! Of course!</p><h3>Overt Overhaul</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_ly1SNT8NPM9Q0XuFRA8Tg.png" /></figure><p>I’m DONE. I tried modifying the CYPHER query to fit the data. Now I’ll modify the input for the query to fit the original query. 😑</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/0*m2-bYo_XgG9Sz7GP.jpg" /></figure><p>Forget the whole script. I want to make this happen for Starboy song first. Will loop the whole process when I finish mining this one as expected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/799/0*DelsBCugactbnFDh.jpg" /></figure><p>Look at the CSV I manually made at the initial stage of the project:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/478/1*Aip7fg2F4BzchGua1ixCwA.png" /><figcaption>This is how I need to make the data</figcaption></figure><p>After several minutes of coding, I got to:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/478/1*Gdhz3O0C09tWvBOLEfHn3w.png" /></figure><p>And</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/478/1*cc5YMGwpu3s684yudDcLhg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XDrvsXYmYyNJKIbjdmSKIw.png" /></figure><p>This is because of the way I’m fundamentally handling the lyrics from API. I don’t get separate lines, but perhaps I can use ‘\n’ as delimiter to split lines into list of line elements.</p><h3>Code and Notes:</h3><ul><li><a href="https://github.com/D3V4N5H/100DaysOfMLcode/blob/master/log.md#day-28-october-16-2018">D3V4N5H/100DaysOfMLcode</a></li><li><a href="https://github.com/D3V4N5H/StarBoy/commits/master/StarBoy.py">D3V4N5H/StarBoy</a></li><li><a href="https://github.com/D3V4N5H/StarBoy/commits/master/Mining%20Top%20Tracks%20in%20india.py">D3V4N5H/StarBoy</a></li></ul><h3>MORE :</h3><h3>Devansh&#39;s #100DaysOfMLcode</h3><p>The Making of Flubber Weebo for songs!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=79f9dca5e092" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ML Day 21–27 : Week 4]]></title>
            <link>https://medium.com/@D3V4N5H/ml-day-21-27-week-4-ae8d39df7d96?source=rss-e53210527945------2</link>
            <guid isPermaLink="false">https://medium.com/p/ae8d39df7d96</guid>
            <category><![CDATA[100daysofmlcode]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[neo4j]]></category>
            <dc:creator><![CDATA[Devansh Trivedi]]></dc:creator>
            <pubDate>Fri, 19 Oct 2018 14:16:53 GMT</pubDate>
            <atom:updated>2018-10-19T14:16:53.334Z</atom:updated>
            <content:encoded><![CDATA[<h3>ML Day 21–27 : Week 4</h3><h4>Failure and Support</h4><h3>Unhealthy Boilerplate</h3><p>I wanted to import the words from the JSON, but the queries I worked on before, used CSV file. Luckily, I had parsed the data into a neat dictionary. But now I needed to pass lyrics from it to the Graph.</p><p>At first, this seemed easy. But later I noticed it didn’t work. I even tried to code another query to pass word pairs instead of whole lines. For this, I needed to iterate TWO value in a list at the same time.</p><p>So I tried to find a way to do that, but that didn’t do quite as expected:</p><p><a href="https://stackoverflow.com/questions/16789776/iterating-over-two-values-of-a-list-at-a-time-in-python">iterating over two values of a list at a time in python</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RviphAsgEvXUglkxHIfPPQ.png" /><figcaption>At first sight, this looks right</figcaption></figure><blockquote>It had to take the second word again next time!</blockquote><h4>So I implemented it the old school way:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AslOeIDHTxKTJqpZ-9ga0Q.png" /><figcaption>Yes, now that’s more like it!</figcaption></figure><p>For the first time, I couldn’t do something, despite my 4 Hours of Hard Work. I felt disappointed and stuck !!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TNo5tcqd_kzhHf-NNZm6_Q.png" /></figure><p>But thanks to <a href="https://medium.com/u/3865848842f9">Michael Hunger</a>, we got this beautiful graph!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/542/1*D-zOLDjnUye4dYl0HQxxPw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*erx46Md6cKcMMiBmk4aL1A.png" /></figure><p>As I was wondering why the word frequency from the API fetched lyrics didn’t match that of the CSV lyrics, it was comforting to recollect that I don’t get FULL lyrics from the free account. Hence the difference.</p><p>Despite that, my mining was very accurate if you compare the following to the <a href="http://sync.dvnsh.com/mlday5">mining from original FULL lyrics</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2z8ie97_76UdkNE5jtN4rw.png" /></figure><h3>Bypassing Label limitations using APOC</h3><p>Currently, we’re only able to see numbers on nodes. I’d like to know</p><ol><li>What word that node represents,</li><li>Which song is that cluster of nodes for !</li></ol><p>Adding a label with the word as value was easy:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C3co359pY_qWF_StT6Q2OQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/841/1*2EZnNfbQYxY9M7uRc35dbQ.png" /></figure><p>But Song Names often had multiple words, which at the time seems to be a technical limitation of Cypher.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h29V40TwABc9VvOCr6OW6w.png" /><figcaption>Read: “Invalid input ‘L’. Expected whitespace, comment, NodeLabel, MapLiteral, a parameter, ‘)’ or a relationship pattern”</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CIC318-ixrUtv3X1ZcXOZg.png" /><figcaption>And so I couldn’t PASS it as a Label</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f7oUcqJr7aa8oLl-6yRjOA.png" /><figcaption>It simply raised an Exception!</figcaption></figure><h3>Another Day, Another Angel</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*BevnChbjKycACriAQS8w7g.png" /></figure><p><a href="https://markhneedham.com/">Mark Needham</a>, Developer Relations Engineer at <a href="https://medium.com/u/26bec2028463">Neo4j</a>, suggested using something interesting:</p><blockquote>APOC: Awesome Procedures On Cypher</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/208/0*UTy3q0AXWCSi0zBM.gif" /></figure><p>There’s actually a fun <a href="https://github.com/neo4j-contrib/neo4j-apoc-procedures#apoc-name-history">story</a> about the acronym. Apoc was the technician and driver in the Matrix movie !</p><p>The APOC library consists of about 450 procedures and functions to help with many different tasks in areas like data integration, graph algorithms or data conversion.</p><p>And that helped!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MCy-EENusz531dRODAHU_Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hNHWb_HMq7CjN_C5rQBi4w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3cNQ84VZsyu8BX5jtLMoEg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/446/1*4y_w-9-DGRrpyxiYHU8Myw.png" /></figure><p>So I made a loop to do this for all the Top Tracks in India:</p><ol><li>Get name of Each Song</li><li>Get lyrics of that song</li><li>Clean the lyrics</li><li>Create Node Pairs</li><li>Log when all the words of song’s lyrics are merged into nodes of the graph</li><li>Mine only those nodes in the graph</li><li>Log that the song was mined</li><li>Get weighted keywords from the graph</li><li>Change Label to Song Name using APOC</li></ol><p>Labelling songs by their names allowed me to mine each song individually, as opposed to a graph where nodes from words of all songs were pooled.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n-GyAV1pyetoSLUNv7taVg.png" /></figure><h3>When AI is drunk, and drives under influence</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hgB4cffBAt3WHiAHAQqHpw.gif" /><figcaption>Do Dikhte hai = Do you see double?</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ey6bxvuGBSmbRBCE6xx7Cw.png" /><figcaption>Notice Duplicate Nodes</figcaption></figure><p>Even though I added UNIQUE keyword in the query, at least two nodes were created for each word, as the query was run for the next word pair (see 2nd picture in the article for reference).</p><h4>Interesting note:</h4><blockquote>Most weighted pairs are the same words!</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iRRrmQ2AKtapT70o_2DlbQ.png" /></figure><h3>What I tried but didn’t work:</h3><ol><li>Defining the query AGAIN at every iteration in the loop, concatenating each word pair into it (hard coding it explicitly)</li><li>Passing lines instead of words with Field Terminator as a single horizontal white space character</li><li>Using Unwind to access multiple neighbouring values in the line (it seems I can only access and iterate over one element)</li></ol><h3>Code</h3><p><a href="https://github.com/D3V4N5H/StarBoy/commits/master/Mining%20Top%20Tracks%20in%20india.py">D3V4N5H/StarBoy</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ae8d39df7d96" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ML Day 14–20 : Week 3]]></title>
            <link>https://medium.com/@D3V4N5H/ml-day-14-20-week-3-e8880bec9aa8?source=rss-e53210527945------2</link>
            <guid isPermaLink="false">https://medium.com/p/e8880bec9aa8</guid>
            <category><![CDATA[music]]></category>
            <dc:creator><![CDATA[Devansh Trivedi]]></dc:creator>
            <pubDate>Mon, 08 Oct 2018 14:21:40 GMT</pubDate>
            <atom:updated>2018-10-08T14:32:21.039Z</atom:updated>
            <content:encoded><![CDATA[<h3>ML Day 14–20 : Week 3</h3><h4>Making it all work: RECONnaissance</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AbiK2HI6sdRElcGs0c6QgA.png" /><figcaption>These are the TOP songs and artists in India</figcaption></figure><p>Then I went a step ahead and found the top songs OF top artists. That’s sooo… META!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WO6vMykxfAWjzhdzB5LXRw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6pk1hkfAwAYWdilXnPn6iQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*7AfiIIzTjtxr0p39Fbsgpw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/524/1*pUgGJE5LnAFZ0ZKUFbUx3g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VZN_DoBEMTRwvWwq4f3AYQ.png" /></figure><h3>Getting Release Date METADATA</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m4MfwybchwjeRx3Nj4tVlA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P8e7vLxdMwa58wwb_0LJqg.png" /></figure><p>After parsing the Dates and Formatting:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sjSzCoNDNP23yHxa8YAPSA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dQIPMSOMDwIzLtO9Hizr3A.png" /></figure><h4>Breaking news</h4><p>If you sing Honey Singh songs, your choice is not so bad as a lot of Indian Music Users prefer it too. He’s in the top artists!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZqlgK8s-WehciK_WhzCcIQ.png" /></figure><h3>Reducing the time of execution</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/817/1*w5RV92Ki50n77Og0wXucMw.png" /></figure><p>This META loop was taking too long for the script to give output.</p><blockquote>I found the data in previous calls</blockquote><p>So I removed the function I was using to get Release Date of Each Track of Each Popular Artist, and coded a function to iterate over data from existing calls.</p><p>Now I will be able to find out the Trending Tracks FASTER!!!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/711/1*bCG2Y5nXbSAE4DJZECkBsA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/542/1*Nv8oF8kXKm8pCDjf4gJ44w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/566/1*fVtDSt26CsMIZh2_YZKKQg.png" /></figure><h3>UPS and DOWNS</h3><p>(That was not a pun). Using driver is a pain when debugging.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1W7JxZKQDYnZxf5s6-hWAA.png" /></figure><h3>Getting the Lyrics</h3><p>I fetched lyrics of Top Songs in <a href="https://twitter.com/hashtag/India?src=hash">#India</a> songs to train the AI</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*azOxppgyZRv6cIAPp2M31A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YmbV7hsw2y6TCEJ37SL3AA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Il-u8Fi90mgxNvvatnZf5A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*roO6g9PFvmwDpXgBLXpObA.png" /></figure><h4>Then I fixed it to read clearly and parse it</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-ldK1gYC-kL9PfEvs00_MA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/919/1*OuNbnIeriOE0qCbiyNsqyg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-5KE-hqZV2qS1lC9RF-DCg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K3Zs5oMI8yeiIsEwI1nsng.png" /></figure><p>For AI, I will need to clean this and process further for WHITESPACES like empty lines and “…” and the FREE API notice before I generate keywords.</p><h3>I DID IT!</h3><h4>Getting lyrics is now officially automated !!</h4><blockquote>Genius.com <a href="https://genius.com/3317959">resources</a> helped understand parts of the lyrics</blockquote><p>The data in my file is analogous to the CSV file that I manually generated at the beginning of the project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Nl5SwRtZZos8j55l58u-tA.png" /></figure><p>Then I realised that the background vocal music in the brackets is not really contributing in the teaching AI what the lyrics means</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UwfVkcm1dWyEOsefyo9cAg.png" /><figcaption>Notice the (ha-ha-ha)</figcaption></figure><p>So I got rid of it from the lyrics and also removed commas ( , ).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9pYMgmFe04A8ujZsnhWvdQ.png" /><figcaption>I like this. do you?</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e8880bec9aa8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ML Day 7–13 : Week 2]]></title>
            <link>https://medium.com/@D3V4N5H/ml-day-7-13-week-2-f908407eeac5?source=rss-e53210527945------2</link>
            <guid isPermaLink="false">https://medium.com/p/f908407eeac5</guid>
            <category><![CDATA[music]]></category>
            <dc:creator><![CDATA[Devansh Trivedi]]></dc:creator>
            <pubDate>Mon, 01 Oct 2018 20:21:46 GMT</pubDate>
            <atom:updated>2018-10-01T20:21:46.457Z</atom:updated>
            <content:encoded><![CDATA[<h4>API data → Song Attribute Graph</h4><p>By now I have managed to summarise the song. I can find ‘key’ words from a song, along with their weights.</p><blockquote>But understanding the lyrics is not how we choose our songs, do we? Sometimes, we just like the artist; or the genre (like Punjabi Beats), or the topic (like patriotic songs); or its timing (like songs from your childhood).</blockquote><p>Lo and Behold the :</p><blockquote>“Song Attribute Graph Data Model”</blockquote><p>I’m advocating content based recommendations at this stage of the project. I had to guess what parameters can be taken into account to find patterns that match the environment. To keep my project simple, I have chosen Release Date, Featuring Actor, Genre AND Keywords.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rj9esfnK_vqkXxL0u73LPw.png" /><figcaption>How I expect the AI to know about the song</figcaption></figure><p>Remember, I’m attempting Content-based Recommendation first. That means tracking the user’s preferences over time. This will work better over time. It grows on you, just like Music!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*9vcneTjII_mY4wNu0zCLkA.png" /><figcaption>Now that we have both, let the fun begin!</figcaption></figure><h3>Bring the Data over here!</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bFYFjD5yjPx8A6bldR134A.png" /><figcaption>#PoweredByMusixMatch</figcaption></figure><p>It took long enough to get ahold of these, but it was worth the wait!</p><h3>What I CAN do:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i1J6T8AcBochnY8bmV0yGw.png" /></figure><p>I’m planning to use the following to compute relevance:<br>chart.tracks.get<br>chart.artists.get<br>matcher.track.get<br>track.get<br>artist.albums.get<br>matcher.lyrics.get<br>track.lyrics.get</p><h3>What I CAN NOT do:</h3><h4>I’d need a commercial plan for the following:</h4><blockquote>track.search</blockquote><blockquote>track.subtitle.get</blockquote><blockquote>matcher.subtitle.get</blockquote><h3>The PlayGround</h3><p>MusixMatch has an API Playground. Authenticate using a key to test responses.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dqhwk7JA3sC8T36codwXeQ.png" /></figure><p>track.snippet.get didn’t work for starboy track_id 144134659, not even in the playground 😭. “snippet_body” was empty! They don’t have it. 🤬</p><h4>Word to the Wise</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7NcBOnoT4RFaFSoNhRnc3w.png" /></figure><h3>Show me the CODE !!</h3><p>I know. Almost everyone who reads my articles wants to know what I did. Here’s how I did it:</p><blockquote>I’m leveraging local insights. You location plays a part in defining your environment.</blockquote><p>Here is the response for top artists in US:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gUPbPwJGpgL-7vzwYq4SFw.png" /><figcaption>…And that’s where I also find Genre</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*83Cl3ENWlvoZj5gxXa2TAA.png" /></figure><p>So top songs from one of these artists is a good idea for a US user:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xwWpF7hipOkpnCLx4R2T6A.png" /><figcaption>Fetching Top Artists in US</figcaption></figure><p>Side note: Look what I found</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/757/1*HjF329bKweU9ZRp-0mHNjQ.png" /><figcaption>Please Beware! You are warned!</figcaption></figure><h3>Subtitle of this Article</h3><p>Now I will combine both python files : The one that fetches data and the one that fires queries to generate nodes in the graph database to associate data.</p><p>All that fancy JSON is not going to understand itself. We need to add this to the ‘Song Attribute Graph’. So I need a function to do what I did: Add the nodes with the data. Like this query:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/410/1*U5wcgViDHmgEbqwRVGoDdg.png" /></figure><p>I wanted to create separate functions for queries to add user, add song, add metadata and link. But because of the limitation of query variables, their scope only exists with the query. So I had to create a common function. If I still wanted to create separate ones, I’d need to MATCH/MERGE them before making associations to a new variable, which would further complicate the code. I will have a workaround in Python to remedy this by creating query templates and loops.</p><p>So I did that for a bunch of users with just the Artist relationship.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rAAXVC-folonBX9xSufiLg.png" /><figcaption>Users 0, 1, 2, 3, 4, 5.</figcaption></figure><p>And then some.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kf7hjstpqoh4WfnHRzwPAA.png" /></figure><p>I divvied up the graph above into following upper and lower halves, so you can read the data inside.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OJaMF1R1GmLyrrbmwNOqNw.png" /><figcaption>Upper Half</figcaption></figure><p>…And</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xFJARGCsflCJUuY_Q3iw9g.png" /><figcaption>Lower Half</figcaption></figure><h3>No!! You’re not going to believe this!</h3><p>Today I met a Music Studio Owner 🎤 — 🎧, Sir R. K. Bhanushali. He was surprised to find all my response data had only English songs and none in Hindi.</p><p>The algorithm should</p><ol><li>Find which languages are prominent at user’s location and whether or not user knows / likes songs in that language</li><li>Find the code for that language to use in the API call</li><li>Add top tracks in each language to the Graph</li></ol><p>Luckily, I don’t need to do all that if I use MusixMatch charts</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AbiK2HI6sdRElcGs0c6QgA.png" /></figure><p>Now that I have the language agnostic recommendations, I will fetch their lyrics, snippets and metadata. I need more metadata, as I couldn’t get some info about tracks from MusixMatch API.</p><h3>Links to work:</h3><ul><li><a href="https://github.com/johnymontana/nlp-graph-notebooks/blob/master/Content%20Recommendation.ipynb">johnymontana/nlp-graph-notebooks</a></li><li><a href="https://github.com/D3V4N5H/StarBoy/commit/b48685dfa9f8a13574be17c33f666fc340185d8a">Create UserPreferenceAndSongAttributesGraph.cql · D3V4N5H/StarBoy@b48685d</a></li><li><a href="https://github.com/D3V4N5H/StarBoy/commit/2dd791e57beaf098c6879e8c50f3b6e831b53c70">Create User Preference and Song Attributes Graph · D3V4N5H/StarBoy@2dd791e</a></li><li><a href="https://github.com/D3V4N5H/StarBoy/blob/master/mxm%20chart.artists.get.py">D3V4N5H/StarBoy</a></li><li><a href="https://github.com/D3V4N5H/StarBoy/commit/61e9663bd47823fd453d3970cf08a3664f256065">Day 11: Names of Top Artists in India from JSON · D3V4N5H/StarBoy@61e9663</a></li><li><a href="https://github.com/D3V4N5H/StarBoy/blob/master/userArtistSongMap.py">D3V4N5H/StarBoy</a></li><li><a href="https://github.com/D3V4N5H/StarBoy/commit/2bbfafbfbf6a8d2d75d7d5c859c4ba35b9e0b9e8">Fixed Artists for India and Added Top Tracks · D3V4N5H/StarBoy@2bbfafb</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f908407eeac5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ML Day 6]]></title>
            <link>https://medium.com/@D3V4N5H/ml-day-6-9cc1a05bf6aa?source=rss-e53210527945------2</link>
            <guid isPermaLink="false">https://medium.com/p/9cc1a05bf6aa</guid>
            <category><![CDATA[javascript]]></category>
            <dc:creator><![CDATA[Devansh Trivedi]]></dc:creator>
            <pubDate>Mon, 24 Sep 2018 18:22:25 GMT</pubDate>
            <atom:updated>2018-09-24T18:22:25.196Z</atom:updated>
            <content:encoded><![CDATA[<h4>Graph based Summarization and Keyword Extraction</h4><p>To know <strong>what</strong> the song is all about, I need to summarize it. In graph-based summarization, I will explore a specific approach: <strong>Opinion mining.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*2wXAlHtxse-itT5rFrNGcQ.png" /></figure><p>Here the yellow node (smiley) is the largest and hence most important because of the number of other nodes that point to it in the graph</p><h3>TextRank</h3><p>This is like running PageRank on our Text Adjacency Graph. It has Numpy and Scipy dependencies:</p><p><a href="https://github.com/summanlp/textrank">https://github.com/summanlp/textrank</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/438/1*jkzxgSF64-qK1JGD1Q6juQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*g9pGzOYtXA0vxdO0bL3v1g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*8UD0ND0QllwWfGCf4hcxCw.png" /></figure><h4>Here’s how it went:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XtSJy5HRjMhaJ5F0oxf1cw.png" /><figcaption>Output from Full Lyrics</figcaption></figure><p>This is bad. Good, but bad…</p><p>It worked. But the song is ‘Starboy’ and that’s not in the summary. 🤔</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ag_ZkZenhVluoUo1LDvq4Q.png" /><figcaption>Keywords</figcaption></figure><p>Cool. ‘Starboy’ spotted. Better than before. But I will not settle for this.🤨</p><h3>My Query made it work !</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9USqBSOmZLKjuzeBLHRp8Q.png" /><figcaption>IT’S ALIVE! IT’S ALIVE!</figcaption></figure><p>The chorus had the most weight. Look at the first few lines! 🎉 🎊</p><h3>Links to work:</h3><ul><li><a href="https://github.com/D3V4N5H/StarBoy/blob/master/TextRank.py">D3V4N5H/StarBoy</a></li><li><a href="https://github.com/D3V4N5H/StarBoy/commit/97fc56baf9c71e07e20a2175a10f770b617efe9a">Graph based Summarization and Keyword Extraction · D3V4N5H/StarBoy@97fc56b</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9cc1a05bf6aa" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ML Day 5]]></title>
            <link>https://medium.com/@D3V4N5H/ml-day-5-579899732cf8?source=rss-e53210527945------2</link>
            <guid isPermaLink="false">https://medium.com/p/579899732cf8</guid>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Devansh Trivedi]]></dc:creator>
            <pubDate>Sun, 23 Sep 2018 18:02:46 GMT</pubDate>
            <atom:updated>2018-09-23T18:02:46.061Z</atom:updated>
            <content:encoded><![CDATA[<h4>Mining Paradigmatic Word Associations using Jaccard Index to compute similarity</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1007/1*GpVC3sWbmBwMda1-jeDfLQ.png" /><figcaption>Example in Neo4j video</figcaption></figure><p>Knowing preceding and succeeding words is not much helpful (yet), as it only projected an obvious fact, that the ‘articles’ are common (meaning frequent) in the text adjacency graph. I need a way to mathematically find context from the text corpus.</p><p>There are two common types of word associations in natural language processing:</p><p><strong><em>paradigmatic and syntagmatic</em></strong></p><p><strong>Paradigmatic</strong>: words A and B are paradigmatically related if they can be substituted for each other. This indicates they belong in the same class, such as “Monday” and “Thursday” or “Cat” and “Dog.</p><p><strong>Syntagmatic</strong>: words that can be combined with each other, such as “cold” and “weather”.</p><p>To compute a measure of paradigmatic relation we take an aggregation of the similarity of relative contexts:</p><blockquote>Sim(“Cat”, “Dog”) =</blockquote><blockquote>Sim(Left1(“Cat”), Left1(“Dog”))</blockquote><blockquote><strong><em>+</em></strong></blockquote><blockquote>Sim(Right1(“Cat”), Right1(“Dog”))</blockquote><h3>We can use the Jaccard index as a measure of similarity</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*xVmkFQAiIqsRsO_Bu2N1hQ.png" /><figcaption>Also called Intersection over Union</figcaption></figure><p>A revision to the set theory (very silly, I know. Today <a href="https://twitter.com/hashtag/AddTheWordSillyToAnything?src=tren">#AddTheWordSillyToAnything</a> is trending on Twitter 😜)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*5VQoQquK9HRMKm4zAa6i9g.png" /><figcaption>Intersection</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*Paiv0jZv11clx0nK4aej1A.png" /><figcaption>Over Union</figcaption></figure><blockquote>0: if no words in common<br>1: if the sets are equal</blockquote><h3>A visual Example — Object Detection</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*e7XbLr7A6HF18dcv9EDqpQ.jpeg" /><figcaption>Jaccard Distance</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*2LPQLE87SJBRCSXhpow9sA.png" /><figcaption>Visualising the Ratio</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*JoyOqNm42i90sM2Y7VWBYA.png" /><figcaption>Inference from values</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*i7ZjtVO3aU4pyW2EBeZkbw.png" /><figcaption>Similarity of asymmetric binary attributes</figcaption></figure><h3>The dilemma</h3><p>The whole point of using the Python driver was the ability to leverage the language. But in this case, the Cypher + Python approach won’t allow identifying the most paradigmatically similar words. For example, I can only ask “what is the paradigmatic similarity between words X, Y”. What we really want to ask is “what are the word pairs that are most paradigmatically similar?” For this I will use a pure Cypher approach to compare word pairs and update the graph with a :RELATED_TO relationship. This relationship will store the similarity score. Once this query runs we can query for word relations using this new :RELATED_TO edge.</p><p>The query is 21 lines long. It took 6.581 seconds to fire it. 14762 properties were set and same number of relationships were created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4lk5SCLAPua3P3OM4cG-CQ.png" /></figure><p>After all this hard work, my query says <strong>‘starboy’ is not related to anything</strong>, even though the song is titled “StarBoy”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4hgD5Tvzpb7d0hVkVb51DQ.png" /></figure><p>Not only that, <strong>nothing is related TO starboy</strong> either!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yMrGls27IEMNhDj-Qo6DOg.png" /></figure><p>However, the song is all about <strong>‘style’</strong>, it seems!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3mRSAvxGwVSe3UVOd1mxUw.png" /></figure><blockquote>This is kinda okay, because I don’t expect the AI to play this song when someone says “starboy”, as that event is highly unlikely.</blockquote><p>Perhaps, the words with most ZEROS can tell us what the song is NOT about. All the words in the phrase <strong>“cleaner than your church shoes”</strong> had many zeros, here’s their score:</p><blockquote>cleaner: 121 (all zeros, no other values)</blockquote><blockquote>than: 119<br>“of than” : 0.2<br>“out than”: 0.33</blockquote><blockquote>church: 119<br>“league church”: 0.5<br>“whole church”: 0.5</blockquote><blockquote>your: 118<br>“the your”: 0.1<br>“of your”: 0.125<br>“these your”: 0.2</blockquote><blockquote>shoes: 114<br>“me shoes”: 0.25<br>“too shoes”: 0.3333333333333333<br>“you shoes”: 0.3333333333333333<br>“mood shoes”: 0.5<br>“hefty shoes”: 0.5<br>“SV shoes”: 0.5<br>“cry shoes”: 0.5</blockquote><p>But we all knew that this song was not about “cleaner church shoes”. Now let’s see if 1.0 score can offer any interesting insights about it; something that can help the AI decide when to play the song, or whether or not it is relevant to the environment.</p><blockquote>such pairs with r.paradig == 1.0 are:<br>“end” “test”<br>“tease” “hurt”<br>“cup” “style”<br>“kill” “take”</blockquote><p>This kinda makes sense; the lines containing these pairs were repeated throughout the lyrics, and the pairs are in adjacent lines. Hmm. Good.</p><h3>Fun for Thoughts:</h3><p>Enough coding for today. Now I will share funny observations with silly captions…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*14RH8pAK5H9cS3JyjGr1JQ.png" /><figcaption>data helped detect his deceptiveness as it reveals he didn’t really fuck his mother</figcaption></figure><p>Daft Punk: +1</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lg0g4chuJlZrMyAXHkDfMg.png" /><figcaption>if you ignore the foul language, notice the last line: he is very chivalrous, the value with “she” is 0</figcaption></figure><p>Daft Punk: +2</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/1*6t5nSOO8QnkJ-vZMFO4-eg.png" /><figcaption>AI is ambivalent if the song is flirtatious in nature! (50% score)</figcaption></figure><p>Noted.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/718/1*vF6iiK7tcal1CBvGLhO9gA.png" /><figcaption>This song is kinda about the ‘kinda money’ that you don’t have but DP does.</figcaption></figure><p>AI: +1</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/743/1*QVnW4MB_l_cbI3bwjnuL6A.png" /><figcaption>Man’s got style, but the AI is too neutral to judge him 🤥☹️😑</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/767/1*B4fOK6IEvH2Y0u6U5U22xQ.png" /><figcaption>AI is learning so fast, it switched 1st line in the 4th line 🤣🤣🤣</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/696/1*MK5rImxDErz8YVaXSK2fPw.png" /><figcaption>I’d say this was a close try to appeal Brad Pitt, reminiscing his best moments as a tribute</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/691/1*nXC0T69vA6GfS5bda26qMQ.png" /><figcaption>LOL… Wish dying was that easy. For many, it’s painful.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/932/1*aqfRKdp3ciokM6mMSWDxBw.png" /><figcaption>We kill my love ? What do you love, AI? Who do you favor?</figcaption></figure><p>My AI is in love with Daft Punk. Maybe it realised it lost by 1 point and is ruthless about it 🤦🏻‍♂️</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/1*YYarZCHyvumuNxRI_2aX6g.png" /><figcaption>We should all get better shoes, a pair cleaner than cars! 😝</figcaption></figure><p>Ask someone today: Which is cleaner, your shoes or my car?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/1*8uqlV1UUlSiw7_Xkzm7G9Q.png" /><figcaption>I’m going to take this as a compliment. And ask you not to give it to anyone else too.</figcaption></figure><p>I’m a Classic Man!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/1*Cln50ec1P0r7tpI1gmGScg.png" /><figcaption>Okay, this is between the AI and Daft Punk. No comments, none of our business.</figcaption></figure><p>This indicates my AI is a female. “Girls get loose when they hear this song”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/932/1*h8aabxRwXjdyAaWhEeYomw.png" /><figcaption>AI is desperate to learn English. Very desperate.</figcaption></figure><p>Exactly 16.66% desperation is a lot for a computer. Believe me.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/933/1*Hdm-WvbPAW8IRw5bCGUpiw.png" /><figcaption>AI learns to rhyme from a song</figcaption></figure><p>What?!! It learned and started Poetry?!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/729/1*TW-47_lZw_m0nvW40mW5hQ.png" /><figcaption>It’s saying that I’m out of this world</figcaption></figure><p>AI flattering me. Stop reading this blog, these are our personal moments ✋🏼 (I blushed a little)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/1*-J08mjrq1QBog-xlLOjwKg.png" /><figcaption>“For me” is the right way to say that</figcaption></figure><p>English is not easy. ( for a computer 🤪 )</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/907/1*Ef3LWcZZeMjfVCYjjBZ7qw.png" /><figcaption>Wait, it’s trying something</figcaption></figure><p>I’ll let you know when it’s done. Reminds me of quotes from <strong>Star Wars</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FBQ4yd2W50No%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DBQ4yd2W50No&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FBQ4yd2W50No%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/3cdbcc3360466005e13689d99026606d/href">https://medium.com/media/3cdbcc3360466005e13689d99026606d/href</a></iframe><p>and a similar one in <strong>Matrix</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F5mdy8bFiyzY%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D5mdy8bFiyzY&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F5mdy8bFiyzY%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/00dcae8540d36b3b90c561461594c71b/href">https://medium.com/media/00dcae8540d36b3b90c561461594c71b/href</a></iframe><h3>Link to work:</h3><p><a href="https://github.com/D3V4N5H/StarBoy/commit/bf55b72c9c85170b2f14980bc574d8b7dcd79929">https://github.com/D3V4N5H/StarBoy/commit/bf55b72c9c85170b2f14980bc574d8b7dcd79929</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=579899732cf8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ML Day 4]]></title>
            <link>https://medium.com/@D3V4N5H/ml-day-4-a916e223bfee?source=rss-e53210527945------2</link>
            <guid isPermaLink="false">https://medium.com/p/a916e223bfee</guid>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Devansh Trivedi]]></dc:creator>
            <pubDate>Sat, 22 Sep 2018 12:37:42 GMT</pubDate>
            <atom:updated>2018-10-02T18:57:49.421Z</atom:updated>
            <content:encoded><![CDATA[<h4>Mining Word Associations</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/893/1*vEcTGwNTAPs3jju_JT9Yvg.png" /><figcaption>Current state of my Graph Database</figcaption></figure><p>The error I was facing while trying to keep a list of word pairs as the keys of the dictionary wpDict, was that there’s no built-in hash function for lists (by design).</p><blockquote>TypeError: unhashable type: ‘list’</blockquote><p>I’m glad about my brainstorming yesterday, as I just found out that dictionaries are implemented as hash tables! So my implementation is as efficient as it ideally should be.</p><blockquote>Then I used the inbuilt tuple() instead of explicit assignment of firstWord and secondWord variables. So they are gone for good.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/855/1*pnCn405vKdYmthvuwa2Tqw.png" /></figure><h3>Paradigmatic Similarity</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*vMyYef1Ry0-5UfUiSAYVzg.png" /><figcaption>Word Adjacency Graph</figcaption></figure><p>Consider this example from Neo4j’s official video:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qv8U6okOhW9j0_DenEsvSQ.png" /></figure><p>I implemented <strong>Right1</strong> and <strong>Left1 </strong>and will eventually implement Right2 and Left2 when it makes sense to do so.</p><h3>Left1</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/778/1*-bwXK4WNu2g4FetkNLnpfA.png" /><figcaption>for “my” it returned “up” and “love”</figcaption></figure><p>This means “up my” and “love my” word pairs are there in the lyrics and hence the graph. Then I figured I could count them !</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/722/1*dHgh80-0RQrvEtMIjqwXbg.png" /><figcaption>So that returned 2 for “my”</figcaption></figure><p>Then I decided to find out which word had the <strong>highest</strong> number of distinct left words. Hmmm… Let’s see if you can guess it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/909/1*UsByqftRr98AycFXUU2xOQ.png" /><figcaption>“I” had 6 matches</figcaption></figure><p>Wait for it, this is second highest</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/713/1*sfaiRR_IuDRzKWqKD_Rw1A.png" /><figcaption>You can imagine “I” occurring after these words</figcaption></figure><p>Phrases such as “man I” are in lines like “man I love my baby” and “competition I” in anastrophes like “No competition I don’t really listen”.</p><p>Also, I should be able to detect a sentence, when it ends, so adjacency doesn’t have unexpected correlations like “cup I”, where the actual sentence ends at “cup” and another begins with “I”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/726/1*eSOz52wD1JTnQlJjP1271w.png" /><figcaption>“a” had 11 matches</figcaption></figure><p>Yes, this one’s the winner! 🎉</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/1*vhkkcgyAnuMXW14Jg_9KVA.png" /><figcaption>Remember these words were followed by “a” ?</figcaption></figure><p>Word pairs “I’m a”, “Like a”, “in a”, “need a” are very common</p><h3>Right1</h3><p>This will make even more sense and as it gave me a valuable insight in how we can observer how our language boils down to the basic rules of grammar!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/912/1*vDqlk0lfgEVzqMjiuPlXDg.png" /><figcaption>7 distinct words were detected that were in the right side of “the”</figcaption></figure><p>“The” is an article (in grammar terms).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/742/1*5mb1P5a42zDVpn3zL8UL9Q.png" /><figcaption>Try to put “The” before each of them</figcaption></figure><p>The first line of the song is: “I’m tryna put you in <strong>the worst</strong> mood ah” !</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/1*funApNfnjfBfxnqK89HzxA.png" /><figcaption>“a” had 11 matches (AGAIN! Pure coincidence, as it is the number of distinct words)</figcaption></figure><p>All this has started to make sense, as “a” and “the” are articles (grammar). They’re always used with a noun. So it makes sense that they would have the highest right-adjacency word pair frequency.</p><h3>Links to work:</h3><ul><li><a href="https://github.com/D3V4N5H/StarBoy/blob/master/starboy.csv">D3V4N5H/StarBoy</a></li><li><a href="https://github.com/D3V4N5H/StarBoy/blob/master/CypherQueries">D3V4N5H/StarBoy</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a916e223bfee" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ML Day 3]]></title>
            <link>https://medium.com/@D3V4N5H/ml-day-3-17d215953adf?source=rss-e53210527945------2</link>
            <guid isPermaLink="false">https://medium.com/p/17d215953adf</guid>
            <category><![CDATA[data-structures]]></category>
            <dc:creator><![CDATA[Devansh Trivedi]]></dc:creator>
            <pubDate>Fri, 21 Sep 2018 18:17:28 GMT</pubDate>
            <atom:updated>2018-10-02T18:58:32.123Z</atom:updated>
            <content:encoded><![CDATA[<h4>The list key conundrum</h4><p>I considered Data Structures like Hash Table (with multiple key values mapped to a value) and Linked List for storing the data for word pairs and their frequencies. At one point, I thought it was a good idea to develop a new data structure, for which I watched a FreeCodeCamp Full Course for Beginners of Data Structures and Algorithms.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Ft2CEgPsws3U%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dt2CEgPsws3U&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Ft2CEgPsws3U%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/221153ef58b219921ae0b88eeda776a4/href">https://medium.com/media/221153ef58b219921ae0b88eeda776a4/href</a></iframe><p>I settled for saving the word pairs as a list in a Python dictionary as keys and their frequency count as values.</p><blockquote>But in Python it is not possible to use a list as a key for a dictionary, as the key of a dictionary MUST BE immutable</blockquote><p>So I used a <em>tuple</em> instead. For this, in each iteration of the loop, I split the word-pair result from the list in the ‘Record object’ into a tuple of two variables, <strong>firstWord</strong> and <strong>secondWord</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ilGBoQzP2sBMs3U6v0CkXQ.png" /><figcaption>Word-Pair Frequency into Python Dictionary with tuples as keys and count as values</figcaption></figure><h3>Link to work:</h3><p><a href="https://github.com/D3V4N5H/StarBoy/commit/05c96ab331569f1831943aa3d168921935e88888">https://github.com/D3V4N5H/StarBoy/commit/05c96ab331569f1831943aa3d168921935e88888</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=17d215953adf" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>