<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sanat Dutta on Medium]]></title>
        <description><![CDATA[Stories by Sanat Dutta on Medium]]></description>
        <link>https://medium.com/@sanatdutta?source=rss-b74763478118------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*6kLxjhzYTvW5tG_fZsYRQQ.png</url>
            <title>Stories by Sanat Dutta on Medium</title>
            <link>https://medium.com/@sanatdutta?source=rss-b74763478118------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 22:16:32 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sanatdutta/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Midnight Hack 5: Using Machine Learning to categorize Spotify playlists]]></title>
            <link>https://medium.com/data-science/midnight-hack-5-using-machine-learning-to-categorize-spotify-playlists-57dc492fc3e6?source=rss-b74763478118------2</link>
            <guid isPermaLink="false">https://medium.com/p/57dc492fc3e6</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[spotify]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Sanat Dutta]]></dc:creator>
            <pubDate>Sun, 20 Jun 2021 22:29:56 GMT</pubDate>
            <atom:updated>2021-06-20T22:29:56.247Z</atom:updated>
            <content:encoded><![CDATA[<h4>Classifying and clustering Spotify tracks using KNN &amp; Random Forest.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-hw260nIuxWnd0uuYkIWpA.jpeg" /></figure><p>It probably takes only a second to add a track to a playlist, but as a representative of elite procrastinators, I refuse to do so. Instead, when I like a song, I just hit the like button and leave it there forever.</p><p>Occasionally, you get an urge to bring some order into your life, which usually happens at 2 am. So you sit down, open <a href="https://www.spotify.com/">Spotify</a>, shuffle-play your liked songs, and try to add them to playlists one by one, only to get overwhelmed, quit and order food online in the middle of the night.</p><p>Here I’m going to show you how to not be like me and use Machine Learning to your advantage. In this article, we’re going to cover both supervised and unsupervised approaches. In the unsupervised approach, we’ll cluster the tracks using song features. In the supervised approach, we’ll use our existing playlists to train a model to understand the relationship between the tracks and playlists and use this model to categorize liked tracks to one of these playlists.</p><p>We will also cover automating playlist creation and adding tracks to playlists using Spotify APIs. In case you’re unfamiliar with the Spotify authentication and scopes, I suggest you checkout my previous <a href="https://betterprogramming.pub/midnight-hack-episode-4-automating-spotify-mapping-spotify-functions-to-keyboard-shortcuts-3ea5302bb2cb">article</a>.</p><p><strong>Data Preparation:</strong></p><ul><li>Fetching liked tracks and generating audio features.</li></ul><p><strong>Unsupervised approach:</strong></p><ul><li>Categorize liked tracks into clusters using <a href="https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm">KNN</a>.</li><li>Creating playlists based on cluster groups and adding tracks to playlists.</li></ul><p><strong>Supervised approach:</strong></p><ul><li>Fetch existing playlists and tracks and generate training data.</li><li>Train a <a href="https://en.wikipedia.org/wiki/Random_forest">Random Forest Model</a> for track classification.</li><li>Use the trained model to categorize liked tracks into existing playlists.</li></ul><p><strong>Data Preparation:</strong></p><p>We can easily access Spotify APIs via <a href="https://spotipy.readthedocs.io/">spotipy</a> package. Check the article mentioned above to get Spotify client_id and client_secret.</p><p>Once we have our Spotify client ready, we can use all Spotify APIs via the client. Here we can use current_user_saved_tracks() API to get a list of paged liked tracks. Most of Spotify’s list APIs are paged to limit API overloads. After handling the paging logic and we get a complete list of our liked tracks.</p><p>Spotify stores a lot of tracks metadata. Here we’re only saving the id and the name of the tracks.</p><p>For generating the audio features, initially I had thought of downloading the audio waveforms for all the tracks and using libraries like <strong>librosa </strong>to generate audio features manually but while going through the Spotify API documentation, I came across audio_features() API which returns features like danceability, energy, key, loudness and 7 other audio features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wRryXFa6nERfVCvxaXb6Ug.png" /></figure><p>For better accuracy, we should normalize the features and each with the correct function. For example, the audio level is in decibels, which is on a log scale so min-max scaling wouldn’t be accurate. For this project, I did not normalize the features.</p><p>I used only these features for the automation but additional features like locale/language info, track length etc. can also be used for better model accuracy.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0c13a7bb8ba3a69db3c8cd5bb8543763/href">https://medium.com/media/0c13a7bb8ba3a69db3c8cd5bb8543763/href</a></iframe><p><strong>Unsupervised approach:</strong></p><p>Now that we have track data, we can cluster tracks. But before grouping, we need to identify how many clusters (K) should be generated. We can do this using the <a href="https://en.wikipedia.org/wiki/Determining_the_number_of_clusters_in_a_data_set">Elbow Method</a>. In this method, a model is fitted with varying number of K values. The cost value is plotted against the K number and where the curve breaks, is considered the optimum K value. It’s not a set rule, but it narrows down the K range for better clustering.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*9OuZoLhTQlPYteq3BARE2w.png" /></figure><p>As you can see in the plot above, curve breaks around K=3 but the saturation doesn’t start until K=5. I used K=5 and grouped my tracks into 5 playlists.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4bd3be107c84a381e76734c21b076157/href">https://medium.com/media/4bd3be107c84a381e76734c21b076157/href</a></iframe><p>Since I’m using 11 audio features, It’ll be difficult to visualize the 11 dimensional vectors on 2d plots. So I used <a href="https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html">TSNE</a> to reduce the dimensionality of audio vectors for cluster plotting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*iJRgufRzUE2wM7_vKaozQg.png" /></figure><p>In the plot above, each point represents a track, and each color represents a playlist. Now that we have mapped our liked tracks to groups, we can use Spotify APIs to create playlists for each group and add respective tracks to playlists.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/449/1*j1toYYih7xikr15oTmYkpQ.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3efce0539d3fcf84f8386519dc1fef3b/href">https://medium.com/media/3efce0539d3fcf84f8386519dc1fef3b/href</a></iframe><p>Here’s a screenshot of automatically created playlists with organized tracks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JzbqRvBCC_iow-SLMZt9QQ.jpeg" /></figure><p><strong>Supervised approach:</strong></p><p>Unsupervised KNN approach was simple and fast but it also has problems. It assumes that all features are equally important and we don’t have any metric to identify which feature is more important than the others.</p><p>In the unsupervised approach, we fitted a model to group tracks based on audio features, but can we teach it to Love? Show emotions? Probably not. But we can surely teach it to learn the relationship between an existing playlist and its tracks.</p><p>When we already have a couple of playlists, we have a target variable. We can identify the user logic and infer the weights user subconsciously used to categorize a certain track to a playlist. Of course, here we assume that the user also used the same audio features subconsciously. Therefore, relevant features are important, but we’ll work with what we have.</p><p>First, we need to prepare training data on which we’ll train our Machine Learning model.</p><p>I chose some of my playlists, which had a reasonable number of tracks. The more data we have, the better.</p><p>This data structure is the same as the liked tracks data but here each track is mapped to a playlist.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k8-xZA8YVVpu0Cks0LnYzA.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4442b521589bc8e12e5920dee2188229/href">https://medium.com/media/4442b521589bc8e12e5920dee2188229/href</a></iframe><p>With the training data in hand, I trained a <a href="https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html">Random Forest Classifier</a> on 70% of the data using mostly default configuration. Here max_depth can be increased for better accuracy, but it may cause the model to overfit on the training data.</p><p>Upon testing on the remaining 30% data, the model accuracy came out to be around 67% which is decent considering the limited dataset and minimal hyper-parameter tuning.</p><p>Now we can use this trained this model to predict and assign playlists for the liked tracks and use the same APIs we used in the previous section to add tracks to playlists.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/1*6H88s0-bXzxuWUDQH1FEyw.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2f2fe52abf8e538b535b64bcba87064c/href">https://medium.com/media/2f2fe52abf8e538b535b64bcba87064c/href</a></iframe><p><strong>Footnote:</strong></p><p>The approaches discussed here are nowhere perfect and are merely a proof of concept. We can vastly increase the accuracy of the models by using relevant features, appropriate data normalization, testing different classifiers and using appropriate hyper-parameters.</p><p>Until next time…</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=57dc492fc3e6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/midnight-hack-5-using-machine-learning-to-categorize-spotify-playlists-57dc492fc3e6">Midnight Hack 5: Using Machine Learning to categorize Spotify playlists</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Midnight Hack Episode 4: Automating Spotify]]></title>
            <link>https://medium.com/better-programming/midnight-hack-episode-4-automating-spotify-mapping-spotify-functions-to-keyboard-shortcuts-3ea5302bb2cb?source=rss-b74763478118------2</link>
            <guid isPermaLink="false">https://medium.com/p/3ea5302bb2cb</guid>
            <category><![CDATA[spotify]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[windows]]></category>
            <dc:creator><![CDATA[Sanat Dutta]]></dc:creator>
            <pubDate>Fri, 04 Dec 2020 09:41:50 GMT</pubDate>
            <atom:updated>2021-06-20T12:36:51.795Z</atom:updated>
            <content:encoded><![CDATA[<h3>Midnight Hack 4: Automating Spotify</h3><h4>Mapping Spotify functions to keyboard shortcuts</h4><figure><img alt="Spotify logo and wordmark with an image of a grinning hacker using a laptop" src="https://cdn-images-1.medium.com/max/1024/1*OtaS_CK6UPLMtuaOoCs7Ag.jpeg" /><figcaption>Image credit: Author via <a href="https://www.spotify.com/in/">Spotify</a> and <a href="https://pixabay.com/illustrations/hacker-hack-security-cyber-4685929/">Hacker Hack Security — Free image on Pixabay</a></figcaption></figure><h3>TLDR;</h3><p>In case you don’t want to read through the entire post, here’s what we’re building. We’re building an automation pipeline to automatically save a currently playing Spotify track to user playlists, based on the shortcut key pressed. The execution is device independent, meaning you could be playing the track on your mobile phone, and with the press of Alt+1 on your computer, you could add the track to, say, Playlist 1, or with Alt+2 you could add it to Playlist 2.</p><p>The execution is also a background process, so it doesn’t affect your active working window or whatever work you’re doing on the computer.</p><p>Okay, let’s start.</p><p>Why do I need this? Well, I did some groundwork. When you’re working, in order to switch to Spotify, clicking on three dots on the song, finding the playlist I want to add it to, and saving it to a playlist took me about 13 seconds. Let’s round it to ten seconds because I’m a sloth.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ke7R7fwaTfdjzmdMkbJ7YA.png" /></figure><p>For all of the 912 combined songs in my playlist, that’s a total of 2.53 hours I could’ve used to think about what I’m doing with my life.</p><p>Also, it’s mildly inconvenient to switch applications to do this task. So let’s automate this. Here’s how we do it:</p><figure><img alt="diagram showing development flow of the app" src="https://cdn-images-1.medium.com/max/921/1*cuQVlhCvfp29xRhuBCKsog.jpeg" /><figcaption>Development flow</figcaption></figure><h3>1. Get Spotify API Keys</h3><p>Go to <a href="https://developer.spotify.com/dashboard/applications">https://developer.spotify.com/dashboard/applications</a> and create a new application. Save the client_id and client_secret.</p><p>Go to “edit settings” and add a redirect URL such as <a href="https://example.com/callback">https://example.com/callback</a>. This will be used for OAuth and for generating a user token.</p><p>Since we need to generate a user token only once, there’s no need to have a live server to catch the redirected URL. We can just copy the redirected URL and create a user token. More on this in the next step.</p><figure><img alt="Spotify for developers dashboard showing the Autoplaylist app has been created" src="https://cdn-images-1.medium.com/max/1024/1*c-2hOeyWhX-6Z5TZGMGczw.png" /><figcaption>Screenshot</figcaption></figure><h3>2. Generate Hotkey-Playlist Mapping config</h3><p>Now that we have the client_id and client_secret, we can get a user token, a list of user playlists, and playlist IDs. I used a Spotify Python library called <a href="https://spotipy.readthedocs.io/en/2.16.1/">spotipy</a><strong><em>.</em></strong></p><p>I created a small script to authenticate the user, list user playlists, and prompt the user to select playlists they want to map the hotkeys to. I’ve used Alt+num, but it can be anything you want.</p><p>Here’s the gist for the same:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3eb8a0f0bbc1040dea47ab8811dcfeb1/href">https://medium.com/media/3eb8a0f0bbc1040dea47ab8811dcfeb1/href</a></iframe><p>The script generates a config file playlist_mapping.config<strong>, </strong>which is used for hotkey-playlist lookup at the time of track saving.</p><p>Here’s what the script looks like in action:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*ThklxIY6yYt1fEgjfm2r1A.gif" /></figure><h3>3. AutoHotKey Script</h3><p>In order to monitor the hotkey triggers, we need an input monitoring service. <a href="https://www.autohotkey.com/">AutoHotKey</a> is a really good tool for this. Once the AutoHotKey script is running, it monitors the hotkeys and triggers appropriate actions specified in the AHK script. Plus you can also add AHK scripts to startup so you don’t have to run the AHK script every time you boot up your computer.</p><p>There is more than one way to do this, but I’ve used the Run<strong><em> </em></strong>command, which runs an executable or program in a specific directory.</p><p>Here’s the cmd format:</p><p>&lt;hotkey&gt;::Run, &lt;command_to_execute&gt;, &lt;dir_to_execute_in&gt;</p><p>And here’s my script:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f93a532061e502f57e97eb6d887d0bf7/href">https://medium.com/media/f93a532061e502f57e97eb6d887d0bf7/href</a></iframe><p>! stands for <em>Alt</em>. So, when I press Alt+1, the AHK script triggers the python3w spotify_api.py 1<strong> </strong>command in the D:\_DEV_\Python\spotify_auto_playlist directory.</p><p>Note that I’m using python3w<strong> </strong>and not python3. python3w runs the script in the background.</p><p>Once you have the script saved, right-click and run the service.</p><h3>4. Python Script to Save Current Track to a Playlist</h3><p>This script receives a positional argument from the AHK script which is used to identify the hotkey. Using the hotkey-playlist lookup config we generated in the first step and the current track ID through the Spotify API, we can add the current track to any playlist previously defined by the user.</p><p>Here’s the script:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/02b953b45d0c336071b3b7d2a3c4eb19/href">https://medium.com/media/02b953b45d0c336071b3b7d2a3c4eb19/href</a></iframe><p>That’s it.</p><h3>Here’s the Product in Action</h3><p>On the left side, we have the application log, which displays the hotkey pressed and the info about which track is being saved to which playlist. On the right side, we have the Spotify desktop player.</p><p>The app log and Spotify player shown here are just for the demo. The song could be playing on your mobile phone, and if you press Alt+1, it’ll save the currently playing track to the playlist defined in the first slot by the user.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fplayer.vimeo.com%2Fvideo%2F487235189%3Fapp_id%3D122963&amp;dntp=1&amp;display_name=Vimeo&amp;url=https%3A%2F%2Fvimeo.com%2F487235189&amp;image=https%3A%2F%2Fi.vimeocdn.com%2Fvideo%2F1008340638_1280.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=vimeo" width="1280" height="720" frameborder="0" scrolling="no"><a href="https://medium.com/media/30e2f4f4a1bd214203fefb583fab1657/href">https://medium.com/media/30e2f4f4a1bd214203fefb583fab1657/href</a></iframe><p>If you have any suggestions or comments, please do write below.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3ea5302bb2cb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/better-programming/midnight-hack-episode-4-automating-spotify-mapping-spotify-functions-to-keyboard-shortcuts-3ea5302bb2cb">Midnight Hack Episode 4: Automating Spotify</a> was originally published in <a href="https://betterprogramming.pub/">Better Programming</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Midnight Hack Episode 3: Scraping LinkedIn connections using JavaScript and Python]]></title>
            <link>https://sanatdutta.medium.com/midnight-hack-episode-3-scraping-linkedin-connections-using-javascript-and-python-b698b714b864?source=rss-b74763478118------2</link>
            <guid isPermaLink="false">https://medium.com/p/b698b714b864</guid>
            <category><![CDATA[hacking]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[linkedin]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[javascript]]></category>
            <dc:creator><![CDATA[Sanat Dutta]]></dc:creator>
            <pubDate>Thu, 14 May 2020 17:03:48 GMT</pubDate>
            <atom:updated>2021-06-20T12:38:28.302Z</atom:updated>
            <content:encoded><![CDATA[<h3>Midnight Hack 3: Scraping LinkedIn connections using JavaScript</h3><p>For reasons, I needed to export a list of my LinkedIn network contacts specifically in the Pharmaceutical Industry.</p><p>You can filter and view your contacts on LinkedIn but it neither lets you export the data nor has official API support for the same.</p><p>So, we’ll be using our logged-in browser session to make unofficial API calls to scrape the contacts.</p><p>I’ve heard, if you say something is for Educational Purposes, you can pretty much get away with anything. So yeah, this is for <strong>Educational Purpose</strong> :)</p><p>Let’s begin by going to the LinkedIn Contacts page and apply all the required filters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qPtyaTDa5tpuZxHNUJcdBQ.png" /><figcaption>Source: Screenshot</figcaption></figure><p>Now open DevTools on your browser and switch to Network Tab. Upon basic inspection, it shouldn’t be difficult to identify the API responsible for fetching contacts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8-1Q_ELH4zMpmUZzqwCOfw.png" /><figcaption>Source: Screenshot</figcaption></figure><p>Now you can make the same API request from the browser console and get the paged contacts without even implementing OAuth :)</p><p>Let’s make a simple REST call from the console tab and check out the response.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/83d883b3ee7141b01be18e9c9cc11bff/href">https://medium.com/media/83d883b3ee7141b01be18e9c9cc11bff/href</a></iframe><p>Upon making the request, you’ll find that the request errors out because of failed CSRF check.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/718/1*76AHKUSXVR8eBU-OYYZc2A.png" /><figcaption>Source: Screenshot</figcaption></figure><p>This is a counter-measure to stop unwanted API calls from logged-in clients. But since you’re already logged in and you can see the request headers for the previously made API call though LinkedIn page, you can easily copy the <strong>csrf-token </strong>and other request headers to your API request.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*v3L7u3KtW8YLdu22-3FPHg.jpeg" /><figcaption>Source: <a href="https://i.kym-cdn.com/entries/icons/original/000/022/138/highresrollsafe.jpg">https://i.kym-cdn.com/entries/icons/original/000/022/138/highresrollsafe.jpg</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/863/1*gYeYRP7oqX9GhHX5qrPUrw.png" /><figcaption>Source: Screenshot</figcaption></figure><p>Switch to header tab and copy csrf-token and other request headers and make the API call again.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/95a680714f9b169db384b02d3257a5ed/href">https://medium.com/media/95a680714f9b169db384b02d3257a5ed/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/839/1*JcT-3x9qZ8gmFdgAmgNthw.png" /><figcaption>Source: Screenshot</figcaption></figure><p>And …… <strong>Voilà</strong></p><p>This only returns 10 paged results and for some reason, the API response became uncertain when I tried to increase the count by changing the request parameter so let’s just stick to 10 responses and add a loop to fetch all the contacts.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/575ebc0c4245d4cb73976b2df3eb31c3/href">https://medium.com/media/575ebc0c4245d4cb73976b2df3eb31c3/href</a></iframe><p>This piece of code will loop the request with increasing page count and save fetched contacts as an array. Since we’re restless and we haven’t added any delays between API requests, the requests are terminated from the server-side after 1000 or so contact fetches.</p><p>To avoid this you can add a delay between requests or re-run the same script with start index as 1000. But for now, 1000 contact will do.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/531/1*oEQBuEuUarsub4oafB3NMw.png" /><figcaption>Source: Screenshot</figcaption></figure><p>Now, let’s filter out the unwanted fields and export it as CSV.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e720b386b28d30797492a4792732de14/href">https://medium.com/media/e720b386b28d30797492a4792732de14/href</a></iframe><p>And….. here’s the result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bK_m43xWq0wY7vZwlWEA4g.png" /><figcaption>Source: Screenshot</figcaption></figure><p>Until next time….</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b698b714b864" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Midnight Hack Episode 2: Contextual search & grouping of Emojis using Document Embedding]]></title>
            <link>https://medium.com/data-science/midnight-hack-episode-2-contextual-search-grouping-of-emojis-using-document-embedding-9bee5bbed6cc?source=rss-b74763478118------2</link>
            <guid isPermaLink="false">https://medium.com/p/9bee5bbed6cc</guid>
            <category><![CDATA[visualization]]></category>
            <category><![CDATA[doc2vec]]></category>
            <category><![CDATA[emoji]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Sanat Dutta]]></dc:creator>
            <pubDate>Sun, 16 Feb 2020 16:16:29 GMT</pubDate>
            <atom:updated>2022-11-07T07:21:10.922Z</atom:updated>
            <content:encoded><![CDATA[<h3>Midnight Hack 2: Contextual search &amp; grouping Emojis using Document Embedding</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bZ09tZQ9YvbxFtgSLgHV0Q.png" /><figcaption>Citation: <a href="http://www.satandigital.com/apps/EmojiPlot/">http://www.satandigital.com/apps/EmojiPlot/</a></figcaption></figure><p>Hello there !!! and welcome to yet another stupid project of mine.</p><p>So I was using <a href="https://www.whatsapp.com/"><strong>Whatsapp</strong></a><strong> </strong>the other day and found this cool little feature of finding emojis using keywords really interesting. Now, I’m pretty sure they didn’t use Machine Learning to achieve this and it’s most probably a simple map of hardcoded keywords and emojis.</p><p>But what if you could go a step further and not limit the search functionality to those predefined keywords but search emojis based on the contextual meaning of the search keyword.</p><p>Too lazy to read through? <strong>You can view the EmojiPlot here:</strong> <a href="http://www.satandigital.com/apps/EmojiPlot/">http://www.satandigital.com/apps/EmojiPlot/</a></p><p>The plot has zoom and pan functionality. Click and drag the mouse to zoom and use the shift key to pan.</p><p><strong>And the scraped dataset is hosted here:</strong> <a href="https://github.com/AtomicSpider/Emoji-Dataset">https://github.com/AtomicSpider/Emoji-Dataset</a></p><p>This project has 4 parts:</p><ol><li>Prepare Emoji Dataset</li><li>Generate vector or numerical representation of Emojis</li><li>Plot emojis in 2D space</li><li>Write a search script</li><li><strong>Prepare Emoji Dataset:</strong></li></ol><p>A simple google search led me to <a href="https://unicode.org/">https://unicode.org/</a> where they had a comprehensive list of all the emojis, images, Unicode ids, base64 data URIs and names.</p><p>I wrote a simple web scraping script to download all the images and metadata and created a master dataset JSON which also had reference to image files.</p><p>Dataset can be found here: <a href="https://raw.githubusercontent.com/AtomicSpider/Emoji-Dataset/master/dataset/dataset.json">https://raw.githubusercontent.com/AtomicSpider/Emoji-Dataset/master/dataset/dataset.json</a></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/38c89f79119b1b8f1b0635a7b7204770/href">https://medium.com/media/38c89f79119b1b8f1b0635a7b7204770/href</a></iframe><p><strong>2. Generate vector or numerical representation of Emojis:</strong></p><p>I came across <a href="https://radimrehurek.com/gensim/models/word2vec.html"><strong>word2vec</strong></a><strong>,</strong> a two-layer Neural Network which can take words and generate a multi-dimensional vector representation of the words.</p><p>The document equivalent of word2vec, <a href="https://radimrehurek.com/gensim/models/doc2vec.html"><strong>doc2vec</strong></a><strong>, </strong>works in a similar way. doc2vec takes documents or a group of words and generates a multi-dimensional vector representation of it.</p><p>I decided to use doc2vec, extracted keywords from emoji names and generated a 300-dimensional vector for each emoji.</p><p>Emoji names can directly be used as input to doc2vec model but I went ahead and used <strong>nltk </strong>(natural language toolkit) to remove stopwords from the names to achieve better accuracy.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7439f93196b879cbd8159e7e825248ab/href">https://medium.com/media/7439f93196b879cbd8159e7e825248ab/href</a></iframe><p><strong>3. Plot emojis in 2D space:</strong></p><p>Cool !!! Now that we have the vector data, plotting should be easy right? right??</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2Fd10dMmzqCYqQ0%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2Fd10dMmzqCYqQ0%2Fsource.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2Fd10dMmzqCYqQ0%2Fgiphy-downsized-large.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="245" frameborder="0" scrolling="no"><a href="https://medium.com/media/9b72bf133da8ecab4f8014bcf1dca23e/href">https://medium.com/media/9b72bf133da8ecab4f8014bcf1dca23e/href</a></iframe><p>Actually, this part took quite a bit of time for me to figure out. The first problem being: How do you even plot 300-dimensional vectors?</p><p>After quite a bit of research, I decided to use <a href="https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html"><strong>tSNE</strong></a><strong> </strong>and reduce the dimensionality of the generated emoji vectors.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fc5958071a1b0aa4c8023185dfbc77a7/href">https://medium.com/media/fc5958071a1b0aa4c8023185dfbc77a7/href</a></iframe><p>Right !!! Now that we have the plot data, generating a simple scatter plot with Emojis would be easy right? right?? right???</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2F26DOzMVt4i9GzzxyE%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2F26DOzMVt4i9GzzxyE%2Fsource.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2F26DOzMVt4i9GzzxyE%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="303" frameborder="0" scrolling="no"><a href="https://medium.com/media/3882e7e1fa822b93c890636a94412147/href">https://medium.com/media/3882e7e1fa822b93c890636a94412147/href</a></iframe><p>Well, the problem is, I wanted to use Emojis as custom markers for scatter points. Sadly python did not have great support for that.</p><p>There’s a library in R, <strong>ggplot</strong> which can be used with <a href="https://github.com/dill/emoGG"><strong>dill/emoGG</strong></a><strong> </strong>to generate emoji markers but the plot didn’t look great. Also, there was no pan and zoom functionality.</p><p>So I decided to export the data and plot using <a href="https://www.highcharts.com/"><strong>HighCharts</strong></a><strong>. </strong>Thanks to my colleague Vinoth, who helped me figure out how to use custom markers in HighCharts.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/563abf88dc1384003780a2482864d553/href">https://medium.com/media/563abf88dc1384003780a2482864d553/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8bf1508b5d3ac60bcaa0e7d43713c811/href">https://medium.com/media/8bf1508b5d3ac60bcaa0e7d43713c811/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bZ09tZQ9YvbxFtgSLgHV0Q.png" /><figcaption>Citation: <a href="http://www.satandigital.com/apps/EmojiPlot/">http://www.satandigital.com/apps/EmojiPlot/</a></figcaption></figure><p>The plot page can be accessed here: <a href="http://www.satandigital.com/apps/EmojiPlot/"><strong>http://www.satandigital.com/apps/EmojiPlot/</strong></a></p><p>As you can see, the approach we took was able to group emojis pretty well. All the flags can be seen grouped together on the left, animals towards the middle top, food on the right top, face emojis on the right and so on.</p><p><strong>4. Write a search script:</strong></p><p>For searching emojis from keywords, we can use the same doc2vec model to convert search keywords to a vector and find the nearest neighbors of that vector from the emoji vector list.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1caaabd434918b8ef22c316b970da167/href">https://medium.com/media/1caaabd434918b8ef22c316b970da167/href</a></iframe><p><strong>Result:</strong></p><p>if you do a search for happy emotions on the <strong>WhatsApp </strong>emoji keyboard, it does not produce any results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jmfrNB56frn0tG9pwayJXw.jpeg" /><figcaption>Citation: screenshot captured from my phone</figcaption></figure><p>But with our approach, we’re able to search related emojis with contextual information rather than hardcoded keyword mapping.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/448/1*C21Q0_Eh4GpPRUGRR_yFQQ.png" /><figcaption>Citation: screenshot captured from my laptop</figcaption></figure><p>This concludes my sleep-deprived night 😁.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9bee5bbed6cc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/midnight-hack-episode-2-contextual-search-grouping-of-emojis-using-document-embedding-9bee5bbed6cc">Midnight Hack Episode 2: Contextual search &amp; grouping of Emojis using Document Embedding</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Midnight Hack Episode 1: Visualizing my Swiggy Order History]]></title>
            <link>https://medium.com/data-science/midnight-hack-episode-1-visualizing-my-swiggy-order-history-f1cce25cbff6?source=rss-b74763478118------2</link>
            <guid isPermaLink="false">https://medium.com/p/f1cce25cbff6</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[swiggy]]></category>
            <category><![CDATA[visualization]]></category>
            <dc:creator><![CDATA[Sanat Dutta]]></dc:creator>
            <pubDate>Sat, 31 Aug 2019 11:08:59 GMT</pubDate>
            <atom:updated>2021-06-20T12:39:15.563Z</atom:updated>
            <content:encoded><![CDATA[<h3>Midnight Hack 1: Visualizing my Swiggy Order History</h3><p>Don’t have the patience to read though? Worry not.</p><p>Here’s the <a href="https://swiggy-order-dashboard.herokuapp.com/"><strong>Visualization Dashboard</strong></a><strong>, </strong><a href="https://www.youtube.com/watch?v=M5qhpyvUYhw"><strong>Dashboard Preview</strong></a><strong> </strong>and here’s the project <a href="https://github.com/AtomicSpider/Swiggy-Dashboard"><strong>Github Link</strong></a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WSqoOZwryXVOJw0djaZsvw.jpeg" /></figure><p>It’s 11 pm and as I’m on my way to order multiple midnight meals, I thought to myself :</p><blockquote>I don’t really know my ordering habits. is there a trend? Can I derive some meaningful insights from my ordering history?</blockquote><p>As I browse through the dessert options on Swiggy (An Indian Food Order &amp; Delivery App), I’m trying to figure out how can I get my order history data. Now, I’ve learned from past experiences that asking nicely just works out sometimes. Let’s try that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MQ-HkZkspvq5jLywARSHbg.jpeg" /></figure><p>So yeah, that didn’t pan out but where there’s a will, there’s a way.</p><p>I can probably intercept network requests for <a href="https://www.swiggy.com/">Swiggy</a> App on the order history page. I know apps like <a href="https://play.google.com/store/apps/details?id=org.sandroproxy.drony">Drony</a> can do that. But I’ll have to develop a custom app in order to iterate through all the order pages and aggregate the results. Agggggghhhhhhh…… seems like too much work. Let’s call this plan-B. Will come back to it if nothing else works out.</p><p>I guess Web would be a better choice. Relatively easier to hack. I just hope they still have the order history page on their website.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bR0BoZ_tD0E18wuTmch87Q.png" /></figure><p>Yayyyyy, they have a fully functional website. I thought they had gone completely mobile. Well, good for me. They even have the order history page. I’m in tears right now. Dev tears :’)</p><p>Let’s pop out chrome Dev Tools and look at some network requests, shall we?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GPs_pFt5LTL8TAOM2uKu7w.png" /><figcaption>Order list response</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yufZaAqf0SebcszVaM3qSg.png" /><figcaption>GET request headers</figcaption></figure><p>Oh, that sweet sweet JSON response :)</p><p>Looks like order_id is the parameter for pagination.</p><p>They seem to be using CSRF tokens. Looking at the request headers, seems like they’re using cookies to maintain session and using CSRF token to avoid third party client requests. I thought they’ll be using some simple JWT token, so I was going to use python for network requests but I guess that’s not an option.</p><p>Better to hit the APIs through the browser. Not sure if that’ll work but let’s give it a try.</p><pre>var xmlHttp = new XMLHttpRequest();<br>xmlHttp.open( “GET”, “<a href="https://www.swiggy.com/dapi/order/all?order_id=">https://www.swiggy.com/dapi/order/all?order_id=</a>&quot;, false );<br>xmlHttp.send( null );<br>console.log(xmlHttp.responseText);</pre><p>While I’m logged in, this simple JS snippet should be able to fetch order data using session cookies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K_2ZprJ7DrRIiDQHF9SqkA.png" /><figcaption>Order response through JS snippet</figcaption></figure><p>And it worked :’)</p><p>I didn’t expect it to work with all the security measures they had taken but heyyyy…..</p><p>Now all I have to do is modify the script so that it can paginate through all the pages and can give me the aggregated data.</p><pre>order_array=[]<br>order_id=’’<br>page = 1</pre><pre>try {<br>    while(true){<br>        var xmlHttp = new XMLHttpRequest()<br>        xmlHttp.open( “GET”, “<a href="https://www.swiggy.com/dapi/order/all?order_id=">https://www.swiggy.com/dapi/order/all?order_id=</a>&quot;+order_id, false )<br>        xmlHttp.send( null )<br>        resText=xmlHttp.responseText<br>        var resJSON = JSON.parse(resText)<br>        order_id=resJSON.data.orders[resJSON.data.orders.length-1].order_id<br>        order_array=order_array.concat(resJSON.data.orders)<br>        console.log(“On page: “+page+” with last order id: “+order_id)<br>        page++<br>    }<br>}<br>catch(err) {<br>    console.log(order_array)<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CPCXH2C58lSGCC3xmjdiIw.png" /><figcaption>Complete order history</figcaption></figure><p>Total of <strong>53 </strong>pages and <strong>518 </strong>orders.</p><p>*Looks at his floppy tummy.</p><p>Not bad. Not bad at all. *cries internally</p><p>Now, I could’ve added some code to save the data in a file, but there’s an easier way. Right-click on the data&gt; Store as a global variable. Use command copy to copy data in the clipboard.</p><pre>copy(temp1)</pre><p>Paste it in notepad and save. And Voilà, now you have your entire <a href="https://www.swiggy.com/">Swiggy </a>history.</p><p>It’s <strong>5MB </strong>of <strong>text data</strong> with a lot of information. Initially, I was thinking of deriving some insights in text form, but, looking at the data, it’s better to create a visual dashboard.</p><p>I don’t really have any experience in creating visualization but after going through a ton of documentation and evaluating tools like J<strong>upyter Notebooks, Matplotlib, Plotly, Dash</strong>; I decided to go with Dash. It has a pretty straight forward layouting process and works well with python.</p><p>It’s 4:20 PM, Next day. After an entire night of analysis and design, here’s the final result.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FM5qhpyvUYhw%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DM5qhpyvUYhw&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FM5qhpyvUYhw%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/6983633ef13ce61c655ecda03ffb5160/href">https://medium.com/media/6983633ef13ce61c655ecda03ffb5160/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I0xSMGOyeADny1KW4McDsA.png" /><figcaption>Screencap of Visualization Page</figcaption></figure><p>You can check out my Visualization Dashboard <a href="https://swiggy-order-dashboard.herokuapp.com/"><strong>here</strong></a>. If you’d like to visualize your own <a href="https://www.swiggy.com/">Swiggy </a>order history, all the required resources are hosted <a href="https://github.com/AtomicSpider/Swiggy-Dashboard"><strong>here</strong></a>.</p><p>I think the most useful insight I got from this project is how much money I’ve spent on food and that too from one single source. I had no clue about the magnitude of money I had spent via <a href="https://www.swiggy.com/">Swiggy</a>.</p><p>Until next time …</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f1cce25cbff6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/midnight-hack-episode-1-visualizing-my-swiggy-order-history-f1cce25cbff6">Midnight Hack Episode 1: Visualizing my Swiggy Order History</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>