<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Fanny Kassapian on Medium]]></title>
        <description><![CDATA[Stories by Fanny Kassapian on Medium]]></description>
        <link>https://medium.com/@fannykassapian?source=rss-417fc9684734------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*feho3OJ_9Ib8zppnxQm65Q.png</url>
            <title>Stories by Fanny Kassapian on Medium</title>
            <link>https://medium.com/@fannykassapian?source=rss-417fc9684734------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 07:09:25 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@fannykassapian/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to build a non-geographical map #2]]></title>
            <link>https://medium.com/data-science/how-to-build-a-non-geographical-map-2-340256ad9f16?source=rss-417fc9684734------2</link>
            <guid isPermaLink="false">https://medium.com/p/340256ad9f16</guid>
            <category><![CDATA[leaflet]]></category>
            <category><![CDATA[code]]></category>
            <category><![CDATA[maps]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[Fanny Kassapian]]></dc:creator>
            <pubDate>Fri, 11 Jan 2019 13:17:08 GMT</pubDate>
            <atom:updated>2019-01-23T13:01:49.109Z</atom:updated>
            <content:encoded><![CDATA[<h4>Or how to turn a scatter plot into an interactive map (with Leaflet and JavaScript)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OyDngHAOswm4ogMq8xI-IA.png" /><figcaption>This guy’s body may seem odd and his chair disproportionate, but aren’t maps a distortion of reality, anyway? :)</figcaption></figure><p><em>Maps are powerful design objects for data visualization. They emphasize the spatial relationship between elements and the comparison of variables. In the context of non-geographical data, web maps can enhance traditional scatter plots.</em></p><p><em>🔗 Link to </em><a href="https://towardsdatascience.com/how-to-build-a-non-geographical-map-1-8d3373e83d6c"><em>#part 1: How to map similarities with dimensionality reduction</em></a></p><h3>#Part 2: From scatter plot to interactive map</h3><p>At the center of data visualization, is communication. The ability to transform a frightening table into a compelling story. Often, the ultimate goal is to drive insights for problem-solving or decision making. In most cases, users don’t get to play with data and take a passive stance. But, what if you want to empower your audience, spark their curiosity and allow them to explore data?</p><p>We use maps to navigate the unknown. They give us the confidence to get off the beaten path. We deviate from the initial route and make a stop at a café or a detour by a museum. This is also how we use the web. We make a search, one link leads to another, and we end up developing an interest in something we did not even know existed a few clicks ago.</p><p><strong><em>Could we convey data in the same serendipitous way?</em></strong></p><p>In the <a href="https://towardsdatascience.com/how-to-build-a-non-geographical-map-1-8d3373e83d6c">first part of this series</a>, we used dimensionality reduction to map similarities in a 2D space. We ended up with a new coordinate system that allowed us to visualize our data in a scatter plot (see <a href="https://plot.ly/~fanny_kassapian/1/#/">graph (Plotly)</a>).</p><p>In this article, we will transform it into an interactive map, including a search bar and a side panel. We’ll be using Leaflet and some additional JavaScript.</p><blockquote><strong>This article might be useful to anyone who wants to build a web map, no matter the type of data. If that’s your case, jump to step 2.</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*Rt0GmMiIffHocHiqFEKM8Q.gif" /><figcaption>&gt;&gt; <a href="https://codepen.io/fannykassapian/pen/qJOvbB">See code &amp; end result (CodePen)</a></figcaption></figure><h3>Step 1. Build a JSON of coordinates</h3><p>If you followed along the <a href="https://towardsdatascience.com/how-to-build-a-non-geographical-map-1-8d3373e83d6c">mapping of non-geographical data</a>, you now have an embedding — a (very, very long) Numpy array of coordinates representing your data in 2D.</p><p><strong><em>To plot all these points of data on a Leaflet map, we need them stored into a JavaScript object.</em></strong></p><p>To do so, we can transform our DataFrame into a JSON string and handle it with JavaScript.</p><p>If you only have a few points of data to plot, then you may want to skip this step and store them manually in your JavaScript.</p><p>Also, bear in mind that each user will “load” the entire JSON every time they open the page. This will affect your website’s performance. Depending on the amount of data and traffic you have, you may prefer to load the content dynamically from a data base.</p><p><strong>1/ Create a DataFrame</strong></p><p>In this example, I will be using:</p><ul><li>the <a href="https://towardsdatascience.com/how-to-build-a-non-geographical-map-1-8d3373e83d6c">embedding resulting from the mapping of occupations based on skills and knowledge similarity</a> and</li><li>the <a href="https://www.onetcenter.org/database.html">ONET open database</a>.</li></ul><p>Our JSON must contain each marker’s coordinates and some data, such as its title and description, that we’ll use to update the panel’s content.</p><p>Below is a quick description of the steps that I followed to create the DataFrame. You can find the <a href="https://github.com/fannykassapian/python_json_coordinates/blob/master/UMAP.ipynb">full notebook here (Github)</a>:</p><ul><li>After loading the <a href="https://www.onetcenter.org/dictionary/23.1/excel/occupation_data.html">Occupation Data.xlsx</a> file from O*NET, I created two series. I used the O*NET SOC Code as their common index and occupations titles and descriptions as their respective values. Then, I combined them into a single DataFrame.</li><li>Finally, I created a dictionary of all occupations titles and their respective array of coordinates (based on the UMAP embedding) and mapped it with the DataFrame’s title column.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sETEjB0itnSiELpy1Icyjw.png" /><figcaption>This is what your DataFrame should look like.</figcaption></figure><p><strong>2/ Convert into JSON</strong></p><ul><li>Pandas has a built-in function that enables to convert an object to a JSON string:</li></ul><pre>yourdataframe.to_json(&#39;newfile.json&#39;, orient=&#39;index&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QyVignTJ8Lr2uc5VlM7saw.png" /><figcaption>This is what the JSON looks like.</figcaption></figure><p>For more details, you can find the <a href="https://github.com/fannykassapian/python_json_coordinates/blob/master/UMAP.ipynb">full notebook here (Github)</a>.</p><h3>Step 2. Create a map with Leaflet</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oCnDlju_lObf4u21VYteIQ.png" /><figcaption><a href="https://leafletjs.com/">Leaflet.js</a> is an open-source JavaScript library for responsive &amp; interactive maps. It offers wonderful features and a beautiful default design.</figcaption></figure><p>To create this map, you need <a href="https://leafletjs.com/download.html"><strong>Leaflet</strong></a><strong>, </strong><a href="https://getbootstrap.com/docs/4.1/getting-started/introduction/"><strong>Bootstrap</strong></a><strong>, </strong><a href="https://code.jquery.com/"><strong>jQuery</strong></a><strong> and </strong><a href="https://cdnjs.com/libraries/jqueryui"><strong>jQuery-ui</strong></a>. Make sure to include the links to their CSS and JS libraries in your code.</p><p><strong>1/ Create map</strong></p><p>In this article, I will only use the very basic settings of Leaflet and focus on panel interactivity with the map. However, there are plenty of amazing things you can do with Leaflet, <a href="https://leafletjs.com/examples.html">it’s worth a look</a>.</p><p>Below is a quick set up. Alternatively, you can follow their <a href="https://leafletjs.com/examples/quick-start/">quick start guide</a>.</p><ul><li>In your HTML, create an empty container: &lt;div id=&quot;mapDemo&quot;&gt; &lt;/div&gt;</li><li>The map container must have a defined height. Here, I set it to 100% of the viewer’s screen: #mapDemo {width: 100vw; height: 100vh;}</li><li>In your JS, create the map: map = L.map(&#39;mapDemo&#39;);. Because we’ll have a panel on the left, let’s place the zoom controls at the bottom right of the screen: map.zoomControl.setPosition(&#39;bottomright&#39;);</li></ul><p><strong>2/ Plot markers</strong></p><p>The data is stored under a JSON file with the same structure as a JavaScript dictionary. Note that I based my <a href="https://codepen.io/fannykassapian/pen/qJOvbB">codePen example</a> on a JavaScript dictionary with only 3 elements and 3 features (“title”, “coords” (coordinates) and “description”).</p><ul><li>First, store your JSON file in a JavaScript dictionary var markers = {}:</li></ul><pre>var markers = $.getJSON(&quot;map_data.json&quot;);</pre><ul><li>Then, loop through the dictionary to store each marker’s features under markerOptions. Plot them with addTo(map):</li></ul><pre>$.each(markers, function(key, val) {<br>  var markerOptions = {markerId: key,<br>                       markerTitle: val[&#39;title&#39;],<br>                       markerText: val[&#39;description&#39;]};</pre><pre>  var marker = L.marker(val[&#39;coords&#39;], markerOptions).addTo(map);<br>});</pre><ul><li>To see the markers, set a map view within your data coordinates bounds:</li></ul><pre>var bounds = [[-10,-10], [10,10]];<br>map.fitBounds(bounds);</pre><h3>Step 3. Lay the groundwork</h3><p>Before tackling interactivity, let’s get the HTML and CSS out of the way.</p><p><strong>1/ HTML</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e2BvYmZOiNyQ9d5_kAFK4g.png" /><figcaption>Main elements and their ids. &gt;&gt; <a href="https://codepen.io/fannykassapian/pen/qJOvbB">See CodePen</a></figcaption></figure><ul><li>In the &lt;body&gt; of your HTML, create an input group with a form control (#searchTerm) and a button (#searchBtn).</li><li>Below, add a button (#panelBtn) . It will allow the user to open/close the panel at all times.</li><li>For the panel, add a &lt;div&gt; with a close button (#closeBtn) and the content that will be updated with markers’ data (like a title and a paragraph).</li></ul><p><strong>2/ CSS</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zdmn2n5kDpE8a8MiB4Ql0w.png" /><figcaption>This map is mobile first. I only included a media query for screens that are 500px wide and above, but you may want to refine that.</figcaption></figure><ul><li>In your CSS, make sure that all the elements that should sit on top of the map have a z-index higher than 400 (400 being the z-index of the map).</li><li>By default, the #panel is hidden and is only displayed when .active. Set the left property to the same size as the width of the #panel element to hide it, and switch it to 0 when #panel.active (to show it).</li><li>For the panel height, define where the the element should start and end with top and bottom properties. Note that, if instead of setting bottom, you define a height, the scroll won’t work as you expect. The #panel will continue “outside” of the visible screen (in spite of overflow-y:scroll;).</li><li>On mobile and small screens, the panel and input group take full width. So, I set width:100vw; but change it to width:500px; for screens that are more than 500px wide.</li></ul><p>You can find the <a href="https://codepen.io/fannykassapian/pen/qJOvbB">complete code here (codePen)</a>.</p><p><strong>3/ Data structures</strong></p><p>We have to jump ahead a little and think about the data we will need to access to display the right output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7NPk_K_VZEYiQIY9NNdugg.png" /><figcaption>Overview of the events that require access to markers’ data. In blue are the functions and in turquoise their respective arguments.</figcaption></figure><h4>Dictionaries</h4><p>Let’s call markerOnClick() the function that is called when a marker is clicked, and search() the function that is called after a search is entered (either by clicking on a suggested marker title or by clicking the search button).</p><p>These functions have a bulk of steps in common. To avoid writing the same code twice, we encompass these steps under the updatePanel() function.</p><p>To update the panel with the title and text attached to each markerId, updatePanel() must have access to some other marker features. Likewise, search() is a function of markerTitle but needs access to markerId to call updatePanel().</p><p>For functions to be able to access those features, we need dictionaries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OyksBUBiQTDSxYMPxnxUdA.png" /><figcaption>These are the key-value pairs that we need.</figcaption></figure><ul><li>Create 3 dictionaries: var titleToId = {}; var idToTitle = {}; var idToText = {};</li><li>Then, append them as you loop through the markers{} dictionary:</li></ul><pre>$.each(markers, function(key, val) {<br>...<br>  titleToId[val[&#39;title&#39;]] = key;<br>  idToTitle[key] = val[&#39;title&#39;];  <br>  idToText[key] = val[&#39;description&#39;];<br>...<br>});</pre><h4>List</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2soGh0R1DLEOZUrDkNaAaw.png" /></figure><p>When the user types a search, it calls autocomplete(), which suggests all the possible marker titles based on user input. Thus, the data source for this function must be a list of marker titles:</p><ul><li>Create a list of marker titles var titlesList =[]; that you append within the markers loop:</li></ul><pre>$.each(markers, function(key, val) {<br>...<br>titlesList.push(val[&#39;title&#39;]);<br>...<br>});</pre><h3>Step 4. Orchestrate interactivity</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IAGJ-Gx77DUrML-54ZDJ3w.png" /><figcaption>Overview of the functions associated with panel interactivity.</figcaption></figure><p><strong>1/ Click on marker</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Sl1RWzvrF83yKAqMkGX0tQ.png" /><figcaption>When a marker is clicked: the search bar is updated with the marker’s title, the “open panel” button changes to “close panel”, and the panel opens and displays some content attached to that marker.</figcaption></figure><h4><strong><em>updatePanel()</em></strong></h4><ul><li>Define<strong> </strong>updatePanel():</li></ul><pre>var updatePanel = function(mId){<br>  $(&#39;#panel&#39;).addClass(&#39;active&#39;);<br>  $(&quot;#panelBtn&quot;).text(&#39;&lt; Close Panel&#39;);<br>  var markerTitle = idToTitle[mId];<br>  var markerText = idToText[mId];<br>  $(&#39;#panelTitle&#39;).text(markerTitle);<br>  $(&#39;#panelText&#39;).text(markerText);<br>};</pre><h4><strong><em>markerOnClick()</em></strong></h4><ul><li>And call updatePanel() within markerOnClik() :</li></ul><pre>var markerOnClick = function(){<br>  var mId = this.options.markerId;<br>  var markerTitle = idToTitle[mId];<br>  $(&#39;#searchTerm&#39;).val(markerTitle);<br>  updatePanel(mId);<br>};</pre><ul><li>In the loop, add the onClick event to all markers. Set the popup options (popupContent) and bind the popup to marker click:</li></ul><pre>$.each(markers, function(key, val) {<br>...<br>  marker.on(&#39;click&#39;, markerOnClick);<br>  var popupContent = val[&#39;title&#39;];<br>  marker.bindPopup(popupContent);<br>...<br>});</pre><p>Now, when the user clicks on a marker, the corresponding popup window opens.</p><p><strong>2/ Search input</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9k0wg9_Jp9RsKzXx0lKqKQ.png" /><figcaption>When the user types in the search bar, the autocomplete suggests a few marker titles. From there, the user can either select one of those suggestions or confirm her search. Either way, the output is the same as when a user clicks on a marker (see 1/ Click on marker).</figcaption></figure><h4><strong><em>autocomplete()</em></strong></h4><ul><li>Here, we define the options for autocomplete(). Also, when an item is selected from the autocomplete, we store its value in #searchTerm and call search():</li></ul><pre>$(function() {<br>   $(&quot;#searchTerm&quot;).autocomplete({<br>     source: titlesList,<br>     minLength: 1,<br>     select: function(event, ui) { <br>       $(&quot;#searchTerm&quot;).val(ui.item.label);<br>       console.log(ui.item.label);<br>       search();<br>     }<br>   });<br> });</pre><h4><strong><em>search()</em></strong></h4><p>Now, if the user clicks on a suggestedmarkerTitle, search() and updatePanel() are called. However, without an actual click on the marker, the popup won’t show.</p><p>A solution is to attach a different layer to each marker. We can then apply Leaflet’s openPopup() method to the layers within the search() function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iBtOAcqJvVM7dbG9TzrDKg.png" /><figcaption>LayerGroup is a method from the Layer class used to group several layers and handle them as one &gt;&gt; see <a href="https://leafletjs.com/reference-1.4.0.html#layergroup">documentation</a></figcaption></figure><ul><li>Create a group of layers var layers = L.layerGroup().addTo(map); and update it within the markers loop layers.addLayer(marker)</li><li>A layer is attached to each marker, but we need a dictionary that will allow us to retrieve a marker’s layer based on its markerId: var idToLayer = {};:</li></ul><pre>$.each(fgLayers._layers, function(key,val){<br>  idToLayer[val.options.markerId] = key;<br>});</pre><ul><li>We can finally define the search() function. It gets the markerTitle from the search field, finds its corresponding marker, opens its popup window and calls updatePanel():</li></ul><pre>var search = function(){<br> $(&quot;#searchTerm&quot;).blur(); <br> var markerTitle = $(&quot;#searchTerm&quot;).val();<br> var markerId = titleToId[markerTitle];<br> var layerId = idToLayer[markerId];<br> var layer = layers._layers[layerId];<br> layer.openPopup();</pre><pre> updatePanel(markerId);<br>}</pre><p><strong>3/ Open-close panel</strong></p><p>After users have made a search or clicked a marker, we want them to be able to close the panel and go back to the map at all times. We also want to leave them the option to reopen the panel, from the map directly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KYNVZtyTKmTHd8a8CytONw.png" /><figcaption>To close the panel, the user can either click the close button of the panel or click the “close panel” button on the map.</figcaption></figure><h4>togglePanel()</h4><p>The #panelBtn button allows the user to successively open or close the panel. Its content should change depending on which action (open or close) it allows the user to do next. We can achieve this by using jQuery’s toggleClass method.</p><ul><li>Add an onClick event to the button: &lt;button onClick = &quot;togglePanel()&quot;;&gt; and define togglePanel():</li></ul><pre>var togglePanel = function() {<br>  $(&#39;#panel&#39;).toggleClass(&#39;active&#39;);             <br>  $(&quot;#panelBtn&quot;).text(($(&quot;#panelBtn&quot;).text() == &#39;Open Panel &gt;&#39;) ? &#39;&lt; Close Panel&#39; : &#39;Open Panel &gt;&#39;); <br>}</pre><h4>closePanel()</h4><p>The #closeBtn button allows the user to close the panel when it is open, and updates the #panelBtn inner HTML.</p><ul><li>Add an onClick event to the button: &lt;button onClick = &quot;closePanel()&quot;;&gt; and define closePanel():</li></ul><pre>var closePanel = function() {<br>  $(&#39;#panel&#39;).removeClass(&#39;active&#39;);<br>  $(&#39;#panelBtn&#39;).text(&#39;Open Panel &gt;&#39;);<br>}</pre><h3>Wrap up</h3><h4>Thank you for reading 😃</h4><p>In this series, we’ve covered how to use dimensionality reduction to build a scatter plot that uses distance as a proxy for similarity and turned it into an interactive map.</p><p>I had a lot of fun writing those articles and illustrating them, and learnt a lot along the way. I hope you’ve found it useful, and I look forward to seeing more amazing maps out there.</p><h4>🔗 Link to <a href="https://towardsdatascience.com/how-to-build-a-non-geographical-map-1-8d3373e83d6c">#part 1: How to map similarities with dimensionality reduction</a></h4><h4>👉 Check out how I use this in practice: <a href="http://www.tailoredpath.com/">www.tailoredpath.com</a></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=340256ad9f16" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-to-build-a-non-geographical-map-2-340256ad9f16">How to build a non-geographical map #2</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to build a non-geographical map #1]]></title>
            <link>https://medium.com/data-science/how-to-build-a-non-geographical-map-1-8d3373e83d6c?source=rss-417fc9684734------2</link>
            <guid isPermaLink="false">https://medium.com/p/8d3373e83d6c</guid>
            <category><![CDATA[maps]]></category>
            <category><![CDATA[jobs]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[Fanny Kassapian]]></dc:creator>
            <pubDate>Thu, 30 Aug 2018 11:04:02 GMT</pubDate>
            <atom:updated>2019-06-19T09:07:35.833Z</atom:updated>
            <content:encoded><![CDATA[<h4>Or how to map similarities for serendipitous data exploration (with Python)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*algP9YGeEfWsJEqASMattQ.png" /></figure><p><em>Maps are powerful design objects for data visualization. They are commonly used to emphasize the spatial relationship between elements and compare multiple variables. In the context of non-geographical data, dimensionality reduction techniques allow us to map similarities in a two-dimensional space without compromising the richness of the initial information.</em></p><p><em>🔗 Link to </em><a href="https://towardsdatascience.com/how-to-build-a-non-geographical-map-2-340256ad9f16"><em>#part 2: How to turn a scatter plot into an interactive map</em></a></p><h3>#Part 1: Dimensionality reduction &amp; visualization</h3><h3><strong><em>Step 1. Set up features and initial dimensions</em></strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U6JqXTIZk4VVXAiiQRmq8A.png" /><figcaption><em>In a geographical map, each element’s coordinates are inherently defined by its position. An element’s position is made of three parameters: latitude, longitude and elevation. To represent locations on Earth, the geographic coordinate system uses either a spherical coordinate system (globe) or a cartesian coordinate system (flat map). To transform the Earth’s three-dimensional space into a two-dimensional map, we use map projection techniques, the most common of which is the Mercator’s cylindrical projection.</em></figcaption></figure><p>To visualize your non-geographical data in a map-like way, you need to <strong>think of each element (or point on the map), as defined by a certain set of features (or attributes).</strong></p><p>This set of features must be <strong>constant</strong> across your dataset. Meaning that each element can be described by assigning different values to the same features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BQ9LkNUGPuP6zfEx-Q0R0g.png" /><figcaption>This is what your dataset should look like.</figcaption></figure><p>To identify what attributes would best describe your data, you can ask yourself:</p><p><strong><em>“What intuitively makes two elements similar, and what differentiates them?”</em></strong></p><p>On the map, the closer together, the more similar two elements are. The further away, the more different.</p><h4><strong>Example - </strong>mapping occupations based on skills and knowledge similarity</h4><p>In this example, I want to represent job titles depending on their similarity in terms of required skills and knowledge, using <a href="https://www.onetcenter.org/database.html">ONET open database</a>.</p><p>I’m quite lucky, because this dataset is very well organized. All occupations are described by the same set of skills (same with knowledge), and each of them is broken down by importance (how important this skill is to perform the job) and level (required level of expertise for the job). The “Data Value” columns assigns the respective rating.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fun9BDbCT06PBJKBL84tNg.png" /><figcaption><a href="https://www.onetcenter.org/dictionary/23.0/excel/skills.html">ONET skills.xlsx</a></figcaption></figure><p>I defined each skill and knowledge item as a single, separate feature:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MYVXuEadEp8_767Spj3Jaw.png" /><figcaption>This is what my dataset should look like after cleaning</figcaption></figure><p>Thanks to the quality of the original dataset, this part was fairly easy:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FmRNAz8vzM6vv5n5O9DepQ.png" /><figcaption>📝&gt;&gt; Check out the full notebooks <a href="https://github.com/fannykassapian/python_dimensionality_reduction">here</a></figcaption></figure><p>I am now left with a matrix whose row vectors are jobs (n=640) and whose columns are skills and knowledge parameters (p=134).</p><p>In machine learning, “dimensionality” simply refers to the number of features (or attributes, or variables) in your dataset.</p><p><strong><em>Since each element is described by 134 parameters, we need a 134-dimensional space to fully represent it… Hard to picture, isn’t it?</em></strong></p><p>Let’s see how we can factorize this matrix in order to reduce its representation to a 2-dimensional space.</p><h3>Step 2. Reduce dimensionality</h3><blockquote><strong>W<em>e want to create a lower dimensional representation where we keep some of the structure from the higher dimensional space.</em></strong></blockquote><p>Instinctively, we would like points that were close to each other in the original space to be close to each other in 2D. The same goes for far away points.</p><p>Doing this by hand would be an impossible task. So I looked around for dimensionality reduction algorithms.</p><h4>No free lunch</h4><p>Before we go over a few of them, note that <strong>no one algorithm fits every problem.</strong> This simple (but often forgotten) fact is humorously called the <strong>“no free lunch” theorem</strong>. Each model relies on some assumptions to produce a simplification of reality. But in certain situations, those assumptions fail. Hence, they produce an inaccurate version of reality.</p><p>Consequently, you must pick a few appropriate algorithms depending on the constrains of your data and on the nature of your problem. Then, try them out to see which one works best for you.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*itwuf1H2flZ2EicUr6vFGQ.png" /><figcaption>Dimensionality reduction — Purpose &amp; method (far from exhaustive)</figcaption></figure><p>In my project, for instance, I am using dimensionality reduction for the sole purpose of data visualization, not as a preliminary step before applying a pattern recognition algorithm. Also, I want to produce a map. Therefore, I am interested in the accuracy of the distance between points as a proxy for similarity, but I don’t really care about the interpretability of the new set of axes.</p><p>This leads us to <strong>feature extraction algorithms</strong>.</p><h4><strong><em>a. Linear transformation</em></strong></h4><ul><li><strong><em>Principal Component Analysis (PCA)</em></strong></li></ul><p>A well known and commonly used linear transformation algorithm is Principal Component Analysis (PCA).</p><p>In the matrix above, each job is described by 134 dimensions. It is very likely that to some extent, columns are linearly correlated. When two columns are perfectly correlated, I only need one to describe the other. To reduce the level of inaccuracy of my new representation while dropping dimensions, I would better keep the dimensions that describe most of the variation in my dataset. In other words, get rid of the unnecessary and noisy dimensions, and keep only the most informative ones. But still, dropping 132 dimensions is a lot.</p><p>Instead of identifying the dimensions where there is the most variance, PCA identifies the <strong>directions</strong> where there is the most variance, i.e., where the data is most spread out.</p><p>These directions are called <strong>principal components</strong> (or eigenvectors), and are made of “combinations” of the initial dimensions. Hence, they are much more informative than any single dimension alone, but unlike initial dimensions, they can’t be labeled with a particular feature (hence difficult to interpret).</p><p>PCA finds a <strong>new coordinate system</strong> made of a pair of <strong>orthogonal (uncorrelated) principal components with the highest variance</strong>, and assigns new values to each point in the dataset to position them accordingly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oXQ01435SVuc1jhs" /><figcaption>Illustration of what PCA does</figcaption></figure><p>By nature, <strong>PCA preserves the global structure of the data</strong>, because it finds the coordinate system that will produce the highest variance for the whole dataset, in a “one size fits all” manner. It does not take into account the initial position of points relatively to each other. Rather, it focuses on spreading the data as much as possible in view of the dimensionality reduction.</p><p>If you’re having a hard time visualizing what I just explained, take a look at <a href="http://setosa.io/ev/principal-component-analysis/">this awesome interactive visualization of PCA</a>.</p><p>Ok, let’s try it out:</p><p><strong>1/ Create a “new” set of features with PCA</strong></p><p>First, you need to import PCA from the <a href="http://scikit-learn.org/stable/">scikit-learn</a> library.</p><pre>from sklearn.decomposition import PCA</pre><p>Then, set n_components. If you want to build a two-dimensional coordinate system, you set it up so that PCA finds 2 principal components.</p><pre>pca = PCA(n_components=2)</pre><p>Use fit_transform to fit the model with your DataFrame (X) and apply the dimensionality reduction.</p><pre>pc = pca.fit_transform(X)</pre><p>PCA builds an array of 640 pair of coordinates along the first and second principal components respectively.</p><p><strong>2/ Visualize with Plotly</strong></p><p>Once you’ve imported plotly…</p><pre>from plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplot</pre><pre>init_notebook_mode(connected=True)</pre><pre>import plotly.graph_objs as go</pre><p>…draw a scatter plot using the first and second element of each pair of coordinates as x and y respectively:</p><pre>data = [go.Scatter(x=pc[:,0], y=pc[:,1], mode=’markers’)]</pre><p>My data is not labeled, so the only way to check if its representation is accurate is to hover over points and check if their neighbors “make sense”.</p><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fplot.ly%2F%7Efanny_kassapian%2F3%2F&amp;src=https%3A%2F%2Fplot.ly%2F%7Efanny_kassapian%2F3.embed%3Fautosize%3Dtrue&amp;type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=plot" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/27132322d98f481bac8f92d33c4624a4/href">https://medium.com/media/27132322d98f481bac8f92d33c4624a4/href</a></iframe><p>As you can see, the data is well spread out, but there seem to be a lot of “outliers”. Note that my judgement of whether the algorithm works well with this dataset is highly subjective to my own expectations of the end visualization.</p><p>For this project, I am highly interested in preserving the local structure of the data. I want clusters that were neighbors in the dataset to remain neighbors after the dimensionality reduction. But as mentioned above, that’s not PCA’s best asset.</p><h4><em>b. Non-linear transformation</em></h4><p>The non-linear dimensionality reduction (NLDR) approach assumes that in the high-dimensional space, <strong>the data has some underlying low-dimensional structure lying on an embedded non-linear manifold.</strong></p><p>I’m not a mathematician, and topology is quite a complex field, so I’ll try to explain it metaphorically.</p><p><strong>A manifold resembles Euclidean space near each point.</strong> An Euclidean space is a space where the very first theorems you learnt in geometry, like Thales’s theorem, apply. Two-dimensional manifolds include surfaces (they resemble the Euclidean plane near each point).</p><p>NLDR assumes that, <strong>if the embedded manifold is two-dimensional</strong> (even though the data is 3 -or more- dimensional), <strong>then the data can be represented on a two-dimensional space too.</strong></p><p>A good way to picture it is by looking at your duvet before you’ve made your bed. Imagine you had a rough night and your bed is messy. If you were to describe each point on the duvet, you would need three dimensions. But the duvet resembles the Euclidean plane near each point. So, it can be flattened into a plane (two-dimensional).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LaeepsvlX9BVmYpB" /><figcaption>Illustration of non-linear dimensionality reduction, from 3D to 2D</figcaption></figure><p><strong>Non-linear algorithms are good at preserving the data’s local structure</strong> because they adapt to the underlying data, performing different transformations on different regions of the manifold.</p><ul><li><strong><em>T-distributed Stochastic Neighbor Embedding (t-SNE)</em></strong></li></ul><p>TSNE is a recent, but popular, non-linear visualization method based on local distance measurements, and therefore <strong>preserving close neighbors</strong>. However, <strong>it does not necessarily preserve global structure</strong>. It means it does not always capture the distance between clusters. In the context of my project, that could be an issue.</p><p>If you want to understand better how t-SNE works: <a href="https://distill.pub/2016/misread-tsne/">this article explains how to read t-SNE’s results correctly</a>.</p><p><strong>1/ Create the embedding</strong></p><p>Just like you did for PCA, import SNE from the scikit-learn library, set n_components to 2 and create the embedding with fit_transform:</p><pre>from sklearn.manifold import TSNE</pre><pre>tsne = TSNE(n_components=2)</pre><pre>embedding = tsne.fit_transform(X)</pre><p><strong>2/ Have a look at the result:</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fplot.ly%2F%7Efanny_kassapian%2F5%2F&amp;src=https%3A%2F%2Fplot.ly%2F%7Efanny_kassapian%2F5.embed%3Fautosize%3Dtrue&amp;type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=plot" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/dc8fa08f25ad2cf9120543af822df6d2/href">https://medium.com/media/dc8fa08f25ad2cf9120543af822df6d2/href</a></iframe><p>It looks pretty good already. Let’s try one last method.</p><ul><li><strong><em>Uniform Manifold Approximation and Projection (UMAP)</em></strong></li></ul><p>UMAP, a new-comer, tries to <strong>balance between preserving both local and global distances</strong>, which is exactly what I need.</p><p>UMAP is fairly easy to use, and there are a few parameters that you can adjust (the defaults are quite good):</p><ul><li>n_neighbors: constrains the size of the local neighborhood to will look at when attempting to learn the manifold structure of the data</li><li>n_components: indicates the number of dimensions of the target space</li><li>min_dist: sets the minimum distance between points in the low dimensional representation</li><li>metric: controls how distance is computed in the ambient space of the input data (feature space)</li></ul><p>For more details, you can find the documentation <a href="https://umap-learn.readthedocs.io/en/latest/">here (link to UMAP)</a>.</p><p><strong>1/ Create the embedding</strong></p><p>Once you’ve imported UMAP, you can create the embedding and use the UMAP and fit_transform functions to create the final array of coordinates (full notebook <a href="https://github.com/fannykassapian/python_dimensionality_reduction/blob/master/UMAP.ipynb">here</a>):</p><pre>import umap</pre><pre>embedding = umap.UMAP(n_neighbors=15, n_components=2, min_dist=0.3, metric=’correlation’).fit_transform(X.values)</pre><p><strong>2/ Visualize</strong></p><p>The distribution of points seems pretty logical to me. Clusters are logically positioned relatively to one another :</p><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fplot.ly%2F%7Efanny_kassapian%2F1%2F&amp;src=https%3A%2F%2Fplot.ly%2F%7Efanny_kassapian%2F1.embed%3Fautosize%3Dtrue&amp;type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=plot" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/c266683cbbbc4673b24295f0241f2de7/href">https://medium.com/media/c266683cbbbc4673b24295f0241f2de7/href</a></iframe><h3>Wrap up</h3><h4>Thank you for reading to the end 😃</h4><p>So far, we’ve built a scatter plot with a coordinate system that doesn’t really have any obvious meaning in itself but that uses distance as a proxy for similarity among data features. The visualization is quite basic but it lays out the premises of a map.</p><p>Next, we’ll see <a href="https://towardsdatascience.com/how-to-build-a-non-geographical-map-2-340256ad9f16">how to give it a format that resembles a map in both its aspect and its use (link to #part 2)</a>.</p><p>In the meantime, feel free to share your inspiring non-geographical maps and other dimensionality reduction methodologies.</p><h4>🔗 Link to <a href="https://towardsdatascience.com/how-to-build-a-non-geographical-map-2-340256ad9f16">#part 2: How to turn a scatter plot into an interactive map</a></h4><h4>👉 Check out how I use this in practice: <a href="http://www.tailoredpath.com/">www.tailoredpath.com</a></h4><h4>📫 Let me know what you think: <a href="/service/mailto:tailoredpath@gmail.com">tailoredpath@gmail.com</a></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8d3373e83d6c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-to-build-a-non-geographical-map-1-8d3373e83d6c">How to build a non-geographical map #1</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>