<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Elliot Humphrey on Medium]]></title>
        <description><![CDATA[Stories by Elliot Humphrey on Medium]]></description>
        <link>https://medium.com/@elz1582?source=rss-13e1322246bb------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*qaeGYf2hX7-XPMcoKNS71g.jpeg</url>
            <title>Stories by Elliot Humphrey on Medium</title>
            <link>https://medium.com/@elz1582?source=rss-13e1322246bb------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 10:15:11 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@elz1582/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Spotting Spatiotemporal Patterns in Earthquake Data]]></title>
            <link>https://medium.com/data-science/spotting-spatiotemporal-patterns-in-earthquake-data-b07068b84314?source=rss-13e1322246bb------2</link>
            <guid isPermaLink="false">https://medium.com/p/b07068b84314</guid>
            <category><![CDATA[earthquake]]></category>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[geospatial]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[hands-on-tutorials]]></category>
            <dc:creator><![CDATA[Elliot Humphrey]]></dc:creator>
            <pubDate>Wed, 10 Jan 2024 18:47:07 GMT</pubDate>
            <atom:updated>2024-01-10T18:47:07.921Z</atom:updated>
            <content:encoded><![CDATA[<h4>Use density-based clustering and survival analysis to estimate when earthquakes occur</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sayjdUvygDYDHJvs" /><figcaption>Photo by <a href="https://unsplash.com/@eli_from_prague?utm_source=medium&amp;utm_medium=referral">Eliška Motisová</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>Introduction</h4><p>Whilst we have a fairly good understanding of where and why earthquakes occur, understanding when an earthquake will happen is very challenging. In this article we will use historical earthquake data and a combination of clustering and survival analysis to answer the following questions:</p><p><em>“How are earthquake events spatially distributed?”</em></p><p><em>“If an earthquake happens, what’s the probability of another earthquake happening within the next hour?”</em></p><p><em>“Are there regional differences in the time between earthquake events?”</em></p><h4>Getting the data</h4><p>We will be using data from the USGS[<a href="https://usgs.gov/">1</a>] that records earthquake events (earthquake data resides in the Public Domain and is provided courtesy of the U.S. Geological Survey). They have a nice <a href="https://earthquake.usgs.gov/fdsnws/event/1/">API</a> that means we can get earthquake data in a straightforward way in Python:</p><pre>import http.client<br>import pandas as pd<br>import json<br><br>url = &#39;/fdsnws/event/1/query&#39;<br>query_params = {<br>    &#39;format&#39;: &#39;geojson&#39;,<br>    &#39;starttime&#39;: &quot;2020-01-01&quot;,<br>    &#39;limit&#39;: &#39;10000&#39;,<br>    &#39;minmagnitude&#39;: 3,<br>    &#39;maxlatitude&#39;: &#39;47.009499&#39;,<br>    &#39;minlatitude&#39;: &#39;32.5295236&#39;,<br>    &#39;maxlongitude&#39;: &#39;-114.1307816&#39;,<br>    &#39;minlongitude&#39;: &#39;-124.482003&#39;,<br>}<br>full_url = f&#39;https://earthquake.usgs.gov{url}?{&quot;&amp;&quot;.join(f&quot;{key}={value}&quot; for key, value in query_params.items())}&#39;<br><br>print(&#39;defined params...&#39;)<br><br>conn = http.client.HTTPSConnection(&#39;earthquake.usgs.gov&#39;)<br>conn.request(&#39;GET&#39;, full_url)<br>response = conn.getresponse()</pre><p>In our API request we have a few parameters:</p><ul><li><strong>limit </strong>— <em>The maximum number of earthquake events we want</em></li><li><strong>starttime</strong> — <em>When should the earliest earthquake event be</em></li><li><strong>minmagnitude</strong> — <em>The minimum magnitude of an earthquake event</em></li><li><strong>maxlatitude</strong> — <em>Maximum latitude</em></li><li><strong>minlatitude</strong> — <em>Minimum latitude</em></li><li><strong>maxlongitude</strong> — <em>Maximum longitude</em></li><li><strong>minlongitude</strong> — <em>Minimum longitude</em></li></ul><p>We will set the minimum magnitude to 3, as this is typically what can be felt at the surface, and provide latitude and longitude coordinates to make a bounding box around the US state of California. California is home to the famous <strong>San Andreas Fault</strong>, so will have plenty of earthquake events for us to analyse.</p><p>To handle the GeoJSON response we use the following code:</p><pre>if response.status == 200:<br>    print(&#39;Got a response.&#39;)<br>    data = response.read()<br>    print(&#39;made the GET request...&#39;)<br>    data = data.decode(&#39;utf-8&#39;)<br>    json_data = json.loads(data)<br>    features = json_data[&#39;features&#39;]<br>    df = pd.json_normalize(features)<br><br>    if df.empty:<br>        print(&#39;No earthquakes recorded.&#39;)<br>    else:<br>        df[[&#39;Longitude&#39;, &#39;Latitude&#39;, &#39;Depth&#39;]] = df[&#39;geometry.coordinates&#39;].apply(lambda x: pd.Series(x))<br>        df[&#39;datetime&#39;] = df[&#39;properties.time&#39;].apply(lambda x : datetime.datetime.fromtimestamp(x / 1000))<br>        df[&#39;datetime&#39;] = df[&#39;datetime&#39;].astype(str)<br>        df.sort_values(by=[&#39;datetime&#39;], inplace=True)<br>else:<br>  print(f&quot;Error: {response.status}&quot;)</pre><p>This will return a Pandas DataFrame where each row is an earthquake event, with columns describing event properties (i.e., latitude, longitude, magnitude, id etc.).</p><p>For the geospatially-minded out there, you’ll recognise that the API call coordinates represent a <em>bounding box</em> of earthquakes (i.e., finds earthquakes in a boxed area), however we only care about <strong>earthquakes in California</strong>. Therefore we will need to filter out all the earthquakes that did not occur in California (e.g., earthquake events in Nevada &amp; Oregon). To do this we will use the convenient OSMNX package to get a <strong>MultiPolygon of the California border</strong>:</p><pre>import osmnx<br>import geopandas as gpd<br><br>place = &quot;California, USA&quot;<br>gdf = osmnx.geocode_to_gdf(place)<br># Get the target geometry<br>gdf = gdf[[&quot;geometry&quot;, &quot;bbox_north&quot;, &quot;bbox_south&quot;, &quot;bbox_east&quot;, &quot;bbox_west&quot;]]</pre><p>Next we’ll convert our earthquake coordinates into <strong>Point geometries using Shapely</strong>, and then perform a <strong>spatial join</strong> to filter our non-Californian earthquakes:</p><pre>from shapely.geometry import Point<br># Convert to a GeoDataFrame with Point geometry<br>geometry = [Point(xy) for xy in zip(df[&#39;Longitude&#39;], df[&#39;Latitude&#39;])]<br>earthquake_gdf = gpd.GeoDataFrame(df, geometry=geometry, crs=&#39;EPSG:4326&#39;)<br><br># Filter to keep only points within the California bounding box<br>points_within_california = gpd.sjoin(earthquake_gdf, gdf, how=&#39;inner&#39;, predicate=&#39;within&#39;)<br><br># Extract latitude, longitude etc.<br>df = points_within_california[[&#39;id&#39;, &#39;Latitude&#39;, &#39;Longitude&#39;, &#39;datetime&#39;, &#39;properties.mag&#39;]]</pre><p>Now to see our first map of Californian earthquakes colour-coded by magnitude:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ILpZJ5dbYHSXlEJOFdhgzA.png" /><figcaption>Map of earthquakes in California. Data from the USGS. Image by Author.</figcaption></figure><p>Excellent, we now have a map of California earthquakes!</p><h4>Spatial Clustering of earthquakes</h4><p>Looking at the earthquakes plotted on the map earlier you can see that events are spatially arranged in a linear SE-NW orientation, where you can see around 2–3 clear linear-shaped groupings of earthquakes. This makes sense because:</p><ol><li>Earthquakes occur due to movements along faults, which are linear features (i.e., cracks in the Earth’s crust).</li><li>The orientation of earthquake events matches the orientation of the San Andreas Fault zone.</li></ol><p>Our goal is now to cluster these earthquake events based on their location, so that we can produce spatial clusters associated with regional faults.</p><p>For clustering we will use <strong>HDBSCAN</strong> (Hierarchical Density-Based Spatial Clustering of Applications with Noise) which is a density-based clustering algorithm useful for certain types of datasets, such as those with irregularly shaped clusters and varying cluster densities. This is relevant to our dataset since earthquake events can occur in <strong>irregular shaped clusters</strong> (i.e., along faults of different orientations) and in <strong>different densities</strong> (i.e, some fault zones are more active than others). HDBSCAN is also <strong>robust to outliers</strong>, which is this case can be earthquakes that occur far away from fault zones.</p><p>Through some experimentation the following parameters produce okay results (<em>a bit of trial and error involved, but guided by our prior assumptions on how earthquakes should be clustered</em>):</p><pre># Fit HDBSCAN<br>clusterer = HDBSCAN(min_cluster_size=200, metric=&#39;haversine&#39;, min_samples=20, cluster_selection_epsilon=0.05)<br>result_df[&#39;cluster_label_hdbscan&#39;] = clusterer.fit_predict(data_scaled)<br><br># Find the number of unique cluster labels<br>num_clusters = result_df[&#39;cluster_label_hdbscan&#39;].nunique()<br><br># Create a list of colors<br>colors = [&#39;lightgray&#39;, &#39;red&#39;, &#39;blue&#39;, &#39;green&#39;][:num_clusters]<br><br># Create a Folium map centered at the mean coordinates<br>map_center = [result_df[&#39;Latitude&#39;].mean(), result_df[&#39;Longitude&#39;].mean()]<br>mymap = folium.Map(location=map_center, zoom_start=6)<br><br># Add markers for each data point with cluster color<br>for _, row in result_df.iterrows():<br>    cluster_color = colors[row[&#39;cluster_label_hdbscan&#39;] + 1]  # Map cluster label to color<br>    folium.CircleMarker(<br>        location=[row[&#39;Latitude&#39;], row[&#39;Longitude&#39;]],<br>        radius=2,<br>        color=cluster_color,<br>        fill=True,<br>        fill_color=cluster_color,<br>        fill_opacity=0.7,<br>        popup=f&#39;Cluster: {row[&quot;cluster_label_hdbscan&quot;]}&#39;<br>    ).add_to(mymap)<br>mymap</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qjRvTgjMWO2zGxAD3NFz3w.png" /><figcaption>Earthquake locations color-coded by HDBSCAN cluster. Image by Author.</figcaption></figure><p>Super, we have now used HDBSCAN to cluster our earthquake events into three areas, which corresponds with our prior regional knowledge. <em>Note that some earthquakes are considered outliers and coloured grey.</em></p><h4>Examining the time between earthquakes</h4><p>Since we have the date and time of each earthquake we can calculate <strong>the time between each event</strong> (i.e., the time duration between two earthquakes), which is required for our Survival Analysis in the next section. We have already sorted our values by the <em>‘datetime’ </em>column so they’re in chronological order, now we need to calculate the time difference:</p><pre>from datetime import datetime<br><br># Convert &#39;time&#39; column to datetime objects<br>df[&#39;time&#39;] = pd.to_datetime(df[&#39;datetime&#39;])<br><br># Sort dataframe by time<br>df.sort_values(by=[&#39;time&#39;], inplace=True)<br><br># Calculate time elapsed between consecutive events<br>df[&#39;time_elapsed&#39;] = df[&#39;time&#39;].diff().shift(-1)</pre><p>Since there will be no time difference for the last earthquake (i.e., the most recent) we can calculate the difference from a time we’ll assume as the present day:</p><pre># Assuming present time is &#39;2024-01-08 16:06:00.000&#39;<br>present_time = pd.to_datetime(&#39;2024-01-08 16:06:00.000&#39;)<br><br># For the last event of each group, replace NaN with time between event and present time<br>df[&#39;time_elapsed&#39;].fillna(present_time - df[&#39;time&#39;], inplace=True)</pre><p>Finally we will add a new column indicating whether an earthquake was followed by another earthquake. Whilst this seems slightly redundant, given that only the last earthquake is affected by this, it highlights an interesting point about using survival analysis for these types of tasks (which we’ll cover later):</p><pre># Label events based on whether they happened or not<br>df[&#39;event_happened&#39;] = df[&#39;time_elapsed&#39;].apply(lambda x: 1 if pd.Timedelta(days=0) &gt; 0 else 0)</pre><p>Here is a histogram of the elapsed time between earthquake events:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XArsalhXeq1QYzm_s4KWNg.png" /><figcaption>Histogram of earthquake interval times. Image by Author.</figcaption></figure><p>We can see that the majority of earthquakes occur in short succession of each other (approximately 2.5 hours of each other), with far fewer earthquakes occurring at longer time intervals apart (i.e., the maximum time between earthquakes in this dataset is 350 hours).</p><h4>Survival Analysis of Earthquake Time Intervals</h4><p>We have our earthquake clusters, as well as the time between earthquakes, so now we use Survival Analysis to tell us about the probability of earthquake occurrences. Our goals will therefore be to:</p><ol><li>Use survival analysis to create curves that show the probability of an earthquake occurring in time, given that an earthquake has just occurred.</li><li>Compare our probability curves across our already clustered earthquake regions, to see if there are any similarities/differences in regional earthquake activity.</li></ol><p><em>What is survival analysis?</em></p><p>There are plenty of well-written articles [<a href="https://towardsdatascience.com/the-complete-introduction-to-survival-analysis-in-python-7523e17737e6">2</a>,<a href="https://medium.com/the-researchers-guide/survival-analysis-in-python-km-estimate-cox-ph-and-aft-model-5533843c5d5d">3</a>,<a href="https://towardsdatascience.com/introduction-to-survival-analysis-6f7e19c31d96">4</a>] covering this, but in short, survival analysis is a statistical technique used to analyse time-to-event data, typically in the context of biomedical or observational studies. It focuses on estimating and modelling the time until an event of interest occurs, such as death, failure, or another specific outcome, providing insights into the probability of events happening over time and the impact of covariates on the event occurrence.</p><p><strong>In our case, an earthquake has happened and we want to know the probability of another earthquake occurring over time. </strong><em>Note: we make the assumption that earthquake events are independent, as well as only retrieving events with a minimum magnitude of 3, which is a limitation of this work since it oversimplifies earthquake dynamics.</em></p><p>The <strong>Kaplan-Meier estimator</strong> is a non-parametric method used in survival analysis to estimate the survival function, which will give us our probabilities. We will use the package <a href="https://pypi.org/project/lifelines/"><strong>‘lifelines’</strong></a><strong> </strong>to fit the Kaplan-Meier estimator to the earthquakes in each cluster, to produce a probability curve:</p><pre>import plotly.graph_objects as go<br>from lifelines import KaplanMeierFitter<br><br># Initialize Kaplan-Meier Fitter<br>kmf = KaplanMeierFitter()<br><br># Create a Plotly Figure<br>fig = go.Figure()<br><br>color_map = {0: &#39;red&#39;, 1: &#39;blue&#39;, 2: &#39;green&#39;, -1: &#39;gray&#39;}<br><br># Fit and plot survival curves for each cluster excluding cluster -1<br>for label, group in result_df.groupby(&#39;cluster_label_hdbscan&#39;):<br>    if label != -1:<br>        durations = group[&#39;time_elapsed&#39;].dt.total_seconds() / 3600  # Convert hours to days<br>        event_observed = group[&#39;event_happened&#39;].values<br>        kmf.fit(durations=durations, event_observed=a)<br>        <br>        # Plot survival function<br>        fig.add_trace(go.Scatter(<br>            x=kmf.survival_function_.index,<br>            y=1-kmf.survival_function_.KM_estimate,<br>            mode=&#39;lines&#39;,<br>            name=f&#39;Cluster {label}&#39;,<br>            line=dict(color=color_map[label])<br>        ))<br><br># Customize the plot<br>fig.update_layout(<br>    xaxis_title=&#39;Elapsed Time (hours)&#39;,<br>    yaxis_title=&#39;Probability of experiencing an earthquake&#39;,<br>    legend_title=&#39;Cluster Labels&#39;,<br>    width=800,  # Adjust width<br>    height=500,  # Adjust height<br>    margin=dict(l=50, r=20, t=50, b=50),  # Adjust margins<br>    legend=dict(x=0.8, y=0.99),  # Adjust legend position<br>)<br><br># Show the plot<br>fig.show()</pre><p>Before seeing the results, notice the ‘event_observed’ parameter that uses the ‘event_happened’ column we created earlier. Our final earthquake is an example of a right-censored data point, since there was no following earthquake, and is therefore treated as a partial observation.</p><p>Now to view the curves:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KJhOrffKycO6u9RSnkhvKQ.png" /><figcaption>Survival analysis probability curves for clustered earthquake regions. Image by Author.</figcaption></figure><p>Each cluster of earthquakes has its own coloured curve, where the X axis represents the time elapsed after an earthquake, and the Y axis represents the probability of an earthquake occurring. Some interesting observations:</p><ul><li>The shape of the curves are similar.</li><li>The initial steepness of the curves varies by cluster location.</li></ul><p>Lets focus on the initial time after an earthquake has occurred, since most of our data shows that earthquakes happen in relatively quick succession:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XMFBNfXF11v_9A03QIrwLA.png" /><figcaption>Survival analysis probability curves for clustered earthquake regions. Image by Author.</figcaption></figure><p>This suggests that the <strong>time between earthquakes is not the same regionally</strong>. Looking at the graph we can say that once an earthquake event has happened, the probability of another earthquake happening within ten hours is approximately 35% for cluster 0, 60% for cluster 1, and 48% for cluster 2. This shows that earthquakes in the region that belong to cluster 1 happen in quick success relative to the other two areas.</p><p>We can double check this by looking at a histogram of elapsed time between earthquakes for each cluster location:</p><pre>import plotly.graph_objects as go<br>from plotly.subplots import make_subplots<br>import numpy as np<br><br># Filter data<br>filtered_df = result_df[result_df[&#39;cluster_label_hdbscan&#39;] != -1]<br><br># Create subplots in a 2 by 2 grid<br>fig = make_subplots(rows=2, cols=2, subplot_titles=[f&#39;Cluster {label}&#39; for label in filtered_df[&#39;cluster_label_hdbscan&#39;].unique()],<br>                    shared_xaxes=&#39;all&#39;, shared_yaxes=&#39;all&#39;)<br><br># Create histograms for elapsed time for each cluster<br>for i, (label, group) in enumerate(filtered_df.groupby(&#39;cluster_label_hdbscan&#39;)):<br>    # Calculate histogram<br>    hist_data, bin_edges = np.histogram(group[&#39;time_elapsed&#39;].dt.total_seconds() / 3600, bins=20)<br><br>    # Add histogram trace to subplot<br>    fig.add_trace(go.Bar(<br>        x=bin_edges,<br>        y=hist_data,<br>        opacity=0.5,<br>        name=f&#39;Cluster {label}&#39;,<br>        marker=dict(color=color_map[label])<br>    ), row=(i // 2) + 1, col=(i % 2) + 1)<br><br>    # Customise subplot<br>    fig.update_xaxes(title_text=&#39;Elapsed Time (hours)&#39;, row=(i // 2) + 1, col=(i % 2) + 1)<br>    fig.update_yaxes(title_text=&#39;Frequency&#39;, row=(i // 2) + 1, col=(i % 2) + 1)<br><br># Update layout<br>fig.update_layout(<br>    showlegend=False,<br>    margin=dict(l=50, r=50, t=50, b=50),  # Adjust margins<br>    height=600,<br>    width=800,<br>)<br><br># Show the plot<br>fig.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vb-TEoVKoTUlnMGPVFzwew.png" /><figcaption>Histograms of elapsed time between earthquakes in clustered regions. Image by Author.</figcaption></figure><p>As shown by the probability curves, cluster 1 has longer intervals between earthquake events relative to clusters 2 and 3.</p><h4>Conclusion</h4><p>This work utilised <strong>spatial clustering </strong>and <strong>survival analysis</strong> to unravel temporal and geographic patterns within earthquake data. By employing techniques such as <strong>HDBSCAN</strong> for spatial clustering and the <strong>Kaplan-Meier</strong> estimator for survival analysis, we gained valuable insights into the <strong>regional variations in time between earthquakes</strong>, which was translated into probability curves that can be useful for risk assessment and preparedness in earthquake-prone regions.</p><p>Thanks for reading!</p><p><em>Disclaimer: This article should only be used for learning purposes.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b07068b84314" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/spotting-spatiotemporal-patterns-in-earthquake-data-b07068b84314">Spotting Spatiotemporal Patterns in Earthquake Data</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Finding Patterns in Convenience Store Locations with Geospatial Association Rule Mining]]></title>
            <link>https://medium.com/data-science/finding-patterns-in-convenience-store-locations-with-geospatial-association-rule-mining-5bad7e833299?source=rss-13e1322246bb------2</link>
            <guid isPermaLink="false">https://medium.com/p/5bad7e833299</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[hands-on-tutorials]]></category>
            <category><![CDATA[maps]]></category>
            <category><![CDATA[geospatial]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Elliot Humphrey]]></dc:creator>
            <pubDate>Sat, 01 Apr 2023 03:57:51 GMT</pubDate>
            <atom:updated>2023-04-01T03:57:51.867Z</atom:updated>
            <content:encoded><![CDATA[<h4>Understanding spatial trends in the location of Tokyo convenience stores</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GHV-W8hVdvphUxpf" /><figcaption>Photo by <a href="https://unsplash.com/@shams_of_tabiriz?utm_source=medium&amp;utm_medium=referral">Matt Liu</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>When walking around Tokyo you will often pass numerous convenience stores, locally known as “konbinis”, which makes sense since there are <strong>over </strong><a href="https://www.statista.com/statistics/810901/japan-convenience-store-numbers/"><strong>56,000</strong> </a>convenience stores in Japan. Often there will be different chains of convenience store located very close to one another; it is not uncommon to see stores around the corner from each other or on opposite sides of the street. Given Tokyo’s population density, it is understandable for competing businesses to be forced closer to each other, however, <strong>could there be any relationships between which chains of convenience stores are found near each other?</strong></p><h3>Defining the task</h3><p>The goal will be to collect location data from numerous convenience store chains in a Tokyo neighbourhood, to understand if there are any relationships between which chains are co-located with each other. To do this will require:</p><ul><li>Ability to query the location of different convenience stores in Tokyo, in order to retrieve each store’s name and location</li><li>Finding which convenience stores are co-located with each other within a pre-defined radius</li><li>Using the data on co-located stores to derive association rules</li><li>Plotting and visualising results for inspection</li></ul><p>Let’s begin!</p><h3>Step One: Extracting data using OpenStreetMap</h3><p>For our use case we want to find convenience stores in Tokyo, so first we’ll need to do a little homework on what are the common store chains. A quick Google search tells me that the main stores are <strong>FamilyMart, Lawson, 7-Eleven, Ministop, Daily Yamazaki and NewDays</strong>.</p><p>Now we know what we are searching, lets go to <a href="https://osmnx.readthedocs.io/en/stable/">OSMNX</a>; a great Python package for searching data in OpenStreetMap (OSM). According the OSM’s schema, we should be able to find the store name in either the <strong><em>‘brand:en’</em></strong> or <strong><em>‘brand’</em></strong> field.</p><p>We can start by importing some useful libraries for getting our data, and defining a function to return a table of locations for a given convenience store chain within a specified area:</p><pre>import geopandas as gpd<br>from shapely.geometry import Point, Polygon<br>import osmnx<br>import shapely<br>import pandas as pd<br>import numpy as np<br>import networkx as nx<br><br>def point_finder(place, tags):<br>    &#39;&#39;&#39;<br>    Returns a dataframe of coordinates of an entity from OSM.<br><br>            Parameters:<br>                    place (str): a location (i.e., &#39;Tokyo, Japan&#39;)<br>                    tags (dict): key value of entity attribute in OSM (i.e., &#39;Name&#39;) and value (i.e., amenity name)<br>            Returns:<br>                    results (DataFrame): table of latitude and longitude with entity value <br>    &#39;&#39;&#39;<br><br>    gdf = osmnx.geocode_to_gdf(place)<br>    #Getting the bounding box of the gdf<br>    bounding = gdf.bounds<br>    north, south, east, west = bounding.iloc[0,3], bounding.iloc[0,1], bounding.iloc[0,2], bounding.iloc[0,0]<br>    location = gdf.geometry.unary_union<br>    #Finding the points within the area polygon<br>    point = osmnx.geometries_from_bbox(north,<br>                                       south,<br>                                       east,<br>                                       west,<br>                                       tags=tags)<br>    point.set_crs(crs=4326)<br>    point = point[point.geometry.within(location)]<br>    #Making sure we are dealing with points<br>    point[&#39;geometry&#39;] = point[&#39;geometry&#39;].apply(lambda x : x.centroid if type(x) == Polygon else x)<br>    point = point[point.geom_type != &#39;MultiPolygon&#39;]<br>    point = point[point.geom_type != &#39;Polygon&#39;]<br><br>    results = pd.DataFrame({&#39;name&#39; : list(point[&#39;name&#39;]),<br>                            &#39;longitude&#39; : list(point[&#39;geometry&#39;].x),<br>                            &#39;latitude&#39; : list(point[&#39;geometry&#39;].y)}<br>                            )<br><br>    results[&#39;name&#39;] = list(tags.values())[0]<br>    return results<br><br>convenience_stores = place_finder(place = &#39;Shinjuku, Tokyo&#39;,<br>                                  tags={&quot;brand:en&quot; : &quot; &quot;})</pre><p>We can pass each convenience store name and combine the results into a single table of store name, longitude and latitude. For our use case we can focus on the <strong>Shinjuku</strong> neighbourhood in Tokyo, and see what the abundance of each convenience store looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2ECGMC3vfYYug2HX5cxiRA.png" /><figcaption>Frequency count of convenience stores. Image by author.</figcaption></figure><p>Clearly FamilyMart and 7-Eleven dominate in the frequency of stores, but how does this look spatially? Plotting geospatial data is pretty straightforward when using <a href="https://kepler.gl/"><strong>Kepler.gl</strong></a>, which includes a nice interface for creating visualisations which can be saved as html objects or visualised directly in Jupyter notebooks:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pTt26bK2PtX3_ylQ_38EGg.png" /><figcaption>Location map of Shinjuku convenience stores, colour coded by store name. Image by author.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mFGqa3Xh99ikHjmelK-IDg.png" /><figcaption>Location map of Shinjuku convenience stores, colour coded density in a two minute walking radius (168m). image by author.</figcaption></figure><h3>Step Two: Finding nearest neighbours</h3><p>Now that we have our data, the next step will be to find nearest neighbours for each convenience store. To do this, we will be using Scikit Learn’s <a href="https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.BallTree.html">‘BallTree’ class</a> to find the names of the nearest convenience stores within a two minute walking radius. We are not interested in how many stores are considered nearest neighbours, so we will just look at which convenience store chains are within the defined radius.</p><pre># Convert location to radians<br>locations = convenience_stores[[&quot;latitude&quot;, &quot;longitude&quot;]].values<br>locations_radians =  np.radians(locations)<br><br># Create a balltree to search locations<br>tree = BallTree(locations_radians, leaf_size=15, metric=&#39;haversine&#39;)<br><br># Find nearest neighbours in a 2 minute walking radius<br>is_within, distances = tree.query_radius(locations_radians, r=168/6371000, count_only=False, return_distance=True)<br><br># Replace the neighbour indices with store names<br>df = pd.DataFrame(is_within)<br>df.columns = [&#39;indices&#39;]<br>df[&#39;indices&#39;] = [[val for val in row if val != idx] for idx, row in enumerate(df[&#39;indices&#39;])]<br><br># create temporary index column<br>convenience_stores = convenience_stores.reset_index()<br># set temporary index column as index<br>convenience_stores = convenience_stores.set_index(&#39;index&#39;)<br># create index-name mapping<br>index_name_mapping = convenience_stores[&#39;name&#39;].to_dict()<br><br># replace index values with names and remove duplicates<br>df[&#39;indices&#39;] = df[&#39;indices&#39;].apply(lambda lst: list(set(map(index_name_mapping.get, set(lst)))))<br># Append back to original df<br>convenience_stores[&#39;neighbours&#39;] = df[&#39;indices&#39;]<br><br># Identify when a store has no neighbours<br>convenience_stores[&#39;neighbours&#39;] = [lst if lst else [&#39;no-neighbours&#39;] for lst in convenience_stores[&#39;neighbours&#39;]]<br><br># Unique store names<br>unique_elements = set([item for sublist in convenience_stores[&#39;neighbours&#39;] for item in sublist])<br># Count each stores frequency in the set of neighbours per location<br>counts = [dict(Counter(row)) for row in convenience_stores[&#39;neighbours&#39;]]<br><br># Create a new dataframe with the counts<br>output_df = pd.DataFrame(counts).fillna(0)[sorted(unique_elements)]</pre><p>If we want to improve the accuracy of our work, we could replace the haversine distance measure for something more accurate (i.e., walking times calculated using <a href="https://networkx.org/">networkx</a>), but we’ll keep things simple.</p><p>This will give us a DataFrame where each row corresponds to a location, and a binary count of which convenience store chains are nearby:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7xDw4gWmhlnOief4nW8swA.png" /><figcaption>Sample DataFrame of convenience store nearest neighbours for each location. Image by author.</figcaption></figure><h3>Step Three: Association Rule Mining</h3><p>We now have a dataset ready to perform association rule mining. Using the <a href="https://github.com/rasbt/mlxtend">mlxtend</a> library we can derive association rules using the <a href="https://www.geeksforgeeks.org/apriori-algorithm/">Apriori algorithm</a>. There is a minimum <strong>support</strong> of 5%, so that we can examine only the rules related to frequent occurrences in our dataset (i.e., co-located convenience store chains). We use the metric ‘lift’ when deriving rules; <strong>lift</strong> is the ratio of the proportion of locations that contain both the antecedent and consequent relative to the expected support under the assumption of independence.</p><pre>from mlxtend.frequent_patterns import association_rules, apriori<br><br># Calculate apriori<br>frequent_set = apriori(output_df, min_support = 0.05, use_colnames = True)<br># Create rules<br>rules = association_rules(frequent_set, metric = &#39;lift&#39;)<br># Sort rules by the support value<br>rules.sort_values([&#39;support&#39;], ascending=False)</pre><p>This gives us the following results table:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sXnwZr4E0Ap0cdrB7dRCkg.png" /><figcaption>Association rules for convenience store data. Image by author.</figcaption></figure><p>We will now interpret these association rules to make some high level takeaway learnings. To interpret this table its best to read more about Association Rules, using these links:</p><ul><li><strong><em>Complete Guide to Association Rules </em></strong>— <a href="https://towardsdatascience.com/association-rules-2-aa9a77241654">https://towardsdatascience.com/association-rules-2-aa9a77241654</a></li><li><strong><em>Association Rules with Apriori Algorithm — </em></strong><a href="https://towardsdatascience.com/association-rules-with-apriori-algorithm-574593e35223">https://towardsdatascience.com/association-rules-with-apriori-algorithm-574593e35223</a></li><li><strong><em>Understanding Association Rules — </em></strong><a href="https://towardsdatascience.com/a-simple-way-to-understand-association-rule-from-the-customer-basket-analysis-use-case-c7bcd75bdec1">https://towardsdatascience.com/a-simple-way-to-understand-association-rule-from-the-customer-basket-analysis-use-case-c7bcd75bdec1</a></li></ul><p>Okay, back to the table.</p><p>Support is telling us how often different convenience store chains are actually found together. Therefore we can say that 7-Eleven and FamilyMart are found together in ~31% of the data. A lift over 1 indicates that the presence of the antecedent increases the likelihood of the consequent, suggesting that the locations of the two chains are partially dependent. On the other hand, the association between 7-Eleven and Lawson shows a higher lift but with a lower confidence.</p><p>Daily Yamazaki has a low support near our cutoff and shows a weak relationship with the location of FamilyMart, given by a lift slightly above 1.</p><p>Other rules are referring to combinations of convenience stores. For example when a 7-Eleven and FamilyMart are already co-located, there is a high lift value of 1.42 that suggests a strong association with Lawson.</p><h3>So why is this useful?</h3><p>If we had just stopped at finding the nearest neighbours for each store location, we would not have been able to determine anything about the relationships between these stores.</p><p>An example of why geospatial association rules can be insightful for businesses is in determining new store locations. If a convenience store chain is opening a new location, association rules can help to identify which stores are likely to co-occur.</p><p>The value in this becomes clear when tailoring marketing campaigns and pricing strategies, as it provides quantitative relationships about which stores are likely to compete. Since we know that FamilyMart and 7-Eleven often co-occur, which we demonstrate with association rules, it would make sense for both of these chains to pay more attention to how their products compete relative to other chains such as Lawson and Daily Yamazaki.</p><h3>Conclusion</h3><p>In this article we have created geospatial association rules for convenience store chains in a Tokyo neighbourhood. This was done using data extraction from OpenStreetMap, finding nearest neighbour convenience store chains, visualising data on maps, and creating association rules using an Apriori algorithm.</p><p>Thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5bad7e833299" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/finding-patterns-in-convenience-store-locations-with-geospatial-association-rule-mining-5bad7e833299">Finding Patterns in Convenience Store Locations with Geospatial Association Rule Mining</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Geospatial Site-Selection Analysis Using Cosine Similarity]]></title>
            <link>https://medium.com/data-science/geospatial-site-selection-analysis-using-cosine-similarity-dcde4be458e7?source=rss-13e1322246bb------2</link>
            <guid isPermaLink="false">https://medium.com/p/dcde4be458e7</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[coffee-shop]]></category>
            <category><![CDATA[location]]></category>
            <category><![CDATA[geospatial]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Elliot Humphrey]]></dc:creator>
            <pubDate>Fri, 29 Jul 2022 18:11:36 GMT</pubDate>
            <atom:updated>2022-07-29T18:11:36.565Z</atom:updated>
            <content:encoded><![CDATA[<h4>Identifying similarities between geographic areas based on neighbourhood amenities</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gdZDWn4bLvXj1TB5" /><figcaption>Photo by <a href="https://unsplash.com/@wyron?utm_source=medium&amp;utm_medium=referral">Wyron A</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3><strong>Introduction</strong></h3><p>Location is paramount for businesses that operate <strong>physical locations</strong>, where it is key to be located close to your target market.</p><p>This challenge is often the case for franchises that are expanding into new areas where it is important to understand the <em>‘fit’</em> of a business in a new area. The aim of this article is to explore this idea in more detail, <strong>to evaluate the suitability of a new location for a franchise based on the characteristics of areas where existing franchises are located</strong>.</p><p>To achieve this we will be taking data from <strong>OpenStreetMap</strong> of a popular coffee shop franchise in Seattle, to use information about the surrounding neighbourhood to identify new prospective locations that are similar.</p><h3><strong>Framing the problem</strong></h3><p>To approach this task there are a few steps that need to be considered:</p><ol><li><em>Finding existing franchise locations</em></li><li><em>Identifying nearby amenities around these locations (which we will assume gives us an idea about the neighbourhood)</em></li><li><em>Finding new potential locations and their nearby amenities (repeating steps 1 &amp; 2)</em></li><li><em>Evaluating the similarity between potential and existing locations</em></li></ol><p>As this task is geospatial, using OpenStreetMap and packages like <strong><em>OSMNX</em></strong> and <strong><em>Geopandas</em></strong> will be useful.</p><h3><strong>Collecting the data</strong></h3><p><strong><em>Finding existing franchise locations</em></strong></p><p>As mentioned, we will use a popular coffee shop franchise to define the existing locations of interest. Collecting this information is rather straightforward using OSMNX, where we can define the geographic place of interest. I have set the <strong>place of interest as Seattle (USA)</strong>, and defined the name of the franchise using the name/brand tag in OpenStreetMap.</p><pre>import osmnx<br>place = &#39;Seattle, USA&#39;<br>gdf = osmnx.geocode_to_gdf(place)</pre><pre>#Getting the bounding box of the gdf<br>bounding = gdf.bounds<br>north, south, east, west = bounding.iloc[0,3], bounding.iloc[0,1], bounding.iloc[0,2], bounding.iloc[0,0]<br>location = gdf.geometry.unary_union</pre><pre>#Finding the points within the area polygon<br>point = osmnx.geometries_from_bbox(north, south, east, west, tags={brand_name : &#39;coffee shop&#39;})<br>point.set_crs(crs=4326)<br>point = point[point.geometry.within(location)]</pre><pre>#Making sure we are dealing with points<br>point[&#39;geometry&#39;] = point[&#39;geometry&#39;].apply(lambda x : x.centroid if type(x) == Polygon else x)<br>point = point[point.geom_type != &#39;MultiPolygon&#39;]<br>point = point[point.geom_type != &#39;Polygon&#39;]</pre><p>This gives us the locations of existing franchise locations with our area:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2A5NVBlJuykDDWm__z6JmA.png" /><figcaption>Map of existing franchise locations (blue points) within the defined Seattle area (purple polygon). Image by Author.</figcaption></figure><p>Looking at the existing locations makes us wonder about the following:</p><ol><li><em>What is the density of the franchise in this areas?</em></li><li><em>What is the spatial distribution of these locations (clustered close together or evenly spread out)?</em></li></ol><p>To answer these questions we can calculate the franchise density using the defined area polygon and the count of existing franchises, which gives us 0.262 per SqKm. <em>Note: large areas in the polygon are water, therefore the density will appear much lower here than in reality…</em></p><p>For measuring how dispersed these locations are relative to each other we can calculate the distance to the nearest neighbour using <em>Sklearn’s </em><strong><em>Ball</em>Tree:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XDpxI9qMXGK9zoIssTVn6Q.png" /><figcaption>Map of existing franchise locations, coloured by distance to nearest neighbour (in Km), within the defined Seattle area (purple polygon). Image by Author.</figcaption></figure><p>Nearest neighbours can also be shown as a histogram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_9b67IxEmVwqbM9xBMMHxw.png" /><figcaption>Nearest neighbour histogram. Image by Author.</figcaption></figure><p>It looks like the majority of locations exist with 800m of each other, which is apparent when looking at the map and seeing the high density of existing locations in the city centre.</p><p><strong><em>What about the amenities surrounding these locations?</em></strong></p><p>We first need to get all the amenities within an area of interest and define a radius around each existing location, that can be used to identify nearby amenities. This can be achieved using another BallTree, however querying points based on a specified radius (which I’ve set as 250m):</p><pre>from sklearn.neighbors import BallTree</pre><pre>#Defining the tree based on lat/lon values converted to radians<br>ball = BallTree(amenities_points[[&quot;lat_rad&quot;, &quot;lon_rad&quot;]].values, metric=&#39;haversine&#39;)</pre><pre>#Querying the tree of amenities using a radius around existing locations<br>radius = k / 6371000<br>indices = ball.query_radius(target_points[[&quot;lat_rad&quot;, &quot;lon_rad&quot;]], r = radius)<br>indices = pd.DataFrame(indices, columns={&#39;indices&#39;})</pre><p>When we query OSM and use the BallTree to find nearby amenities we are left with the indices of each amenity within the radius of an existing franchise location. Therefore we need to extract the amenity type (e.g., restaurant) and count each occurrence to get a processed dataframe like the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qtUYjcmadnpgxjydL8lgcA.png" /><figcaption>Example dataframe summarising amenities within a radius around existing locations. Each row represents an existing franchise location. Image by Author.</figcaption></figure><p>Now we can see the most popular amenities located near existing franchise locations in a sorted bar chart:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xjEZ_zQ37D-hWKd7kPQd1w.png" /><figcaption>Bar chart of amenities within the radii of existing franchise locations. Image by Author.</figcaption></figure><p>It appears that our coffee shop franchise is dominantly located proximal to other locations that serve food/drinks, along with a few other minority amenities like ‘charging station’. This gives us the total count for all existing locations, <strong><em>but are the distribution of amenities the same?</em></strong></p><p>We can apply quick <strong>PCA</strong> and <strong>DBSCAN</strong> clustering to see how existing franchise locations cluster relative to each other (using a min_sample value of 3):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/1*vlpeERV3kcJvuk5D8JUfCA.png" /><figcaption>DBSCAN clustered franchise locations, coloured by cluster label. Image by Author.</figcaption></figure><p>There is a dominant cluster towards the left, however other smaller clusters exist too. This is important as it tells us that existing franchise locations <strong>vary</strong> based on their surrounding amenities and <strong>do not conform</strong> <strong>to a single ‘type’ of neighbourhood</strong>.</p><h3><strong>Finding new locations</strong></h3><p>Now that we have created a dataset for existing franchise locations, we can now produce a similar dataset for new potential locations. We can randomly select nodes that exist in our area of interest using the graph extracted by OSMNX, as points will be constrained to existing paths accessible for walking:</p><pre>G = osmnx.graph_from_place(place, network_type=&#39;walk&#39;, simplify=True)<br>nodes = pd.DataFrame(osmnx.graph_to_gdfs(G, edges=False)).sample(n = 500, replace = False)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L59PsvGsioYO736hWYZ7MA.png" /><figcaption>Randomly sampled locations (blue) that we will compare against the existing franchise locations. Image by Author.</figcaption></figure><p>Finding nearby amenities for each of these potential locations can be achieved by repeating the previous steps…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eP6YOnKFVvxA9e51SGjObA.png" /><figcaption>A slightly messy location map highlighting existing franchises (black), new prospective locations (dark green), and all amenities in the area (coloured by amenity type).</figcaption></figure><p><strong>Measuring the similarity between existing and potential locations</strong></p><p>This is the part we’ve been waiting for; measuring the similarity between existing and our potential locations. We will use the <strong>pairwise cosine similarity</strong> to achieve this, where each location consists of a vector based on the diversity and quantity of amenities nearby. Using cosine similarity offers two benefits in this geospatial context:</p><ol><li><strong>Vector lengths do not need to match</strong> = We can still measure similarities between locations with different types of amenities.</li><li><strong>Similarity is not based on just the frequency of amenities</strong> = Since we also care about the diversity of amenities, not just the magnitude.</li></ol><p>We calculate the cosine similarity of a potential new location against all other existing locations, which means that we have multiple similarity score.</p><pre>max_similarities = []<br>for j in range(len(new_locations)):<br>similarity = []<br>for i in range(len(existing_locations)):<br>cos_similarity = cosine_similarity(new_locations.iloc[[j]].values, existing_locations.iloc[[i]].values).tolist()<br>similarity.extend(cos_similarity)<br>similarity = [np.max(list(chain(*similarity)))]<br>average_similarities.extend(similarity)<br>node_amenities[&#39;averaged similarity score&#39;] = max_similarities</pre><p><strong>So how do we define what is similar?</strong></p><p>For this, we can <strong>remember that existing locations do not form a single cluster,</strong> meaning there is <strong>heterogeneity</strong>. A good analogy can be when evaluating a person can join a friendship group:</p><ol><li>A friendship group typically will contain different people with varying traits, and not a group of people with identical traits.</li><li>Within a group, people will share more or less traits with different members of the group.</li><li>A new potential member doesn&#39;t necessarily need to be similar to everyone in the group to be considered a good fit.</li></ol><p>Because of this, we chose the <strong>maximum</strong> similarity score when comparing with existing locations, as this tells us that the potential location is <strong>similar to at least one other existing franchise locatio</strong>n. Taking an average would lead to lower score, since variation in nearby amenities exists between franchise locations. We can now plot the results, <strong>coloured by similarity score</strong>, to see what areas could be new franchise locations:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v5Tg99PP6WbdcfQ2bIf7Rg.png" /><figcaption>Potential locations coloured by similarity score (where yellow represents high % similarity and purple means no similarity), alongside existing franchise locations (black). Image by Author.</figcaption></figure><p>Our similarity scores for locations in the city centre all show strong similarities with existing locations, which makes sense, therefore what we’re really interest in are the locations with <strong>high similarity scores</strong> that are positioned <strong>far away from existing franchise locations</strong> (black points on the map).</p><h3><strong>Final thoughts</strong></h3><p>We have used <strong>geospatial</strong> techniques to evaluate the potential locations for a coffee shop franchise to expand into new locations, using <strong>cosine similarity</strong> scores based on <strong>nearby amenities</strong>. This is just a small piece of a bigger picture, where factors such as population density, accessibility etc. should also be taken into account.</p><p>Keep an eye out for the <strong>next article</strong> where will start to mature this idea further with some modelling. <strong>Thanks for reading!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dcde4be458e7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/geospatial-site-selection-analysis-using-cosine-similarity-dcde4be458e7">Geospatial Site-Selection Analysis Using Cosine Similarity</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimising the Location of Food Delivery Couriers Using Geopandas and OpenStreetMap]]></title>
            <link>https://medium.com/data-science/optimising-the-location-of-food-delivery-couriers-using-geopandas-and-openstreetmap-e771433e5e9f?source=rss-13e1322246bb------2</link>
            <guid isPermaLink="false">https://medium.com/p/e771433e5e9f</guid>
            <category><![CDATA[maps]]></category>
            <category><![CDATA[geospatial]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[Elliot Humphrey]]></dc:creator>
            <pubDate>Fri, 04 Feb 2022 04:32:20 GMT</pubDate>
            <atom:updated>2022-02-04T04:32:20.372Z</atom:updated>
            <content:encoded><![CDATA[<h4>Where should couriers position themselves to have the best access to the maximum number of restaurants?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*l0700yHxBs_G0Bh1" /><figcaption>Photo by <a href="https://unsplash.com/@rowanfreeman?utm_source=medium&amp;utm_medium=referral">Rowan Freeman</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>A food delivery courier in the ‘gig economy’ wants to handle as many food pickups and deliveries as possible. <strong>This article will look at how using geospatial tools such as OpenStreetMap and Geopandas can be used to optimise the location of food delivery couriers to be best positioned in areas with a high density of restaurants, increasing the chance of food delivery pickups.</strong></p><p>If we step into the shoes of a food delivery courier we can ask ourselves the following question:</p><blockquote>“Where should I be located to receive more food delivery pickups from restaurants?”</blockquote><p>This question relates to <strong>finding the area(s) with the highest density of potential opportunities</strong> (i.e. food delivery pickups). For simplicity we will not consider the popularity of a restaurant, as this will of course affect the number of food deliveries, and therefore stick to thinking about things in a spatial context, such as the location of restaurants.</p><h3>Importing and installing the relevant libraries</h3><p>We will use the <em>osmnx</em> library to query <strong>OpenStreetMap</strong> and retrieve the data we need. Additional examples will help us with handling geospatial data, mainly <em>geopandas</em>. We can start by installing and importing the libraries we need:</p><pre>#Installing and importing libraries<br>!pip install osmnx<br>!pip install contextily folium==0.12.1.post1 geopandas==0.10.2 matplotlib mapclassify==2.4.3 pygeos==0.12.0 pyproj==3.2.1</pre><pre>import geopandas as gpd<br>from shapely.geometry import Point, Polygon<br>import contextily as cx<br>import osmnx as ox<br>import shapely<br>import pandas as pd</pre><h3>Extracting restaurant data</h3><p>Using osmnx we will start by finding all the restaurants in a city; we’ll use Oslo, Norway as our test case since the number of restaurants will be smaller (and more manageable) compared to larger cities. This is very straightforward since luckily Oslo can be readily recognised as a place:</p><pre>#choosing area of interest<br>place = &#39;oslo&#39;</pre><pre>#Selecting amenities and extracting information<br>tags = {&quot;amenity&quot;:&quot;restaurant&quot;}<br>food = ox.geometries_from_place(place, tags=tags)<br>print(food.shape)</pre><pre>food = food[[&#39;name&#39;, &#39;geometry&#39;]]</pre><p>This produces a geodataframe with a shape <em>(527, 114)</em>. There are lots of redundant columns of metadata about these restaurants that we can remove since we only care about the location, which is stored as either a <em>Point</em> or <em>Polygon</em> in the ‘Geometry’ column.</p><h3><strong>Data processing</strong></h3><p>Since polygons are also returned as part of this geodataframe, we can quickly convert these to Points by taking the <strong>centroid of the polygon</strong>. All our rows of restaurant data will then correspond to a pointset of latitudes and longitudes:</p><pre>#Convert polygons to points<br>food[&#39;geometry&#39;] = food[&#39;geometry&#39;].apply(lambda x : x.centroid if type(x) == Polygon else x)</pre><p>For the sake of plotting we will change the crs of the pointset coordinates, which can make it easier when plotting on basemaps (e.g. from Contextily), and clean up our columns. Finally we can use Geopanda’s <em>explore() </em>to produce an interactive map for visualising restaurant data:</p><pre>#Simplifying the df and converting crs in case of plotting on other map tiles<br>food = food.to_crs(epsg=3857)<br>food.explore()</pre><figure><img alt="A map of plotted restaurant data" src="https://cdn-images-1.medium.com/max/1024/1*eb0eXHM7Dd4IG8Hro3qMUA.png" /><figcaption>Plotted restaurant data from OSM in Oslo. Image by Author.</figcaption></figure><p>So now we can see where restaurants are distributed. As expected, there are areas where restaurants occur in higher densities. These will be of interest to our hypothetical delivery couriers, since they should be better positioned to receive orders from more restaurants. Let’s zoom in to see how restaurants are distributed in the centre:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h04awF-YoLVYYPg-kGs_xg.png" /><figcaption>Plotted restaurant data from OSM in Oslo. Image by Author.</figcaption></figure><p>We’ve moved into the specifics of our problem statement, which is selecting a place(s) on our map of Oslo where couriers will be available to receive orders from more restaurants. <strong><em>Where do you think that location would be on the above map?</em></strong> Take a guess and find out at the end…</p><p>In this situation, we will say that a delivery courier is available to receive an order from a restaurant if they are within a five minute walk, which translates to about 400 meters. If we <strong>add a 400 meter buffer</strong> around all our restaurants, we can get a better picture of where these buffers <strong>overlap</strong> (i.e. locations that are within range of multiple restaurants will appear more opaque):</p><pre>food[&#39;geometry&#39;] = food.buffer(400)<br>food.explore(style_kwds={&#39;stroke&#39;:False, &#39;fillOpacity&#39;:0.05})</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9gcSKdMUwoznaYfl8fTidg.png" /><figcaption>Plotted restaurant buffers from OSM in Oslo. Image by Author.</figcaption></figure><h3>Counting restaurant intersections</h3><p>We’re getting a better idea of where to be positioned now, but how can we improve this? It would be nice to <strong>quantify</strong> the areas which have more overlapping restaurants buffers. This solution already exists in <a href="https://pro.arcgis.com/en/pro-app/latest/tool-reference/analysis/count-overlapping-features.htm">ArcGIS Pro</a>, which creates polygons and counts the number of intersecting features, but I found a great solution that works well <a href="https://gis.stackexchange.com/questions/387773/count-overlapping-features-using-geopandas">here</a>. This now gives a geodataframe of <strong><em>newly created intersection polygons</em></strong>, which includes an intersection count. We can sort these polygons by the number of intersects and convert the polygons to points to get the top ten locations which intersect the most restaurant buffers:</p><pre>#Count the overlapping features of restaurant buffers<br>food_intersections = count_overlapping_features(food)</pre><pre>food_intersections = food_intersections.sort_values(&#39;count_intersections&#39;, ascending=False)<br>food_intersections[&#39;geometry&#39;] = food_intersections.centroid</pre><pre>#Plot results<br>food_intersections[:10].explore(marker_type = &#39;circle&#39;, marker_kwds = {&#39;radius&#39;:20, &#39;fill&#39;:True})</pre><h3>The optimised locations for restaurant food pickup are…?</h3><p>This gives us the following locations which, based on the buffer radius and location of Oslo restaurants, position delivery couriers to be available to the highest number of restaurants:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cZQFa0H7oOzFEi6TbT8hOQ.png" /><figcaption>Map of locations that feature the highest number of restaurant buffer intersections. Image by Author.</figcaption></figure><p>Interestingly, we see that we have a single cluster of location points (i.e lots of intersection polygons close together) as well as two much smaller areas either side to the west and east. Selecting the top 10 centroids based on intersection count means that each of these centroids intersects approximately 25 restaurants. <strong><em>So, how did that compare to your guess? </em></strong>We can now recommend to our hypothetical food delivery couriers that they have a higher chance of receiving food pickup orders if they are located in the areas we discovered.</p><h3>Summary</h3><p>In this article we have thought about <strong>how food delivery couriers can be better positioned to receive more pickup orders from restaurants</strong> and<strong> </strong>reduce the time spent idle. By extracting restaurant data, creating hypothetical ‘availability’ buffers, and find areas where the most buffers overlap, we can pinpoint optimum locations to wait for pickup orders. OpenStreetMap is a great dataset, so why now try this out in another city and see what the distribution of best spots looks like.</p><p><strong>Thanks for reading!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e771433e5e9f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/optimising-the-location-of-food-delivery-couriers-using-geopandas-and-openstreetmap-e771433e5e9f">Optimising the Location of Food Delivery Couriers Using Geopandas and OpenStreetMap</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Geospatial Data to Discover Volcanoes in Hawaii]]></title>
            <link>https://medium.com/data-science/using-geospatial-data-to-discover-volcanoes-in-hawaii-67a15c18fa8f?source=rss-13e1322246bb------2</link>
            <guid isPermaLink="false">https://medium.com/p/67a15c18fa8f</guid>
            <category><![CDATA[geospatial]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[segmentation]]></category>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[hands-on-tutorials]]></category>
            <dc:creator><![CDATA[Elliot Humphrey]]></dc:creator>
            <pubDate>Tue, 04 Jan 2022 22:58:17 GMT</pubDate>
            <atom:updated>2022-01-05T16:45:36.849Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsdatascience.com/tagged/hands-on-tutorials">Hands-on Tutorials</a></h4><h4>Geospatial image segmentation using topographic data and Transfer Learning</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*80-CNOVtGYRZ4xelwIpfEQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@marcszeglat?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Marc Szeglat</a> on <a href="https://unsplash.com/s/photos/volcano-hawaii?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Practical examples of semantic image segmentation, which aims to identify features on a pixel-wise basis, range from <a href="https://smartlabai.medium.com/deep-learning-based-semantic-segmentation-in-simulation-and-real-world-for-autonomous-vehicles-e7fe25cef816">self-driving cars</a> to <a href="https://towardsdatascience.com/deep-learning-for-ship-detection-and-segmentation-71d223aca649">detecting ships in satellite images</a>. But what about practical applications which <strong>do not</strong> use these conventional image datasets?</p><p>In this article I’ll be walking through a <strong>geospatial example of semantic segmentation</strong>, where we’ll be <strong>mapping cinder cones in Hawaii</strong> using Digital Elevation Model (DEM) data.</p><p>We’ll start with a quick introduction to <strong>what cinder cones are</strong>, and how to quickly <strong>create a labelled dataset</strong> using <a href="https://qgis.org/en/site/index.html">QGIS</a>. Then we’ll go through some <strong>data processing</strong> and model selection / training before <strong>testing our trained model</strong> on a new data.</p><h3>So what are cinder cones?</h3><p>Based on the definition by the <a href="https://pubs.usgs.gov/gip/volc/types.html">United States Geological Survey</a>:</p><blockquote>“Cinder cones are the simplest type of volcano. They are built from particles and blobs of congealed lava ejected from a single vent. … Most cinder cones have a bowl-shaped <em>crater</em> at the summit…”</blockquote><p>But a picture is worth a thousand words so here’s a nice photograph of some cinder cones:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u41U2gwTL0rEA-Yx9LSw_A.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@jeffkingla?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Jeff King</a> on <a href="https://unsplash.com/s/photos/volcanic-cone?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>So we now know what we’re roughy looking for from a spatial point of view; <strong>cone-shaped features with steep sides and a circular crater in the middle.</strong></p><p><strong>Let’s move on to seeing what this looks like in our dataset.</strong></p><h3>Data Loading and Labelling</h3><p>The dataset we will be using for this is DEM data (aka topographic data) from the largest island in the Hawaiian Islands (provided by the USGS¹), where we are able to see the morphology of cinder cones. We will map the cinder cones found on <a href="https://www.google.ru/maps/place/Mauna+Kea+Ice+Age+Natural+Area+Reserve/@19.8235951,-155.5825901,11.18z/data=!4m13!1m7!3m6!1s0x7bffdb064f79e005:0x4b7782d274cc8628!2sHawaii,+USA!3b1!8m2!3d19.8967662!4d-155.5827818!3m4!1s0x7953bdf95561b15d:0x582987cd20952faf!8m2!3d19.7915039!4d-155.4748835!5m1!1e4"><strong>Mauna Kea</strong></a>, a dormant volcano, to build a model which will be used to map cinder cones on the nearby volcano <a href="https://www.google.ru/maps/place/Hual%C4%81lai/@19.6811034,-155.9165149,12.38z/data=!4m13!1m7!3m6!1s0x7bffdb064f79e005:0x4b7782d274cc8628!2sHawaii,+USA!3b1!8m2!3d19.8967662!4d-155.5827818!3m4!1s0x79540a70234d7c0f:0xdf4d56417549044!8m2!3d19.6890817!4d-155.8646894!5m1!1e4"><strong>Hualālai</strong></a>.</p><p>QGIS is a great application for handling geospatial applications, and will be used for labelling the cinder cones. Loading in the data, which comes as a .geotiff, and changing the output to 10m contours helps us visualise the cinder cones (<strong><em>hint: look for the small concentric contour circles that are tightly spaced</em></strong>):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S35PK4DUbIwzTtWhLYyOWQ.png" /><figcaption>Contour map (at 10m intervals) of cinder cones on the southern flank of Mauna Kea. Image by author.</figcaption></figure><p>We can see the cinder cones based on our previously defined spatial criteria, so now <strong>we need to map them</strong> to create our <strong>labelled masks</strong>. For simplicity we can assume that cinder cones are round, therefore using circular polygons will be okay as labels, and create a shapefile of circles. We can start loading in our DEM geotiff and shapefile labels:</p><pre>#Loading a geotiff shapefile<br>def readimage():<br> print(‘reading image…’)<br> with rasterio.open(path_image, ‘r’) as ds:<br> arr = ds.read()<br> arr = np.transpose(arr, (1, 2, 0))<br> #Clip negative values to 0 <br> arr = arr.clip(0)<br> print(arr.shape)<br> return arr</pre><pre>#Loading the shapefile mask<br>def readmask(link):<br>print(‘reading mask…’)<br>geo = gpd.read_file(link)<br>with rasterio.open(path_image) as src:<br>raster = src.read()<br>geo = geo.to_crs(src.crs)<br>out_image, out_transform = rasterio.mask.mask(src, geo.geometry, filled=True)<br>masks = out_image[0,:,:]<br>#Set the mask labels to 1 and the rest to 0<br>masks[np.where(masks&lt;=0)] = 0<br>masks[np.where(masks&gt;0)] = 1<br>masks = masks.astype(np.int8)<br>masks = np.expand_dims(masks, axis=2)<br>return masks</pre><h3>Data Preprocessing</h3><p>Our DEM array covers the entire island, so we’ll need to do some cropping around Mauna Kea where we have our labelled mask. Also we will need to take our DEM array, which is 2D, and <strong>turn it into a 3D array</strong> with three channels. Why do we do this? Later we’ll be building a segmentation model using a <strong>pre-trained encoder</strong>, which typically requires a 3-channel image input (i.e. a colour image), so this ensures compatibility:</p><pre>#Crop both the image and masks to the same Mauna Kea area<br>image = image[ymin:ymax, xmin:xmax,:]<br>masks = masks[ymin:ymax, xmin:xmax,:]<br><br>#Stack the image array and normalise<br>image = np.dstack((image, image, image))<br>image = (image — image.min())/(image.max() — image.min())<br> <br>original_size = image.shape</pre><p>This gives us a final shape of <strong>(4000, 6500, 3)</strong>. Perfect. Let’s have a look and visualise the image and mask together:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8KkTmlsgBGmJym7Phe_6Ig.png" /><figcaption>Image of elevation data (left) and labelled cinder cone masks (right). Image by author.</figcaption></figure><p>Since we now how duplicated the same image three times, we can try to make these images more useful by applying some filters. We’ll <strong>enhance the contrast</strong>, so that the cinder cones stand out more compared to the flatter surroundings, and <strong>add a sobel filter to highlight the circular cone shapes:</strong></p><pre>#Contrast enhancing<br>image_eq = exposure.equalize_adapthist(image, clip_limit=0.05)</pre><pre>#Sobel filter and normalising<br>image_sobel = filters.sobel(np.squeeze(image))<br>image_sobel = (image_sobel — image_sobel.min())/(image_sobel.max() — image_sobel.min())</pre><pre>#concatenate standard image, equalised and sobel together<br>images = np.dstack((image[:,:,0], image_sobel[:,:,0], image_sobel[:,:,0]))</pre><p>Now that we’ve made some extra use out of those three channels we now need to <strong>subdivide</strong> this large array into smaller arrays which will be fed into our model. We will split our array so the output looks like:</p><blockquote>(No. samples, height, width, No. channels)</blockquote><pre>#Making image tiles<br>size = 224<br>step = int(size/2)</pre><pre>patch_arr = skimage.util.view_as_windows(image, (size, size, layer.shape[2]), step = step)<br>output = patch_arr.reshape((-1,) + (size, size, layer.shape[2]))</pre><p>This is repeated for both the input image and masks so that everything is complimentary. Finally these leaves us with the following inputs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zJP3qx5o0oyOY_Q_aIx4eA.png" /><figcaption>Image tiles and masks from Mauna Kea DEM data. Image by author.</figcaption></figure><p>Quite a few interesting images here! Our final array shape is <strong>(1938, 224, 224, 3);</strong> not many examples to build a model which further supports the need for using a pre-trained model. A final step will be to split the data between training and validation sets:</p><pre>x_train, x_val, y_train, y_val = train_test_split(images, masks, test_size=0.2, shuffle=True, random_state=123)<br>print(x_train.shape)<br>print(x_val.shape)<br>print(y_train.shape)<br>print(y_val.shape)</pre><pre>y_train = tf.cast(y_train, tf.float32)<br>y_val = tf.cast(y_val, tf.float32)</pre><h3><strong>Model Building and Training</strong></h3><p>Now we’re ready to build a model! We’ll be using a <strong>UNET model</strong> , which appears to be the go-to choice of segmentation architecture, with a pre-trained encoder available in <a href="https://keras.io/api/applications/">Keras</a>. I’ve chosen to use the <strong>InceptionResNetV2</strong> as an encoder due to its performance on the imagenet dataset, but we could use any of the pre-trained models.</p><pre>input_shape = (size, size, 3)<br>inception = InceptionResNetV2(include_top = False, weights = &#39;imagenet&#39;, input_tensor = input_shape)</pre><pre>inception.summary()<br>layer_names = [layer.name for layer in model.layers]</pre><p>Loading the pre-trained encoder with our input dimensions and getting a list of layer names for the encoder will help to construct the decoder part of the UNET, since we want to add skip connections between the encoder and decoder. Setting ‘include_top’ to False ensures that we can build a matching decoder. The decoder will restore the output of the encoder back to the original input image dimensions so that we have a matching segmentation mask:</p><pre>x = Conv2DTranspose(num_filters, (2,2), strides=2, padding=’same’)(inputs)<br>x = Conv2D(size, 2, padding=’same’, dilation_rate = 1, kernel_initializer = ‘he_normal’)(x)<br>x = BatchNormalization()(x)<br>x = Dropout(0.2)(x)<br>x = Activation(‘LeakyReLU’)(x)</pre><p>Stacking these blocks with decreasing filter sizes will make our decoder, and now for the skip connections. Skip connections can be added to the activation layers where the input layer size corresponds to <strong>224, 112, 56 and 28. </strong>The layer size may not directly correspond to these dimensions depending on the model used, so padding may be required like in the following example:</p><pre>skip_connection_2 = inception.get_layer(index = 3).output <br>skip_connection_2 = ZeroPadding2D(( (1, 0), (1, 0))  (skip_connection_2)</pre><p>Here we see that the fourth layer (index 3) needs to be padded to match our desired shape.</p><p>These skip connections will link the encoder and decoder by concatenating the output from the specified activation layers with our transposed decoder blocks, for example:</p><pre>x = Concatenate()([x, skip_connection_2])</pre><p>Once the decoder is built we finish the model with a final output layer, using a sigmoid activation since our task is <strong>binary segmentation</strong> (i.e. cinder cone or not).</p><pre>outputs = Conv2D(1, (1,1), padding=’same’, activation=’sigmoid’)(last_layer)</pre><p>When compiling the model, the loss function should take into account class imbalance, since the number of cinder cones isn&#39;t high relative to the background area, therefore we can use the <strong>Tversky Loss</strong>. There’s a good number of different loss functions readily available <a href="https://www.kaggle.com/bigironsphere/loss-function-library-keras-pytorch">here</a> which helps with experimenting with different losses. Our metric will be the <strong>Jaccard Similarity Coefficient </strong>to measure the percentage of overlapping pixels between our input mask and segmentation mask.</p><pre>model.compile(tf.keras.optimizers.Adam(5e-5), <br>loss= tversky_loss, metrics=[jaccard_coefficient],)</pre><pre>reduce_lr = ReduceLROnPlateau(monitor=’loss’, factor=0.2, patience=15, verbose=1, mode=’max’, min_lr=1e-6)<br>early = tf.keras.callbacks.EarlyStopping(monitor=’val_loss’, patience=10)</pre><pre>history = model.fit(x_train, y_train, epochs=100, batch_size = 32, validation_data = (x_val,y_val), callbacks = [reduce_lr, early])</pre><p>Using a low learning rate of 1e-5 and ReduceLROnPlateau during training for 100 epochs allowed the model to achieve a validation IoU of just <strong>over 80%</strong>, which isn’t too bad. Maybe this could be further improved with a more bespoke labelling strategy (ie. not simplifying the labels as circles) or different encoder model. Freezing different encoder layers could also be experimented with.</p><h3>Testing the model at another volcano</h3><p>Moving to the <strong>Hualālai </strong>volcano, where we will see how well our segmentation model recognises cinder cones. Let’s have a look at this new volcano:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yv_Lkhw31B06ptsDoVOZ0w.png" /><figcaption>Normalised DEM of the Hualālai<strong> </strong>volcano. Image by author.</figcaption></figure><p>We need to apply all the same preprocessing steps like we did with our Mauna Kea inputs, so that everything is compatible with our model. After cropping the area, applying filters and dividing the array into tiles, we have a shape of (420, 224, 224, 3). One thing to mention is that <strong>we don’t need to have a step when tiling the images</strong> since we aren’t interested in increasing the size of our samples here.</p><p>Let’s take a quick look at an example input tile like before:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TvIr0LWJdC49Bfmn4Lcguw.png" /><figcaption>Image tiles and masks from Hualālai<strong> </strong>DEM data. Image by author.</figcaption></figure><p>After using our model to make predictions:</p><pre>y_pred = model.predict(tiled_Hualalai_image)</pre><p>We can take a quick look at some of the segmented tiles before <strong>recombining the image tiles</strong> to form a segmentation mask with the same dimensions as the original image:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5RvQK92zlX7e5I9XhmYaDw.png" /><figcaption>Predicted segmentation mask with input image tiles. Image by author.</figcaption></figure><p>Predictions seem okay, but it looks like the model struggles to segment the small cinder cones, which could be due to the size of the cinder cones in our Mauna Kea training set (which are typically larger).</p><p>We can now view the segmentation image overlaid on the original input image. To make the cinder cones easier to spot, we’ll overlay the predicted segmentation masks over the sobel filter image (<em>where the cinder cones appear as small dots / rings</em>):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cAcIaElT60KCgPuJoTgnjw.png" /><figcaption>DEM data with sobel filter (left) combined with overlaid segmentation masks (right). Image by author.</figcaption></figure><p>Looking at the image as a whole looks a lot better:</p><blockquote>The location of the masks broadly matches the NW-SE orientation of the cinder cones along the volcano;</blockquote><blockquote>Cinder cones of various sizes are segmented (i.e. not just the large ones);</blockquote><blockquote>The shapes of each cinder cone broadly matches the shape of the segmentation masks.</blockquote><h3>Summary</h3><p>Overall we’ve worked through a <strong>novel example of using image segmentation</strong> with geospatial datasets to identify cinder cones in Hawaii. Since we can convert our segmentation masks back into the original coordinates from the DEM input data, we now have the ability to <strong>turn our segmentation masks into real datasets</strong> for further analysis (<strong>i.e what’s the distribution of cinder cone size? How does the density of cinder cones vary?</strong>).</p><p>Thanks for reading!</p><p><em>[1] U.S. Geological Survey, 2013, USGS 13 arc-second n20w156 1 x 1 degree: U.S. Geological Survey. Data available from U.S. Geological Survey, National Geospatial Program. Data from The National Map is free and in the public domain.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=67a15c18fa8f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/using-geospatial-data-to-discover-volcanoes-in-hawaii-67a15c18fa8f">Using Geospatial Data to Discover Volcanoes in Hawaii</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Identifying signs of tiredness using sequence classification]]></title>
            <link>https://medium.com/data-science/identifying-signs-of-tiredness-using-sequence-classification-f312e0adb6c?source=rss-13e1322246bb------2</link>
            <guid isPermaLink="false">https://medium.com/p/f312e0adb6c</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[hands-on-tutorials]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[keras]]></category>
            <dc:creator><![CDATA[Elliot Humphrey]]></dc:creator>
            <pubDate>Tue, 31 Aug 2021 17:25:31 GMT</pubDate>
            <atom:updated>2021-09-07T19:45:45.513Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsdatascience.com/tagged/hands-on-tutorials">Hands-on Tutorials</a></h4><h4>Training a model to classify tiredness using wearable health data</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0kFCtjIpEF6AwlQZg-WG4Q.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@crisaur?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Cris Saur</a> on <a href="https://unsplash.com/s/photos/tired?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Imagine you’ve been driving on the road for a while and it started to get dark. You feel your eyes getting heavier and know you should stop for a rest, but want to keep going so you can get to your destination on time…</p><p>This is a situation that I’m sure a lot of people have experienced at some point, a typical example of <strong>‘drowsy driving’</strong>, that leaves us <a href="http://1.https//www.nsc.org/road-safety/safety-topics/fatigued-driving">3X more likely to be in a road accident</a>.</p><p><strong>What if there was a way we could identify when we start to feel tired?</strong></p><blockquote>This is the problem statement we will try to solve using my personal <a href="https://www.fitbit.com/global/us/home"><strong>Fitbit</strong></a> data. Another example I’ve written using Fitbit data can be found <a href="https://towardsdatascience.com/does-machine-learning-know-when-im-scared-anomaly-detection-in-wearable-health-data-72f537dd4370">here</a>.</blockquote><h3><strong>The Challenge</strong></h3><p>We will use sleep and heart rate data collected from my <strong>personal Fitbit watch</strong>. For this task sleep data gives an idea about when I started to sleep, and can be used as a point to delineate between being awake and asleep. Furthermore this helps us estimate a time interval for being tired; for convenience this will be <strong>defined as the last thirty minutes before sleep</strong>. Defining these different stages helps to label our heart rate data, which measures heart beats per minute.</p><p>This gives us a <strong>labelled univariate sequence</strong> of heart rate measurements classified by sleep interval, which will be used as an input into our sequence classification model. Our model will be a GRU built using Keras.</p><p>After building and training our sequence classifier we’ll see how well it performs on some unseen data.</p><h4><strong>Data Loading and Preparation</strong></h4><p>Sleep and heart rate data comes as a mix of timestamped values stored in either a .csv or .json which represents either a day of month, therefore we need to load all these files and append them together.</p><pre>#Import sleep data data and combine files<br>def Dataimport():<br>    datasets = [&#39;sleep_files *.csv&#39;]<br>    for datatype in datasets:<br>    <br>        file_list=[]<br>        path = &#39;folder pathway&#39;<br>        os.chdir(path)<br>        for file in glob.glob(datatype):<br>            file_list.append(file)<br>        dfs = [] <br>        for file in file_list:<br>            data = pd.read_csv(path + file) <br>            print(&#39;Reading: &#39; + str(file))<br>            dfs.append(data)<br>        concatenated = pd.concat(dfs, ignore_index=True)<br>        concatenated = concatenated[[&#39;sleep_start&#39;,&#39;sleep_end&#39;]]<br>    return concatenated</pre><pre>sleepdata = Dataimport()</pre><p>This gives us a table with the time when sleep started and ended, with a value like this:</p><pre>Sleep start: 2021-01-12 22:10:00      Sleep end: 2021-01-13 05:37:30</pre><p>We can split the date and time since we will later need to find <em>only the days where both sleep data and heart rate data are recorded</em>:</p><pre>#Splitting the date and time<br>def Datasplit(sleepdata):<br>    sleepdata[&#39;date_start&#39;] = sleepdata[&#39;sleep_start&#39;].str.split(&#39;T&#39;, 1, expand=True)[0]<br>    sleepdata[&#39;time_start&#39;] = sleepdata[&#39;sleep_start&#39;].str.split(&#39;T&#39;, 1, expand=True)[1]<br>    sleepdata[&#39;date_end&#39;] = sleepdata[&#39;sleep_end&#39;].str.split(&#39;T&#39;, 1, expand=True)[0]<br>    sleepdata[&#39;time_end&#39;] = sleepdata[&#39;sleep_end&#39;].str.split(&#39;T&#39;, 1, expand=True)[1]<br>    sleepdata[&#39;start_of_sleep&#39;] = pd.to_datetime(sleepdata[&#39;date_start&#39;] + &#39; &#39; + sleepdata[&#39;time_start&#39;])<br>    sleepdata[&#39;end_of_sleep&#39;] = pd.to_datetime(sleepdata[&#39;date_end&#39;] + &#39; &#39; + sleepdata[&#39;time_end&#39;])<br>    sleepdata = sleepdata[[&#39;start_of_sleep&#39;, &#39;end_of_sleep&#39;]]<br>    sleepdata = sleepdata.sort_values(by=&quot;start_of_sleep&quot;)<br>    return sleepdata</pre><pre>sleepdata = Datasplit(sleepdata)</pre><p>For importing heart rate data we can reuse the same <em>Dataimport()</em> function by replacing the filename to include <em>“heart_rate” </em>in the filename. This data will need to be cleaned in a slightly different way, since we want to remove all unnecessary strings and resample the time series to be at the same resolution as the sleep data (i.e. in minutes):</p><pre>#cleaning the columns of the heart rate data<br>def HRclean(heart):<br> heart = heart.sort_values(by=”dateTime”)<br> heart = heart.set_index(‘dateTime’)<br> heart[“value”] = heart[“value”].apply(str)<br> heart[“value”] = heart[“value”].str.split(“{‘bpm’:”).str[1]<br> heart[“value”] = heart[“value”].str.split(“,”, 1, expand = True)[0]<br> heart[“value”] = heart[“value”].astype(int)<br> heart = heart.resample(‘1Min’).mean()<br> heart[‘value’] = heart[‘value’].round(0)<br> heart[‘date’] = heart.index<br> heart = heart[[‘date’, ‘value’]]<br> return heart</pre><pre>heartdata = Dataimport()<br>heart = HRclean(heartdata)</pre><p>Since we will define our ‘tired’ interval as thirty minutes before sleep, we can already imagine there will be <strong>a large class imbalance </strong>compared to the time awake and asleep. To mitigate this we can filter the heart rate data to a smaller interval, say from early evening to the early hours of the morning:</p><pre>#selecting only values in the evening (times around the tiredness since mornings are irrelevant)<br>heart = heart.between_time(‘19:00’, ‘03:00’)<br>heart[“only_date”] = [d.date() for d in heart[“date”]]<br>sleepdata[“only_date”] = [d.date() for d in sleepdata[“start_of_sleep”]]</pre><pre>#Identifying rows where sleep data exists for the given day of heart rate data<br>heart[‘sleep_data_exists’] = pd.Series(heart.only_date.isin(sleepdata.only_date).values.astype(int), heart.date.values)<br>heart = heart[heart[‘sleep_data_exists’] == 1]</pre><p>We can now label the heart rate data as being either asleep or awake based on the date-time range of sleep:</p><pre>#for each HR row, need to see if that time was during sleep or not, and label as 1 or 0<br>def Labelling(heart):<br> print(‘labelling the data…’)<br> heart[‘sleep_label’] = 0</pre><pre>#for each heartrate value, for each dt range, if hr date in dt range (per row), =1 else = continue<br> for i in range(len(heart)):<br> print(str(i) + ‘ of ‘+ str(len(heart)))<br> for j in range(len(sleepdata)):<br> if heart[‘date’][i] &gt;= sleepdata[‘start_of_sleep’][j] and heart[‘date’][i] &lt;= sleepdata[‘end_of_sleep’][j]:<br> heart[‘sleep_label’][i] = 1<br> else:<br> continue<br> return heart</pre><pre>heart = Labelling(heart)</pre><p>The final preparation step will be to label our ‘tired’ class:</p><pre>#selecting the time n rows before sleep starts<br>idx = heart.index.get_indexer_for(heart[heart[&#39;sleep_label&#39;] == 1].index)<br><br>subset = heart.iloc[np.unique(np.concatenate([np.arange(max(i-30,0), min(i-30+1, len(heart)))<br>                                            for i in idx]))]<br>subset = subset[subset.sleep_label == 0]<br>heart[&#39;tired_label&#39;] = pd.Series(heart.date.isin(subset.date).values.astype(int))</pre><pre>#cleaning the final labels into numerical values<br>heart[&#39;label&#39;] = pd.Series()<br>heart[&#39;label&#39;][heart.tired_label == 1] = 2<br>heart[&#39;label&#39;][heart.sleep_label == 1] = 1<br>heart[&#39;label&#39;] = heart[&#39;label&#39;].fillna(0)<br>heart = heart.dropna()</pre><h4><strong>Exploring the data</strong></h4><p>We can now begin to look at our dataset in more detail. Let’s begin by importing useful libraries. It’s then a good idea to visualise some of our labelled heart rate data so <em>we can</em> <em>get an idea about how heart rate varies per class</em>:</p><pre>import pandas as pd<br>import numpy as np<br>from sklearn.model_selection import train_test_split<br>from tensorflow import keras<br>import matplotlib.pyplot as plt<br>import scipy.stats as stats<br>import seaborn as sns<br>from tensorflow.keras import backend as K<br>from keras.layers import Dense, Dropout</pre><pre>sns.set(rc={&quot;figure.figsize&quot;:(30, 8)})<br>sns.scatterplot(x = df.index, y = df[&#39;heart_value&#39;][:1500], hue = df.label, palette = &#39;Spectral&#39;)<br>plt.xlabel(&quot;Heart Rate (BPM)&quot;)<br>plt.ylabel(&quot;Time (Minutes)&quot;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wzd0cCc3fZ_aSGA839jbSQ.png" /><figcaption>Time series plot of heart rate data (BPM) coloured by state.</figcaption></figure><p>We can clearly see that <strong>heart rate varies between being asleep and being awake</strong>. Heart rate values labelled as ‘tired’ are transitionary and therefore are less distinct, which will make our modelling interesting. Despite narrowing the time interview we still have an class imbalance that needs to be addressed later.</p><p>It’s also a good idea to look at the distribution of heart rate values per class:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8N0aW48u6NNAVGkE8Dx_eQ.png" /><figcaption>Histograms comparing the heart rate distribution for different classes</figcaption></figure><p>And see their summary statistics:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QF481_h5AmgbUaDLGdv47Q.png" /><figcaption>Summary table for heart rate data per class</figcaption></figure><p>In this case, <strong>0 is awake, 1 is asleep and 2 is tired</strong>. The ‘tired’ class has a mean similar to the ‘asleep’ class however has a standard deviation similar to the ‘awake’ class. This echoes our first pass observations from looking at the heart rate time series.</p><h4><strong>Sequence Preprocessing</strong></h4><p>Now let’s divide our data into fixed length sequences. <strong>Our sequences will be 30 minutes</strong> long to align with the length of our tired interval. We also need to split our data:</p><pre>def create_dataset(X, y, time_steps, step):</pre><pre>Xs, ys = [], []</pre><pre>   for i in range(0, len(X) - time_steps, step):<br>v = X.iloc[i:(i + time_steps)].values<br>labels = y.iloc[i: i + time_steps]<br>Xs.append(v)<br>ys.append(stats.mode(labels)[0][0])<br>   return np.array(Xs), np.array(ys).reshape(-1, 1)</pre><pre>interval = 30<br>X_train_full, y_train_full = create_dataset(x1, y1, 1, interval)</pre><pre>X_train, X_test, y_train, y_test = train_test_split(X_train_full, y_train_full, test_size=0.05, stratify = y_train_full, random_state=42, shuffle = True)</pre><p>We can use Keras to convert our labels to encoded categories, and further split our dataset so we have separate training (76%), validation (19%) and testing sets (5%)…a rather strange split looking back in hindsight:</p><pre>y_train1 = keras.utils.to_categorical(y_train, num_classes = None)<br>y_test1 = keras.utils.to_categorical(y_test, num_classes = None)</pre><pre>X_train = X_train.astype(np.float32)<br>y_train1 = y_train1.astype(np.float32)</pre><pre>#making the validation set separately<br>X_train, X_val, y_train1, y_val = train_test_split(X_train, y_train1, test_size=0.2, stratify = y_train1, random_state=42, shuffle = True)</pre><p>We use the ‘stratify’ argument in <em>train_test_split() </em>so that our labels are distributed proportionately per class. To help with class imbalance we will use <strong>class weights</strong>, where our model will weight a correct ‘tired’ prediction higher than the other classes. The weight will be proportional to the number of observations in the dataset:</p><pre>class_weight = {3: 1., 1: 1., 2: int((sum(y_train1.iloc[:,0]) + sum(y_train1.iloc[:,2])) / sum(y_train1.iloc[:,1]))}</pre><p>In this case, our ‘tired’ class has <strong>13x more weight </strong>compared to being awake or asleep. Since this is a <strong>multiclass sequence classification</strong> we will use a F1 score to measure accuracy. A useful guide on this as well as code for a Keras implementation can be found <a href="https://neptune.ai/blog/implementing-the-macro-f1-score-in-keras"><strong>here</strong></a>.</p><h4><strong>Model Building</strong></h4><p>Finally onto the model building. As mentioned <strong>we will use a GRU mode</strong>l since its <strong>faster</strong> to train. We build a model using <strong>Keras</strong> as follows, making sure to add Dropout layers to minimise overfitting, and passing the output sequence from one GRU to the next. Our GRU layers will start with <strong>32 units</strong> that will be multiplied in successive layers. We use an <strong>Adam</strong> optimiser, set the activation to <strong>softmax</strong> (since we have multiple classes) and measure the <strong>cross entropy loss</strong>:</p><pre>def create_gru_model(unit):<br>  inputs = keras.Input(shape=(X_train.shape[1],X_train.shape[2]))<br>  x = layers.GRU(unit*1, activation=&#39;tanh&#39;, return_sequences=True)  (inputs)<br>  x = layers.Dropout(0.25)(x)<br>  x = layers.GRU(unit*2, activation=&#39;tanh&#39;, return_sequences=True)(x)<br>  x = layers.Dropout(0.25)(x)<br>  x = layers.GRU(unit*2, activation=&#39;tanh&#39;, return_sequences=True)(x)<br>  x = layers.Dropout(0.25)(x)<br>  x = layers.GRU(unit*3, activation=&#39;tanh&#39;)(x)</pre><pre>outputs = layers.Dense(y_train1.shape[1], activation=&quot;softmax&quot;)(x)<br>model = keras.Model(inputs, outputs)</pre><pre>opt = keras.optimizers.Adam(learning_rate=1e-3)<br>model.compile(loss=&#39;categorical_crossentropy&#39;, optimizer= opt, metrics=[custom_f1])</pre><pre>  return model</pre><pre>model_2 = create_gru_model(32)<br>history2 = model_2.fit(X_train, y_train1, validation_data = (X_val, y_val), epochs = 200, batch_size = 256, shuffle= False, class_weight=class_weight)</pre><p>After 200 epochs of training and validation we can see the F1 score and model loss:</p><pre>sns.set(rc={&quot;figure.figsize&quot;:(12, 12)})<br>plt.plot(history2.history[&#39;custom_f1&#39;])<br>plt.plot(history2.history[&#39;val_custom_f1&#39;])<br>plt.ylabel(&#39;F1 Score&#39;)<br>plt.xlabel(&#39;Epoch&#39;)<br>plt.legend()<br>plt.show()</pre><pre>plt.plot(history2.history[&#39;loss&#39;])<br>plt.plot(history2.history[&#39;val_loss&#39;])<br>plt.ylabel(&#39;Loss&#39;)<br>plt.xlabel(&#39;Epoch&#39;)<br>plt.legend()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-FS1nutqAO9GbR0t3moGcw.png" /><figcaption>Model loss and F1 score after 200 epochs</figcaption></figure><p>A <strong>final validation score above 0.95</strong> seems pretty good. Finally we will predict on our test set and evaluate the results in a confusion matrix. We’ll need to use np.argmax() to help us when dealing with multiple classes:</p><pre>y_pred = np.argmax(model_2.predict(X_test), axis=1)<br>y_pred = np.expand_dims(y_pred, axis=-1)</pre><pre>from sklearn.metrics import plot_confusion_matrix<br>matrix = confusion_matrix(y_test, y_pred)</pre><pre>sns.heatmap(matrix, annot=True, fmt=’g’, yticklabels= [ ‘Awake’, ‘Asleep’, ‘Tired’], xticklabels= [ ‘Awake’, ‘Asleep’, ‘Tired’])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/322/1*XEx9V7T7bXetxabKYWQq3w.png" /><figcaption>Confusion matrix on unseen test dataset</figcaption></figure><p>It appears the large majority of classes were correctly predicted. We see some incorrect classifications between being tired and awake, which is understandable from the mean and histograms we previously saw.</p><h4><strong>Summary / Next steps</strong></h4><p>We’ve taken a univariate time series of heart rate data, combined with labels derived from sleep data, <strong>to create a sequence classification model able to tell when we’re asleep, awake or tired</strong>. Our model has performed well, but maybe there are <strong>additional aspects</strong> that could be taken into account:</p><ul><li>How would a model perform for binary sequence classification (i.e. tired / not tired)?</li><li>Would data measuring the number of steps taken help to differentiate between being awake and tired (assuming that physical activity decreases when tired)?</li><li>If we expanded our time interval to also record tiredness when we wake up, would it look the same or different compared to ‘evening tiredness’?</li></ul><p>There’s a lot more to do and explore, I’ll keep you updated with my progress. Thanks for reading!</p><p><em>Note: This article does not claim to predict or prevent tiredness-related accidents, and should only be read and used as a data science learning exercise.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f312e0adb6c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/identifying-signs-of-tiredness-using-sequence-classification-f312e0adb6c">Identifying signs of tiredness using sequence classification</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Does Machine Learning know when I’m scared? Anomaly detection in wearable health data]]></title>
            <link>https://medium.com/data-science/does-machine-learning-know-when-im-scared-anomaly-detection-in-wearable-health-data-72f537dd4370?source=rss-13e1322246bb------2</link>
            <guid isPermaLink="false">https://medium.com/p/72f537dd4370</guid>
            <category><![CDATA[keras]]></category>
            <category><![CDATA[anomaly-detection]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[wearables]]></category>
            <dc:creator><![CDATA[Elliot Humphrey]]></dc:creator>
            <pubDate>Sun, 02 May 2021 17:21:15 GMT</pubDate>
            <atom:updated>2021-05-02T17:21:15.118Z</atom:updated>
            <content:encoded><![CDATA[<h4>An opportunity to build autoencoders in Keras using real-life Fitbit measurements.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m9zX3GyK0ZJu6lhiFS1wtw.jpeg" /><figcaption>Image by Author</figcaption></figure><p>Wearable health devices like Fitbit offer large amounts of information about our fitness and day-to-day activities. This provides a <strong>great opportunity</strong> to work with large datasets, where personal information can be used to gain a deeper understanding of practical real-world machine learning applications beyond examples like the <a href="https://archive.ics.uci.edu/ml/datasets/human+activity+recognition+using+smartphones">Human Activity Recognition</a> dataset.</p><p>The dataset used in this article is from my personal <a href="https://www.fitbit.com/global/us/home">Fitbit</a> device, where calories and heart rate data are recorded as a time series over a period of nearly three weeks. The goal will be to build and use an autoencoder model using Keras to <strong>identify anomalies based on heart rate and calories over time</strong>.</p><h3><strong>Summary of work:</strong></h3><p>· We will begin with loading in the datasets and seeing how heart rate and calories vary during our sample period.</p><p>· We will also examine the dataset that contains anomalies to help us understand what potential anomalies could look like. This will come in useful once we begin to evaluate our model’s performance.</p><p>· Data will be cleaned, scaled and reshaped to be compatible with our autoencoder model.</p><p>· Our model will be built and evaluated to determine prediction error, that will later be used to determine whether anomalies exist.</p><p>· Finally, we will look into the detected anomalies to try and unravel why they might have occurred.</p><h3><strong>Loading and visualising:</strong></h3><p>The dataset was downloaded from my personal Fitbit account, where files are exported in a JSON format. Information about downloading your own data can be found <a href="https://help.fitbit.com/articles/en_US/Help_article/1133.htm">here</a>. We can start by loading these files to produce DataFrames of heart rate and calories, which are both time series data, using Pandas after we import all the relevant modules:</p><pre>#Importing the relevant modules<br>import glob<br>import numpy<br>import os<br>import pandas as pd<br>import numpy as np<br>import seaborn as sns; sns.set()<br>import matplotlib.pyplot as plt<br>from tensorflow import keras<br>from sklearn.preprocessing import MinMaxScaler<br>sns.set_palette(“hls”, 2)</pre><pre>#Loading in the heart rate and calories data<br>heart = pd.DataFrame()<br>calories = pd.DataFrame()<br>datasets = [‘heart_rate-2021–03-*.json’, ‘calories-2021–02-*.json’]<br>for datatype in datasets:<br>    file_list=[]<br>    path = ‘/Physical Activity/’<br>    os.chdir(path)<br>    for file in glob.glob(datatype):<br>        file_list.append(file)<br>    dfs = []<br>    for file in file_list:<br>        data = pd.read_json(path + file)<br>        print(‘Reading: ‘ + str(file))<br>        dfs.append(data)<br>    concatenated = pd.concat(dfs, ignore_index=True)<br>    concatenated[“value”] = concatenated[“value”].apply(str)<br>    if concatenated[‘value’].str.contains(“bpm”).any():<br>        heart = pd.concat([heart, concatenated], axis = 1)<br>    else:<br>        calories = pd.concat([calories, concatenated], axis = 1)</pre><p>Let’s have a look at the data to see what kind of cleaning we may need to do:</p><pre>print(heart.head(5))<br>print(calories.head(5))</pre><pre>dateTime value<br>0 2021–03–08 21:00:07 {‘bpm’: 57, ‘confidence’: 3}<br>1 2021–03–08 21:00:17 {‘bpm’: 56, ‘confidence’: 3}<br>2 2021–03–08 21:00:32 {‘bpm’: 56, ‘confidence’: 3}<br>3 2021–03–08 21:00:42 {‘bpm’: 57, ‘confidence’: 3}<br>4 2021–03–08 21:00:57 {‘bpm’: 58, ‘confidence’: 3}</pre><pre>dateTime value<br>0 2021–02–18 00:00:00 1.19<br>1 2021–02–18 00:01:00 1.19<br>2 2021–02–18 00:02:00 1.19<br>3 2021–02–18 00:03:00 1.19<br>4 2021–02–18 00:04:00 1.19</pre><p>Heart rate is measured in beats per minute (BPM) and sampled at irregular intervals of 10–15 seconds. Calories data measures the amount of calories used, which is calculated using a combination of a person’s <a href="https://help.fitbit.com/articles/en_US/Help_article/1141.htm#:~:text=How%20does%20my%20Fitbit%20device%20calculate%20calories%20burned%3F,to%20estimate%20your%20calories%20burned.">basal metabolic rate (BMR) and activity rate</a>. We can change the sample frequency of the heart rate data so that both datasets record observations per minute. The value columns should also have the text removed in order to be read as floats:</p><pre>#Cleaning the training data<br>heart = heart.sort_values(by=&quot;dateTime&quot;)<br>heart = heart.set_index(&#39;dateTime&#39;)<br>heart[&quot;value&quot;] = heart[&quot;value&quot;].apply(str)<br>heart[&quot;value&quot;] = heart[&quot;value&quot;].str.split(&quot;{&#39;bpm&#39;:&quot;).str[1]<br>heart[&quot;value&quot;] = heart[&quot;value&quot;].str.split(&quot;,&quot;).str[0]<br>heart[&quot;value&quot;] = heart[&quot;value&quot;].astype(int)<br>heart = heart.resample(&#39;1Min&#39;).mean()<br>heart[&#39;value&#39;] = heart[&#39;value&#39;].round(0)<br>calories = calories.sort_values(by=&quot;dateTime&quot;)<br>calories[&quot;value&quot;] = calories[&quot;value&quot;].astype(float)<br>calories.columns = [&#39;time&#39;, &#39;value&#39;]<br>calories = calories.set_index(&#39;time&#39;)</pre><p>We can now merge heart rate and calories together to create a single DataFrame of time series data covering the sample period. Creating an empty DataFrame with a DateTimeIndex covering our sample period will make things more convenient:</p><pre>#Merge the datasets together<br>empty = pd.DataFrame(pd.date_range(start=&#39;2021-03-01&#39;, end=&#39;2021-03-20&#39;, freq=&#39;1min&#39;))<br>empty.columns = [&#39;datetimeindex&#39;]<br>empty = empty.set_index(&#39;datetimeindex&#39;)<br>from functools import reduce<br>df = reduce(lambda left,right: pd.merge(left,right,left_index=True, right_index=True), [heart, calories, empty])<br>df.columns = [&#39;Heart Rate (BPM)&#39;, &#39;Calories Used&#39;]<br>df = df.dropna()</pre><p>Our training data will come from a consecutive sample period. The test data comes from a day approximately in the middle of our sample, therefore we can assume there is some continuity in conditions affecting heart rate and calories data:</p><pre>#Splitting the test day<br>df_test = df.loc[(df.index &gt; &#39;2021-03-12 00:00:00&#39;) &amp; (df.index &lt;= &#39;2021-03-12 23:59:59&#39;)]<br>df = df.loc[(df.index &lt; &#39;2021-03-12 00:00:00&#39;) | (df.index &gt; &#39;2021-03-12 23:59:59&#39;)]</pre><p>We are now at a stage to finally visualise what the Fitbit data looks like, and will start with the training data. The gap in the training data represents the test day that we previously extracted:</p><pre>#Visualising training and testing data<br>datasets = [df, df_test]<br>for dfs in datasets:<br>    fig, axs = plt.subplots(nrows=2, ncols=1, figsize=(12,6))<br>    axs[0].plot(dfs[&#39;Heart Rate (BPM)&#39;])<br>    axs[1].plot(dfs[&#39;Calories Used&#39;])<br>    plt.setp(axs[0].get_xticklabels(), visible=False)<br>    axs[0].tick_params(axis=&#39;x&#39;, rotation=70)<br>    axs[1].tick_params(axis=&#39;x&#39;, rotation=70)<br>    axs[0].set(ylabel=&#39;Heart Rate (BPM)&#39;)<br>    axs[1].set(ylabel= &#39;Calories Used&#39;, xlabel=&quot;Date&quot;)<br>    plt.tight_layout()<br>    plt.show</pre><p>Finally, some graphs!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R_oZFUYb4ND1PSBxlSkMbg.png" /><figcaption>Training dataset of heart rate and calories used over time</figcaption></figure><p>The dataset show a cyclical trend in both heart rate and calories; both datasets increase in the morning, remain fairly stable during the day, then gently decrease in the evening. This is an expected trend as it correlates with waking up, working during the day, then relaxing in the evening before bed.</p><p>Moving onto the test dataset which will be used to validate whether anomalies are detected. This data covers a single day, where we can try to guess what would be a potential anomaly based on what we have seen in the training data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fbW42m4JEes46kISt8wcxA.png" /><figcaption>Test dataset of heart rate and calories used over time</figcaption></figure><p>Interestingly the test data for this day has two peaks where heart rate increases significantly in the afternoon, which doesn’t correlate with an increase in calories. This would indicate that the <strong>increase in heart rate is likely not associated with a change in exercise</strong>. The added benefit of working with personal data is that I know what happened on this day; a trip to the hospital. As someone who isn’t a huge fan of hospital visits, my Fitbit has clearly captured my nervousness which makes it an excellent example of anomaly we will later try to identify.</p><p>We can make a simple correlation matrix between heart rate and calories to see their relationship:</p><pre>#Make a correlation coefficient<br>print(df.corr())</pre><pre>                 Heart Rate (BPM)  Calories Used<br>Heart Rate (BPM)     1.000000       0.102251<br>Calories Used        0.102251       1.000000</pre><p>There is a<strong> slightly positive correlation</strong> between heart rate and calories used. According to Fitbit’s <a href="https://help.fitbit.com/articles/en_US/Help_article/1141.htm#:~:text=How%20does%20my%20Fitbit%20device%20calculate%20calories%20burned?,to%20estimate%20your%20calories%20burned.">documentation</a> on how calories are calculated, heart rate influences how calories are calculated therefore our observed positive correlation is to be expected.</p><h3><strong>Preprocessing:</strong></h3><p>Getting back to the code, we now need to scale our data so that we can reduce the impact of any outliers within our training data that may impede a model’s ability to detect anomalies. We will use the MinMaxScaler as this will change both the heart rate and calories data to be between 0 and 1, which will be important later when we calculate prediction errors.</p><pre>#Scale the data<br>scaler = MinMaxScaler()<br>scaler = scaler.fit(df)<br>scale_train = pd.DataFrame(scaler.transform(df))<br>scale_test = pd.DataFrame(scaler.transform(df_test))</pre><p>Now the data has been scaled, we need to <strong>reshape</strong> to be compatible with our autoencoder model. The input to our model will be a three dimensional array with a shape according to the number of samples, the length of the sequences, and the number of features. We will use a sequence of 30 time steps (aka half an hour time interval) as we assume that anomalies are short lived.</p><pre># reshape to [samples, time_steps, n_features]<br>def create_dataset(X, y, time_steps=1):<br>    Xs, ys = [], []<br>    for i in range(len(X) - time_steps):<br>        v = X.iloc[i:(i + time_steps)].values<br>        Xs.append(v)<br>        ys.append(y.iloc[i + time_steps])<br>    return np.array(Xs), np.array(ys)</pre><pre>time_steps = 30<br>X_train, y_train = create_dataset(scale_train, scale_train, time_steps)<br>X_test, y_test = create_dataset(scale_test, scale_test, time_steps)<br>print(X_train.shape, y_train.shape)</pre><p>This produces a train and test set with a shape of (23333, 30, 2) (23333, 2).</p><h3><strong>Model building:</strong></h3><p>Our data has been successfully reshaped and is now ready for building our autoencoder. This article won’t go into the details of autoencoders and LSTMs, but I highly recommend checking this information out:</p><ul><li><a href="https://towardsdatascience.com/anomaly-detection-using-autoencoders-5b032178a1ea">Anomaly Detection using Autoencoders</a></li><li><a href="https://towardsdatascience.com/lstm-for-time-series-prediction-de8aeb26f2ca">LSTM for time series prediction</a></li></ul><p>The autoencoder will be composed of LSTM layers split into an encoder and decoder. Firstly the model will encode the input sequence of data into a <strong>simplified representation of its key features</strong>, then a decoder will learn how to covert this simplified representation back into its original input state. Using an autoencoder approach allows us to <strong>create a representation of ‘normal’ features within a dataset</strong>. Therefore when an anomalous sequence is input, the model will be unable to reconstruct its simplified representation which will lead to a high model error.</p><p>As our input uses two features, the error from the combined prediction (i.e. the error from both heart rate and calories sequence reconstructions) will be used as our method for detecting anomalies.</p><p>Our model architecture uses multiple LSTM layers in the encoder and decoder, separated by a RepeatVector layer which repeats a simplified representation given by the last encoder LSTM layer. A final TimeDistributed later at the end of the decoder rebuilds the input back to its original shape.</p><pre>#model building<br>model = keras.Sequential()<br>model.add(keras.layers.LSTM(256, input_shape=(X_train.shape[1], X_train.shape[2]), return_sequences=True, name=&#39;encoder_1&#39;))<br>model.add(keras.layers.Dropout(rate=0.2))<br>model.add(keras.layers.LSTM(128, return_sequences=True, name=&#39;encoder_2&#39;))<br>model.add(keras.layers.Dropout(rate=0.2))<br>model.add(keras.layers.LSTM(64, return_sequences=False, name=&#39;encoder_3&#39;))<br>model.add(keras.layers.RepeatVector(n=X_train.shape[1], name=&#39;encoder_decoder&#39;))<br>model.add(keras.layers.LSTM(64, return_sequences=True, name=&#39;decoder_1&#39;))<br>model.add(keras.layers.Dropout(rate=0.2))<br>model.add(keras.layers.LSTM(128, return_sequences=True, name=&#39;decoder_2&#39;))<br>model.add(keras.layers.Dropout(rate=0.2))<br>model.add(keras.layers.LSTM(256, return_sequences=True, name=&#39;decoder_3&#39;))<br>model.add(keras.layers.TimeDistributed(keras.layers.Dense(units=X_train.shape[2])))<br>model.compile(loss=&#39;mae&#39;, optimizer=&#39;adam&#39;)<br>model.summary()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/1*Uv-WlhcCAfJKOJOfD8fcdQ.png" /><figcaption>Model summary</figcaption></figure><p>The model will use the Adam optimiser and measure the mean average error ‘MAE’ loss. We will fit the model to our training data; <em>note that training data is used for both the input and label when fitting our model</em>. A batch size of 256 will be used for a training duration of 25 epochs.</p><pre>#fitting on training data<br>history = model.fit(X_train, X_train, epochs=25, batch_size=256,<br>                    validation_split=0.1,verbose=1,shuffle=False)<br>#plotting loss<br>fig=plt.figure()<br>plt.plot(history.history[&#39;loss&#39;], label=&#39;Training loss&#39;)<br>plt.plot(history.history[&#39;val_loss&#39;], label=&#39;Validation loss&#39;)<br>plt.ylabel(&#39;Loss&#39;)<br>plt.xlabel(&#39;No. Epochs&#39;)<br>plt.legend()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dc7JkYVrZsOxx475IYl60A.png" /><figcaption>Training &amp; validation loss during training</figcaption></figure><p>We will now use the MAE between the training data and predictions to show a distribution of losses, that will serve as our way to <strong>define an anomaly</strong>. MAE is calculated per time step from a combination of both heart rate and calories data.</p><pre>#predicting on test data<br>X_pred = model.predict(X_train)<br>X_pred_2d = pd.DataFrame(X_pred[:,0,:]).astype(float)<br>X_pred_2d.columns = [&#39;HR Pred&#39;, &#39;Calories Pred&#39;]<br>X_train_2d = pd.DataFrame(X_train[:,0,:]).astype(float)<br>X_train_2d.columns = [&#39;HR Test&#39;, &#39;Calories Test&#39;]</pre><pre>#Plot the test data together<br>fig, axs = plt.subplots(4, figsize=(12,6))<br>axs[0].plot(X_pred_2d[&#39;HR Pred&#39;])<br>axs[1].plot(X_train_2d[&#39;HR Test&#39;])<br>axs[2].plot(X_pred_2d[&#39;Calories Pred&#39;])<br>axs[3].plot(X_train_2d[&#39;Calories Test&#39;])<br>plt.setp(axs[0].get_xticklabels(), visible=False)<br>plt.setp(axs[1].get_xticklabels(), visible=False)<br>plt.setp(axs[2].get_xticklabels(), visible=False)<br>axs[0].tick_params(axis=&#39;x&#39;, rotation=70)<br>axs[1].tick_params(axis=&#39;x&#39;, rotation=70)<br>axs[0].set(ylabel= &#39;HR Prediction&#39;)<br>axs[1].set(ylabel= &#39;HR Training&#39;)<br>axs[2].set(ylabel= &#39;Calories Prediction&#39;)<br>axs[3].set(ylabel= &#39;Calories Training&#39;, xlabel= &#39;Time Step (per minute)&#39;)<br>plt.tight_layout()</pre><pre>#calculate error<br>predictions = pd.concat([X_pred_2d[&#39;HR Pred&#39;], X_pred_2d[&#39;Calories Pred&#39;]], axis = 1)<br>train_inputs = pd.concat([X_train_2d[&#39;HR Test&#39;], X_train_2d[&#39;Calories Test&#39;]], axis = 1)anomaly = pd.DataFrame(np.abs(predictions.values - train_inputs.values))<br>anomaly = anomaly.mean(axis=1)</pre><pre>ax = sns.distplot(anomaly, bins=50, kde = True)<br>ax.set_title(&#39;Training Data Loss Distribution&#39;)<br>ax.set_xlabel(&#39;Loss&#39;)<br>ax.set_ylabel(&#39;Frequency&#39;)<br>fig = ax.get_figure()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7joOnXOedsTz5ROJDptmzw.png" /><figcaption>Training data loss distribution</figcaption></figure><p>After looking at the loss distribution we can see a long tail exists with higher loss values, indicating times when the model has struggled to reconstruct the input sequence. To define the loss threshold which will be used to judge whether a prediction is anomalous, we will find the loss in the training data at the <strong>99th percentile</strong>.</p><pre>thres = round(numpy.quantile(anomaly, 0.99),3)<br>print(&#39;99th percentile loss value from training: &#39; + str(thres))</pre><pre>99th percentile loss value from training: 0.177</pre><h3><strong>Predictions:</strong></h3><p>Now that we have built our model and <strong>defined a loss threshold of 0.177</strong>, we can move on to testing our model to see whether it can detect the anomalies we recognised earlier in the test data. We will reuse our model with the previously reshaped test data and repeat the same process for calculating the prediction loss.</p><pre>#Predicting<br>X_pred = model.predict(X_test)<br>X_pred = pd.DataFrame(X_pred[:,0,:]).astype(float)<br>X_pred.columns = [&#39;HR Pred&#39;, &#39;Calories Pred&#39;]<br>X_test_data = pd.DataFrame(X_test[:,0,:]).astype(float)<br>X_test_data.columns = [&#39;HR Test&#39;, &#39;Calories Test&#39;]</pre><p>We use the previously defined loss threshold as a cut-off for defining a data point as anomalous. To better visualise our anomalous points we combine the anomaly column with the original test dataset to help put things into context.</p><pre>difference = pd.DataFrame(np.abs(X_pred.values - X_test_data.values))<br>difference[&#39;mae loss&#39;] = difference.mean(axis=1)<br>difference[&#39;threshold&#39;] = thres<br>difference[&#39;anomaly&#39;] = difference[&#39;mae loss&#39;] &gt; difference[&#39;threshold&#39;]<br>difference[&#39;index&#39;] = difference.index</pre><pre>X_pred[&#39;index&#39;] = X_pred.index<br>X_test_data[&#39;index&#39;] = X_test_data.index<br>X_test_data = X_test_data.join(difference[&#39;anomaly&#39;])</pre><p>The heart rate and calories data can be returned to their original scale using the previously made scaler, which will help us better understand our results:</p><pre>X_test_data_original = pd.DataFrame(scaler.inverse_transform(X_test_data[[&#39;HR Test&#39;,&#39;Calories Test&#39;]]))<br>X_test_data = pd.concat([X_test_data, X_test_data_original], axis = 1)<br>X_test_data.columns = [&#39;HR Test&#39;, &#39;Calories Test&#39;, &#39;Index&#39;, &#39;Anomaly&#39;, &#39;Heart Rate (BPM)&#39;, &#39;Calories Used&#39;]</pre><p>Our final steps will be focused on visualising the prediction results. We will begin by plotting the average loss of each data point, colour coded by whether an anomaly was identified:</p><pre>plt = sns.lmplot(x=&#39;index&#39;, y=&#39;mae loss&#39;, data=difference, <br>           fit_reg=False, hue=&#39;anomaly&#39;, scatter_kws={&quot;s&quot;: 10}, legend=True, legend_out=False, height=5, aspect=2)<br>plt.set(xlabel=&#39;Time Steps (per minute)&#39;, ylabel=&#39;MAE Loss&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qjyD4O7PWU-Ft9ODp8yVBQ.png" /><figcaption>Predicted Mean Absolute Error over time</figcaption></figure><p>As we can see, our previously defined anomaly cut-off has been used to <strong>detect anomalous data points</strong>.</p><p>Looking at the distribution of model loss during the day in our test data, it is clear that the likelihood of anomalies occurring in the early morning are minimal since the loss on data points below 500 minutes (aka 08:30am are consistent low). This makes sense as I’m asleep and obviously not going to the gym in the middle of the night.</p><p>Things become slightly different however during the rest of the day where the distribution of loss becomes larger and some data points are flagged as anomalous. The larger distribution of loss in the afternoon (aka between 720–1140 minutes) can potentially be attributed to variations in my afternoon schedule.</p><p>The anomalous points are spread into multiple time intervals. To understand why these anomalous events occur, we can look at the heart rate and calories dataset:</p><pre>plt = sns.lmplot(x =&#39;Index&#39;, y=&#39;Heart Rate (BPM)&#39;, scatter_kws={&quot;s&quot;: 10}, data=X_test_data, <br>           fit_reg=False, hue=&#39;Anomaly&#39;, legend=True, legend_out=False, height=5, aspect=2)<br>plt.set(xlabel=&#39;Time Steps (per minute)&#39;, ylabel=&#39;Heart Rate (BPM)&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vXRmpEDDf1WnnM3kbJ1jMQ.png" /><figcaption>Test dataset of heart rate coloured by detected anomalies</figcaption></figure><p>The heart rate data has been colour coded by anomalies detected in the previous loss plot. We can see that obvious anomalies are when my heart rate is anomalously high, reaching over 140BPM, forming two peaks between 800–1000 minutes. This fits nicely with the time of my aforementioned hospital visit.</p><p>Interestingly we can see that anomalies also occur during normal heart rates (based on the neighbouring heart rate values). Our model uses both heart rate and calories which suggests that these anomalies are likely associated with calories used. We need to look into the calories data to gain more information:</p><pre>plt = sns.lmplot(x =&#39;Index&#39;, y=&#39;Calories Used&#39;, scatter_kws={&quot;s&quot;: 10}, data=X_test_data, <br>           fit_reg=False, hue=&#39;Anomaly&#39;, legend=True, legend_out=False, height=5, aspect=2)<br>plt.set(xlabel=&#39;Time Steps (per minute)&#39;, ylabel=&#39;Calories Used&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cdtBFhfRn4HyreexGug3iA.png" /><figcaption>Test dataset of calories used coloured by detected anomalies</figcaption></figure><p>Our calories data shows that anomalies are detected during peaks, typically above six calories per minute, which are likely when I am walking. When we look at the anomalies associated with high calories we see they occur during times when heart rate is considered normal. This relationship, <strong>high calories but average heart rate</strong>, can be considered anomalous as typically most calories will be used when heart rate is higher (also supported by the positive correlation between heart rate and calories).</p><p>The inverse relationship is observed when heart rate is high as calorie data is low, suggesting that heart rate is high due to reasons not associated with physical activity. <strong>As this is my own data I know that in the afternoon I went to the hospital and, due to a fear of needles, my heart rate was very high. This is a great example of an anomaly and the reason why I selected this day to be used as our test dataset.</strong></p><h3><strong>Conclusion:</strong></h3><p>We have learnt how to <strong>develop an autoencoder model using LSTM networks</strong> to detect anomalies in health time series data. We used our model to learn what was considered normal behaviour, and use this to <strong>detect potential anomalies</strong>. The model was tested on a day of anomalous data and correctly identified some interesting trends between heart rate and calories, associated with spontaneous walks and a <strong>fearful trip to the hospital!</strong></p><p>This project hopefully highlights the fun of working on personal data, letting us interpret the model outcomes to a much higher level compared to some classic readily available datasets. Thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=72f537dd4370" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/does-machine-learning-know-when-im-scared-anomaly-detection-in-wearable-health-data-72f537dd4370">Does Machine Learning know when I’m scared? Anomaly detection in wearable health data</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>