<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by MARTIN YU on Medium]]></title>
        <description><![CDATA[Stories by MARTIN YU on Medium]]></description>
        <link>https://medium.com/@zidongyu26?source=rss-3339ed5adf2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*uWBVFDibS25lRgU8EI5zwA.jpeg</url>
            <title>Stories by MARTIN YU on Medium</title>
            <link>https://medium.com/@zidongyu26?source=rss-3339ed5adf2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 08:26:54 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@zidongyu26/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Exploring the spatial distribution of Airbnb listings in Hong Kong, Part 5]]></title>
            <link>https://medium.com/@zidongyu26/exploring-the-spatial-distribution-of-airbnb-listings-in-hong-kong-part-5-b06f3d4158d3?source=rss-3339ed5adf2------2</link>
            <guid isPermaLink="false">https://medium.com/p/b06f3d4158d3</guid>
            <category><![CDATA[geography]]></category>
            <category><![CDATA[hong-kong]]></category>
            <category><![CDATA[spatial-analysis]]></category>
            <category><![CDATA[airbnb]]></category>
            <category><![CDATA[gis]]></category>
            <dc:creator><![CDATA[MARTIN YU]]></dc:creator>
            <pubDate>Thu, 27 Feb 2020 15:23:48 GMT</pubDate>
            <atom:updated>2020-02-27T15:23:48.112Z</atom:updated>
            <content:encoded><![CDATA[<p>Article 5: Revealing the hidden stories of spatial autocorrelation</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0V82_xk7tSKx80FywvQPKg.jpeg" /><figcaption>Chess board, Retrieved from: <a href="https://chessantiques.com/product/reproduction-antique-rosewood-and-holly-chessboard/">Source</a></figcaption></figure><p>Previously, two articles have been posted for explaining the cluster phenomenon existing in the data points of Airbnb listings in Hong Kong. Yet, using clustering technique for coordinates can only indicate the cluster of the location of points instead of some associated attributes with the points, therefore, in this article, some methods will be introduced for exploring how similar closer objects are to other nearby objects, for their associated attributes.</p><p>Generally, for any choropleth map (theme map), it is critical for us to explore the distribution/pattern of the location that represents the selected value(attribute). In the field of statistics, people always use correlation for indicating the relationship between two variable (features), however, from the perspective of geospatial data, for identifying the pattern of the map, a method called “Spatial Autocorrelation” can be used for recognizing and summarizing the principal characteristics of the geospatial data.</p><h3>What is spatial autocorrelation?</h3><p>Spatial autocorrelation is used to explain the presence of systematic spatial changes in given variables, helping people to learn how similar closer objects are to other neighbourhood objects. According to the explanation provided by <a href="https://gisgeography.com/tobler-first-law-of-geography/">gisgeography.com</a>, it is reasonable for us to utilize Moran’s Index for measuring the spatial correlation, and this index can be classified and understood as three indications for positive value, negative value, and zero, demonstrating that similar values cluster together, dissimilar values cluster together, or no autocorrelation, respectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*m8A8NHGf1QclDuJj2WuWoQ.png" /><figcaption>Explanation of spatial autocorrelation, <a href="https://gisgeography.com/tobler-first-law-of-geography/">Source</a></figcaption></figure><p>By using Moran’s Index, geographers can understand and examine the Tobler’s First Law of Geography, telling that everything is related to everything else and near things are more related than distant things.</p><p>Some techniques used below are retrieved by this medium article and you can find more detail about spatial autocorrelation from:</p><p><a href="https://towardsdatascience.com/what-is-exploratory-spatial-data-analysis-esda-335da79026ee">What is Exploratory Spatial Data Analysis (ESDA)?</a></p><h3>Spatial autocorrelation with python</h3><p>Import basic data-analysis libraries first:</p><pre>import pandas as pd<br>import numpy as np<br>import matplotlib<br>import matplotlib.pyplot as plt<br>import seaborn as sns<br>import geopandas as gpd<br>%matplotlib inline</pre><p>Import the data of Airbnb listing and boundaries for three selected districts: Central &amp;Western, Wan Chai, as well as Yau Tsim Mong.</p><pre>data = pd.read_csv(&quot;HK/top3.csv&quot;).drop(&#39;Unnamed: 0&#39;, axis = 1)<br>data = data.drop([&#39;host_id&#39;, &#39;minimum_nights&#39;, &#39;number_of_reviews&#39;, &#39;calculated_host_listings_count&#39;, &#39;availability_365&#39;], axis=1)<br>geo_data  = gpd.read_file(&quot;HK/Hong_Kong_18_Districts/Hong_Kong_18_Districts.shp&quot;).drop(&#39;TCNAME&#39;, axis = 1)<br>geo_data = geo_data.loc[geo_data[&#39;ENAME&#39;].isin({&#39;CENTRAL &amp; WESTERN&#39;, &#39;WAN CHAI&#39;, &#39;YAU TSIM MONG&#39;})]</pre><p>Since we want to process all data using Geo-Pandas, latitude and longitude of Airbnb listing data need to be zipped and converted the to a point-coordinate format, using gpd.GeoDataFrame() function:</p><pre>data = gpd.GeoDataFrame(data,crs={&#39;init&#39; :&#39;epsg:4326&#39;}, geometry=gpd.points_from_xy(data.longitude, data.latitude))</pre><p>It should be noted that we here will import polygon grid, namely fishnet, which is preprocessed by ArcGIS using fishnet function:</p><pre>grid = gpd.read_file(&quot;HK/Grid_3.shp&quot;)</pre><p>Have a look about the grid fishnet:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/763/1*yFmkOzYZy88nLZMCg8aePg.png" /><figcaption>The grid map of study area.</figcaption></figure><p>By focusing on the map details, the unit of the grid map showing above is meter instead of degree of latitude or longitude as usual we see, because this is a projected map over an assumedly flat surface, thus, we have to convert this coordinate system to be same as other geospatial data in this article, by using to_crs() function.</p><pre># Check the crs of imported data<br>data.crs<br>geo_data.crs</pre><p>And them, we got the coordinate reference system of epsg: 4326, so we will convert the grid map to this crs by typing:</p><pre>grid = grid.to_crs({&#39;init&#39;:&#39;epsg:4326&#39;}) </pre><p>Now, we can plot all points of Airbnb listing over the grid map for checking whether all data in one crs:</p><pre>fig, ax = plt.subplots(figsize=(12,10))<br>grid.plot(color=&quot;white&quot;, edgecolor=&#39;black&#39;, ax= ax)<br>data.plot(markersize=1,ax=ax)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/754/1*Htm8XE5y11IMufD5D9uICg.png" /><figcaption>Scatter map of Airbnb listings over grid map</figcaption></figure><p>Taking a careful look at the points along edges of the polygon, we will there are some errors existing on the conversion of the CRS system, if we want to enhance the accuracy, we had better do those processes using ArcGIS.</p><p>The next step is to spatially join the points and the grid map, i.e. attach data (attributes) from one feature to another feature, in accordance with their spatial relationship. By using GeoPandas, it is very easy and simple:</p><pre>joined_data = gpd.sjoin(data, grid, op = &#39;within&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/1*ZkYHzLteA9ferIGEV_BrOA.png" /><figcaption>The output of spatially joined data</figcaption></figure><p>As we can see, each point is assigned to a specified cell of grid map, indicating by column of index_right. So, we can group all cells and therefore compute the total number of Airbnb listing for each cell:</p><pre>num_per_pix = joined_data.groupby([&#39;index_right&#39;]).count()[[&#39;id&#39;]]<br>num_per_pix = num_per_pix.join(grid)<br>num_per_pix = num_per_pix.drop(&quot;Id&quot;, axis = 1).rename(columns={&quot;id&quot;: &quot;count&quot;})<br>num_per_pix = gpd.GeoDataFrame(num_per_pix, geometry= &#39;geometry&#39;)<br>num_per_pix.head(5)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/530/1*SESBMuyd7udn90vKb3tYRQ.png" /><figcaption>The output of number of Airbnb listings per cell unit</figcaption></figure><h3>Spatial Weights and Spatial Lag</h3><p>Import libraries of exploratory spatial data analysis and splot :</p><pre>pip install esda<br>pip install splot</pre><pre>import pysal<br>from pysal.lib  import weights<br>from pysal.explore import  esda<br>from esda.moran import Moran, Moran_Local</pre><pre>import splot<br>from splot.esda import moran_scatterplot, plot_moran, lisa_cluster</pre><pre>import splot<br>from splot.esda import moran_scatterplot, plot_moran, lisa_cluster</pre><p>Time for determining the spatial weight and spatial lag:</p><p>Spatial weight: “Spatial statistics integrate space and spatial relationships directly into their mathematics (area, distance, length, or proximity, for example). Typically, these spatial relationships are defined formally through values called spatial weights structured as a <a href="http://resources.esri.com/help/9.3/arcgisdesktop/com/gp_toolref/spatial_statistics_toolbox/modeling_spatial_relationships.htm">spatial weights matrix file</a>.”[Retrieved from <a href="http://resources.esri.com/help/9.3/arcgisdesktop/com/gp_toolref/spatial_statistics_toolbox/spatial_weights.htm">Source </a>provided by ESRI Help]</p><p>Spatial lag: the variable (i.e. feature) to measure the relationship in spatial weights matrix, for our study cases, the total number of Airbnb listings per cell, is the spatial lag.</p><p>Code for define a spatial weight and lag:</p><pre>w = weights.Queen.from_dataframe(num_per_pix)<br>w.transform = &quot;R&quot;</pre><pre>num_per_pix[&quot;count_total&quot;] = weights.lag_spatial(w, num_per_pix[&quot;count&quot;])<br>num_per_pix.head()</pre><h3>Spatial Autocorrelation</h3><p>In terms of global spatial autocorrelation, a scatterplot will be presented to display the overall pattern of this dataset. By using Moran(), we can easily compute the Moran’s Index:</p><pre>y = num_per_pix[&quot;count&quot;]<br>moran = Moran(y, w)<br>print(&#39;The Moran\&#39;s I = &#39; + str(moran.I) + &#39;\nThe significane = &#39; + str(moran.p_sim))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/331/1*9dZ5zK50YstZTZ4_Jl5TbA.png" /><figcaption>Moran’s Index and its significance</figcaption></figure><p>And then produce the scatterplot:</p><pre>fig, ax = moran_scatterplot(moran, aspect_equal=True)<br><br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/1*vZLCmsM3s3lJlw7UYXlUOA.png" /><figcaption>Moran Scatterplot</figcaption></figure><p>As shown above, the Moran’s Index 0.69 indicates that there is a positive spatial autocorrelation within the dataset, meaning the positive correlation in a cell among nearby cells which may have the similar number of Airbnb listings. Moreover, the red line in scatterplot also states the relationship between the count of listings per cell and spatial locations (spatial weight).</p><p>As for local spatial autocorrelation, Local Indicators of Spatial Association (LISA) Moran_Local(y, w)will be employed to depict the detailed groups for each cluster including high density with high density, high density with low density, low density to high density and its opposite.</p><pre># calculate Moran Local <br>m_local = Moran_Local(y, w)<br># Plot<br>fig, ax = moran_scatterplot(m_local, p=0.01)<br>ax.set_xlabel(&#39;Number of Listings&#39;)<br>ax.set_ylabel(&#39;Spatial Lag of Num&#39;)<br>plt.text(1.95, 0.5, &quot;HH&quot;, fontsize=25)<br>plt.text(1.95, -1.5, &quot;HL&quot;, fontsize=25)<br>plt.text(-1.5, 1, &quot;LH&quot;, fontsize=25)<br>plt.text(-1.3, -1.3, &quot;LL&quot;, fontsize=25)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/1*YxxOIOJ0_yTTZh2qjb_q_w.png" /><figcaption>Moran Local Scatterplot</figcaption></figure><p>Referring to the plot above, the Moran’s Index is mainly contributed by the group of high density to high density, explaining why the index is 0.69. For other groups, it seems not too much connection (spatial correlation) between the count of listings and locations.</p><p>The final step is to map all group into our grid map:</p><pre>fig, ax = plt.subplots(figsize=(14,12))<br>grid.plot(color=&quot;white&quot;, edgecolor=&#39;black&#39;, ax= ax)<br>lisa_cluster(m_local, num_per_pix, p=0.01, figsize = (16,12),ax=ax)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/949/1*4amNtYgYn-O_I1XR3iGAEA.png" /><figcaption>Grid map with classified groups</figcaption></figure><h3>Conclusion</h3><p>Again, the techniques used above are referred and learned by this medium article and you can find more detail about spatial autocorrelation from:</p><p><a href="https://towardsdatascience.com/what-is-exploratory-spatial-data-analysis-esda-335da79026ee">What is Exploratory Spatial Data Analysis (ESDA)?</a></p><p>You can also click my Github repository of Airbnb Hong Kong for the original code of 5 articles in my medium:</p><p><a href="https://github.com/MartinYee/AirbnbHongKong">MartinYee/AirbnbHongKong</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b06f3d4158d3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring the spatial distribution of Airbnb listings in Hong Kong, Part 4]]></title>
            <link>https://medium.com/@zidongyu26/exploring-the-spatial-distribution-of-airbnb-listings-in-hong-kong-2465d8c7e667?source=rss-3339ed5adf2------2</link>
            <guid isPermaLink="false">https://medium.com/p/2465d8c7e667</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[transportation]]></category>
            <category><![CDATA[hong-kong]]></category>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[travel]]></category>
            <dc:creator><![CDATA[MARTIN YU]]></dc:creator>
            <pubDate>Tue, 04 Feb 2020 16:50:44 GMT</pubDate>
            <atom:updated>2020-02-22T16:19:11.070Z</atom:updated>
            <content:encoded><![CDATA[<p>Article 4: Using unsupervised ML classification techniques to explore clustering phenomenon.</p><blockquote>My previously story of clustering:</blockquote><p><a href="https://medium.com/@zidongyu26/exploring-the-spatial-distribution-of-airbnb-listings-in-hong-kong-a4abbb626e4e">Exploring the spatial distribution of Airbnb listings in Hong Kong</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/1*Mx54czmTDd5oUbedAufRJA.jpeg" /><figcaption>The high-density residential mansions in Tin Shui Wai</figcaption></figure><p>Clustering is an Unsupervised Machine Learning technique, involving that grouping of point features.</p><blockquote>“In theory, data points that are in the same group should have similar properties and/or features, while data points in different groups should have highly dissimilar properties and/or features( <a href="https://towardsdatascience.com/the-5-clustering-algorithms-data-scientists-need-to-know-a36d136ef68">Seif, 2018</a>)”</blockquote><p>In this article, two clustering skills will be employed that they are K-Means clustering algorithm and DB-Scan clustering algorithm.</p><p>K-Means clustering algorithm is an iterative clustering algorithm. Its steps are to randomly select K objects as the initial cluster center, then calculate the distance between each object and each seed cluster center, and assign each object to the neighbour cluster center with the nearest distance. Before fitting the K-Means algorithm to our data, we need to predefine K, the number of cluster centers we want, and then all data points will be clustered into K clusters.</p><p>The DB-Scan clustering algorithm is a clustering algorithm based on density space. Unlike the K-Means algorithm, it does not need to determine the number of clusters but infers the number of clusters based on the data. It can generate clusters for any shape.</p><p>For both clustering algorithm, the latitude and longitude will be imported as the properties or features and then classified.</p><p>For more information about clustering, you can refer to the below link:</p><p><a href="https://towardsdatascience.com/the-5-clustering-algorithms-data-scientists-need-to-know-a36d136ef68">The 5 Clustering Algorithms Data Scientists Need to Know</a></p><h3>K-Means Clustering Algorithm</h3><p>First of all, we plot data using geopandas:</p><pre># Import related library<br>import geopandas as gpd</pre><pre># Importing the Hong Kong 18 districts boundary GeoJSON file<br>map_data = gpd.read_file(&quot;HK/Hong_Kong_18_Districts/Hong_Kong_18_Districts.shp&quot;)</pre><pre><br># Dropping the unwanted column<br>map_data.drop(&quot;TCNAME&quot;, axis = 1, inplace = True<br></pre><pre># Extract data<br>map_data = map_data.loc[map_data[&#39;ENAME&#39;].isin({&#39;CENTRAL &amp; WESTERN&#39;, &#39;WAN CHAI&#39;, &#39;YAU TSIM MONG&#39;})]</pre><pre># Prepare for the location of label<br>map_data[&quot;center&quot;] = map_data[&quot;geometry&quot;].centroid<br>top3_points = map_data.copy()<br>top3_points.set_geometry(&quot;center&quot;, inplace = True)</pre><p>And, read the data generated by previous articles:</p><pre>data = pd.read_csv(&quot;HK/top3.csv&quot;)<br># Dropping out the unwanted columns<br>data = data.drop({&#39;Unnamed: 0&#39;, &#39;host_id&#39;, &#39;minimum_nights&#39;,&#39;number_of_reviews&#39;, &#39;calculated_host_listings_count&#39;, &#39;availability_365&#39;, &#39;price&#39;}, axis = 1)</pre><p>Plot all data points as scatter map:</p><pre>map_top3 = map_data.plot(figsize=(12,10), color = &#39;whitesmoke&#39;, edgecolor=&#39;black&#39;, linewidth = 1, legend = True)<br>texts = []<br>plt.scatter(data[&#39;longitude&#39;], data[&#39;latitude&#39;], c=&#39;blue&#39;, marker = &#39;.&#39;, alpha = 0.4)<br>for x, y, label in zip(top3_points.geometry.x, top3_points.geometry.y, top3_points.ENAME):<br>    texts.append(plt.text(x, y, label, fontsize = 8, size = 12, color = &#39;red&#39;, weight =&#39;bold&#39;))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/753/1*on5vDo7I5fDWd14NU_j-_Q.png" /><figcaption>The scatter map of Airbnb listing data for certain districts in Hong Kong.</figcaption></figure><p>The first clustering algorithm we use is the K-Means:</p><pre># import KMeans<br>from sklearn.cluster import KMeans</pre><p>For the district-level, we assume that 3 clusters are existing within our study area, for the YAU TSIM MONG, CENTRAL &amp; WESTERN, and WAN CHAI, respectively</p><pre># create np array for data points<br>points = data[[&#39;longitude&#39;, &#39;latitude&#39;]].to_numpy()</pre><pre># Create kmeans object<br>kmeans = KMeans(n_clusters=3)<br># Fit kmeans object to data<br>kmeans.fit(points)<br># print location of clusters learn by keams object<br>print(kmeans.cluster_centers_)<br># save new clusters for chart<br>y_km = kmeans.fit_predict(points)</pre><p>Time for plotting the result of clustering:</p><pre># Define the base map<br>map_top3 = map_data.plot(figsize=(12,10), color = &#39;whitesmoke&#39;, edgecolor=&#39;black&#39;, linewidth = 1, legend = True)<br>texts = []</pre><pre># Plotting out all points<br>plt.scatter(points[:,0], points[:,1],c=y_km, cmap=&#39;Paired&#39;)</pre><pre># Adding label for each district<br>for x, y, label in zip(top3_points.geometry.x, top3_points.geometry.y, top3_points.ENAME):<br>    texts.append(plt.text(x, y, label, fontsize = 8, size = 12, color = &#39;red&#39;, weight =&#39;bold&#39;))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/887/1*yM8oCYz1QaIQHXBlYtegMw.png" /><figcaption>The result generated by K-Means clustering algorithm, for K = 3</figcaption></figure><p>Looking so good, and the result generally met our anticipation, as the map shown above, all data points are divided into 3 parts (since K, the number of clusters, is 3 set initially), demonstrating that the distribution of Airbnb listings within one district may have same geographical patterns.</p><p>Now, let’s zoom in to Yau Tsim Mong district, along with the auxiliary of Tsuen Wan line, depicting the relationship between Airbnb listings and MTR stations.</p><pre>data_yau = data[data.neighbourhood == &#39;YAU TSIM MONG&#39;]<br># create np array for data points<br>points_yau = data_yau[[&#39;longitude&#39;, &#39;latitude&#39;]].to_numpy()<br>yau_points = top3_points[top3_points.ENAME == &#39;YAU TSIM MONG&#39;]<br># Extract map of YAU TSIM MONG<br>map_yau = map_data.loc[map_data[&#39;ENAME&#39;].isin({&#39;YAU TSIM MONG&#39;})]</pre><p>And add the Tsuen Wan Line (MTR):</p><pre>tsuen_wan_line = {<br> &#39;station&#39;:[&#39;Tsim Sha Tsui Station&#39;, &#39;Jordan Station&#39;, &#39;Yau Ma Tei Station&#39;, &#39;Mong Kok Station&#39;, &#39;Prince Edward Station&#39;],<br>        &#39;latitude&#39;:[22.297560, 22.3048588, 22.313351, 22.319232, 22.324611],<br>        &#39;longitude&#39;:[114.172209, 114.17172, 114.170572, 114.169349, 114.168269]<br>}<br>tsuen_wan_line = pd.DataFrame(tsuen_wan_line)<br>tsuen_wan_line</pre><p>Since that we don’t know the optimal number of clusters(K) for Yau Tsim Mong, we will create a for loop to plot a series of maps with a range of K, for exploring the best K value.</p><pre># Define the range of K<br>K = [3,4,5,6]</pre><pre>for i,k in enumerate(K):<br>    # Create kmeans object<br>    kmeans = KMeans(n_clusters=k)<br>    # Fit kmeans object to data<br>    kmeans.fit(points_yau)<br>    # print location of clusters learn by keams object<br>    #print(kmeans.cluster_centers_)<br>    # save new clusters for chart<br>    y_km = kmeans.fit_predict(points_yau)<br>    <br>    map_yau.plot(figsize=(6,5), color = &#39;whitesmoke&#39;, edgecolor=&#39;black&#39;, linewidth = 1, legend = True)<br>    plt.scatter(points_yau[:,0], points_yau[:,1],c=y_km, cmap=&#39;Accent&#39;)<br>    plt.plot(tsuen_wan_line.longitude, tsuen_wan_line.latitude, color=&#39;purple&#39;, marker=&#39;o&#39;, linestyle=&#39;dashed&#39;, linewidth=3, markersize=12)<br>    for x, y, label in zip(yau_points.geometry.x, yau_points.geometry.y, yau_points.ENAME):<br>        texts.append(plt.text(x, y, label, fontsize = 8, size = 12, color = &#39;red&#39;, weight =&#39;bold&#39;))<br>    plt.title(&quot;The clustering result of k= &quot; + str(k))<br>    plt.xlabel(&quot;Longitude&quot;)<br>    plt.ylabel(&quot;Latitude&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/431/1*1EPYanHMJiZNPKpCeuUjmg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/424/1*YRpTnCMGpzvnUEbgDRtSVg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/422/1*aOHhBMRO2XOILlwkFXCfwA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/421/1*UwxlPfdUNCV8emWHatY0Bg.png" /></figure><p>Based on the visual assessment, the map of K = 6 may have a better performance compared to the plots with other K values, thus, we furthermore plot the clustering map with K =6, in a larger figure size.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/617/1*-EeOi5PRkghDR1pz--VbBQ.jpeg" /><figcaption>The result generated by K-Means clustering algorithm, for K = 6, within Yau Tsim Mong district.</figcaption></figure><p>Here, on the basis of the clustering map generated by K-Means algorithm, as displayed above, some meaningful suggestions can be concluded and learned, first, by generating 6 clusters in total, 5 of them are along the MTR Tsuen Wan line, demonstrating that the direction of urban planning in Hong Kong possibly is a rail-based transit-oriented type. Secondly, for the points with cyan color, isolating from the remaining clusters, we can draw an observation that one or more than one communities may exist and cluster in those areas like East Tsim Sha Tsui. The third point suggests that the topmost MTR station Prince Edwards in this map may share the cluster with other area, as presented in green color.</p><h3>DB-Scan Clustering Algorithm</h3><p>Import the wanted libraries</p><pre>import numpy as np<br>from sklearn.cluster import DBSCAN<br>from sklearn.datasets.samples_generator import make_blobs<br>from sklearn.neighbors import NearestNeighbors<br>from sklearn.cluster import DBSCAN<br>sns.set()</pre><p>For using DB-Scans algorithm, it requires two parameters as input: the minimum number of neighbour points(min_sample) and the neighbour radius(epsilon, eps). In this article, the value of the neighbour radius will be firstly obtained by using K-distance plot:</p><pre>neigh = NearestNeighbors(n_neighbors=3)<br>nbrs = neigh.fit(points_yau)<br>distances, indices = nbrs.kneighbors(points_yau)</pre><pre>distances = np.sort(distances, axis=0)<br>distances = distances[:,1]<br>plt.plot(distances)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/1*eJPcMNzLDgquDri9EcAmAA.png" /><figcaption>The K-distance plot with n_neighbors = 3</figcaption></figure><p>Based on the k-distance plot, the elbow point of the k-distance plot may be the ideal value of eps, besides, since we are using the latitude and longitude as the clustering features, it is reasonable to use haversine metric and ball tree algorithm to compute the great circle distance between points, this come from the tutorial <a href="https://geoffboeing.com/2014/08/clustering-to-reduce-spatial-data-set-size/">https://geoffboeing.com/2014/08/clustering-to-reduce-spatial-data-set-size/</a>.</p><pre>dbscan = DBSCAN(eps=2/6371., min_samples=15, algorithm=&#39;ball_tree&#39;, metric=&#39;haversine&#39;)<br># Fit kmeans object to data<br>dbscan.fit(points_yau)<br># save new clusters for chart<br>y_db = dbscan.fit_predict(points_yau)<br>labels = dbscan.labels_<br>num_of_clusters = len(set(labels))<br>    <br>map_yau.plot(figsize=(12,10), color = &#39;whitesmoke&#39;, edgecolor=&#39;black&#39;, linewidth = 1, legend = True)<br>plt.scatter(points_yau[:,0], points_yau[:,1],c=y_db, cmap=&#39;gist_ncar&#39;, alpha = 0.4)<br>plt.plot(tsuen_wan_line.longitude, tsuen_wan_line.latitude, color=&#39;red&#39;, marker=&#39;.&#39;, linestyle=&#39;-.&#39;, linewidth=3, markersize=20)<br>plt.title(&quot;The DBScan clustering result of num_of_clusters = &quot; + str(num_of_clusters) + &#39; ,min_samples = 15&#39;)<br>plt.xlabel(&quot;Longitude&quot;)<br>plt.ylabel(&quot;Latitude&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/748/1*w1aTuJMY3yMOnA0Bf3NofA.png" /></figure><p>Unlike the result generated by using K-Means algorithm, the result of using DB-Scan algorithm pays more attention to the space density of the data points, identifying the arbitrary shape of the Airbnb listings, however, the bulk of the clusters still have some pattern, from the perspective of Tsuen Wan line. Except for the station of Prince Edwards, other MTR stations both are surrounded by continuously clustered areas, coloring by dark blue, green, green, light blue. Besides, in contrast with the result of K-Means, the clustered area is smaller and more dispersed, revealing the existing of micro-scale and cohesive communities in these areas.</p><h3>Conclusion</h3><p>Ultimately, in this article, two unsupervised clustering algorithms have been applied for exploring some hidden pattern of Airbnb listings, from the view of rail-based transit-oriented development. And, we did find some interesting patterns within our study area, proving the assumption made by other scholars that Hong Kong is a typical sample of using a railway-based transit-oriented developed strategy for urban planning, affecting the distribution of residence(Loo et al., 2010).</p><p><strong>Reference:</strong></p><p>Loo, B. P., Chen, C., &amp; Chan, E. T. (2010). Rail-based transit-oriented development: lessons from New York City and Hong Kong. <em>Landscape and Urban Planning</em>, <em>97</em>(3), 202–212.</p><blockquote><em>Some of my previous articles:</em></blockquote><ul><li><a href="https://medium.com/@zidongyu26/exploring-the-spatial-distribution-of-airbnb-listings-in-hong-kong-a4abbb626e4e">Exploring the spatial distribution of Airbnb listings in Hong Kong</a></li><li><a href="https://medium.com/analytics-vidhya/exploring-the-spatial-distribution-of-airbnb-listings-in-hong-kong-5e10e1b8fe30">Exploring the spatial distribution of Airbnb listings in Hong Kong</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2465d8c7e667" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring the spatial distribution of Airbnb listings in Hong Kong, Part 3]]></title>
            <link>https://medium.com/@zidongyu26/exploring-the-spatial-distribution-of-airbnb-listings-in-hong-kong-a4abbb626e4e?source=rss-3339ed5adf2------2</link>
            <guid isPermaLink="false">https://medium.com/p/a4abbb626e4e</guid>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[hong-kong]]></category>
            <category><![CDATA[airbnb]]></category>
            <category><![CDATA[mapping]]></category>
            <category><![CDATA[spatial-analysis]]></category>
            <dc:creator><![CDATA[MARTIN YU]]></dc:creator>
            <pubDate>Fri, 31 Jan 2020 16:06:15 GMT</pubDate>
            <atom:updated>2020-02-22T16:16:30.410Z</atom:updated>
            <content:encoded><![CDATA[<p>Article 3: Revealing the cluster phenomenon of sharing lodgings, in terms of public transport nodes, on the basis of Folium.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FGCuXKPxfndE3qhmZu_uHQ.jpeg" /><figcaption>The Entry of Cross-Harbour Tunnel, Hung Hom, Hong Kong.</figcaption></figure><h3>Introduction</h3><p>It is known that Hong Kong has a highly developed and complex pubic transport network system, providing a punctual and safe transport service to people living in Hong Kong. Based on the travel survey conducted by the Government of HK’s, more than 90 percent of daily commute trips rely on the public transport system, which is the highest rate all over the world(Lam and Bell, 2002). Besides, some scholars still suggested that Hong Kong is a typical sample of using a railway-based transit-oriented developed strategy for urban planning, affecting the distribution of residence(Loo et al., 2010).</p><p>Therefore, in this article, combining with the consideration of Mass Transit Railway (MTR) in Hong Kong, on the basis of <a href="https://python-visualization.github.io/folium/">Folium</a>, some data visualization, and clustering techniques will be employed to reveal some hidden stories of Airbnb lodging lists’ spatial distribution.</p><h3>Data Pre-processing</h3><p>Before fitting the data into the model, as the previous articles’ strategies, it is critical to preprocess the raw data, retaining the concise portion of the data for the next step.</p><pre># import the libraries need to be used<br>import pandas as pd<br>import numpy as np<br>import matplotlib<br>import matplotlib.pyplot as plt<br>import seaborn as sns<br>import geopandas as gpd<br>%matplotlib inline</pre><p>As usual, import some basic and general-used libraries, as shown above. And then, read the data coming from previous articles:</p><pre># Read the data which come from my previous articles<br>data = pd.read_csv(&quot;HK/top3_dist.csv&quot;)<br>data = data.drop({&#39;Unnamed: 0&#39;, &#39;host_id&#39;, &#39;minimum_nights&#39;,&#39;number_of_reviews&#39;, &#39;calculated_host_listings_count&#39;, &#39;availability_365&#39;, &#39;price&#39;}, axis = 1)<br>data.head(2)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/542/1*RWChZw39neBmKeHAmWv1YA.png" /><figcaption>The screenshot of the output data</figcaption></figure><h3>Location Analysis by Using Folium</h3><p>Prior to employing unsupervised clustering techniques provided by <a href="https://scikit-learn.org/"><strong><em>scikit-learn</em></strong></a><strong><em> </em></strong>module, we firstly learn and use some basic mapping functions with <a href="https://python-visualization.github.io/folium/"><strong><em>Folium</em></strong></a><strong><em>.</em></strong></p><p>In this part, the data we would use is located within the YAU TSIM MONG district, so, we have to extract the wanted data:</p><pre>data_yau = data[data.neighbourhood == &#39;YAU TSIM MONG&#39;]</pre><p>Import folium:</p><pre>import folium<br>from IPython.display import display</pre><p>Create a data frame of main MTR stations in study area, in this case, we would use Tsuen Wan line:</p><pre>tsuen_wan_line = {<br>	&#39;station&#39;:[&#39;Tsim Sha Tsui Station&#39;, &#39;Jordan Station&#39;, &#39;Yau Ma Tei Station&#39;, &#39;Mong Kok Station&#39;, &#39;Prince Edward Station&#39;],<br>        &#39;latitude&#39;:[22.297560, 22.3048588, 22.313351, 22.319232, 22.324611],<br>        &#39;longitude&#39;:[114.172209, 114.17172, 114.170572, 114.169349, 114.168269]<br>}</pre><pre>tsuen_wan_line = pd.DataFrame(tsuen_wan_line)<br>tsuen_wan_line</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/361/1*2uons_yw0YF4_vYejLn5dA.png" /><figcaption>MTR stations of Tsuen Wan Line within Yau Tsim Mong</figcaption></figure><pre># Convert the coordinates of stations to list<br>coordinates  = list(zip(tsuen_wan_line.latitude, tsuen_wan_line.longitude))</pre><p>Define a map centered around Yau Tsim Mong district and plot all data of entire apartment and home, as point features:</p><pre>hk_map_en = folium.Map(location=[22.31,114.18],zoom_start = 13)</pre><pre>data_yau[data_yau.room_type == &#39;Entire home/apt&#39;].apply(lambda row:folium.CircleMarker(location=[row[&quot;latitude&quot;], row[&quot;longitude&quot;]],radius=1,color=&#39;red&#39;,fill=False,popup=&#39;Entire home/apt&#39;,fill_opacity=0.6).add_to(hk_map_en), axis=1)</pre><p>And then, draw out the Tsuen Wan Line of MTR:</p><pre>folium.PolyLine(locations=coordinates,weight=5,color = &#39;purple&#39;).add_to(hk_map_en)</pre><pre># Plotting the points of the MTR stations in our study area<br>for lat, lng, label in zip(tsuen_wan_line.latitude, tsuen_wan_line.longitude, tsuen_wan_line.station.astype(str)):<br>    folium.CircleMarker(<br>            [lat, lng],<br>            radius=4,<br>            color=&#39;blue&#39;,<br>            fill=True,<br>            popup=label,<br>            fill_color=&#39;blue&#39;,<br>            fill_opacity=1<br>        ).add_to(hk_map_en)</pre><p>Finally, save the output as html format:</p><pre>hk_map_en.save(&quot;map_entire.html&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/946/1*GZj0vdK24wYyfqAYf40twQ.png" /><figcaption>The distribution of entire apartment &amp; home in Yau Tsim Mong area.</figcaption></figure><p>As the map of the distribution of the entire apartment &amp; home in Yau Tsim Mong area, shown above, it can be concluded that a great number of Airbnb listings are located along the Tsuen Wan MTR line, indicating that the rail-based transit direction may affect the development of residence projects strongly. In addition, this map presents that Tsim Sha Tsui, Jordan as well as Mong Kok areas may have higher residence density than other areas in Yau Tsim Mong district.</p><p>Same processed were conducted for the Airbnb listings of the private room, and the output map shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/767/1*TWip85zL6qbptwdt0xTagg.png" /><figcaption>The distribution of private room in Yau Tsim Mong area.</figcaption></figure><p>Generally speaking, the pattern of the distribution of private rooms is similar to the previous map of the entire home &amp; apartment, however, for the private room type, it is clear to find out that the density around MTR stations is obviously higher than the map of theentire unit.</p><p>By using this mapping tool, given by Folium, we can easily and quickly plot all data on out study area as point features, furthermore, we are also able to estimate the accuracy of the record points, taking an example, referring to the above map, few Airbnb lodgings are located in the Victoria Harbour (the sea surface), little bit weird.</p><h3>Using MarkerCluster to group markers into different clusters</h3><p>Inside Folium, clustering function is also provided by grouping markers into different clusters, by applying the MarkerCluster object. The subsequent part depicts all types of Airbnb listings within Yau Tsim Mong district, and present them by grouping into clusters.</p><pre>from folium import plugins</pre><pre># create a map object of Yau Tsim Mong district<br>hk_map = folium.Map(location=[22.31,114.18],zoom_start = 13)</pre><p>As ever, plot Tsuen Wan line on map:</p><pre># And then, drawing out the Tsuen Wan Line of MTR, one of most important MTR route in HK<br>folium.PolyLine(locations=coordinates,weight=5,color = &#39;purple&#39;).add_to(hk_map)</pre><pre># Plotting the points of the MTR stations in our study area.<br>for lat, lng, label in zip(tsuen_wan_line.latitude, tsuen_wan_line.longitude, tsuen_wan_line.station.astype(str)):<br>    folium.CircleMarker(<br>            [lat, lng],<br>            radius=4,<br>            color=&#39;blue&#39;,<br>            fill=True,<br>            popup=label,<br>            fill_color=&#39;blue&#39;,<br>            fill_opacity=1<br>        ).add_to(hk_map)</pre><p>Instatiate a mark cluster object for Aribnb listings:</p><pre>listings = plugins.MarkerCluster().add_to(hk_map)</pre><pre># Display the listings inside Yau Tsim Mong<br>for lat, lng, label, in zip(data_yau.latitude, data_yau.longitude, data_yau.room_type):<br>    if label != 0:<br>        folium.Marker(<br>            location=[lat,lng],<br>            icon=None,<br>            popup=label<br>        ).add_to(listings)</pre><p>Save the cluster map to local:</p><pre>hk_map.save(&quot;map_cluster.html&quot;)</pre><h3>Analysis and Discussion</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/783/1*xd2tOv8ty4W6PoLyIakmSA.png" /><figcaption>The clustered map at a relatively smaller scale</figcaption></figure><p>Based on the above map, it is evident to indicate that bulk of the Airbnb listings is located along with the Tsuen Wan MTR line, additionally, quite a few lodgings stay away from the Tsuen Wan line, grouping on Hung Hom area, a famous residence community in Kowloon, Hong Kong.</p><p>Moving to the map with a larger scale,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/1*LXRMGYbxbPRNGRQhTn0p-g.png" /><figcaption>The clustered map at a relatively largeer scale</figcaption></figure><p>When we zoom in to a larger scale, we did find something different, as shown above, despite that the majority of listings is still along Tsuen Wan line, Tai Kok Tsui and Eastern Tsim Sha Tsui provide a certain amount of Airbnb lodgings, this is probably because some high-density residential projects carried in those areas.</p><p>What should be mentioned here is that I just plotted a portion of the Tsuen Wan line on the map of study area, other MTR stations are neglected in this case study like East Tsim Sha Tsui Station, Hung Hom Station, Austin Station and so on…</p><p>Once, I attended a lecture of urban science and system and an urban planning expert of Hong Kong had said:</p><blockquote>“Above 90 percent of dwellings is located within 500 meters of MTR stations.”</blockquote><h3>Conclusion</h3><p>Ultimately, in this article, we have used clustering function provided by Folium, to explore the distribution of Airbnb listings, from the perspective of the clustering phenomenon. Consequently, the clustered map generated by Folium has suggested that the bulk of Airbnb lodgings are aggregated around some MTR stations of Tsuen Wan line, demonstrating the idea of rail-based transit-oriented development and enhancing the sustainability of transportation activities.</p><p>Reference:</p><p>Lam, W. H., &amp; Bell, M. G. (2002). <em>Advanced modeling for transit operations and service planning</em>.</p><p>Loo, B. P., Chen, C., &amp; Chan, E. T. (2010). Rail-based transit-oriented development: lessons from New York City and Hong Kong. <em>Landscape and Urban Planning</em>, <em>97</em>(3), 202–212.</p><blockquote>My previous articles:</blockquote><ul><li><a href="https://medium.com/@zidongyu26/exploring-the-spatial-distribution-of-airbnb-listings-in-hong-kong-816a3f56aa2c">Exploring the spatial distribution of Airbnb listings in Hong Kong</a></li><li><a href="https://medium.com/analytics-vidhya/exploring-the-spatial-distribution-of-airbnb-listings-in-hong-kong-5e10e1b8fe30">Exploring the spatial distribution of Airbnb listings in Hong Kong</a></li></ul><blockquote>For future article:</blockquote><p>The fourth article of this series will be uploaded shortly, talking about how to utilize unsupervised classification to clustering data points.</p><p>My GitHub’s repository for this series is still under construction and will be available soon.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a4abbb626e4e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring the spatial distribution of Airbnb listings in Hong Kong, Part 2]]></title>
            <link>https://medium.com/analytics-vidhya/exploring-the-spatial-distribution-of-airbnb-listings-in-hong-kong-5e10e1b8fe30?source=rss-3339ed5adf2------2</link>
            <guid isPermaLink="false">https://medium.com/p/5e10e1b8fe30</guid>
            <category><![CDATA[geospatial]]></category>
            <category><![CDATA[landuse]]></category>
            <category><![CDATA[tourism]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[linear-regression]]></category>
            <dc:creator><![CDATA[MARTIN YU]]></dc:creator>
            <pubDate>Sun, 05 Jan 2020 15:55:56 GMT</pubDate>
            <atom:updated>2020-02-22T16:16:47.831Z</atom:updated>
            <content:encoded><![CDATA[<p>Article 2: Examining the Bid-Rent Theory, in term of Airbnb Listings</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D31ksNO6SQleQ1uQhQBfiw.jpeg" /><figcaption>The Bid-rent curve</figcaption></figure><h3>Introduction</h3><p>As aforementioned article noted, Bid Rent Theory, one of the iconic study cases in urban land-use discipline, was designed by William Alonso and published in 1964, basically explaining the relationship between the price and the land demand in case of a change in distance from an assumed central business district (CBD) and, the principle of the bid rent theory is that land rent would be lower where the areas are getting farther from the central area of the city, which is based on the extension of the von Thunen model on urban land uses(Alonso, 1964).</p><p>To examine the Bid Rent Theory, from the perspective of sharing economy lodging, in this article, we are gonna do some data pre-processing, model building and then attempt to find out the relationship between the room prices and distance from the central point of study area, testifying the hypothesis that established and given by the Bid Rent Theory.</p><h3>Dara Pre-processing</h3><p>Before fitting the data into the model, it is essential to process the data, retaining the useful part of the data for next step.</p><pre># import the libraries need to be used<br>import pandas as pd<br>import numpy as np<br>import matplotlib<br>import matplotlib.pyplot as plt<br>import seaborn as sns<br>import geopandas as gpd<br>%matplotlib inline</pre><p>And then, based on the result of theprevious article, since Yau Tsim Mong, Central &amp; Western and Wan Chai districts can provide more than 2000 lodgings, giving a large enough sample for our examination, therefore, we retrieve the data of that 3 districts.</p><pre># import the data processed and used by last article<br>data = pd.read_csv(&quot;HK/top3.csv&quot;).drop(&#39;Unnamed: 0&#39;, axis = 1)</pre><pre>geo_data  = gpd.read_file(&quot;HK/Hong_Kong_18_Districts/Hong_Kong_18_Districts.shp&quot;).drop(&#39;TCNAME&#39;, axis = 1)</pre><p>Using head() to take a peek about the imported data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4ZX4T60W9c6vQlTyxryafg.png" /></figure><p>We find some unnecessary columns inside the data and remove the extreme value of price, so drop them down:</p><pre># Filter out extremely high price and drop some useless columns<br>data = data[data.price &lt;= 	     6000].drop([&#39;host_id&#39;,&#39;minimum_nights&#39;,&#39;number_of_reviews&#39;                ,&#39;calculated_host_listings_count&#39;,&#39;availability_365&#39;], axis = 1)</pre><pre>geo_data = geo_data.drop([&#39;OBJECTID&#39;,&#39;Shape__Are&#39;,&#39;Shape__Len&#39;], axis = 1)</pre><blockquote>Now, 2 things we should do that are firstly locating the central point of each district as well as secondly computing the distance between each host and its corresponding central point:</blockquote><blockquote><strong>Central point of each district:</strong></blockquote><p>For the central point, here, based on my experience, I will assume that the Peninsula Hong Kong, the Central Plaza and the HSBC building, are the central point of YAU TSIM MONG, WAN CHAI, AND CEENTRAL&amp;WESTERN, respectively, and therefore obtain centroid coordinates from google map.</p><pre>geo_data.loc[geo_data.ENAME == &#39;YAU TSIM MONG&#39;,&quot;geometry_lat&quot;] = 22.295062<br>geo_data.loc[geo_data.ENAME == &#39;YAU TSIM MONG&#39;,&quot;geometry_lon&quot;] = 114.171865<br>geo_data.loc[geo_data.ENAME == &#39;WAN CHAI&#39;,&quot;geometry_lat&quot;] = 22.280025 <br>geo_data.loc[geo_data.ENAME == &#39;WAN CHAI&#39;,&quot;geometry_lon&quot;] = 114.173477<br>geo_data.loc[geo_data.ENAME == &#39;CENTRAL &amp; WESTERN&#39;,&quot;geometry_lat&quot;] = 22.280641<br>geo_data.loc[geo_data.ENAME == &#39;CENTRAL &amp; WESTERN&#39;,&quot;geometry_lon&quot;] = 114.159040</pre><blockquote><strong>The distance between host and central point:</strong></blockquote><p>Before we calculate the distance between two lat/long coordinates, we have to merge data with geo_data together, combining the host location and central points.</p><pre># Merge two data frames into one data frame<br>top_3 = pd.merge(data, geo_data, how=&#39;inner&#39;, left_on=&#39;neighbourhood&#39;, right_on=&#39;ENAME&#39;).drop([&#39;ENAME&#39;,], axis = 1).drop([&#39;geometry&#39;], axis=1)<br>top_3.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*dTzPDjZkyRzDZyFBnwRI_A.png" /><figcaption>The head rows of merged data</figcaption></figure><p>And then, we import the geopy library for calculating distance:</p><pre>import geopy.distance</pre><p>What should be care here is that, two types of the algorithm are usually used to calculate the distance: the great circle and the geodesic, for the great circle, it computes the great circle distance based on the surface of a sphere while the geodesic calculates the distance based on the surface of an ellipsoid.</p><p>Generally speaking, the algorithm of geodesic distance is more accurate than the great circle algorithm, since the earth’s surface is actually an ellipsoid instead of a sphere as human’s expectation. However, in our case, due to the microscale of our study area, the difference using different distance algorithms does not play an important role, thus, here, I will show the solutions of both two methods but only employ the geodesic distance to represent the distance.</p><pre># Using two different algorithm to calculate distance<br>top_3[&#39;distance_geodesic&#39;] = top_3.apply(<br>    (lambda row: geopy.distance.geodesic(<br>        (row[&#39;latitude&#39;], row[&#39;longitude&#39;]),<br>        (row[&#39;geometry_lat&#39;], row[&#39;geometry_lon&#39;])<br>    ).m),<br>    axis=1<br>)</pre><pre>top_3[&#39;distance_great_circle&#39;] = top_3.apply(<br>    (lambda row: geopy.distance.great_circle(<br>        (row[&#39;latitude&#39;], row[&#39;longitude&#39;]),<br>        (row[&#39;geometry_lat&#39;], row[&#39;geometry_lon&#39;])<br>    ).m),<br>    axis=1<br>)</pre><h3><strong>Linear Model</strong></h3><p>First, we import the libraries we need:</p><pre># Import the libraries of linear regression<br>from sklearn.linear_model import LinearRegression<br>from sklearn import metrics</pre><p>Because the different types of Airbnb lodging may affect the price, we have to divide and fit the data for their own room types. Extract all room types:</p><pre># Find out all unique values of room_type<br>set(top_3.room_type)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*aOZse00VLTO_kfxq5R7aZg.png" /></figure><p>Obviously, there are totally three room types for entire home/apartment, private room as well as shared room.</p><p>For fitting a linear model and drawing out the relationship, here, I am gonna use YAU TSIM MONG district as a demonstration, containing the largest population of 4698 instances:</p><pre># Extract the data of YAU TSIM MONG<br>yau = top_3[top_3.neighbourhood == &#39;YAU TSIM MONG&#39;]</pre><pre># Divide the data by their room types<br>yau_en = yau[yau.room_type == &#39;Entire home/apt&#39;]<br>yau_pr = yau[yau.room_type == &#39;Private room&#39;]<br>yau_sh = yau[yau.room_type == &#39;Shared room&#39;]</pre><p>Here, we use the entire home/apartment in YAU TSIM MONG as example:</p><pre># Define the X and Y, and then fit them into linear model<br>x1 = yau_en[[&#39;distance_geodesic&#39;]]<br>y1 = yau_en[[&#39;price&#39;]]<br>model = LinearRegression()<br>model.fit(x1,y1)</pre><pre># Get the predicted y from trained model<br>yau_en_pred = model.predict(x1)</pre><p>Repeat the code above for other room types as well as other two districts.</p><p>Private room in YAU TSIM MONG:</p><pre>x2 = yau_pr[[&#39;distance_geodesic&#39;]]<br>y2 = yau_pr[[&#39;price&#39;]]<br>model = LinearRegression()<br>model.fit(x2,y2)</pre><pre>yau_pr_pred = model.predict(x2)</pre><p>Shared room in YAU TSIM MONG:</p><pre>x3 = yau_sh[[&#39;distance_geodesic&#39;]]<br>y3 = yau_sh[[&#39;price&#39;]]<br>model = LinearRegression()<br>model.fit(x3,y3)</pre><pre>yau_sh_pred = model.predict(x3)</pre><p>Finally, plot all data points as well as the line fitted by linear regression:</p><pre># Plot outputs<br>plt.figure(figsize=(9,9))</pre><pre>plt.scatter(yau_en.distance_geodesic, yau_en.price, marker = &#39;.&#39;, color = &#39;lightsteelblue&#39;, alpha = 0.5 ,label = &#39;Entire home/apt&#39;)<br>plt.scatter(yau_pr.distance_geodesic, yau_pr.price, marker = &#39;.&#39;, color = &#39;lightcoral&#39;, alpha = 0.5, label =&#39;Private room&#39;)<br>plt.scatter(yau_sh.distance_geodesic, yau_sh.price, marker = &#39;.&#39;, color = &#39;lightgreen&#39;, alpha = 0.5, label =&#39;Shared room&#39;)</pre><pre>plt.plot(x1, yau_en_pred, color = &#39;blue&#39;, label =&#39;Entire home/apt \n coef: 0.118&#39;)<br>plt.plot(x2, yau_pr_pred, color = &#39;red&#39;, label =&#39;Private room \n coef: -0.026&#39;)<br>plt.plot(x3, yau_sh_pred, color = &#39;darkgreen&#39;, label =&#39;Shared room \n coef: 0.070&#39;)</pre><pre>plt.gca().set_xlabel(&#39;Distance to centroid point(unit:m)&#39;)<br>plt.gca().set_ylabel(&#39;Price per night(HK$)&#39;)<br>plt.legend()<br>plt.title(&#39;The relationship between price and diatance to centre in YAU TSIM MONG&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/648/1*RvXGLkWKrLQQ8UrXWtCu4g.png" /><figcaption>Line and scatter plot of the relationship between price and distance to centre</figcaption></figure><h3>Analysis and Discussion</h3><p>Referring to the plot shown above, we can figure out something interesting, for example, concerning on the type of entire home/apartment and the blue colour for line and light blue colour for data points, it is evident that the price of entire home/apartment is higher than the other two types of room, meeting our expectations.</p><p>For the private rooms and shared rooms with the line and points of red and green colours, separately, the difference between them is smaller compared to the difference of entire home/apartment, indicating that these types of rooms may be the priority choices for low-income tourist that spend less money on their lodging, particularly for the shared rooms.</p><p>Overall, in this case, we can see that the price of all Airbnb lodgings decreased along with the distance to central point, presenting a slightly negative trend and matching the assumption of Bid-Rent theory. Also in this graph the line is fairly smooth, so we can say that the price decline gently with the coefficients -0.043, -0.034 and -0.061, telling us the distance may just have a tiny influence on the price. In fact, the factors affecting the room price are various including nearby infrastructures, public transport, service, review, decoration and so on.</p><h3>Conclusion</h3><p>In this article, to testify the Bid Rent theory, a simple linear regression model has been applied for revealing the relationship between Listing’s prices and its distance to the subjectively central point, demonstrating that the lodging price of sharing economy may also follow the pattern of residential price, from the perspective of Bid Rent model.</p><p>However, the distance to a central point is not the only factor affecting the price, other factors can have huge influences on price-setting in Airbnb, the R squared value of the training model is extremely small and then only a tiny portion of price can be explained by distance, the input data.</p><p>Reference:</p><p>Alonso, W. (1964). Location and land use. Toward a general theory of land rent. <em>Location and land use. Toward a general theory of land rent.</em></p><blockquote>My previous article:</blockquote><p><a href="https://medium.com/@zidongyu26/exploring-the-spatial-distribution-of-airbnb-listings-in-hong-kong-816a3f56aa2c">Exploring the spatial distribution of Airbnb listings in Hong Kong</a></p><blockquote>For the next article:</blockquote><p>Honestly, I am little bit busy nowadays, so for the next post, maybe started in late January.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5e10e1b8fe30" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/exploring-the-spatial-distribution-of-airbnb-listings-in-hong-kong-5e10e1b8fe30">Exploring the spatial distribution of Airbnb listings in Hong Kong, Part 2</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring the spatial distribution of Airbnb listings in Hong Kong, Part 1]]></title>
            <link>https://medium.com/@zidongyu26/exploring-the-spatial-distribution-of-airbnb-listings-in-hong-kong-816a3f56aa2c?source=rss-3339ed5adf2------2</link>
            <guid isPermaLink="false">https://medium.com/p/816a3f56aa2c</guid>
            <category><![CDATA[mapping]]></category>
            <category><![CDATA[gis]]></category>
            <category><![CDATA[geospatial]]></category>
            <category><![CDATA[hong-kong]]></category>
            <category><![CDATA[sharing-economy]]></category>
            <dc:creator><![CDATA[MARTIN YU]]></dc:creator>
            <pubDate>Wed, 01 Jan 2020 12:31:32 GMT</pubDate>
            <atom:updated>2020-02-22T16:17:04.041Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Article 1: Basic exploratory data analysis</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*daIDocp3rk3hcJ5DlDr9rg.png" /><figcaption>The screenshot of Airbnb Hong Kong</figcaption></figure><h3>About this series</h3><p>In the past few years, the sharing economy had become highly popular, and they have provided alternative choices for people, affecting many traditional business sectors like tourism (Airbnb), transportation (Uber, Dockless Bike Sharing), retail (eBay). To date, Airbnb has approximately 150 million users and its business been operated in more than 81000 cities within 191 countries, providing more than 5 million spaces for tourists to choose (<a href="https://muchneeded.com/airbnb-statistics/">Airbnb Statistics, 2019</a>). In terms of Hong Kong, as the top destination city for visitors in Asia, Airbnb offers more than 10,000 rooms or departments for tourists across all 18 districts in Hong Kong, which are mainly concentrated on three downtown districts: Wan Chai, Western &amp; Central, and Yau Tsim Mong.</p><blockquote>Therefore, it maybe an interesting topic that learns the lodging logistics of sharing economy in the tourism industry.</blockquote><p>For this paper, the first section of this series, we will focus on the basic data exploratory skills and basic geo-visualization skills, which are necessary and the pre-processing part for all data exploratory analysis. For further content, some machine learning skills including regression, classification, and clustering will be shown, trying to present a comprehensive combination of Machine Learning and Geospatial Analysis.</p><blockquote>Now, let us start by using Jupyter Notebook:</blockquote><p>As for the libraries we need:</p><pre>import pandas as pd<br>import numpy as np<br>import matplotlib<br>import matplotlib.pyplot as plt<br>import seaborn as sns<br>%matplotlib inline</pre><p>For the data, we download the data of our target city from the <a href="http://insideairbnb.com/">http://insideairbnb.com/</a>, a website provide Airbnb data of major cities all over the world. And then, we import the data we downloaded using pd.read_csv:</p><pre>data = pd.read_csv(&quot;HK/listings.csv&quot;)</pre><p>After we import data, it is good to have a quick overview and summary of the whole data, indicating the columns, type as well as <strong><em>the number of non-null value:</em></strong></p><pre># output the basic info<br>data.info()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/557/1*wNFc0StouNY7hIr0K1EUxQ.png" /><figcaption>The output of data.info()</figcaption></figure><p>Also, for the null values, we can use .isnull().sum for looking the missing values:</p><pre>data.isnull().sum()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/367/1*plPuv45QbRvh1gOjqWt98g.png" /></figure><p>Reviewing the columns considerably, some columns with null values are unneeded and useless in this analysis process, so we have to drop some unnecessary columns:</p><pre>data = data.drop([&#39;neighbourhood_group&#39;,&#39;last_review&#39;, &#39;reviews_per_month&#39;, &#39;host_name&#39;, &#39;name&#39;], axis=1)<br>data.isnull().sum()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/335/1*NNx5lxlnTg5X3f7KGX9z-g.png" /></figure><p>Now we are gonna check the numeric variables by using pandas.DataFrame.describe(), from the overall perspective, for generating descriptive statistics summary that includes the shape of data’s distribution, helping us find out some “weird” and unrealistic data, and then remove them:</p><pre>data.describe()</pre><pre># clean the data with price = 0<br># clean the data with minimun_night greater than 100<br>data = data[data.price != 0]<br>data = data[data.minimum_nights &lt;= 100] </pre><blockquote>Geo-Visualization</blockquote><p>For the libraries will be used in this part, I will import folium, geopandas and others:</p><pre>import folium as fl<br>from folium import plugins<br>from folium.plugins import HeatMap<br>import geopandas as gpd<br>import descartes<br>from descartes.patch import PolygonPatch</pre><p>Let’s begin with a heatmap first, for looking the whole distribution of Airbnb listings across Hong Kong:</p><pre>hk_map = fl.Map(location=[22.38,114.13],zoom_start = 10)<br>data_loc = data[[&#39;latitude&#39;,&#39;longitude&#39;]].values<br>HeatMap(data = data_loc.tolist(), radius=8, max_zoom=10).add_to(hk_map)<br>hk_map</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/711/1*hjXSzpjXlAlOYlrUPp6_UA.png" /><figcaption>The heatmap of Airbnb listings distribution.</figcaption></figure><p>Referring to the output heatmap shown above, it is clear to note that the Airbnb listings are mainly concentrated on Hong Kong island as well as Kowloon (the deep red colour), and besides, areas of Tin Shui Wai, Sha Tin and Cheung Sha Wan also provide a great number of lodges for tourists.</p><p>However, the heatmap above does not indicate the comparison between each district. Therefore, next, the GeoJson file of Hong Kong 18 districts boundary will be imported through gpd.read_file.</p><pre># Importing the Hong Kong 18 districts boundary GeoJSON file<br>map_data = gpd.read_file(&quot;HK/Hong_Kong_18_Districts/Hong_Kong_18_Districts.shp&quot;)</pre><pre># Uppercase all districts&#39; name to match the HK&#39;s shapefile<br>data[&quot;neighbourhood&quot;] = data[&#39;neighbourhood&#39;].str.upper()</pre><pre># Dropping the empty column<br>map_data.drop(&quot;TCNAME&quot;, axis = 1, inplace = True)</pre><p>After importing the boundary file, we have to prepare the listings data that will be used to join with boundary file later. For example, I am gonna explore the numbers of listings, median prices, mean prices of each district.</p><pre># Creating a dataframe of listing counts and median price by borough<br>neighbour_price = pd.DataFrame(data.groupby(&#39;neighbourhood&#39;).size())</pre><pre>neighbour_price.rename(columns={0: &#39;number_of_listings&#39;}, inplace=True)<br>neighbour_price[&#39;median_price&#39;] = data.groupby(&#39;neighbourhood&#39;).price.median().values<br>neighbour_price[&#39;mean_price&#39;] = data.groupby(&#39;neighbourhood&#39;).price.mean().values</pre><p>Now, time for joining two dataframe and then checking some rows of output data frame:</p><pre># Joining the data frames<br>neighbour_map_data = map_data.set_index(&#39;ENAME&#39;).join(neighbour_price)</pre><pre>neighbour_map_data.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cW_ic1sGZ35YK0OB_cZnKA.png" /></figure><p>Looking good! And time for plotting map for the median price of listings and the number of listings in each district:</p><pre># Plotting the median price in each district<br>fig2, ax2 = plt.subplots(1, figsize=(12,5))<br>neighbour_map_data.plot(column=&#39;median_price&#39;, cmap=&#39;Reds&#39;, ax=ax2)<br>ax2.axis(&#39;off&#39;)<br>sm = plt.cm.ScalarMappable(cmap=&#39;Reds&#39;, norm=plt.Normalize(vmin=neighbour_map_data.median_price.min(), vmax=neighbour_map_data.median_price.max()))<br>sm._A = [] # Creates an empty array for the data range<br>cbar = fig2.colorbar(sm)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/594/1*eU9BmKflSjKI2mNwjfiX1g.png" /><figcaption>The Median Price of Airbnb Listings in Hong Kong, Grouped by Districts</figcaption></figure><pre># Plotting the number of listings in each district<br>fig1, ax1 = plt.subplots(1, 1, figsize=(8,5))<br>neighbour_map_data.plot(column=&#39;number_of_listings&#39;, cmap=&#39;Reds&#39;, ax=ax1)<br>ax1.axis(&#39;off&#39;)<br>ax1.set_title(&#39;Number of Airbnb Listings&#39;)<br>sm = plt.cm.ScalarMappable(cmap=&#39;Reds&#39;, norm=plt.Normalize(vmin=0, vmax=5000))<br>sm._A = [] # Creates an empty array for the data range<br>cbar = fig1.colorbar(sm)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*W-Y2mYDQWJ7W0C_BthYnwg.png" /><figcaption>The Number of Airbnb Listings in Hong Kong, Grouped by Districts</figcaption></figure><p>For the choropleth map generated above, we can easily find that 3 districts of Hong Kong are distinguished from remaining districts, for offering more than 2000 rooms, which are Yau Tsim Mong, Central &amp; Western and Wan Chai.</p><p>Therefore, we pick up that 3 top districts that offer most listings, specifying their locations, precise shapes.</p><pre># Obtain the index of the target districts<br>top3 = data.groupby(&quot;neighbourhood&quot;).price.count().sort_values(ascending=False).head(3).index<br>top3 = top3.tolist()</pre><pre># Extract the corresponding rows<br>top3 = neighbour_map_data.loc[top3]</pre><pre># And then, plot out the their shape <br>base = neighbour_map_data.plot(figsize=(14, 12), color = &#39;white&#39;, edgecolor=&#39;black&#39;, linewidth = 0.5)<br>top3.plot(ax = base, color=&#39;r&#39;, edgecolor=&#39;black&#39;, linewidth = 0.5 )</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/751/1*x8riN4l2-UE5p_TLlYxk7Q.png" /></figure><p>Zoom in and label that three districts:</p><pre>top3[&quot;center&quot;] = top3[&quot;geometry&quot;].centroid<br>top3_points = top3.copy()<br>top3_points.set_geometry(&quot;center&quot;, inplace = True)</pre><pre>map_top3 = top3.plot(figsize=(12,10), color = &#39;whitesmoke&#39;, edgecolor=&#39;black&#39;, linewidth = 1, legend = True)<br>texts = []</pre><pre>for x, y, label in zip(top3_points.geometry.x, top3_points.geometry.y, top3_points.index):<br>    texts.append(plt.text(x, y, label, fontsize = 8))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/751/1*V6frVHXYXcFrlEfHLyUO-g.png" /></figure><p>Finally, for the further exploration, it is a good operation and essential for us to export the data we have extracted from raw data.</p><pre>top3.to_file(&#39;top3.shp&#39;)</pre><h3>Conclusion:</h3><p>we have seen a number of data-exploration techniques, which are easy and straightforward to understand and use. Nevertheless, there are lots of skills that have not shown in this article but still significant and useful, and Medium has provided numerous tutorials about data analysis.</p><blockquote><strong>For the next article:</strong></blockquote><p>Bid Rent Theory, one of the iconic study cases in urban land-use discipline, was designed by William Alonso and published in 1964, basically explaining the relationship between the price and the land demand in case of a change in distance from an assumed central business district (CBD) and, the principle of the bid rent theory is that land rent would be lower where the areas are getting farther from the central area of the city, which is based on the extension of the von Thunen model on urban land uses(Alonso, 1964).</p><p>To examine the Bid Rent Theory, from the perspective of sharing economy, some primary and simple Machine Learning skills will be introduced in the next article.</p><p><em>Reference:</em></p><p>Alonso, W. (1964). Location and land use. Toward a general theory of land rent. <em>Location and land use. </em>Toward a general theory of land rent.</p><p>Airbnb Statistics (2019): Users, Booking, Financials &amp; Fun Facts. (2019). Retrieved 8 October 2019, from https://muchneeded.com/airbnb-statistics/</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=816a3f56aa2c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>