<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Anthony Galtier on Medium]]></title>
        <description><![CDATA[Stories by Anthony Galtier on Medium]]></description>
        <link>https://medium.com/@anthony.galtier?source=rss-4d6214013c99------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ATcoV9X2KuKjvnMZnYbtOA.jpeg</url>
            <title>Stories by Anthony Galtier on Medium</title>
            <link>https://medium.com/@anthony.galtier?source=rss-4d6214013c99------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 08:20:39 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@anthony.galtier/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Do pictures in real estate listings actually help us estimate the list price ?]]></title>
            <link>https://medium.com/ilb-labs-publications/do-pictures-in-real-estate-listings-actually-help-us-estimate-the-list-price-9e63b1be9926?source=rss-4d6214013c99------2</link>
            <guid isPermaLink="false">https://medium.com/p/9e63b1be9926</guid>
            <category><![CDATA[real-estate]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Anthony Galtier]]></dc:creator>
            <pubDate>Wed, 14 Sep 2022 07:54:07 GMT</pubDate>
            <atom:updated>2022-09-14T07:54:07.310Z</atom:updated>
            <content:encoded><![CDATA[<h3>Do pictures in real estate listings actually help us estimate the list price ?</h3><p>Estimating housing real estate price is quite a common topic, with an important literature on estimating prices based on a set of numeric and categorical features describing the characteristics of a property such as the location, surface, land size, number of bedrooms, age of the building… These hedonic approaches are usually sufficient to estimate the price range but lack precision.</p><p>Few have worked however to see if other types of data carry complementary information, enabling a more precise price estimation. In our <a href="https://medium.com/@anthony.galtier/list-price-prediction-with-scraped-real-estate-data-df5e49f14547">previous posts</a>, <a href="https://medium.com/@anthony.galtier/fine-tuning-bert-for-a-regression-task-is-a-description-enough-to-predict-a-propertys-list-price-cf97cd7cb98a">we showed that the textual descriptions of the property allow us to reach similar list price estimation performances as with tabular data</a>. In this article, we will explore the photos of a property and see whether they contribute to improving the performance of the usual feature-based models.</p><p>To do so, we test two approaches, a first extracting an explicit set of features with traditional computer vision techniques, and a second relying on a Convolutional Neural Network to embed the information carried by the images and then compare each’s performance to a simple benchmark model based exclusively on the tabular set of features.</p><p><em>For more details, you can check out </em><a href="https://gitlab.com/institut-louis-bachelier-datalab/real-estate-list-price-prediction-using-images"><em>the code developed for this article here</em></a><em>.</em></p><h3>The data set</h3><p>The data set used in this article is the same as the one used in our previous posts. The data set consists of <strong>46K listings</strong> for which we collected<strong> 27 numeric and categorical features</strong> (location, surface, number of rooms, exposition…), a textual description and <strong>1 to 6 photos</strong>. This data was scraped for French real-estate websites.</p><p>The photos collected are the photos displayed on the listing’s web page. In most cases, they are photos showing different views of the property, mostly indoor views for apartments and a mix of wide outdoor views and indoor views of specific rooms for houses and larger property types. These photos usually highlight the most notable characteristics of the property (swimming pools, verandas, large windows…). That said, angles, lighting and subject can differ significantly from one listing to another, some photos even containing watermarks, text, or logos.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*_n2pFq8ZkjKRCeUA5ykKWg.png" /><figcaption>Screenshot of a typical French real estate listing</figcaption></figure><p>Extracting valuable information from these images is not a straightforward task. To simplify things a bit, we chose to restrict our study to apartments and houses listed between 20K and 2M euros with less than 10 bedrooms. This leaves us with 69% (32K) of the original 46K listings.</p><p>This data set was then randomly split into a training set with 80% of listings (26K) and a test set with the remaining 20% (6K) of the listings kept aside to evaluate the performance of the different models.</p><h3>Benchmark</h3><p>Before diving into the photos, we create a model based exclusively on the numeric and categorical features describing the property. The price prediction performance of this model will be our benchmark. We will use it as a point of reference to evaluate how the performance is affected when information extracted from the listing’s photos is fed as well into the model.</p><h4>Model</h4><p>The model consists of a pipeline of pre-processing transformations followed by a final regression estimator. The pre-processing steps deal with outliers and reduce the columns specifying the number of rooms, bathrooms, and bedrooms into a single feature by principal component analysis (PCA). For the final estimator, we use a CatBoost regressor model, which is a supervised boosted tree-based model specifically adapted for data sets with categorical variables. The model was trained on a selection of 17 numeric and categorical variables of the train set with a 5-fold grid search cross validation to optimize the parameters of the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/778/1*cny6rBXBQOblT5x_Tnbg1Q.png" /><figcaption>Variable distributions as observed in the train set</figcaption></figure><h4>Results</h4><p>The performance of the benchmark model was then evaluated on the test set. Predictions are off on average by 73.2K euros or 25.8% in relative terms. The median error of 41.5K euros or 16.9% in relative terms is better than the average absolute errors suggesting the model sometimes makes unusually large errors for a few properties. The R squared score reaches a value of 0.79 indicating that 79% of the variability in the price variable is explained by our models, which is quite satisfactory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/343/1*AXYanAepchtDEQURpcSLpQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*RB4elWRNp_1oAmHBKN0pLQ.png" /><figcaption>Benchmark model performance scores on the test set (left) and top 5 variables by feature importance (right)</figcaption></figure><p>Like most decision tree-based ensemble learning models (Random Forests, XGBoost…), CatBoost provides a measure of “feature importance” that measures how each feature contributes on average to the model’s prediction. When interpreting these values, one must keep in mind that the “feature importance” is shared proportionally between correlated features.</p><p>The most discriminating features for our benchmark model are the location and size of the property which makes sense. In the graph above, we should not interpret however that the size is less important than the location. Indeed, the feature importance of the size, number of rooms and land size variables are slightly understated due to their correlation.</p><h3>With simple image feature extraction techniques</h3><p>For each listing, we have 1 to 6 photos with most (70%) having 6 photos. Since over 99% of the photos are JPEG color images, they can easily be represented in <a href="https://en.wikipedia.org/wiki/RGB_color_model">RGB format</a> where each pixel’s color is represented by a triplet of values corresponding to the quantity of red, green, and blue. The photos can also be represented in the <a href="https://en.wikipedia.org/wiki/HSL_and_HSV">HSV color space</a>, an alternative format to the RGB model where the triplet of values for each pixel’s color describes its tint (“hue”), saturation and brightness (“lightness” or “value”).</p><p>The idea is thus to extract a set of simple features from these representations, to append them to the previously used numeric and categorial features and see if they enable the model to better predict the list price.</p><h4>Extracted features</h4><p>For each photo, we chose to extract shape related features such as the orientation (portrait, square or landscape) and the aspect ratio (4/3, 16:9…) and color related features (main color, RGB skewness, saturation, brilliance…). These image level features were then aggregated to the listing level by mean, frequency or other statistical measures depending on the nature of the feature.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/421/1*CFJ0x0dW68t1qn_4uAujCA.png" /><figcaption>Extracted image feature distributions aggregated at listing level as observed on the train set</figcaption></figure><p>Although these features do not explicitly describe the subject of the photos, they do carry some information. We noticed for example that photos of apartments tend to be less brilliant and saturated than photos of houses with less yellow and more red. There are also more portrait-oriented photos of houses than of apartments. However, is this enough to improve the performance of our model?</p><h4>Model</h4><p>We used the same Catboost regressor and the same training and test observations as for the benchmark model but adding 18 features extracted from the images to the original data set. The parameters of the model were optimized with a 5-fold randomized grid search cross-validation procedure, just like the benchmark model.</p><h4>Results</h4><p>This new model’s predictions are now off by an absolute average of 75.8K euros or 26.3% in relative terms. The median absolute error standing at 43.4K or 17.6% in relative terms here again remains better than the mean. The R squared as well stays unchanged at 0.78. Thus, the features extracted from the images did not improve the performance of the model, they may even have degraded it. That said, the difference in performance is small, it could just be due to the inherent variability of the model.</p><p>Looking at the feature importance graph, we see that the most influential features remain the same ones as for the benchmark. None of the features extracted from the photos appear to be of any importance in the model’s decision-making process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/362/1*qZWxPc8fWw_eGMzKkPc11w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/563/1*I14VobrYD7MXMqD-Q2Ms9w.png" /><figcaption>Image model performance scores on the test set (left) and top 5 variables by feature importance (right)</figcaption></figure><h3>With deep learning</h3><h4>Embedding images with a CNN</h4><p>Our first deep learning approach is just like the previous one, but we use a fine-tuned CNN to extract features from the photos instead. The 138K photos associated to the 26K training listings are used to optimize the parameters of a CNN composed of a ResNet34 followed by a few regression layers. The weights of the CNN are fine-tuned on the task of predicting from each image the price of the corresponding listing.</p><p>After the fine-tuning step, we realized that predictions can vary significantly from one image to another. To mitigate this, we decided to retain only the average of the three median predictions from the photos of the same listing. It is this average prediction that we append as a new variable to the benchmark data set. That data is then used to train a Catboost model following the same procedure as previously.</p><h4>Results</h4><p>Already during the fine-tuning step, we noticed that the CNN struggled to make good list price predictions from induvial photos. Despite our attempt to mitigate the variability of the CNNs predictions by removing outliers, aggregating only the median predictions, our CNN outputs failed to improve the overall price prediction performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/379/1*N3PvUhk7MDpS30drHCWbFA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/1*pQ_87AJInopBACHI00yXMw.png" /><figcaption>CNN based image model performance scores on the test set (left) and top 5 variables by feature importance (right)</figcaption></figure><p>The MAE at 73K euros and MAPE at 25.8% are not significantly different from the benchmark. The median absolute error at 41.8K and median percentage error at 16.8% are also no better than the benchmark. Furthermore, the feature importance of the average price prediction extracted from the photos is low suggesting that it does not contribute much to the model’s predictions.</p><h4>End to end neural network</h4><p>We tried a second deep learning approach that consists in a two-branch neural network where one branch performs an embedding of the photos, the other performs an embedding of the numeric and categorical variables, and a common part performs the regression from the concatenation of two embeddings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/280/1*1AmCPjF6y5JYPy5jsw670Q.png" /><figcaption>Schematic representation of the end to end deep learning model</figcaption></figure><p>In the hope of making the entire network converge, the architecture and parameters of the image embedding, and tabular branch are each optimized separately first. For the image branch, we reuse the previously optimized parameters of the ResNet34 CNN. The common regression part of the network is optimized last, with the parameters of the two pre-trained branches frozen.</p><h4>Results</h4><p>We noticed immediately during the training of the embedding branches that the network could learn to predict prices well from the tabular data but struggled a lot more with the photos, just like in the previous section. When we put everything together in the two-branch network, the embeddings learned from the images seemed only to add noise to the training procedure. Indeed, the two-branch model learns slower and does not outperform the simple feed forward network of the tabular branch.</p><h3>Conclusions, limitations, and next steps</h3><p>The main conclusion we take out from our different attempts is that extracting insightful information from the photos published with a real estate listing is a challenging task. None of our proposed implementations have been capable of leveraging on the images to improve the benchmark model that relies only on numerical and categorical features.</p><p>Furthermore, unlike our<a href="https://medium.com/@anthony.galtier/fine-tuning-bert-for-a-regression-task-is-a-description-enough-to-predict-a-propertys-list-price-cf97cd7cb98a"> previous work with textual data</a>, we witnessed a lack of performance even when optimizing a CNN to directly predict the list price from the photos. This shows that it is not just difficult to improve the performance of the benchmark, it is inherently difficult to extract useful information from the photos.</p><p>The task of predicting a list price from photos of a property is thus a difficult one, even for a human when you think about it. Images, unlike textual descriptions or tabular data sets, do not explicitly highlight the key facts and value drivers of a listing, most commonly, the size and location of the property. Images also seem to carry more noise and superfluous information than other data types. The diversity of subjects and angles probably make it hard for algorithms to generalize and predict the price accurately.</p><p>That said, there is surely some value to extract from certain photos that highlight specific value-driving characteristics of the property. Alone, such information may not be enough, but we can expect it to be complementary to the usual housing variables.</p><p>An improvement we should consider next time is to be more specific in the selection of photos used to train our models. Semi-supervised learning could come in handy here to help us automatically select images that a human would deem useful to predict the price. Another idea would be to use domain expertise to identify certain value driving characteristics of properties that do not exist in the tabular data set (presence of a pool, a lawn, brightness of the rooms…) and then build classifiers to extract these features from the photos.</p><h3><em>Acknowledgements</em></h3><p><em>I would like to give credit and a special thanks to </em><a href="https://www.linkedin.com/in/chahnez-chouba-49b8881a5/"><em>Chahnez Chouba</em></a><em> for her more than significant contribution to the content of this article at the </em><a href="https://www.linkedin.com/company/institut-louis-bachelier/mycompany/verification/"><em>Institut Louis Bachelier</em></a><em> Datalab.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9e63b1be9926" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ilb-labs-publications/do-pictures-in-real-estate-listings-actually-help-us-estimate-the-list-price-9e63b1be9926">Do pictures in real estate listings actually help us estimate the list price ?</a> was originally published in <a href="https://medium.com/ilb-labs-publications">ILB Labs publications</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fine-tuning BERT for a regression task: is a description enough to predict a property’s list price?]]></title>
            <link>https://medium.com/ilb-labs-publications/fine-tuning-bert-for-a-regression-task-is-a-description-enough-to-predict-a-propertys-list-price-cf97cd7cb98a?source=rss-4d6214013c99------2</link>
            <guid isPermaLink="false">https://medium.com/p/cf97cd7cb98a</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[bert]]></category>
            <dc:creator><![CDATA[Anthony Galtier]]></dc:creator>
            <pubDate>Mon, 13 Sep 2021 12:10:48 GMT</pubDate>
            <atom:updated>2022-12-12T16:17:04.827Z</atom:updated>
            <content:encoded><![CDATA[<p>The purpose of this article is to provide a practical example of fine-tuning BERT for a regression task. In our case, we will be predicting prices for real-estate listings in France.</p><p><a href="https://medium.com/@anthony.galtier/list-price-prediction-with-scraped-real-estate-data-df5e49f14547">In a previous post</a>, we constructed a supervised regression model based on an LGBM to predict the list price from a tabular set of numerical and categorical variables. Today, we will compare BERT’s performance at predicting the same list prices based solely on the textual description of the properties.</p><h4>The data set</h4><p>The data set we are using today is the same as in the <a href="https://medium.com/@anthony.galtier/list-price-prediction-with-scraped-real-estate-data-df5e49f14547">previous article</a>. It includes data from <strong>788K listings </strong>scraped from French real-estate websites. Alongside the numerical and categorical features used in the benchmark LGBM model, we collected a <strong>textual description of the property for sale</strong>.</p><p>These descriptions tend to repeat at least part of the information contained in the other features but in the form of natural language. The information is often presented in a subjective manner, putting forward the most attractive features of the property. These descriptions can also provide details that are specific to the property, that do not fit into any set of common features, such as “renovated by an architect”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*60silFfTAocSPjNpPdXqSA.png" /></figure><p>All in all, we are led to believe that the textual feature should somehow, on it’s own, be enough to achieve good price prediction performances. But how do we process it?</p><p>Luckily for us, Natural Language Processing is a very active field of research. Transformer architecture and attention mechanisms were first presented in the <a href="https://papers.nips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf">2017 paper called “Attention is all you need”</a> and <a href="https://arxiv.org/pdf/1810.04805v2.pdf">in 2018, Google presented a breakthrough transformer-based language model: BERT</a>.</p><p>This post will not detail the inner workings of Transformers, attention mechanisms and BERT but if you want to learn about it, here are a few references that will help you understand the key concepts:</p><ul><li>If you like videos, check out CodeEmporium’s video about <a href="https://www.youtube.com/watch?v=TQQlZhbC5ps">Transformers </a>and <a href="https://www.youtube.com/watch?v=xI0HHN5XKDo">BERT</a>,</li><li>If you prefer reading, have a look at this <a href="https://medium.com/inside-machine-learning/what-is-a-transformer-d07dd1fbec04">post</a>.</li><li>You can also watch ChrisMcCormickAI’s <a href="https://www.youtube.com/watch?v=FKlPCK1uFrc">series on BERT</a> and his tutorial on <a href="https://mccormickml.com/2019/07/22/BERT-fine-tuning/">fine tuning BERT for document classification</a> that inspired some of the content of this post.</li></ul><p>Often, blogposts, articles and tutorials explain how to use BERT for supervised classification problems such as document classification. In this article we will see how to use BERT for a regression task.</p><h4>BERT and CamemBERT</h4><p>As you have probably noticed by now, the textual data we collected is in French… Luckily, researchers at Facebook and Inria teamed up to produce <a href="https://aclanthology.org/2020.acl-main.645.pdf">CamemBERT</a>. No, not cheese…, but a “<em>state-of-the-art language model for French based on the </em><a href="https://ai.facebook.com/blog/roberta-an-optimized-method-for-pretraining-self-supervised-nlp-systems/"><em>RoBERTa architecture</em></a><em> pretrained on the French sub-corpus of </em><a href="https://traces1.inria.fr/oscar/"><em>OSCAR</em></a><em>” </em>as stated on their <a href="https://aclanthology.org/2020.acl-main.645.pdf">website</a>.</p><h4>The transformers library</h4><p>For our use-case, we will use the <a href="https://huggingface.co/transformers/">transformers library</a> implementation of CamemBERT. The transformers library, developed by <a href="https://huggingface.co/">HuggingFace</a>, seems to have become the go-to library for pre-trained transformer architecture models like BERT in PyTorch.</p><p>HuggingFace manages to keep the same or at least a very similar interface from one model to another in the transformers library. This article’s code can thus easily be adapted to fit your needs if your are working on text in another language or just want to try another model out.</p><h4>Pre-processing data for BERT</h4><p>We will be using the same training and validation data as in our previous post. To know more about the data and how it is pre-processed, you can check out our <a href="https://medium.com/@anthony.galtier/list-price-prediction-with-scraped-real-estate-data-df5e49f14547">previous post</a>. Let’s import and pre-process our data sets again and this time focus on the “description” variable.</p><pre>import pandas as pd<br>from preprocessing import preprocessing_pipeline</pre><pre>train_path = &quot;train_data.csv&quot;<br>val_path = &quot;test_data.csv&quot;</pre><pre>train_data = pd.read_csv(train_path, sep=&#39;,&#39;, index_col=0)<br>val_data = pd.read_csv(val_path, sep=&#39;,&#39;, index_col=0)</pre><pre>train_data = preprocessing_pipeline.fit_transform(train_data)<br>val_data= preprocessing_pipeline.transform(val_data)</pre><pre>df = train_data[[&#39;id_annonce&#39;, &#39;description&#39;, &#39;prix&#39;]]</pre><p>The descriptions average a length of 154 words and 928 characters. Ignoring <a href="https://github.com/explosion/spaCy/blob/604272353543b618672c4b857e95dd72d822955b/spacy/lang/fr/stop_words.py#L47">stop words</a>, the most common words are clearly in the lexical field of real estate: house, bedroom, kitchen, land…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*ClnoRD1ywqHVwyG1aHVYFA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-2XklAETx5tbw2g-Pmb3Vg.png" /><figcaption>Description length distribution based on the whole training set (left) and most common word lemmas in a random sample of 20K descriptions excluding stop words (right)</figcaption></figure><p>Common embedding approaches in NLP such as Word2Vec or FastText usually require lemmatization and stop words to be removed, but this is not the case for BERT. The general structure of the text should actually not be modified since BERT relies on it to learn and interpret context.</p><p>Our pre-processing will thus be limited to:</p><ul><li>Converting text to lower-case</li><li>Standardising representations of a same entity such as “€”, “euro” and “euros” or “m2” and “m²”,</li></ul><pre>import re</pre><pre>def treat_euro(text):<br>    text = re.sub(r&#39;(euro[^s])|(euros)|(€)&#39;, &#39; euros&#39;, text)<br>    return text</pre><pre>def treat_m2(text):<br>    text = re.sub(r&#39;(m2)|(m²)&#39;, &#39; m²&#39;, text)<br>    return text</pre><ul><li>Cleaning out certain patterns that are unlikely to be meaningful such as URLs, phone numbers, emails and bank account references.</li></ul><pre>def filter_ibans(text):<br>    pattern = r&#39;fr\d{2}[ ]\d{4}[ ]\d{4}[ ]\d{4}[ ]\d{4}[ ]\d{2}|fr\d{20}|fr[ ]\d{2}[ ]\d{3}[ ]\d{3}[ ]\d{3}[ ]\d{5}&#39;<br>    text = re.sub(pattern, &#39;&#39;, text)<br>    return text</pre><pre>def remove_space_between_numbers(text):<br>    text = re.sub(r&#39;(\d)\s+(\d)&#39;, r&#39;\1\2&#39;, text)<br>    return text</pre><pre>def filter_emails(text):<br>    pattern = r&#39;(?:(?!.*?[.]{2})[a-zA-Z0-9](?:[a-zA-Z0-9.+!%-]{1,64}|)|\&quot;[a-zA-Z0-9.+!% -]{1,64}\&quot;)@[a-zA-Z0-9][a-zA-Z0-9.-]+(.[a-z]{2,}|.[0-9]{1,})&#39;<br>    text = re.sub(pattern, &#39;&#39;, text)<br>    return text</pre><pre>def filter_ref(text):<br>    pattern = r&#39;(\(*)(ref|réf)(\.|[ ])\d+(\)*)&#39;<br>    text = re.sub(pattern, &#39;&#39;, text)<br>    return text</pre><pre>def filter_websites(text):<br>    pattern = r&#39;(http\:\/\/|https\:\/\/)?([a-z0-9][a-z0-9\-]*\.)+[a-z][a-z\-]*&#39;<br>    text = re.sub(pattern, &#39;&#39;, text)<br>    return text</pre><pre>def filter_phone_numbers(text):<br>    pattern = r&#39;(?:(?:\+|00)33[\s.-]{0,3}(?:\(0\)[\s.-]{0,3})?|0)[1-9](?:(?:[\s.-]?\d{2}){4}|\d{2}(?:[\s.-]?\d{3}){2})|(\d{2}[ ]\d{2}[ ]\d{3}[ ]\d{3})&#39;<br>    text = re.sub(pattern, &#39;&#39;, text)<br>    return text</pre><p>Surely more text cleaning can be done but let’s stick with this for now.</p><pre>def clean_text(text):<br>    text = text.lower()<br>    text = text.replace(u&#39;\xa0&#39;, u&#39; &#39;)<br>    text = treat_m2(text)<br>    text = treat_euro(text)<br>    text = filter_phone_numbers(text)<br>    text = filter_emails(text)<br>    text = filter_ibans(text)<br>    text = filter_ref(text)<br>    text = filter_websites(text)<br>    text = remove_space_between_numbers(text)<br>    return text</pre><pre>df[&#39;cleaned_description&#39;] = df.description.apply(clean_text)</pre><h4>Tokenization</h4><p>BERT takes as input sequences of equal length. Input sequences must thus be padded or truncated at a given length with special characters explicitly indicating the actual start and end of the sequence. The special start character is referred to as the “[CLS]” token. We’ll talk about it again later.</p><p>These input sequences are then split into two :</p><ul><li>A sequence of “input ids”, mapping the words to tokens from the vocabulary the model was pre-trained with,</li><li>A binary sequence of “attention masks” that indicate whether the “input id” at a given index is a word or padding.</li></ul><p>The transformer library provides a tokenizer module that does it all for us:</p><pre>from transformers import CamembertTokenizer</pre><pre>tokenizer = CamembertTokenizer.from_pretrained(&#39;camembert-base&#39;)</pre><pre>encoded_corpus = tokenizer(text=df.cleaned_description.tolist(),<br>                            add_special_tokens=True,<br>                            padding=&#39;max_length&#39;,<br>                            truncation=&#39;longest_first&#39;,<br>                            max_length=300,<br>                            return_attention_mask=True)</pre><pre>input_ids = encoded_corpus[&#39;input_ids&#39;]<br>attention_mask = encoded_corpus[&#39;attention_mask&#39;]</pre><p>The maximal input sequence length supported by BERT is 512 tokens but using shorter input sequences tends to improve and speed up the training process. We’ll take a maximum input sequence of 300. To avoid any description being truncated and because it concerns only a small proportion of the observations, we chose to put aside descriptions that required truncation given a maximum length of 300.</p><pre>import numpy as np</pre><pre>def filter_long_descriptions(tokenizer, descriptions, max_len):<br>    indices = []<br>    lengths = tokenizer(descriptions, padding=False, <br>                     truncation=False, return_length=True)[&#39;length&#39;]<br>    for i in range(len(descriptions)):<br>        if lengths[i] &lt;= max_len-2:<br>            indices.append(i)<br>    return indices</pre><pre>short_descriptions = filter_long_descriptions(tokenizer, <br>                               df.cleaned_description.tolist(), 300)</pre><pre>input_ids = np.array(input_ids)[short_descriptions]<br>attention_mask = np.array(attention_mask)[short_descriptions]<br>labels = df.prix.to_numpy()[short_descriptions]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EH7iRuYwvyKkBGIg_j9PVg.png" /><figcaption>Tokenization of an input sequence for BERT</figcaption></figure><h4>Input formatting</h4><p>Our data is already split into a training and a validation set. The validation set will be used exclusively to evaluate the performance of the final trained model. The training set is the same 300K observation set we used in the <a href="https://medium.com/@anthony.galtier/list-price-prediction-with-scraped-real-estate-data-df5e49f14547">previous post</a>. To monitor the performance of the model during training, another 10% of the training data is put aside into a separate test set.</p><p>It is important to use a separate test and validation set since the performance metrics observed during training could influence certain decisions regarding the training process such as the number of epochs and thus bias the final evaluation of the model.</p><pre>from sklearn.model_selection import train_test_split</pre><pre>test_size = 0.1<br>seed = 42</pre><pre>train_inputs, test_inputs, train_labels, test_labels = \<br>            train_test_split(input_ids, labels, test_size=test_size, <br>                             random_state=seed)</pre><pre>train_masks, test_masks, _, _ = train_test_split(attention_mask, <br>                                        labels, test_size=test_size, <br>                                        random_state=seed)</pre><p>In a deep learning regression task, it <a href="https://machinelearningmastery.com/how-to-improve-neural-network-stability-and-modeling-performance-with-data-scaling/">can arguably be helpful for the stability of the training process to have scaled the target variable</a>:</p><pre>from sklearn.preprocessing import StandardScaler</pre><pre>price_scaler = StandardScaler()<br>price_scaler.fit(train_labels.reshape(-1, 1))</pre><pre>train_labels = price_scaler.transform(train_labels.reshape(-1, 1))<br>test_labels = price_scaler.transform(test_labels.reshape(-1, 1))</pre><p>Let’s now convert the training and test sets to PyTorch-friendly input formats.</p><p>For each set, the input ids, masks and labels are each converted to tensors and then put together into a TensorDataset. The TensorDataset is then packed into a DataLoader object which is an iterator that will present the inputs in batches of 32 observations to our model at training.</p><pre>import torch</pre><pre>from torch.utils.data import TensorDataset, DataLoader</pre><pre>batch_size = 32</pre><pre>def create_dataloaders(inputs, masks, labels, batch_size):<br>    input_tensor = torch.tensor(inputs)<br>    mask_tensor = torch.tensor(masks)<br>    labels_tensor = torch.tensor(labels)<br>    dataset = TensorDataset(input_tensor, mask_tensor, <br>                            labels_tensor)<br>    dataloader = DataLoader(dataset, batch_size=batch_size, <br>                            shuffle=True)<br>    return dataloader</pre><pre>train_dataloader = create_dataloaders(train_inputs, train_masks, <br>                                      train_labels, batch_size)</pre><pre>test_dataloader = create_dataloaders(test_inputs, test_masks, <br>                                     test_labels, batch_size)</pre><h4>Model architecture</h4><p>Now for the actual architecture of our model. BERT is composed of an embedding layer, and 12 transformers stacked one after the other.</p><p>For each input sequence, BERT’s output is a same-size sequence of vectors. These vectors are the final hidden states representing each input token. Each of these vectors are made up of 768 floats.</p><p>In <a href="https://arxiv.org/pdf/1810.04805.pdf">BERT’s paper</a>, it is indicated that only the final hidden state of the first token in the output sequence should be used for classification tasks. In other words, only the vector representing the “[CLS]” token mentioned earlier, should be used.</p><p>For our regression task, we will do the same thing but rather than add a dense pooling layer for classification behind it, we will add a dense linear layer with dropout that will serve as our final regression layer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/563/1*3YYXvnnJygt7ImJNZPpC0w.png" /><figcaption>Model architecture</figcaption></figure><h4>Implementing the model in PyTorch</h4><p>The PyTorch code to implement this model is actually quite straightforward. Our CamembertRegressor is a PyTorch nn.Module with two new attributes:</p><ul><li>an instance of the pretrained CamembertModel from the transformers library,</li><li>a single layer regression network taking as input a 768 long input and giving a single output value</li></ul><p>The forward method passes the tokenized input trough the CamembertModel and collects the 768 long vector corresponding to the “Class Label” output token. It then passes that vector through the regression layer that outputs the predicted value.</p><pre>import torch.nn as nn<br>from transformers import CamembertModel</pre><pre>class CamembertRegressor(nn.Module):<br>    <br>    def __init__(self, drop_rate=0.2, freeze_camembert=False):<br>        <br>        super(CamembertRegressor, self).__init__()<br>        D_in, D_out = 768, 1<br>        <br>        self.camembert = \<br>                   CamembertModel.from_pretrained(&#39;camembert-base&#39;)<br>        self.regressor = nn.Sequential(<br>            nn.Dropout(drop_rate),<br>            nn.Linear(D_in, D_out))</pre><pre>    def forward(self, input_ids, attention_masks):<br>        <br>        outputs = self.camembert(input_ids, attention_masks)<br>        class_label_output = outputs[1]<br>        outputs = self.regressor(class_label_output)<br>        return outputs</pre><pre>model = CamembertRegressor(drop_rate=0.2)</pre><h4>Setting up the training environment</h4><p>If a GPU is available, it should be used to accelerate the training process. This code should let PyTorch use a GPU, if available, else it will train the model on CPU.</p><pre>import torch</pre><pre>if torch.cuda.is_available():       <br>    device = torch.device(&quot;cuda&quot;)<br>    print(&quot;Using GPU.&quot;)</pre><pre>else:<br>    print(&quot;No GPU available, using the CPU instead.&quot;)<br>    device = torch.device(&quot;cpu&quot;)</pre><pre>model.to(device)</pre><h4>Optimizer, scheduler and loss function</h4><p>We will define the optimizer and the learning rate scheduler for our training process. We will use the Adam optimizer with a 5e-5 learning rate as was done in the official BERT paper.</p><pre>from transformers import AdamW</pre><pre>optimizer = AdamW(model.parameters(),<br>                  lr=5e-5,<br>                  eps=1e-8)</pre><p>To define our scheduler, we must calculate the total number of training steps which is simply the number of training observations multiplied by the number of epochs. We will fine tune our model on 5 epochs.</p><pre>from transformers import get_linear_schedule_with_warmup</pre><pre>epochs = 5</pre><pre>total_steps = len(train_dataloader) * epochs</pre><pre>scheduler = get_linear_schedule_with_warmup(optimizer,       <br>                 num_warmup_steps=0, num_training_steps=total_steps)</pre><p>Our loss function will be the Mean Squared Error loss, the most common loss function for regression problems.</p><pre>loss_function = nn.MSELoss()</pre><h4>Training loop</h4><p>Training our model consists in repeating the following actions successively for each batch of input data at each epoch:</p><ul><li>Unpack input ids, attentions masks and corresponding target prices,</li><li>Load these onto the GPU or CPU device,</li><li>Reset the gradients of the previous training step,</li><li>Compute the prediction (forward pass),</li><li>Compute the gradients (backpropagation),</li><li>Clip gradients to prevent exploding or vanishing gradient issues,</li><li>Update the model parameters,</li><li>Adjust the learning rate.</li></ul><p>The following code on it’s own is thus enough to train our model:</p><pre>from torch.nn.utils.clip_grad import clip_grad_norm</pre><pre>def train(model, optimizer, scheduler, loss_function, epochs,       <br>          train_dataloader, device, clip_value=2):<br>    for epoch in range(epochs):<br>        print(epoch)<br>        print(&quot;-----&quot;)<br>        best_loss = 1e10<br>        model.train()<br>        for step, batch in enumerate(train_dataloader): <br>            print(step)  <br>            batch_inputs, batch_masks, batch_labels = \<br>                               tuple(b.to(device) for b in batch)<br>            model.zero_grad()<br>            outputs = model(batch_inputs, batch_masks)           <br>            loss = loss_function(outputs.squeeze(), <br>                             batch_labels.squeeze())<br>            loss.backward()<br>            clip_grad_norm(model.parameters(), clip_value)<br>            optimizer.step()<br>            scheduler.step()<br>                <br>    return model</pre><pre>model = train(model, optimizer, scheduler, loss_function, epochs, <br>              train_dataloader, device, clip_value=2)</pre><p>However, it is important to regularly compute, store and log the training loss to monitor the learning process. For the sake of brevity, we did not include such code in the post, but you can check out <a href="https://mccormickml.com/2019/07/22/BERT-fine-tuning/">Chris McCormick’s tutorial</a> for inspiration. We computed the MSE loss as well as the R2 score for every 20 batches of training data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/888/1*A1-KILcxzW_j1hscvYwDSw.png" /><figcaption>MSE loss and R2 score per training step</figcaption></figure><p>It is also essential to measure the loss on the separate test set punctually during the training process to ensure the model is not overfitting. We computed the MSE loss and an R2 score and the separate test set at the end of each epoch by calling the following function.</p><pre>def evaluate(model, loss_function, test_dataloader, device):<br>    model.eval()<br>    test_loss, test_r2 = [], []<br>    for batch in test_dataloader:<br>        batch_inputs, batch_masks, batch_labels = \<br>                                 tuple(b.to(device) for b in batch)<br>        with torch.no_grad():<br>            outputs = model(batch_inputs, batch_masks)<br>        loss = loss_function(outputs, batch_labels)<br>        test_loss.append(loss.item())<br>        r2 = r2_score(outputs, batch_labels)<br>        test_r2.append(r2.item())<br>    return test_loss, test_r2</pre><pre>def r2_score(outputs, labels):<br>    labels_mean = torch.mean(labels)<br>    ss_tot = torch.sum((labels - labels_mean) ** 2)<br>    ss_res = torch.sum((labels - outputs) ** 2)<br>    r2 = 1 - ss_res / ss_tot<br>    return r2</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/899/1*6UDgwwKS4DcwustQYp326w.png" /><figcaption>MSE and R2 score on training versus validation sets at each epoch</figcaption></figure><p>It also comes in handy to measure and log the duration of the training steps. We trained our model with 150K training observations on a GPU accelerated Google Colab notebook, it took about 1h30 per epoch.</p><h4>Performance</h4><p>Now that our model is trained, let’s implement a function to collect some predictions. We will collect our model’s prediction on the test set and use these to compute the same performance metrics as for our benchmark model.</p><pre>def predict(model, dataloader, device):<br>    model.eval()<br>    output = []<br>    for batch in dataloader:<br>        batch_inputs, batch_masks, _ = \<br>                                  tuple(b.to(device) for b in batch)<br>        with torch.no_grad():<br>            output += model(batch_inputs, <br>                            batch_masks).view(1,-1).tolist()[0]<br>    return output</pre><p>We will use the validation set we put aside straight from the start to evaluate the final performance of our model. The descriptions of the validation set have to go through the same pre-processing steps as the training data before we are able to make predictions.</p><pre>val_set = val_data[[&#39;id_annonce&#39;, &#39;description&#39;, &#39;prix&#39;]]<br>val_set[&#39;cleaned_description&#39;] = \<br>                val_set.description.apply(clean_text)</pre><pre>encoded_val_corpus = \<br>                tokenizer(text=val_set.cleaned_description.tolist(),<br>                          add_special_tokens=True,<br>                          padding=&#39;max_length&#39;,<br>                          truncation=&#39;longest_first&#39;,<br>                          max_length=300,<br>                          return_attention_mask=True)</pre><pre>val_input_ids = np.array(encoded_val_corpus[&#39;input_ids&#39;])<br>val_attention_mask = np.array(encoded_val_corpus[&#39;attention_mask&#39;])<br>val_labels = val_set.prix.to_numpy()<br>val_labels = price_scaler.transform(val_labels.reshape(-1, 1))<br>val_dataloader = create_dataloaders(val_input_ids, <br>                         val_attention_mask, val_labels, batch_size)</pre><pre>y_pred_scaled = predict(model, val_dataloader, device)</pre><p>Our CamemBERT’s outputs are scaled, we will use the same price scaler we fitted in the pre-processing step to convert them back to euros.</p><pre>y_test = val_set.prix.to_numpy()<br>y_pred = price_scaler.inverse_transform(y_pred_scaled)</pre><p>One of the downsides of BERT is that it is relatively slow for prediction. Nevertheless, it manages to challenge the performances of our benchmark model. It actually outperforms it in terms of Mean Squared Error and comes very close to it in terms of mean and median absolute error and reaches a similar R2 score. In terms of absolute percentage error, it is also very close.</p><pre>from sklearn.metrics import mean_absolute_error<br>from sklearn.metrics import median_absolute_error<br>from sklearn.metrics import mean_squared_error<br>from sklearn.metrics import mean_absolute_percentage_error<br>from sklearn.metrics import r2_score</pre><pre>mae = mean_absolute_error(y_test, y_pred)<br>mdae = median_absolute_error(y_test, y_pred)<br>mse = mean_squared_error(y_test, y_pred)<br>mape = mean_absolute_percentage_error(y_test, y_pred)<br>mdape = ((pd.Series(y_test) - pd.Series(y_pred))\<br>         / pd.Series(y_test)).abs().median()<br>r_squared = r2_score(y_test, y_pred)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0KVmLlCpncIqo64JEChVSg.png" /><figcaption>Fine tuned BERT price performance metrics</figcaption></figure><h3>Recommended next steps: combining numerical, categorical and textual features</h3><p>An interesting next step would be to construct a model that exploits both the tabular set of features used in the previous post and the textual description used in this one. With more information at hand, we would expect such a model to achieve better performances. That said, a good deal of the information is redundant. The challenge thus relies in extracting information from the features and text that is actually complementary and pertinent for the prediction task.</p><p>One approach would be to use text embedding techniques to extract meaningful quantitative features from the text. The embedding features can then be appended to the other numerical and categorical features and used with any supervised regression model.</p><p>Another approach would be to fine-tune a similar BERT based network that includes the numerical and categorical features. These features could for example be appended to the final hidden states before being sent through the regression layer.</p><p>For both approaches, great care will have to be taken to manage the curse of dimensionality. Indeed, we can expect the dimension of the text embedding features, or the dimension of the final hidden states, to be significantly larger than the dimension of the numerical and categorical features. Dimensionality reduction or weighting methods could thus come in handy.</p><h3>Conclusion</h3><p>In our previous article, we used a tabular set of numerical and categorical features to predict list prices. Today, we showed that using only the textual feature, we are capable of reaching very similar performances. These results are satisfactory, far better than random, but can surely be improved.</p><p>Most of the posts and tutorials we come across about fine-tuning BERT focus on supervised classification tasks. Through this price prediction use-case, we showed that it can effectively be used for supervised regression tasks too.</p><h4><em>Acknowledgements</em></h4><p><em>I would like to give a special thanks to Kawtar Zaher, Anne-Marie Heng, Lorraine Hickson and Louis Boulanger for their great contribution to the this article.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cf97cd7cb98a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ilb-labs-publications/fine-tuning-bert-for-a-regression-task-is-a-description-enough-to-predict-a-propertys-list-price-cf97cd7cb98a">Fine-tuning BERT for a regression task: is a description enough to predict a property’s list price?</a> was originally published in <a href="https://medium.com/ilb-labs-publications">ILB Labs publications</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[List price prediction with scraped real-estate data]]></title>
            <link>https://medium.com/ilb-labs-publications/list-price-prediction-with-scraped-real-estate-data-df5e49f14547?source=rss-4d6214013c99------2</link>
            <guid isPermaLink="false">https://medium.com/p/df5e49f14547</guid>
            <category><![CDATA[price-prediction]]></category>
            <category><![CDATA[regression]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[real-estate]]></category>
            <dc:creator><![CDATA[Anthony Galtier]]></dc:creator>
            <pubDate>Mon, 06 Sep 2021 09:58:22 GMT</pubDate>
            <atom:updated>2022-12-12T16:17:34.121Z</atom:updated>
            <content:encoded><![CDATA[<p>This is the first of two articles about using data scraped from the web and machine learning to predict the list price of real-estate listings.</p><p>This first article focuses on a tabular set of numerical and categorical features we collected describing the listings. We will explore these features and show how they can be processed and used in a supervised regression model to predict the list price.</p><p>In the second article, we will focus on a textual description variable we collected and see how we can leverage the latest advances in NLP to achieve similar or better list price prediction performances. Through this practical example, we will be showing how BERT can be fine-tuned for a regression task.</p><h4>The data set</h4><p>The data set used in this article was scrapped from French real-estate websites.</p><p>This dataset is different from usual housing datasets since it focuses on list prices and not transaction prices. We can expect some bias between the two, list prices are expected to be on average higher than effective transaction prices. The biggest advantage of this dataset is that we have many features that are often absent from housing datasets.</p><p>It includes data from <strong>788K listings </strong>such as the one below for which <strong>31 features</strong> were collected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/704/1*8DOFgatRpvVRZb3Qmj0bIA.png" /></figure><p>The features we have at hand are:</p><ul><li>the list price which will be our target variable,</li><li>the property type (house, apartment, condo, mansion…),</li><li>the location (latitude, longitude, city, department),</li><li>the size (living area and land area when applicable),</li><li>the number of rooms, bedrooms, bathrooms…</li><li>energy performance indicators (energy and greenhouse gas emissions)</li><li>the number of photos attached to the listing,</li><li>indicators whether there is a garden, a balcony, air conditioning…</li><li>and lastly, a free text form description of the property in French. We will talk more about this textual feature and NLP in the second article.</li></ul><h3>Feature exploration</h3><h4>Train test split</h4><p>The exploratory data analysis is expected to influence our pre-processing and modelling decisions. To minimize our bias when evaluating performance later, we will do a train test split straight away and focus on the training set.</p><pre>import pandas as pd<br>from sklearn.model_selection import train_test_split</pre><pre>file_path = &quot;real_estate_data.csv&quot;<br>data = pd.read_csv(file_path, <em>sep</em>=&#39;,&#39;, <em>index_col</em>=0)</pre><pre>train_data, test_data = train_test_split(data, <em>train_size</em>=<em>int</em>(3e5),    <br>                                         <em>random_state</em>=42)</pre><pre>df = train_data</pre><p>The data set we are dealing is voluminous, in it’s original CSV file format it takes up 1.68GB. Training complex machine learning models on such large data with limited computing resources will be time consuming. For this reason, we will only consider 300K observations for training out of the original 788K. The rest will be kept for evaluation purposes. Thus, we assume that the random sample of ~40% taken from our set is representative of the problem domain.</p><h4>Duplicate values</h4><p>With this one-liner, we can confirm that none of the rows are exact duplicates of each other across all features.</p><pre>df.duplicated(<em>keep</em> = False).sum()<br># returns 0</pre><p>That said, certain features alone should not contain any duplicate values by nature. This is the case for the listing ID (“<em>id_annonce</em>” variable) and arguably for the textual description of the listing (“<em>description</em>” variable).</p><p>It turns out 1.25% of the rows have duplicate listing IDs and 21% have duplicate descriptions. Since none of these rows are duplicates of each other across all features we looked into which features tend to differ for ID or description duplicates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*Vqx1uQwwAJPxIV0g4xhzGw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*ZuyDLOgNEkghB3NemXGd6Q.png" /><figcaption>Number of distinct values for listings with duplicate IDs or descriptions (only displaying values ≥ 2)</figcaption></figure><p>Having listings with a same ID but a different price (“<em>prix</em>”), number of rooms (“<em>nombre_pieces</em>”), size (“<em>superficie</em>”)… seems incoherent. Likewise, we would not expect properties with a same description to have a different ID, price, number of rooms (“<em>nombre_chambres</em>”)… These duplicates are thus removed from the set altogether.</p><pre>df.drop_duplicates(subset = &quot;id_annonce&quot;, keep=False, inplace=True)<br>df.drop_duplicates(subset = &quot;description&quot;, keep=False, inplace=True)</pre><p>Removing these duplicates reduces the size of the training set by 21.12%.</p><h4>Missing values</h4><p>Missing value rates in our features can quickly be calculated with the following line of code:</p><pre>df.isna().sum()/len(df)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/717/1*fyt4XEOuiqGWNH5OQqyZ-w.png" /><figcaption>Missing value rates after dropping duplicates</figcaption></figure><p>To avoid using artificial values in our target variable, any row where the price is missing is discarded.</p><p>To guarantee coherent geographical analyses later on and since there are only few, rows with missing latitude and longitude values are also removed.</p><pre>df = df.dropna(subset = [&#39;prix&#39;, &#39;latitude&#39;, &#39;longitude&#39;])</pre><p>Part of the missing values in living area (“<em>superficie</em>”) and land size (“<em>superficie_terrain</em>”) can be imputed based on the type of property (“<em>type_bien</em>”). We can assume that appartements (“<em>appartement</em>”), lofts and single rooms (“<em>chambre</em>”) have a land size of zero. Likewise, we can assume that plots of land or building sites (“<em>terrain</em>” and “<em>terrain à bâtir</em>”) have a living area of zero.</p><pre>df.loc[df.type_bien.isin([&#39;appartement&#39;, &#39;loft&#39;, &#39;chambre&#39;]) \<br>       &amp; df.superficie_terrain.isna(), &#39;superficie_terrain&#39;] = 0</pre><pre>rows = df.type_bien.isin([&#39;terrain à bâtir&#39;, &#39;terrain&#39;]) \<br>                          &amp; df.superficie_terrain.isna() \<br>                          &amp; ~df.superficie.isna()</pre><pre>df.loc[rows, &#39;superficie_terrain&#39;] = df.loc[rows, &#39;superficie&#39;]<br>df.loc[rows, &#39;superficie&#39;] = 0</pre><p>Doing so, the missing value rate in the living area slightly decreases while in the land size variable it drops from 40% to 9%.</p><p>The energy performance variables (features ending by “<em>_energie</em>” or “<em>_ges</em>”) go by pairs. One is a numerical value (starting with “<em>valeur_</em>”), the other (starting with “<em>categorie_</em>”) is a category “A”, “B”, “C”… based on the latter numerical value. Missing energy performance variables could have been imputed based on the observed value-category correspondence but unfortunately, when one is missing, the other is missing too. It can be visualised using the <a href="https://github.com/ResidentMario/missingno">missingno</a> toolset.</p><pre>import missingno</pre><pre>missingno.matrix(df[[&#39;categorie_energie&#39;, &#39;valeur_energie&#39;,  <br>                     &#39;categorie_ges&#39;, &#39;valeur_ges&#39;]])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qlXwzevVwprY6RFtrenA_Q.png" /><figcaption>Missingno matrix displaying missing observations in a feature as black lines for the energy performance variables</figcaption></figure><p>Missing values in the rest of the features, including the energy performance variables, are thus imputed by default with the median value for numerical features and an “Unknown” modality for categorical ones.</p><pre>from pandas.api.types import is_string_dtype<br>from pandas.api.types import is_numeric_dtype</pre><pre>for col in df.columns:<br>    if col != &#39;prix&#39;:<br>        if is_numeric_dtype(df[col]):<br>            df[col] = df[col].fillna(df[col].median())<br>        elif is_string_dtype(df[col]):<br>            df[col] = df[col].fillna(&#39;Unknown&#39;)</pre><p>After dealing with duplicates and missing values, the training data set is 21.4% smaller than the original 300K observations, standing at 235K observations.</p><h4>Outliers</h4><p>Looking into the distributions of the features, we notice that some values are incoherent or problematic for our price prediction objective.</p><p><strong>Continuous numerical features</strong></p><p>Focusing on continuous numerical variables, we notice that the distribution of the top 1% of the price, energy performance, greenhouse gas emission, living and land size are far off from the remaining 99% of the values. The maximum value for these features is 10e3 to 10e6 times greater than the 99th percentile. Most of these outliers seem incoherent by nature. They are also likely to be problematic when modelling or evaluating performances, especially for the outliers in the target variable.</p><pre>continuous_numerical = [ <br>    &#39;prix&#39;, # price<br>    &#39;valeur_energie&#39;, # energy performance value<br>    &#39;valeur_ges&#39;, # greenhouse gas emission performance value<br>    &#39;latitude&#39;,<br>    &#39;longitude&#39;,<br>    &#39;superficie&#39;, # living area <br>    &#39;superficie_terrain&#39;, # land size<br>    ]</pre><pre>df[continuous_numerical].describe([0.1, 0.25, 0.5, 0.75, 0.9, <br>                                 0.99]).round()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/928/1*XsmgPZGgUy-qrmpqvbaTjQ.png" /><figcaption>Summary of the distribution of the continuous numerical features</figcaption></figure><p><strong>Discrete numerical features</strong></p><p>Likewise, focusing on the discrete numerical values, we notice that the maximum value is an order of magnitude of 10 to 100 times greater than the 99th percentile for most of these features.</p><pre>discrete_numerical = [<br>    &#39;nombre_pieces&#39;, #number of rooms<br>    &#39;nombre_chambres&#39;, #number of bedrooms<br>    &#39;nombre_salles_de_bain&#39;, #number of bathrooms<br>    &#39;etage&#39;, #floor<br>    &#39;nombre_box&#39;, #number of boxes<br>    &#39;nombre_terrasses&#39;, #number of terrasses<br>    &#39;nombre_parkings&#39;, #number of parking spots<br>    &#39;nombre_photos&#39;#number of photos attached to the listing<br>]</pre><pre>df[discrete_numerical].describe([0.1, 0.25, 0.5, 0.75, 0.9, 0.99]).round()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hDQnX9Pdz0RVCBoTAgnoCg.png" /><figcaption>Summary of the distribution of the discrete numerical features</figcaption></figure><p><strong>Outlier removal in numerical features</strong></p><p>The top 1% of the continuous and discrete numerical features with outliers is thus removed from the set. The bottom 1% of the price feature is also deemed incoherent and is discarded from the set.</p><pre>def remove_outliers(df, lower_outliers, q_bottom, upper_outliers, <br>                    q_top):<br>    lower_quantiles = df[lower_outliers].quantile(q_bottom)<br>    for col in lower_outliers:<br>        df = df[df[col] &gt;= lower_quantiles[col]]<br>    upper_quantiles = df[upper_outliers].quantile(q_top)<br>    for col in upper_outliers:<br>        df = df[df[col] &lt;= upper_quantiles[col]]<br>    return df</pre><pre>upper_outliers = [&#39;prix&#39;, &#39;valeur_ges&#39;, &#39;valeur_energie&#39;, <br>                  &#39;superficie&#39;, &#39;superficie_terrain&#39;]<br>upper_outliers += discrete_numerical<br>lower_outliers = [&#39;prix&#39;]</pre><pre>df = remove_outliers(df, lower_outliers, 0.01, upper_outliers, 0.99)</pre><p><strong>Binary features</strong></p><p>Looking at the rates of zeros and ones in the binary variables revealed that four of the features actually carry no information and should thus not be included in the study.</p><pre>binary = [ <br>    &#39;presence_balcon&#39;, # has a balcony<br>    &#39;presence_cave&#39;, # has a cellar<br>    &#39;presence_garage&#39;, # has a garage<br>    &#39;presence_climatisation&#39;, # has air conditionning<br>    &#39;etage_dernier_etage&#39;, # on the top floor<br>    &#39;etage_etage_eleve&#39;, # in the upper floors<br>    &#39;etage_rez_de_jardin&#39;, # on the ground floor<br>    &#39;superficie_multiples_biens&#39;, # combined area of several <br>                                  # properties<br>    &#39;is_neuf&#39;, # is new<br>    ]</pre><pre>binary_rates = 100*df[binary].sum()/len(df)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/717/1*a_GUhgxFGSpw0CK3PHsFbg.png" /><figcaption>Rates of 1s in the binary (0 or 1) features</figcaption></figure><p><strong>Categorical features</strong></p><p>The modality counts of the categorical variables vary significantly. The geographical variables (“department” and “ville”) have over 5K distinct modalities. Encoding such a large number of modalities is not straightforward and since the geographical information is to some extent redundant with the latitude and longitude variables, we chose to discard these features.</p><pre>categorical = [<br>    &#39;categorie_energie&#39;, #energy performance category<br>    &#39;categorie_ges&#39;, #greenhouse gas emission category<br>    &#39;ville&#39;, #city<br>    &#39;departement&#39;, #postal code<br>    &#39;type_client&#39;, #type of client advertising the listing<br>    &#39;source&#39;, #source of the listing<br>    &#39;exposition&#39;, #direction the property is facing<br>    &#39;type_bien&#39;, #property type<br>]</pre><pre>modality_counts = df[categorical].nunique()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/478/1*YVzpTSJe6rzLIaOfKuQlJQ.png" /><figcaption>Modality counts in the categorical features</figcaption></figure><p>The energy performance and greenhouse gas emission features are ordered categorical features, “A” is better than “B” which is better than “C”… We can thus explicitly convert the feature to a discrete numerical ranking. Missing values or “Unknown” values should here too be imputed with the median value.</p><pre>for col in [&#39;categorie_energie&#39;, &#39;categorie_ges&#39;]:<br>        df[col] = df[col].replace({&#39;A&#39;:0, <br>                                   &#39;B&#39;:1, <br>                                   &#39;C&#39;:2, <br>                                   &#39;D&#39;:3, <br>                                   &#39;E&#39;:4, <br>                                   &#39;F&#39;:5, <br>                                   &#39;G&#39;:6, <br>                                   &#39;Unknown&#39;: None})<br>        df[col] = df[col].fillna(value=df[col].median())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/383/1*LGp9R7Yxi-sGzNbVMPJhTw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/383/1*nNtePW2gwmWLf-3hrPWl2g.png" /><figcaption>Encoding of the energy performance and greenhouse gas categories</figcaption></figure><p>The remaining features can be one-hot encoded. Regarding the property type variable, since most of the 28 modalities concern less than 1% of the observations, we will group them under a single “autre” modality, which means “other” in French, to prevent the one-hot encoding from increasing the dimensionality of the training set too much.</p><pre>type_shares = df[&#39;type_bien&#39;].value_counts(normalize = True)<br>minor_types = type_shares[type_shares&lt;0.01].index.tolist()<br>df[&#39;type_bien&#39;] = df[&#39;type_bien&#39;].replace(minor_types, &#39;autre&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/717/1*8epqiDNuJUwdCfXvtFwJ_g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/717/1*IAiAVf8NiyQREAVJLafJ1A.png" /><figcaption>Distribution of the property type variable before and after grouping least represented modalities</figcaption></figure><pre>dummies = [&#39;type_bien&#39;, &#39;type_client&#39;, &#39;source&#39;, &#39;exposition&#39;]<br>df = pd.get_dummies(df, columns=dummies)</pre><p><strong>Textual features</strong></p><p>In the data that was scraped, there are two textual features, the title and the description. The description will be used to build another list price prediction model based on BERT in the next article. To make results comparable, it is best to use the same training and test observations. Outliers in the description feature will thus be dealt with right away.</p><p>Looking at the distribution of their lengths, we remark some incoherencies. As shown below, too short descriptions do not make much sense or do not actually describe the property.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/910/1*N2xh7Oat0MQJgFEW6nkp7Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/774/1*7tYdZYG6GNv7-BtSJpkb4A.png" /><figcaption>Distribution of the length of the property descriptions in number of characters (left) and sample of the shortest descriptions (right)</figcaption></figure><p>Descriptions shorter than 100 characters are deemed to be uninformative and are excluded for the set.</p><pre>df = df[df.description.apply(len)&gt;100]</pre><h4>Multivariate analysis</h4><p>At this point, we can consider our training set to be clean enough and ready for the actual modelling phase. Before diving into it, let’s do a quick multivariate analysis, looking at the Spearman correlations of the training features. Since we have many binary or discrete features in our data set, we chose to look at the Spearman correlation coefficient because it focuses on relative ranks rather than relative values like the Pearson coefficient does.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/1*x2CnT_Zz_hNYtRB-lm9U-w.png" /><figcaption>Spearman correlation coefficients between feature, only displaying coefficients with an absolute value &gt;0.2</figcaption></figure><p>As one could expect, the number of rooms, bedrooms, bathrooms… are positively correlated. Energy performance and greenhouse gas emission values and categories are also unsurprisingly positively correlated. Some property type indicator features are positively or negatively correlated with numerical features. Obviously, the “building plot” indicator is negatively correlated with the number of rooms, bathrooms… but interestingly, the “appartement” property type indicator is positively correlated with the longitude feature whereas it is negatively correlated with the “house” indicator suggesting that there may be more houses in the West and more apartements to the East. Further analyses would be needed to confirm this.</p><h4>Pre-processing pipeline summary</h4><p>Throughout the exploration phase of the training set, we constructed a pre-processing procedure that deduplicates the data, deals with missing values and outliers, selects and encodes the features.</p><p>We packed all these pre-processing steps into a scikit-learn pipeline module. For the sake of brevity, it is not included in this blog post.</p><p>71.5% of the original 300K observations are deemed pertinent for the modelling task. The training set thus stands at 215K observations of 66 features.</p><h3>Supervised regression model</h3><p>Our objective is to predict the list price of a real-estate listing based on the pre-processed features of our data set. The list price is a continuous numerical variable. We will thus chose a supervised regression model to predict it. In this article, we will focus on the numerical and categorical variables. The next post will show how the textual variable can be used for the same task.</p><p>To start off, both the training and test sets should be pre-processed and the features that are not to be considered in the model should be put aside.</p><pre>from preprocessing import preprocessing_pipeline</pre><pre>train_data = preprocessing_pipeline.fit_transform(train_data)<br>test_data = preprocessing_pipeline.transform(test_data)</pre><pre>train_data = train_data.drop([&#39;titre&#39;, &#39;description&#39;,&#39;id_annonce&#39;, <br>                              &#39;ville&#39;, &#39;departement&#39;], axis = 1)<br>test_data = test_data.drop([&#39;titre&#39;, &#39;description&#39;,&#39;id_annonce&#39;, <br>                            &#39;ville&#39;, &#39;departement&#39;], axis = 1)</pre><p>We recommend choosing a gradient boosted tree supervised regression model for this price prediction task. We tried some linear regression methods such as Lasso, Ridge or ElasticNet but came to realise that they were outperformed by models such as XGBoost and Light GBM. Since <a href="https://towardsdatascience.com/lightgbm-vs-xgboost-which-algorithm-win-the-race-1ff7dd4917d">LGBMs are faster to train</a>, we stuck with that.</p><p>LGBM, Light Gradient Boosting Machine, is a distributed tree-based gradient boosting framework. If you know nothing about gradient boosting:</p><ul><li><a href="https://www.youtube.com/watch?v=3CC4N4z3GJc">StatQuest’s videos</a> can give you a grasp of what it’s about.</li><li>For more detail, you check out <a href="https://medium.com/@pushkarmandot/https-medium-com-pushkarmandot-what-is-lightgbm-how-to-implement-it-how-to-fine-tune-the-parameters-60347819b7fc">this post</a> or <a href="http://lightgbm.readthedocs.io/">read the docs</a>.</li><li>To see how it compares to XGBoost, check out<a href="https://towardsdatascience.com/catboost-vs-light-gbm-vs-xgboost-5f93620723db"> this post</a>.</li></ul><pre>from lightgbm import LGBMRegressor</pre><pre>estimator = LGBMRegressor(random_state = 42)</pre><p>To rapidly tune the parameters of the LGBM model we use scikit-learn’s RandomizedSearchCV module. This method computes the cross-validated scores of the model for different combinations of parameters sampled randomly from a set or a distribution of possible parameters. It is less computationally costly than trying out all the possible combinations as is done with the GridSearchCV module yet it provides similar results when enough parameter combinations are taken.</p><pre>from sklearn.model_selection import RandomizedSearchCV</pre><pre>parameters = {<br>    &#39;num_leaves&#39; : [10, 30, 50, 100, 200],<br>    &#39;max_depth&#39;: [None, 5, 10, 20, 50],<br>    &#39;n_estimators&#39;: [150, 200, 400, 600],<br>    &#39;learning_rate&#39;: [0.05, 0.1, 0.25, 0.5]<br>    }</pre><pre>model = RandomizedSearchCV(estimator, parameters, random_state=42, <br>                           scoring = &#39;r2&#39;, <em>n_iter</em> = 50)</pre><p>It took about 11 minutes to fit and tune the model, trying out 50 random combinations of the parameters listed above on CPU with 215K training observations and 5 fold cross validation.</p><pre>X_train = train_data.drop([&#39;prix&#39;], axis = 1)<br>y_train = train_data.prix</pre><pre>model.fit(X_train, y_train)</pre><h3>Performance</h3><p>Now that the list price prediction model is trained, it is time to evaluate it’s performance on the training set. To do so, we compute the price predictions for the pre-processed observations of the test set.</p><pre>X_test = test_data.drop([&#39;prix&#39;], axis = 1)<br>y_test = test_data.prix</pre><pre>y_pred = model.best_estimator_.predict(X_test)</pre><p>The scikit-learn library offers a variety of metrics to evaluate the performance of a supervised regression model. Most of the metrics summarise the distribution of the errors made by the model (MAE, MSE, MAPE…). It is interesting to look at these metrics alongside the <a href="https://en.wikipedia.org/wiki/Coefficient_of_determination">R squared</a> metric that quantifies the degree of any linear correlation between the predictions and true observed target values.</p><pre>from sklearn.metrics import mean_absolute_error<br>from sklearn.metrics import median_absolute_error<br>from sklearn.metrics import mean_squared_error<br>from sklearn.metrics import mean_absolute_percentage_error<br>from sklearn.metrics import r2_score</pre><pre>mae = mean_absolute_error(y_test, y_pred)<br>mdae = median_absolute_error(y_test, y_pred)<br>mse = mean_squared_error(y_test, y_pred)<br>mape = mean_absolute_percentage_error(y_test, y_pred)<br>mdape = ((pd.Series(y_test.to_numpy()) \<br>         - pd.Series(y_pred))\<br>         / pd.Series(y_test.to_numpy())).abs().median()<br>r_squared = r2_score(y_test, y_pred)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/923/1*jsL-ezEMZD5Kp3Ir2dL-5w.png" /><figcaption>LGBM price prediction model performance</figcaption></figure><p>This model was able to predict the list price of the 311K properties from the separate test set averaging an absolute error of 62K euros with a median absolute error at 29K euros.<br>For this task, it makes sense to talk about error in relative terms, a 50K euro error on a million-euro property is not as bad as the same error on a 100K property. The Mean Average Percentage Error (MAPE) thus comes in handy. On average, the model predictions are off by 27% of the actual price but the Median Average Percentage Error is only 15%. The R squared score is also satisfactory, reaching 0.82.</p><h3>Recommended next steps: explainability and robustness</h3><p>An interesting complement to this study would be to explain how the model is actually predicting the list price of the property. A first step we can take in this direction is to look at the feature importance of the fitted LGBM.</p><pre>feature_importances = \<br>            pd.DataFrame(model.best_estimator_.feature_importances_,<br>                 index = X_train.columns, columns = [&#39;importance&#39;] \<br>                 ).sort_values(&#39;importance&#39;, ascending=False)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/234/1*V0ZnIVUOsuvMKEqF3ndejQ.png" /><figcaption>Top 10 features by feature importance for the fitted LGBM model</figcaption></figure><p>With an LGBM, like other tree based ensemble learning models, the “importance” of each feature in the prediction decision can be measured as the decrease in the scoring metric at each split involving that feature weighted by the probability of reaching that split.</p><p>In the case of this model, the “feature importance” values suggest that the prediction value depends mostly on the location and size of the property which makes sense. Interestingly, the energy performance features and number of photos also seem to play a role. That said, conclusions based on feature importance values are to be taken with a pinch of salt, they can be misleading. When features are correlated, their features importance value is shared between them, thus understating their potential predictive power.</p><p>A more sophisticated and robust method to analyse the influence of features could be to look at SHAP values. They can provide information about both the magnitude and direction of the value of a feature can have on the final prediction.</p><ul><li>You can read <a href="https://towardsdatascience.com/explain-your-model-with-the-shap-values-bc36aac4de3d">more about SHAP values in this post</a>.</li></ul><p>It would also be interesting to perform some robustness tests, comparing performances of our model for different housing types, for different areas, different countries… We could also test our model for robustness in time. How would this model perform on listings that are several years old or on future listings?</p><h3>Conclusion</h3><p>In this article we explored real-estate data and constructed a model that is capable of predicting the list price of a real-estate property based on a set of numerical and categorical features scraped from online listings. The performance we got out of this model is satisfactory, it is a lot better than random but there remains room for improvement.</p><p>In our next post, we will exploit the textual information contained in the description of the real-estate property to predict the list price. We will see if we can leverage the power of NLP to achieve similar if not better results than those we got today.</p><p>The purpose of the second article is to provide a practical example of how Google’s BERT can be fine-tuned for a regression task. Usually most posts focus on fine-tuning BERT for classification, but we will be showing that it can easily be adapted to regression tasks too.</p><h4>Acknowledgements</h4><p><em>I would like to give a special thanks to Kawtar Zaher, Anne-Marie Heng, Lorraine Hickson and Louis Boulanger for their great contribution to the this article.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=df5e49f14547" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ilb-labs-publications/list-price-prediction-with-scraped-real-estate-data-df5e49f14547">List price prediction with scraped real-estate data</a> was originally published in <a href="https://medium.com/ilb-labs-publications">ILB Labs publications</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>