<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Diogo Matos Chaves on Medium]]></title>
        <description><![CDATA[Stories by Diogo Matos Chaves on Medium]]></description>
        <link>https://medium.com/@diogomatoschaves?source=rss-dccf9e76f185------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*gveT1L-uDNimBL5HeXyXYQ.png</url>
            <title>Stories by Diogo Matos Chaves on Medium</title>
            <link>https://medium.com/@diogomatoschaves?source=rss-dccf9e76f185------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 15:14:50 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@diogomatoschaves/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to Implement a Backtester in Python]]></title>
            <link>https://medium.com/@diogomatoschaves/how-to-implement-a-backtester-in-python-030b968f6e8d?source=rss-dccf9e76f185------2</link>
            <guid isPermaLink="false">https://medium.com/p/030b968f6e8d</guid>
            <category><![CDATA[investing]]></category>
            <category><![CDATA[trading]]></category>
            <category><![CDATA[algorithmic-trading]]></category>
            <category><![CDATA[backtesting]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Diogo Matos Chaves]]></dc:creator>
            <pubDate>Sun, 11 Feb 2024 17:19:15 GMT</pubDate>
            <atom:updated>2024-02-12T01:15:57.145Z</atom:updated>
            <content:encoded><![CDATA[<h4>And which library to use if you don’t want to code one yourself</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uDkS64QQc3uJYJVQEt8MLw.png" /></figure><h3>Introduction</h3><p><em>Note: Understanding how backtesting works is quite beneficial for creating profitable strategies, but it is not required to re-invent the wheel when there’s no need; hence you can use a </em><a href="https://github.com/diogomatoschaves/stratestic"><strong><em>library</em></strong></a><em> I authored which implements the logic we discuss here, along with many other features. Check it out!</em></p><p>In the world of algorithmic trading, backtesting plays a crucial role in the development and evaluation of trading strategies. It allows traders and developers to assess how a given strategy would have performed on historical data, providing insights into its strengths and weaknesses. There are different approaches to backtesting, with two primary methods being <strong>vectorized</strong> and <strong>iterative</strong> backtesting—which we’ll explore in more detail in this guide.</p><p>At its core, Backtesting involves applying a <strong>trading strategy</strong> to historical market data to evaluate its performance, in essence simulating executing the trades we would have done—according to the strategy—if we were replaying the past.</p><p>Backtesting is a crucial step in the development process for several reasons. It helps validate a strategy’s effectiveness, assesses risk, and provides a quantitative basis for decision-making. Neglecting thorough backtesting can lead to the implementation of flawed strategies, resulting in significant financial losses.</p><p>Having said that, it is not a magic bullet. A trading strategy performing well in a backtest does not automatically guarantee success in trading with live money, so it is recommended to proceed with caution and test strategies in a paper trading account first.</p><p>In this guide, we’ll delve into the basics of implementing a backtester—both in a vectorized and iterative way. So without further ado, let’s dive right into it!</p><h3>Implementation</h3><p>Below we’ll go step by step, and you can follow along on the <a href="https://gist.github.com/diogomatoschaves/53b7af03eb193a66ef1c380ba961ac50">notebook</a> as well.</p><h4>Loading the data and calculate the returns</h4><p>We’ll first load the sample data we’ll use for this exercise, which consists of 100 rows of hourly OHLC data for <em>Bitcoin</em>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0e27eff62426fe6cfc4dfefc5d46d9a6/href">https://medium.com/media/0e27eff62426fe6cfc4dfefc5d46d9a6/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/854/1*If_J1NsFQ8hePURdGc_PrQ.png" /><figcaption>First 5 rows of the data dataframe</figcaption></figure><p>We can then calculate the logarithmic returns for this dataset. The column we’ll be using for the prices is the close column, which is the price that the asset has when each time period ends.</p><p><em>Note: Using logarithmic returns has several advantages, with the most prominent being that it allows us to take the sum of both long and short returns in order to get the total strategy return. I’ll delve into this topic in more detail on a subsequent post</em>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7d8a8c6cfbf5037382282fa0c34b6b5c/href">https://medium.com/media/7d8a8c6cfbf5037382282fa0c34b6b5c/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/994/1*ARtJYfqgoN4hgxPP-fAB6A.png" /><figcaption>First 5 rows of data with returns</figcaption></figure><h4>Calculate the strategy signals</h4><p>The next step is to calculate the signal for the specific strategy being tested. For this example we’ll be using a Simple Moving Average strategy, which outputs a <strong>Long</strong> signal whenever the price goes above the moving average, and a <strong>Short</strong> signal whenever it goes below. The moving average window used in this example is <em>20</em>, but obviously this can be changed to whatever value we want — as long as it is below the total number of rows in the data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3baeb8a01e00a80ef4be4bbe5fdc75e7/href">https://medium.com/media/3baeb8a01e00a80ef4be4bbe5fdc75e7/href</a></iframe><p>We’ve calculated the moving average with a period of 20, and then assigned a <strong>1 </strong>for the periods where the close price is higher than the moving average, and a <strong>-1 </strong>for the periods where it is below. We then drop the rows where we have missing values. This leaves us with the following dataframe.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OPrF5s6Agg0HhXDfSKbEnQ.png" /><figcaption>First 5 rows of data with signal</figcaption></figure><h4>Vectorized Backtesting</h4><p>Now that we have all the required ingredients, we can calculate the strategy returns according to its signals. For this we only need to multiply the return on each row by the signal from the previous row, since we are calculating each signal with the price at the end of each period.</p><p>Breaking down this operation, it’s as if we get to the end of a period (say to the end of the first period above, 2023-02–09 16:00), calculate the corresponding signal according to the price on which that period closed (the close price), then wait for the next period to end (in this case, for 1 hour to pass since we are using hourly data), and then multiply the return of that period with the position we had during that period (long or short). In a vectorized fashion, this equates to the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c02a0ffe20b0c2e2d8d2998e30295878/href">https://medium.com/media/c02a0ffe20b0c2e2d8d2998e30295878/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T0hoqtQnGH_UGiL1jXG4cw.png" /><figcaption>First 5 rows of data with strategy returns</figcaption></figure><p>As we can see now we have a strategy_returns column, which is the same as the returnscolumn, but has the opposite sign since the signal for these rows is always -1.</p><p>We can now calculate the total return of this strategy, as well as for the <em>buy &amp; hold </em>strategy—which is the same as having a strategy with only long positions. This is where the logarithmic returns come in handy, as now we can get the total return by taking the exponential of the cumulative sum of the returns.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f599fa16ef352d8a8d1aed29e141741f/href">https://medium.com/media/f599fa16ef352d8a8d1aed29e141741f/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9ElV7si11uP4GvxNp_P6Tg.png" /><figcaption>Last 5 rows of the data with the cumulative returns</figcaption></figure><p>Finally, if we want to get the return for the whole backtesting we just have to take the corresponding value from the last row. Hence our backtest outputs a total cumulative return of <em>1.020306</em> for our strategy, and of <em>0.968810</em> for the <em>buy &amp; hold</em> strategy, which means that our strategy would have provided us with a positive return, whereas just holding the asset would lead to a negative return.</p><p><strong>Taking into account trading costs</strong></p><p>In order to take incorporate trading costs into the backtestand assuming the trading costs come as a percentage of the total trade value—, we need only to subtract the trading cost from the strategy return, since a return is already a percentage. For this we need an auxiliary column trades, which registers the trades performed per row. So for example if we are going from a <em>neutral</em> to a <em>long</em> position it would be 1 trade, whereas from a <em>long</em> to a <em>short </em>position<em> </em>equates to 2 trades. The total trading cost is then the multiplication of the number of trades by the unitary trading cost.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e368115dff1778cb0c458e6a00ddd2ac/href">https://medium.com/media/e368115dff1778cb0c458e6a00ddd2ac/href</a></iframe><p>The above would result in the following results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SYL2zADPl36YSqow3gakdg.png" /><figcaption>Last 5 rows of results with trading costs</figcaption></figure><p>One can see that the total return with a 0.1% trading cost brings our final cumulative return to 1.010895, which is, as expected, lower than the return for the strategy without taking into account trading costs.</p><h4>Iterative Backtesting</h4><p>Iterative backtesting, also known as event-driven backtesting, follows the same principles in theory, but as the name indicates, instead of calculating the returns and other quantities in a vectorized fashion, we’ll iterate through the data and calculate those values for each row of data.</p><p>Iterative backtesting offers the advantage of allowing us to implement complex logic that would be difficult to implement in a vectorized way or outright not possible. The iterative backtester will shine in these more complex logic cases. But this comes at the expense of slower execution and more boilerplate code just to make the same backtest as we did in the vectorized example.</p><p>Below is a simple implementation of such a backtester, and it is divided in a few functions for modularity. The main logic happens in a for loop, shown below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cd753f5c1e6fe5ebbdacdbac31acd822/href">https://medium.com/media/cd753f5c1e6fe5ebbdacdbac31acd822/href</a></iframe><p>We start with a few initializations and then move inside the for loop, where we iterate over the data, compute the signal according to the strategy (in this case the Moving Average strategy is hard-coded), and then call another function trade, where we actually execute the trading logic.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0aa7a68f8e9254434be88209d8afde58/href">https://medium.com/media/0aa7a68f8e9254434be88209d8afde58/href</a></iframe><p>The trade function has a few <em>if else</em> conditions in order to determine what happens next. So if for example our current position is short, and we have a <em>long</em> signal incoming, we would first execute a trade with the current units—in order to bring the current position back to neutral, followed by another trade with the full account balance, in order to bring us to a long position. The same logic in reverse follows if we have a <em>short</em> signal. The trade function makes use of the buy_asset and sell_asset functions, shown below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/aed4fe60bc5b15dba8a85f4ef33f7d52/href">https://medium.com/media/aed4fe60bc5b15dba8a85f4ef33f7d52/href</a></iframe><p>These functions execute the atomic logic behind buying or selling the asset, which includes updating the current balance and units accordingly, converting the amount to units if required.</p><p>This is the minimum logic behind backtesting a strategy iteratively, and it can get considerably more complex once one starts to add more features. In that case, a class-based implementation of an Iterative Backtester can bring considerable advantages as the logic gets more complex and more variables need to be kept in sync between methods calls.</p><p>Starting with an equity of 1000 USD, if we execute the backtester following our moving average strategy we would end up with 1019.65 USD.</p><p><strong>Taking into account trading costs</strong></p><p>In order to take into account the trading costs, we have to think as if we’re paying a premium on the asset, so in essence paying a higher price if we are buying the asset, and receiving a lower price if we are selling the asset. The modified buy_asset and sell_asset functions would be:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/442cf4b5bd26cbcba74cac5d8c679770/href">https://medium.com/media/442cf4b5bd26cbcba74cac5d8c679770/href</a></iframe><p>The final balance would be 1009.59 USD, a value that is, as expected, less than when we did not take into account the trading costs.</p><h3>Conclusion</h3><p>In this article we saw the basic building blocks behind implementing a backtester in python, both in a vectorized and iterative fashion. We saw that the vectorized way is a lot simpler to implement, but it comes at the cost of limited logic implementation, and vice versa for the iterative way.</p><p>The implementations for each case above are the minimum that is required, and it can quickly get a lot more complex once you start bringing leverage into the equation, or you add features such as optimization, combining strategies, etc.</p><p>Implementing your own backtester for sure will give you a nitty gritty understanding of how backtesting and trading strategies work, but it takes time to create something valuable and well tested. Hence, If you don’t feel like implementing a backtester from scratch, I recommend you to use <a href="https://pypi.org/project/stratestic/"><strong>stratestic</strong></a>, a library I’ve been developing, showcasing all the above mentioned features and regularly updated with more.</p><p>Hope you had fun with this!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=030b968f6e8d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing GeneAl: a Genetic Algorithm Python Library]]></title>
            <link>https://medium.com/data-science/introducing-geneal-a-genetic-algorithm-python-library-db69abfc212c?source=rss-dccf9e76f185------2</link>
            <guid isPermaLink="false">https://medium.com/p/db69abfc212c</guid>
            <category><![CDATA[genetic-algorithm]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[optimization]]></category>
            <dc:creator><![CDATA[Diogo Matos Chaves]]></dc:creator>
            <pubDate>Wed, 24 Jun 2020 17:07:08 GMT</pubDate>
            <atom:updated>2020-06-24T23:37:19.516Z</atom:updated>
            <content:encoded><![CDATA[<h4>In this post, I’ll introduce <a href="https://github.com/diogomatoschaves/geneal">GeneAl</a>, a python library for solving optimisation problems with genetic algorithms (GA). And in the process, we’ll get to know the theory behind them and see how they work under the hood with python examples.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h8Xtpd1mp2QHvmWrnP28VQ.jpeg" /><figcaption>Image by <a href="https://pixabay.com/users/qimono-1962238/?utm_source=link-attribution&amp;amp;utm_medium=referral&amp;amp;utm_campaign=image&amp;amp;utm_content=1811955">Arek Socha</a> via <a href="https://pixabay.com/?utm_source=link-attribution&amp;amp;utm_medium=referral&amp;amp;utm_campaign=image&amp;amp;utm_content=1811955">Pixabay</a></figcaption></figure><p>Genetic algorithms (GA) are an optimization and search technique based on the principles of genetics and natural selection, in essence mimicking the natural evolution process that we observe in life. Their general principle is based on the concept of having an <strong>initial</strong> population composed of several individuals — with each representing a particular solution to the problem — and allow it to evolve to a state that maximizes its overall <strong>fitness</strong>, using three main operators: <strong>selection</strong>, <strong>crossover</strong> and <strong>mutation</strong>. We’ll look into these aspects a bit more in detail below.</p><p>Genetic Algorithms are nothing short of fantastic, as they can be applied to many kinds of optimization problems and find solutions to complex functions for which we do not have a mathematical expression. This comes at a cost of computational complexity though, as, for large populations, we’ll have to evaluate the fitness of all individuals at every generation. If the fitness function is expensive, the algorithm run will be slow.</p><p>GA’s can be divided into <em>Binary</em> and <em>Continuous</em>, depending on the type of problem we’re optimizing for. Potentially all problems could be broken down as having their variables (<em>genes)</em> represented by binary strings, but in general, if the input space is real-valued, it makes more sense to use a <em>continuous</em> GA.</p><p>As there are fewer examples for continuous GA out there, the examples shown here will be for that version of GA.</p><p><strong>Initialization</strong></p><p>The search starts with a random population of <em>N </em>individuals. Each of those individuals corresponds to a <em>chromosome</em>, which encodes a sequence of genes representing a particular solution to the problem we’re trying to optimize for. Depending on the problem at hand, the genes representing the solution could be bits (0’s and 1’s) or continuous (real valued). An example of a real-valued <em>chromosome </em>representing a solution to a given problem with 9 variables (<em>genes</em>) is shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3X7QtogVmtWnguQMsYpPTg.png" /><figcaption>Example of an individual’s chromosome</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e7eb8836d5467373d135957e27906d48/href">https://medium.com/media/e7eb8836d5467373d135957e27906d48/href</a></iframe><p><strong>Fitness</strong></p><p>The <em>fitness </em>of each individual<em> </em>is what defines what we are optimizing for, so that, given a <em>chromosome</em> encoding a specific solution to a problem, its fitness will correspond to how well that particular individual fares as a solution to the problem. Therefore, the higher its fitness value, the more optimal that solution is.</p><p>After all, individuals have their fitness score calculated, they are sorted, so that the fittest individuals can be selected for crossover.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f460518d5b58ebef6b962b5c215ab7e9/href">https://medium.com/media/f460518d5b58ebef6b962b5c215ab7e9/href</a></iframe><p><strong>Selection</strong></p><p>Selection is the process by which a certain proportion of individuals are selected for mating between each other and create new offsprings. Just like in real-life natural selection, individuals that are fitter have higher chances of surviving, and therefore, of passing on their genes to the next generation. Though versions with more individuals exist, usually the selection process matches two individuals, creating pairs of individuals. There are four main strategies:</p><p><em>pairing</em>: This is perhaps the most straightforward strategy, as it simply consists of pairing the top fittest chromosomes two-by-two (pairing odd rows with even ones).</p><p><em>random</em>: This strategy consists of randomly selecting individuals from the mating pool.</p><p><em>roulette wheel</em>: This strategy also follows a random principle, but fitter individuals have higher probabilities of being selected.</p><p><em>tournament</em>: With this strategy, the algorithm first selects a few individuals as candidates (usually 3), and then selects the fittest individual. This option has the advantage that it does not require the individuals to be sorted by fitness first.</p><p>A python implementation for the <em>roulette wheel</em> strategy is shown on the snippet below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/59a604dfb393b833851eed97ab3dca10/href">https://medium.com/media/59a604dfb393b833851eed97ab3dca10/href</a></iframe><p><strong>Crossover</strong></p><p>This is the step where new offsprings are generated, which will then replace the least fit individuals in the population. The idea behind crossing over individuals is that, by combining different genes, we might produce even fitter individuals, which will be better solutions to our problem. Or not, and in that case, those solutions won’t survive to the next generations.</p><p>In order to perform the actual crossover, each of the pairs coming from the selection step are combined to produce two new individuals each, which will both have genetic material from each of the parents. There are several different strategies for performing the crossover, so for brevity, we’ll only discuss one of them.</p><p>Supposing we have a problem defined by 9 variables, if we have 2 parents and we choose randomly the crossover gene as index 3, then each of the offsprings will be a combination of each parent, as shown in the diagram below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*76lEcLZ2ayCxvaKg-wXJbg.png" /><figcaption>Diagram showing how parents are crossed over to generate new offspring</figcaption></figure><p>The crossover gene of each offspring is calculated according to the rule given by:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-ZWo8Cctykxaa99ASX9MNQ.png" /><figcaption>Equation for calculating new crossover genes</figcaption></figure><p>Where <em>β</em> will be a random number between 0 and 1. The python code for the crossover is given below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/73eaadf1ef3a27ae7eb84784abee2139/href">https://medium.com/media/73eaadf1ef3a27ae7eb84784abee2139/href</a></iframe><p><strong>Mutation</strong></p><p>Mutation is the process by which we introduce new genetic material in the population, allowing the algorithm to search a larger space. If it were not for mutation, the existing genetic material diversity in a population would not increase, and, due to some individuals “dying” between generations, would actually be reduced, with individuals tending to become very similar quite fast.</p><p>In terms of the optimization problem, this means that without new genetic material the algorithm can converge to local optima before it explores an enough large size of the input space to make sure that we can reach the global optimum. Therefore, mutation plays a big role in maintaining diversity in the population and allowing it to evolve to <em>fitter</em> solutions to the problem.</p><p>The most simple way we can do this is, given a certain <em>mutation rate</em>, to randomly choose some individuals and some genes and assign a new random number to those positions. This is exemplified in the diagram and code snippet below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MWhlDik-dgaCUwPyxp739A.png" /><figcaption>Mutation of two genes in an individual</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a9dd8b01780b07c0acb2762a7091fcdd/href">https://medium.com/media/a9dd8b01780b07c0acb2762a7091fcdd/href</a></iframe><p><strong>Solver</strong></p><p>Now it’s time to tie it all together. Using the operators that we defined above, the algorithm can now solve the problem, with the actual main cycle of the algorithm being implemented in just a few lines of code. The flowchart of the algorithm, as well as an example implementation in python are shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GMTie35Nnnxciazxek66Qw.png" /><figcaption>Flowchart of the Genetic Algorithm</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/55f14121bf939f82c8f4da8cd26a5859/href">https://medium.com/media/55f14121bf939f82c8f4da8cd26a5859/href</a></iframe><h3>Introducing GeneAl</h3><p><a href="https://github.com/diogomatoschaves/geneal">GeneAl</a> is a python library implementing Genetic Algorithms, which can be used and adapted to solve many optimization problems. One can use the provided out-of-the-box solver classes — <strong>BinaryGenAlgSolver</strong> and <strong>ContinuousGenAlgSolver</strong> — , or create a custom class which inherits from one of these, and implements methods that override the built-in ones. It also has support for solving the (in)famous <strong>Travelling Salesman Problem</strong>.</p><p>For brevity, we’ll only see how to use the continuous version — keeping in line with this post — , but for more details, check out the README of this project.</p><p>The first thing would be to install the package, which can be made through pip, as such:</p><pre>pip install geneal </pre><p>After the installation completes, one is ready to use it. So let’s see how we can use the <strong>ContinuousGenAlgSolver </strong>class.</p><p>As a bare minimum, the class requires the user to provide the number of genes present in the problem, as well as to provide the custom defined fitness function. For convenience, the library provides some default fitness functions that can be used for testing.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9b1859d3c81b5d40adbb15c7bb02130e/href">https://medium.com/media/9b1859d3c81b5d40adbb15c7bb02130e/href</a></iframe><p>With the initialization done above, the class will solve the problem using default values for all the parameters. If we wish to have more control over the algorithm run, we will want to adjust these, and that can be done as shown below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b221eb86371a45ed53d5eb417b013bca/href">https://medium.com/media/b221eb86371a45ed53d5eb417b013bca/href</a></iframe><p>Finally, this class allows the user to specify the type of problem — if the possible values are integers or floats — , as well as the variables’ limits, in order to limit the search space.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/25d0c65faaca74b9f12cfcedbc4f2d96/href">https://medium.com/media/25d0c65faaca74b9f12cfcedbc4f2d96/href</a></iframe><p>This completes this short introduction to the library. If you want to know more, check out the <a href="https://github.com/diogomatoschaves/geneal">GitHub repository</a>, which has more information :)</p><p>Thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=db69abfc212c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/introducing-geneal-a-genetic-algorithm-python-library-db69abfc212c">Introducing GeneAl: a Genetic Algorithm Python Library</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Estimating the Population of Arbitrarily Chosen Areas]]></title>
            <link>https://medium.com/data-science/estimating-the-population-of-arbitrarily-chosen-areas-533e81c2104e?source=rss-dccf9e76f185------2</link>
            <guid isPermaLink="false">https://medium.com/p/533e81c2104e</guid>
            <category><![CDATA[geospatial]]></category>
            <category><![CDATA[openstreetmap]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[population]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Diogo Matos Chaves]]></dc:creator>
            <pubDate>Thu, 14 May 2020 18:58:11 GMT</pubDate>
            <atom:updated>2020-05-19T18:34:35.581Z</atom:updated>
            <content:encoded><![CDATA[<h4>Using only OpenStreetMap (OSM) features</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tq4NJv_GXgjvS1keSlI9hg.png" /><figcaption>Examples of population estimations using the model described in this post</figcaption></figure><h3>Introduction</h3><p>Information about the size of a given place or country’s population has been sought after since ancient times. Old empires and states would go to great lengths to roughly estimate the numbers of inhabitants in their realms, so as to ascertain the military forces which could be recruited or the numbers of persons from whom taxes could be collected for example.</p><p>In more recent times, methods for estimating the population have improved greatly and became more refined, with most governments nowadays performing periodic census enumerations, which are then supplemented by records of births, deaths and migrations, as well as schools attendance, taxation lists, voting registers, etc. Despite this, many countries and places still lack accurate information about their populations, and even in places where that information is known, there is a lack of publicly available information at a fine-grained level, where usually only data at a relatively large administrative level (eg. village, council, etc) is known or accessible.</p><p>With that in mind, in this post we will examine a versatile method for estimating the population of any given area in the world, using only features from one of the largest and most accessible public datasets in the world: <a href="https://www.openstreetmap.org/#map=18/51.48934/-0.13251&amp;layers=D"><strong>OpenStreetMap</strong></a> <strong>(OSM)</strong>.</p><p>OSM is a massive collaborative map of the world built and maintained mostly by volunteers, and it has a wealth of information about all sorts of things, such as <strong>roads</strong> (motorway, primary, residential, etc), <strong>buildings</strong> (accommodation, commercial, religious, etc), <strong>amenities</strong> (restaurants, bars, entertainment, transportation, etc), <strong>shops</strong>, <strong>leisure areas</strong>, <strong>sports places</strong>, and a wealth of other features which can be extracted either by using an API (such as the <a href="https://dev.overpass-api.de/overpass-doc/en/">overpass</a> API), or, as was the case in this project, by extracting them directly from an OSM file using a tool such as <a href="https://osmcode.org/pyosmium/">PyOsmium</a>.</p><p>The population training data we’ve used in this project came (indirectly) from Facebook’s <a href="https://research.fb.com/downloads/high-resolution-settlement-layer-hrsl/">High Resolution Settlement Layer</a>, which estimates the population of a given area using satellite imagery. Although perhaps said model can achieve higher accuracy, performing predictions for new user-defined areas would probably mean having access to the image processing algorithm used in the first place, which is very likely not publicly available, let alone the probably high computing resources required for that task.</p><p>Therefore, there are obvious advantages on having a model that can make the same kind of predictions with a relatively small set of features, and based solely on publicly available data (OSM). This <a href="https://github.com/diogomatoschaves/osm-population-predictor">project’s</a> main goal was to provide such a model as an end result.</p><h3>Methodology</h3><p>This <a href="https://github.com/diogomatoschaves/osm-population-predictor">project</a> consisted of three distinct but closely related major steps: <em>data collection, data analysis </em>and<em> model training</em>.</p><p><em>Note: as you’ll see, there were two somewhat separate workflows performed, one with a more basic set of OSM features (which was part of the original sourced dataset), and another one with an extended set of OSM features (which were added to the original data). The latter will be the main focus throughout this present analysis, but we’ll talk about the former as well.</em></p><p><strong>Data Collection</strong></p><p>We started with an existing dataset <strong>[1]</strong>, which consisted of multiple hexagons with a 400m edge spanning the entire earth (minus some shadow areas). These contained aggregated data from Facebook’s <a href="https://research.fb.com/downloads/high-resolution-settlement-layer-hrsl/">High Resolution Settlement Layer</a> regarding population data, combined with some OSM features, such as the number of OSM objects, number of buildings, number of OSM users who mapped the area, total length of roads, among others.</p><p>From that huge dataset, a smaller subset area encompassing Great Britain (England, Scotland and Wales) was selected and filtered for this model, resulting in a dataset consisting of ~300k hexagons. This area was chosen due to its relatively large spread between high density urban areas and lower density rural areas, as well as being relatively well mapped. A snapshot of the resulting dataset can be visualised below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N0CPnFMZr7mYTq0O5FtyiA.png" /><figcaption>Red areas indicate areas of higher population</figcaption></figure><p>As the OSM features present in this data proved insufficient for explaining enough variance (as we’ll see in the <strong>results</strong> section), a considerable effort was put into building an <a href="https://github.com/diogomatoschaves/osm-feature-extractor">application</a> that, given a base OSM file and a GeoJSON polygon(s) defining the area(s) of study, can automatically retrieve several features and map those to the provided polygon(s).</p><p>The OSM data model is such that there are <strong>nodes</strong>, <strong>ways</strong> and <strong>areas</strong>. Nodes<strong> </strong>are the basic building blocs of all other more complex structures, and when they have associated tags, they are included as a <em>count. </em>Ways represent a collection of nodes representing lines, and are used to map roads for example. As such, they are mapped as a <em>length. </em>Finally, areas are a collection of nodes starting and ending in the same location, and are used to map buildings or any other structure, and hence, are mapped as an <em>area.</em></p><p>Thus, in the end this equated to roughly 120 extracted features, which basically correspond to most of the information that can be extracted from an OSM map. As an example, in the figure below all purple tags would correspond to shops, which would then be divided by category (such as food, clothing, beauty, etc), whereas the dark yellow ones would equate to amenities, which again would be split by category (sustenance, entertainment, transportation, etc).</p><p><em>For an in-depth explanation of the rationale behind the feature collection, visit the corresponding </em><a href="https://github.com/diogomatoschaves/osm-feature-extractor"><em>project</em></a><em> and the </em><a href="https://wiki.openstreetmap.org/wiki/Map_Features"><em>OSM wiki</em></a><em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yyq4gtJp8uQRmr9_lFgDNQ.png" /><figcaption>Example of OSM tags</figcaption></figure><p>After collecting the data, some data cleaning was performed, which included removing columns with only one value and removing outliers. As there were no categorical variables, there was no need for one-hot encoding. Following this, the data was ready to be analysed and for a model to be built from.</p><p><strong>Data Analysis</strong></p><p>After the data collection and preparation step, an exploratory analysis was performed, in order to get a sense of the data and the relation between the target variable <em>population </em>and the other independent variables. Since we were building a regression model, one of the most important characteristics of the data is the correlation between the dependent (target) and the independent (features) variables. Perhaps unsurprisingly, it was found that the variables highway_residential_length<em>, </em>landuse_residential_area and<em> </em>highway_bus_stop_count had the highest correlations, as these variables are associated with residential areas, and thus, populated places. On the opposite end, the variables with highest negative correlation were natural_area and landuse_agricultural_area, which was again expected, as these type of features are associated with low population density. The figure below shows the full correlation results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*RzVbI_oi26Ueh5to9tjG4w.png" /><figcaption>Correlation between all features and population</figcaption></figure><p>Taking a closer look at the relation between some of these variables<em>, </em>a strong correlation between highway_residential_length<em> </em>and<em> </em>population<em> </em>is observed<em>, </em>but<em> </em>weaker between building_accommodation_count and<em> </em>population<em>. </em>For the variables with negative correlation, we can observe that the relation is not linear but parabolic, which gives us already an indication that perhaps polynomial terms of the variables will provide a better fit to the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*opE_i3drYisacjpDQUMm9Q.png" /></figure><p>Visualising the data spatially also brings an interesting perspective, confirming the correlations we see in the scatter plots. As expected, urban centres have higher concentration of residential roads, railways and buildings, whereas the areas with higher agricultural land use or areas marked as natural have higher concentrations away from urban centres.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*mlXOoZDs67-Cz6wrd6ePYA.jpeg" /><figcaption>Spatial distribution of selected features. Redder areas indicate higher values</figcaption></figure><p><strong>Model Training</strong></p><p>After analysing the data, a machine learning pipeline was put together in order to perform some preprocessing and transformations on the data and then fit the training data into the model, with a 20–80% test – training split.</p><p>One of the transformations in the pipeline was to normalise all the features and target variable by the <em>area</em> corresponding to each training example, as that proved a better solution for making the results more generalisable to larger areas. Below the steps of the pipeline are briefly described:</p><ol><li><em>calculate area</em>: This step calculates the <em>area</em> of the input data if it is not calculated already.</li><li><em>feature selector</em>: This step selects the model features + the <em>area</em>, as the latter will be used in the next step.</li><li><em>area normaliser</em>: This step normalises each feature by the area corresponding to that training example.</li><li><em>remove columns</em>: This step removes unrequired columns; in this case the <em>area.</em></li><li><em>feature mapping</em>: This step adds features which are the polynomial terms of the input features, up to a specified degree (if it’s 1, then no polynomial terms are added). There’s the option to include only the exponent terms or also interaction terms between variables.</li><li><em>feature scaling</em>: This step makes the features values all in the same scale, by subtracting the mean and dividing by the standard deviation.</li><li><em>regressor</em>: This step is when the regression algorithm is actually applied, and the data is fitted into the model.</li></ol><h3>Results</h3><p>As mentioned before, two different models were built: a first approach with the sourced dataset containing few OSM features, and a second one using the dataset augmented with more features using a custom-built <a href="https://github.com/diogomatoschaves/osm-feature-extractor">application</a>. For testing the performance of the models two different metrics were chosen: the <em>r²</em> <em>score</em>, which gives us a measure of the ratio of the variance in the data explained by the model, and the <em>mean absolute error</em>, which gives us an indication of how well the model is doing in a measurable quantity — in our case <em>inhabitants / km².</em></p><p>For the first approach at building the model, a combination of linear (<strong>Lasso</strong> and <strong>Ridge</strong>) and ensemble (<strong>Random</strong> <strong>Forest</strong> and <strong>Gradient</strong> <strong>Boosting</strong>) regressors were tried. After performing hyperparameter tuning, it was seen that the best results were achieved using a combination of up to 3rd degree polynomial terms and a <strong>Gradient</strong> <strong>Boosting </strong>regressor, with a corresponding <em>r²</em> <em>score</em> of ~0.67 and a <em>mean absolute error</em> of ~158 <em>inhabitants / km². </em>In order to get a glimpse of how we were faring in terms of bias-variance, the learning curves were analysed, and as suspected we could observe that, even using polynomial terms to build more complex relations between the variables the model was suffering from high bias, where both the training and cross validation errors were converging at a relatively high value. Therefore, in order to build a better performing model, we would not need to add more training data, but to include a larger set of features, one that could explain more of the variance in the target variable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j-pJ-UuiC8wDQwRLZcNTCg.png" /></figure><p>Thus, motivated by these learnings, more data was acquired (as detailed in <em>Data Collection</em>), making us ready to apply the machine learning pipeline to this enriched dataset. As was the case in the previous approach, again a combination of linear (<strong>Lasso</strong> and <strong>Ridge</strong>) and ensemble (<strong>Random</strong> <strong>Forest</strong> and <strong>Gradient</strong> <strong>Boosting</strong>) regressors were tried. Following a hyperparameter optimisation, a combination of up to 2nd degree polynomial terms and a <strong>Gradient</strong> <strong>Boosting </strong>regressor achieved once more the best results, with a <em>r²</em> <em>score</em> of ~0.9 and a <em>mean absolute error</em> of ~91.8 <em>inhabitants / km². </em>This was a great increase from the first approach! This proved the effort of acquiring more features worth it. Having a look again at the learning curves, we see that the training and cross validation errors are still converging (indicating a certain bias in the model), but to a sizeably lower value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xWI6sWP-PEzLqr18VPLFkg.png" /></figure><p>In this case though, a comparison of actual and estimated population values was performed for some places / cities*. When that was done, it was seen that the best results (lower average error between actual and predicted) were achieved using a <strong>Lasso</strong> regressor with a regularisation value of 1, indicating that we are perhaps overfitting on the data somehow when using the <strong>Gradient Boosting</strong> regressor.</p><p>*<em>In order to calculate the population for custom defined places, one first needs to use </em><a href="https://github.com/diogomatoschaves/osm-feature-extractor"><em>osm-feature-extractor</em></a><em> and map the osm features of the provided area. Then, this intermediate step can be fed into the model, which will output the population density (inhabitants / km²). Finally, by multiplying the result by the area one gets the total population.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j8U7vJSbpdkONUE06qzdtg.jpeg" /><figcaption>Comparison of model results between Lasso (left) and Gradient Boosting (right) models for several places</figcaption></figure><p>Examining further the models, the feature importance of that model shows that indeed the most important feature by far is highway_residential_length, as opposed to the more distributed coefficient values of the Lasso regressor. This points to the fact that the <strong>Lasso</strong> regressor model is likely better at generalising to unseen cases as it takes into account more features, and hence less prone to overfitting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qXTZSkq9twvGHt1hz4SDLQ.jpeg" /><figcaption>Coefficients / features importance comparison between Lasso (left) and Gradient Boosting (right) models</figcaption></figure><h3>Conclusion</h3><p>In this project we set out to build a model that could accurately predict the population of a given area, based solely on features extracted from OSM data. For this we started first with a dataset taken from <strong>[1]</strong>, which was characterised by having few features, and thus, resulting in a high bias model. Based on this, we extended the base data with extracted features taken from OSM files using a custom-built <a href="https://github.com/diogomatoschaves/osm-feature-extractor">application</a>.</p><p>We saw that the addition of more features, although risking adding some noise to the model, provided a much improved predictive capacity, and allowed us to greatly reduce the mean absolute error. Also, with this approach we have a way of deterministically extracting all the required model input features from OSM files, as opposed to just using the initial data, which we did not have out of the box a way of determining the features for new areas without having to perform some sort of reverse engineering.</p><p>We saw that, even though the <strong>Gradient Boosting</strong> regressor model performed better on the test data, the <strong>Lasso</strong> regressor performed better on actual population estimates for several places / cities, achieving a maximum difference of 26%.</p><p>In this model, we used data points that spread across the UK, which spans areas of very high density (such as London), as well as areas of less density and of farmland. This should enable the model to extend well to most cases, although probably not so well if we were to use it in areas of very high population density. Also, the model is dependent on how well mapped the area in the study is. In most of the western world, the OSM extracts are quite well mapped, but it’s less so the case in less developed places — although the main centres of population, in general, tend to be well mapped.</p><p>Despite these facts, we saw that the model estimated the population of actual places and cities with relatively low error, and we now have a model that can make rough estimates on the population of any given area based solely on its OSM features.</p><p><em>To see the analysis in more detail check out the </em><a href="https://github.com/diogomatoschaves/osm-population-predictor"><em>Github repository</em></a><em>.</em></p><h3>References</h3><p><strong>[1] - </strong><a href="https://data.humdata.org/dataset/kontur-population-dataset">Kontur Population: Global Population Density for 400m H3 Hexagons</a></p><p><strong>[2] - </strong><a href="http://ad-publications.informatik.uni-freiburg.de/GIS_population_BSW_2015.pdf">Bast, Hannah, 2015. Fine-Grained Population Estimation</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=533e81c2104e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/estimating-the-population-of-arbitrarily-chosen-areas-533e81c2104e">Estimating the Population of Arbitrarily Chosen Areas</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Berlin House Rental Market Analysis]]></title>
            <link>https://medium.com/data-science/berlin-house-rental-market-analysis-18dfc0c2ca9?source=rss-dccf9e76f185------2</link>
            <guid isPermaLink="false">https://medium.com/p/18dfc0c2ca9</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[berlin]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[house-prices]]></category>
            <dc:creator><![CDATA[Diogo Matos Chaves]]></dc:creator>
            <pubDate>Tue, 17 Mar 2020 22:41:01 GMT</pubDate>
            <atom:updated>2020-03-19T02:25:25.725Z</atom:updated>
            <content:encoded><![CDATA[<h4>Am I paying a fair price?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*auUACR0hEleBd7rD0wb2Mg.png" /></figure><h3><strong>Introduction</strong></h3><p>Not so long ago, Berlin used to be a place associated with low living costs, attracting many people looking for a high quality of life at cheap prices. However, perhaps due to that popularity and other macro-economic shifts, those days became a thing of the past.</p><p>Ask anyone living in Berlin at the moment and they will tell you how difficult it is to get a house rental contract, not only due to the scarcity of offers and plenty of demand for apartments, but also due to the soaring prices, even despite state regulation efforts.</p><p>With the above setting the tone and having found myself in the situation of looking for a new place to live, I decided to use a data-driven approach to help me with my search and give me the confidence of making good decisions based on data. Thus, in this post I’ll describe some of my findings in this process.</p><p>For a period of a few months I collected some data on the market from the website <a href="http://www.immobilienscout24.de/">immobilienscout24</a> for certain areas of Berlin — some areas were not included as I was not looking to live there. With the data I collected, I was interested in answering several questions to help me in my search, but the main ones were:</p><ul><li>What are the most and less expensive areas?</li><li>What is the most popular time to publish a new listing?</li><li>Am I paying a fair price?</li><li>What makes some listings more attractive than others?</li></ul><p>I’ll try to answer these questions in this post, so read on!</p><h4><strong>What are the most and less expensive areas?</strong></h4><p>I wanted to see where my budget allowed me to live in and therefore knowing the average asking price for a given area was the first step.</p><p><em>It’s important to clarify what </em><strong><em>warm price</em></strong><em> means: In Germany house prices are usually split in cold and warm prices, which correspond to the price before and after heating costs, as well as upkeep, garbage collection and other maintenance costs. Therefore, the rental cost we’re most interested in is the warm price.</em></p><p>The bar plot below shows in descending order the most expensive areas in Berlin from this dataset. We can see that somewhat expectedly, the most expensive areas are <strong>Mitte, Tiergarten, Friedrichshain, Prenzlauer Berg and Kreuzberg</strong>, which are typically more emblematic parts of the city and thus more sought after. On the other end, we have <strong>Wedding, Schöneberg, Lichtenberg and Treptow,</strong> which can be considered somewhat outside the city center and therefore less expensive as expected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*wehL7Do9A_MSuAxOKzEmqw.png" /></figure><p><strong>What is the most popular time to publish a new listing?</strong></p><p>As the market is very competitive and listings disappear very quickly, I was interested in knowing what was the most popular time for new listings to appear, so that I could be more ready to respond fast.</p><p>On the plot below we can see the distribution of times listings were posted on. It was observed that the most popular times to post new listings were in the morning, with a peak just before lunch time (12:00–13:00).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*8XPbhkMdA-pXxFojp45ofg.png" /></figure><p><strong>Am I paying a fair price?</strong></p><p>Perhaps the most interesting question of all, I wanted to know if a listing was asking a fair price, so that I could have a base for my decision to apply for an apartment or, even more importantly, to take an offer.</p><p>To accomplish this I built a Linear Regression model to predict the final (warm) price, using as features whether an apartment has a balcony, garden or built-in kitchen, if it is new or recently renovated, if the offer is from a private individual or real estate agency, the total living area, the number of rooms or the district where it is located, among other variables.</p><p>The results are shown below, where the relative strength of the indicators is plotted (both positive and negative) in descending order.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ciUizINkYFAAEcFH_qH5eg.png" /></figure><p>It can be concluded that the strongest indicators are if an apartment is located in Wedding or Schöneberg, which affect negatively the price. Somewhat unsurprisingly, an apartment having a built-in kitchen or garden is strongly correlated with higher prices, but perhaps more surprisingly, the number of rooms has a negative coefficient, meaning that the larger the number of rooms, the lower the final price.</p><p>Even though the coefficient for living space is relatively low compared to the others, one has to take into account that this is a continuous variable with a much higher variation than the boolean variables (which vary between 0 and 1). Therefore, for every unit increase in living area (1 m2), the price is expected to increase around 6 EUR.</p><p>With this model, I became more confident in my decisions and was able to gauge if the asking price of an apartment was fair or not.</p><p><strong>What makes some listings more attractive than others?</strong></p><p>As a bonus, I was also interested in knowing what characteristics of a listing are more important to know if a listing will be attractive or not. For that I used the time a listing stays online as a metric, with the lower the online time the higher the attractiveness and interest.</p><p>To achieve this I ran a Linear Regression model to predict the time a listing stays online, using as features the same ones used in the previous linear model, plus the warm price and the normalised difference between warm and cold prices. The results of the model are shown in the plot below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6CtFgoTMvi_G1BKRkS7vbw.png" /></figure><p>In this case, if a coefficient is positive, then it is associated with more time online, and vice versa. It can be concluded for example that if a listing is a new build (or has been renovated), has more rooms or larger living space, has a balcony or built-in kitchen, or is located in the neighbourhoods Schöneberg, Tiergarten, Mitte, Treptow or Kreuzberg, then it stays less time online (receives more interest and goes to someone more quickly). On the other hand, a listing stays longer online if the price difference or warm price is higher, or if it is located in the perhaps less attractive neighbourhoods of Lichtenberg, Wedding or Neukölln. Somewhat surprisingly, a listing having a garden is associated with more time online.</p><h3>Conclusion</h3><p>In this article, we looked into some insights given by analysing a dataset consisting of apartments published online for rent in Berlin. More specifically, we:</p><ul><li>Saw that the most expensive surveyed neighbourhood was <strong>Mitte</strong>, and the least one was <strong>Wedding</strong>.</li><li>Saw that the <strong>most popular time</strong> for publishing new listings was in the <strong>morning</strong>.</li><li>Observed that the <strong>most important predictors</strong> for the warm price were if a listing is not located in the neighbourhoods of <strong>Wedding</strong> or <strong>Schöneberg</strong>, and if it has a <strong>built-in kitchen </strong>or a<strong> garden.</strong></li><li>Concluded that the characteristics that are mostly associated with more interest in a listing are if it is <strong>new</strong>, has a <strong>built-in kitchen</strong> or a <strong>balcony</strong> as well as having <strong>more rooms and living space</strong>. As for the most sought after areas, we saw that the most attractive neighbourhoods were <strong>Schöneberg</strong>, <strong>Tiergarten</strong>, <strong>Mitte</strong>, <strong>Treptow</strong> and <strong>Kreuzberg.</strong></li></ul><p>Hope this can help you in your future search for an apartment in Berlin! To see the analysis in more detail check out the <a href="https://github.com/diogomatoschaves/berlin-house-prices-analysis">Github repo</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=18dfc0c2ca9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/berlin-house-rental-market-analysis-18dfc0c2ca9">Berlin House Rental Market Analysis</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>