<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Opendoor on Medium]]></title>
        <description><![CDATA[Stories by Opendoor on Medium]]></description>
        <link>https://medium.com/@joseph.gomez?source=rss-efdb2be1f68d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*nh0uDoJUX3GVKV8wmLYm7Q.jpeg</url>
            <title>Stories by Opendoor on Medium</title>
            <link>https://medium.com/@joseph.gomez?source=rss-efdb2be1f68d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 05:40:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@joseph.gomez/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Five of our favorite talks from RailsConf 2019]]></title>
            <link>https://medium.com/opendoor-labs/five-of-our-favorite-talks-from-railsconf-2019-5bd276d61dd3?source=rss-efdb2be1f68d------2</link>
            <guid isPermaLink="false">https://medium.com/p/5bd276d61dd3</guid>
            <category><![CDATA[ruby-on-rails]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[real-estate]]></category>
            <dc:creator><![CDATA[Opendoor]]></dc:creator>
            <pubDate>Tue, 03 Dec 2019 22:59:51 GMT</pubDate>
            <atom:updated>2020-02-05T16:05:14.983Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SagfZFlGmeY4RGUGsu-7Dg.png" /><figcaption>Members of the Opendoor Engineering at RailsConf 2019</figcaption></figure><p><em>By: Vijay Edupuganti</em></p><p>At <a href="https://www.opendoor.com/">Opendoor</a>, we use Rails extensively for our business logic and tooling, in addition to other services written in Python and Go. This past May, six members of our engineering team traveled to Minneapolis for <a href="https://railsconf.com/">RailsConf</a>.</p><p>While there were a ton of great topics covered, here are five of our favorite talks from the conference.</p><h4><strong>1. Opening Keynote — David Heinemeier Hansson</strong></h4><p><a href="https://youtu.be/VBwWbFpkltg">https://youtu.be/VBwWbFpkltg</a></p><p>As with every RailsConf, the opening keynote was given by none other than DHH. To kick things off, he lightheartedly remarked that he had prepared for the talk for close to a year. Safe to say, he did not disappoint as the hour-long talk was filled with interesting insights on topics ranging from stoicism, to how the concept of<a href="https://en.wikipedia.org/wiki/Jubilee_(biblical)"> Jubilee</a> applies to Rails contributors. I had never really considered how Bill Gates and Richard Stallman were similar in approach, or how I should feel about tech companies not always contributing back to the framework.</p><p>One of his most interesting points was that there is often an adverse relationship between developer happiness and consumerism. He used many examples to illustrate this.</p><p>Open source maintainers can be pressured to be subservient to the people they are supporting, trying to please rather than work on what’s most interesting. Engineers often make technology choices for scientific and business reasons, considering performance and scale, but not always humanistic reasons like what language is most enjoyable to work in. Overly grandiose mission statements are created sometimes to the detriment of work-life balance.</p><p>At the very least, watch the talk to find out why no one at a file cabinet company talks of changing the world! It’s well worth a listen.</p><h4><strong>2. The Elusive Attribute — Chris Salzberg</strong></h4><p><a href="https://youtu.be/PNNrmNTQx2s">https://youtu.be/PNNrmNTQx2s</a></p><p>foo.bar_changed?</p><p>How exactly does this line of code work? I <em>kind of </em>knew. Most Rails developers have an idea of what an attribute is, but can largely trust Rails metaprogramming to make the correct decision. That wasn’t good enough for Chris though.</p><p>Going into the talk, I assumed an attribute was essentially synonymous with a database column or a method defined on a certain model. Using introspection, Chris took us through the fundamentals of the attribute methods that Rails generates for us. From there, he showed us the exact process Rails uses to determine an attribute’s value: sometimes relying on a generated attribute method, sometimes relying on a database column, and sometimes even relying on the record returned from the database.</p><p>The talk was an excellent technical deep dive and an opportunity for Chris to pitch cleaning up some of the attribute method matching logic under the hood.</p><p>Bonus: A<a href="https://github.com/rails/rails/pull/36005"> PR</a> that Chris referenced in the talk was recently merged into Rails!</p><h4><strong>3. The Selfish Programmer — Justin Searls</strong></h4><p><a href="https://youtu.be/k5thkp4ZXSI">https://youtu.be/k5thkp4ZXSI</a></p><p>Antisocial. Egotistical. Irresponsible. Three stereotypes of a solo programmer right?</p><p>Justin Searls seems to think so. Specifically, he argues that these characteristics are actually ideal when working alone. This talk was filled with useful and interesting techniques to drive solo projects through to the finish line. Debating how to handle authentication in your Rails app? Consider implementing it from scratch rather than using devise. Seeing the same code repeat itself in multiple places? That’s fine, punt on refactoring. And when in doubt, always use Heroku.</p><p>A must watch talk for those with side projects and wavering motivations, or those with aspirations to work on something on their own. Also, our favorite slide deck of the conference!</p><h4><strong>4. Code Spelunking — Jordan Raine</strong></h4><p><a href="https://youtu.be/LiyLXklIQHc">https://youtu.be/LiyLXklIQHc</a></p><p>A common complaint amongst those who work with Rails codebases revolves around “Rails magic.” There’s nothing more frustrating than furiously searching through StackOverflow and poring over unclear logs to debug an issue pertaining to something you didn’t even write. Jordan Raine’s talk delved into techniques to navigate through these moments, especially when the issue involves digging through core Rails source code.</p><p>In his talk, Jordan cited a study in which researchers found that only 5% of an engineer’s workday was spent writing code. On the flip side, over 80% of time was spent comprehending and navigating through code, from understanding the scope of what to work on, to actually finding answers to questions. As a result, when trying to level up and advance as an engineer, improving the ability to find the correct answer is more important than anything else.</p><p>Give it a listen to discover some nifty methods to get unstuck.</p><h4><strong>5. Fixing Flaky Tests Like a Detective — Sonja Peterson</strong></h4><p><a href="https://youtu.be/qTyoMg_rmrQ">https://youtu.be/qTyoMg_rmrQ</a></p><p>While tests should be deterministic, it’s crazy how often a build pipeline can look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PgAlR-w6Blalq2Q4" /></figure><p>The picture above can be a common one in large Rails codebases. Often, fixing these flaky tests consists of blindly re-trying or going down many fruitless rabbit holes.</p><p>We enjoyed Sonja Peterson’s talk as it detailed a systematic approach for any engineer, of any level, to dive into fixing a flaky test. The approach outlined five common causes of brittle tests — async code, order dependency, time, unordered collections, and randomness — along with gems and techniques to easily avoid introducing them. A great talk for those interested in writing better tests or improving the quality of an existing test suite.</p><p><strong>About Us:</strong></p><p>Here at Opendoor, we are trying to simplify the home transaction. Selling a home has traditionally been a very stressful and time-intensive process. By selling directly to Opendoor, sellers are able to avoid the many hassles of listing, such as showings, buyer financing fall-through risk, and months of uncertainty. We also support the full customer journey from <a href="https://www.opendoor.com/">selling</a> to <a href="https://buywith.opendoor.com/">buying</a> to <a href="https://www.opendoor.com/w/trade-in">trading-in</a>.</p><p><em>Opendoor is hiring! Check out our</em><a href="https://www.opendoor.com/jobs"><em> jobs page</em></a><em> for open roles in data science, engineering, and more.</em></p><p>We operate in 21 markets and counting, including Minneapolis! During our visit, we were able to visit the local Opendoor operations office and even check out a few homes that we own.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BDJpEHKHsZ_yagl8" /></figure><p><strong>Acknowledgments:</strong></p><p>RailsConf 2019 was exceptionally well planned and we want to extend thanks to the organizers — Ruby Central, Inc. — and sponsors for putting the conference together. Additional props to Confreaks for the videos, all the speakers and incredible engineers we met over the week. <a href="https://www.youtube.com/playlist?list=PLE7tQUdRKcyaOq3HlRm9h_Q_WhWKqm5xc">See the full list of talks here</a>.</p><p><a href="https://www.opendoor.com/jobs">The simple way to sell or buy a home.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5bd276d61dd3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/opendoor-labs/five-of-our-favorite-talks-from-railsconf-2019-5bd276d61dd3">Five of our favorite talks from RailsConf 2019</a> was originally published in <a href="https://medium.com/opendoor-labs">Open House</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Taming missing features at serving time]]></title>
            <link>https://medium.com/opendoor-labs/taming-missing-features-at-serving-time-3e2d0d9ed674?source=rss-efdb2be1f68d------2</link>
            <guid isPermaLink="false">https://medium.com/p/3e2d0d9ed674</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[real-estate]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Opendoor]]></dc:creator>
            <pubDate>Thu, 03 Oct 2019 21:36:26 GMT</pubDate>
            <atom:updated>2020-02-05T16:10:41.009Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NHuKybh3Wlljzh8sgu9cQA.png" /></figure><p><em>By: Leo Pekelis</em></p><p><em>“We skated over the matter of where our data comes from in the first place, and what we plan to ultimately do with the outputs from our models.” — </em><a href="https://d2l.ai/"><em>D2L</em></a></p><p>At Opendoor, the Data Science team has built predictive models to power multiple business applications. In previous blog posts, we discussed <a href="https://medium.com/opendoor-labs/why-ensembling-works-the-intuition-behind-opendoors-home-pricing-d12c17c24f96">valuing homes</a> in order to make competitive offers to home sellers, as well as <a href="https://medium.com/opendoor-labs/liquidity-modeling-in-real-estate-using-survival-analysis-cfd0ce01522a">forecasting how long our inventory takes to sell </a>to accurately estimate liquidity risk.</p><p>Transience in real estate markets is a challenge to all such models. If prices of homes have increased in the past month, then a model trained with last month’s list prices as a feature has difficultly operating this month because it is now forced to make predictions out-of-sample. More subtle and insidious examples abound.</p><p><a href="https://d2l.ai/">Dive into Deep Learning</a> classifies this phenomena<em> covariate shift </em>and attaches a mathematical framing. Imagine we are tasked with learning a conditional distribution 𝑝(<strong>𝐲</strong>|<strong>𝐱</strong>) of some output(s) <strong>𝐲</strong>, given some other input(s) <strong>𝐱</strong>. Even if the conditional distribution stays the same between development and deployment, a shift in the input distribution 𝑝(<strong>𝐱</strong>) can degrade deployed performance.</p><p>In this blog post, I’ll talk about a very specific example of <em>covariate shift,</em> and a neat data augmentation trick to tame it, that works with any modeling architecture.</p><h3>The data</h3><p>To serve as a reproducible and practical example, I’ll use the house price dataset from Kaggle, freely available at:</p><p><a href="https://www.kaggle.com/c/house-prices-advanced-regression-techniques/data">https://www.kaggle.com/c/house-prices-advanced-regression-techniques/data</a></p><p>The training set contains 1460 house prices of sales in Ames, Iowa from the period of 2006–2010. Our objective will be to build a model that predicts the sale price using other features in the dataset.</p><p>A self-contained jupyter notebook that reproduces all the results shown below can be found <a href="https://gist.github.com/lpekelis/690ce481f98ed5c2e793c40b6755f4ea">here</a>.</p><h3>Occlusion: missing features at serving time (the problem)</h3><p>Keeping the example going, now imagine that we want to deploy our model into production so that we can use it to predict the market value and make a competitive offer to homeowners wanting to sell their home to Opendoor. There is just one wrinkle…</p><p>The overall quality score (a feature in the model identified by OverallQual) requires the expert eye of an Opendoor technician to physically survey the home, score it, and submit their report. Furthermore, we don’t always have a technician’s report finished by the time we need to make an offer. This is an example of serving occlusion, where we at times need to make predictions with features missing at serving time, but will eventually see them in time for model training.</p><p>The OverallQual feature is important for the model. It improves the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_absolute_error.html">mean absolute error</a> (MAE) of our ability to predict sale prices by 12% when used alongside 31 other features. It is doing more than its fair share of the work and is actually the most predictive feature in the model.</p><p>But what happens when some percentage of OverallQual values are not available?</p><p>In the following simulation, I train a simple model that scales all features to have mean 0 and variance 1, imputes any missing feature values with 0 (the overall training set mean), and finally fits a Poisson regression to explain SalePrice with 32 numeric features in the data. The Poisson regression is mostly to emphasize that all this works for any modeling architecture, and also because home prices (rounded to the nearest $1k) roughly follow a Poisson distribution. I’ll refer to this model as the “naive” estimator.</p><p>I run 100 simulations. For each, I train the naive estimator, then mask a random fraction of OverallQual values in the training set (serving set is training set plus masking), use it to predict SalePrice, and calculate MAE. The violin plots below illustrate the distribution of MAE across different amounts of occlusion. Even though the estimator can serve results when features are occluded, it doesn’t mean it does it well. In fact, if more than 30% of OverallQual values are occluded at serving time, it’s better to not include the feature at all in the naive estimator!</p><p>Since we train on the 100% visible dataset, and especially since we have been calculating training (or in-sample) error this whole time (only who will be occluded is unknown at training time), surely we can do better.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/641/0*LvTra0N5j5LDjGj6.png" /></figure><h3>Build multiple models (one solution)</h3><p>Some motivation for an improvement comes from the correlation of OverallQual with other columns in the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/535/0*_Kf21l7jBuoJPXxx.png" /></figure><p>Besides SalePrice, OverallQual is also highly correlated with other features we are using as inputs in the model. For example, it is positively correlated with YearBuilt. However, we naively impute 0 or average quality for all homes, no matter when they’re built, leading to poor predictions. What we should be doing is imputing with the level of quality we expect, given the other feature values that we observe. Luckily, removing the OverallQual feature from the model does roughly that.</p><p>This suggests building two models: one with the OverallQual feature and one without. Then at serving time we can simply pick which model we use to predict depending on whether or not quality is available. I’ll call this the “multiple models” estimator.</p><p>Running the same simulation using the multiple models estimator recovers the behavior we expect. Multiple models always performs better than the naive estimator without the OverallQual feature (orange line) and has less and less improvement over it as more of the feature is occluded at serving.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/753/0*4uRUgJHcBwXiUpaD.png" /></figure><h3>Data stacking (an upstream solution)</h3><p>One defect of the multiple models estimator is that it increases serving complexity. When we deploy it into production, we’ll also have to build a ModelSwitcher module that chooses between our models based on the input. While the switching logic is simple in this example, multiple sources of occlusion across multiple features is harder to resolve, and any additional overhead is a potential source of failure in a critical part of the system.</p><p>A one model solution is to use data stacking. First, concat two copies of our training data, but replace all the values of OverallQual in the second copy with nan (or the missing indicator of your choice). Second, add an indicator feature for occlusion that is 0 over the 1st copy and 1 over the 2nd copy. Third, add interactions between the indicator and all other features besides OverallQual.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rBrts5nNmfTXkOxX.png" /></figure><p>Now our model has both kinds of examples to learn, occluded and non-occluded, and it has enough flexibility to encode all the differences between predicting when a feature is occluded and when it’s visible. Intuitively, the interactions allow features correlated with OverallQual to impute it when it is missing. I’ll call this model the “data stacking” estimator.</p><p>On the same 100 simulations of occlusion, the stacked estimator achieves almost identical MAE to the multiple models estimator.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/753/0*PdIQE0fxGJAiALem.png" /></figure><p>Look to the appendix below for more simulation results.</p><h3>Conclusion</h3><p>Data stacking is a slightly more technical way to solve for serving occlusion, but it pushes code in the right direction — upstream to the feature and model training pipelines. This typically demands fewer <a href="https://en.wikipedia.org/wiki/High_availability#%22Nines%22">nines</a> of uptime. Also, the interaction terms (steps 2 and 3) are not strictly necessary in architectures that do automated feature engineering, like random forests or deep neural nets.</p><p>At Opendoor, data scientists and engineers work closely together to truly solve problems end to end. If this type of work interests you, <a href="https://www.opendoor.com/jobs">find out more at our careers page</a>.</p><h3>Appendix</h3><p><strong>More simulation results</strong></p><p>Furthermore, the predictions of the two estimators are very similar — 90% of predictions are less than $2500 apart, which is 10% of the naive MAE. And even the coefficients are roughly the same up to an arbitrary scaling between feature and coefficient — 90% of coefficient values have less than 20% difference.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/809/0*v0wzxmExK3zbAsY9.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/595/0*P8g09ptuhgt7sdqV.png" /></figure><p><strong>A note on other imputation techniques</strong></p><p>There are lots of other imputation techniques I could have used. Many are substantially more powerful than the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.impute.SimpleImputer.html">SimpleImputer</a> in the naive estimator above. For example, the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.impute.IterativeImputer.html">IterrativeImputer</a> included in <a href="https://scikit-learn.org/stable/index.html">sklearn</a> predicts missing feature values from a multivariate model fit to all other features, in a round-robin fashion. Combining it with step 1 of Data Stacking gives very similar results to full Data Stacking, though it is computationally more complex.</p><p>While the efficiency of imputation algorithms is an interesting topic on its own (and maybe enough content for a separate blog post), my main goal here was to remind us data scientists that augmenting data is a valid and sometimes powerful item in our toolbox.</p><p><a href="https://www.opendoor.com/jobs">The simple way to sell or buy a home.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3e2d0d9ed674" width="1" height="1" alt=""><hr><p><a href="https://medium.com/opendoor-labs/taming-missing-features-at-serving-time-3e2d0d9ed674">Taming missing features at serving time</a> was originally published in <a href="https://medium.com/opendoor-labs">Open House</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Liquidity modeling in real estate using survival analysis]]></title>
            <link>https://medium.com/opendoor-labs/liquidity-modeling-in-real-estate-using-survival-analysis-cfd0ce01522a?source=rss-efdb2be1f68d------2</link>
            <guid isPermaLink="false">https://medium.com/p/cfd0ce01522a</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[real-estate]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Opendoor]]></dc:creator>
            <pubDate>Thu, 15 Aug 2019 21:20:00 GMT</pubDate>
            <atom:updated>2020-02-05T16:54:06.925Z</atom:updated>
            <content:encoded><![CDATA[<p>By: <em>David Lundgren &amp; Xinlu Huang</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Eu2sPkre0b2uG-YDBiZGsQ.png" /></figure><p>Originally published at <a href="https://www.opendoor.com/w/blog/liquidity-modeling-real-estate-survival-analysis">Opendoor.com</a>:</p><p><a href="https://www.opendoor.com/w/blog/liquidity-modeling-real-estate-survival-analysis">Liquidity modeling in real estate using survival analysis | Opendoor</a></p><p><strong><em>Gloria Gaynor: </em></strong><em>At first I was afraid, I was petrified…</em></p><p>At <a href="https://www.opendoor.com/">Opendoor</a>, we buy and sell thousands of homes a month. The longer a home remains in our inventory, the more exposure we have to macroeconomic shifts. Moreover, property taxes, home financing, and maintenance fees all increase with time. These costs — which are sometimes referred to as “liquidity risk” — directly impact our unit economics. It is critical that we’re able to understand and predict how long a house will take to sell.</p><p>In this post, we’ll discuss modeling the holding time of our homes using a technique called<a href="https://en.wikipedia.org/wiki/Survival_analysis"> survival analysis</a>. This dramatically improves our ability to respond to changing market conditions, and to more accurately predict our unit economics.</p><h3>Describing the Data</h3><p>Opendoor analyzes a large collection of historical market transactions. We have details about the individual homes that sell — like the number of bedrooms and bathrooms they have, or their square footage — as well as information about each particular listing, like the sale date, sale price, when the home listed, and at what price.</p><p>We also look at similar data for homes that are currently active on the market, or that are taken off the market without transacting (we call these “delistings”). For these listings, we observe a lower-bound for their time-on-market, but not necessarily their true value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*N3g49AgpGsmgp229.png" /></figure><p>The image above is an example of Opendoor’s dataset. Each row is a listing on the market. We have a variety of home-level details (e.g., square feet) and listing-level details (list price) for each transaction. Our target variable is days-on-market, which we do not get to observe for all listings (some are still active, awaiting buyers, while others delist without ever selling).</p><h3>Framing as Regression</h3><p>Given our goal of predicting how long it will take to sell each home, the first approach you might try is to directly predict the variable days-on-market as a function of home and listing features. To do this, we’ll remove transactions that haven’t yet closed, or end up dropping off the market without selling — since we don’t know their actual label — and train a model using the remaining data points.</p><p>One way we understand our model’s performance is to take the predictions for all homes that list in a certain month, and then measure the accuracy of these predictions after a suitable amount of time has elapsed (for example, a year later). Below we plot the average error by predicted days-on-marketon a synthetic hold-out set of data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HkZUC4hH8lOENTh1.png" /></figure><p>The plot above shows the average errors of a (simulated) group of homes listed around the same time of predictions ordered by their expected time on market. The deviation between the solid “linear regression” line and the dashed “unbiased model” line in this simulation demonstrates the systematic bias in a model that results from dropping censored observations. The riskiest transactions end up being the most poorly explained!</p><p>As you can see from the plot of the average error of sale dates, our model is negatively biased on homes that take a long time to sell, resulting in under-predicting listings that take a long time to close. These are the riskiest homes, and we’re doing a poor job at predicting them — our model is too optimistic!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HQ2w1pUZTRpfTm1e.png" /></figure><p>The grayed listings above are censored observations, indicating partial information for our target.</p><p>By removing unsold homes from our training data, we end up introducing a significant bias into our model. Opendoor’s transactions are systematically different from standard market sales — we <em>must </em>sell our homes. The active or delisted homes that we discarded are like<a href="https://en.wikipedia.org/wiki/Abraham_Wald"> Wald’s planes that didn’t return</a>; they’re<a href="https://en.wikipedia.org/wiki/Survival_analysis#Censoring"> censored observations</a>. This is dangerous because it indicates a discrepancy between our offline backtested metrics and our online metrics. Our machine learning objective function doesn’t match our business objective function.</p><h3>Reframing as Classification</h3><p>Another way of viewing the problem is as a binary classification problem: a home is either “risky” or “not-risky”. Homes that sell fast are considered less risky. Listings that move slowly, or that never close at all, are highly risky. The only thing required is choosing a threshold for “risky”, such as homes that don’t close within 100 days on the market. Many time-to-event prediction problems, like churn prediction, are handled in this manner (we often ask “did a user churn or not”, rather than “when, if ever, did the churn occur”).</p><p>It’s a simple trick — just a variable transformation and a switch from regression to classification. By looking at the problem this way, we side-step a lot of the censoring issues from before. We get to use data from delistings (that have survived at least 100 days), homes that we’ve defined to be inherently risky, as well as active listings that have exceeded our risk threshold.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*w901BFOqmuibSgE_.png" /></figure><p>By changing our target, and reframing the problem as classification allows us to use more data, particularly about homes that are still active (52 Downtown Ave) or that withdrew from the market without selling (90 Outskirts Lane). However, this approach still throws away information. By quantizing our target, we treat all homes in the same category as equivalent, which is unlikely to be true (Is a listing that closed in 7 days really equivalent to one that closed in 99 days?).</p><p>Additionally, we discard homes that haven’t exceeded the risk threshold; these are by definition recent transactions. This causes us to respond sluggishly to evolving market conditions. If the market rapidly turns, our model will lag the change significantly, unintentionally increasing the risk the company takes on.</p><p>Finally, it’s not clear how to interpret this as a non-technical stakeholder, “<em>Okay, this home is riskier, but how much riskier?</em>”. It’s difficult to answer questions like “<em>When should I expect to finish clearing this cohort of homes?</em>”.</p><h3>A General Recipe for Survival Analysis</h3><p>Fortunately there’s a sub-area of statistics that’s focused on modeling problems exactly like this,<a href="https://en.wikipedia.org/wiki/Survival_analysis"> survival analysis</a>. By slightly re-framing our classification problem, we can avoid the issues mentioned above, and leverage a range of techniques related both to<a href="https://data.princeton.edu/wws509/notes/c7s6"> discrete time models</a> as well as classification problems.</p><p>To do this, we leverage the fact that our target variable is discrete, and expand our dataset to include examples of listing-day pairs. Each day a home is on the market without closing is marked as a negative example, and whenever we observe a sale, we label it as a positive example. The censored active transactions and delistings get full representation in this formulation. They simply have no corresponding positive examples.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uj8EZLVGYu3m21Vo.png" /></figure><p>While this dramatically increases the size of our dataset, the ecosystem for solving large-scale classification problems is mature and well-understood.</p><p>To map back to our original problem of predicting the days-on-market for a home, we predict the probability of a sale on a fixed number of listing-days for the particular property. This corresponds to the conditional probability of a listing closing on a particular day, <strong><em>given</em></strong> that it survived on the market until then (in survival analysis parlance, this is the “<a href="https://en.wikipedia.org/wiki/Failure_rate#Failure_rate_in_the_continuous_sense">hazard rate</a>”). With these probabilities, we can stitch together the probability mass function (pmf) for a sale occurring on a specific day and therefore the expected days until sale, with the following relations.</p><p>sale_probability(day, home) := ( hazard(day, home) * product(1 — hazard(previous_day, home) for previous_day in 0…day)) expected_days_to_sale(home) := sum(sale_probability(day, home) * day for day in 0…MAX_NUMBER_OF_DAYS)</p><p><a href="https://gist.github.com/jggomez88/e38ad2027e402b5b584fefb3b0d3a4ba/raw/37c56654deb9bb70072fea75bef26304e75ca7e4/days-on-market.py">view raw</a><a href="https://gist.github.com/jggomez88/e38ad2027e402b5b584fefb3b0d3a4ba#file-days-on-market-py">days-on-market.py</a> hosted with</p><p>by <a href="https://github.com/">GitHub</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hITratfXeMOY1JBE.png" /></figure><p>Our classifier predicts the conditional probability of a home selling in a particular period conditioned on that home not selling before that. To convert hazard to the more standard pmf, we compute the probability that the home failed to sell on any of the previous days it was on market, and then multiply by the probability that it sold on the specific day in question (i.e, the hazard).</p><p>One major advantage of this approach is that it provides an intuitive mechanism for adding time-varying features. To code features like the number of active competitors, the current list price of a home, or whether a home is active during a seasonally slow period (e.g. Thanksgiving), we simply add new features for this behavior with the appropriate instantaneous value of the attribute.</p><p>A second benefit with this structure is that it is immediately intuitive to business stakeholders and industry partners. In real estate, a key market-level indicator is the inventory turnover rate. This measures the ratio of sales in a given period to the number of active listings in that same period. By defining our classification problem in the listing-day space, we model this pervasive and widely-understood relationship. To convert from our unit-level representation to this aggregate view, we can take the mean of sold-in-the-next-day grouped by time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3oFSgobHKQrrDJf7.png" /></figure><p>Operating in the listing-day space provides a natural framework for adding time-varying features. In the example above, 410 Main Street undergoes a series of price drops over time, from the initial list price of $200,000 down to $170,000. To encode this behavior, we just add a feature for “current list price” and assign the value to be the list price on the specific day.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*le45cUTADur0G82f.png" /></figure><p>The chart above shows the inventory turnover rate (also referred to as “clearance rate”) in Phoenix for three consecutive years. The rate is defined as the ratio of sales in a given period to the total number of active listings in that same period. As we can see, for most of the 2018, homes were clearing faster than in previous years, but the seasonal dip this year appears to be more dramatic than in the past.</p><p>At Opendoor, our mission is to <strong><em>empower</em></strong> everyone with the <strong><em>freedom to move</em></strong>. A key enabler of this is our ability to accurately understand, model, and price liquidity risk. This methodology has improved our ability to accurately forecast our cost structure and dynamically respond to changing market conditions. Because of this, we can charge fairer and more transparent fees.</p><h3>Special Thanks</h3><p>Thanks to Chris Said, Jackson Gorham, Frank Xia, Todd Small, Sherwin Wu, Mike Chen, Jules Landry-Simard, Joseph Gomez, and many others for their helpful comments and contributions!</p><p><a href="https://www.opendoor.com/jobs">Join our team</a> and build cool things with real estate data. Or learn more about <a href="https://www.opendoor.com/">Opendoor</a> and the other challenges our <a href="https://www.opendoor.com/w/blog/category/technology">data science team</a> is working on.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cfd0ce01522a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/opendoor-labs/liquidity-modeling-in-real-estate-using-survival-analysis-cfd0ce01522a">Liquidity modeling in real estate using survival analysis</a> was originally published in <a href="https://medium.com/opendoor-labs">Open House</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why we use S3Guard with S3 as a filesystem for Spark]]></title>
            <link>https://medium.com/opendoor-labs/why-we-use-s3guard-with-s3-as-a-filesystem-for-spark-6cf464f8bacd?source=rss-efdb2be1f68d------2</link>
            <guid isPermaLink="false">https://medium.com/p/6cf464f8bacd</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[real-estate]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Opendoor]]></dc:creator>
            <pubDate>Thu, 15 Aug 2019 21:18:54 GMT</pubDate>
            <atom:updated>2020-02-05T16:14:19.710Z</atom:updated>
            <content:encoded><![CDATA[<p>By: <em>Gustavo Torres, Vincent Ketelaars, and Chris Said</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CeeyV5H1YV1sKaAULXSXMw.png" /></figure><p>Originally published at <a href="https://www.opendoor.com/w/blog/why-s3guard-with-s3-as-a-filesystem-spark">Opendoor.com</a>:</p><p><a href="https://www.opendoor.com/w/blog/why-s3guard-with-s3-as-a-filesystem-spark">Why we use S3Guard with S3 as a filesystem for Spark | Opendoor</a></p><p><em>Your tables are missing rows, and your jobs are failing mysteriously. It feels like some malign entity is at work. Here’s one solution…</em></p><p>After some diligent debugging, your team has figured out that the problems originated from interactions with <a href="https://aws.amazon.com/s3/">S3</a>, the ubiquitous storage service from Amazon. S3 is an object store, and it allows you to easily upload and download massive amounts of data.</p><p>Like many companies, we at Opendoor use S3 as a storage dump. But we also use it as a <em>filesystem </em>to interact with Big Data processing systems like <a href="https://spark.apache.org/">Spark</a> and <a href="https://dask.org/">Dask</a> (Read more on <a href="https://www.opendoor.com/w/blog/imputing-text-data-using-markov-random-fields">how we use Spark</a>)</p><p>We’ll discuss an important challenge with distributed system consistency when using S3 as a filesystem. But before we get to that, let’s start with why we chose S3 over other alternatives like <a href="https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html">HDFS</a>.</p><h3>Why use S3 as a filesystem?</h3><p>There are some good reasons why you would use S3 as a filesystem, rather than other options such as <a href="http://cassandra.apache.org/">Cassandra</a>, <a href="https://kudu.apache.org/">Kudu</a>, or the more commonly used HDFS.</p><ul><li><strong>Fully managed<br></strong>S3 is fully managed by AWS, and has ridiculously high up time. If S3 goes down, chances are there are other business that are going down as well too!</li><li><strong>Cost<br></strong>It’s much cheaper than hosting your own HDFS, Cassandra, Kudu</li><li><strong>Developer Experience<br></strong>Good API support for uploading and downloading</li><li><strong>Elasticity<br></strong>You need a bit more space, no problem!</li></ul><p>These are compelling reasons to use S3 as a filesystem. However, if you are considering making the switch, it’s important to keep in mind a key limitation to S3 when it comes to <em>consistency</em><strong>.</strong></p><h3>What is consistency?</h3><p>A distributed filesystem is <em>consistent</em> if actions made on one node are immediately available on a different node. That is, a consistent distributed filesystem acts as if it were running in a single process. For example, HDFS is a robust filesystem since it guarantees that if one node writes a file, another node could discover that file immediately after.</p><p><strong>What level of consistency does S3 provide?</strong></p><p>Unlike HDFS, S3 provides <em>read-after-write</em> and <em>eventual</em> <a href="https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel">consistency</a>. Read-after-write consistency means that after you have created a new object, subsequent requests will return that same object. Eventual consistency, however, applies in all other scenarios, meaning that your updates will only eventually become available consistently. Until that time though, your requests could get returned in either the old state or the new!</p><p>These scenarios include overwriting an existing key or the deletion of the key. There is even a scenario where, if you query an object before you create it, that write will also be <em>eventually consistent</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/0*BoVWWqyBQJC0f_L8.png" /></figure><h3>S3 can cause missing rows</h3><p>When we started to use S3 as a filesystem for Spark jobs, we occasionally noticed issues with our results. Further investigation into intermediate artifacts revealed missing rows in our data.</p><p><strong>So how did this happen?</strong></p><p>Some of the intermediate results for our Spark jobs are very large, and splitting up those objects makes it easier to deal with them. The diagram below shows Task 1 successfully writing two split files to S3 in the same directory.</p><p>But, when Task 2 requires the output of Task1 and asks for a list of all the files, the returned list only contains a.txt. This causes Task 2 to continue working with only half of the intended data set. If however Task 2 were to wait a little longer, S3 would have managed to propagate the changes throughout, and a subsequent list call would have returned both files, i.e. the entire data set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/0*zckfLMtX-9USJxRZ.png" /></figure><h3>S3Guard as a solution</h3><p>To deal with this, Hadoop provides a tool called <a href="https://hadoop.apache.org/docs/r2.9.1/hadoop-aws/tools/hadoop-aws/s3guard.html">S3Guard</a>, which uses <a href="https://aws.amazon.com/dynamodb/">DynamoDB</a> as a consistent metadata layer where all read requests should go before hitting the S3 API. It takes advantage of the fact that S3 can provide read after write consistency if you know the filename you are looking for.</p><p>The only downside of using S3Guard when writing to S3 is the fact that all writers and readers must use this metadata layer when writing/reading so that the DynamoDB table does not go out of sync with S3 contents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HEh2vRP5-3j3OF2H.png" /></figure><p>Be careful when implementing this, because only Hadoop understands the s3a protocol that is required for S3Guard. You will get errors if you try it with an AWS native interface!</p><h3>Success!</h3><p>S3Guard successfully solved our problems with missing rows, which we were able to confirm by verifying that the number of rows written matched the number of rows we expected.</p><p>If you’re interested in complex data processing problems, <a href="https://www.opendoor.com/jobs">come join us at Opendoor</a>. Or learn more about <a href="https://www.opendoor.com/w/blog/category/technology">the challenges</a> our team is solving.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6cf464f8bacd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/opendoor-labs/why-we-use-s3guard-with-s3-as-a-filesystem-for-spark-6cf464f8bacd">Why we use S3Guard with S3 as a filesystem for Spark</a> was originally published in <a href="https://medium.com/opendoor-labs">Open House</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding how busy roads affect home values]]></title>
            <link>https://medium.com/opendoor-labs/understanding-how-busy-roads-affect-home-values-79ea116b4b43?source=rss-efdb2be1f68d------2</link>
            <guid isPermaLink="false">https://medium.com/p/79ea116b4b43</guid>
            <category><![CDATA[openstreetmap]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[real-estate]]></category>
            <dc:creator><![CDATA[Opendoor]]></dc:creator>
            <pubDate>Thu, 15 Aug 2019 21:17:54 GMT</pubDate>
            <atom:updated>2020-02-05T16:15:12.793Z</atom:updated>
            <content:encoded><![CDATA[<p>By: <em>Zain Shah &amp; Zach Gottesman</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d8NrExW47XphIFkC8Z7XMA.png" /></figure><p>Originally published at <a href="https://www.opendoor.com/w/blog/understanding-how-busy-roads-affect-home-values">Opendoor.com</a>:</p><p><a href="https://www.opendoor.com/w/blog/understanding-how-busy-roads-affect-home-values">Understanding how busy roads affect home values | Opendoor</a></p><p>At Opendoor, we strive to make moving easier, providing sellers with more certainty through competitive, all-cash offers. We’re able to do this using a combination of human and machine intelligence. Specifically, we supplement data-driven home value predictions with local market knowledge from real estate experts.</p><p>Any realtor will tell you that one of the most significant factors to consider when estimating a home’s value — if not the most significant — is its location. Some simple examples of external location factors include the amenities of the neighborhood and walkability. For us, one of the most important, and surprisingly difficult to compute factors, is whether a home is exposed to a busy road (and subsequently, exposed to road noise) [<a href="https://collateralanalytics.com/blog/2018/09/20/the-impact-of-noise-on-residential-property-value/">1</a>, <a href="https://aresjournals.org/doi/abs/10.5555/1521-4842.18.1.35">2</a>, <a href="https://www.sciencedirect.com/science/article/pii/0022460X82904965">3</a>].</p><p>Here’s how we went about automatically including this information in our models, using both local real estate knowledge and data.</p><h3>The challenge</h3><p>In the past, our valuation model often disagreed with the offers made by our real estate experts on homes affected by busy roads. We found that our human valuators and our model made busy road adjustments very differently. In fact, busy road adjustments accounted for nearly 11% of our major valuation discrepancies between our model and local experts. With the prevalence and magnitude of our automated valuation misses, we were determined to engineer a smarter busy road feature and enhance our existing road data.</p><h3>Previous Approaches</h3><p>Opendoor uses a freely available collaborative dataset known as OpenStreetMap for our road geometries. You can think of OpenStreetMap (OSM) as something like the Wikipedia for road data. OSM is fantastic for open source mapping projects and contains particularly comprehensive road geometry data, which makes it great for geospatial operations.</p><p>Unfortunately, the data from OSM is not designed to meaningfully describe traffic volume. The best we can do with the OSM vector data alone is guess the traffic volume of a road based on its road type. In other words, if the road is marked as a “motorway” (OSM-speak for large highway) then it should indicate high traffic. If a road is marked as a “residential road” that should indicate low traffic volume.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_0AxPu4usZxe9qiF.png" /></figure><p>These OSM road labels are only intended to distinguish how a road should be rendered on a map. The labels do not <em>necessarily</em> represent traffic volume. We don’t want to attribute incorrect traffic densities to roads because it could inaccurately skew our valuation model’s prediction on a customer’s home.</p><p>Generally, our market operators agree with the busy road adjustments inferred from OSM when evaluating roads classified as freeways and highways. However, that agreement breaks down when we examine the adjustments our model makes for homes on arterial roads, connector routes, and main streets in neighborhood subdivisions. The latter road types’ traffic patterns are much more nuanced and more difficult to get from publicly available data.</p><h3>Pitfalls of Other Datasets</h3><p>We investigated several alternative traffic data sets including data from the Department of Transportation, traffic data from private companies, and noise data from the Bureau of Transportation. To our dismay, we found that those datasets didn’t contain the exact data we needed.</p><p>Our use case demands high coverage for all sorts of roads, even small roads in residential subdivisions. What we need is a dataset granular enough to describe the small road networks people live within, which is where the majority of our mis-valuations occurred.</p><h3>Be the dataset you wish to see in the world</h3><p>Opendoor sits on a treasure chest of internal data about home values and the adjustments made when comparing a home to other comps. If you’re not familiar with making adjustments for comparable home, see our guide on <a href="https://www.opendoor.com/w/guides/how-opendoor-calculates-the-value-of-your-home">how Opendoor calculates home values</a>.</p><p>Not only do we have local real estate experts deliberating on thousands of home values in each market, but we also have direct data about market dynamics from being one of the largest home sellers in the markets we operate in.</p><p>The offers we’ve made on our customers’ homes contain valuable information from our experts on which roads affect home values the most, but this data is restricted to home level information, and therefore difficult to generalize to new offers.</p><p>However, with a trusted dataset of road geometries, we can actually generalize this info to offers on new homes we’ve never seen before, so long as we can figure out which road is responsible for a given adjustment. This way we won’t repeat the same mistake for a new home, so long as it’s on a road we previously made an offer on. Even better, this data would only improve over time as our offers dataset grows.</p><h3>We need adjacency</h3><p>To build this dataset, we need to better understand which road a home’s adjustment was made for, as well as which homes should be affected by a given road. In other words, we don’t actually know which road is responsible for a given home selling $10K less than it should, so we need to infer this information.</p><p>In the past, we concluded that a home is affected by a busy road only by its distance to the nearest busy road (+ a cutoff). The distance + cutoff approach does not work well for dense or sparse collections of homes.</p><p>For instance, if <em>home A</em> is close to a big highway, but <em>home B</em> is sandwiched between <em>home A</em> and the highway, we could still possibly make a busy road adjustment for <em>home A</em> when we should not have. Just the same, we miss out when a home is next to the highway but beyond our cutoff distance. With such an imprecise association between roads and homes, we dilute this valuable information by spreading it across all the roads nearby.</p><p>The more accurately we can attribute a certain valuation difference to a specific road, the more valuable our dataset will be. At first, it may seem reasonable to look at the street name on a home’s address, or any road touching the lot the house sits on. Unfortunately, that wouldn’t account for cases where a highway runs alongside a home’s backyard. There may be a field in between, but the traffic could still be an ear-and-eye-sore.</p><p>To this end, we settled on defining the association as one of adjacency–only homes “adjacent” to a road are affected by it. For this, we need a robust association which can account for any roads that the home might be adjacent in any direction.</p><h3>Voronoi Diagrams</h3><p>We determine adjacency using something called a <a href="https://en.wikipedia.org/wiki/Voronoi_diagram">Voronoi Diagram</a>. A Voronoi diagram is a partition of a given space based on a set of points using an algorithm called Delaunay triangulation. The algorithm effectively divides the space into regions for each point so that everything in a given region is closer to its point than any of the other points.</p><p>This fulfills our requirements nicely because we want to know whether a home is closer to a given road than anything else, irrespective of a specific distance cutoff. If home A is closer to a busy road than home B, then its Voronoi region will be touching the road, while home B’s Voronoi region will not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*df_3-sJtpnVoLdct.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qCWyK3zfRkyoX9PE.png" /><figcaption>Source image left: Veroni diagrams Wikipedia</figcaption></figure><h3>Zooming Out</h3><p>Our adjacency calculation is performed by collecting all the road geometry coordinates and home coordinates into a set of points for a given market. We then generate a Voronoi diagram from the set of all points (road and home coordinates).</p><p>From this Voronoi diagram, we look at all the home points and compute whether their Voronoi region touches a Voronoi region of a road point. If a home region touches a road region, then that home is said to be adjacent to that road, because the region of space closer to that road than anything else is also closer to the home than anything else.</p><p><strong>At a high level, we generate our dataset by doing the following:</strong></p><ol><li>For a given OSM road geometry, we find homes adjacent to the road.</li><li>We then look for previous human adjustments on any of those adjacent homes</li><li>From the previous human adjustments, we compute the average road adjustment value</li><li>We label the OSM road geometry with the average road adjustment value (which we can then use for future offers on any homes adjacent to that specific road geometry).</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Rj82M650b8lMSRUe.png" /></figure><h3>When Data Improves, the Customer Wins</h3><p>So, how did this new busy road data concoction affect the performance of our valuation model? As mentioned before, this dataset grows over time as we collect more human adjustment data for any given market. Keeping that in mind, it should not be surprising we found this data is most beneficial for the markets we’ve operated in the longest, such as Phoenix and Dallas.</p><p>With a quick competitive offer on any <a href="https://www.opendoor.com/w/faq/what-types-of-homes-does-opendoor-purchase">eligible home</a>, Opendoor can empower anyone with the freedom to move. Knowing how homes near high traffic roads perform on the market allows us to operate more efficiently and reach more customers, which at the end of the day is what matters most.</p><p>Interested in using real estate data to build more transparency into the biggest financial purchase of a lifetime? <a href="https://www.opendoor.com/jobs">Join our team</a> or learn more about the <a href="https://www.opendoor.com/w/blog/category/technology">challenges we’re solving</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=79ea116b4b43" width="1" height="1" alt=""><hr><p><a href="https://medium.com/opendoor-labs/understanding-how-busy-roads-affect-home-values-79ea116b4b43">Understanding how busy roads affect home values</a> was originally published in <a href="https://medium.com/opendoor-labs">Open House</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why ensembling works: The intuition behind Opendoor’s home pricing]]></title>
            <link>https://medium.com/opendoor-labs/why-ensembling-works-the-intuition-behind-opendoors-home-pricing-d12c17c24f96?source=rss-efdb2be1f68d------2</link>
            <guid isPermaLink="false">https://medium.com/p/d12c17c24f96</guid>
            <category><![CDATA[real-estate]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Opendoor]]></dc:creator>
            <pubDate>Sat, 01 Dec 2018 00:43:05 GMT</pubDate>
            <atom:updated>2020-02-05T16:16:06.939Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VKAenipyUn-fggpszhXgdg.png" /></figure><p><em>Written by Chris Said, Jules Landry-Simard</em></p><p>At Opendoor, we leverage machine learning to price homes that customers sell to us. Because we make real offers on homes worth hundreds of thousands of dollars, it is critical that our models make accurate price estimates.</p><p>If our estimates are too low, we are unfair to our customers. If our estimates are too high, we might pay more for the home than the market would allow us to sell it for, and as a result we might lose money. Neither of these outcomes are good, and that’s why we always work hard to improve our models.</p><p>A popular way to improve model accuracy is to build multiple models and then compute a weighted average of their estimates. For example, if one model says that a home is worth $295K and another model says the home is worth $305K, we might average the two estimates together to make a final estimate of $300K. This approach is called <em>ensembling.</em></p><p>When it comes to improving accuracy, ensembling is effective. What is probably most surprising is that adding a new model to the ensemble usually helps even if the new model is inferior to all of the original models.</p><h3>If our models were people</h3><p>The simplest way to think about ensembles is to imagine that you are in a meeting with two real estate professionals. Both are trying to guess how much a home is worth, and it is your job to reach a conclusion. Amy is smart and has a lot of experience, and she thinks the value is $230K. Bob is newer and doesn’t know as much, and he thinks the value is $220K. Who is closest to the actual market price?</p><p>Amy is more likely to be accurate than Bob, but you probably shouldn’t rely only on her answer. Even though Bob doesn’t know as much, he brings a new perspective. Maybe he notices something about the home that Amy has missed. The best thing for you to do is to <em>compromise</em> between the two estimates. Since Amy has more expertise, you should put more weight on her estimate, so that your compromise comes closer to her, maybe around $227.</p><p>In some sense, Amy and Bob are both “models”. By choosing a price that is a weighted average of their estimates, you are acting like an ensemble model and will tend to make better decisions. Of course, this approach won’t always lead to better decisions. Sometimes the best answer may have been much closer to Amy’s estimate or closer to Bob’s estimate, or perhaps even more extreme than either one of them. But on average you will tend to make better decisions if you factor in both of their beliefs than if you only pay attention to one.</p><h3>Putting some numbers to it</h3><p>Let’s test this with some real numbers. Imagine you have two models, A and B. Sometimes the models make estimates that are 5% too high and sometimes they make estimates that are 5% too low. There is no correlation between the errors of the models. That is, whenever Model A is too high, it is equally likely that Model B will be too high or too low.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uW-dVzmOMj4eyz8g.jpg" /></figure><p>As you can see in the table above, these models will on average both be 5% off of what we estimate the market price to be.</p><p>However, if you average the two models together, something interesting happens. Whenever both models are too low or too high, the ensemble is also too low or too high. But whenever one model is too low and the other model is too high, the ensemble estimate becomes perfect! As a result, the ensemble model will on average be about 2.5% off of the market price, which is better than either of the original models alone.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YwHtoM8QYAJ5dCSr.jpg" /></figure><p>Even if you made one of the models slightly worse, for example by turning all the 5’s into 6’s in Model B, the Ensemble would still outperform either model alone, which helps explain why introducing an inferior model can sometimes help. Of course, at a certain point, a model can become so bad that it does more bad than good. But in most cases, adding a new model tends to help.</p><h3>What this looks like in model space</h3><p>A final way to explain ensembling is with graphs and a little bit of linear algebra.</p><p>In the plot below, we show some simulated errors for 300 homes based on two different models. Each point represents a home. The errors for Model A are shown on the vertical axis and the errors for Model B are shown in the the horizontal axis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*Qa8VD04G2_yATPOn.png" /></figure><p>Look at the homes in the top right quadrant. For these homes, both of the models have positive error. As a result, these homes tend to obtain very little benefit from ensembling, as the average estimate will also have positive error. That’s why we call this quadrant the “Meh” Quadrant.</p><p>Now look at homes in the top left quadrant. For these homes, Model A gave a positive error and Model B gave a negative error. Because the errors largely cancel each other out, these homes benefit a lot from ensembling. That’s why we call this quadrant the Good Quadrant.</p><p>A similar story plays out in the bottom two quadrants.</p><p>Quadrants are a useful way to think about this, but to be a bit more precise we need to use a little bit of linear algebra. When you compute an average of two values, you multiply each of the values by 0.5 and then you add them up. One way of thinking about this is that you are taking a dot product of the vector of values with the averaging vector [0.5, 0.5]. And a good way to think about a dot product is that it is proportional to a <em>projection</em> of one vector on to the other. Without getting into the details too much, an average of two numbers is equivalent to 71% of the projection on to [0.5, 0.5]</p><p>To show how this plays out graphically, the plot below shows the averaging vector ([0.5, 0.5]), which points to the top right. We have also highlighted one of the homes in dark. For this home, the error from Model A was positive, and the error from Model B was negative. The average of the errors (i.e. the error of the ensemble) is proportional to the projection of the home onto the averaging vector. The projection for this example home is shown as the pink vector pointing to the top right. Notice that it is very small, meaning that the ensemble error for this home (0.091) is also small.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*o9VAIF__ZoN8f3yS.png" /></figure><p>The two models shown in the graph above had errors that were not very correlated. But sometimes two models can have errors that are very correlated, meaning that when one model has e.g. positive error, the other model is also very likely to have positive error. When this happens, the models become somewhat redundant, and there is not much advantage to averaging.</p><p>Some insight can be obtained by visualizing models with different degrees of correlation. The animation below shows a fixed Model A, with different variants of Model B, ranging from perfectly correlated with Model A to totally uncorrelated with Model A.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*RxpDlgNgC7mjEURp.gif" /></figure><p>Pay attention to the dots in the top right “Meh” Quadrant when the errors are perfectly correlated. Notice that when the errors begin to separate, some dots move right and some dots move left. But critically, notice that <em>more</em>dots move left than move right. This includes the example dot. The dots that move left reduce their projection on the averaging vector, and thus have improved error.</p><p>Why is this happening? The dots in the top right quadrant represent homes on which Model A had positive errors. If you have another model value these homes it’s likely that this other model will have lower (less positive) errors. While a minority of homes might obtain even higher error from the second model, most will get lower error. And critically, the less correlated the second model is with Model A, the bigger the magnitude of this delta, and the bigger the benefit of averaging.</p><p>That’s why at Opendoor we strive to use models that are minimally correlated with each other.</p><h3>Opendoor and ensemble models</h3><p>Ensembling has always been core to the way Opendoor approaches the very hard problem of valuing a home sight unseen. One way we use multiple “models” is through our hybrid pricing approach, where we have both automated and human-led valuations on a significant fraction of offers. This allows us to scale rapidly while maintaining the accuracy required to deliver a great experience to our customers.</p><p>We have also built infrastructure to allow data scientists to prototype, backtest and efficiently implement new valuation models. Among all the models we currently use in production, some of them even have sub-components that are ensemble models themselves!</p><p>If you’re interested in building new models and helping people move rapidly and freely, <a href="https://www.opendoor.com/jobs">consider joining our team</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d12c17c24f96" width="1" height="1" alt=""><hr><p><a href="https://medium.com/opendoor-labs/why-ensembling-works-the-intuition-behind-opendoors-home-pricing-d12c17c24f96">Why ensembling works: The intuition behind Opendoor’s home pricing</a> was originally published in <a href="https://medium.com/opendoor-labs">Open House</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>