Here at Loft, we are continually building our understanding of the real estate market, bringing data and the experience of our partners and founders together. We like to exercise our creativity by exploring and adapting existing concepts with data and algorithms.

One of the most established concepts in the real estate market is that of the neighbourhood. Each person has a different relationship with this concept: some people care little about which neighbourhood they live in, as long as it is close to work; others were born, grew up and created roots in the same neighbourhood, or identify with its lifestyle.

At Loft, in addition to identifying which neighbourhoods meet the different profiles of our clients, we are interested in understanding neighbourhoods by more functional aspects, such as size, population density, presence of green areas, and more. In particular, we were interested in knowing where the pockets of real estate value are located in the city. That is, to find out how to divide the city into parts such that property prices in each part are very similar.

Thus, we exercise our understanding of the market by creating a new value-focused neighbourhood concept, comparing new neighbourhood lines with existing ones.

Simplicity and effectiveness: decision trees

A decision tree is a machine learning algorithm that learns through rule induction. It partitions the training dataset aiming at reducing the dispersion of its target variable. In our case of neighbourhood partitioning, this algorithm fits very well, because we want to do just that: divide the city in partitions (splits in latitude and longitude data) such that homes in these partitions have low dispersion of price per square meter.

In a nutshell, we arrive at the result as follows:

• We prepared a database of real estate listings that contains estimates of price per meter of flats in 2018. At Loft, in addition to the listing database, we use several other data sources for pricing, including more up-to-date and accurate sources such as registrations and transactions on our marketplace. For the purposes of our exercise, however, the data collected is sufficient.
• Each row in our database is a building, containing the median price per metre of the apartments advertised in the building:

• We apply a rotation transformation to the latitude and longitude data. This way the tree can make diagonal splits.

• We fit the tree on the rotated lat-longs, with the constraint that we must have at least 200 buildings per partition.

• We publish the result using kepler.gl, which allows us to see the partitions and the average price per metre of the partition.

Results

Below we publish the map with the results, where you can switch between the layer of clusters (partitions) and average price per cluster. It’s worth mentioning again that we used a listings data source from 2018. Therefore, the prices on this map may not accurately reflect current market prices. Nevertheless, that does not defeat the purpose of using this database for making comparisons between neighbourhoods.

Large-scale WebGL-powered Geospatial Data Visualization Tool

Example: Jardins

Each person has a slightly different definition, but it is common to refer to the neighbourhoods between Avenida Paulista, Rua Estados Unidos, Avenida Rebouças and Avenida Brigadeiro Luiz Antônio as “Jardins”. Loft was born in these neighbourhoods. Our first office and first apartments were in this area. We are therefore very fond of this region and have a lot of experience operating there. We generally evaluate the behaviour of our algorithms in Jardins to see if they “make sense”.

It is common to divide Jardins into two different neighbourhoods:

• Jardim América, between Avenida Rebouças and Avenida Nove de Julho
• Jardim Paulista, between Avenida Nove de Julho and Avenida Brigadeiro Luiz Antônio

The algorithm, on the other hand, divided Jardins into 4 different partitions, dividing both Jardim América and Jardim Paulista in two (the central line runs through Alameda Casa Branca, almost at the division of the two neighbourhoods):

When we change the visualisation to the price per metre of each of the partitions, we see a pattern of higher valuation in the Jardim América region and in the vicinity of Rua Estados Unidos, a fact that had been heard in the market a lot, and has now been discovered and confirmed through data :).

Conclusion

Finally, we show this last visualisation of the average price per partition in São Paulo. We see several interesting patterns: (a) Vila Nova Conceição and a part of Moema coming together into a single zone; (b) different partitions in Higienópolis; (c) price increases around Ibirapuera Park; (d) a mini-high-value zone between Consolação and Bela Vista, and others.

Feel free to explore the map we have provided above and create your understanding of the market together with us!

We continue to use data science to address the complexity of the housing market, reinvent the consumption of housing, and eliminate, for our clients, the friction in the process of switching homes.

Do you want to join Loft and build the future of real estate?

Apply for our open positions! — https://jobs.lever.co/loft/

Using Machine Learning to find value zones in São Paulo was originally published in Loft on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introducing XGBoost Survival Embeddings (xgbse), our survival analysis package built on top of XGBoost

“There are two cultures in the use of statistical modeling to reach conclusions from data. One assumes that the data are generated by a given stochastic data model. The other uses algorithmic models and treats the data mechanism as unknown.” — Leo Breiman, Statistical Modeling: The Two Cultures

Survival Analysis is a branch of statistics concerned with modeling the duration of events, most importantly providing tools to deal with (and not simply discard) censored data, partial but useful information about durations. For instance, if an apartment is on the market for 90 days, we don't know how long it will take to actually sell it, but we do know that it took at least 90 days (where 90 days is the time of censoring). Survival analysis is useful for a wide range of applications, such as predictive maintenance, customer churn, credit risk, asset liquidity risk, and many others.

The most common output of survival analysis models is a survival curve, which shows the cumulative probability of survival (event not happening) over time. Other models may output risk, which is a number proportional to how fast the curve decays, or an expected time, obtained by direct optimization or by numerically integrating the curve.

At Loft, we're a big user of survival analysis models, as the journey of finding a new home is rather long in Brazil, with several milestones along the way (visits, offers, deeds, renovation, etc). Particularly, as Loft operates an iBuyer (using its own equity to buy, renovate, and sell apartments), we're interested in modeling the time to sell an apartment, as holding it for too long represents the biggest risk we are subject to. The models that produce these estimates are used to make million-dollar decisions and thus have strict requirements on discrimination performance and calibration.

Therefore, we invested a lot of time exploring survival analysis packages. In our quest we found that most implementations embraced one of two cultures:

1. Models with sound statistical properties (such as calibrated survival curves), but lacking in expressiveness and computational efficiency
2. Highly efficient and expressive models, but lacking some statistical properties (producing only risk or uncalibrated expected survival times — XGBoost fits here)

Given this scenario, we embarked on a journey to unite the two cultures in a single package that would meet our requirements: XGBoost Survival Embeddings (xgbse). And now we’re open sourcing it!

We chose XGBoost as our "base model" for its efficient and easy-to-use implementation, and leveraged some tricks to add nice statistical properties to it. We provide more details about XGBoost's shortcomings and these tricks next.

XGBoost Shortcomings for Survival Analysis

The XGBoost implementation provides two methods for survival analysis: Cox and Accelerated Failure Time (AFT). When it comes to ordering individuals by risk (discrimination), both show competitive performance (as measured by C-index, the ROC AUC equivalent for survival) while being lightning fast.

However, we can observe shortcomings when it comes to other desirable statistical properties. Specifically, three properties are of concern:

• prediction of survival curves rather than point estimates
• estimation of confidence intervals
• calibrated (unbiased) expected survival times

Let us take the AFT implementation as an example (as Cox only outputs risk and not time or survival). The AFT model outputs a value that should be interpreted as the expected survival time for each sample. To compute this time, it assumes an underlying distribution for times and events, controlled by the aft_loss_distribution and aft_loss_distribution_scale hyperparameters. As they control hard-wired assumptions of the model, we would expect that they could drastically change its output.

To confirm this is true, let us perform an experiment using the METABRIC dataset. We will test different values for aft_loss_distribution_scale while keeping aft_loss_distribution as "normal" and check how this will affect model performance. We test the following values: [0.5, 1.0, 1.5]. The following gist gives a rough idea on how to run this simple experiment but you can find the full code here. The results are very interesting, if not alarming:

aft_loss_distribution_scale: 1.5
C-index: 0.645
Average predicted survival time: 203 days
----
aft_loss_distribution_scale: 1.0
C-index: 0.648
Average predicted survival time: 165 days
----
aft_loss_distribution_scale: 0.5
C-index: 0.646
Average predicted survival time: 125 days
----

In all three scenarios, we can build models with C-index results close to the state of the art [8]. However, our predictions change drastically, with two models showing an average predicted survival time difference of 78 days. This difference would perhaps be understandable if we were analyzing a single sample. But we're looking across the full validation dataset where the average prediction should be fairly stable regardless of which model we use.

If we plot these results alongside an unbiased survival estimator such as the Kaplan Meier we can check that for each step of 0.5 in aft_loss_distribution_scale we move roughly one decile to the right in the curve. Also, we don't see a full survival curve: XGBoost only outputs point time-to-event predictions (no confidence intervals either).

Given these results, we raise the question: "what model should we trust?". Although we could try to optimize for some error measure such as L1-loss [9], we would like more guarantees for applications that are dependent on stable and calibrated time-to-event estimates. Having only point estimates, no confidence intervals, and no "calibration by design" mines trust and prevents shipping survival analysis models to production.

Improving XGBoost by using it as a feature transformer

Although in need of an extension for better statistical properties, XGBoost is undoubtedly a powerhouse. C-index results show that the model has great discrimination performance, being competitive with the state of the art. We just need to adapt how we use it.

Besides being leveraged for prediction tasks, Gradient Boosted Trees (GBTs) can also be used as feature transformers of the input data. Trees in the ensemble perform splits on features that discriminate the target, encoding relevant information in their structure. In particular, the terminal nodes (leaves) at each tree in the ensemble define a very nice feature transformation (embedding) of the input data.

This kind of tree ensemble embedding has very convenient properties:

1. sparsity and high-dimensionality: trees deal with nonlinearity and cast original features to a sparse, high-dimensional embedding, which helps linear models perform well when trained on it. For instance, a logistic regression trained on the embedding (with features as one-hot encoded leaf indices) can show comparable performance to the actual ensemble, with the added benefit of probability calibration (see [1], [2], and [3])
2. supervision: trees also work as a noise filter, performing splits only through features that have predictive power. Thus, the embedding actually has a lower intrinsic dimension than the input data (see [4]). This mitigates the curse of dimensionality and allows a kNN model trained on the embedding (using hamming distance) to have comparable performance to the actual ensemble. With neighbor-sets at hand, we can use robust estimators such as the Kaplan-Meier to get predictions.

These two properties are the foundation upon which we built xgbse. We show how we put them to good use next.

Logistic regressions, time windows, embedding as input

Our first approach, XGBSEDebiasedBCE, takes inspiration from the multi-task logistic regression method in [5], the BCE (binary cross-entropy) approach in [6], and the probability calibration ideas from [1], [2] and [3].

It consists of training a set of logistic regressions on top of the embedding produced by the underlying XGBoost model, each predicting survival at different user-defined discrete time windows. The classifiers remove individuals as they are censored, with targets that are indicators of surviving at each window.

As the embedding is sparse and high dimensional, the linear models actually can improve the ensemble performance, while also adding nice statistical properties to it, such as probability calibration (as logistic regressions are calibrated by design).

The naive approach tends to give biased survival curves, due to the removal of censored individuals. Thus, we made some adaptations such that logistic regressions estimate the di/ni term (point probabilities) in the Kaplan-Meier formula and then use the KM estimator to get nearly unbiased survival curves.

This way, we can get full survival curves from XGBoost, and confidence intervals with minor adaptations (such as performing some rounds of bootstrap). Training and scoring of logistic regression models is efficient, being performed in parallel through joblib, so the model can scale to hundreds of thousands or millions of samples.

Kaplan-Meier on nearest neighbors

As explained in the previous section, even though the embedding produced by XGBoost is sparse and high dimensional, its intrinsic dimensionality actually should be lower than the input data. This enables us to "convert" XGBoost into a nearest neighbor model, where we use hamming distance to define similar elements as the ones that co-occurred the most at the ensemble terminal nodes. Then, at each neighbor-set we can get survival estimates with robust methods such as the Kaplan-Meier estimator.

We recommend using dart as the booster to prevent any tree to dominate variance in the ensemble and break the leaf co-occurrence similarity logic. We built a high-performing implementation of the KM estimator to calculate several survival curves in a vectorized fashion, including upper and lower confidence intervals based on the Exponential Greenwood formula [7].

However, this method can be very expensive at scales of hundreds of thousands of samples, due to the nearest neighbor search, both on training (construction of search index) and scoring (actual search).

Single tree, and Kaplan-Meier on its leaves

As a simplification to XGBSEKaplanNeighbors, we also provide a single tree implementation. Instead of doing expensive nearest neighbor searches, we fit a single tree via XGBoost and calculate KM curves at each of its leaves.

It is by far the most efficient implementation, able to scale to millions of examples easily. At fit time, the tree is built and all KM curves are pre-calculated, so that at scoring time a simple query will suffice to get the model’s estimates.

However, as we’re fitting a single tree, predictive power may be worse. That could be a sensible tradeoff, but we also provide XGBSEBootstrapEstimator, a bootstrap abstraction where we can fit a forest of XGBSEKaplanTree's to improve accuracy and reduce variance.

Does it solve the problem?

Now we return to the first example and check how XGBEmbedKaplanNeighbors performs:

aft_loss_distribution_scale: 1.5
C-index: 0.640
Average probability of survival at [30, 90, 150] days: [0.9109, 0.6854, 0.528]
----
aft_loss_distribution_scale: 1.0
C-index: 0.644
Average probability of survival at [30, 90, 150] days: [0.9111, 0.6889, 0.5333]
----
aft_loss_distribution_scale: 0.5
C-index: 0.650
Average probability of survival at [30, 90, 150] days: [0.913, 0.6904, 0.5289]
----

As measured by the average probability of survival in 30, 90 and 150 days the model shows good results, with comparable C-index results and predictions being very close and independent of aft_loss_distribution_scale choice. Visually, the comparison of the average model predictions to a Kaplan Meier yields much better results:

Much less bias, full survival curves and stability! Although is too harsh to claim that the problem is solved, we believe that the we could build a strong survival analysis package and bring the "two cultures" much closer together.

Try it for yourself!

XGBoost Survival Embeddings shows great results in several survival analysis benchmarks, outperforming vanilla XGBoost and parametric methods in most cases. It is pip installable and scikit-learn compatible. There are more features to try, such as confidence intervals, extrapolation of survival curves, explainability through prototypes, metrics, and more. We’re committed to improving the lib and adding new content in the future.

For more details, please check the the library on github, the docs, and examples! We invite the community to contribute. Please help by trying it out, reporting bugs, and letting us know what you think!

By Davi Vieira, Guilherme Marmerola, Gabriel Gimenez and Vitor Estima.

References

[1] X. He, J. Pan, O. Jin, T. Xu, B. Liu, T. Xu, Y. Shi, A. Atallah, R. Herbrich, S. Bowers, and J. Q. Candela. Practical Lessons from Predicting Clicks on Ads at Facebook (2014). In Proceedings of the Eighth International Workshop on Data Mining for Online Advertising (ADKDD’14).

[2] Feature transformations with ensembles of trees. Scikit-learn documentation at https://scikit-learn.org/.

[3] G. Marmerola. Calibration of probabilities for tree-based models. Personal Blog at https://gdmarmerola.github.io/.

[4] G. Marmerola. Supervised dimensionality reduction and clustering at scale with RFs with UMAP. Personal Blog at https://gdmarmerola.github.io/.

[5] C. Yu, R. Greiner, H. Lin, V. Baracos. Learning Patient-Specific Cancer Survival Distributions as a Sequence of Dependent Regressors. Advances in Neural Information Processing Systems 24 (NIPS 2011).

[6] H. Kvamme, Ø. Borgan. The Brier Score under Administrative Censoring: Problems and Solutions. arXiv preprint arXiv:1912.08581.

[7] S. Sawyer. The Greenwood and Exponential Greenwood Confidence Intervals in Survival Analysis. Handout on Washington University in St. Louis website.

[8] H. Kvamme, Ø. Borgan, and I. Scheel. Time-to-event prediction with neural networks and Cox regression. Journal of Machine Learning Research, 20(129):1–30, 2019.

[9] H. Haider, B. Hoehn, S. Davis, R. Greiner. Effective Ways to Build and Evaluate Individual Survival Distributions. Journal of Machine Learning Research 21 (2020) 1–63.

xgbse: Improving XGBoost for Survival Analysis was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Na Loft, acreditamos que o mercado imobiliário está quebrado e precisa ser reinventado com tecnologia e dados. Queremos mudar a forma como as pessoas consomem moradia, com o foco em melhorar a experiência de pessoas compradoras e vendedoras, atingindo a visão de Freedom Living.

Nesse contexto, nosso time de ciência de dados (Data & Decision Science) tem a missão de transformar nossos dados em algoritmos que suportem nossos produtos e tomada de decisão em prol desta visão. Somos responsáveis pela geração de resultados de negócio e protagonistas nessa jornada: tornar o mercado imobiliário mais líquido, com menos assimetria de informação, conectando pessoas vendedoras e compradoras por meio da tecnologia.

Com o tempo nossos projetos evoluíram, e conseguimos montar melhor nosso quebra-cabeça de tomada de decisão em um processo orgânico que deu origem ao que apelidamos de nosso “ecossistema” de algoritmos, que continua evoluindo. Vamos mostrá-lo nesse post.

Nosso ecossistema de algoritmos

O mercado imobiliário tem diversas complexidades. Nele, são realizadas as maiores transações das vidas das pessoas, geralmente relacionadas a sonhos ou grandes mudanças. Clientes sofrem com pouca transparência de preços, dificuldade de achar um novo lar por conta de silos de informação e dificuldade de vender o lar atual por falta de visibilidade do mercado.

Nesse cenário, investimos fortemente em dados para resolver esses problemas, possibilitando construir confiança entre agentes do mercado. Construindo essa confiança, transformamos nosso marketplace em um motor de liquidez, ajudando pessoas compradoras a encontrar rápido seu apartamento dos sonhos, e pessoas vendedoras a obter propostas rapidamente. Suportando essa máquina temos uma série de modelos e políticas de tomada de decisão, formando nosso ecossistema de algoritmos.

Para trazer uma ajuda visual ao artigo, montamos a seguinte apresentação com diagramas do nosso ecossistema. Fique a vontade para usar como material de suporte. Vamos lá!

Estruturando o histórico de transações imobiliárias

Em países onde temos alta liquidez de imóveis e baixa dispersão de preços geralmente encontramos um sistema centralizado de anúncios e transações de imóveis. Nos Estados Unidos, esse sistema se chama MLS (Multiple Listings System). No Brasil, esses dados se encontravam em silos, nos cartórios. Trabalhamos intensamente com as associações de registradores de imóveis para mudar esse cenário, fazendo bons avanços.

Dessa forma, a primeira etapa da nossa máquina envolve catalogar as informações dos imóveis contidas em uma certidão chamada matrícula. Isso envolve digitalizar a matrícula, converter a imagem digitalizada para texto e aplicar algoritmos de NLP para encontrar as informações de interesse (entidades) nesse texto. Ao final do processo, teremos uma base de transações centralizada e estruturada.

Definindo uma referência de preços

Uma vez em posse de uma base de transações confiável, podemos utilizá-la para construir modelos de valoração de imóveis ou, na sigla em inglês, AVMs (Automated Valuation Models), em conjunto com outras fontes de dados como anúncios no mercado, transações no nosso próprio marketplace e a nossa feature store de imóveis (com informações de localização, condomínios, estado de conservação do apartamento e muitas outras).

Esses modelos entregam uma referência de preços de mercado, um ponto de partida para clientes determinarem o preço de anúncio dos seus imóveis, a depender da necessidade de rapidez na venda desses ativos. Além disso, essa referência de preços é consumida por diversos outros times na Loft, para ajudar na experiência de clientes em diversos outros pontos de contato. Esse é o primeiro passo para diminuirmos a assimetria de informação entre participantes do mercado, aumentar liquidez, e estabilizar preços (que atualmente são muito voláteis).

Operando fundos imobiliários

Com nossas referências de preço disponíveis, nosso ecossistema se torna mais diverso do ponto de vista de aplicações. O primeiro negócio que a Loft operou foi o de iBuyer (instant Buyer), que consiste em comprar, reformar e vender apartamentos, onde assumimos risco investindo em apartamentos de pessoas vendedoras que querem liquidez rápida e reformamos para pessoas compradoras que querem um apartamento pronto para morar.

Nesse processo, a Loft se torna proprietária dos imóveis até eles serem vendidos, financiando a compra deles por meio de fundos imobiliários, que atualmente têm centenas de ativos na carteira. Uma parte significativa do capital desses fundos foi fornecida por pessoas que decidiram investir seu patrimônio na visão da Loft e esperam retornos compatíveis aos riscos que elas assumiram.

Além do risco de variações de preço de mercado, o principal fator de risco para nossos ativos imobiliários é a liquidez (tempo até vender o imóvel). Dessa forma, investimos bastante energia na construção de modelos de liquidez imobiliária. Os modelos entregam estimativas de tempo até venda dos imóveis no portfolio, permitindo calcular o risco (e retorno esperado) desses ativos.

Mais especificamente, as métricas de risco e retorno dos ativos são derivadas de um simulador que calcula os custos de carregamento e as receitas associadas de cada apartamento. Com isso podemos escolher bons investimentos (inclusive diversificando o portfolio!), visando bons retornos para clientes dos nossos fundos e propostas justas para clientes do iBuyer.

Monitorando a saúde do nosso marketplace

Melhorar o motor de liquidez do iBuyer resolve somente uma parte do desafio de portfolio. Temos também a gestão de todos os outros apartamentos que estão listados na plataforma e não são de nossa propriedade, representando a maioria dos anúncios no marketplace.

Esses apartamentos pertencem a clientes que confiaram na Loft para conseguir boas propostas rápido. Do outro lado, pessoas compradoras querem encontrar o apartamento dos sonhos rápido e fazer bons negócios. Em qualquer marketplace, essas forças (de oferta e demanda) se equilibram e na Loft não é diferente. Contudo, dado o alto ticket médio e baixo giro do mercado imobiliário, precisamos garantir que o mercado tem a capacidade de se equilibrar sozinho e muitas vezes dar alguns empurrões para que ele se mantenha saudável.

Nesse cenário, antes de tudo, precisamos ter boas unidades de monitoramento: saber quais “produtos” existem na nossa vitrine, quais estão em falta e quais temos que trabalhar mais na sua divulgação. Portanto, começamos construindo um modelo de clusterização de apartamentos, que usa informações de demanda na plataforma Loft e outros registros para definir quais apartamentos são similares do ponto de vista da demanda.

Uma vez com o modelo de clusterização, podemos fazer o monitoramento da saúde do marketplace em diferentes clusters. Quando notamos desequilíbrio na plataforma tentamos acionar alavancas visando retornar a níveis saudáveis.

Além disso, a partir do modelo de clusterização conseguimos obter um serviço para determinar, dado um apartamento, quais são seus comparáveis, atuando como um ponto de partida para recomendação no site.

Transmitindo a voz da pessoa compradora através de dados

No cálculo de equilíbrio do marketplace, temos relativamente nítida nossa métrica de oferta, o número de apartamentos na vitrine. Contudo, a métrica de demanda é menos trivial. Para isso, temos um time dedicado a entender a demanda, transmitindo a voz de pessoas compradoras através de dados.

Usando informações de navegação na plataforma, construímos algoritmos de previsão de demanda, indicando o engajamento esperado de pessoas compradoras na nossa vitrine e fechando a lacuna no cálculo de equilíbrio no marketplace.

Além de ajudar no fornecimento da informação de demanda esperada, o modelo de previsão de demanda serve de fonte de dados para modelos de recomendação no site, junto com o serviço de comparáveis derivado do modelo de clusterização. Assim ajudamos clientes na busca dos seus apartamentos dos sonhos.

Por fim, nossa última alavanca (ativa) de demanda é mídia de performance (propaganda em redes sociais e outros sites). Dada a complexidade do problema, começamos construindo uma plataforma de experimentação (o MASP!) para validar hipóteses de melhorias nas nossas campanhas e diminuir nosso CAC (custo de aquisição de clientes).

Contudo, não enxergamos CAC como algo necessariamente ruim, mas como um investimento que precisa ser utilizado de forma inteligente, como uma alavanca de demanda em prol de clientes do marketplace. Assim, fechamos o ciclo de oferta e demanda na Loft.

Ajudando a pessoa vendedora a alavancar a liquidez do seu imóvel

Fechamos nosso ecossistema de algoritmos com um produto de dados responsável por expor inteligência para ajudar anunciantes no nosso site, baseado em algoritmos construídos para entregar recomendações para alavancar a liquidez dos seus imóveis.

Nesta missão, logo encaramos nosso primeiro desafio. Tínhamos a hipótese de que uma das principais alavancas para o sucesso de um anúncio é a qualidade das fotos do imóvel, sendo isso uma das principais formas de capturar a atenção de pessoas compradoras. Portanto, precisávamos caracterizar as imagens no nosso site de forma precisa e escalável.

Assim nasceram nossos modelos de "tagueamento" de imagens. Usando dados rotulados por Lofters, construímos modelos que conseguem caracterizar as imagens dos apartamentos (em termos de qualidade, ângulo, e outros), insumo que passou a ser importante para diversos times na Loft.

Com os dados de caracterização das imagens e diversas outras fontes como clusterização, demanda e precificação, criamos um “perfil competitivo” do anúncio para sugerir melhorias visando alavancar a liquidez do apartamento.

Pensamentos finais

Esperamos que tenha gostado da leitura e que esse documento tenha ajudado você a conhecer mais um pouco sobre o time de Data & Decision Science na Loft. Ressaltamos a importância de criar as condições certas — alta intensidade de aprendizado, cultura de compartilhamento de conhecimento e foco em resultado de negócio — para o surgimento desse ecossistema, tema que vamos abordar mais em um próximo post.

Valeu!

Ficou com vontade de fazer parte da Loft?

Se inscreva nas nossas vagas! — https://jobs.lever.co/loft/

Data & Decision Science Loft: nosso ecossistema de algoritmos was originally published in Loft on Medium, where people are continuing the conversation by highlighting and responding to this story.

Aqui na Loft, estamos em contínua construção do nosso entendimento do mercado imobiliário, juntando dados com a experiência dos nossos parceiros e fundadores. Gostamos de exercitar nossa criatividade explorando e adaptando conceitos existentes com dados e algoritmos.

Um dos conceitos mais estabelecidos no mercado imobiliário é o de bairro. Cada pessoa tem um relacionamento diferente com esse conceito: alguns se importam pouco com qual bairro morar, desde que seja perto do trabalho; outros nasceram, cresceram e criaram raízes no mesmo bairro (alô Mooca!), ou se identificam com o estilo de vida dele (Santa Ceciliers ❤).

Na Loft, além de identificar quais bairros atendem aos diferentes perfis dos nossos clientes (que fica para um próximo post!), nos interessa entender bairros por aspectos mais funcionais, como metragem, densidade demográfica, arborização e outros. Em particular, despertou nosso interesse saber onde estão os bolsões de valor imobiliário na cidade. Ou seja, descobrir como dividir a cidade em partições tal que os preços dos imóveis em cada uma delas sejam muito parecidos.

Assim, exercitamos nosso entendimento do mercado ao criar um novo conceito de bairro focado em valor, comparando as novas linhas de bairros com as existentes.

Simplicidade e eficácia: árvores de decisão

Uma árvore de decisão é um algoritmo de machine learning que aprende através da indução de regras, construindo um particionamento da base de treino visando a redução de dispersão da sua variável alvo. No nosso caso de particionamento de bairros, esse algoritmo encaixa muito bem, pois estamos querendo justamente isso: dividir a cidade em partições (quebras a partir de dados de latitude e longitude) tal que os imóveis nessas partições apresentem baixa dispersão de preço por metro quadrado.

Explicamos intuitivamente (e visualmente, em menos de 8 minutos) como o algoritmo funciona neste vídeo do Loft Storm. De uma maneira resumida, chegamos no resultado da seguinte forma:

• Preparamos uma base de anúncios imobiliários que contém estimativas de preço por metro de apartamentos em 2018. Na Loft, além de bases de anúncios, utilizamos diversas outras fontes de dados para precificação, incluindo fontes mais atualizadas e precisas como matrículas e transações no nosso marketplace. Para os propósitos do nosso exercício, contudo, os dados coletados são suficientes.
• Cada linha na nossa base de dados é um prédio, contendo a mediana do preço por metro dos apartamentos anunciados no prédio:

• Aplicamos uma transformação de rotação nos dados de latitude e longitude. Assim a árvore pode fazer quebras diagonais.

• Ajustamos a árvore nos lat-longs rotacionados, com a restrição de termos no mínimo 200 prédios por partição.

• Publicamos o resultado usando o kepler.gl, que nos permite enxergar as partições e o preço por metro médio da partição.

Resultados

Abaixo publicamos o mapa com os resultados, onde você pode alternar entre a camada dos clusters (partições) e preço médio por cluster. Vale ressaltar novamente que usamos uma fonte de dados de anúncios de 2018. Portanto, os preços nesse mapa podem não refletir exatamente os preços de mercado. Apesar disso, a comparação entre bairros mantém válida a utilização dessa base para esse exercício.

Large-scale WebGL-powered Geospatial Data Visualization Tool

Exemplo: Jardins

Cada um tem uma definição um pouco diferente, mas é comum nos referirmos como “Jardins” os bairros que se situam entre a Avenida Paulista, a Rua Estados Unidos, a Avenida Rebouças e a Avenida Brigadeiro Luiz Antônio. A Loft nasceu nesses bairros. Nosso primeiro escritório e primeiros apartamentos transacionados foram nessa área. Temos, portanto, muito carinho por essa região, e muita experiência operando nela. Geralmente avaliamos o comportamento dos nossos algoritmos nos Jardins para ver se eles “fazem sentido”.

É comum dividir os Jardins em dois bairros diferentes:

• Jardim América, entre a Avenida Rebouças e a Avenida Nove de Julho
• Jardim Paulista, entre a Avenida Nove de Julho e a Avenida Brigadeiro Luiz Antônio

O algoritmo, em contrapartida, dividiu os Jardins em 4 partições diferentes, dividindo tanto o Jardim América quanto o Jardim Paulista em dois (a linha central passa na Alameda Casa Branca, quase na divisão dos dois bairros):

Quando mudamos a visualização para o preço por metro de cada uma das partições, vemos um padrão de valoração mais alta na região do Jardim América e nas proximidades da Rua Estados Unidos, padrão que se ouvia falar muito no mercado, e agora foi descoberto e confirmado por meio de dados :).

Conclusão

Por fim, deixamos essa última visualização do preço médio por partição em São Paulo. Vemos diversos padrões interessantes: Vila Nova Conceição e uma parte de Moema se juntando em uma única zona, diferentes partições em Higienópolis, aumento de preços ao redor do Parque Ibirapuera, uma mini-zona de alto valor entre Consolação e Bela Vista, e outros.

Fique a vontade para explorar o mapa que disponibilizamos anteriormente e criar seu entendimento do mercado junto com a gente!

Seguimos usando ciência de dados para lidar com a complexidade do mercado imobiliário, reinventar o consumo de moradia, e eliminar, para os nossos clientes, a fricção no processo de mudar de lar. Mudar faz parte da gente!

Ficou com vontade de fazer parte da Loft?

Se inscreva nas nossas vagas! — https://jobs.lever.co/loft/

Encontrando zonas de valor em SP com machine learning was originally published in Loft on Medium, where people are continuing the conversation by highlighting and responding to this story.

Loft’s mission is to disrupt the real-estate market, transforming painful and bureaucratic processes into simple and delightful experiences. There’s plenty of evidence that we’re going to get there, and a lot of reasons for anyone to accept this challenge: we've just raised our Series C, and entered unicorn territory after just one and a half years since foundation; Florian and Mate experiences as second-time entrepreneurs speak for themselves; our team and investors mean business, and we have been elected by LinkedIn as one of the Top 3 startups to work in Brazil.

However, on top of all that, I want to add my view on specifically why you, as a Data Scientist, should join Loft. You and I know that Data Science has yet to fully mature in industry, and while some companies might seem very cool at first, data science work in them may be very frustrating.

Therefore, I want to clear things up and show you that at Loft you’ll do meaningful, stimulating data science work that will actually drive business decisions and help Loft achieve its goals.

1. You can focus on doing your best Data Science work

We have excellent infrastructure and a world-class data engineering team, with whom you’ll work in a close partnership. This means you’ll spend less time cleaning and tidying data and more time performing analysis and driving business decisions. Our stack (AWS, Databricks, Spark, and others) allows us to use state-of-the-art machine learning models and ship them company-wide with little overhead.

You’ll have freedom to choose a setup that works for you. That includes choosing between Mac or PC, using a second monitor, etc. You can work at your desk or at our rooftop, with one of the best views in São Paulo. You’ll book meeting rooms seamlessly through Google Calendar, and will have brainstorm sessions using one of many whiteboards across the office. You’ll have fruits, snacks and coffee ready for you at any time.

2. There’s a strong, company-wide buy-in and data-driven culture

Loft’s goal is to leverage technology and data to disrupt the real estate market. Thus, being data-driven is one of Loft’s core values, so you won’t be hindered by politics and lack of buy-in. If you can use data to convince people of making a decision (which should be what you’re best at), there’s nothing else to stop you from having an impact.

Loft spent a small fortune buying data and partnering with companies that can bring us new data capabilities. We have access to real transaction values (which are very hard to get here in Brazil), which come in the form of a messy and expensive document called Matrícula. We invested a lot of our engineering team’s time to build the stack to process Matrículas and transform them in valuable, structured data ready for analysis. We acquired Spry, a company that works similarly to Amazon’s Mechanical Turk, but for data collection in the real world: we pay people to physically go to neighborhoods and collect information about real-estate demand, demographics, and more.

Florian, Loft co-CEO, even joked that “sometime we’ll need to take losses in some transactions, so the algorithms can learn”. Even though this is a joke and it’s very unlikely to happen, I’m confident that the company’s willing to experiment in order to learn and make the best decisions.

3. You won’t work in an isolated team

Loft adopts Spotify’s engineering model with Squads, Tribes, Chapters and Guilds. On your daily routine, you’ll work in a Squad, alongside a multidisciplinary team of business analysts, designers and developers.

Additionally, you’ll be very close to your colleagues at the Data Science Chapter, who will have your back on technical issues. Our team has a mix of backgounds in engineering, statistics, economics and business. Diversity matters to us: we always take into account how "decorrelated" our candidates' thinking is from the rest of the group. As a company, we've set a goal of fulfilling 30% of our tech positions with women for the first quarter of 2020, and we plan to add other types of diversity in the near future.

The DS Chapter holds a number of rituals for decision-making and training. The first is our Study Group: weekly training sessions to explore foundational machine learning, statistics, business and computer science concepts. We also encourage Data Scientists to explore new algorithms, read papers, and have small research projects through our LobsterWorks initiative, staying in touch with the state-of-the-art and sharing knowledge with colleagues. Finally, we hold weekly all hands meetings to discuss issues common to all DSs, using the High Stakes Meetings format, to keep them productive and inclusive.

Loft holds several internal and external meetups. In particular, we have an internal event called Tech & Beers (yes, you got that right!) where we discuss bleeding edge issues in product, design, engineering and data science. We're also building a culture of hosting internal hackatons (HackDays) for people to work on a side project outside their squad priorities and solve different problems.

4. You’ll solve the hardest, most interesting problems in real-estate

You’ll solve problems that no one in Brazil (and perhaps around the world) has solved before. We built the first Automated Valuation Model trained with real transactions in Brazil, and we’re just beginning. Here are some examples of questions you’ll try to answer:

• How to find comparable houses when we have hundreds of dimensions to compare them? What criteria should we use?
• What index should we use to bring older transactions values to current values, at the most granular level possible?
• What should be Loft’s price premium, in order to add liquidity to the market and reduce friction in the process?
• How do we account for risk in our models, given that price predictions won’t be perfect?
• How can we explain prices generated by a black box model? Or, can we build interpretable models that achieve the same level of accuracy?
• How long it takes for an apartment to sell? How price plays a role in liquidity?
• How many kinds of apartments exist? What makes apartments "different" from one another?
• Which apartments should be in our portfolio so we meet our business goals? In which markets we should bet?
• How to help clients find the apartment of their dreams, through our website? How to recommend apartments for the client to visit?
• How to engaje brokers in our platform, supplying intelligence to help them make more deals?

And much more! Loft is just getting started and the possibilities for new business models (and new questions that you can answer using data science!) are endless.

What are you waiting for?

If this article made sense to you, I strongly advise you to apply to our Data Scientist positions and come build the future of real estate with us. We’re looking forward to it!!!!

Hope the article helped! If you have any questions, please don’t hesitate getting in touch with me or the crew.

Why you should join Loft as a Data Scientist was originally published in Loft on Medium, where people are continuing the conversation by highlighting and responding to this story.