Es de esperarse. Son demasiadas palabras de moda en una sola frase…

El propósito de este artículo es desglosar esa frase y convencer a nuestros queridos lectores (en especial aquellos ajenos al mundo de ML) de que las posibilidades de aplicación son variadas y de alto impacto para la experiencia de nuestros usuarios compradores, como también de nuestros asesores inmobiliarios (agentes vendedores).

Arranquemos y vamos por partes.

0. ¿Cómo así que un marketplace inmobiliario?

A veces cuando me preguntan qué es LH y no tengo mucho tiempo para explicar, digo simplemente que somos una inmobiliaria digital, es decir, una inmobiliaria con una componente tecnológica muy significativa. El concepto de inmobiliaria es más o menos claro para la gente de a pie. Gráficamente es algo así:

El objetivo de una inmobiliaria tradicional es hacer el puente (intermediación) entre un comprador de vivienda y un vendedor.

Sin embargo, esta diagrama no captura la dimensión completa de las necesidades que busca suplir La Haus. La Haus es un marketplace inmobiliario, que podría representarse mejor como algo así:

Un marketplace inmobiliario es una plataforma (digital) que conecta y facilita la interacción entre al menos tres participantes. Por un lado está la oferta de vivienda que, como veremos, es abundante pero tiene problemas de calidad en la información disponible sobre ella. Luego están los asesores inmobiliarios, que deben conocer la oferta y acompañar al posible comprador en la transacción. Finalmente está el usuario comprador que necesita acceso directo a una base de amplia de oferta de calidad, herramientas de búsqueda inteligente para no perderse en ella y, muy posiblemente también, el acompañamiento de un asesor que entienda sus necesidades. En una plataforma inmobiliaria como LH, el usuario comprador es el rey y es hacia quien la mayor parte de los esfuerzos de desarrollo de tecnología y producto van dirigidos.

1. Formulando preguntas para mejorar la experiencia de compra de vivienda

Para cada uno de los elementos en el diagrama anterior hay varias preguntas que quisiéramos responder de forma cuantitativa.

Para la vivienda (1):

• ¿Qué tan atractiva, comercialmente hablando, es una vivienda dada para un usuario comprador promedio?
• ¿Qué tan buena es la calidad de la información de una vivienda?

Posibles respuestas a cualquiera de estas preguntas serían: baja, media, alta, o una calificación numérica de 1 a 10. Dependiendo de la respuesta a estas preguntas podríamos, por ejemplo, promover algunas viviendas en los resultados de búsqueda que presentamos a los clientes.

Es decir, una pregunta cuantitativa de este tipo y su respuesta no tienen valor en sí mismas. El valor vendrá de las acciones comerciales o características de producto que se puedan derivar de ellas y que conduzcan a mejorar o hacer más eficiente la experiencia de compra desde la perspectiva del cliente.

Para el asesor (2):

• ¿Qué tan eficiente es un asesor?
• ¿Cuál es la calidad de su servicio?

Nuevamente, las respuestas a estas preguntas podrán ser usadas para gestión de la fuerza comercial y el control de calidad del servicio en general.

Para el usuario (3):

• ¿Qué tan listo está un usuario para comprar una vivienda?
• Asumiendo que ya está listo, ¿qué tan alta es su intención de compra o probabilidad de que sí realice una compra a través de nuestra plataforma en un horizonte de tiempo dado?

Por otro lado, para cada una de las interacciones entre participantes en la plataforma, es decir, para cada uno de los los lados del triángulo del diagrama anterior, también podemos formularnos preguntas.

Para la interacción asesor — vivienda (A):

• ¿Qué tanto conoce el asesor X la vivienda Y? Es decir, ¿cuál es la probabilidad de que un asesor X sea exitoso vendiendo la vivienda Y?

Para la interacción asesor— usuario (B):

• ¿Qué tan probable es que el asesor X cierre una transacción de compra de vivienda con el usuario Z, garantizando además un excelente experiencia para dicho usuario?
• ¿Entre todos los asesores disponibles para atender un usuario dado, cuál sería el más adecuado (o mejor ‘match’) para atenderlo?

Y finalmente, tal vez las más cruciales son las que se refieren a la interacción usuario — vivienda (C):

• ¿Qué tanto se ajusta una vivienda Y a las necesidades del usuario Z?
• ¿Entre los miles de viviendas disponibles en La Haus, cuáles son las 5 o 10 que más alta probabilidad tienen de convertirse en la vivienda de los sueños del usuario Z?

Un sistema que responda la pregunta anterior es lo que se conoce como un motor de recomendación y es una componente esencial en cualquier e-commerce así como en plataformas de distribución de contenido, (p. ej. Netflix, Google o Spotify.)

Ahora, en una inmobiliaria tradicional estas preguntas las responden los asesores o los gerentes comerciales a partir de su intuición. La intuición humana, al ser el destilado de años de experiencia, es en general una buena guía, pero tiene dos problemas grandes: 1. Está sesgada por la experiencia subjetiva de una sola persona, y 2. tal vez lo más grave, es sumamente costosa de adquirir o construir.

2. ¿Qué es el machine learning y qué le aporta a un marketplace inmobiliario?

El machine learning (ML), también conocido en español como aprendizaje de máquina, aprendizaje automático o aprendizaje estadístico, es un conjunto de técnicas y algoritmos que permiten generar, de forma numérica y objetiva, respuestas a preguntas como las de la sección anterior. Todas las técnicas del aprendizaje estadístico tienen en común que toman como entrada un conjunto de datos históricos con muchos casos ejemplo. Este conjunto de datos se conoce como la tabla o base de ‘entrenamiento’.

En la rama del aprendizaje estadístico que se conoce como aprendizaje supervisado, cada caso de ejemplo consiste de datos (históricos) acerca de una entidad de interés, p. ej. un usuario comprador, y que además vienen acompañados de información sobre un resultado de interés para el negocio, p. ej. si el usuario realizó una visita (física) a algún inmueble o si cerró una transacción de compra de vivienda. La palabra ‘supervisado’ no significa que este tipo de algoritmos requieran supervisión constante para operar. Esta simplemente hace alusión a que el algoritmo recibe información histórica sobre el ente principal, en nuestro ejemplo el usuario, pero también la ‘respuesta correcta’, no conocida a priori, sobre qué pasó luego con ese usuario, p. ej si visitó o no, o si compró o no.

En el enfoque estadístico tradicional los datos conocidos a priori acerca de la entidad de interés (usuario, propiedad o asesor) se conocen como covariables, variables explicativas, o variables independientes, regresoras, predictoras o más coloquialmente como las Xs (equis). Por otro lado, a la variable resultado, la que indica si hubo visita, compra, click en un resultado de búsqueda o en una recomendación, se le conoce tradicionalmente como la variable dependiente o variable objetivo, target variable, o más coloquialmente, como ‘la Y’ (ye).

Para fijar ideas veamos las que podrían ser las variables predictoras Xs y la variable objetivo Y, para un usuario comprador en La Haus:

Ahora, ¿qué es lo que ‘aprende’ un algoritmo de ML?
En resumidas cuentas un algoritmo de ML lo que hace es inferir una relación funcional entre X y Y. Es decir, el algoritmo aprende una función f para la cual se cumpla la siguiente ecuación:

Hay que admitir que esta descripción es bastante abstracta. Para entenderla mejor, recordemos que una función no es más que una regla que produce un resultado (respuesta a una pregunta) a partir de unas variables de entrada. Esta regla puede implementarse de formas muy diversas. Una de las más fáciles de entender es un árbol de decisión. Un posible árbol de decisión para estimar si la probabilidad de que un usuario compre vivienda con LH es alta o baja, sería el siguiente:

Como vemos, la estructura de una árbol de decisión es simple. Consiste de un conjunto de nodos de decisión. En cada uno de estos se formula una pregunta con respuesta binaría (Sí / No) que se responde simplemente mirando una de las variables predictoras X y posiblemente un valor de referencia o umbral.

Para generar una respuesta, el árbol de decisión se recorre de arriba abajo, empezando por el nodo que aparece más arriba y visitando otros nodos en sucesión. En cada nodo se formula una pregunta, se responde a partir de una de las variables X y el umbral, y se escoge el siguiente nodo (rama) según si la respuesta es Sí o No.

Un árbol de decisión es una implementación concreta de una regla, o función, que calcula una respuesta Y, en este caso la probabilidad de compra, a partir de las variables de entrada X, en este caso los datos de un usuario.

En los tiempos previos al ML, la construcción un árbol de decisión era labor de un humano, experto en el problema dado. Para el caso que venimos describiendo, este humano sería un gerente comercial o un asesor senior. Una vez este experto hubiera especificado completamente un árbol, ya era tarea de un programador implementar este árbol como un función, en código. Esto permitía aplicar la evaluación o scoring a una base de nuevos clientes, a un costo marginal insignificante.

Armados de técnicas de ML es posible mejorar este enfoque. El algoritmo de ML es capaz de construir un árbol de decisión automáticamente, a partir de una base de entrenamiento que contenga unos cuantos miles de ejemplos, y lo hará de manera óptima, es decir, construirá el árbol que mejor se ajuste o aproxime los datos, en un sentido cuantitativo preciso y objetivo.

Una nota sobre terminología

El árbol producido por el algoritmo de ML para el ejemplo anterior, es un ejemplo de un modelo de ML. Como comentamos arriba un modelo no es más que una función que recibe una entrada (datos de un usuario, asesor o vivienda ) y produce una salida. A la salida se le llama generalmente predicción pese a que en muchas ocasiones no se refiere a un hecho en el futuro, sino simplemente a un dato desconocido a priori.

No hay que confundir un modelo de ML con un algoritmo de ML. Este último término denota un procedimiento genérico que recibe como entrada un conjunto de datos de entrenamiento (una tabla de ejemplos) y produce como resultado un modelo de ML. De esta forma un algoritmo de ML es en realidad un algoritmo de orden superior, o meta-algoritmo, ya que su salida es otro algoritmo.

3. Conclusión

En este artículo hemos visto cómo el ML nos ayuda a construir modelos, a partir de datos históricos, de forma eficiente. Estos modelos sirven para responder, de forma objetiva, preguntas cuantitativas sobre usuarios, agentes y propiedades en un marketplace inmobiliario. Las respuestas obtenidas a partir de estos modelos, añadirán valor al negocio de LH en la medida en que estas conduzcan a hacer las experiencias de búsqueda y compra de vivienda más eficientes y placenteras para usuarios compradores y asesores.

En una próxima entrega haremos un doble-clic a los diferentes modelos de machine learning ya desarrollados en LH y a las preguntas de negocio que ellos responden.

En La Haus y LaHaus.mx seguimos trabajando de forma incansable para ser los mejores en producto y tecnología en Colombia y Latinoamérica. Cualquier contribución, corrección o sugerencia a este artículo será recibida con mucha gratitud. Me puedes escribir a mateorestrepo@lahaus.com. Si quieres aplicar a un trabajo con La Haus este es link para hacerlo.

Aplicando machine learning en un marketplace inmobiliario was originally published in Building La Haus on Medium, where people are continuing the conversation by highlighting and responding to this story.

This series of three posts is meant to serve as an accompanying script for the Prezi presentation “Data science, AI, ML, DL and all that jazz”. In the first part of the series, we delved into the concept of data science. This part is focused on the presentation’s second chapter which aims at giving a non-technical explanation of what AI is and how it differs from ML, give lots of examples, and also explain what AI isn’t (yet). As you surely have noticed, AI is nowadays the buzzword of choice of almost every company out there who is selling data analytics and ML related services, such as Yuxi Global :).

The presentation can be found here:

What is AI?

This chapter is divided into five sections:

1. Three popular AI applications. In which, we exemplify how AI is used in today’s world, before venturing to give a formal definition.
2. Defining AI. Aimed at the formalism-obsessed, this section comments on the difficulties of giving a satisfactory definition of “Artificial Intelligence” and then attempts to give a one that is pragmatic.
3. A catalog of AI sub-problems. Armed with the intuitions developed through looking at the applications and the concepts introduced in the definition, we proceed to decompose AI into several sub-fields, according to the types of problems each of these tries to solve.
4. Limitations of AI. In this short section we point out that AI, at least in the narrow form that we have managed to develop so far, is not all-powerful and has serious limitations.
5. What AI isn’t. In this final section, we carry out some expectation management aimed at decision takers who have watched too many science fiction movies about robots taking over the world and enslaving humans.

Three popular AI applications

Rather than jumping right away into a definition of AI in an abstract, academic and dry way, we would like to first review three popular applications of AI. Some of these might as well be considered plain machine learning (ML) and not really that “intelligent”. However, being ML a subset of AI, any ML application is also an AI application, in the same way that cooking desserts is an example of the broader category of “cooking”. These applications will serve to introduce some of the general concepts involved in our definition of AI.

Application 1. Self-driving cars.

Autonomous vehicles are probably the pinnacle of AI nowadays. If you think about it, aself-driving car is actually a robot, although it might not look like one. It perceives (gathers data about) its environment through a rich set of sensors (cameras, radars, lidars, GPS) and it is also endowed with actuators that allow it to perform actions (steering, accelerating) to change its own situation in — or otherwise affect — the environment. Self-driving cars constitute a prime example of the general concept of “autonomous agents”, which will be key in properly defining AI later on.

Beyond sensors and actuators, which roughly correspond to the input and output layers, there is an intermediate processing layer that interprets the raw data coming from the sensors and turns it into information/knowledge that is ultimately used to solve the problem of driving safely and taking you from A to B. This layer is of course what gives a self-driving car its smarts and combines quite a few key AI techniques. To list just three, we have:

• Computer vision: This refers to a set of relatively low level techniques used to turn raw pixels captured by an optical or infrared camera into actual detection of lane lines, obstacles, pedestrians, deer, traffic signs, etc. This is really a pure ML problem, currently solved with Deep Learning techniques and specialized hardware (GPUs).
• Search and planning: These are employed, among other things to find the most convenient route from A to B, subject to constraints given by obstacles, traffic, road closings, speed limits, weather, etc… Incidentally, this is one of the problems solved by AI that cannot be solved through pure ML techniques, proving that there is more to AI than just ML.
• Decision making under uncertainty: as sensor measurements are not absolutely exact and the available information about the surrounding environment is necessarily incomplete, any autonomous engine is forced to make decisions under uncertainty almost all the time. This is not a common feature of computer systems pre-AI, or even something that most people would think computers can do. Luckily, during the last half of the 20th century there was a big deal of progress in the field of computational Bayesian statistics to enable inherently deterministic computers to deal with uncertainty.

Application 2. Personalized Content Recommendation

Although sometimes invisible, recommendation systems are virtually everywhere, starting from e-commerce sites — think Amazon — and going all the way to digital content distribution platforms such as Netflix and Spotify, as well as social networks. Whenever you see messages of the sort ‘based on your recent viewing / purchasing history, we think you would also like Y’ you can be sure there is a recommendation system in action.

The use of this kind of system has permeated even traditional content providers, recently turned digital. Most people don’t realize it but the front page of the online version of the New York Times is different for each reader. Recommendation algorithms determine the content each user sees in a personalized way.

The goal of a recommendation system is to sift through a vast sea of information or content items to select those that are most likely to be relevant to a particular user or subset of users (i.e. those residing in a location or belonging to a particular age segment). This improves the user experience and overall engagement with the platform’s content.

Technically, a recommendation system can be seen as an application of one of the following general AI techniques:

• Search: The system is tasked with finding the most relevant content subject to some user-specific constraints, which are usually derived from the past history of user interactions.
• Predictive machine learning: Given the record of past user interactions (examples), we tune a user-specific model that then allows us to make predictions about the likelihood that user X will like a (previously unseen) item Y.

Application 3: Computer vision

We briefly touched on this application as part of our discussion of self-driving cars. This is, however, a much wider field the has already started to revolutionize the manufacturing and security industries, among others. Much of the recent surge in popularity gained by AI is due to success in this area. To enrich your computer vision vocabulary we would like to draw distinctions between several terms that are sometimes used interchangeably by management folk:

• Object detection: This refers to locating, that is, drawing bounding boxes, around all instances of a particular class of object, e.g. pedestrian, in a given image. Variants of this can be used to find defects in images of manufactured pieces, for instance.
• Object classification: Given an image of a single isolated object, identify which class of object it is. In other words, is this a picture of a fruit, a person, a table, another car…?
• Face recognition: Given a picture of a face, tell me who it is.
• Image segmentation: Given a picture of a complex scene, separate the different components (sets of pixels) corresponding to each individual object in it, even if you can’t tell me what they are.
• Flow detection: Given a video, identify in each frame, the objects that are moving, and how fast…

Defining AI

An attempt at defining AI is fraught with cultural, conceptual and even philosophical difficulties, among which we can count:

1. Necessarily flawed depictions of AI from science fiction movies and the corresponding misconceptions they have generated in popular culture. We will comment more on these later in the section on what AI isn’t.
2. The recent resurgence of AI has brought with it a lack of precision if not downright abuse of the term by marketers and C-suite executives. This is in fact one of the main reasons for creating this presentation. In commercial settings, the term “AI” is often used to refer to many things that used to be called by other names at least in the narrower academic circles where they were developed. Some examples of these are:
   Machine Learning: As we have already pointed out, this is but a strict subset of AI.
   Business intelligence/analytics: not really AI, although it can serve to guide smarter decisions by humans.
   Expert systems: These are based on painstakingly collecting many if-then rules from human experts in a given domain and having a computer apply them as needed. Although they belong to the beginnings of AI, much has happened since their early development
   Some statistical tools: even classical statistical tools such as linear or logistic regression are sometimes labeled as AI, in the interest of marketing!
3. To add insult to injury this situation is worsened by the fact that even researchers of AI have no precise definition of their field of study. We cannot blame them if we consider that they are aiming at a moving target. Many products that were once considered prime examples of AI, such as OCR systems (that extract digital text from scanned documents or handwritten zip codes from letter envelopes) or spell-checkers are now considered by many to be “regular software”, not worthy of the “AI inside” label. As an old joke claims “AI is all cool things that computers can’t yet do”. As all jokes, this one is rooted in the commonly held opinion that, as soon as a reasonably efficient algorithmic approach or computer architecture for solving a problem is designed, said problem automatically falls out of the realm of AI and gets incorporated into the field of “classical” software engineering.
4. Last but not least, we shall mention that the mere presence of the word intelligence in “Artificial Intelligence” opens up a semantic and philosophical can of worms that plague any attempt at satisfactorily defining the phrase. Millennia-old open questions about the “true nature” of consciousness, thought, mind and soul immediately pop up. Of course, we won’t even try to attack these here. Suffice it to say that any coherent and accepted definition of AI, as the one we will present in the next section, will necessarily avoid touching on the dicey topic of what “biological” or “organic intelligence” actually entails.

A(n imperfect) definition

Here is a pragmatic definition of AI. Due to the philosophical difficulties just mentioned, it is necessarily imperfect. It is operational in the sense that it focuses on the observable attributes that intelligent agents must exhibit, rather than on what intelligence means.

Definition: AI is the branch of computer science tasked with the design and construction of intelligent agents.

An intelligent agent is in turn any agent that is (mostly):

• autonomous (that is, able to perform tasks in a complex environment without constant guidance by a user) and
• adaptable (able to improve performance by learning from experience)
• able to sense and efficiently abstract features of its environment (perception)
• able to act “rationally”. In this context rational behavior is any that is directed towards maximizing a quantitative and objective utility function or performance measure.

Two corollaries of this definition are that intelligent agents necessarily comprise (and can almost be reduced to):

• programs that make smart decisions — where “smart” is just a simpler word for rational in the sense made precise above.
• systems able to mimic (some aspects of) human cognition.

A list of AI sub-fields

Armed with the previous definition, however imperfect, we are better equipped to review and understand some of the sub-fields of AI. Here is a short catalog:

• Search and Information retrieval: This doesn’t just refer to internet search, e.g. Google or Shazam, but to a broader set of problems in which our intelligent agent — called in this context “the search engine” — has to navigate an environment consisting of a large collection of data items to maximize a utility function — a quantitative measure of each item’s relevancy to a query.
• Planning: This is the problem of devising a plan or sequence of actions that lead a complex system from an initial state A to a desired state B, subject to a set of constraints. A non-trivial example of this could be that of creating a fair and optimal landing schedule for a set of planes at a busy airport during hours of congestion.
• Natural Language Processing (NLP) and speech recognition: these are the key ingredients of any chatbot or conversational digital assistant. The key capability here is the “understanding” of utterances or written phrases input by a human to detect their intent (ask a question, place an order, etc.) In this context, the intelligent agent is the bot and the environment is the chatroom, on one side, and the set of information services, DBs or APIs used to answer the question or execute the human’s request, on the other.
• Automated reasoning: These are systems that try to carry out logical thinking or deduction from a set of premises to a conclusion. A few years ago, IBM Watson became famous for winning Jeopardy against previous human champions. To be sure, the game wasn’t just about reasoning but also contained a heavy information retrieval component. More recently, a so-called “legal AI” was able to spot issues in NDA agreements, with higher accuracy than a set of experienced lawyers. Granted, this example is also on a gray zone between natural language processing and automated reasoning.
• Computer vision: We discussed this sub-field at length as the last of the three prototypical applications in the first section of this post.
• Any machine learning (ML) problem: Last but not least we should mention there is a large class of problems that don’t fall directly in any of the sub-fields above but are solvable through ML techniques. To be able to apply ML, the key ingredient is having lots of data in the form of examples from which generic algorithms can spot patterns and later make predictions. We will have a lot more to say about ML in the third post of this series. So, stay tuned!

Limitations of AI

As we hinted at in a previous section, the word “intelligent” is problematic and even misleading. A computer vision system that is able to detect melanoma from a skin photograph can be called intelligent but it is not capable of performing any other task that an intelligent person would do.

This is why the more pedantic among us make a distinction between narrow and general artificial intelligence. An AI system is called narrow if its ability to solve one problem tells us nothing about the ability to solve another problem, even if, from a human’s perspective, both problems are closely related. The key observation is that narrow AI systems cannot transfer general principles from one problem to another.

So far we have only been able to develop narrow AI. Artificial general intelligence, often abbreviated AGI, refers to machines that can handle any intellectual task and that can transfer knowledge or skills from one task to another. Unfortunately (or fortunately?) AGI is still in the realm of science fiction. There is little consensus as to whether AGI will be achieved in the foreseeable future. Despite the impressive advances in the field of deep learning for things that not far ago where considered impossible, (i.e. AlphaGo), it is the opinion of this researcher that AGI is at least 50 years away, if not more..

In conclusion, the main limitation of AI is that AI agents are typically only able to optimize their decisions and actions to achieve a singular goal. The human element still plays a key role in understanding the context of this goal, prioritizing work across different goals, and communicating the outcomes of AI-powered models effectively.

What AI isn’t (yet…)

To end this post, here is a list of things AI isn’t — at least yet:

• Robots that talk with a metallic voice (and that are able to scheme against humans). E.g. Terminator / Skynet.
• Conscious / feeling machines: No matter how good some machines might become in the near future at imitating feelings, we shall not forget that we are still far from replicating consciousness in digital computers. For starters we are not really sure what consciousness entails.
• Able to replace any human at any job. Although there is an increasing number of jobs that are being effectively automated, there is still a large array of more creative or strategically oriented jobs that won’t be taken over by machines in the medium to long term or possibly ever.
• General intelligence: able to proactively abstract knowledge and skills from one domain and transfer it. See the previous section.
• Able to (significantly) update themselves proactively/freely. Despite adaptability being one of the defining characteristics of AI agents, significant updates that allow them to tackle problems that are qualitatively different from the ones for which they were originally designed, are still out of their reach.
• A countable noun. Advise: do not talk about “an AI”. AI is a scientific discipline, like mathematics or biology.

Conclusion

In this post we have given a high-level but detailed overview of the field of AI, its possibilities and limitations, from a “business” point of view. To put some order in our ideas we also attempted a precise and practical definition of AI, that focuses on observable and objectively quantifiable goals rather than on hard questions about the mind and the true meaning of intelligence.

Stay tuned for the third and last post in this series in which we will narrow our focus on the areas of machine learning and deep learning, through which AI has regained preponderance in recent years.

References

• A free online introduction to artificial intelligence for non-experts
• Artificial Intelligence Course | Udacity
• AI vs. Lawyers: The Ultimate Showdown - LawGeex
• 20 top lawyers were beaten by legal AI. Here are their surprising responses
• Geoffrey Hinton and Demis Hassabis: AGI is nowhere close to being a reality

Explaining data science, AI, ML and deep learning to management — a presentation and a script —… was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

If you are a data scientist, a machine learning engineer, an AI specialist, or however the heck you want to call yourself, chances are that at some point in your professional career you have encountered at least one manager that doesn’t understand the differences between some of the concepts listed in the title. After all, why should he?! Chances are that you have struggled to explain those differences without having to employ technical jargon, and you have alienated your already confused manager by doing so.

Fear no more! We, at Yuxi Global, have developed a nearly perfect solution for this kind of situation. As the Head of Data Analytics at Yuxi Global, I was tasked with the construction of a Prezi presentation to aid in explaining what “Data Science”, “Artificial Intelligence”, “Machine Learning” (ML) and “Deep Learning” (DL) mean to medium and upper management folks. The ultimate goal was to help them guide their strategic as well as commercial decisions about what services and kinds of solutions we want to develop for clients in these areas. After doing this we thought it would be valuable to share our work with the world, to save every data scientist out there many hours of intense labor with a presentation software.

The presentation link is just below this paragraph. Further below, we also share a script that presents the material in a fully linear fashion. Of course, you can pick bits and pieces on which to focus the presentation and jump between them. That was one of the reasons for choosing Prezi as our presentation solution.

Presentation’s Script for “Analytics / Data Science / AI / ML / DL and all that jazz” — Part 1 of 3

Part 1: What is data science / data analytics?

This part consists of two chapters. The first one focuses on trying to define the somewhat sneaky concept of “Data Science” by yielding not only one but three related definitions. The idea is that the presenter may choose the definition that best resonates with his own background and understanding of the subject or the one that will be most easily assimilated by his audience. The second chapter attempts to explain why data science is a tendency nowadays and it wasn’t before, which is odd given how natural or fundamental the concept of data science is in light of the definitions.

Chapter 1: Three definitions

Definition 1: Perhaps one of the best ways to understand “data science” is as a process, namely that of using data to understand things. Among the “things” we would like to better we can list:

• Your customers
• Your business processes
• How your product is consumed / perceived / liked by your customers.

The key words here are “data” and “understanding”. Data is the main input, understanding is the ultimate goal. Thus, data science goes far beyond business intelligence, that often stops at measuring and presenting what has happened without delving into why it happened or how to control or affect what will happen in the future. Another simple way to put this is that one lets the “data tell a story”. After, all humans have always tried to make sense of the world around them by creating narratives that connect the mere facts in logically coherent — if sometimes fantastic — ways, e.g. myths and legends.

Yet another way to put this definition into words is that data science equates to uncovering insights, patterns and trends that are *hidden* behind the data. Some people like this definition because it is short and mentions two nice buzzwords: “trends” and “insights”. The meaning of the former is pretty clear to me. Most people have a very distinct mental picture of what a trend is. However, insight is a much more abstract notion with a rather unclear meaning. This is illustrated by the fact that this word doesn’t have a distinct translation into Spanish. From the translations given you would think that it means “understanding” in some contexts and “knowledge” in others… What I do like about this wording is that it introduces the key notion that data by itself is just the beginning, and that there are important pieces of the puzzle hidden behind the data and which one wants to uncover.

This brings us to a further wording of the same concept, that I especially like: data science means transforming data into knowledge, that can guide decisions. A wonderful image to bring home the (for some people unexpected) distinction between data and knowledge is the “tomato analogy”. Here it goes:

The key insight here is that data in itself has practically zero value when it comes to making decisions. Data is no more than a, sometimes huge, collection of recorded facts about some entities, such as your customers, your sales transactions or your web interactions. Although it might cost you and your company a lot of money, time and effort, to manage this repository of facts, said cost does not translate automatically into value.

Information can be found at the next level, after adding a further layer of value on top of data, in the form of systematization or structure within a rich context. The business related aspect of your data only shows up here for the first time. Data becomes information, only insofar as it has a relevant meaning within the context of your business.

Now onto knowledge. The defining characteristic of knowledge is that it gives you a competitive advantage, or as the old apothegm reads “knowledge is power”. The way you gain such an advantage is by making better decisions than your competitors. In other words, one doesn’t have (valuable) knowledge about something unless said knowledge can inform your decisions and lead you to more desirable outcomes for your business.

The bit about wisdom is really a joke, in case that isn’t clear. As someone once said, anything learned with humor is never forgotten.

Definition 2: Data science is just science.

This definition emphasizes the fact that science has been around for centuries. In all that time, the recipe applied — a.k.a. the scientific method —has been one and the same:

1. Start by observing some phenomenon and record those observations, i.e. produce data.
2. Formulate hypotheses and questions and try to confirm and answer them, based on the data at hand. Strictly, speaking one can never validate a hypothesis, unless it is formulated as the negation of another one. The most one can hope for is to either reject or fail to reject it. However, I advise you to never discuss this subtlety in front of management as it will most likely lead to confusion and them erroneously thinking that you are a cheap philosopher.
3. Given enough data and confirmed hypotheses, one can sometimes venture to build a model. A model is a much more elaborate conceptual / mathematical artifact that further encodes or summarizes your understanding of the process or system in question. A strong enough model could even be used to generate forward looking predictions.
4. For some subjects of study, it is possible to design experiments (e.g. A/B tests) to really test the effects of certain conditions on your system in a reproducible way. This is really the only sensible way to establish cause-and-effect relationships.

In short, under this definition, data science is no more than the application of the centuries old (and proven!) scientific method to solve business problems. Incidentally, this definition helps us see that the “data” bit in “data science” is really redundant, as there has never been any science that doesn’t deal with data!

Definition 3: Data science as an umbrella term.

This is probably the least deep but also the most direct definition. It reduces data science to a direct sum of some of its tactical components. Under this definition, data science would be a recent umbrella term that groups up several crafts such as:

• Artificial Intelligence: Some people claim that good data science requires employing at least a pinch of AI. We don’t necessarily agree with this statement. We will have much more to say about this in Part 2 of this series.
• Machine Learning: This will be the subject of Part 3
• Statistics: really a tool of any scientific endeavor and the only way to turn a “soft” science into a “hard” science.
• ETLs, data cleanup: ETL refers to extraction, transformation and load or the stages previous to data exploration, formulation of hypothesis and construction of models. These are undoubtedly the least glamorous but undoubtedly necessary aspects of data science. However, it has become fashionable to classify these within the realm of “data engineering”.
• Data Visualization: essential for the “story telling” aspect of data science mentioned in the first definition, as well as for the successful communication of results derived from models or from simple initial explorations of the input data.
• Algorithms and Infrastructure: Aimed for efficient manipulation as well as processing of data. This is the “number crunching” involved in the second stage (after ETLs and cleanup) of almost any data scientific project. Algorithms and infrastructure are really a tool and a means yet not an end per se of data science efforts. Their development and theoretical study really belong in the Computer Science department.
• Efficient data storage and retrieval architectures: Quite similar to the previous one. This aspect relates to the technology (both hardware and software) to efficiently acquire, store and retrieve data. Data science necessarily finds support in this field.
• Business knowledge / understanding of the application domain: This is an often overlooked aspect of data science. I personally have experienced many instances of data science projects failing due either to the people from the “business side” taking for granted that the data scientist completely gets the business dynamics and all of its implicit assumptions and constraints for granted, or the data scientist himself completely disregarding these and focusing only on the data analytics methods.

Chapter 2: Why is data science “a thing” nowadays?

Given all that was mentioned in the previous chapter about data science being no more than the natural application of an old method to the field of business in order to make it better and get an advantage, the obvious natural question is then: why didn’t it happen earlier?

The answer in our opinion boils down to two closely related and relatively recent developments:

• The advent of the Internet economy and the explosion in mobile apps has caused a deluge of data waiting to be turned into value. As a result of this there are many companies or so called platforms, whose business consists of shifting bits around. Data is thus the main asset for many of these companies. The same applies to many traditional industries that have made the transition to the digital world — think newspapers, TV, hospitality, food delivery, insurance, etc…

• The sharp decrease in costs associated with data storage and processing. According to this source and this other source, the cost of a hard-drive per GB dropped from around $100 USD in 1997 to $1 USD in 2005 and then again to around $0.03 USD in 2017. This has but fueled trends such as “Big Data” and “distributed computing”.

To a lesser extent, the following factors, which we view as a consequence of the two we just mentioned, have contributed to the popularity of Data science and the resurgence of ML, AI, and Deep Learning (ML):

• The abundance of open-source tools. The convenience of sharing source code via public repositories has made open-source software the king of data science and almost completely supplant traditional expensive closed-source solutions such as SAS or IBM SPSS.
• The development of a wealth of innovative ML and DL algorithms The need to be able to process data sets that grow every year and to be able to extract value from non-tabular data (e.g. image or sound files) prompted this.
• On the hardware side, the availability of GPUs that are used to run the heavy computations required by deep learning algorithms. This was an almost accidental but fortunate side-effect of the popularity of gaming. Thanks gamers!

Conclusion for part 1

In this first part, we presented the general context of data science as seen through three different optics. We also commented on the environmental changes that directly contributed to data science applied to information centric industries.

Stay tuned for part two in which we shall explain what AI is and how it is really much more than just ML.

References

• Think Like a Data Scientist: Tackle the data science process step-by-step
• a history of storage cost - matt komorowski
• Cisco Visual Networking Index: Global Mobile Data Traffic Forecast, 2017-2022 Q&A

Explaining data science, AI, ML and deep learning to management — a presentation and a script —… was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

I recently attended the O’Reilly Artificial Intelligence Conference 2018. I am writing this post to share my experiences, and some of the lessons I learned with friends and colleagues. I comment on what I thought were the main themes in the conference: AI for customer support, Human in the loop and the impact of AI in the workforce, deep learning (DL) and reinforcement learning (RL), the cloud to deploy AI and automatic machine learning (Auto-ML). I finish with some thought-provoking quotes I collected from the speakers.

Also, here are the slides of many of the presentations.

The main themes of the conference

AI for customer support

A big fraction of the companies that are actually employing AI or ML to put their data to work are doing it in the fields of customer support and customization, or user experience. Notable examples of this are:

• Uber, who employed deep learning to develop natural language processing models to classify and suggest responses to customer tickets, as embodied in their Customer Obsession Ticket Assistant (COTA-v2).
• AT&T, who developed an LSTM network (a type of recurrent neural network) to produce a model of the sequence of touch-points (channels) that a customer will follow when interacting with the company, as well as the outcome of those interactions (whether they bought a product or service and through what channel).
• Blue Shield of California, who sees the chatbot they have put in place as an integral part of their omni-channel customer service strategy. They emphasized, however, that the most successful chatbots don’t work by themselves. They are always built on top of a good data infrastructure foundation that includes well developed APIs and micro-services.

Human-in-the-loop and the impact of AI on the workforce

There was a lot said about how AI shouldn’t (completely) replace humans but rather make their work more efficient, powerful and gratifying. Also, at least for the time being, humans are still instrumental in the development of AI solutions, due to the simple fact that most expert knowledge is contained in the brains of human experts, and they are the only ones capable of accurately doing the costly labeling of thousands of data points required to train ML algorithms.

There are also intermediate approaches, such as active learning, based on the idea of collaboration between humans and machines in a semi-supervised way. That is, let machines handle the easy routine cases while routing the difficult/edge cases to human experts. Of course, this is practical in applications in which difficult cases not amenable to automated decisions are easy to identify and constitute a minority of all cases.

Deep learning and reinforcement learning are all the rage

It’s probably no news to most readers that deep learning has positioned itself as the de facto architecture for ML models. Perhaps due to the conference happening in California and the fact that Google was one of the conference sponsors, there were tons of presentations about TensorFlow. By the way, after the conference I talked to some good friends of mine who are actually DL researchers and they very much vouch for PyTorch as a nicer framework for running DL experiments.

Peter Norvig gave a wonderful keynote presentation about the amazing applications of DL to scientific problems that non-DL-experts have been able to pull-off in recent years. There has been everything from applications in astronomy (gravitational lenses, exoplanet detection), passing through applications in medicine (assessing cardiovascular risk factors with CV, high-school students identifying cancer, and going all the way to agriculture (identifying sick cassava plants with a DL-based app, tracking cows).

There was a very nice talk by two young Microsoft researchers ( Danielle Dean and Wee Hyong Tok) on the “best kept secret” in deep learning: “transfer learning.” In a nutshell, transfer learning is a technique to leverage the knowledge encoded in a deep network that has been carefully and painstakingly trained by experts on a large data set, to solve a different problem. To give an example, a deep network trained to solve an object classification task can be refurbished with relatively little work to solve another computer vision task, such as that of texture classification. This is achieved by using the network as a featurizer and only retraining the last layer on a possibly small data set.

The cloud is (pretty much) the only place to deploy AI/ML solutions

Given the complex hardware of software configurations required by most advanced ML and AI solutions, it should come as no surprise that cloud-based infrastructures are preponderant.
One of the nicest contributions in this regard was the keynote given Levent Besik, from Google, in which he outlined the company's efforts to “democratize AI and to make it easy and useful for all developers & users”, regardless of their degree of expertise in the technical or scientific aspects of AI. Google’s offering includes three levels of abstraction. At the lowest level, one finds platform (ml-engine, dataflow and dataproc cloud services) and library (TensorFlow, Keras, Spark) level, for developers who want to create solutions “from scratch.” Then come the AI building blocks, which are essentially mature APIs to solve well defined problems, such as language translation, speech transcription, or image recognition. At the highest level we can find template solutions such as product recommendation engine or a customer contact center.

Amazon Web Services (AWS) offers a set of services and platforms, for developing and deploying ML, along the same general lines. I got the feeling, however, that some components of AWS offering, such as Amazon Sagemaker are more mature, complete, and overall better designed than their Google counterparts.

Auto-ML

If you are a data-scientist or AI solutions developer, there is a chance that you too will be replaced by a machine in the not too distant future. Auto-ML technologies promise to abstract and automate end-to-end the most complicated aspects of ML-model construction going from data-pre-processing and going all the way to hyper-parameter tuning, model selection and deployment. In a nutshell, an Auto-ML service takes as input a data-set (which might be dirty and doesn’t even have to include labels) and outputs an already deployed ML-model, complete with a REST-API and everything!

H2O is another company that showed interesting developments in this direction.

Given that I consider myself an ML practitioner (I abhor the term data-scientist), it shall come as no surprise to you that I am still very skeptical about Auto-ML technologies. It’s hard to imagine a fully automated process that manages to carry-out all the difficult experience- and intuition-based decisions involved in the creation of an ML model. Ingenious feature engineering, for instance, is a particularly tricky thing to automate. However, the recent advances in Auto-ML, as exhibited by Google, in which they have managed to develop neural nets that design other neural nets, make me doubt myself.

Quotes

To finish, I will leave you with some quotes that I found either interesting, illuminating, or provoking:

“Many of the AI things that we do have no business value beyond marketing” — Ben Taylor.

“Eight out of ten of all ML workloads run on AWS” — Hagay Lupesgo, AWS.

“[Out of] 20 million developers, 1 million are data scientists and 1000 deep-learning researchers” — Levent Besik, Google Cloud.

“Some companies have many ‘good ideas’ [for AI/ML use] but can’t quantify value. If the idea starts with ‘woulnd’t it be cool…?’, it’s probably a bad idea.” — Ben Taylor.

“82% of organizations are in some stage of considering adopting AI. It’s very easy to do pilots but very hard to deploy them… Executives are focusing on customer retention and satisfaction, customer acquisition cost reduction. ” — Manish Goyal, IBM

“Barriers to implement AI: Lack of skilled resources or technical expertise, Regulatory constraints. Legal, security, privacy concerns about the use of data and information.” — Manish Goyal, IBM

“In the future you are going to see affordable, intelligent, cloud-powered, personalized prosthetic devices” — Joseph Sirosh, Microsoft

“We’re moving into a world where machines and software can analyze (see patterns that were always hidden before); optimize (tell a plane which altitude to fly each mile to get the best fuel efficiency); prophesy (tell you when your elevator will break and fix it before it does); customize (tailor any product or service for you alone) and digitize and automate just about any job. This is transforming every industry.” — Tom Friedman ,Author of Earth is Flat, New York Times

“ All ML does is provide inferences. Scientific method to the business. Inferences are easy for business people to relate to… Data science doesn’t lend itself well to agile methodologies” — Carlos Escapa, AWS

Impressions and Lessons from the O’Reilly AI Conf 2018 was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this post and the next, we will look at one of the trickiest and most critical problems in Machine Learning (ML): Hyper-parameter tuning. After reviewing what hyper-parameters, or hyper-params for short, are and how they differ from plain vanilla learnable parameters, we introduce three general purpose discrete optimization algorithms aimed at search for the optimal hyper-param combination: grid-search, coordinate descent and genetic algorithms. We report on the results of an experiment in which we use each of these to come up with good hyperparameters on an example ML problem taken from Kaggle. Although we focus on optimizing XGBoost hyper-parameters in our experiment, pretty much all of what we will present applies to any other advanced ML algorithms.

What are hyper-parameters?

The most powerful ML algorithms are famous for picking up patterns and regularities in the data by automatically tuning thousands (or even millions) of so-called “learnable” parameters. For instance, in tree-based models (decision trees, random forests, XGBoost), the learnable parameters are the choice of decision variables at each node and the numeric thresholds used to decide whether to take the left or right branch when generating predictions. In neural networks, the learnable parameters are the weights used on each connection to amplify or negate the activation from a previous layer into the next one.

Hyper-parameter tuning and its objective

Learnable parameters are, however, only part of the story. In fact, they are the easy part. The more flexible and powerful an algorithm is, the more design decisions and adjustable hyper-parameters it will have. These are parameters specified by “hand” to the algo and fixed throughout a training pass. Further, the algo typically does not include any logic to optimize them for us. In tree-based models, hyper-parameters include things like the maximum depth of the tree, the number of trees to grow, the number of variables to consider when building each tree, the minimum number of samples on a leaf, the fraction of observations used to build a tree, and a few others. For neural networks, the list includes the number of hidden layers, the size (and shape) of each layer, the choice of activation function, the drop-out rate and the L1/L2 regularization constants.

From a computational point of view, supervised ML boils down to minimizing a certain loss function (e.g. the MSE or the classification error) that depends on the data training input data (Xt, Yt), the learnable params which we denote by a, and the hyper-params. The crucial observation here is that this minimization is done by letting only the learnable parameters vary, while holding the data and the hyper-params constant. To choose the optimal set of hyper-params, the usual approach is to perform cross-validation. Given a hyper-param vector h, its quality is assessed by evaluating the loss function on a held-out set of validation data using the best set of learnable parameters, a*, for that value of h. Symbolically, we have

Introducing the Hyper-Parameter Grid

One important thing to note about hyper-parameters is that, often, they take on discrete values, with notable exceptions being things like drop-out rates or regularization constants. Thus, for practical reasons and to avoid the complexities involved in doing hybrid continuous-discrete optimization, most approaches to hyper-parameter tuning start off by discretizing the ranges of all hyper-parameters in question. For example, for our XGBoost experiments below we will fine-tune five hyperparameters. The ranges of possible values that we will consider for each are as follows:

{"learning_rate"    : [0.05, 0.10, 0.15, 0.20, 0.25, 0.30 ] ,
 "max_depth"        : [ 3, 4, 5, 6, 8, 10, 12, 15],
 "min_child_weight" : [ 1, 3, 5, 7 ],
 "gamma"            : [ 0.0, 0.1, 0.2 , 0.3, 0.4 ],
 "colsample_bytree" : [ 0.3, 0.4, 0.5 , 0.7 ] }

For an explanation of what each of these means in the context of XGBost we refer the reader to this post.

Notice that despite having limited the range for the (continuous) learning_rate hyper-parameter to only six values, that of max_depth to 8, and so forth, there are 6 x 8 x 4 x 5 x 4 = 3840 possible combinations of hyper parameters. This discrete subspace of all possible hyper-parameters is called the hyper-parameter grid. In what follows, we will use the vector notation symbol h = [h0, h1, …, hp] to denote any such combination, that is, any point in the grid.

Three Hyper-param optimization methods.

Exhaustive Grid Search (GS)

Exhaustive grid search (GS) is nothing other than the brute force approach that scans the whole grid of hyper-param combinations h in some order, computes the cross-validation loss for each one and finds the optimal h* in this manner. Generally, the use of lexicographic ordering, that is, the dictionary order imposed on hyper-param vectors, is discouraged and a different order should be considered. The reason is that with lexicographic ordering there is a high chance that the search will focus on a rather uninteresting part of the search space for a rather long time. An interesting alternative is scanning the whole grid in a fully randomized way that is, according to a random permutation of the whole grid . With this type of search, it is likely that one encounters close-to-optimal regions of the hyper-param space early on. We show some evidence of this in the example below.

Coordinate descent (CD)

Coordinate descent (CD) is one of the simplest optimization algorithms. It’s an iterative algorithm, similar to gradient descent, but even simpler! The basic idea is that, at each iteration, only one of the coordinate directions of our search vector h is altered. To pick which one, we examine each coordinate direction turn and minimize the objective function by varying that coordinate and leaving all the other constant. Then we pick the direction that yields the most improvement. The algorithm stops when none of the directions yields any improvement. To learn all the details we refer the reader to the Wikipedia article.

The main advantage of this simple approach is that it is easily adaptable to the fully discrete case for which the arbitrary directions given by gradient descent cannot be easily used. We provide an implementation here.

Genetic algorithm

Genetic algorithms (GAs) are a whole class of optimization algorithms of rather general applicability and are particularly well adapted for high-dimensional discrete search spaces. A genetic algorithm tries to mimic nature by simulating a population of feasible solutions to a(n optimization) problem as they evolve through several generations and survival of the fittest is enforced. There are two basic mechanisms for generating a new generation from the previous one. One is cross-breeding in which two individuals (feasible solutions) are combined to produce two offspring. The other one is mutation. With a given mutation probability, any individual can change any of their params to another valid value. Survival of the fittest is enforced by letting fitter individuals cross-breed with higher probability than less fit individuals. The fitness of an individual is of course the negative of the loss function.
For the whole details we refer the reader to Wikipedia or to AIMA, Part II Chapter 4. Our own implementation is available here.

An experiment

To get a feeling for each of the hyper-param optimization approaches mentioned above, we performed an experiment using actual data taken from the Home Credit Default Risk binary-classification competition, currently being held on Kaggle. We only considered the application_(train|test) data sets and performed a rather minimal data preparation step (see function prepare_data here). Further, to keep training and validation times short and allow for full exploration of the hyper-param space in a reasonable time, we sub-sampled the training set, keeping only 4% of the records (approx. 12,000). For validation, we used a separate 3.2% of the records (approx. 10,000). Even with this small amount of data, a single round of training and validation takes around 3 seconds, for a given set of hyper-params, which is not negligible considering that we intend to scan the whole hyper-param grid. For the reasons just mentioned, the cross-validation AUC values we get are not competitive with the ones at the top of the leader board, which surely are making use of all available records in all data sets provided by the competition.

As mentioned before, we chose XGBoost as our machine-learning architecture. Due to the outstanding accuracy obtained by XGBoost, as well as its computational performance, it is perhaps the most popular choice among Kagglers and many other ML practitioners for purely “tabular” problems such as this one.

Now, for each of the three hyper-param tuning methods mentioned above, we ran 10,000 independent trials. Each trial first resets the random seed to a new value, then initializes the hyper-param vector to a random value from our grid, and then proceeds to generate a sequence of hyper-param vectors following the optimization algorithm being tested. When testing GS, the trial just goes through hyper-param vectors according to a random permutation of the whole grid. With CD, the generated hyper-param vectors are all the ones tried out in intermediate evaluations of the CD algorithm. When CD gets stuck at a local optimum, it gets restarted to a new initial vector uniformly at random. With the genetic algo, the hyper-param vectors are the ones contained in the “DNA” of every individual generated in every successive generation.

By the way, we were able to run these many trials despite the cost of each CV-loss evaluation thanks to a technique called memoization (https://en.wikipedia.org/wiki/Memoization). Essentially, once the CV-loss of a hyper-param vector was evaluated for the first time, we kept it in a look-up table shared among the three methods and never had to re-evaluate it again.

While running each trial, we keep a log of all the hyper-param vectors tried and their corresponding cross-validation losses. The following plots show the behavior of a typical trial, for each of the three methods, The y-axis shows the negative of the CV loss, which in this case is just the CV_AUC metric which we are maximizing. The X-axis keeps track of “time”, that is, the total number CV-AUC evaluations up to a point.

The orange “envelope” line keeps track of the “running best,” which is the best AUC value seen among all function evaluations prior to a point. In a sense, this line is all we care about, as every method will always return the best of all hyper-param vectors evaluated and not simply the last one. Notice how different all three methods look at this level. GS succeeds in gradually maximizing the AUC by mere chance. The deliberate “effort” that coordinate descent and the genetic algorithm make in finding progressively better values is apparent from their plots as well.

However, one randomized trial does not tell the whole story. Let’s look at the “envelope” lines for each of 30 independent trials of each of the three methods.

Interesting… The envelope lines of the grid-search and genetic algos quickly cluster towards the top, unlike the CD one… This means that, with high probability, almost all trials of GS and genetic are likely to give near optimal solutions quicker than CD. On the other hand, the overall best solution is achieved around function evaluation #40 by one of the CD trials, and we don’t see the same for the other methods. Also coordinate descent beats the other two methods after function evaluation #100 or so, in that all of the 30 trials are nearly optimal and show a much smaller variance!

Let’s look at a final plot. What if we take the 10% percentile of the running-best AUC among all 10000 trials? This number is interesting because it gives us a sense of how the running-best of most trials evolves as a function of the number of evaluations. Another way to put it is that it gives us a 90% confidence bound on the worst we can expect from each method. Here it is:

Aha! This is more compelling evidence of what was mentioned above as it tells us that,with probability at least 90%, a randomized trial of coordinate descent will have seen, by CV-AUC evaluation #90, a hyper-param vector with an AUC of around 0.740 or better, whereas the corresponding value for GS is only around 0.738. This might seem like a minor difference, but it is quite significant as far as ML metrics go. Perhaps more telling than that is the fact that, up until the point it plateaus, the CD curve in the plot above has a distinctly higher slope than the GS curve.

The GA, on the other hand, seems to be dominated by GS throughout. This perhaps shouldn’t surprise us. On the one hand GA’s depend on several adjustable parameters (which would be hyper-hyper-parameters in our formulation) such as generation size, mutation rate, ordering of the instance variables to build a DNA sequence and the function linking AUC to the probability of cross-breading. For our experiment, we only tried a couple of settings for these. It is likely that with different settings GA would have beat GS. On the other hand, it’s known (see, for instance AIMA , at the end of page 148 ) that genetic algorithms work best when there are contiguous blocks of genes (hyper-params in our case) for which there are certain combinations of values that work better on average. It is not clear if this is the case in for the rather arbitrary ordering of XGBoost hyper-params that we have chosen.

Conclusion

In this post we clarified the distinction between params and hyper-params in the context of supervised ML and showed, through an experiment, that optimizing over hyper-params can be a tricky and costly business, in terms of computing time and electricity. We see, however, that there are approaches such as coordinate descent that go beyond plain vanilla grid search in terms of speed and quality of the discovered hyper-param vectors.

Stay tuned for a second post in this series in which we shall try out three other optimization approaches: Bayesian optimization, HyperOpt and auto-tune.

La Haus México - Casas y departamentos en venta en CDMX

Doing XGBoost hyper-parameter tuning the smart way — Part 1 of 2 was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Is it possible to process big data on a laptop? Most people would answer with a blatant no. In this post, I would like to convince you that not only is it possible but, in some cases, it can be convenient.

But first, let’s get back to the basics. What does big data analysis entail? Well, according to one of the simplest and most accepted operational definitions, big data computation happens whenever you need to process a data set that doesn’t completely fit in the RAM of a single computer.

As per this definition, big data computation doesn’t have to be distributed among many machines. If you only have one machine and your dataset fits in its hard disk, but not in your RAM, you are facing a big data challenge. This situation calls for techniques from the area of out-of-core algorithms. The name stems from the fact that the only possible approach in this case is to load one chunk of data into memory, do something with them, store (intermediate) results to disk, load another chunk of data into memory, rinse and repeat.

But wait! — you would say — writing and reading intermediate results to disk is very expensive and time-consuming. You're 100% percent right. In fact, a comparison of the time needed to read 1000 MB of data from different media goes as follows:

We see there is a two+ orders of magnitude difference between the transfer rate of DDR to main memory vs. that of a traditional hard disk. Note, however, that not all hard disks were created equal, a solid-state drive being only around a factor of 10 slower than memory.

But wait! — you would say — having to divide my data into chunks and then coordinate the writing and re-reading of intermediate results to and from disk seems like daunting, tedious and error prone task. Again, my dear reader, you are right! The good news is that, in the Python ecosystem, there already exist good quality, high-level libraries that take care of all the nitty gritty details for you. And the best thing is: their APIs are very similar to the ones you might be used to by now, namely Pandas.

Enter Turicreate and SFrames

About two years ago I had my first contact with Graphlab Create, a closed-source Python library for easy exploration, visualization and development of machine-learning models from large data sets, both structured and unstructured. Despite its misleading name, Graphlab Create was pretty awesome. Its only problem: it required an expensive license to use. Dato, the company behind Graphlab Create, was then acquired by Apple and became Turi. This acquisition had two nice side-effects: 1) Graphlab Create was re-branded as “Turi Create”, a more abstract yet less misleading name. 2) Turi Create was open-sourced. Most of Turi Create’s functionality is built on top of the SFrame class, which is very similar in concept to a Pandas DataFrame, except that it can be out-of-core.

A small (but significant!) case study: stalker-stalkee pair detection

In the remainder of this article, I will report on my experience trying out Turi Create and SFrame on a tricky problem using real data.

The data

The data we will use for our experiment comes from the (now inexistent) Gowalla social networking site. Two data nice data sets coming from this site are available here. We will be looking at the biggest one, which contains the event-log of “check-ins” of Gowalla’s users to a set of locations. This data set contains 6.44 million records, each containing a single check-in and just a few columns, of which we will pick only 3: user_id, location_id and checkin_ts (the second-resolution timestamp of the check-in event). The total size of the data set in uncompressed plain text format is just below 380 MB, which cannot be considered big data by today’s standards, but please read on…

The problem and its (theoretical) solution

We will use Turi Create to attack what could be termed the “stalker-stalkee detection problem” on this data set. In this problem, we are asked to identify pairs of users (E, R) that maximize the ‘stalking measure between E and R’. The stalking measure between E and R is defined as the number of distinct locations where there was ever a check-in by user E (the stalkEE) followed by a check-in by user R (the stalkER).

As you can see, this is not an ML problem or even a problem requiring sophisticated statistical techniques. It’s just a “data munging” problem requiring simple structured data manipulations.

In fact, here is what the solution to this problem looks like using Pandas:

However, trying to run this code on a laptop or PC with the amount of RAM that is usual these days, (say 16GB), will result in a MemoryError exception…

But, why is this a big data problem?

To answer this question, we have to go into a bit of the minutia of what the code does. The first thing is to index the check-ins by location_id (remember that in pandas a single value for a key can refer to more than one row). This will make the following computation easier.

Then comes the tricky part, for each location we want to consider all pairs of check-ins where the check-in time stamp of the first user in the pair strictly precedes that of the second user. The code thus generates chin_pairs, a data frame containing all pairs of check-ins for the same location and then filters it to enforce the conditions just described, to generate pairs_filtered.

This is the tricky part because it implies a location-wise “quadratic” computation time- and memory requirement. And it is precisely here where the naïve Pandas solution blows up memory-wise. Sadly, or rather, interestingly, there is no way to get around this. When we run the out-of-core solution below, we will also get the exact measurement of how big the intermediate data sets in this computation are.

What we are witnessing here is a problem that falls in the category of big data not because its input data set is big, but because an intermediate step can overflow our machine’s memory.

Turi Create to the rescue

We will try out Turi Create to implement and run the same algorithm. Luckily, Turi Create’s designers mimicked a good part of Pandas’s API and the translation is almost direct!

First, we load the data set directly from the tab-delimited text file and select the three columns of interest:

Next, we generate the pairs of check-ins that satisfy the conditions of our detection algorithms. Along the way, we rename the columns in the data frame of pairs, to identify columns containing potential stalkee and stalker ids.

The rest is smooth-sailing.

The code above produces the following output.

And voilá! The whole thing ran to completion with no memory error. The total computation time for this script on a 2-core machine, with just 4 GB RAM and 16 GB SSD, running Anaconda Python 3.6 on Ubuntu, was 1703 seconds, i.e. slightly below half an hour.

Measurements of the size of the intermediate results (using len()) yield the following:

Notice that each record in our data set takes up 56 bytes of space: 16 bytes for the user_ids, 8 for the location_id , 16 for the two check-in timestamps, and (at least?) 8 bytes of overhead implied by the Pandas index. Thus, the first intermediate result, chins_ps, would take up at least 56 B x 562 MM = 31e9 B or about 30 GB. This clearly explains the MemoryError exception we got with Pandas.

A note on alternatives

There is at least one very promising alternative to Turi Create out there, called Dask. Unfortunately, Dask hasn’t even reached version 1.0, and in a preliminary experiment I wasn’t able to reproduce the result above with it. Apparently, it requires some careful fine-tuning of some configuration parameters, which was beyond my humble infrastructure configuring abilities. If I ever figure out how to make Dask work for this problem, I will certainly write a post about it.

Conclusion

In this post we saw that doing big data computation in a single, regular sized computer makes sense and is feasible by employing out-of-core algorithms. Further, the use of high-level and mature libraries such as Turi Create on Python makes it convenient. If you have an SSD, you can even do it efficiently for moderately sized datasets, say, less than 100 GB.

More where this came from

This story is published in Noteworthy, where thousands come every day to learn about the people & ideas shaping the products we love.

Follow our publication to see more product & design stories featured by the Journal team.

This post will be shorter. There are four sections.

• In Section 1, we give links to code implementations
• In Section 2, we make a side by side comparison of our algorithms features with LIME’s
• In Section 3, we illustrate the use of our algorithm on a didactic example.
• In Section 4, we present our conclusions.

1. Code implementations

We have implemented the method above in the form of a very short Python module tcxp (tree-classifier explanation) which depends on the scikit-learn implementation of decision tree classifiers and random forests. The code can be found on Yuxi Global’s publicly shared data analytics repository, here.

The provided Python code is meant for illustrative and pedagogical purposes only, as it is certainly not the most optimized implementation possible. It contains a pair of nested loops: one over instances x[i] and another over the nodes of the path followed by those instances in the given tree. The latter loop can be done with through vectorization by using np.take and np.diff. Furthermore, the whole thing could be easily ported to Cython to produce a truly optimized version.

We have also produced a Spark 2.2 implementation of the code, dependent on Spark’s own ML library (spark-ml), which can be found here.

2. A side by side comparison to LIME

The following table summarizes some of the key features of our explanation producing algorithm and its implementation as compared to LIME. The details on each of these are contained in Section 3 of the first part.

3. An example

Let’s illustrate the use of our Python module by means of an example.

We consider Kaggle’s classic Titanic: Machine Learning from Disaster binary classification task. In this task we are given a training set consisting of 891 records, each containing information about one of the passengers of the Titanic. The fields include sex, age, passenger class, the fare paid, the number of siblings and parents traveling with the passenger and the port at which they embarked. Along with these, there is a binary column ‘Survived’ which tells us which passengers survived the ship’s sinking. The goal of the task is to train a supervised model that predicts this class as well as possible from the passenger variables given.

The following code imports the necessary classes from scikit-learn to train a random forest classifier as well as the rf_explain function from the tcxp module we have developed.

The last line performs minor preprocessing of the training data. The result looks as follows.

Training a random forest can be done in a few lines now:

The last line computes the accuracy which incidentally is about 85%. Not bad for a few lines of work. However, it is not the highest accuracy that we are looking for. We want to visualize why the model makes the predictions it makes.

This is accomplished through a single function call to the rf_explain_function

This call returns (1), a matrix of dimension 891 by 9 (the number of features) containing the explanation arrays as rows, and (2), the p[0] probability which is the a priori probability that a passenger survived the Titanic incident without taking into consideration any of their characteristics.

Now let’s take a look at a few passengers and see whether their predictions and explanations

Passenger 1 (second row in table above) is predicted to have 0.963 probability of survival and in fact she survived. The explanation array is as follows: tc_exps[1,:] =[0.11 0.01 0.02 -0.01 0.11 0.00 0.28 0.03 0.02], but it is better to see it in a plot:

That this passenger survived is hardly a surprise as first-class female passengers had a very high probability of survival. The explanation array shows those characteristics together with the fare (highly correlated with Pclass) as the most determinant characteristics of the prediction.

Passenger 14 (second to last row in table above) didn’t make it though, and our model correctly predicts a relatively low probability of survival. To explain why, let’s look at the explanation plot:

In this case the fact that she was travelling in third class negatively impacted her chances of survival.

A more interesting case is presented by passenger 305 (last row of table shown above), who survived even though he was male:

We see that the class he was travelling on, but mostly his young age as well as the fact that he was travelling with his parents and siblings, contributed to his probability of survival of 67.7%.

Conclusion

As the last section shows, our tree classifier explanation method is likely to produce sensible explanations and to illuminate predictions of otherwise blackbox ML architectures. This method, together with other more general methods such as LIME, yields some hope that someday these kinds of advanced algorithms may come out of their current bad reputation. As with any other explanation method, ours opens avenues for smarter prescriptive actions coming out of predictions of such models.

WHY did your model predict THAT? (Part 2 of 2) was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this post, we present a simple and scalable approach for producing easily interpretable explanations for the individual predictions of tree-based binary classifiers. This post is divided into three sections:

• In Section 1, we motivate the need for case by case explanations of the predictions made by machine-learning classifiers and comment on how the most advanced methods are usually the least transparent.
• In Section 2, we review previous work in this area, in particular, the LIME package
• In Section 3, we present our explanation-generating algorithm in detail and comment on its generality and scalability

In the second part, we will provide links actual code implementations of the algorithm presented here, make a side by side comparison with LIME and look at an example application. Go there, if you want to see our algorithm in action.

1. Advanced machine learning models and their explanatory power (or lack thereof)

Machine learning models have often been criticized for their lack of transparency. All but the simplest model architectures (i.e. linear / logistic regression) implement rather complicated non-linear prediction functions that combine all the available features and the interactions amongst them in ways that are difficult to visualize and explain. Thus, for example, the question why did your model deny credit to customer X? cannot be satisfactorily answered if the prediction is produced by a random forest, a boosted-forest or a neural-network.

As others have noted before, being able to understand the reasons behind the predictions produced by a model is important to establish trust on the part of humans who consume the model results and those who ultimately will be impacted by those results.

The black-box nature of advanced ML models has led to at least two undesirable consequences:

• In some cases, in which individual prediction explanations are vital (more on this below), data scientists have reverted to the simplest model architectures at the cost of significantly impacting prediction accuracy.
• There has been a public and regulatory reaction against advanced machine learning models as exemplified by the Algorithmic Fairness provisions of the General Data Protection Regulation (GDPR) that is about to become law in Europe.

A naïve data-scientist would argue that it doesn’t matter why a prediction is what it is, as long as the classifier achieves a very high degree of accuracy or some other performance measure. This isn’t true in many cases. There are situations, in which we must decide on the best course of action on a case by case basis. Think, for example, customer churning or loan-recovery. More often than not, the best course of action for a given case depends on which variables most influenced the prediction for that particular case. Therefore, it becomes vital to be able work-out what those variables were, and quantify the degree to which each of them influenced said prediction.

2. Previous work

For some of the architectures, particularly random-forests, there are well known ways to come up with global measures of importance for the input variables. The term global is meant to emphasize that they produce a single measure of importance for each variable and do so by aggregating all training cases. This is not satisfactory as, in the majority of situations, the set of important decision variables depends on the particular case being examined. Hence, what we really need are local or, even better, case-by-case measures of variable importance.

From here on, we will use the term explanation in a precise sense: Given a set of evaluation instances x[i], i = 0, 1, …, and a binary classifier C, assigning to each instance a probability p[i] = C(x[i]) that it belongs to the positive class, an explanation for that prediction is an array e of real numbers of dimension f, where f is the number of features (variables). In symbols,

In this array, the j-th component is interpreted as a measure of the importance or contribution of the j-th feature to the probability p[i]. Thus, an explanation is a function of two things; the classifier used to produce a prediction and the instance to which the classifier is applied.

This formalism is easily extended to the case of k-class classification. In this case, the explanation is a matrix of dimension k-by-p, where the l-th row uis interpreted as the contributions of each of the features that the probability that the instance belongs to class l.

There have already been important contributions in this direction. Perhaps the most notable and best known is the LIME package (video, paper) which can be used to produce explanations for the predictions of any classifier. LIME stands for Local Interpretable Model-agnostic Explanations. It achieves model-agnosticity y generating a locally linear ML model for each instance x[i] to be explained and casting the feature importances of that model as an explanation of the form above. The way to generate this locally linear model is by first generating M (of the order of hundreds) samples of the neighborhood around x[i], evaluating the classifier on each of them and then fitting a model on the M predictions thus generated.

The obvious advantage of this approach is that it works for any model. But this generality comes at great computational cost. Namely, the number of operations required to produce explanations for all of the evaluation instances is M times larger than the number of operations required by the simple computation of classifier predictions.

In the next sections we address this shortcoming and propose a simple and efficient algorithm for producing explanations for tree-based classifiers.

The authors of LIME claim that the local model generated for each example is faithful in some sense. However, it is not clear to yours truly that this means anything in the case of tree-based models, which produce piecewise constant classification functions. In this kind of models, any sufficiently local model is necessarily going to miss the dependency of the decision function on the any independent variable. On the other hand, any model that takes into account neighbors of a point that fall within different branches of some of the trees is in danger of not being local enough to be faithful.

3. A simple algorithm for producing explanations for single tree predictions

First, let’s consider the case of a simple binary classification tree T, which has already been trained on a set of examples. Let t denote the total number of nodes in this tree, indexed from 0 to t-1, with 0 corresponding to the tree’s root. A key fact is that on each node in this tree there is a single feature on which the decision to either go left or right is based. Thus we have corresponding features {f[0], f[1], …,f[t -1]}, each being a simple index in the set {0,1, …, f-1}

Given an instance x[i] to be evaluated and explained, the way to make a prediction is to run it through the tree, following a path of nodes that ends in a leaf.

The exact path of course depends on the feature values of the given instance. Let this path be described by the sequence of indices of the nodes that it goes through:

Now, for any node n, let’s define p[n] as the empirical probability that an instance in the training set belongs to the positive class given that the instances’s path goes through node n.

Notice that, with the notation above, the tree’s estimate of the probability that x[i] is in the positive class is given by

On the other hand p[0] — recall 0 denotes the tree’s root — corresponds to the unconditioned probability that x[i] belongs to the positive class.

Following x[i]’s path through the tree we can decompose its probability estimate as the sum of p[0] and a series of probability increments

Notice that the s-th probability delta corresponds to the segment of the path that went from node n[s -1] to node n[s] and, as such, can be attributed to the feature used at n[s -1] to decide to go the latter node.

Based on this observation we can define an explanation array as follows:
The j-th component, for j in {0,1, … p-1}, of the explanation array is obtained as the sum of probability deltas attributed to feature j from the previous sum. In symbols,

Notice that, because of this definition and Eq. 3 above, we have the following nice identity

3.1 Extension to tree ensemble classifiers

The astute reader will surely have realized at this point that the definition of the explanation array presented above naturally extends to any classifier that produces a prediction as a linear combination of individual tree predictions. Let’s suppose we are given such a classifier C, consisting of A trees, T[0], T[1], … , T[A-1], corresponding weights w[0], w[1], … , w[A-1], and prediction function given by

Then, an explanation array can be computed as a weighted sum of the explanation arrays computed for each of the individual trees:

3.2 A note on complexity and scalability

Assuming that the conditional probabilities p[n] are available, the cost of computing all components of the explanation array e[j], for j in 0, …, f, for any given instance, x[i] by means of eq. 4 is just O(n_c) (recall n_c is the length of the path in the tree), as it is enough to make a single pass through x[i]’s and update the appropriate component of e as one goes along. Hence, computing an explanation array for an instance has essentially the same complexity as computing that instance’s prediction.

Incidentally, both the scikit-learn tree estimation procedure as well as the Spark 2.2 ml library implementation of Decision trees, produce and keep within the estimated trees these values of p[.].

If they weren’t available, the cost of computing them will essentially be the same as that of computing predictions of for all training samples while at the same time updating two counter variables at each node of every tree, one for the total number of instances and another one for the number of instances belonging to the positive class whose paths go through that node.

In the second part, we will provide links to actual code implementations of the algorithm presented here, make a side by side comparison with LIME and look at an example application. Go there, if you want to see our algorithm in action.

WHY did your model predict THAT? (Part 1 of 2) was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.