<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by DelgadoPanadero on Medium]]></title>
        <description><![CDATA[Stories by DelgadoPanadero on Medium]]></description>
        <link>https://medium.com/@delgadopanadero?source=rss-7462ba410e23------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Z2rwz2m6o0GTuwdHstNgFw.jpeg</url>
            <title>Stories by DelgadoPanadero on Medium</title>
            <link>https://medium.com/@delgadopanadero?source=rss-7462ba410e23------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 13:33:26 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@delgadopanadero/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[TaEl modelo fundacional sobre datos tabulares!! ✨]]></title>
            <link>https://medium.com/@delgadopanadero/tael-modelo-fundacional-sobre-datos-tabulares-d8620f1e4f34?source=rss-7462ba410e23------2</link>
            <guid isPermaLink="false">https://medium.com/p/d8620f1e4f34</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[mls]]></category>
            <category><![CDATA[tabpfn]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[DelgadoPanadero]]></dc:creator>
            <pubDate>Thu, 16 Jan 2025 14:02:14 GMT</pubDate>
            <atom:updated>2025-01-20T16:31:40.830Z</atom:updated>
            <content:encoded><![CDATA[<h3>TabPFN - El modelo fundacional sobre datos tabulares!! ✨💥</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/441/0*Bk9RCgCHL4HD7XG1" /></figure><p>El otro día mi colega <a href="https://www.linkedin.com/in/javier-echavarren-suarez/">Javier Echavarren Suarez</a> me descubrió <a href="https://www.linkedin.com/feed/hashtag/?keywords=tabpfn&amp;highlightedUpdateUrns=urn%3Ali%3Aactivity%3A7285570194876338176">#<strong>TabPFN</strong></a>. TabPFN es un modelo fundacional para datos tabulares que supera a modelos tradicionales (así es, supera a modelos como <strong>RandomForest</strong> y <strong>GradientBoosting</strong>) en precisión, pero lo más increíble de todo es que NO REQUIERE ENTRENAMIENTO</p><h3>¿Cómo funciona?</h3><p>Nos es que use <a href="https://www.linkedin.com/feed/hashtag/?keywords=llms&amp;highlightedUpdateUrns=urn%3Ali%3Aactivity%3A7285570194876338176"><strong>LLMs</strong></a> como GPT y pase los datos como “contexto”. Tampoco es una red neuronal que tengas que diseñar a mano y entrenar como se hacía antiguamente. Básicamente es un modelo fundacional que toma datos tabulares como input y ha aprendido a qué fijarse y cómo gracias a los mecanismos de atención. De esta manera lo único que requiere es una ventana histórica de datos como input, pero no un entrenamiento en sí</p><h3>¿Por qué tienes que mirarlo?</h3><p>Creo que no es el típico modelo que nace y muere a los dos días, creo que con este modelo se está buscando (y consiguiendo) un “<strong>AGI</strong>” de datos tabulares. Además ya estamos empezando a ver que están saliendo otras librerías como <a href="https://www.linkedin.com/feed/hashtag/?keywords=shapiq&amp;highlightedUpdateUrns=urn%3Ali%3Aactivity%3A7285570194876338176">#<strong>Shapiq</strong></a> para obtener la explicabilidad de este tipo de datos</p><h3>Reflexión</h3><p>En el anterior post que escribí hablaba de cómo el futuro para la adopción de la IA en las empresas pasa por la integración del almecenamiento, modelado y consumo de datos no estructurados en los <a href="https://www.linkedin.com/feed/hashtag/?keywords=datawarehouse&amp;highlightedUpdateUrns=urn%3Ali%3Aactivity%3A7285570194876338176">#<strong>DataWarehouse</strong></a> gracias a los LLM.</p><p>Creo que en un futuro veremos como herramientas como TabPFN y los LLM se combinan para poder “procesar” automáticamente datos tabulares y textuales. Y esta es la verdadera <a href="https://www.linkedin.com/feed/hashtag/?keywords=multimodalidad&amp;highlightedUpdateUrns=urn%3Ali%3Aactivity%3A7285570194876338176">#<strong>multimodalidad</strong></a> que importa a las empresas</p><p><strong>TabFPN</strong>: <a href="https://github.com/PriorLabs/TabPFN/tree/main">https://github.com/PriorLabs/TabPFN/tree/main</a><br><strong>Shapiq</strong>: <a href="https://github.com/mmschlk/shapiq">https://github.com/mmschlk/shapiq</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d8620f1e4f34" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[  ]]></title>
            <link>https://medium.com/@delgadopanadero/-75a83d814a43?source=rss-7462ba410e23------2</link>
            <guid isPermaLink="false">https://medium.com/p/75a83d814a43</guid>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[DelgadoPanadero]]></dc:creator>
            <pubDate>Wed, 08 Jan 2025 12:57:55 GMT</pubDate>
            <atom:updated>2025-01-08T12:57:55.035Z</atom:updated>
            <content:encoded><![CDATA[<h4>La nueva forma de 𝗖𝗔𝗚arla haciendo proyecto de IA <strong>Generativa</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/659/1*igaTDzBDwf1-rNQfpXXpZg.png" /></figure><h3>Introducción</h3><p>El mes pasado se publicó en Arxiv un paper proponiendo una nueva arquitectura de GenAI: Caché Augmented Generation (CAG). Como pasa con cualquier tema de IA, mi feed se ha llenado de posts evangelizando sobre el tema, asíque aquí vamos a hacer lo contrario… vamos a cuestionar el CAG</p><h4>¿Qué es el CAG?</h4><p>Básicamente es una evolución de la arquitectura RAG. Al igual que el RAG, el CAG también se busca utilizar bases de datos documentales como contexto a un LLM para mejorar las respuestas, sin embargo, CAG utiliza memoria caché, reduciendo mucho los tiempos de respuesta y mejorando el real time (aunque el verdadero cuello de botella del real time no está ahí)</p><h4>El reto de las empresas</h4><p>Ok, el CAG obviamente hace un consumo más rápido de la información al estar cacheada pero… ¿Realmente es ese el problema que están teniendo las empresas a la hora de productivizar casos de uso? No, no lo es.</p><p>A grandes rasgos, los dos retos que veo yo en las empresas de españa para la mayoría de funciones de negocio y niveles de capitalización son 2</p><ul><li>La <strong>conceptualización</strong> (ser capaz de ver un producto más allá del chatbot)</li><li>La <strong>observabilidad</strong> (tanto a nivel de desarrollo como de datos como de ejecución)</li></ul><h4>Los problemas del CAG</h4><p>Ya no solo es que no crea que el CAG no resuelva los problemas reales que se encuentran las empresas, es que además abre un nuevo problema: el gobierno de los datos cacheados (de hecho la tendencia tecnológica apunta a que el futuro del GenAI no esté en mejorar los LLM sino el gobierno/calidad/metadatado de los datos)</p><h3>Conclusión</h3><p>Los retos de la IA Generativa no son técnicos, así que no busques vueltas de tuerca tecnológicas a problemas que aún no tienes, solo te llevará a tener más fricciones a la hora de adoptar la IA. El CAG es una buena propuesta cuando el reto es de real time en aplicaicones satélite</p><p>Si quieres más consejos sobre estrategia tecnológica de IA y GenAI, sígueme porque este 2025 me he propuesto publicar mucho contenido :)</p><p>#GenAI #IA #RAG #CAG #Data #AI #ArtificialIntelligence #OpenAI</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=75a83d814a43" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Modelo de Assessment de IA]]></title>
            <link>https://medium.com/@delgadopanadero/modelo-de-assessment-de-ia-36e37fc838af?source=rss-7462ba410e23------2</link>
            <guid isPermaLink="false">https://medium.com/p/36e37fc838af</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[consulting]]></category>
            <category><![CDATA[strategy]]></category>
            <dc:creator><![CDATA[DelgadoPanadero]]></dc:creator>
            <pubDate>Wed, 14 Aug 2024 13:06:58 GMT</pubDate>
            <atom:updated>2024-08-14T13:06:58.479Z</atom:updated>
            <content:encoded><![CDATA[<h4>Cómo definir una estretagia empresarial de Inteligencia Artificial sea cual sea el estado en que se encuentre actualmente.</h4><h3>1. ¿Por qué un assesment?</h3><p>Un assessment es un tipo de proyecto de consultoría tecnológica donde el objetivo no es el desarrollo de un producto o solución de software, sino que consiste en <strong>asesorar a una empresa</strong> a la hora de definir una estrategia y/o plan ante un problema.</p><p>En el ámbito de la IA, las empresas ya saben el valor que aporta como catalizador tecnológico, sin embargo, la empresas <strong>no saben</strong> <strong>por donde empezar a hacer IA</strong>, qué tipos de perfiles necesitan, qué herramientas tienen que adquirir, que consideraciones tienen que tener a nivel legal, qué datos necesitan…</p><p>Es aquí donde un assessment de IA aporta mucho valor, porque permite definir una estrategia de IA para una empresa, en base al punto en que se encuentra y las necesidades que tienen, en lugar de empezar a desarrollar soluciones sin saber que si realmente van a tener un retorno de la inversión.</p><h3>2. ¿Cómo se hace un assessment?</h3><p><strong>¿Cuál es el entregable de un assessment? </strong>Como hemos dicho, el entregable e un assessment no es software, el objetivo se dar conocimiento. Es por ello que el entregable suele ser documentación con todo el conocimiento desarrollado sobre el assessment.</p><p><strong>¿Cómo se puede asesorar a una empresa? </strong>Generalmente, entregar documentación sin más contexto no suele tener calado. Además, no existe una estrategia común para todas las empresas. Para ello, más allá de simplemente entregar documentación, la idea es definir un conjunto de sesiones/dinámicas de trabajo. Estas dinámicas se suelen organizar cronológicamente en 3 fases entre 1–2 semanas cada una.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/896/1*DGSw7NYO4U8F692TDgURKw.png" /><figcaption>Las fases de un assessment suelen ser similares a las de cualquier dinámica de Desing Thinking. La diferencia es que aquí el objetivo no es pensar sobre una aplicación sino sobre una estrategia.</figcaption></figure><ul><li><strong>1º Fase — Inmersión: </strong>Estas dinámicas el objetivo es entender el estado actual a nivel de adopción tecnológica</li><li><strong>2º Fase — Ideación: </strong>Estas dinámicas el objetivo es entender los principales problemas que tiene actualmente a la hora la empresa y cuales son los principales casos de uso</li><li><strong>3º Fase — Planificación: </strong>Una vez se han identificado qué necesitades tiene una empresa a nivel de IA, es necesario finalizar con una última fase en la que se plantea un plan de ejecución.</li></ul><h3>3. Sesiones de Inmersión</h3><p>Como comentábamos, estas dinámicas tienen como objetivo entender el estado actual de la empresa en términos de IA. Aún no se hará ninguna propuesta de mejora aunque se vea clara, ya que el objetivo es implemente hacer un diagnóstico.</p><h4>3.1. Identificación del nivel de adopción de IA</h4><p><strong>Objetivo: </strong>El objetivo de esta sesión es extraer todas las iniciativa actuales (desarrolladas, como en curso, como en to-do) de IA que tiene la empresa. En el caso de que no tenga ninguna, puede ser interesante hablar de aquellas que habían pensado</p><p><strong>Asistentes: </strong>Áreas de Negocio, PMs y Tech Lead (cualquier que pueda identificar las iniciativas de IA)</p><p><strong>Duración: </strong>2h</p><p><strong>Metodología: Dinámica de trabajo.</strong> Suele ser muy difícil conseguir que una empresa sea capaz de listar fácilmente todas las inicitivas de IA y cualificarlas, por ello suele ser interesante definir una dinámica para que salgan de manera natural. Para ello, proponemos la siguiente dinámica</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*1EivBiXRB5yNH3d5km-eeA.png" /></figure><p>El objetivo de la dinámica es crear un tablón compartido con 3 columnas con los “tipos de IA” y 3 filas con el “nivel de exposición” de la IA (llamamos nivel bajo a pluggins internos y componentes con bajo impacto en el negocio y nivel alto si la operativa/interacción con clientes, depende de esas componentes).</p><p>Durante la dinámica, todas las personas invitadas a la reunión pondrán posits con todas las iniciativas (hechas, en curso o pensadas) que tengan identificadas. De esta manera, construiremos un mapa completo de iniciativas actuales de IA de la empresa</p><p><strong>Conclusiones: ¿</strong>Cómo interpretar los resultados?</p><ul><li>Una empresa que tenga un “nicho de conocimiento” seguramente tenga solamente casos de usos identificados en una columna</li><li>Una empresa que tenga detectados pocos casos de uso, seguramente tenga casos de uno en una de las líneas</li><li>Una empresa que tenga un nivel de adopción alto, puede tener tanto muchos casos de uso a lo largo del tablón como pocos casos de uso muy maduros</li></ul><h4>3.2. Mapa de sistemas y ciclo de vida del dato</h4><p><strong>Objetivo: </strong>El objetivo de esta sesión consiste en entender el conjunto de sistemas que son objeto de utilizar datos/IA como el flujo que siguen desde su origen hasta que son consumidos.</p><p><strong>Asistentes: </strong>Data team, IA team (cualquier persona que pueda darnos un conocimiento de los sistemas que componen ahora mismo el stack de la empresa y para qué se usan)</p><p><strong>Duración: </strong>2h</p><p><strong>Metodología: Dinámica de trabajo. </strong>El objetivo de esta dinámica consiste en tener un diagrama de todos los flujos que relacionan los orígenes de datos, cómo se procesan y/o se usan por sistemas de IA y finalmente cómo se consume toda esta información. Ademés suele ser relevante incluir qué dependencias (humanas, técnicas, basadas en otros datos) tienen todos estos flujos.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aSiRwiueihFeChuAgsvXjQ.png" /></figure><p>La idea durante la sesión es que los miembros del equipo técnico pinten todos los flujos que tienen detectados. A diferencia de la sesión anterior, aquí no se ponen posits de manera independiente, sino que cada</p><p>En el caso de que el número de flujos sea muy grande, puede ser buena idea pintar los flujos sobre diferentes diagramas o incluso cada flujo en un diagrama independiente.</p><p><strong>Conclusiones: </strong>¿Cómo interpretar los resultados?</p><ul><li>Una empresa que tenga muchos flujos independientes es una empresa que es propensa a tener silos de conocimiento.</li><li>Una empresa que tiene muchas dependencias suele ser una empresa que tiene mucha deuda técnica</li><li>Una empresa que tiene muchos flujos cruzados suele ser una empresa que tiene una metodología muy asentada</li><li>Una empresa que tenga un alto número de iniciativas de IA, tendrá muchos postits en la sección de “sistemas de IA” y seguramente, la dinámica requiera una sección a parte de “catálogo de modelos”.</li></ul><h4>3.3. Entregables</h4><p>La combinación de la identificación del nivel de adopción junto con el mapa de sistemas nos permite identificar de manera más o menos clara qué datos tiene ahora mismo la empresa, cómo los consume y cómo los usa para hacer IA.</p><ul><li><strong>Resultados de las sesiones</strong></li><li><strong>Documentación ASIS: </strong>Documentación detallada con el diagnóstico del estado actual de la empresa en términos de IA. Es muy imporantate ser suficiéntemente detallista en la descripción de este documento, ya que nos servirá como punto de partida sobre el que apoyarnos en las siguientes fases.</li></ul><h3>4. Sesiones de Ideación</h3><p>En esta fase las dinámicas deberán tener como objetivo entender cuales son las necesidades que tiene actualmente la empresa. A diferencia de la fase anterior, aquí deberemos llevar un rol más “procativo” a proponer ideas, intentar animar a la gente a pensar ideas…</p><h4>4.1. Pain Points</h4><p><strong>Objetivo: </strong>El objetivo de esta sesión consiste en detectar los principales que tienen actualemente trabajando con datos e IA para el desarrollo de nuevos modelos de IA.</p><p><strong>Asistentes: </strong>Áreas de Negocio, PMs y Tech Leads</p><p><strong>Duración: 2</strong>h</p><p><strong>Metodología: </strong>Al igual que una dinámica de Design Thinking, una manera de levantar todas las necesidades y problemas del estado actual que tenga la empresa en términos de IA, es plantearlo por medio de historias de usuario, de manera que cada persona de la sesión exponga las necesidades que tiene identificadas</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*TfNHcd9ThWCd4hYFskOavA.png" /></figure><p>Otra forma de enfocar esta sesión (sobretodo en empresas que tengan un algo nivel de adopción de IA) es el de hacer un análisis DAFO, de manera que también podamos entender, no solamente las necesidades, sino también los puntos que se tienen identificados como puntos fuertes en el equipo</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/694/1*5OwvbyRNJxN8WBW3AH2DRQ.png" /></figure><p>Además de esto, hay veces que es necesario “guiar” la sesión detectando pain points que creamos que puede haber en base a lo que hemos visto en las sesiones de inmersión</p><p><strong>Conclusiones: </strong>Después de que en las sesiones anteriores hayamos ententido el mapa de sistemas y casos de uso, el objetivo de esta sesión es entender realmente qué problemas tienen y cómo impactan</p><h4>4.2. Casos de uso del sector</h4><p><strong>Objetivo: </strong>El objetivo de esta sesión consiste presentar casos de uso relavantes para la empresa tanto por el sector en el que se encuentran (casos de uso que otras empresa de su sector hacen) como por el punto en el que se encuentran (tipos de tecnología que ya usan)</p><p><strong>Asistentes: </strong>Áreas de Negocio, PMs y</p><p><strong>Duración: </strong>1h</p><p><strong>Metodología: </strong>Presentación de casos de uso con diálogo abierto</p><p><strong>Conclusiones: </strong>El objetivo de la presentación sería tener una idea de qué casos de uso les interesan más y con qué tipo de problemáticas se sienten más identificados</p><h4>4.3. Sesiones temáticas y talleres</h4><p>En esta secciones incluimos, en el caso de que fuera necesario, sesiones específicas para dudas concretas que pueda tener la empresa con lo que se haya visto en las sesiones anteriores. Algunos de estas sesiones temáticas pueden ser:</p><ul><li><strong>IA Act y aspectos legales: </strong>Entender en qué consiste la regulación actual de Inteligencia Artificial y saber cómo asegurar su cumpliment</li><li><strong>Estrategia de plataforma: </strong>En el caso de que la empresa ya tenga iniciaticas de ML arrancadas, seguramente le interesa más saber cómo desplegar soluciones. La estrategia de plataforma introduce cómo crear una metodología de despliegue</li><li><strong>Benchmark de herramientas: </strong>Tando si se tiene un stack tecnológico claro, como si no, puede ser interesante listar las alternativas que existen en el mercado y cuál es mejor analizando cada una de ellas por separado.</li></ul><h4>4.3 Entregables</h4><ul><li><strong>Resultados de las sesiones</strong></li><li><strong>Documentación ASIS ampliado: </strong>Documentación de la sesión anterior añadiendo los pain points detectados.</li></ul><h3>5. Sesiones de planificación</h3><p>Tras las dos fases anteriores, al inicio de esta fase tenemos que tener pensada una propuesta de solución a los problemas que se han planteado anteriormente. El objetivo de esta fase es plantear la solución propuesta y recoger feedback.</p><h4>5.1. Presentación de la Propuesta TOBE</h4><p><strong>Objetivo: </strong>El objetivo de esta dinámica es simplemente presentar una propuesta de solución a los problemas que se han levantado. Algunas de las soluciones propuestas como TOBE pueden ser:</p><ul><li><strong>Propuesta de arquitectura</strong></li><li><strong>Propuesta de equipo</strong></li><li><strong>Propuesta de casos de uso</strong></li></ul><p><strong>Metodología: Presentación. </strong>Aunque esta dinámica consiste en una presentación, es muy importante recoger feedback y ver que los asistentes se sienten identificados con la solución. No se trata de convencer de la propuesta. Una solución buena, pero que no es compartida por los asistentes, no es una buena solución</p><p><strong>Duración: 1h</strong></p><h4>5.2. Presentación de Estrategia de implantación</h4><p><strong>Objetivo: </strong>Una vez hemos definido una propuesta de solución, la última sesión consiste en presentar una estrategia de implantación. Esto dependerá de el tipo de propuesta que hayamos hecho. Algunos tipo de estrategia de implemetación pueden ser los siguientes.</p><ul><li><strong>Estrategia de migración</strong></li><li><strong>Roadmap de crecimiento de equipo</strong></li><li><strong>Priorización de casos de uso</strong></li></ul><p><strong>Metodología: Presentación</strong> de la estrategia de implamentación diseñada previo a la sesión.</p><p><strong>Duración: </strong>1h</p><h4>5.3. Entregables</h4><ul><li><strong>Propuesta TOBE: </strong>Documento en el que de detalla una solución y/o estrategia de IA para la empresa. Es importante esta propuesta refleje todas las ideas y painpoint detectados en la fase de ideación</li><li><strong>Estrategia de implantación: </strong>Además de una propuesta TOBE, para que el plan sea aplicacble, es necesario definir un roadmap y/o estratecia de implementación. Es imporatnte que este roadmap parta del diagnóstico realizado en la fase de inmesión y del documento de ASIS.</li></ul><h3>6. Conclusiones</h3><p>Un assessment consiste en un tipo de proyecto donde el objetivo es asesorar a una empresa sobre un tema concreto. Para que este tipo de proyecto tenga éxito es muy importante crear una serie de dinámicas, agenga y documentos que aseguren que realmente la empresa entiende la solución y le es útil para empezar a ejecutar un plan justo después del assessment.</p><p>En este documento hemos explicado en particular cómo hacer un assessment de IA y hemos propuesto un modelo de assessment y una serie de dinámicas para cada una de sus fases. No obstante, es importante tener en cuenta que cada empresa puede tener una circunstancia distinta y siempre habrá que adaptar este modelo a ese caso concreto.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=36e37fc838af" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[El color del dinero y el valor del trabajo.]]></title>
            <link>https://medium.com/@delgadopanadero/el-color-del-dinero-y-el-valor-del-trabajo-535173cce61b?source=rss-7462ba410e23------2</link>
            <guid isPermaLink="false">https://medium.com/p/535173cce61b</guid>
            <dc:creator><![CDATA[DelgadoPanadero]]></dc:creator>
            <pubDate>Fri, 12 Jul 2024 08:13:45 GMT</pubDate>
            <atom:updated>2024-07-12T08:13:45.920Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/575/0*cl_6DqUrS2KE3glr" /><figcaption>El color del dinero (1986). Eddie (Paul Newman) intentándole pervertir a un joven Vicent (Tom Cruise) haciéndole creer que el valor de su trabajo se mide en dollares y no en la virtuosidad de su trabajo.</figcaption></figure><p>Esta semana me vuelvo a cambiar de trabajo una vez más. Una vez más he vuelto a tener un motivo para no estar contento con mi trabajo. En estos últimos 7 años, he estado dado de alta en 6 empresas distintas y he trabajado eventualmente para otras 4. Todo esto me ha llevado a hacerme preguntas como ¿Por qué me muevo tanto?¿Es algo bueno o es algo malo? o más bien,¿Por qué la gente, en el sector IT cambia tanto de trabajo?</p><p><strong>¿Donde quedaron carreras profesionales como las de mi padre?</strong> 42 años en una misma empresa, creciendo, con sus más y sus menos, pero desde luego, sintiéndose realizado como profesional ¿No pueden las empresas dar un plan de crecimiento profesional como el de antaño?¿Han cambiado los estándares de los trabajadores?¿Que espera un trabajador del trabajo?</p><p>Si bien es cierto que tanto la concepción como las tareas que constituyen los trabajos de la sociedad actual ha cambiado tanto en los últimos 50 años, existen diferentes motivos que pueden justificar esta situación: la inflacción, la era digital, la globalización… mi objetivo en este ensayo consiste en desarrollar <strong>una explicación desde la perspectiva del trabajador</strong> y no desde la sociedad en la que vive</p><p>A lo largo de este ensayo, analizaré como creo (en mi experiencia) que ha cambiado “el trabajo” para el trabajador y <strong>cómo se mide el valor</strong> de su trabajo.</p><h3>El nuevo capital</h3><p>No es la primera vez en la historia que existen cambios en el concepto del trabajo. Durante la revolución industrial ya vimos lo que Marx llamó el <strong>paso de obrero a proletario</strong>. Independientemente de la ideología política que tengas tú (sí tú, lector), sí que es innegable que Marx tenía razón en esto. El trabajador dejó de ser ese artesano que hacía una silla y era capaz de generar valor con su trabajo, para ser alguien que apretaba una tuerca en una cadena de montaje y por tanto, su trabajo valía solo lo que valía apretar una tuerca (entre poco y nada).</p><p>¿Cual es el motivo de que existe este cambio? Marx lo tiene claro, el capital. “<strong>El capital” es esa fábrica sin la cual los trabajadores no pueden ejercer. </strong>Es gracias a la cual, se consigue la super-productividad pero que hace que el trabajo pase de ser un oficio que se aprende con los años a ser un completo inutil toda la vida.</p><p>Hoy en día, gracias a que la tecnología no es algo físico, como una fábrica, parece que <strong>ya no existe “el capital”</strong>, <strong>cualquiera en su casa puede desarrollar una aplicación</strong> sin necesidad de “una fábrica”. Sin embargo, el hecho de que la tecnología avance tan rápido, hace que aunque tengas la capacidad física de poder “artesanía de software” no tengas tiempo para ello. Porque sí, así es amigos, el problema es que el conocimiento está democratizado, pero el tiempo para adquirirlo es finito, el día tiene 24 horas y el año 365 días.</p><blockquote>Tu valor no es el conocimiento que tienes, sino la rapidez con la que puedes aprender el conocimiento de otro</blockquote><p>Por lo tanto, bajo mi opinión, la limitación del tiempo ha hecho que el <strong>nuevo capital sea “la posesión del conocimiento”</strong> y el artesano de software que podía desarrollar lo que quisiese, ahora es un proletario de código que solamente puede apretar las tuercas de los servicios de AWS, GCP y demás productos controlados por un puñado de empresas.</p><h3>¿Cual es el valor de mi trabajo?</h3><p>En resumen, creo que en estos años, con el hecho de que hayan aparecido cloud vendors y el hecho de que salga más rentable comprar productos prefabricados, el desarrollador se está conviertiendo en un proletario del código, pero en cambio, lejos de devaluarse los salarios en el sector IT, siguen subiendo cada vez más ¿Cómo es esto posible?</p><p>No sé dar una respuesta clara a lo anterior, lo que sí que creo es que es lo que pasa cuando creas un “mercado del conocimiento”. Los filósofos se convierten en sofistas y tienen que inventar nuevo conocimiento, <strong>no por el objetivo de seguir creando, sino con el objetivo de seguir vendiendo</strong>.</p><blockquote>La tecnología ha pasado de crearse para funcionar a crearse para poder venderse.</blockquote><p>En “El color del dinero” (película perfecta para ver un domingo de tarde) veíamos como un virtuoso jugardor de billar, Paul Newman, veía más rentable elegir cuándo tenía que perder aposta para ganar más dinero, que actuar en virtud a su profesión y buscar siempre la victoria. Del mismo modo creo que pasa con la tecnología (y sobretodo las empresas de consultoría), que <strong>confunden la virtud de “aportar valor en la tecnología” con el valor de la facturación.</strong></p><h3>La virtuosidad en el trabajo</h3><p>Después de toda reflexión, creo que el factor común que he tenido en todos estos años es que siempre he trabajado en consultoría tecnológica y he visto como la preventa precede al conocimiento, pero sobretodo como la venta sustituye al valor. Me he visto haciendo cosas que no entendía por qué hacía, me he sentido traicionando y mi esencia de trabajador.</p><p>Después de leer la Apología de Sócrates, creo que no veo maldad en esta dislexia moral, la entiendo aunque no la comparta y creo que quien la comparte es que, debido a su falta de tiempo, debido a que solo tiene en la cabeza ese bug del servicio de Azure Functions que tiene que resolver de aquí a final de este sprint, no tiene ni tiempo ni ganas a pararse a entender cual es el valor de lo que hace. Porque tampoco lo necesita. Porque es un proletario de código y no un artesano que tenga que entender el valor de su trabajo.</p><blockquote>Me pregunto si es posible una consultoría que tenga como objetivo resolver problemas y no vender soluciones.</blockquote><p>¿Considero entonces que gano poco? Depente, existe una dicotomía. Si lo mides en función al valor de mi trabajo, con respecto a lo que aporto a la sociedad, gano demasiado. Si lo mido en función a venderme como profesional, a dedicar innumerables horas (muchas fuera del trabajo) a aprender cosas que yo mismo me cuestiono, creo que gano poco.</p><h3>Conclusión</h3><p>La tecnología se ha convertido en la nueva industria, al igual que eran las fábricas en el siglo XVIII así que tienes que elegir. O bien eres el artesano que trabaja viendo el valor de su trabajo, o aceptas ser parte de una cadena de montaje.</p><p>Me pregunto si es posible una consultoría tecnológica que pase del sofismo a la filosofía, que tenga como objetivo resolver problemas y no vender soluciones. No sé si la encontraré, pero sí sé que es lo que tengo que buscar y hasta entonces… seguiré buscando.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=535173cce61b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The AI is not an Humanism]]></title>
            <link>https://medium.com/@delgadopanadero/the-ai-is-not-an-humanism-735fcf921df?source=rss-7462ba410e23------2</link>
            <guid isPermaLink="false">https://medium.com/p/735fcf921df</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[ml-so-good]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[phylosophy]]></category>
            <category><![CDATA[science]]></category>
            <dc:creator><![CDATA[DelgadoPanadero]]></dc:creator>
            <pubDate>Sun, 08 Jan 2023 10:55:43 GMT</pubDate>
            <atom:updated>2023-01-08T10:55:43.621Z</atom:updated>
            <content:encoded><![CDATA[<h4>The AI have reach the ability to think, but it has not reach the consciousness about it yet.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*F9IKdyY0iikfgUL0.png" /></figure><h3>Introduction to the problem of artificial intelligence</h3><p>Artificial intelligence (AI) has come to stay, but so have the fears and uncertainties it brings with it. At first, it seems very curious that it is capable of solving problems that require abstract capabilities, to the point where <strong>we tend to subconsciously humanize it</strong> with phrases such as “How smart it is!” or “It is thinking about it…”. But then we see that it is more complex than all that.</p><p>Beyond the anecdotal, the problem comes when we see its decision-making ability facing problems that <strong>we thought it was necessary to have a human mind</strong>; the ability to draw, to write… or even ability to understand, are examples of this. Machines were already capable of adding and multiplying very quickly, memorizing a lot of information, and working without rest. Still, they were not capable of solving problems that required that abstract reasoning ability typical of humans.</p><p>As humans, the first fear that arises is “Will it replace us?”, and when we think a little more about problems like autonomous driving or medical applications, the next doubt is “What if an AI makes a mistake and harms someone?”. Due to these two issues, in recent years, the state of the art of AI has focused mainly on making <strong>explainable AI</strong> (i.e. that a human can understand and supervise) or on <strong>ethical AI</strong> (i.e. introducing rules in its learning that ensure that its knowledge meets our standard of what is morally correct).</p><h3>The real problem</h3><p>However, despite all this, it has not changed our way of seeing them, the fear is still there, regardless of whether we understand and supervise them.</p><blockquote>Will they replace us?</blockquote><p>The answer is clear and obvious, <strong>yes, it will</strong> (at least in many tasks). And if I say it with such certainty it is because this has already happened. The printing press replaced the scribe or the harvester replaced the man with the scythe. Strangely, this did not worry us, no one felt threatened by a printing press but by AI yes.</p><blockquote>What if an AI makes a mistake and harms someone?</blockquote><p>Obviously, people will die in a traffic accident due to a decision made by an AI that was driving a car, just as people already die due to people driving. We know that the probability of having an accident in an autonomous car is <strong>much lower than if a human is driving</strong>, however, that does not make most people feel comfortable with an AI driving.</p><h3>Existentialism is a Humanism</h3><p>Second World War, Paris, 1943. We all have the image of a Nazi army devastating Europe by force, but the reality was not like that: “<em>The Germans did not force civilians to get out of the way on the sidewalks. They offered seats to elderly women on the subway. They show great affection for children and the sick. They are always extremely polite. — Jean-Paul Sartre</em>”. However, does that make the Nazis less aware of what the New Order entailed?</p><p>This is how Sartre saw how, with good words, the reaction capacity of French society was nullified. How the Parisians stopped living to survive and how day by day they were giving up a little more freedom to the point of no longer being who they were. It was in this context that Sartre defined existentialism, as <strong>a vision of humanism</strong> in which the most important quality of people is freedom.</p><blockquote>Man is condemned to be free since once in the world, he is responsible for all his actions.</blockquote><p>The basis of existentialist thought is to realize that what defines us is not “an essence” with which we are born, what defines us are our actions. It didn’t matter how kind the Nazis were, it didn’t matter if they were just pawns. They were aware of what they were doing and therefore responsible, in the same way, that they were also aware of what they “were not doing” to avoid it. It is this vision of <strong>consciousness as something innate to humanity </strong>that allows us to fully hold someone responsible for their actions.</p><h3>AI is not Humanism</h3><p>Most people do not know about existentialist philosophy, but that does not make it any less present in our humanity. All people have consciousness and therefore understand the consequences of their decisions. We have <strong>this idea so rooted in our humanity</strong> that all our social, legal (or even emotional) system is based on the need for everyone to be held responsible for the consequences of their decisions.</p><p>This idea posed no problem until AI arrived. Because AI is capable of making decisions, of “thinking” but <strong>AI has no capacity for consciousness </strong>yet<strong> .</strong> This is where the real problem of AI lies.</p><blockquote>AI is condemned to not be free since even though it can act as it pleases, it will never be conscious of it.</blockquote><p>It doesn’t matter if we understand the decisions that AI makes, it doesn’t matter if it replaces or not a human being or if it damages. The real problem is the need of people to hold someone responsible for their actions and <strong>the impossibility of holding AI responsible</strong>, as we would with a human, due to its lack of consciousness. This did not happen with the previous advances that have occurred so far, the printing press was not able to print anything on its own, and cars were not able to drive on their own. However, AI is capable of “thinking” on its own.</p><h3>Conclusion</h3><p>The real fear of humans towards AI is not based on reasons such as not understanding how it thinks or whether or not it is capable of replacing us, but rather on <strong>the difference between humans and AI, which is that AI can act without being responsible for it</strong>. Will we ever be able to give AI consciousness? Who will validate that it has that consciousness? Will each AI have its own consciousness? These are questions that we still do not know, but they are the true questions that we need to answer.</p><p><a href="https://medium.com/mlearning-ai/mlearning-ai-submission-suggestions-b51e2b130bfb">Mlearning.ai Submission Suggestions</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=735fcf921df" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[La IA no es un humanismo]]></title>
            <link>https://medium.com/@delgadopanadero/la-ia-no-es-un-humanismo-80c9c4d7358f?source=rss-7462ba410e23------2</link>
            <guid isPermaLink="false">https://medium.com/p/80c9c4d7358f</guid>
            <category><![CDATA[tecnologia]]></category>
            <category><![CDATA[filosofia]]></category>
            <category><![CDATA[inteligencia-artificial]]></category>
            <dc:creator><![CDATA[DelgadoPanadero]]></dc:creator>
            <pubDate>Sat, 07 Jan 2023 11:46:22 GMT</pubDate>
            <atom:updated>2023-01-08T08:55:37.423Z</atom:updated>
            <content:encoded><![CDATA[<h4>Las IA puede tener capacidad de pensar, pero aún no han conseguido tener consciencia de ello.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*pZ00RDfJ8PoI-XDlE0Qyhg.png" /></figure><h3>Introducción al problema de la IA</h3><p>La Inteligencia Artificial (IA) ha venido para quedarse, pero también lo han hecho los miedos e incertidumbres que trae consigo. Porque de primeras nos parece muy curioso que sea capaz de resolver problemas que requieren capacidades de abstracción, hasta el punto que <strong>tendemos a humanizarla</strong> subconscientemente con frases cómo “Que lista es!” o “Se lo está pensando…”. Pero luego vemos que es más complejo que todo eso.</p><p>Más allá de lo anecdótico, el problema viene cuando vemos que su capacidad de decisión llega a enfrentarse a problemas que creíamos que era necesario ser humano; la capacidad de dibujar, de escribir… o incluso la capacidad de entender, son ejemplo de ello. Las máquinas ya eran capaces de sumar y multiplicar muy rápido, de memorizar mucha información y de trabajar sin descansar, pero no eran capaces de resolver problemas que requerían esa capacidad de <strong>razonamiento abstracto</strong> propia de los humanos.</p><p>Como humanos, el primer miedo que nos surge es “<em>¿Nos va a sustituir?</em>”, y cuando pensamos un poco más en problemas como la conducción automática o las aplicaciones médicas la siguiente duda es “¿<em>Y si una IA se equivoca y hace daño a alguien?</em>”. Debido a estas dos cuestiones, estos últimos años, el estado del arte de la IA se ha centrado principalmente en hacer que la<strong> IA explicable</strong> (i.e. que un humano pueda entender y supervisar) o en <strong>IA ética</strong> (i.e. introducir en su aprendizaje reglas que aseguren que su conocimiento pasa nuestro standard de lo que es moralmente correcto).</p><h3>El verdadero problema</h3><p>Sin embargo, a pesar de todo esto, no ha cambiado nuestra forma de verlas, el miedo sigue estando ahí, independientemente de que las entendamos y las supervisemos.</p><blockquote>¿Nos van a sustituir?</blockquote><p>La respuesta es rotunda y evidente, <strong>sí, lo hará</strong> (al menos en muchas tareas). Y si lo digo con tanta seguridad es porque esto ya ha pasado antes. La imprenta sustituyó al escriba o la cosechadora al hombre con la hoz. Extrañamente, eso no nos preocupó, nadie se sintió amenazado por una imprenta pero en cambio por la IA sí.</p><blockquote><em>¿Y si una IA se equivoca y hace daño a alguien?</em></blockquote><p>Obvio que habrá gente que morirá en un accidente de tráfico debido a una decisión de una IA que conducía un coche, al igual que ya muere gente debido a personas que conducen. Sabemos que la probabilidad de tener un accidente en un coche autónomo es mucho <strong>más baja que si conduce un humano</strong>, sin embargo, eso no le vale a la mayoría de las personas para sentirse cómodos con que una IA conduzca.</p><h3>El existencialismo es un humanismo</h3><p>Segunda guerra mundial, París, 1943. Todos tenemos la imagen de un ejército Nazi arrasando por la fuerza Europa, pero la realidad no fue así: <em>“Los alemanes no obligaban a los civiles a abrirles paso en las aceras. Ofrecían asientos a las ancianas en el metro. Muestran un gran afecto por los niños y les dan palmaditas en la mejilla. Se les había dicho que se comportaran correctamente” — Jean-Paul Sartre. </em>Sin embargo ¿Hace eso a los Nazis menos conscientes de lo que suponía el Nuevo Orden?</p><p>Así fue cómo Sartre vio cómo, con buenas palabras, se anulaba la capacidad de reacción de una sociedad. Cómo los parisinos dejaron de vivir por sobrevivir y cómo día a día iban cediendo un poco más de libertad hasta el punto de dejar de ser quién eran. En este contexto fue cuando Sartre definió el existencialismo, cómo una <strong>visión del humanismo</strong> en la que la cualidad más importante de las personas es la libertad.</p><blockquote>El hombre está condenado a ser libre, ya que una vez en el mundo, es responsable de todos sus actos.</blockquote><p>La base del pensamiento existencialista está en darnos cuenta de que lo que nos define no es “una esencia” con la que nacemos, lo que nos define son nuestros actos. Daba igual la amabilidad de los Nazis, daba igual que solo fuesen peones. Eran conscientes de lo que hacían y por ello responsables, del mismo modo de que también eran conscientes de lo que “no hacían” para evitarlo. Es esa <strong>visión de</strong> <strong>consciencia </strong>como algo innato a la humanidad, lo que nos permite responsabilizar a alguien completamente de sus acciones.</p><h3>La IA no es un humanismo</h3><p>La mayoría de personas no conocen la filosofía existencialista, pero no por ello está menos presente en nuestra humanidad. Todas las personas tienen consciencia y por ello entienden las consecuencias de sus decisiones. Tenemos esta idea tan arraigada en nuestra humanidad que todo nuestro sistema social, legal (o incluso emocional) está basado en la necesidad de responsabilidad a todo el mundo de las consecuencias de las decisiones.</p><p>Esta idea no planteaba ningún problema hasta que llegó la IA. Porque la IA es capaz de tomar decisiones, de realizar acciones de “pensar” y sin embargo,<strong> no tiene capacidad de tener consciencia</strong> de ello. Aquí es donde surge el verdadero problema de la IA.</p><blockquote>La IA está condenada a no ser libre, ya que aún pudiendo actuar como quiera, nunca va a ser consciente de ello.</blockquote><p>Da igual que entendamos las decisiones que toma la IA, da igual sustituya o no a un ser humano o que dañe. El verdadero problema está en la necesidad de las personas de poder responsabilizar a alguien de sus actos y la <strong>imposibilidad de responsabilizar a la IA, </strong>como haríamos con un humano, debido a su falta de consciencia. Esto no pasaba con los anteriores avances que ha habido hasta ahora, la imprenta no era capaz de imprimir nada por si sola, los coches no eran capaces de conducir por sí solos. Sin embargo, la IA es capaz de “pensar” por sí misma.</p><h3>Conclusión</h3><p>El verdadero miedo de los humanos a la IA no está en razones como no entender cómo piensa o que sean capaces o no de sustituirnos, sino en <strong>a diferencia de los humanos, las IA pueden actuar sin ser responsables de ello </strong>¿Seremos algún día capaz de dotarlas de consciencia?¿Quién validará que tiene esa consciencia?¿Cada IA tendrá su propia consciencia? Son preguntas que aún no sabemos, pero son las verdaderas preguntas que necesitamos responder.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=80c9c4d7358f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A different Introduction to Time Series analysis]]></title>
            <link>https://medium.com/@delgadopanadero/a-different-introduction-to-time-series-analysis-dbe7850b0bea?source=rss-7462ba410e23------2</link>
            <guid isPermaLink="false">https://medium.com/p/dbe7850b0bea</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ml-so-good]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <dc:creator><![CDATA[DelgadoPanadero]]></dc:creator>
            <pubDate>Sat, 21 May 2022 09:08:01 GMT</pubDate>
            <atom:updated>2022-05-22T11:38:42.818Z</atom:updated>
            <content:encoded><![CDATA[<h3>A Different Introduction to Time Series analysis</h3><h4>Key concepts to understand Time Series and choose the best predictive model</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*fHDtncFj0mqqel7E" /><figcaption>Image from unsplash <a href="http://twitter.com/tobbes_rd">@tobbes_rd</a></figcaption></figure><h3>Introduction — What is a Time Series?</h3><p>A Time Serie is an ordered sequence of measures of a magnitude along the time, where each measure has a certain time associated with it. Along with this post, we are going to use the dataset from the Kaggle challenge <a href="https://www.kaggle.com/c/demand-forecasting-kernels-only"><strong>Store Item Demand Forecasting</strong></a>. The daily mean sales of this dataset have the following pattern</p><pre>df.set_index(&#39;date&#39;).plot()<br>plt.title(&#39;Store Item Demand Forecasting&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/964/1*bm3A4Hyjtn5dQm_dkCzt9g.png" /><figcaption>Image by the author: Plot of a time serie. The y-axis is always the variable that is measured, while the x-axis are their correspondent times.</figcaption></figure><p>One of the main objectives when working with Time Series is not only to understand its historical behavior but to be able to predict the future values of the series. For doing so, there are many Machine Learning models and consequently, it is not easy to know which one to use for a given dataset. Along with this post, we will understand the weakness and strengths of each of these models depending on the data nature.</p><h3>1. Statistical Analysis — Understanding the behavior of a Time Serie</h3><p>There are many statistics and metrics to characterize a Time Serie. Nevertheless, we are going to focus on those related to knowing which kind of predictive model should we use to forecast future data.</p><h4>1.1 Characterize the Time Serie — Stationary Analysis</h4><p>A Time Serie, <em>Y(t)</em>, is said to be stationary if its statistical properties do not change along the time. This concept is important because the future values are going to have the same behavior. The previous does not imply that the data is going to be predictable nor the vice-versa (for instance, a Time Serie with trend or seasonality is not stationary) but in the worst scenario, we can estimate the expected value with a certain variance at any given point of time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*SpLWMmKcFY3BJMMj.png" /><figcaption>Image from the principles of stationarity, Source: <a href="https://beingdatum.com/time-series-forecasting/">BeingDatum</a></figcaption></figure><p>Consequently,<strong> </strong>the Store Item Demand Forecasting dataset, Y(t), is not stationary. Even though sometimes a Time Serie is not stationary, some transformations can be made to convert it into stationary, for example, by removing the effect of the trend with an<strong><em> n-order differentiation</em></strong>. Given a series Y(t), the n-order differentiation is defined as</p><blockquote>Y’(t) = Y(t) — Y(t-1) — … — Y(t-n)</blockquote><p>Making a 1-order differentiation in the last year of our dataset we obtain the following</p><pre>df.sales=df.sales.diff()<br>df.plot()<br>plt.title(&#39;Rolling diff Demand&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/963/1*EwK6Y57szpIUMkRWeC6oHg.png" /><figcaption>Image by the author: In this image we can see the 1-order differentiation of the original serie from March to December of 2017. As we can see, the mean value is arround 0, while the varienace and range keeps stable from -25 to 7 sales. This means that we can consider this time interval as a stationary time serie.</figcaption></figure><p>Despite that, sometimes, the data keeps non-stationary after an n-order differentiation because the growth is not linear. For instance, if we see the whole Time Series Y(t) and not just the last year we can see that it is not stationary.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/888/1*eYQ5RdQoUSBKf4gx94Gatw.png" /><figcaption>Image by the author: 1-order differentiation of the whole time serie. We can the that along the entire history, that the variance does not keep stable, but it fluctuates with the past of the time. It means that the time serie is not stationary.</figcaption></figure><p>In those cases, we can try to make a <strong><em>log</em></strong> or <strong><em>exponential </em></strong>transformation of Y(t) beforehand. With this transformation, we can finally see that our Time Series Y(t) is stationary</p><blockquote>Y’(t) = log(Y(t))</blockquote><blockquote>Y’(t) = exp(Y(t))</blockquote><pre>df.sales=np.log(df.sales).diff()<br>df.set_index(&#39;date&#39;).plot()<br>plt.title(&#39;Rolling diff Demand&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/881/1*dBVj66WnU_HBW8ax62u_fA.png" /><figcaption>Image by the author: 1-order differentiation with a logarithmic correction of the values of the time serie. The logarithmic correction is usefull when the time serie follows an exponential growth. Now we are pretty sune that the time serie is stationary.</figcaption></figure><p>Now we can see that the variance and the mean of the series keep constant, which means that it is stationary. There are some points on particular days where the values diverge from the Time Series behavior, this kind of values are called <strong>residuals. </strong>Residuals are generally caused by a concrete temporary cause and do not change the pattern of the whole Time Series, so we do not have to worry about them.</p><h4>1.2 Autocorrelacion and Partial Autocorrelation (ACF PACF)</h4><p>To predict future values, we do not only want to understand the time evolution of the statistical properties of Y(t) but also to understand how related are the future values with past values. To do so, we use the ACF and the PACF.</p><p>The <strong><em>ACF</em></strong> is the correlation of the Time Serie Y(t) with itself in a previous moment in time of the series. Computing the correlation with <em>k</em> past values of the series, we can check which of the <em>k</em> previous values are correlated with the current value</p><blockquote>ACF = correlation( Y(t), Y(t-k) )</blockquote><pre>from statsmodels.graphics.tsaplots import plot_acf<br>plot_acf(df.sales)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*j6OZHQksGKG4metUCClXJw.png" /><figcaption>Image by the author: Partial autocorrelation. We can see that all of the 33 previous day are correlated with the current value (correlation&gt;0.25). It means that the current value is related with at least, the 33 previous values.</figcaption></figure><p>Similarly, <strong><em>PACF</em></strong> performs the correlation of <em>Y(t)</em> with a past value <em>Y(t-k)</em> but removing the effect of the intermediate values: <em>Y(t-1), Y(t-2), … Y(t-k+1):</em></p><blockquote>PACF = correlation(Y(t), Y(t-k) | Y(t-1), Y(t-2), … Y(t-k+1))</blockquote><pre>from statsmodels.graphics.tsaplots import plot_pacf<br>plot_pacf(df.sales)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/964/1*vN1YTvPINNHbVkPn8rJB1w.png" /><figcaption>Image by the author: Partial Autocorrelation. Hera we can see that the most important days are the last 8 days and peak of correlation each 7 days. It means that the time serie has a recurrence of 7 days.</figcaption></figure><p>The main difference between both is that ACF with <em>Y(t-k)</em> takes into account not only its dependence on Y(t) but also all the intermediate between <em>Y(t)</em> and <em>Y(t-k)</em> indirectly. While the PACF with <em>Y(t-k)</em> only takes into account the relationship with<em> Y(t) </em>and <em>Y(t-k).</em></p><h4>1.3 Noise and Residuals</h4><p>Sometimes it is common that the variance of the data is very high to see any pattern. This kind of pattern is called a <strong>noisy</strong> <strong>series. </strong>There is not an immediate solution to solve this problem. One possibility is to aggregate the data at a higher data level (for instance at a day level instead of an hourly level). Another common solution to handle the data is to compute the <strong>moving</strong> <strong>average</strong> of the series.</p><pre>df.rolling(window=7).mean().plot()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X0jbhSEtgq_jzo7zabLmBA.png" /><figcaption>Image by the author: Comparison of the original time serie and the window averaged time serie.</figcaption></figure><p>In our case it is not required because we are working with daily data and want to predict the daily sales. If our data was the hourly sales, it would be interesting to aggregate it into the daily sum.</p><h4>1.4 Seasonality</h4><p>Normally, the value of Y(t) is not only related to the near previous values but it can have a long period of recurrence. This kind of behavior is called seasonality. Moreover, it is common that the data does not have only one seasonality contribution but many of them at the same time (for instance a yearly seasonality, a monthly seasonality,…).</p><blockquote>For instance, the bookings in a hotel in summer are more related to the last summer bookings rather than the bookings from the previous month</blockquote><p>Assuming that the seasonality has a constant period we can calculate these periods of each contribution of the seasonality by using the <em>Fourier Series Decomposition</em><strong><em>. </em></strong>First of all, it is convenient to <strong><em>remove the effect of the trend </em></strong>from the data. If the data has a linear trend we can do that with linear regression.</p><pre>trend = LinearRegression(<br>    ).fit(df.index.values.reshape([-1,1]),df.sales<br>    ).predict(df.index.values.reshape([-1,1]))</pre><pre>df.sales = df.sales-trend</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OOBVgNvBUQbuAV9x-vg7dQ.png" /><figcaption>Image by the author: Removing the trend of the time serie. We use a linear regression because after the log transformation, the trend follows a linear growth.</figcaption></figure><p>With the series Y(t), without the trend, we can make the Fourier decomposition analysis. With the analysis, we can see the periodicity of the seasonality.</p><pre>from scipy.fftpack import fft<br>plt.plot(fft(df.sales.values))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/961/1*BCEEtKV3-FXdpX_XOoh53w.png" /><figcaption>Image by the author</figcaption></figure><h4>1.5 Results analysis</h4><p>With the analysis we have seen the following properties of the series of Sales:</p><ul><li>Stationary behavior under a logarithmic growth.</li><li>Seasonality periodicity of 8.8 months.</li><li>The last month is highly correlated to the current value.</li><li>Recurrence of 7 days.</li></ul><h3>2. Predictive Models — Forecast the evolution of the Time Serie</h3><p>Now we are going to analyze the different types of models to predict the future of the Time Series of sales. In the previous, we have analyzed the properties of the Time Series of sales and with these properties, we are going to be able to choose the best model.</p><h4>2.1 Autorregresive Models</h4><p>The autoregressive models suppose that the future values are correlated to the past values of the Time Series consequently, the model of the future values as:</p><blockquote>Y(t) = Y(t-1)*a_t-1 + Y(t-2)*a_t-2 +… Y(t-p)*a_t-p</blockquote><p>One of the most famous autoregressive models is <strong>ARIMA</strong>. This model does not only make an <strong>autoregression</strong> of the past values of the series, but also takes into account the errors of the past values through a <strong>moving average</strong>.</p><blockquote>Y(t) = Y(t-1)*a_t-1 + Y(t-2)*a_t-2 +… Y(t-p)*a_t-p +</blockquote><blockquote>+ error(t-1)*b_t-1 + error(t-2)*b_t-2 + … + error(t-q)*b_t-q</blockquote><p>Where error(t) = Y(t) — Y(t-1). This model needs the Time Series to be stationary, so it automatically makes an n-order <strong>differentiation</strong> of the series to convert the series into stationary if needed (it is important to note that it only does the differentiation, not the exponential o log transformation).</p><blockquote>Y(t) = Y(t) — Y(t-1) — Y(t-2) — … — Y(t-d)</blockquote><p>The values p, d, and q from the previous formulas are the parameters required to tune ARIMA. The p and q can be obtained from the last values of partial autocorrelation and correlation respectively with high correlation. The d value is simply the order of the differentiation that keeps the series stationary. However, ARIMA can not handle seasonality. That is why exists an extension of the algorithm called <strong>SARIMA</strong> which can handle seasonality</p><pre>import statsmodels.api as sm</pre><pre>model = sm.tsa.statespace.SARIMAX(df_train.sales,<br>    trend=&#39;t&#39;, freq=&#39;D&#39;, order=(30,1,9)<br>).fit(disp=False, solver=&#39;powell&#39;)</pre><pre>model.predict(365)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/899/1*rrZOIAVUQnW6eNqxgeM1uA.png" /><figcaption>Image by the author: Here we can see the predictions of the SARIMA model. We can see that the model perfectly catchs the seasonality, the trend and the variance of the data. This is normal noting that the time serie had a high autocorrelation and partial autocorrelation and the time serie where pretty stationary after the log and 1-order differentiation transformation.</figcaption></figure><h4>2.2 Regression Models</h4><p>As we have seen in the previous example ARIMA model and its variants are very good at predicting the future in Time Series when there is a high correlation and autocorrelation from past values. This only works if the relationship with past values is a linear dependence and it fails when the relation is nonlinear.</p><p>The solution to handle nonlinear autoregression is to use a non-linear regression model, able to stack values from a window of time to make future predictions. One of those models is the LSTM network (a particular Recurrent Neural Network). You can add the seasonal effect to the data by converting the Time Series into a tabular dataset like this</p><pre>weekday_df = pd.get_dummies(<br>    df.date.dt.dayofweek.to_numpy(), prefix=&#39;weekday&#39;)</pre><pre>df = pd.merge(weekday_df, df, how=&#39;inner&#39;, on = &#39;date&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/901/1*uNvhVg02hZF_bn5PxqO2ow.png" /><figcaption>Image by the author: Reshape of the origial time serie into a tabular structure. We have added the months for each day to let the model learn the seasonality due to each month of the year.</figcaption></figure><p>To take the near past values as the autoregressive models, we have to build a moving window stack of the near values. The LSTM model is made by stacking the window values together in an array, but other regression models (such as RandomForest) are made differently.</p><pre>x = np.zeros((len(df)-seq_size, seq_size, df.shape[-1]-1))<br>y = np.zeros((len(df)-seq_size, 1))<br>    <br>for i in range(len(df)-seq_size):<br>    x[i,:,:]=df[i:(i+seq_size),0:-1]<br>    y[i,:]=df[i+seq_size, -1:]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/271/1*l0hn7YRXfOBjINd-h7PZsg.png" /><figcaption>Image by the author</figcaption></figure><p>One advantage of this kind of model is that does not require making a deep analysis of the Time Series beforehand because they are able to handle nonlinear relationships by themself.</p><pre>from keras.models import Sequential<br>from keras.layers import LSTM, Flatten,Dropout,Dense</pre><pre>inputshape = (X_train.shape[1], X_train.shape[2])<br>model = Sequential()<br>model.add(LSTM(64, activation=&#39;tanh&#39;, return_sequences=False))<br>model.add(Dense(500))<br>model.add(Dense(1))</pre><pre>model.compile(optimizer=&#39;adam&#39;, loss = &#39;mse&#39;, metrics = [&#39;accuracy&#39;])</pre><pre>model.fit(X_train, y_train, verbose=1, epochs = 10, batch_size = 1,shuffle = False)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/957/1*kvW-Z1DdQc_43APqv7kq9A.png" /><figcaption>Image by the author: Hear we can see the predictions of the LSTM model. we can see that the model capture pretty well the tendency and the variance of the time serie. However, we can see that it fails with the seasonality from half of July to August. This is because we are adding the seasonality to the model with the month number and, as we have seen in the Fourier decomposition, the model follows a strange seasonality of 8.8 which are not entire month. For a more precise result, we should make a finer work of data engineering in the tabular reshape.</figcaption></figure><p>In this case, we can see that we can easily model the Time Series using just the past values of the data because this Time Series can be characterized by the past behavior. However, the evolution of the Time Series depends on other values or even, other Time Series. The advantage of this model is that you can include other values in the model to predict the future behavior of the Time Series.</p><h4>2.3 Fourier Series Models</h4><p>The autoregressive models are good at predicting Time Series where the future values are related to the near previous values (moving average) with a certain periodicity, as in the previous example. However, these models fail to predict Time Series which have no moving average contribution and with multiple seasonalities at the same time (daily, monthly, trimestral, weekly,…).</p><p>Currently one of the most popular models for the Time Series is <strong>Prophet</strong>. The main advantage of the model is that it is able to detect and fit simultaneously <strong>many seasonalities</strong> at the same time. At the same time, it is also able to adjust <strong>main trends</strong> and their breakpoints at the same time (exponential and linear trends).</p><p>For our current dataset, we are going to use the default seasonalities (yearly and weekly) and we are going to add also <strong>add the seasonal behavior </strong>of 8.8 months that we have seen in the Fourier analysis.</p><pre>from prophet import Prophet</pre><pre>model = Prophet(<br>    seasonality_mode=&#39;multiplicative&#39;,<br>    ).add_seasonality(name=&#39;monthly&#39;, period=260, fourier_order=5<br>                     ).fit(df_train)</pre><pre>forecast = model.predict(model.make_future_dataframe(periods=365))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/959/1*GjTfN3pXcVoeIgjgtwH6zw.png" /><figcaption>Image by the author: Here we can see the predictions of the model Prophet.We can see that it captures very well the seasonality and the trend, but it fails a little with the variance. It may be because the seasonality component in this time serie is abrupt. We could model it better by adding this information to the model through breakpoints.</figcaption></figure><p>As we can see in the code, we set the seasonality as “multiplicative”. With the autoregressive model, we have seen the effect of the seasonality as an additive contribution to the Time Series</p><blockquote>Y(t) = Y_trend(t) + Y_season(t)</blockquote><p>However, sometimes the effect of the seasonality changes the trend by a factor rather than an addition. Prophet is able to fit these two kind of behavior</p><blockquote>Y(t) = Y_trend(t) * Y_season(t)</blockquote><p>With Prophet, we can also see the component decomposition of the series, as we can see the different components of the Time Series</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/582/1*dcxaOO_eS6RZ8X6qtnEW0g.png" /><figcaption>Image by the author: Components of the prediction from Prophet</figcaption></figure><h3>3. Conclusions</h3><p>In this post, we have explained some statistical tools to analyze and understand the behavior of a Time Series. Understanding this behavior is important not only for interpreting the past values, but also to know how to model the predictive models to forecast the future values of the time series.</p><p>From all the predictive models, we have defined 3 different kinds of models according to their behavior. There is no best model, but depending on the nature of the Time Series, we should choose which one to use. Here we show a simple table to understand the capabilities of each of the previous models</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/922/1*ZYFSkNDFJS8uigMgnuIEAw.png" /><figcaption>Image by the author: Cheat sheet of the strengths and weakness of each of the models.</figcaption></figure><p>I hope you liked it :)</p><p><a href="https://mlearning.substack.com/">MLearning.ai Art | Datasculptor | Substack</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dbe7850b0bea" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding the foundations of Theoretical Physics]]></title>
            <link>https://medium.com/@delgadopanadero/understanding-the-foundations-of-theoretical-physics-1645106c9018?source=rss-7462ba410e23------2</link>
            <guid isPermaLink="false">https://medium.com/p/1645106c9018</guid>
            <category><![CDATA[science]]></category>
            <category><![CDATA[mathematics]]></category>
            <category><![CDATA[physics]]></category>
            <dc:creator><![CDATA[DelgadoPanadero]]></dc:creator>
            <pubDate>Fri, 04 Mar 2022 09:13:37 GMT</pubDate>
            <atom:updated>2022-03-16T18:12:26.589Z</atom:updated>
            <content:encoded><![CDATA[<h4>The main of this post is to review the reasons that bring the mathematical definition of physics and why we need a more complex understanding than the laws of Newton</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-4uHHQEesH--suMbP1RzoA.png" /><figcaption>Image by the author. This building is in Madrid, in Plaza España. I took this photo in January of 2022. I do not know the history of the building but the symmetry reminds me a lot of the beauty of the mathematical symmetries from physics.</figcaption></figure><h3>1 —Newton’s Mechanics</h3><p>One of the main goals of physics is to understand why everything moves as it does? What is the cause? and Can we predict what is going to happen?</p><p>In the beginning, the unique approach to understanding and predicting the movement of the objects (for instance planets and falling bodies) was to measure every position of the objects in movement and expect that they would behave the same in the future. But… <strong>Newton</strong> appeared! showing that the movements of the planets, the objects, the fluids,… can be understood by forces, and the acceleration of every object is <strong>produced by the forces</strong> that are applied to it</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/261/1*rIRLKY_mI-slgTnDcDOg-Q.png" /><figcaption>Image by the author: Newton’s second law of mechanics. This formula shows that the magnitude and the direction of the acceleration of an object are proportional and with the same direction to the forces applied to it.</figcaption></figure><p>We all have seen this formula before in school but its meaning is deeper than we think. We may think that it only allows us to compute the value of the <strong>acceleration</strong> having the value of the force, however, it is more than that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/251/1*EraTdLb6hCTS4HrGtWjz0w.png" /><figcaption>Image by the author: The formula of the acceleration in terms of the first derivative of the velocity and the second derivative of the position. The derivative just measures how much something changes at each moment of time. For instance, the acceleration measure how much changes the velocity at each moment of time, and the velocity measure how much changes the position at each moment of time.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/345/1*9YnM_B1e8RDJPjeCXQLHWQ.png" /><figcaption>Image by the author: The trajectory of the path described by the object in terms of the time</figcaption></figure><p>Noting that the <strong>acceleration is the second derivative </strong>of the position if the force is a function that can be evaluated at every point of the space, this equation explains how the object is going to move in every point of its movement to the very next moment. It is not about the magnitude of the acceleration but the geometric evolution of the position, so we can obtain the whole <strong>trajectory as the integral</strong> of the acceleration function</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/305/1*4VGRD-3MRBKUm8hFqdilcw.png" /><figcaption>Image by the author: Computation of the trajectory as the integral of the velocity and the velocity as the integral of the acceleration. The integral is the inverse operation of the derivative, so if we know the velocity (the change of the position) at each moment of time, we can get, using the integration, the trajectory of the object.</figcaption></figure><h3>2 — Work and Energy</h3><p>As we have explained, the force is the key to understanding the trajectory at each point, but also to predicting the future of the trajectory. However, sometimes we want to predict <strong>other magnitudes</strong> of the movement beyond the position, for example, the work</p><p>Sometimes we do not want to predict a movement of an object given a force but the opposite: To know how much effort should we make to move the object from one point to another. This magnitude is called <strong>work</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/231/1*IUGUxYShyR-QsYqi3YlP7A.png" /><figcaption>Image by the author: The work, W, is computed as the product of the force applied in each point during the movement, and the distance of the trajectory of the movement. The symbol δ in the formula means that the value work, W, depends not only on the distance but on the trajectory of all the possible paths between the initial and final points.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/212/1*RYKsS_VCd87y3JBEI-ncyQ.png" /><figcaption>Image by the author: Possible paths between the point A and the point B</figcaption></figure><p>In general, the value of the work depends on the trajectory followed by the object, however, for a certain type of forces, the value of the work only depends on the initial and final point, independently on the trajectory. This kind of force is called <strong><em>conservative force</em></strong> and its work can be expressed as the difference of a function of the initial and final point of the trajectory</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/313/1*2hFMbCJKf0I0K4Gl7fLtHw.png" /><figcaption>Image by the author: For a conservative force the work can be expressed as the difference between the potential energy in the initial and final point of the trajectory</figcaption></figure><p>The function U is called <strong>potential energy </strong>and it holds all the information of the force. In fact, because of that characteristic, it can be proven that a conservative force can be expressed as the gradient of the energy function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/220/1*PGugKPxOQdefSaghFcLGTQ.png" /><figcaption>Image by the author: The conservative forces can be expressed as the gradient of a scalar function U. The scalar function is just a function that returns a number (not a vector, as the force) and the symbol ∇ is the gradient (the directional derivative of a scalar function).</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*M6w1wHEQw4jEsh9IBRHCLw.png" /><figcaption>Image by the author: Potential energy of a planet in a gravitational field. It is the sum of the potential energy because of the gravity force and the potential energy because of the rotation.</figcaption></figure><p>But, what does it happens with the work produced by the force? Is it lost? Is it used by something? The answer is no, there is no disappearance of energy because of a force but it is used to move the object, so we can define the energy used to move that object as the <strong>kinetic energy.</strong></p><p>The kinetic energy is expressed mathematically as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/174/1*FeqfKeP6nIbamIhs8Iv9vA.png" /><figcaption>Image by the author: Mathematical definition of kinetic energy. As well as the gradient of the potential energy is the value of the conservative force in each part of the space, the kinetic energy is defined by the acceleration when derived in time.</figcaption></figure><h3>3 — D’Alembert’s Principle</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/1*lZ4esx9otDEgNw4cbfaJyA.png" /><figcaption>Image by the author: In Newton’s approach we compute the trajectory of the ball by summing all the forces (external forces and constrain forces) and making it equal to the acceleration</figcaption></figure><p>The approach of Newton to understand the trajectory of the objects is to compute the evolution of the position (acceleration) in every point given <strong>all the forces that are applied</strong>. For instance, to understand why a ball rolls, we have to take into account two forces, the force that made it move and the force of the ground holding it. Newton&#39;s formula takes the sum of these two forces in every point to compute the very next point of the trajectory. However, this is not the unique approach to studying the trajectory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/1*b8-LpA15o4Da6YZRZ__x8A.png" /><figcaption>Image by the author: In D’Alembert’s approach we compute the trajectory of the ball by taking into account only the external forces.</figcaption></figure><p>In Newton’s approach, some of the forces act as a <strong>constrain </strong>of the movement but do not produce work (for instance the force of the ground that holds the object produces a null work) so they are not the cause of the movement. Consequently, we can also understand the movement by saying that the only net job is produced by <strong>only the forces that produce movement </strong>(external forces). This can be written mathematically as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/293/1*LWZ_ixOspREwmBzCk3rWJw.png" /><figcaption>Image by the author: In this equation, F is only the forces that produce movement. The difference between these forces and the acceleration is the constrains forces. So what this formula tells us is that the work of the constrained forces is zero.</figcaption></figure><p>Rather than understanding the position evolution at each point of the trajectory, we can think about the trajectory of the body as a whole. During all the trajectory, all the acceleration is produced only by the external forces, so the <strong>work of all other forces must be null. </strong>This approach is the principle of D’Alembert.</p><p>Both equations (Newton and D’Alembert) reflect the same result, the unique <strong>difference is the interpretation</strong> and the computation. For simple problems, it is more convenient to use the second Newton equation. The main advantage of D’Alembert’s principle is in the cases where there are many force constraints because it only takes into account the external forces (those that produce movement).</p><h3>4 — The Euler-Lagrange’s Equation</h3><p>As we mentioned, meanwhile the Newton equation has to take into account all the forces (external forces and constrain forces) D’Alembert only takes into account external forces. As we mentioned, in the special where the external forces are conservative (for instance in the case of an object falling because of gravity), we can rewrite the formula of <strong>D’Alembert in terms of the energy (</strong>potential and kinetic). After some mathematics it would be like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/346/1*EQWZ4OHW8B4BR1SQgKaQSw.png" /><figcaption>Image by the author:</figcaption></figure><p>and also, in case the potential energy only depends on the coordinates, if we define the quantity <strong><em>L=T-V</em></strong>, we can write the previous expression as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/270/1*_bI4RC0WY7Nv24fEWo0kLQ.png" /><figcaption>Image by the author: Lagrangian formula</figcaption></figure><p>This is called the Euler-Lagrange equation. Ok, very fancy, but what is the purpose of all of these, and <strong>what is the advantage</strong> of Newton’s or D’Alembert’s approach?</p><p>Basically, the difference is mainly philosophical and how we understand the cause of the movement. In Newton’s equation, we compute the positions of a body as the result of a force, while the Euler-Lagrange’s we build a function that <strong>contains all the information</strong> produced by a source of force (Lagrangian function) and the movement of every object in every part of the space can be derived from that.</p><p>Whereas Newton’s formula tries to solve a concrete problem, Euler-Lagrange tries to reflect the <strong>nature of the force</strong>.</p><h3>5 — Conclusion</h3><p>In this post, we have made a brief overview of the derivation of the Theoretical Physics from the Newton’s Law and what does theoretical physics is needed to answer more questions beyond the computation of the trajectory of a movement.</p><p>The logical sequence that we have made is the following:</p><ul><li>Newton’s law explains the movement of an object because of a force.</li><li>The potential and kinetic energy are needed to measure other features of the movement beyond the trajectory (the work).</li><li>D’Alembert re-interprets the same result of Newton’s laws but using the forces that produce any work along the trajectory</li><li>Rewriting the D’Alembert formula in terms of the energies, rather than forces, we get a formula that has both: the global information of the trajectory of the object and other features beyond the positions, such as the work.</li></ul><p>The physics defined by the Newton laws and the theoretical Physics are the same. The main difference between both is just that one tries to compute the forces and acceleration to solve a certain problem, while the theoretical physics tries (through the energy functions) to understand the characteristic of the source of the force as well as to solve the most general solution.</p><h3>6 - Next Steps</h3><p>In this post, we have just knocked on the doors of theoretical physics. They are more functions and theorems (<strong>Hamiltonian</strong>, Noether theorem,…) that do not try to solve certain trajectories but answer questions of nature such as “is the nature of this problem solvable?”, “How does the solution change with the frame of reference?”, “What are the invariants of the movement?”,…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Tz4zf-ozBi1bnBTjElxGcw.png" /><figcaption>Image by the author: Phase space of the Hamilton canonical variables in a pendulum</figcaption></figure><p>I hope you liked it :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1645106c9018" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Aim of Scikit-Learn 1.0]]></title>
            <link>https://medium.com/data-science/the-aim-of-scikit-learn-1-0-985642811f27?source=rss-7462ba410e23------2</link>
            <guid isPermaLink="false">https://medium.com/p/985642811f27</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[scikit-learn]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[DelgadoPanadero]]></dc:creator>
            <pubDate>Tue, 12 Oct 2021 17:26:31 GMT</pubDate>
            <atom:updated>2021-10-12T17:26:31.369Z</atom:updated>
            <content:encoded><![CDATA[<h4>What are the new features and why are they so important to be the first major release?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iYCCXCfq9iEf3-TLHGwWDg.png" /><figcaption>Image from Unsplash (Mae Mu)</figcaption></figure><p>It has been 14 years and 24 beta releases since the beginning of Scikit-Learn and finally, it has reached its 1.0 release. It may sound a bit strange noting the fact that Scikit-Learn has been used by thousands of companies, data scientists, researchers… for a long time and everyone considers it as the most spread framework for general purpose Machine Learning.</p><p>In this article, I do not want to make an analysis of the new features as many other articles do but to <strong>understand the aim of Scikit-Learn with this release</strong> and what is its strategy for future developments</p><h3>1. Some history</h3><p>Scikit-Learn was born in 2007 first as a Google Summer of Code project and continued being developed in a <strong>researching environment</strong>. Its objective was to serve as a tool to make data analysis without having to focus on any particular technology or code. For this reason, it is based on Python, an open-source language, easy to use, general-purpose, and able to embed C code</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/888/1*rEP4qHt2W65lzBTqpibtvg.png" /><figcaption><strong>Image by the Author</strong>: Scikit-Learn as the Force that the humble Data Jedis need to fight against the Imperial Companies with reliable and efficient library tools.</figcaption></figure><p>Another big problem when working with data is the computational resources in terms of memory and processing, so Scikit-Learn has always made a big effort in improving the algorithm efficiency to allow even the <strong>users with low computational resources </strong>to work with data. Scikit-Learn does it by using statistical approximations and low-level code (Cython).</p><p>Moreover, the key point of Scikit-Learn beyond efficiency and simplicity is the documentation. Many Data scientists (myself) have learned Machine Learning by reading the documentation of Scikit-Learn. It is not expected to just be code documentation but a learning path for Data Science.</p><h3>2. Release 1.0</h3><p>If we see the highlights of the release we can see that there are changes in the API and even some cool new feature</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-h1FjhjhI5PA7R70rD4xcw.png" /><figcaption>Image: Caption from scikit-learn.org</figcaption></figure><p>However, by seeing the <a href="https://scikit-learn.org/stable/whats_new/v1.0.html"><strong>changelog</strong></a> page and plotting the tags extracted by processing the HTML page, we can that the majority of the tags are fixes and API changes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RsOK6kTuJcdku3wWghMn4w.png" /><figcaption><strong>Image by the Author</strong>: Number of features per tag and the code to generate it.</figcaption></figure><p>In the next section, we are going to dive into the three main topics that can sum up all the changes of the release.</p><h3>3. Release highlights</h3><h4>3.1. API Standardization</h4><p>One important pattern in the library interface is that all the <strong>modules tend to be interchangeable</strong>. This means, for instance, if you are trying to build a supervised model, the functions and methods to fit, test, predict and measure the accuracy of this model, are independent of the flavor of the supervised model that you are building (linear regression, decision trees, k-means…).</p><blockquote>Standard interface and signature between objects with the same functionality.</blockquote><p>However, the signature of the module was the same, but the values that the expected were not, between different modules and releases (e.g. “X should be np.matrix or np.array?”, “loss=’ls’ or loss=’mse’?…”). Some highlights to solve this are:</p><ul><li><strong>Signature</strong>: Now they enforce to use of only keyword-only arguments.</li><li><strong>Data types</strong>: new features are working with Pandas (for instance estimators store the feature_names of the pd.DataFrame when training). Meanwhile, the type np.matrix is deprecated.</li><li><strong>Argument values</strong>: Some functions and modules have the same arguments (loss, scaler, criterion,…) but the values it expected were different and this has changed. Some encoders can now accept missing and unknown values.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GUnUMN3ONTfwh3Q4jLKO_Q.png" /><figcaption><strong>Image by the Author</strong>: Now the argument values need to be passed with the argument name. This is a good restriction to avoid problems with legacy bad-designed code.</figcaption></figure><h4>3.2. Computation Performance</h4><p>When you create a library to work with data thinking to be served to everyone, you also have to consider that the majority of people do not have high computational resources, but just <strong>humble home laptops</strong>. That is why Scikit-Learn has always tried to use low-level embedded languages (Cython) to increase performance (such as in SVM and GBDT algorithms).</p><blockquote>Scikit-Learn allows to use Machine Learning in humble laptops with low resources.</blockquote><p>In this release more efficiency enhancements have been made in many functions and modules, for instance:</p><ul><li><strong>Preprocessors: </strong>(StandardScaler, KBinsDiscretizer, PolynomialFeatures)</li><li><strong>Estimators: </strong>(Logistic regression, Neighbors, Cluster, … algorithms)</li><li><strong>Dimensionality Reduction algorithms</strong></li></ul><p>And also, new features that are not a performance enhancement but new models that take advantage of statistical properties to performs faster than the original models:</p><ul><li><strong>Online One-Class SVM: </strong>Use One-Class SVM with stochastic gradient descent and kernel approximations to reduce the complexity from quadratic to linear.</li><li><strong>HistGradientBoostingClassifier: </strong>Gradient boosting implementation where the values of the features are split into splines. The trees learn the spline values instead of the value itself reducing cardinality.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VpzQQ5fdCv7T5xot6NZoYA.png" /><figcaption>Image by the Author: Comparison of the computation time of the Gradient Boosting Classifier and the Histogram Gradient Boosting Classifier and the code to generate the plot.</figcaption></figure><h4>3.3. Measurement</h4><p>It is a common mistake to think about Scikit-Learn just as a Machine Learning framework but the truth is that it goes further. It provides a set of tools not only to develop models but also to measure and understand the predictions of a trained model.</p><blockquote>This point is specially important when you do not just want to create a predictive model but to extract insights from what it has learned</blockquote><p>This release brings some new features that ease the metric computation and also some new metrics:</p><ul><li><strong>Added method:</strong> for metrics and plots from_estimator and from_predictions.</li><li><strong>New metrics:</strong> pinball_loss and tweedie_score.</li><li><strong>New plots</strong> for displaying results CalibrationDisplay.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*N2bFAKthB-v0vPRg.png" /><figcaption>Image from Scikit-Learn documentation</figcaption></figure><h4>3.4. Community</h4><p>Last but not least, are other kinds of changes that are not related to any technical reason but just to solve concerns and needs from the community. Scikit-Learn has always tried to be more than just a library but a complete environment to bring machine learning knowledge to people and define some standards. and a proof of that is the following fact:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/724/1*-ZgiinlpDuP-Tp1wjJr_sA.png" /><figcaption>Caption from <a href="https://github.com/scikit-learn/scikit-learn/issues/16155">Scikit-Learn GitHub issues</a></figcaption></figure><blockquote>Out of over 2100 merged pull requests, about 800 of them are improvements to our <strong>documentation</strong>.</blockquote><p>These updates in the documentation are mainly to increase the quality of the resources to the users and also to supply more needs that they require. This needs can be also for ethical reasons such as the removal of the Boston dataset</p><h3>4. Conclusions</h3><p>In this article, we have covered the main features of release 1.0 of Scikit-Learn and tried to explain their importance from the historical context of the library.</p><p>It may sound strange that the first major release was mainly Fixes and API changes instead of new features, however, understanding the aim of Scikit-Learn, we have seen that its purpose has always been to</p><blockquote>Define a standard of Machine Learning to bring tools to the majority of the people independently if they are researcher, employers, students or just as a hobby.</blockquote><p>and this is exactly what they have achieved with this release.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=985642811f27" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/the-aim-of-scikit-learn-1-0-985642811f27">The Aim of Scikit-Learn 1.0</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My ML Model Fails. Why? Is It the data?]]></title>
            <link>https://medium.com/data-science/my-ml-model-fails-why-is-it-the-data-d8fbfc50c254?source=rss-7462ba410e23------2</link>
            <guid isPermaLink="false">https://medium.com/p/d8fbfc50c254</guid>
            <category><![CDATA[statistical-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[business-intelligence]]></category>
            <dc:creator><![CDATA[DelgadoPanadero]]></dc:creator>
            <pubDate>Wed, 01 Sep 2021 18:46:40 GMT</pubDate>
            <atom:updated>2021-10-13T13:22:18.395Z</atom:updated>
            <content:encoded><![CDATA[<h4>Understand if the model does not perform well because of a bad model selection or because of noise in the training data with a real example.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WpLAABF4-HXlJWYB.jpg" /><figcaption>Image: <a href="https://commons.wikimedia.org/wiki/File:%22The_School_of_Athens%22_by_Raffaello_Sanzio_da_Urbino.jpg">The School of Athens by Raphael 1509</a></figcaption></figure><h3>Introduction</h3><p>One of the most common problems in Machine Learning when you build and train a model and you check its accuracy is “<strong>Is the accuracy the bes</strong>t I can get from the data or could a find a better model?”.</p><p>Also, once your model is deployed, the next common question is “<strong>Why does the model fail</strong> when it does?”. Sometimes neither of these questions can be answered but sometimes we can find preprocessing errors, model bias, and also data leaks by studying the statistical distribution of the model errors.</p><p>In this tutorial, we will explain and demonstrate how to make a statistical study of the model results to <strong>discover the causes of the errors </strong>in an example<strong>.</strong></p><h3>Business Case</h3><p>In this example, we are going to use the data from a <a href="https://www.drivendata.org/competitions/50/worldbank-poverty-prediction/">Driven Data contest</a> to predict from a set of socio-economical variables of a people, whether they live in a poverty status situation or not.</p><p>The value of this business case is not only to be able to <strong>predict the poverty status</strong> with a machine learning model but also to understand how predictive are the socio-economical variables to measure poverty and also to analyze its reasons from the features.</p><h3>Model Training</h3><p>The data consist of a set of nine descriptive variables, four of the categorical and the other five numeric (but one of them seems to be an id so we will discard it).</p><pre>import pandas as pd<br><br>pd.set_option(&#39;display.max_columns&#39;, None)<br>train = pd.read_csv(&#39;train.csv&#39;, index_col=&#39;id&#39;)<br>print(train)</pre><p>which returns</p><pre>  Unnamed: 0 kjkrfgld bpowgknt raksnhjf vwpsxrgk  omtioxzz  yfmzwkru<br>id                                                                          <br>29252  2225    KfoTG    zPfZR    DtMvg      NaN      12.0      -3.0   <br>98286  1598    ljBjd    THHLT    DtMvg    esAQH      21.0      -2.0   <br>49040  7896    Lsuai    zPfZR    zeYAm    ZCIYy      12.0      -3.0   <br>35261  1458    KfoTG    mDadf    zeYAm    ZCIYy      12.0      -1.0   <br>98833  1817    KfoTG    THHLT    DtMvg    ARuYG      21.0      -4.0   <br><br>       tiwrsloh  weioazcf   poor  <br>id                                <br>29252      -1.0       0.5  False  <br>98286      -5.0      -9.5   True  <br>49040      -5.0      -9.5   True  <br>35261      -5.0      -9.5  False  <br>98833      -5.0      -9.5   True  </pre><p>The data distribution can be seen here</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c3nzlYF2ui40BiE3-tyJZQ.png" /><figcaption>Image by author: Pairwise plot of all the features in the dataset colored with the target. In yellow poor=False and violet poor=True.</figcaption></figure><p>After some preprocessing (NaN values imputation, scaling, categorical encoding,…), we are going to train an SVM model (which usually works well in high dimensionality data from the one-hot encoding)</p><h4>Support Vector Machine</h4><pre>from sklearn.pipeline import Pipeline<br>from sklearn.preprocessing import RobustScaler<br>from sklearn.neighbors import KNeighborsClassifier<br><br>model = Pipeline(steps=preprocess+[<br>                 (&#39;scaler&#39;, RobustScaler()),<br>                 (&#39;estimator&#39;, KNeighborsClassifier(n_neighbors=5))])<br><br>model.fit(X_train, y_train)<br>y_pred = model.predict(X_test)<br>print(classification_report(y_test,y_pred))</pre><p>which returns</p><pre>precision    recall  f1-score   support<br><br>       False       0.73      0.77      0.75       891<br>        True       0.70      0.66      0.68       750<br><br>    accuracy                           0.72      1641<br>   macro avg       0.72      0.71      0.71      1641<br>weighted avg       0.72      0.72      0.72      1641</pre><p>An accuracy of 0.72 is not a very good accuracy from a binary classification problem. In contrast, the recall and precision seem to be balanced which makes us think that the model is not a prior bias in favor of any of the categories.</p><h3>Test other models</h3><p>The very next step to try to improve the model is to try other ML models and hyperparameters to see if we find any configuration that improves the performance (or even only check that the performance does not keep constant in a plateau).</p><p>We are going to use two <strong>other models from a different set of family functions</strong>. The KNN model, which is a good option to learn local patterns influences, and also Gradient Boosting Trees, which is one of the ML models with the highest capacity</p><p><strong>K-Nearest Neighbors</strong></p><pre>from sklearn.pipeline import Pipeline<br>from sklearn.preprocessing import RobustScaler<br>from sklearn.neighbors import KNeighborsClassifier<br><br>model = Pipeline(steps=preprocess+[<br>                 (&#39;scaler&#39;, RobustScaler()),<br>                 (&#39;estimator&#39;, KNeighborsClassifier(n_neighbors=5))])<br><br>model.fit(X_train, y_train)<br>y_pred = model.predict(X_test)<br>print(classification_report(y_test,y_pred))</pre><p>which returns</p><pre>precision    recall  f1-score   support<br><br>       False       0.71      0.74      0.72       891<br>        True       0.67      0.63      0.65       750<br><br>    accuracy                           0.69      1641<br>   macro avg       0.69      0.69      0.69      1641<br>weighted avg       0.69      0.69      0.69      1641</pre><h4>Gradient Boosting</h4><pre>from sklearn.pipeline import Pipeline<br>from sklearn.ensemble import GradientBoostingClassifier<br><br>model = Pipeline(steps=preprocess+[<br>                 (&#39;estimator&#39;, <br>                  GradientBoostingClassifier(max_depth=5,<br>                                             n_estimators=100))])<br><br>model.fit(X_train, y_train)<br>y_pred = model.predict(X_test)<br>print(classification_report(y_test,y_pred))</pre><p>which returns</p><pre>precision    recall  f1-score   support<br><br>       False       0.76      0.78      0.77       891<br>        True       0.73      0.70      0.72       750<br><br>    accuracy                           0.74      1641<br>   macro avg       0.74      0.74      0.74      1641<br>weighted avg       0.74      0.74      0.74      1641</pre><p>We can see that both of the other models seem to perform very similarly. This raises the following questions:</p><blockquote>¿Is this the best we can predict with a ML model?</blockquote><h3>Model prediction distribution</h3><p>Beyond the general metrics to check the performance it is also important to analyze the model output distribution. We are going to check the distribution not only in the test dataset but also in the training dataset. The reason for that is because we do not want to see how well our model performs but <strong>if it has learned how to split the training data</strong> too.</p><pre>import matplotlib.pyplot as plt</pre><pre>pd.DataFrame(model.predict_proba(X_train))[1].hist()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*d9_D310c-XeXUMpx5pZYyw.png" /><figcaption>Image by author: Model outputs distribution evaluated over the training set</figcaption></figure><pre>pd.DataFrame(model.predict_proba(X_test))[1].hist()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*BUfVOwR3zbeevk0xC5RIOQ.png" /><figcaption>Image by author: Model output distribution evaluated over the test set</figcaption></figure><p>We can see that there is a <strong>higher peak in the number of predictions of 0</strong>, which suggests that there is a subset of data which the model is pretty sure that its label is 0. Beyond this, the distribution seems to be quite uniform.</p><p>If the model would have known to certainly distinguish between the two labels, the distributions would have two peaks, one near 0, and the other near 1. Consequently, we can see that our model has not learned properly the patterns to distinguish the data.</p><h3>Deviation distribution</h3><p>We have seen that the model has not learned to clearly distinguish between both categories, but we have not seen if it guesses the prediction even though not being confident or it just keeps failing.</p><p>Also, it is important to check if the model fails more in favor of one category or another. To check these two aspects, we can plot the distribution of the deviation of the predictions from the target</p><pre>train_proba = model.predict_proba(X_train)[:,1]<br>pd.DataFrame(train_proba-y_train.astype(int)).hist(bins=50)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*az4pTXSpZUShPq86x1mnyQ.png" /><figcaption>Image by author: Deviation of the model confidence output from its ground truth evaluated over the train set</figcaption></figure><pre>test_proba = model.predict_proba(X_test)[:,1]<br>pd.DataFrame(test_proba-y_test.astype(int)).hist(bins=50)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*a3TwWyT9Q8yasGUjaSCstg.png" /><figcaption>Image by author: Deviation of the model confidence output from its ground truth evaluated over the test set</figcaption></figure><p>From these two images, we can see that the deviation distribution seems to be <strong>symmetric and centered in zero</strong>. The gap is just in zero because the model never returns the exact value of 0 and 1, we do not have to worry about that.</p><p>If the errors from the models came from statistical/measure noise errors from the training data and not bias errors, we would expect that the deviation distribution followed a Gaussian distribution<strong>.</strong></p><p>Our distributions are similar to a <strong>Gaussian distribution</strong> with a higher peak in zero, but this peak can be because of the higher number of zeros predicted by the model (i.e. the model has learned a pattern to distinguish a subset of 0 from the 1 category).</p><h3>Normality Test</h3><p>Before considering that the model predictions deviation can be justified because of statistical noise in the training data, we have to make sure that they follow a Gaussian distribution.</p><pre>import scipy<br><br>scipy.stats.normaltest(train_proba-y_train.astype(int))</pre><p>with returns</p><pre>NormaltestResult(statistic=15.602215177113427, pvalue=0.00040928141243470884)</pre><p>With a p-value=0.0004 we can assume that the deviation of the predictions from the target follows a Gaussian distribution,<strong> </strong>which makes plausible the theory that the model <strong>errors came from noise in the training data</strong>.</p><h3>Model Explainability</h3><p>As we mentioned before, the goal of this business case is not only a model that can predict why it happens and the socio-economical variables related to it.</p><p>The explainable models are those that not only make predictions over unseen data but also give you information about how do the features affect the model (global explainability) and also why a certain prediction has been that (local explainability).</p><p>Nevertheless, the less, the explainability of a model can also be v<strong>ery helpful to understand why it does the predictions</strong> and why it fails. From the Gradient Boosting model we can extract the global explainability as follows:</p><pre>cols = X_train.columns<br>vals= dict(model.steps)[&#39;estimator&#39;].feature_importances_<br><br>plt.figure()<br>plt.bar(cols, vals)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*c3KVA_ijQOhWTvbq59uabw.png" /><figcaption>Image by author: Gradient Boosting feature importances</figcaption></figure><p>Now, we are going to make the same feature importance analysis but just training in a subset of the data. Concretely we are going to train the model with just the clearly zero data (those that the model clearly predicted before as zero) for the zero categories.</p><pre>zero_mask = model.predict_proba(X_train)[:,1]&lt;=0.1<br>one_mask = y_train==1<br>mask = np.logical_or(zero_mask,one_mask)<br>X_train = X_train.loc[mask,:]<br>y_train = y_train.loc[mask]<br>model.fit(X_train,y_train)</pre><p>Now the feature importance is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*jHc0wd_Vnl0T1rBvZUxpyg.png" /><figcaption>Image by author: Gradient Boosting feature importances trained over the subsample of the train set where the model performs the best</figcaption></figure><p>We can see that now, the variables, <em>tiwrsloh,</em> and <em>yfmzwkru</em> have increased their importance while <em>vwpsxrgk</em> has decreased its value. This means that there is a <strong>subset of the population that is clearly not poor </strong>(category 0) and can be characterized by these two variables from the poor variables and <em>vwpsxrgk </em>can be important in many cases but not quietly decisive</p><p>If we plot the filtered values for these two features we see:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/418/1*zrx-sR4LvLzvFLByB7vKaw.jpeg" /><figcaption>Image by author: Segment and characterize the feature area where the model clearly detects non-poverty.</figcaption></figure><p>For these two features, the model has learned to distinguish between two categories meanwhile for other values of these variables, with the entire dataset, the zero categories and the one category are mixed so can not clearly be distinguished.</p><p>We can also characterize a clear subset of <strong>non-poor people from the previous graph as those with <em>tiwrsloh</em>&lt;0 and <em>yfmzwkru</em>&lt;-2</strong>.</p><h3>Conclusions</h3><ul><li>We have analyzed the problem of the detection of poverty in a given dataset and analyze its causes given socio-economical data and found that poverty is not easy to predict, however, we can define some regions that clearly determine the poverty status of the people: <em>tiwrsloh</em>&lt;0 and <em>yfmzwkru</em>&lt;-2.</li><li>We have tried many different models and configurations that fall in a plateau in performance at 0.75. With this and the statistical nature of the distribution of the model predictions and error deviation, we can conclude that the problem is a lack of predictive capacity from the training data to predict the target. So it is not possible to build a better model.</li><li>The reason why the data is not predictive enough may be because of noise, but also because of the lack of some predictive feature to predict poverty beyond the socio-economical features from our dataset.</li></ul><h3>What have we learned?</h3><p>We have tackled the problem where the model does not get a good enough result with a real example. The goal in the situation is to try to understand where does the model fails to understand if the problem is in the data or the model.</p><p>The process to answer this question is:</p><p>1 — Try different family function models and hyperparameter and confirm that all of the fall in a plateau in performance.</p><p>2 — For the best of explainable one, compute the model output distribution and the deviation distribution for the target. If the data is the problem, the output must be uniform and the deviations must follow a Gaussian distribution</p><p>3 — Although the data were the problem, try to find an area where the model performs well from the model outputs and deviations distributions. Try to segment and characterize this area, for instance, retraining the model with this subset and extracting its explainability.</p><p>4 — Moreover of characterizing some subset, we can try to think from the business knowledge if the problem is the statistical/measure noise from the data or from a lack of some feature value which is needed to predict the dependent variable.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d8fbfc50c254" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/my-ml-model-fails-why-is-it-the-data-d8fbfc50c254">My ML Model Fails. Why? Is It the data?</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>