<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dud Díaz on Medium]]></title>
        <description><![CDATA[Stories by Dud Díaz on Medium]]></description>
        <link>https://medium.com/@MrDeds?source=rss-dc5eeba0106c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*q_5vZmZcwI4UpxMAiEYMOg.jpeg</url>
            <title>Stories by Dud Díaz on Medium</title>
            <link>https://medium.com/@MrDeds?source=rss-dc5eeba0106c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 23:52:15 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@MrDeds/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Making a very simple [CI/CD] workflow with Github Actions]]></title>
            <link>https://medium.com/@MrDeds/making-a-very-simple-ci-cd-workflow-with-github-actions-b4ad8ef6cb9b?source=rss-dc5eeba0106c------2</link>
            <guid isPermaLink="false">https://medium.com/p/b4ad8ef6cb9b</guid>
            <category><![CDATA[github]]></category>
            <category><![CDATA[computing]]></category>
            <category><![CDATA[git]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[continuous-integration]]></category>
            <dc:creator><![CDATA[Dud Díaz]]></dc:creator>
            <pubDate>Tue, 12 Nov 2019 01:56:09 GMT</pubDate>
            <atom:updated>2019-11-12T02:02:48.203Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9o7wDZP5pBquR0RE5mRD8w.jpeg" /></figure><p>First of all, I declare myself a total n00b for any DevOps task. Also, my knowledge about Continuous Integration/Delivery is quite limited. Additionally, the project on which I applied CI/CD is very simple, just a script that runs in a Virtual Machine. It’s a powerful script, but nothing very sophisticated. Eventho, I believe this should help anyone with a more robust infrastructure. Any suggestion for making this post better is well received.</p><blockquote>Said that…</blockquote><h3>What are Github Actions?</h3><p>Github just presented some months ago their solution for <a href="https://github.com/features/actions">automating workflows</a>. With this, you can “Run a workflow on any GitHub event”. So, let&#39;s say when somebody makes their first contribution to the team you can greet them. Or maybe something more advanced like Building, testing and deploying an application to a cloud server.</p><figure><img alt="https://github.com/features/actions" src="https://cdn-images-1.medium.com/max/1024/1*91mz0a_BxribRWxVfEG0_Q.png" /><figcaption>Github Actions</figcaption></figure><p>On top of that, you get logs for all the runs you do. There is also a very vibrant open source community making new “Actions” that anybody can use.</p><p>And did I mention is totally free for public repos? (And also free for private ones during beta)</p><h3>What is CI/CD?</h3><p>On the current state of software development, teams tend to make a lot of changes to a codebase. These changes could affect the final product if they are not carefully made and checked. This is where CI/CD can become very helpful.</p><blockquote>CI stands for Continuous Integration and CD Continuous Delivery.</blockquote><p>In essence, I found at it was testing and deploying or not to production automatically. Here’s a diagram that helped me understand what was the difference.</p><figure><img alt="https://www.michielrook.nl/2016/07/continuous-delivery-continuous-deployment/" src="https://cdn-images-1.medium.com/max/640/0*3GljUb9KTwjJOEM2.png" /><figcaption><a href="https://www.michielrook.nl/2016/07/continuous-delivery-continuous-deployment/">https://www.michielrook.nl/2016/07/continuous-delivery-continuous-deployment/</a></figcaption></figure><h3>How to make a GitHub workflow?</h3><p>Ok, so now we now what Actions are and what is CI/CD. Now we want to actually make it happen for our project.</p><p>First of all (as for today), we need to <a href="https://github.com/features/actions/signup/">ask to be in the beta</a>.</p><p>Next, we need to make a Folder inside the .github folder named workflows.</p><p>After that, we can start to make .yml files inside that will include the instructions for our workflows</p><p>Here’s a very simple example of a workflow named ‘Greetings’:</p><pre>name: Greetings</pre><pre>on: [pull_request, issues]</pre><pre>jobs:<br>  greeting:<br>    runs-on: ubuntu-latest<br>    steps:<br>    - uses: actions/first-interaction@v1<br>      with:<br>        repo-token: ${{ secrets.GITHUB_TOKEN }}<br>        issue-message: &#39;Congrats on your first issue!&#39;<br>        pr-message: &#39;Congrats on your first Pull request!&#39;</pre><p>Lets got through the most important parts:</p><p>on:As mentioned above workflows run triggered on events in Github. This workflow is triggered whenever a pull_request or issues happen.</p><p>After the trigger, we declare a job and name it. We define that we want to run this job on a machine using the latest Ubuntu distribution. And start to set the steps for the job.</p><p>- uses: defines the Action we are going to use. Think of it as a function.</p><p>with: the input the Action uses to run. Here there are 3 parameters repo-token ,issue-message and pr-message .</p><h3>Github Secrets</h3><p>Don’t worry, I’m not disclosing any Github secret sauce. But, you might’ve noticed on the repo-token parameter a different value:${{secrets.GITHUB}}. Well, workflows are files that are public to anyone that has access to the repo. And tokens generally are private information, we don’t want anyone that is not authorized to look at them. To solve this we can declare them as ‘Secrets’.</p><blockquote>Settings -&gt; Secrets -&gt; ‘Add a new secret’</blockquote><p>To use them it’s as simple as using this structure:</p><pre>${{ secrets.SECRET_NAME }}</pre><h3>Making the CI workflow</h3><p>So, here’s the main building and testing workflow .yml. If you paid attention to the sections above you shouldn’t have a problem to understand most of it.</p><pre>name: tests</pre><pre>on:<br> pull_request:</pre><pre>jobs:<br> build:<br>  runs-on: ubuntu-latest<br>  steps:<br>  - uses: actions/checkout@master<br>  - uses: actions/setup-python@v1<br>    with:<br>     python-version: &#39;3.7.x&#39; # Version range or exact version of a Python version to use, using semvers version range syntax.<br>     architecture: &#39;x64&#39; # (x64 or x86)</pre><pre>- name: Install dependencies<br>    run: |<br>     python -m pip install --upgrade pip<br>     pip install -r requirements.txt<br>  - name: Run code<br>    env:<br>     email_pass: ${{ secrets.prophets_email_pass}}<br>     token: ${{ secrets.oanda_test_token}}<br>     url: ${{ secrets.oanda_test_url}<br>     tel_token: ${{ secrets.telegram_test_token}}<br>     tele_id: ${{ secrets.telegram_test_chat_id}}<br>     json_google_account: ${{ secrets.json_google_account}}<br>    run: |<br>     cd repo/<br>     python tests/test.py</pre><h3>Adding the CD part to make it work for my VM</h3><p>Ok, now we have somehow solved the first part, but we want to automate everything we can. So, after all checks are passed and the new code is merged in the repo we can pull the new changes into our Virtual Machine. So, we define the workflow using a very <a href="https://github.com/maddox/actions/tree/master/ssh">useful Action</a> to ssh our instance:</p><pre>name: deploy</pre><pre>on:<br> pull_request:<br> types: [closed]<br> branches:<br>  - master<br>jobs:<br> build:<br>  runs-on: ubuntu-latest<br>  steps:<br>  - name: Update Server Repo<br>  uses: maddox/actions/ssh@master<br>  env:<br>   PRIVATE_KEY: ${{ secrets.KEY }}<br>   HOST: ${{ secrets.HOST }}<br>   USER: ${{ secrets.USER }}<br>  with:<br>   args: |<br>    cd repo<br>    git pull</pre><p>You might find some different things here. The thing that is the most important is about the on: section in which we assign a pull_request trigger and precisely order it to run only if the PRs code is merged or closed, on the master branch. If all these conditions are met, then we use the action to connect to our VM and then execute a very simple pull on the base code we just updated. Done!</p><p>Here are some links that helped me:</p><p><a href="https://help.github.com/en/actions/automating-your-workflow-with-github-actions">Actions Docs</a></p><p><a href="https://github.com/sdras/awesome-actions/blob/master/README.md">Awesome Actions</a></p><p><a href="https://github.com/maddox/actions/tree/master/ssh">SSH Action</a></p><p>Follow me on <a href="https://twitter.com/mrdeds">Twitter</a> for more :D</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b4ad8ef6cb9b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AutoML | a simple library for making automatic machine learning models]]></title>
            <link>https://medium.com/resuelve/automl-a-simple-library-for-making-automatic-machine-learning-models-dfb429162191?source=rss-dc5eeba0106c------2</link>
            <guid isPermaLink="false">https://medium.com/p/dfb429162191</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Dud Díaz]]></dc:creator>
            <pubDate>Wed, 19 Dec 2018 18:25:38 GMT</pubDate>
            <atom:updated>2018-12-19T18:25:38.489Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fnmUrKvcymyRo8G_98ugCA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@turner_imagery">Will Turner</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>As a data scientist you always try to get the best model for your problem. Getting a great model, training it and tweaking it to perfection is just part of a bigger project. Also, going through more Machine Learning projects and reading some of other data scientists work, you may find out there is a development cycle. I thought about that, and for our team, this cycle looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EHFCpfqI6hZwAH37qSa9XQ.png" /></figure><p>As many in the tech scene I’ve had a self-taught experience into machine learning. When I was introducing myself into this very young practice, and all the <em>buzzwords</em> associated to it, I found a lot of information, books, videos, posts, MOOCs and tutorials about making algorithms to build models. But, when you get to build those models with your <em>real data</em>; data that is not complete, available, standardized, current, and a list of etceteras that we can summarize in … not clean; you get a lot of time cleaning your datasets. Whereas the creation of the model and even the feature engineering are just a fraction of that. Also, when your models need to be available for real customers/systems it turns out that the tutorial video you saw in Coursera/Youtube or the Github repo guiding you didn’t had that section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/0*UfWShSG_sBfkWO_n.png" /><figcaption>by Bernard on <a href="https://tex.stackexchange.com/questions/313529/changing-the-top-margin-of-the-proof-environment?rq=1">StackExchange</a></figcaption></figure><p>Don’t get me wrong, I believe that finding an appropriate architecture and correctly adjusting the <em>hyperparameters</em> can improve a model greatly. But, I’ve seen it’s the quality and quantity of the data that can create or destroy a model. Also a model that can be used only in a local machine is a good starting point. But, as commented earlier, cleaning, augmenting and deploying a model are not as popular topics as making machine learning algorithms.</p><p>Time and time again the team and I found that we were making the same cleaning process for our datasets in order to make different models. Same cleaning methods for different data.</p><p>We found out that it was not a problem just our team was facing but an issue for a lot of data scientists:</p><blockquote>“The reason data scientists are hired in the first place is to develop algorithms and build machine learning models — and these are typically the parts of the job that they enjoy most. Yet in most companies today, 80 percent of a data scientist’s valuable time is spent simply finding, cleaning and reorganizing huge amounts of data. Without the right cloud tools, this task is insurmountable.” <a href="https://www.infoworld.com/article/3228245/data-science/the-80-20-data-science-dilemma.html">Infoworld</a></blockquote><p>So, thinking like engineers, we thought about automating the repeating part of the job, and not just that but as much as we could from the cycle. So, that was when the idea of making AutoML started.</p><h3>Some considerations before starting…</h3><p>At the time this project started the models developed by the team were mostly from structured data. You can think about it like data that can be on a simple spreadsheet. Things like numbers, dates and strings. Contrary to unstructured data (images, audio, video, text). Also, the predictive models created were for classification purposes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EMhPGtzw8Q64Xtvo" /><figcaption>Photo by <a href="https://unsplash.com/@jeremybishop?utm_source=medium&amp;utm_medium=referral">Jeremy Bishop</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Get data first</h3><p>First part was getting our data to manipulate it. Fortunately, we had already centralized the most of our data into a warehouse, so we just had to query that out. So, the first thing the library needs to work is the query to get data.</p><h3>Wash and rinse</h3><p>After we have the data that we need it is a good idea to get rid of things that can confuse the algorithm on the training stage. To clean the data we used some really simple techniques for excluding features:</p><ul><li>constants</li><li>missing values within a x% threshold</li><li>features with more than y categories, for example.</li></ul><p>We make sure all the data we are getting is in the format and type we are expecting. For instance, we make sure dates are in one format and not as strings.</p><h3>Data++</h3><p>Once we have the data cleaned, let’s go a little further and augment the features we’ve got with some simple transformations:</p><ul><li>Categorical to discrete conversion</li><li>Sums, multiplications and some logical expressions</li><li>Date extractions: Day/day of the week/month</li></ul><h3>Having a good relation</h3><p>We have again a lot of data to train, but in order to determine if a feature is useful we use a <strong>correlation analysis </strong>to eliminate the features that are not as related to the objective variable (the piece of information we want to predict), the <strong>Variance Inflation Factor (VIF) </strong>to reduce multicollinearity, and identify possible <strong>data leakage</strong></p><p><strong><em>Correlation analysis — </em></strong><em>this is a statistical method of evaluation used to study the strength of the linear relationship between two continuous variables (e.g. height and weight)</em></p><p><strong><em>The Variance Inflation Factor (VIF) —</em></strong><em> is another statistical method of evaluation that quantifies the extent of correlation between one predictor and the other predictors in a model.</em></p><p><strong><em>Data leakage — </em></strong><em>when a feature is too related to the objective variable and can be a bad predictor for the model. (e.g. “number of tickets sold” when you try to predict number of passengers)</em></p><p>It is important to do this kind of checks in order to give a model (e.g. Neural network) a better chance to perform better and converge. Also a healthy pause for the data scientist to reflect on the features selected and an opportunity to reduce the dimensionality of inputs for the model. A simpler model is easier to understand and maintain.</p><h3>Model creation</h3><p>Down to this point (if everything went right) we should be having a clean dataset with augmented and related features to our objective variable. Great!</p><p>First we’ve found its very useful to make a very simple classification model. Just to have a ‘base model’ that draws a line in the sand as the barely minimum standard of performance.</p><p>To look for a model that can beat the base model we used three approaches:</p><ul><li>Evolving genetic algorithms</li><li>Random Search</li><li><a href="https://epistasislab.github.io/tpot/">Tpot</a></li></ul><p>We train the best out of the three and measure their performance with some indicators like:</p><ul><li>AUC</li><li>F1 Score</li><li>MCC Score</li></ul><p>The best performer is the winner!</p><h3>Let’s share it with the team</h3><p>Now we have created a model with the best, clean, augmented data, searched for the best architecture, tweaked its parameters, and benchmark it with at least three other models. Cool!</p><p>But a model that runs in your computer is not much of help for the team. Even less helpful for the organization. So, we should deploy it to the cloud.</p><p>Currently AutoML can deploy a trained model to Google’s ML Engine, creating a dedicated autoscaled service with an endpoint to serve as many request as necessary.</p><h3>Thoughts and further work</h3><p>While we are very proud of the result of AutoML, and have used it to develop two models in production. We found many opportunity areas and bugs that can (and should) be fixed in order to make a more robust, faster and better resulting product for everyone. Some ideas are:</p><ul><li>Option to do regressive models</li><li>Option to do a greater variety of models (CNNs, RNNs, Arima, Decision Trees, etc)</li><li>Faster data correlation checking</li><li>More data relationship checking</li><li>Deployment for other cloud vendors</li><li>Dockerized deployment (infrastructure agnostic)</li></ul><p>AutoML is a team work from Francisco Serrano and Dudley Díaz (me) on the AI team at <a href="https://medium.com/u/8c072a087e36">Resuelve tu Deuda</a> in Mexico City, México</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dfb429162191" width="1" height="1" alt=""><hr><p><a href="https://medium.com/resuelve/automl-a-simple-library-for-making-automatic-machine-learning-models-dfb429162191">AutoML | a simple library for making automatic machine learning models</a> was originally published in <a href="https://medium.com/resuelve">Resuelve</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AutoML | Una biblioteca simple para hacer modelos de Machine Learning.]]></title>
            <link>https://medium.com/resuelve/automl-una-biblioteca-simple-para-hacer-modelos-de-machine-learning-bd9529947051?source=rss-dc5eeba0106c------2</link>
            <guid isPermaLink="false">https://medium.com/p/bd9529947051</guid>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[computer-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Dud Díaz]]></dc:creator>
            <pubDate>Wed, 19 Dec 2018 17:31:00 GMT</pubDate>
            <atom:updated>2018-12-19T17:31:00.724Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fnmUrKvcymyRo8G_98ugCA.jpeg" /><figcaption>Foto de <a href="https://unsplash.com/@turner_imagery">Will Turner </a>en <a href="https://unsplash.com/@turner_imagery">Unsplash</a></figcaption></figure><p>Como científico de datos, siempre intentas obtener el mejor modelo para tu problema. Obtener un gran modelo, entrenarlo y ajustarlo a la perfección es solo una parte de un proyecto más grande. A la medida que vas exponiéndote a algunos proyectos de aprendizaje automático y leyendo algunos de los ciclos de desarrollo de los científicos de datos, descubres que hay un ciclo de desarrollo parecido. En general para nuestro equipo se ve así:</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*EHFCpfqI6hZwAH37qSa9XQ.png" /></figure><p>Como muchos en la escena tecnológica, he tenido una experiencia autodidacta en aprendizaje automático. Cuando me estaba introduciendo en esta muy joven práctica, y en todas las palabras de moda asociadas, encontré mucha información, libros, videos, publicaciones, MOOC y tutoriales sobre los algoritmos de creación de modelos. Pero, cuando llegas a construir esos modelos con tus datos reales; datos que no están completos, disponibles, estandarizados, actuales, y una lista de etcéteras que podemos resumir en … datos que no están limpios; te das cuenta de que la mayoría de tu tiempo la pasas limpiando tus conjuntos de datos. Mientras que la creación de tu modelo es algo que toma una fracción de ese tiempo. Adicionalmente, cuando terminas de entrenar esos modelos y deben estar disponibles para clientes/sistemas en producción, resulta que el video tutorial que viste en Coursera / Youtube o el repositorio de Github que te guiaba ya no contaba con esa sección.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*UfWShSG_sBfkWO_n.png" /><figcaption>De Bernard en <a href="https://tex.stackexchange.com/questions/313529/changing-the-top-margin-of-the-proof-environment?rq=1">StackExchange</a></figcaption></figure><p>No me malinterpreten, creo que encontrar una arquitectura adecuada y ajustar correctamente los hiperparámetros puede mejorar un modelo. Sin embargo, considero que es la calidad y la cantidad de datos lo que más impacta para hacer buenos modelos.</p><p>Además, aunque un modelo que solo se puede utilizar en una máquina local es un buen punto de partida, hay muchos pasos a seguir. Pero, como se comentó anteriormente, la limpieza, el aumento de datos y la implementación de un modelo no son temas tan populares como los algoritmos de aprendizaje automático.</p><p>Una y otra vez, el equipo y yo descubrimos que estábamos haciendo el mismo proceso de limpieza para nuestros conjuntos de datos, aún haciendo modelos completamente diferentes. Mismos métodos de limpieza para diferentes datos.</p><p>Descubrimos también que no era un problema exclusivo de nuestro equipo, sino un problema para muchos científicos de datos:</p><blockquote>“The reason data scientists are hired in the first place is to develop algorithms and build machine learning models — and these are typically the parts of the job that they enjoy most. Yet in most companies today, 80 percent of a data scientist’s valuable time is spent simply finding, cleaning and reorganizing huge amounts of data. Without the right cloud tools, this task is insurmountable.” <a href="https://www.infoworld.com/article/3228245/data-science/the-80-20-data-science-dilemma.html">Infoworld</a></blockquote><p>Así que, pensando como ingenieros, decidimos automatizar la parte repetitiva del trabajo, y no solo eso, sino todo lo que pudieramos del ciclo. Ahí fue cuando comenzó la idea de hacer AutoML.</p><h3>Algunas consideraciones antes de comenzar…</h3><p>Al momento de iniciar este proyecto, los modelos desarrollados por el equipo eran en su mayoría de datos estructurados. Puede pensarse como datos que pueden estar en una hoja de cálculo simple. Cosas como números, fechas y cadenas de texto. Contrario a los datos no estructurados (imágenes, audio, video, texto). Además, los modelos creados eran para resolver problemas de clasificación.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*EMhPGtzw8Q64Xtvo" /><figcaption>Photo by <a href="https://unsplash.com/@jeremybishop?utm_source=medium&amp;utm_medium=referral">Jeremy Bishop</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Obtén datos primero</h3><p>La primera parte fue conseguir los datos con los que vamos a trabajar. Afortunadamente, ya habíamos centralizado la mayor parte de nuestros datos en un solo <em>data warehouse</em>, por lo que solo tuvimos que consultarlo. Entonces, lo primero que necesita la biblioteca para trabajar es la capacidad de hacer consultarlo.</p><h3>Lavar y enjuagar</h3><p>Una vez que tenemos los datos que necesitamos, es buena idea deshacerse de las cosas que pueden confundir el algoritmo en la etapa de entrenamiento. Para limpiar los datos utilizamos algunas técnicas realmente simples para excluir variables:</p><ul><li>Eliminación de constantes</li><li>Valores faltantes dentro de un umbral de x%</li><li>variables con más de y categorías, por ejemplo.</li></ul><p>Nos aseguramos de que todos los datos que obtengamos estén en el formato y tipo que esperamos. Por ejemplo, nos aseguramos de que las fechas estén en un formato correcto y no como cadenas de texto.</p><h3>Datos ++</h3><p>Una vez que hayamos limpiado los datos, avancemos un poco más y aumentemos las variables que tenemos con algunas transformaciones simples:</p><ul><li>Discretización de variables categóricas</li><li>Sumas, multiplicaciones y uso de algunas expresiones lógicas</li><li>Extracciones de fecha: día /día de la semana /mes</li></ul><h3>Teniendo una buena relación</h3><p>Ya tenemos una gran cantidad de datos para entrenar, pero para determinar si una variable es útil usamos un <strong>análisis de correlación</strong> para eliminar las variables que no están tan relacionadas con la variable objetivo (<em>la información que queremos predecir</em>), <strong>el factor de inflación de varianza (VIF) </strong>para reducir la multicolinealidad e identificamos posibles <strong>fugas de datos (data leakage)</strong>.</p><p><strong><em>Análisis de correlación —</em></strong><em> este es un método estadístico de evaluación utilizado para estudiar la fuerza de la relación lineal entre dos variables continuas (por ejemplo, altura y peso)</em></p><p><strong><em>El factor de inflación de varianza (VIF) —</em></strong><em> es otro método estadístico de evaluación que cuantifica el grado de correlación entre un predictor y los otros predictores en un modelo.</em></p><p><strong><em>Fuga de datos (data leakage) — </em></strong><em>cuando una característica está demasiado relacionada con la variable objetivo esta puede ser un mal predictor. Un tipo de data leakage común es el caso en que son datos que no están al momento que usarías tu modelo de predicción. (por ejemplo, tomó_antibiótico= True, cuando se quiere predecir enfermedad. La persona puede estar tomando medicina porque </em><strong><em>ya sabe </em></strong><em>que está enferma.)</em></p><p>Es importante realizar este tipo de controles para ofrecer a un modelo (por ejemplo, una red neuronal) una mejor oportunidad de rendir mejor y converger. También crea una pausa saludable para que el científico de datos reflexione sobre las variables seleccionadas y una oportunidad para reducir la dimensionalidad de las entradas para el modelo. Un modelo más simple es más fácil de entender y mantener.</p><h3>Creación de modelos</h3><p>Hasta este punto (si todo ha salido bien) deberíamos tener un conjunto de datos limpio con variables aumentadas y relacionadas con nuestra variable objetivo. ¡Genial!<br>Primero hemos encontrado que es muy útil para hacer un modelo de clasificación muy simple. Solo para tener un ‘modelo base’ que dibuje una línea en la arena como el mínimo estándar de desempeño.<br>Para buscar un modelo que pueda superar al modelo base, utilizamos tres estrategias:</p><ul><li>Algoritmos genéticos evolutivos</li><li>Búsqueda aleatoria</li><li><a href="https://epistasislab.github.io/tpot/">Tpot</a></li></ul><p>Sacamos al mejor modelo de cada estrategia y los evaluamos con algunas métricas:</p><ul><li>AUC (área bajo la curva)</li><li>F1 score</li><li>MCC score</li></ul><p>¡El que mejor desempeño tenga bajo estos parámetros es el ganador!</p><h3>Compartámos con el equipo.</h3><p>Para este momento hemos creado un modelo con los mejores datos, limpios y aumentados; con la mejor arquitectura y parámetros; además lo hemos comparado con con al menos otros tres modelos. ¡Muy bien!<br>Pero un modelo que se ejecuta en tu computadora no es de mucha ayuda para el equipo. Aún menos útil para la organización. Entonces, deberíamos desplegarlo donde todos puedan utilizarlo.</p><p>Actualmente, AutoML puede desplegar un modelo entrenado localmente en el motor de ML de Google, creando un servicio dedicado, auto-escalado y con un endpoint para atender todas las solicitudes que sean necesarias.</p><h3>Pensamientos y trabajos futuros.</h3><p>Si bien estamos muy orgullosos del resultado de AutoML, y lo hemos utilizado para desarrollar dos modelos en producción. Encontramos muchas áreas de oportunidad y ‘bugs’ que pueden (y deben) solucionarse para hacer un producto más robusto, más rápido y mejor para todos. Algunas ideas son:</p><ul><li>Opción de hacer modelos regresivos.</li><li>Opción para hacer una mayor variedad de modelos (CNN, RNN, Arima, árboles de decisión, etc.)</li><li>Comprobación de correlación de datos más eficiente</li><li>Mayor variedad para comprobar correlación de datos</li><li>Implementación para otros proveedores de la nube</li><li>Despliegue agnóstico de infraestructura</li></ul><p>AutoML es un trabajo en equipo de Francisco Serrano y Dudley Díaz (yo) en el equipo de AI en <a href="https://medium.com/u/8c072a087e36">Resuelve tu Deuda</a> en la Ciudad de México, México.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bd9529947051" width="1" height="1" alt=""><hr><p><a href="https://medium.com/resuelve/automl-una-biblioteca-simple-para-hacer-modelos-de-machine-learning-bd9529947051">AutoML | Una biblioteca simple para hacer modelos de Machine Learning.</a> was originally published in <a href="https://medium.com/resuelve">Resuelve</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>