<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Hector Andres Mejia Vallejo on Medium]]></title>
        <description><![CDATA[Stories by Hector Andres Mejia Vallejo on Medium]]></description>
        <link>https://medium.com/@hectormrejia?source=rss-4f706205679a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*MYSlL4J6lnoVEh5J2gqMIw.jpeg</url>
            <title>Stories by Hector Andres Mejia Vallejo on Medium</title>
            <link>https://medium.com/@hectormrejia?source=rss-4f706205679a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 18:28:29 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@hectormrejia/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Low-Code Data Connectors and Destinations]]></title>
            <link>https://medium.com/data-science/low-code-data-connectors-and-destinations-b044128c72ca?source=rss-4f706205679a------2</link>
            <guid isPermaLink="false">https://medium.com/p/b044128c72ca</guid>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[data-connectors]]></category>
            <category><![CDATA[data-ingestion]]></category>
            <category><![CDATA[airbyte]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Hector Andres Mejia Vallejo]]></dc:creator>
            <pubDate>Thu, 10 Oct 2024 06:50:37 GMT</pubDate>
            <atom:updated>2024-10-10T06:50:37.998Z</atom:updated>
            <content:encoded><![CDATA[<h4>Get started with Airbyte and Cloud Storage</h4><h3>Coding the connectors yourself? Think very carefully</h3><p>Creating and maintaining a data platform is a hard challenge. Not only do you have to make it scalable and useful, but every architectural decision builds up over time. Data connectors are an essential part of such a platform. Of course, how else are we going to get the data? And building them yourself from scratch gives you <strong>full control of how you want them to behave</strong>. But beware, with ever-increasing data sources in your platform, that can only mean the following:</p><ul><li>Creating large volumes of code for every new connector.</li><li>Maintaining complex code for every single data connector.</li><li>Functions and definitions between classes may diverge over time, resulting in even more complex maintenance.</li></ul><p>Of course, all three <strong>can be mitigated</strong> with well-defined practices in object-oriented programming. But even still, <strong>it will take many hours of coding </strong>that could be used in later stages to serve your data consumers faster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hFpQzUsqbzcj-QkY" /><figcaption>Data flowing like cars in a highway. Photo by <a href="https://unsplash.com/@stywo?utm_source=medium&amp;utm_medium=referral">Stephan Seeber</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>What if you try low-code connectors?</h3><p>Other options still give you the flexibility to define what data you want to ingest and how with no to very little code involved. With this option, you get:</p><ul><li>Connectors with standardized behavior given the extraction methodology: No divergent classes for two connectors that use REST APIs at their core, for instance.</li><li>Simple, but powerful user interfaces to build connections between sources and destinations.</li><li>Connectors that are maintained by the teams building the tools and the community.</li></ul><p>These benefits allow you to <strong>build data connections in minutes, instead of hours.</strong></p><p>Nevertheless, I am not trying to sell you these tools; if and when you need highly customizable logic for data ingestion, you are going to need to implement it. So, do what is best for your application.</p><h3>The exercise: Airbyte with ADLS Gen2</h3><p>Let’s jump right into it. I am using Azure for this tutorial. You can sign up and get $200 worth of services for free to try the platform.</p><p>We are going to deploy Airbyte Open Source using an Azure Kubernetes cluster and use Azure Storage (ADLS) Gen 2 for cloud storage.</p><h4>Creating the infrastructure</h4><p>First, create the following resources:</p><ul><li><strong>Resource group</strong> with the name of your choosing.</li><li><strong>Azure Kubernetes Services</strong>. To avoid significant costs, set a single node pool with one node. However, that node needs enough resources. Otherwise, the Airbyte syncs won’t start. An appropriate node size is <strong><em>Standard_D4s_v3.</em></strong></li><li><strong>Azure Storage Account</strong>. While creating git, turn on the hierarchical namespace feature so the storage account becomes ADLS Gen2. Now create a storage container with any name you like.</li></ul><blockquote><strong><em>Production Tip: Why the hierarchical namespace? </em></strong><em>Object stores by default have a flat storage environment. This has the benefit of infinite scalability, with an important downside. For analytics workloads, this results in additional overhead when reading, modifying, or moving files as the whole container has to be scanned. Enabling this features brings hierarchical directories from filesystems to scalable object storage.</em></blockquote><h4>Deploying Airbyte to Kubernetes</h4><p>You need to install a few things on your shell first:</p><ul><li><a href="https://learn.microsoft.com/en-us/cli/azure/install-azure-cli">Azure-CLI</a></li><li><a href="https://helm.sh/docs/intro/install/">Helm</a></li><li><a href="https://kubernetes.io/docs/tasks/tools/">Kubernetes command line tools (kubectl)</a></li></ul><p>Now, follow these steps:</p><p>Log into your Azure account using the shell.</p><pre>az login</pre><p>Set the cluster credentials.</p><pre>az aks get-credentials --resource-group &lt;your-resource-group&gt;\<br>                       --name &lt;cluster-name&gt;\<br>                       --overwrite-existing</pre><p>Add remote helm repository and search for the Airbyte chart.</p><pre>helm repo add airbyte https://airbytehq.github.io/helm-charts<br>helm repo update<br>helm search repo airbyte</pre><p>Create a unique Kubernetes namespace for the Airbyte deployments. I called it <strong><em>dev-airbyte</em></strong><em>.</em></p><pre>kubectl create namespace dev-airbyte</pre><p>Deploy Airbyte.</p><pre>helm install airbyte airbyte/airbyte - namespace dev-airbyte</pre><p>Wait a few minutes until the deployment is completed. Run the following command to check if the pods are running:</p><pre>kubectl get pods --namespace dev-airbyte</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nrJTbcyrSKXwIR2mqYbSTQ.png" /><figcaption>Airbyte pods ready! Screen capture taken by me.</figcaption></figure><h4><strong>Accessing the Airbyte web app locally</strong></h4><p>After Airbyte is deployed you can get the container and port, and then run a port forwarding command to map a port in your local machine to the port in the Kubernetes web app pod. This will allow us to access the application using localhost.</p><pre>export POD_NAME=$(kubectl get pods - namespace dev-airbyte -l &quot;app.kubernetes.io/name=webapp&quot; -o jsonpath=&quot;{.items[0].metadata.name}&quot;)<br>export CONTAINER_PORT=$(kubectl get pod - namespace dev-airbyte $POD_NAME -o jsonpath=&quot;{.spec.containers[0].ports[0].containerPort}&quot;)<br>kubectl - namespace dev-airbyte port-forward $POD_NAME 8080:$CONTAINER_PORT<br>echo &quot;Visit http://127.0.0.1:8080 to use your application&quot;</pre><p>If you go to <strong><em>127.0.0.1:8080</em></strong> on your machine, you should see the application. Now, we can start adding data connectors!</p><blockquote><strong><em>Production Tip:</em></strong><em> Port forwarding works only for your local machine and must be done every time the shell is started. However, for data teams in real scenarios, Kubernetes allows you to expose your application throughout a </em><a href="https://learn.microsoft.com/en-us/azure/aks/internal-lb?tabs=set-service-annotations"><em>virtual private network</em></a><em>. For that, you will need to switch to </em><a href="https://docs.airbyte.com/enterprise-setup/"><em>Airbyte Self-managed enterprise</em></a><em> which provides </em><a href="https://docs.airbyte.com/access-management/sso-providers/azure-entra-id"><em>Single Sign-On</em></a><em> with Cloud identity providers like Azure Active Directory to secure your workspace.</em></blockquote><h4>Setting up the data source</h4><p>The provider for the data in this exercise is called <a href="https://www.tiingo.com/documentation/general/overview">Tiingo</a>, which serves very valuable information from the companies in the stock market. They offer <a href="https://www.tiingo.com/documentation/general/overview">a free license</a> that will give you access to the end-of-day prices endpoint for any asset and fundamental analysis for companies in the DOW 30. Be mindful that <strong>with the free license, their data are for your eyes only</strong>. If you want to share your creations based on Tiingo, you must pay for a commercial license. For now, I will use the free version and guide you through the tutorial without showing their actual stock data to remain compliant with their rules.</p><p><a href="https://www.tiingo.com/documentation/general/overview">Create the account</a>. Then, copy the API key provided to you. We are now ready to set up the source in Airbyte.</p><h4><strong>Creating a data source in Airbyte</strong></h4><p>In the Airbyte app, go to <strong><em>Builder &gt; Start from Scratch</em></strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6C-YviFE1R-NStgoWaFFbw.png" /><figcaption>Airbyte connector builder screen. Image captured by me.</figcaption></figure><p>In the API Base URL write <a href="https://api.tiingo.com/tiingo/"><strong><em>https://api.tiingo.com/tiingo/</em></strong></a><strong><em> </em></strong>and for the configuration click on the YAML button. Enter the following:</p><pre>type: ApiKeyAuthenticator<br>inject_into:<br>  type: RequestOption<br>  inject_into: header<br>  field_name: Authorization<br>  api_token: &#39;Token {{ config[&quot;api_key&quot;] }}&#39;</pre><p>This will allow the API token to be inserted in the header of every request. Now, let’s add your first stream by clicking on the plus icon (+) on the left. See the image below for guidance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VkBbiNOI2vu3KEWbovLoPg.png" /><figcaption>Building the data source. Global Configuration. Image captured by me.</figcaption></figure><p><strong>URL and stream partitioning</strong></p><p>At the top write <strong><em>End of Day Prices</em></strong>. This will be our stream name and the URL path will be:</p><pre>daily/{{ stream_partition[&#39;ticker&#39;] }}/prices</pre><p>What is this placeholder between <strong><em>{{}}</em></strong>? These are variables filled by Airbyte at runtime. In this case, Airbyte supports what they call stream partitioning, which allows the connector to <strong>make as many requests as the number of values you have on your partition array</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GINQDmLRsNCsyGMUTUJ2zg.png" /><figcaption>Defining URL path and primary key. Image captured by me.</figcaption></figure><p>Scroll down to parameterized requests, and check the box. In the parameter values dropdown, click User Input, and on the value textbox enter:</p><pre>{{ config[&#39;tickers_arr&#39;] }}</pre><p>Notice that the <strong><em>config</em></strong> variable used here is also referenced in the API Key in the global configuration. This variable holds the user inputs. Moreover, the user input <strong><em>tickers_arr </em></strong>will hold an array of stock IDs.</p><p>Next, on the <strong><em>Current Parameter Value Identifier</em></strong> textbox enter <strong><em>ticker</em></strong>. This is the key that is added to the <strong>stream_partition</strong> variable and references a single stock ID from the array <strong><em>tickers_arr </em></strong>for a single HTTP request. Below you can find screenshots of this process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TTZLLNJo89aYbTnmCZtn1A.png" /><figcaption>Defining the parameterized requests. Image captured by me.</figcaption></figure><p>We are going to test it with 4 stock tickers:</p><ul><li><strong>BA</strong> for Boeing Corp</li><li><strong>CAT</strong> for Caterpillar</li><li><strong>CVX</strong> for Chevron Corp</li><li><strong>KO</strong> for Coca-Cola</li></ul><p>With the stream partitioning set up, the connector will make 4 requests to the Tiingo server as follows:</p><ul><li><a href="https://api.tiingo.com/tiingo/"><strong><em>https://api.tiingo.com/tiingo/</em></strong></a><strong><em>daily/BA/prices</em></strong></li><li><a href="https://api.tiingo.com/tiingo/"><strong><em>https://api.tiingo.com/tiingo/</em></strong></a><strong><em>daily/CAT/prices</em></strong></li><li><a href="https://api.tiingo.com/tiingo/"><strong><em>https://api.tiingo.com/tiingo/</em></strong></a><strong><em>daily/CVX/prices</em></strong></li><li><a href="https://api.tiingo.com/tiingo/daily/KO/prices"><strong><em>https://api.tiingo.com/tiingo/daily/KO/prices</em></strong></a></li></ul><p>Pretty cool, huh?</p><blockquote><strong><em>Production Tip: </em></strong><em>Airbyte supports a parent stream, which allows us to get the list for the partitioning using a request to some other endpoint, instead of issuing the array elements ourselves. We are not doing that in this exercise, but you can check it out </em><a href="https://docs.airbyte.com/connector-development/connector-builder-ui/partitioning#parent-stream"><em>here</em></a><em>.</em></blockquote><p><strong>Incremental Sync</strong></p><p>Airbyte supports syncing data in Incremental Append mode i.e.: syncing only <em>new</em> or <em>modified</em> data. This prevents re-fetching data that you have already replicated from a source. If the sync is running for the first time, it is equivalent to a <a href="https://docs.airbyte.com/using-airbyte/core-concepts/sync-modes/full-refresh-append">Full Refresh</a> since all data will be considered as <em>new</em>.</p><p>To implement this in our connector, scroll to <strong>Incremental Sync</strong> and check the box. In the <strong>cursor</strong> field textbox write date since, according to the documentation, that is the name of the date field indicating when the asset was updated. For the cursor DateTime Formats, enter</p><pre>%Y-%m-%dT%H:%M:%S.%fZ</pre><p>This is the output format suggested by the API docs.</p><p>In the <strong>Start DateTime dropdown</strong> click Custom and on the textbox enter the following:</p><pre>{{ day_delta(-1, format=&#39;%Y-%m-%dT%H:%M:%SZ&#39;) }}</pre><p>It will tell Airbyte to insert the date corresponding to yesterday. For the <strong>End Datetime</strong> leave the dropdown in <strong>Now</strong> to get data from the start date, up until today. The screenshot below depicts these steps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cCrIVuOBhX4zsqNiWK2nUA.png" /><figcaption>Adding Incremental Start Datetime and End Datetime. Image captured by me.</figcaption></figure><p>Finally, check the boxes to inject the start and end time into the outgoing HTTP request. The parameter names should be startDate and endDate, respectively. These parameter names come from Tiingo documentation as well. An example request will now look like:</p><ul><li><a href="https://api.tiingo.com/tiingo/daily/BA/prices?startDate=2024-09-20T13%3A54%3A20.000000Z&amp;endDate=2024-09-23T13%3A54%3A20.000000Z"><strong><em>https://api.tiingo.com/tiingo/daily/BA/prices?startDate=2024-09-20T13%3A54%3A20.000000Z&amp;endDate=2024-09-23T13%3A54%3A20.000000Z</em></strong></a></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b2aclhyHFX2m_ZHhCrf5fw.png" /><figcaption>Start and End Time parameters for our incremental loads. Image captured by me.</figcaption></figure><p><strong>Control Fields</strong></p><p>We are going to insert some information to enrich the data. For this, scroll to the transformations section and check the box. Inside the transformation dropdown, click on Add Field. The path is just the column name to be added, write process_date with the value {{ today_utc() }}. This will just indicate the timestamp for which the records were ingested into our system.</p><p>Now, according to the documentation, the ticker of the asset is not returned in the response, but we can easily add it using an additional transformation. So, for path, write ticker and the value should be {{ stream_partition[‘ticker’] }}. This will add the ticker value of the current stream partition as a column.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-zZ8Gb_A6Ylv5NEbqi_9rg.png" /><figcaption>Adding our control fields to the API response. Image captured by me.</figcaption></figure><p><strong>Testing</strong></p><p>On the Testing values button, enter the list of tickers. A comma separates each ticker: BA, CAT, CVX, KO.</p><p>You should see something similar to the following image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/972/1*mOg1iF3uSD1cyU1FWCmBEQ.png" /></figure><p>Notice the two example partitions. These are two separate, parameterized requests that Airbyte performed. You can also get information about the actual content in your request, the generated schema of the data, and state information.</p><p>Go to the top right corner and click <strong>publish</strong> to save this connector. Give it any name you want, I just called it Tiingo Connector.</p><h4><strong>Connecting Airbyte to the object store</strong></h4><p>Let’s return to our storage service, go to <strong><em>Security + Networking &gt; Access keys</em></strong>. Copy the account name and one of the access keys. Note: we need the access key, not the connection string.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_u_Aqk1BWQFdknmr7ELVSw.png" /><figcaption>Getting the access keys for the azure storage account. Image captured by me.</figcaption></figure><p>Next, go to your Airbyte app, select <strong><em>Destinations&gt; Marketplace,</em></strong> and click <strong><em>Azure Blob Storage</em></strong>. Enter the account name, account key, and leave the other configurations as in the image. Additionally, in the Optional fields, enter the name of the container you created. Next, click on <strong>Set up destination.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aQlVohmw4ENIcb8LLO9fFA.png" /><figcaption>Setting up the destination in Airbyte. Image captured by me.</figcaption></figure><blockquote><strong><em>Production Tip: </em></strong><em>Data assets from your organization need to be secured so that the individuals or teams have access to only the files they need. You can set up role-based access control at the storage account level with the Access Control (IAM) button, and also set Access Control Lists (ACLs) when right clicking folders, containers, or files.</em></blockquote><h4><strong>Creating a connection from source to destination</strong></h4><p>There are four steps to build a connection in Airbyte and it will use the Tiingo Connector and the Azure Storage.</p><p><strong>Defining the source</strong></p><p>In the Airbyte app, go to connections and create one. The first step is to set up the source. Click <strong><em>Set up a new source. </em></strong>Then, on the <strong><em>Custom</em></strong> tab, select the Tiingo connector we just created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q-MVdSG83MhSy6p8l2iZzQ.png" /><figcaption>Creating a source for the connection. Image captured by me.</figcaption></figure><p>It will prompt you to enter the API Keys and stock tickers. Just copy the ones you used while testing the source. Now click on <strong><em>Set up source</em></strong>. It will test the connector with your configuration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QpbqYdqW7ul0mIaZESh1ew.png" /><figcaption>Adding user inputs for the source. Image captured by me.</figcaption></figure><p><strong>Defining the destination</strong></p><p>Once it has passed, we will set up the destination, which is the one created in the above section. At this time, Airbyte will also test the destination.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SM9HnctCTIWac4o2HXukrQ.png" /><figcaption>Adding the destination for the connection. Image captured by me.</figcaption></figure><p><strong>Defining streams</strong></p><p>The third step is to select the streams and the sync mode. As we only defined one stream called <strong><em>End of Day Prices</em></strong>, this is the only one available. As for the sync modes, these are the options available for this exercise:</p><ul><li><strong>Full Refresh | Overwrite:</strong> This mode will retrieve all the data and replace any existing data in the destination.</li><li><strong>Full Refresh | Append</strong>: This mode will also retrieve all of the data, but it will append the new data to the destination. You must deduplicate or transform your data properly to suit your needs afterward.</li><li><strong>Incremental | Append:</strong> This mode requests data given the incremental conditions we defined while building the connector. Then, it will append the data to the destination.</li></ul><p>You can read more about synch modes <a href="https://docs.airbyte.com/using-airbyte/core-concepts/sync-modes/">here</a>. For now, choose <strong><em>Incremental | Append.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5fhye7Xp7nQqY5J-P5c4gQ.png" /><figcaption>Selecting the streams to ingest. Image captured by me.</figcaption></figure><p><strong>Final connection configurations</strong></p><p>Here you can define the schedule you want, plus other additional settings. Click <strong><em>finish and sync</em></strong> to prompt your first data extraction and ingestion.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hTO_-8aXTfX3hXYZsnb1Wg.png" /><figcaption>Running the first synching process. Image captured by me.</figcaption></figure><p>And that’s it! The data is now ingested. Head back to the storage container and you will see a new folder with one CSV file. With the append mode chosen, whenever a sync is triggered, a new file appears in the folder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6V8fnMnTDE__KaLABYoSgw.png" /><figcaption>A new folder with the name of the stream is created. Image captured by me.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wTKvgqwZlzUESXayEr00tA.png" /><figcaption>Data files as a result of multiple syncs in Airbyte. Image captured by me.</figcaption></figure><h3>Conclusion</h3><p>You can clearly see the power of these kinds of tools. In this case, Airbyte allows you to get started with ingesting critical data in a matter of minutes with production-grade connectors, without the need to maintain large amounts of code. In addition, it allows incremental and full refresh modes with append or overwrite capabilities. In this exercise, only the Rest API sources were demonstrated, but there are many other source types, such as traditional databases, data warehouses, object stores, and many other platforms. Finally, it also offers a variety of destinations where your data can land and be analyzed further, greatly speeding up the development process and allowing you to take your products to market faster!</p><p>Thank you for reading this article! If you enjoyed it, please give it a clap and share. I do my best to write about the things I learn in the data world as an appreciation for this community that has taught me so much.</p><p>Till the next time!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b044128c72ca" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/low-code-data-connectors-and-destinations-b044128c72ca">Low-Code Data Connectors and Destinations</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A bit about linear models and regularizers]]></title>
            <link>https://hectormrejia.medium.com/a-bit-about-linear-models-and-regularizers-ec152eda8f08?source=rss-4f706205679a------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec152eda8f08</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Hector Andres Mejia Vallejo]]></dc:creator>
            <pubDate>Wed, 23 Aug 2023 20:58:31 GMT</pubDate>
            <atom:updated>2023-08-23T20:58:31.416Z</atom:updated>
            <content:encoded><![CDATA[<h4>Home-brewed machine learning model series</h4><p>In the past, I would look at all of these linear models in scikit-learn without a clue about how their work. I believe that we should have a sense of the tools we are using. Otherwise, machine learning modeling may end up being a brute-force process to minimize some metrics, severely hurting interpretability and fairness.</p><p>In this post, I will talk about:</p><ul><li><a href="#90ef">Linear models and how to estimate them.</a></li><li><a href="#ad21">The (in)famous bias-variance trade-off</a></li><li><a href="#87f6">Introduction to regularized learning</a></li><li><a href="#d74a">Ridge</a>, <a href="#8a1c">Lasso</a>, <a href="#4d43">Elastic-Net</a>. Weight estimation, characteristics, and limitations.</li></ul><p><a href="https://github.com/hector6298/medium_posts/blob/main/lm/lm.ipynb"><strong>Here</strong></a><strong> is the link to the companion notebook</strong>. Connect with me on <strong>LinkedIn</strong> <a href="https://www.linkedin.com/in/hectorandresmv/">here</a>. Let’s begin!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ryqzR2SszbzkQImu" /><figcaption>Linear road. Linear models. Photo by <a href="https://unsplash.com/@redaquamedia?utm_source=medium&amp;utm_medium=referral">Denny Müller</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Linear models very briefly</h3><p>Given a set of input features and labels, denoted as <strong><em>D:</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/178/1*Sd6IE5JfctA2PZiQVIVFtQ.png" /></figure><pre>import numpy as np<br>import matplotlib.pyplot as plt<br><br># In this example, we are going to include a dataset with only 2 dimensions<br># These are the parameters for the noise<br>mean = 3<br>stddev = 25<br>size = 50<br><br># These are the parameters of the true model<br>beta0 = 3<br>beta1 = 2<br><br># Generating noise for the samples<br>noise = np.random.normal(mean, stddev, size)<br><br># Generating the data using a line plus noise<br>x = np.arange(size)<br>y = beta0 + beta1*x + noise<br><br># Plotting the data<br>plt.scatter(x,y)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/377/1*BvJYosgkh94qr3cEMtV7xA.png" /><figcaption>Fig 1. A sample dataset with a single feature and the targets. From my own authorship.</figcaption></figure><p>Our goal is to learn a mapping from the inputs to the labels. One of the simplest methods is to use a linear model of the form [1]:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/406/1*tgK5xfeGmp5PFWMwc2RlhA.png" /></figure><p>Here <strong><em>yi</em></strong> is assumed to be the weighted sum of the features of <strong><em>xi</em></strong>, plus a residual term<strong><em> </em>ε</strong>. Moreover, the <strong>βs </strong>are the weights (or coefficients) of the model, and <strong><em>x_{i,j} </em></strong>is the feature <strong><em>j</em></strong> of example <strong><em>i</em></strong>.</p><pre># Plotting the &quot;true&quot; linear model<br>x0 = 0<br>x1 = 50<br>y0 = beta0 + beta1*x0<br>y1 = beta0 + beta1*x1<br><br># Plotting the data<br>plt.scatter(x,y)<br>plt.plot([x0,x1], [y0,y1], color=&quot;red&quot;, label=&quot;True Linear model&quot;)<br>plt.legend()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/377/1*w6-ePATW53sBgAt-NikTaA.png" /><figcaption>Fig 2. Plot of the theoretical linear model. From my own authorship.</figcaption></figure><p>We can write this in compact form using matrices:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/128/1*a85xogyzsxvnIT7DFwcg1w.png" /></figure><p>Note that here we are talking about a theoretical “true” model. Out of all the possible linear models, this true model is the best of all. However, in the real world, it is not possible to obtain it as that would require data from the entire population subjected to the analysis. Often, we can only afford to get a sample.</p><p>Instead, what we are doing is finding the estimators:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/102/1*ca_AwlKAs0HGBvMPdfqkzw.png" /></figure><p>That minimize the Mean Squared Error, defined as [1]:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/1*omqY8gR7XWImEo30OEbIIQ.png" /></figure><pre># Plotting the mean squared error<br>for i in range(len(x)):<br>    plt.plot(<br>        [x[i], x[i]], <br>        [true_model(x[i]), y[i]], <br>        color=&#39;green&#39;, <br>        linestyle=&#39;dotted&#39;, <br>        label=&quot;Mean Squared Error&quot; if i == len(x)-1 else None)<br>plt.legend()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/377/1*ZM4KMPY8ndRfHOZVNL-QWw.png" /><figcaption>Fig 3. Plot of the errors against the theoretical model. The Mean Squared Error is the average of all these lines. From my own authorship.</figcaption></figure><p>The MSE is convex and differentiable, which means that the global minimum is found at the zeros of the gradients, and no local minimums exist. Furthermore, following these properties, it is possible to get a closed-form solution to obtain the model coefficients and it is given by:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*TemXu6ktYxSq2fjJ_LLz2A.png" /></figure><pre># Function to fit linear model to data<br>def fit_estimator(x, include_bias=True):<br>    if include_bias:<br>        x = np.column_stack((np.ones((len(x), 1)), x))<br>    return np.linalg.inv(x.T@x)@x.T@y</pre><p>Now our estimated model can be used for inference as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/202/1*N7qMMGfslv38EPeSl0ag5Q.png" /></figure><pre># Fitting the linear model<br>betas = fit_estimator(x)<br>beta0hat = betas[0]<br>beta1hat = betas[1]<br><br># Function used for inference <br>est_model = lambda x: beta0hat + beta1hat*x<br><br># Plotting the data<br>plt.scatter(x,y)<br><br># Plotting the true linear model<br>plt.plot([x0,x1], [est_model(x0),est_model(x1)], color=&quot;orange&quot;, label=&quot;Estimated Linear model&quot;)<br>plt.plot([x0,x1], [y0,y1], color=&quot;red&quot;, label=&quot;True Linear model&quot;, linestyle=&#39;dotted&#39;)<br><br>plt.legend()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/377/1*S81_H5-huSl-ASBeeCHJvQ.png" /><figcaption>Fig 4. Comparison of the true model and an estimated model. From my own authorship.</figcaption></figure><h3>Model Bias vs variance</h3><p>Let’s define two concepts before introducing regularization:</p><ul><li>Model <strong>variance: </strong>It is the spread, or variability, of the model prediction for a given data point.</li><li>Model <strong>bias</strong>: The difference between the expectation of the model prediction and the real target.</li></ul><p>As the number of trainable parameters <strong><em>d</em></strong> increases, it is said that the model becomes increasingly complex. As <strong><em>d </em></strong>approaches the number of training samples<strong><em> n, </em></strong>the <strong>variance</strong> of the model increases, but the <strong>bias</strong> decreases [2].</p><p>A model too complex, with many parameters and small data, will have a small bias but high variance. This means that you would have to be lucky to get a model that produces reliable predictions. Conversely, if the model is not complex enough to describe the data under analysis, due to few features or simple architecture, the variance will be small but the bias will be large. In this case, the model will never reach the target.</p><p>An illustration of this is in Fig 5. As the complexity decreases, so does the number of possible predictions for the same target, but they are also biased away.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m72ksXedBLfPdwWu3nl5PA.png" /><figcaption>Fig 5. Depiction of the bias-variance trade-off. Image generated by me, inspired by [2]. From my own authorship.</figcaption></figure><h3>Regularized learning</h3><p>With regularized models, the objective functions <strong><em>L </em></strong>have the form:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/144/1*0Sa6U-IO68Emcy96bL2kUQ.png" /></figure><p>Where <strong><em>E</em></strong> measures the error for a dataset <strong><em>D. </em></strong>The Mean Squared Error for regression is one example of such measurements. Additionally, there is a new regularization term <strong><em>R, </em></strong>which <strong>penalizes</strong> the <strong>complexity</strong> of the model to <strong>avoid over-fitting</strong> and introduce desirable properties. The two most common regularized models are Ridge and Lasso [2].</p><h3>Ridge</h3><h4>Closed-form estimation of parameters</h4><p>This type uses the Euclidean norm of the weights as the regularization term:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/384/1*knPPzT1ot6vsXow3XYf8xA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/144/1*ylpnMfVZ6yPs3kjq2C7Hww.png" /></figure><p>It is easy to see that both terms are convex and differentiable as they are continuous, soft functions. Thus, the optimum has a closed-form solution:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/1*b0WJ_HJbjYTq_nQKwVvqCg.png" /></figure><pre># Function to fit a ridge model to data (notice the np.eye)<br>def fit_ridge_regressor(x, y, reg_lambda, include_bias=True):<br>    if include_bias:<br>        x = np.column_stack((np.ones((len(x), 1)), x))<br>    return np.linalg.inv(x.T@x + reg_lambda*np.eye(x.shape[1]))@x.T@y</pre><h4>Ridge regularization effect</h4><p>Ridge penalizes large weights, pushing them asymptotically towards zero, but not disabling them as you can see in Figure 6. In this scenario, the variance is reduced and highly correlated variables will have the same weight, which favors interpretability while constraining all the possible models that can be generated from the same dataset.</p><pre>from sklearn.datasets import load_diabetes<br><br># Load the housing dataset<br>dataset = load_diabetes()<br><br># Split the dataset into x and y<br>X = dataset.data<br>y = dataset.target<br><br># Compute model for varying regularization lambdas<br>num_lambdas = 30<br>reg_results = np.empty((X.shape[1]+1,num_lambdas))<br>reg_lambdas = np.linspace(start=0, stop=20, num=num_lambdas)<br>for i, reg_lambda in enumerate(reg_lambdas):<br>    betas = fit_ridge_regressor(X, y, reg_lambda) # fit model here<br>    reg_results[:, i] = betas<br><br># Plot the weights and notice them being pushed towards zero<br>plt.figure(figsize=(15,5))<br>for i in range(reg_results.shape[0]):<br>    plt.plot(reg_lambdas, reg_results[i], linestyle=&#39;dotted&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/885/1*-CPB-t5dl3hvF_mkXmLYiw.png" /><figcaption>Fig 6. The behavior of the Ridge weights as the regularization parameter increases. From my own authorship.</figcaption></figure><h4>Limitations of Ridge</h4><p>For scenarios of very high dimensionality, possibly with many variables that do not provide relevant information to the problem, the model will still consider all of the variables. Ridge will not enforce sparsity in the model, making it one notable drawback.</p><h3>Lasso</h3><h4><strong>Coordinate descent optimization</strong></h4><p>The loss function in Lasso might look very similar to Ridge but has very different properties. The regularization term is given by the sum of the absolute values of the model weights:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/384/1*6ZFpScemOoSCVkRWH7F1cg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/152/1*-cDpqADR6q5RQ7RUOrWvoQ.png" /></figure><p>This means that the overall loss function is not differentiable around zero and a closed-form solution is not possible. However, there are more ways to solve this problem!</p><p>One of the most popular is the <strong>coordinate descent</strong> algorithm which states that if a function can be written as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/1*U1-S4eUtBvZ_3SZJ04_f-g.png" /></figure><p>where <strong><em>g</em></strong> is a convex and differentiable function, and the <strong><em>h_i</em></strong> are the convex, non-smooth parts, then, the optimum can be reached by minimizing along each coordinate axis [3]:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/392/1*e8LZuyDELkpHDSBYSrRN4A.png" /></figure><p>In the case of Lasso, coordinate descent is a perfect fit since:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/338/1*0JezassFylnlKXphNq5Z1A.png" /></figure><p>Note that the <strong>penalization</strong> term is <strong>still not differentiable</strong>. But wait! Now we arrive at the concept of subdifferentials [4]. It is a function that returns a set of values that lie in between the left and right derivatives of the function to be differentiated:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/264/1*IRY8NKyeOWi5DzxSi6n9xQ.png" /></figure><p>Thus, the optimum of the loss function with respect to each coordinate is given by:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/1*xFN7E6duTlF0a3_RPgW3gg.png" /></figure><p>where:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/96/1*H_hCktxD-b-L2_CH4C35tw.png" /></figure><p>Our goal is to solve for the coefficient <strong><em>i. </em></strong>If we further define a variable <strong><em>z_i</em></strong> as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/236/1*1sHQoEs2QklxklLq11h6ug.png" /></figure><p>the solution for lasso is given by soft-thresholding:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/314/1*ch6A9TJkhUI-KQymJSpENQ.png" /></figure><pre>def soft_threshold(z_i,lambda_reg):<br>    if z_i &lt; - lambda_reg:<br>        return (z_i + lambda_reg)<br>    elif z_i &gt;  lambda_reg:<br>        return (z_i - lambda_reg)<br>    else: <br>        return 0<br><br>def fit_lasso_regressor(x, y, reg_lambda, include_bias=True, n_iter=50):<br>    # Random initialization<br>    betas = np.random.uniform(size=x.shape[1])<br>    <br>    # Coordinate descent for n_iter iterations<br>    for i in range(n_iter):<br>        # Minimizing one coordinate at a time<br>        for j in range(X.shape[1]):<br>            x_j = X[:,j].reshape(-1,1)<br>            x_minus_j = np.delete(X,j,1)<br>            betas_minus_j = np.delete(betas, j)<br>            z_j = int((x_j.T @ (y - x_minus_j@betas_minus_j)) / ( x_j.T@x_j))<br><br>            if include_bias and j == 0:<br>                betas[j] = z_j<br>            else:<br>                betas[j] = soft_threshold(z_j, 0.5) <br>    return betas </pre><h4>Lasso regularization effect</h4><p>Lasso, will <strong>enforce sparsity</strong> in the model, pushing some of the weights to be exactly zero, acting as an <strong>automatic feature selector. </strong>This effect can be seen in Figure 7. For instance, when a pair of features are highly collinear, Lasso will disable the weights in one of them. This regularization effect results in a reduction of model complexity and prevents the effects of overfitting.</p><pre># Calculate betas for varying regularization lambdas<br>num_lambdas = 100<br>reg_results = np.empty((X.shape[1],num_lambdas))<br>reg_lambdas = np.linspace(start=0, stop=1000, num=num_lambdas)<br>for i, reg_lambda in enumerate(reg_lambdas):<br>    betas = fit_lasso_regressor(X, y, reg_lambda)<br>    reg_results[:, i] = betas<br><br># Plot the weights and notice them being pushed towards zero<br>plt.figure(figsize=(15,5))<br>for i in range(reg_results.shape[0]):<br>    plt.plot(reg_lambdas, reg_results[i], linestyle=&#39;dotted&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/885/1*GmMDgMBgvIdQc_Uz3mD4PA.png" /><figcaption>Fig 7. The behavior of the Lasso weights as the regularization parameter increases. From my own authorship.</figcaption></figure><h4>Limitiations of Lasso</h4><p>According to the authors of Elastic-Net [5], the Lasso model, by design, has three major shortcomings:</p><ul><li>When the number of features <strong><em>p</em></strong> is greater than the number of observations <strong><em>n</em></strong>, the lasso selects at most <strong><em>n</em></strong> variables before it saturates. This is a limiting feature for a variable selection method.</li><li>If there is a group of variables among which the pairwise correlation is very high, then the lasso tends to select only one variable from the group and does not care which one is selected. This affects model interpretability.</li><li>For the usual <strong><em>n &gt; p</em></strong> situation, if there are high correlations between features, it has been observed that the performance of Lasso is dominated by ridge regression.</li></ul><h3>Elastic-Net</h3><p>The naive Elastic-Net is a regularized method that follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/548/1*H0xzMFGxM999ALyo3B-V-g.png" /></figure><p>As you can see, it includes both Ridge and Lasso regularization terms. However, we can reorganize this equation to include a single regularization hyperparameter and a ratio as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/596/1*V7VAIKDCeVryYvoJmcTyHw.png" /></figure><p>If <strong><em>α=0</em></strong>, then Elastic-Net reduces to Ridge. Conversely, if <strong><em>α=1</em></strong>, it will be equivalent to Lasso. Furthermore, this method was designed to overcome the limitations of the later. Specifically [5]:</p><ul><li>The original loss function can be reorganized into an equivalent problem by playing with the regularization parameters in a way such that we get artificially augmented data with size <strong><em>n+p</em></strong> and rank <strong><em>p</em></strong>. Thus, for the <strong><em>p &gt; n</em></strong> case, the model can select all <strong><em>p</em></strong> features. See the paper for more details here.</li><li>The Elastic-Net regularization penalty is designed to be strictly convex. Thus, it will assign the same values to the regression coefficients if the corresponding features are highly correlated. This is called the grouping effect and solves the interpretability issue of Lasso.</li><li>According to the empirical studies, after optimizing the regularization hyperparameters, Elastic-Net is not outperformed by either Ridge or Lasso, but it keeps both variable selection and shrinkage properties.</li></ul><h4>Estimating the coefficients with coordinate descent</h4><p>The coefficients of Elastic-Net can be obtained using coordinate decent. The optimum of each coordinate is given by:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/618/1*OgfLHoCot793WrcCZzlx3g.png" /></figure><p>where:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/96/1*H_hCktxD-b-L2_CH4C35tw.png" /></figure><p>Solving for the coefficient at coordinate <strong><em>i</em></strong>, we get the soft thresholding operator:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/370/1*_TYReeQnxP86LCwHXAoY_w.png" /></figure><p>where <strong><em>z_i</em></strong> is given by:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/236/1*39ICpAbpxxfd0VS1qW7zWg.png" /></figure><pre>def fit_elastic_net_regressor(x, y, reg_lambda, reg_ratio, include_bias=True, n_iter=50):<br>    # Random initialization<br>    betas = np.random.uniform(size=x.shape[1])<br>    <br>    # Coordinate descent for n_iter iterations<br>    for i in range(n_iter):<br>        # Minimizing one coordinate at a time<br>        for j in range(X.shape[1]):<br>            x_j = X[:,j].reshape(-1,1)<br>            x_minus_j = np.delete(X,j,1)<br>            betas_minus_j = np.delete(betas, j)<br>            z_j = float((x_j.T @ (y - x_minus_j@betas_minus_j))/ ( x_j.T@x_j + (1+reg_lambda*(1-reg_ratio))))<br><br>            if include_bias and j == 0:<br>                betas[j] = z_j<br>            else:<br>                betas[j] = soft_threshold(z_j, reg_lambda*reg_ratio) <br>    return betas</pre><p>And now let’s test this method with fixed ratios and varying lambdas to see how the weights of the models behave. This is just an example. In real life, you should properly tune the hyperparameters to maximize performance.</p><pre># Calculate betas for varying regularization lambdas<br>def try_elastic_net(fixed_ratio, X, y):<br>    num_lambdas = 100<br>    reg_results = np.empty((X.shape[1],num_lambdas))<br>    reg_lambdas = np.linspace(start=0, stop=100, num=num_lambdas)<br>    for i, reg_lambda in enumerate(reg_lambdas):<br>        betas = fit_elastic_net_regressor(X, y, reg_lambda, fixed_ratio)<br>        reg_results[:, i] = betas<br><br>    # Plot the weights and notice them being pushed towards zero<br>    plt.figure(figsize=(15,5))<br>    for i in range(reg_results.shape[0]):<br>        plt.plot(reg_lambdas, reg_results[i], linestyle=&#39;dotted&#39;)</pre><p>You can clearly see that a ratio closer to zero will make the model look closer to Ridge, while a ratio closer to one will resemble Lasso:</p><pre># Models that behaves similar to Ridge (ratio 0.1)<br>try_elastic_net(0.1, X, y)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/885/1*P3ccCdcLB9Z9LS9BRfs4Tw.png" /><figcaption>Fig 8. The behavior of the Elastic-Net weights with a rate parameter closer to 0. Note the similarities with Ridge. From my own authorship.</figcaption></figure><pre># Models that behaves similar to Lasso (ratio 0.9)<br>try_elastic_net(0.9, X, y)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/885/1*kudfrgvtOlFuxSYxST4Uqw.png" /><figcaption>Fig 9. The behavior of the Elastic-Net weights with a rate parameter closer to 1. Note the similarities with Lasso. From my own authorship.</figcaption></figure><h3>Practical exercise using scikit-learn</h3><p>Let’s do a quick exercise on how to use these methods can be used to model datasets. For this exercise, <a href="https://scikit-learn.org/stable/index.html"><strong>scikit-learn</strong></a> will be used since it is the most popular library for classical machine learning algorithms. It is <strong>stable</strong> and under continuous, active development.</p><h4>Splitting and standardizing the dataset</h4><p>In this particular example, we are going to use the diabetes dataset, found in the scikit-learn API. This dataset has</p><ul><li>442 instances</li><li>10 predictor columns (age, sex, body mass, index, glucose, etc)</li><li>One continuous numerical target that represents the progression of the disease.</li></ul><p>Moreover, the first step is to split the data into a train set and a test set. The <strong>train set</strong> is used by the model to learn a function that <strong>maps</strong> the features to the targets. This means that the model has access to these targets.</p><p>On the other hand, the <strong>test set</strong> is used to <strong>assess</strong> the <strong>fitness</strong> of the model and to get a notion of performance on unseen data. In this stage, the model will receive the features, but it will not have access to the targets. However, the targets in the test set are still used to measure the error.</p><pre>import numpy as np<br>import pandas as pd<br>from sklearn.datasets import load_diabetes<br>from sklearn.model_selection import train_test_split, GridSearchCV<br>from sklearn.linear_model import Ridge, Lasso, ElasticNet<br>from sklearn.metrics import mean_squared_error<br><br><br># Load the diabetes dataset<br>diabetes = load_diabetes()<br>X = diabetes.data<br>y = diabetes.target<br><br># Split the dataset into training and testing sets<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)<br><br># Standardize the data separately for training and testing sets<br>scaler = StandardScaler()<br>X_train_scaled = scaler.fit_transform(X_train)<br>X_test_scaled = scaler.transform(X_test)</pre><h4>Hyperparameter optimization and model selection</h4><p>Remember the regularization parameters defined before? well, we cannot just guess what the best values would be. Proper hyperparameter optimization must be performed to get as close as possible to the true model.</p><p>In this scenario, the classic grid search method will be used, which simply fits different models using all the possible combinations of hyperparameters defined in the search space. There are a variety of more sophisticated methods available, but is not the scope of this entry to explore them. I will leave that up to the reader!</p><pre># Parameters for each method<br>ridge_params = {&#39;alpha&#39;: np.logspace(-4, 4, 100)}<br>lasso_params = {&#39;alpha&#39;: np.logspace(-4, 4, 100)}<br>elastic_params = {&#39;alpha&#39;: np.logspace(-4, 4, 100), <br>                  &#39;l1_ratio&#39;: np.linspace(0.1, 0.9, 10)}</pre><p>Behind the scenes, scikit-learn will take the train set and split it further to make an additional validation set. The model will not see the targets, in the same way as in the test set, and it is used for tuning the model. After defining what values we want to explore for the hyperparameter optimization, we want to pick the model that had the least error.</p><p>Why don’t we do this on the test set? Because we don’t want to leak information to the model when assessing its ability for generalization. This would be like telling the model what are the best hyperparameters for that particular sample and it would be cheating!</p><pre>def perform_regression(method, params, X_train, y_train, X_test, y_test):<br>    &quot;&quot;&quot;<br>    Performs hyperparameter optimization, searching for the<br>    best regularization parameters, then makes predictions and<br>    computes regression error on the test set.<br>    &quot;&quot;&quot;<br>    # Optimization<br>    grid = GridSearchCV(method, params, cv=5)<br>    grid.fit(X_train, y_train)<br>    best_model = grid.best_estimator_<br>    best_params = grid.best_params_<br>    # Prediction<br>    predictions = best_model.predict(X_test)<br>    # Test error<br>    mse = mean_squared_error(y_test, predictions)<br>    return best_params, mse</pre><h4>Testing the trained models</h4><p>Finally, after the model is fitted with the best hyperparameters, what is left is to use the test set to measure the error.</p><pre># Perform regression for each method<br>results = []<br>methods = [<br>    (Ridge(), ridge_params),<br>    (Lasso(), lasso_params),<br>    (ElasticNet(), elastic_params)<br>]<br><br>for method, params in methods:<br>    best_params, mse = perform_regression(method, params, X_train_scaled, y_train, X_test_scaled, y_test)<br>    results.append({<br>        &#39;Method&#39;: method.__class__.__name__,<br>        &#39;Best Alpha&#39;: best_params.get(&#39;alpha&#39;, &#39;N/A&#39;),<br>        &#39;Best L1 Ratio&#39;: best_params.get(&#39;l1_ratio&#39;, &#39;N/A&#39;),<br>        &#39;MSE&#39;: mse<br>    })<br><br># Bar chart for MSE comparison<br>plt.figure(figsize=(10, 6))<br>plt.bar(results_df[&#39;Method&#39;], results_df[&#39;MSE&#39;], color=[&#39;blue&#39;, &#39;green&#39;, &#39;purple&#39;])<br>plt.xlabel(&#39;Regression Method&#39;)<br>plt.ylabel(&#39;Mean Squared Error (MSE)&#39;)<br>plt.title(&#39;MSE Comparison: Ridge vs Lasso vs Elastic Net&#39;)<br>plt.xticks(rotation=45)<br>plt.tight_layout()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/424/1*tqYNC9hNRXOii45CmOvvEQ.png" /><figcaption>Fig 10. Plot of the Mean Squared Error using the Diabetes dataset. From my own authorship.</figcaption></figure><p>Note that Lasso here got the least MSE of the three. However, be very careful since this exercise was not meant to be extensive experimentation to draw conclusions about any of the models. In addition, there are many more techniques that can be used for modeling such as support vector machines, or neural networks.</p><p>If you reached this point, I want to thank you for reading and for your support. These posts require quite some work, but I really enjoy doing them. Please give it some claps if you liked it and follow me for future posts.</p><h3>References</h3><p>[1] Berrendero, J. Estimación paramétrica. Universidad Autónoma de Madrid. Madrid, Spain. 2022.</p><p>[2] Alaíz, C. Review of Linear Models. Universidad Autónoma de Madrid. Madrid, Spain. 2022.</p><p>[3] Tibshirani, R. Coordinate Descent. Carnegie Melon University. Pittsburgh, PA. Retrieved from: <a href="https://www.stat.cmu.edu/~ryantibs/convexopt-S15/lectures/22-coord-desc.pdf">https://www.stat.cmu.edu/~ryantibs/convexopt-S15/lectures/22-coord-desc.pdf</a></p><p>[4] Breheny, P. The lasso. University of Iowa. City of Iowa, Iowa. Retrieved from: <a href="https://myweb.uiowa.edu/pbreheny/7600/s16/notes/2-15.pdf">https://myweb.uiowa.edu/pbreheny/7600/s16/notes/2-15.pdf</a></p><p>[5] Zou, H. and Hastie, T., 2005. Regularization and variable selection via the elastic net. <em>Journal of the Royal Statistical Society Series B: Statistical Methodology</em>, <em>67</em>(2), pp.301–320.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec152eda8f08" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Too Many Features? Let’s Look at Principal Component Analysis]]></title>
            <link>https://medium.com/data-science/too-many-features-lets-look-at-principal-component-analysis-62504b791ae9?source=rss-4f706205679a------2</link>
            <guid isPermaLink="false">https://medium.com/p/62504b791ae9</guid>
            <category><![CDATA[dimensionality-reduction]]></category>
            <category><![CDATA[principal-component]]></category>
            <category><![CDATA[data-processing]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Hector Andres Mejia Vallejo]]></dc:creator>
            <pubDate>Sat, 24 Jun 2023 05:25:17 GMT</pubDate>
            <atom:updated>2023-06-24T05:25:17.509Z</atom:updated>
            <content:encoded><![CDATA[<h4>Home-brewed machine learning model series</h4><p><em>The companion repo is available </em><a href="https://github.com/hector6298/medium_posts/tree/main/pca"><em>here</em></a><em>!</em></p><p>The curse of dimensionality is one major problem in machine learning. As the number of features increases, so does the complexity of the model. Moreover, if there is not enough training data, it results in overfitting.</p><p>In this entry, Principal Component Analysis (PCA) will be introduced. First, I will explain why too many features are a problem. Then, the math behind PCA and why it works. Additionally, PCA will be broken down into steps, accompanied by visual examples and code snippets. Moreover, the advantages and disadvantages of PCA will be explained. Lastly, these steps will be encapsulated in a Python class for later use.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fpBsRcFPbDKai8vA" /><figcaption>Projecting a vector into a lower space is like casting a shadow in real life. PCA finds the direction from where to cast these shadows. Photo by <a href="https://unsplash.com/@inbalmarilli?utm_source=medium&amp;utm_medium=referral">inbal marilli</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong>Note for the reader: </strong>If you are not interested in the math explanation and just want to see the practical examples and how PCA works, jump to the section<strong> “</strong><a href="#ea59"><strong>PCA in practice</strong></a><strong>”</strong>. If you are only interested in the Python class head to <strong>“</strong><a href="#6814"><strong>Home-brewed PCA implementation</strong></a><strong>”</strong>.</p><h3>The problem with too many features?</h3><p>Take a look at the feature space in Fig 1. There are few examples to fill all the space, so a model of this data might not generalize well to new, unseen examples.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*cgSYspiNhnpbu7bHIumTBA.png" /><figcaption>Fig 1. An example of 2-dimensional feature space. From my own authorship, inspired by [1].</figcaption></figure><p>What happens if we add another feature? Let’s have a look at the new feature space in Fig. 2. You can see that there is even more empty space than in the previous example. As the number of features increases, the model will overfit the current data. That is why there are techniques to reduce the dimensionality of the data and alleviate this problem. [1]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/392/1*6uhA2HhNaGEK8cFhJ6Nbqw.png" /><figcaption>The same example with an additional feature. From my own authorship, inspired by [1].</figcaption></figure><h3>What is the goal of PCA?</h3><p>In a few words, the purpose of PCA is to extract new, uncorrelated features of lower dimensions that maximize the amount of information kept from the original data. The measure of information in this context is the variance. Let’s see why:</p><p>This technique is based on the assumption that our d-dimensional data point <strong><em>x </em></strong>can be represented by a linear combination of vectors of an <a href="https://en.wikipedia.org/wiki/Orthonormal_basis">orthonormal basis</a> [1]:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/122/1*GkjUkB3128N_34aRAWVqBw.png" /></figure><p>Do not worry, I will explain where we get the vectors of said basis later. Moreover, we can extract a representation <strong><em>x̂ </em></strong>using <strong><em>m </em></strong>out of the<strong><em> d </em></strong>vectors in the combination (<strong><em>m &lt; d</em></strong>):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/122/1*1IQt_HcrAEM8_znqJECEAA.png" /></figure><p>Of course, we are not getting an exact representation since there are fewer features, but at least we try to minimize the loss of information. Let us define the Mean Squared Error (MSE) between the original example <strong><em>x </em></strong>and the approximation <strong><em>x̂</em></strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/318/1*ebryd29bUWV7M75Qbf4HOA.png" /></figure><p>As the summations use the same variables with different cutoffs, then the difference is just the offset:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/270/1*UxKxoeczbHDlzzw-AU6k8Q.png" /></figure><p>We know by our starting hypothesis that <strong><em>x </em></strong>is the sum of orthonormal vectors. Hence, the dot product of these vectors is zero, and each of their Euclidean norms is one. Thus:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/270/1*YN_75DPlH2FRJQrx2wmM_Q.png" /></figure><p>Solving the importance value <strong><em>yi:</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/98/1*52cJnN4VtKKI60WqdKqSeA.png" /></figure><p>Plugging that result into its expectation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/240/1*O78Ll-016I2DOO_GmgpliQ.png" /></figure><p>We can see that if <strong><em>xi </em>is centered (mean equal to zero), then the expectation turns out to be the covariance matrix of the whole data, </strong>and this result is nothing more than the variance in the original space. By choosing the right vectors <strong><em>vi </em></strong>that maximize the variance, we will effectively minimize the representation error.</p><h3>Where does this orthonormal basis come from?</h3><p>As previously stated, we want to get <strong><em>m</em></strong> vectors that maximize the variance:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/408/1*cEahqp18cw-NsDwTQL7lDQ.png" /></figure><p>If we take the whole data matrix, it can be seen that <strong><em>vi</em></strong> is a projection direction. The data is going to be projected in a space of lower dimensionality.</p><p>If we diagonalize the covariance matrix Σ using spectral decomposition we get:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/304/1*VbJne3o7DruBVUpmesuNTA.png" /></figure><p>Where <strong><em>U</em></strong> is a matrix with the normalized eigenvectors of<strong><em> </em></strong>Σ, and Λ is a diagonal matrix that contains the eigenvalues of Σ in descending order. This is possible since Σ is a real, symmetric matrix.</p><p>Furthermore, as Λ only contains non-zero values on the diagonal, we can rewrite the above equation as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/188/1*NvbJmerugLee-VTeAeMmkQ.png" /></figure><p>with:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/296/1*3wbgcWAny7JADCBVd-qj5w.png" /></figure><p>Notice that the vectors in <strong><em>U </em></strong>and vector<strong><em> v </em></strong>are normalized<strong><em>. </em></strong>Thus, when performing the squared dot product of each v with a, we get a value between<strong><em> [0,1] </em></strong>and so<strong><em> w </em></strong>must also be a normalized vector:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/108/1*jErKKeDHs7RMdn7KIhIHVA.png" /></figure><p>From here, interesting properties arise.</p><h4>The first principal component</h4><p>Recall the optimization problem. Since the eigenvalues are ordered and <strong><em>w</em></strong> must be a normalized vector, our best option is to get the first eigenvector with <strong><em>w = (1,0,0,…)</em></strong>. As a consequence, the upper bound is attained when:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/68/1*F1T1Kvl_XZ09vYQOcGjY4w.png" /></figure><p>The projection direction that maximizes the variance turns out to be the <strong>eigenvector</strong> associated with the <strong>largest eigenvalue</strong>!</p><h4>The rest of the components</h4><p>Once the first principal component is set, a new restriction to the optimization problem is added:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/352/1*fPFJidj2kkpCOs-TpkHa4w.png" /></figure><p>It means that the new component <strong><em>v2 </em></strong>must be orthogonal to the previous component, the eigenvector<strong><em> u1</em></strong> so that the information is not redundant. It can be proved that all the d components correspond to the d normalized eigenvectors from Σ associated with the eigenvalues, in descending order. Take a look at <a href="https://verso.mat.uam.es/~joser.berrendero/blog/Componentes_principales.pdf">these notes</a> for formal proof of this claim [2].</p><h3>PCA in practice</h3><p>From the theoretical description above the steps needed to get the principal components of a dataset can be described. Let the initial dataset be a random sample of the following 2D normal distribution:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/386/1*9FT3s3agOwI5BBeoFhMdZw.png" /></figure><pre>from scipy import stats<br>mean = [3,3]<br>var = [[6, 3],<br>       [3, 3.5]]<br>n = 100<br>data_raw = np.random.multivariate_normal(mean, var, 100)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/398/1*4vKX1O-cYisJYp-zdPxfgg.png" /><figcaption>Fig 3. Original point cloud. Non-centered. From my own authorship.</figcaption></figure><h4>1. Centering the data</h4><p>The first step is to move the cloud to the origin of the coordinate system so the data has zero mean. This step is performed by subtracting the sample mean from every point in the dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/312/1*K_Q4Zd-RHWrPdUFnj3Sr6g.png" /></figure><pre>import numpy as np<br>data_centered = data_raw - np.mean(data_raw, axis=0)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/398/1*rfsLIAqf-xzwaYzM4ITwmQ.png" /><figcaption>Fig 4. Centered point cloud. From my own authorship.</figcaption></figure><h4><strong>2. Computing the covariance matrix</strong></h4><p>The variance defined above is the population covariance matrix Σ. In practice, that information is not available to us as we only have one sample. Therefore, we can approximate that parameter by using the sample covariance <strong><em>S</em></strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/594/1*FM0VogpWs50fM9ZuVa-uhg.png" /></figure><p>Recall that the data is already centered. Thus:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/458/1*tVOAMukiaM32i-wgTdqc8Q.png" /></figure><p>We can write this compactly using matrix multiplications. This can also help us vectorize computations:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/166/1*Ik0usw84HDtZIGamaeitFA.png" /></figure><pre>cov_mat = np.matmul(data_centered.T, data_centered)/(len(data_centered) - 1)<br># &gt; array([[5.62390186, 2.47275007],<br># &gt;       [2.47275007, 3.19395349]])</pre><p>The reason for passing the transposed matrix as the first argument in the code is that in the mathematical formulation of the data matrix, the features are in the rows and the subjects in the columns. In the implementation, the opposite happens since in almost every system, the events, subjects, logs, and so forth, are stored in rows.</p><h4>3. Perform the eigendecomposition of the covariance matrix</h4><p>The eigenvalues and eigenvectors a are computed using eig()from <a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.eig.html">scipy</a>:</p><pre>from scipy.linalg import eigh<br>eigvals, eigvecs = eigh(cov_mat)<br><br># Sorting the eigenvalues and eigenvectors<br>indices = eigvals.argsort()[::-1]<br>eigvals, eigvecs = eigvals[indices], eigvecs[:,indices]<br><br>eigvecs<br># &gt; array([[-0.82348021,  0.56734499],<br># &gt;        [-0.56734499, -0.82348021]])</pre><p>As it was explained earlier, the eigenvalues represent the variance of the principal components, and the eigenvectors are the projection directions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/398/1*Rs91Fl1byoKUfrleCcl_gA.png" /><figcaption>Fig 5. Plot of the projection directions and the point cloud. From my own authorship.</figcaption></figure><p>You can see that a new coordinate system is created using the directions of the principal components. Furthermore, the eigenvalues and eigenvectors must be stored to transform new data afterward.</p><h4>4. Enforce determinism</h4><p>The coefficients of the eigenvectors will be always the same, except for their sign. PCA can have multiple valid orientations. Thus, we need to enforce a deterministic result by taking the eigenvector matrix and for each of its columns, applying the sign of the largest absolute value within that column.</p><pre>max_abs_cols = np.argmax(np.abs(eigvecs), axis=0)<br>signs = np.sign(eigvecs[max_abs_cols, range(eigvecs.shape[1])])<br>eigvecs = eigvecs*signs<br>eigvecs<br># &gt; array([[ 0.82348021, -0.56734499],<br># &gt;      [ 0.56734499,  0.82348021]])</pre><h4><strong>5. Extract the new features</strong></h4><p>Each new feature (principal component) is extracted by performing the dot product between each point in the original feature space and the eigenvector:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/126/1*_y3tdMHJWcqbShQPzFfRFQ.png" /></figure><pre>new_features = np.dot(data_centered, eigvecs)</pre><p>For this particular example, after computing the components, the new points in the space are depicted as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/398/1*NMJ3gXz0W2gG2Yr_vEOErA.png" /><figcaption>Fig 6. Plot of the uncorrelated point cloud, using the features from the PCs. Notice that it is just a rotation of the original point cloud. From my own authorship.</figcaption></figure><p>Notice that this result is basically a rotation of the original cloud of points in a way that the attributes are uncorrelated.</p><h4>6. Reduce dimensionality</h4><p>So far, the principal components were computed in full to understand them, in a visual manner. What is left is to choose how many components are needed. We resort to the eigenvalues for this task, since they represent the variance of each principal component.</p><p>The ratio of the variance held by component <strong><em>i</em></strong> is given by:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/138/1*SBS0MImuiTHdJZIsuIZqBA.png" /></figure><p>And the ratio of the variance preserved by choosing m components is given by:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/146/1*5AoqDwEnSYR4f9COLFivzg.png" /></figure><p>If we visualize the variance for each component in our example, we arrive at the following:</p><pre># Variance of each individual component as bars<br>plt.bar(<br>    [f&quot;PC_{i}&quot; for i in range(1,len(eigvals)+1)],<br>    eigvals/sum(eigvals)<br>)<br><br># Percentage held by m components as the line<br>plt.plot(<br>    [f&quot;PC_{i}&quot; for i in range(1,len(eigvals)+1)],<br>    np.cumsum(eigvals)/sum(eigvals),<br>    color=&#39;red&#39;<br>)<br>plt.scatter(<br>    [f&quot;PC_{i}&quot; for i in range(1,len(eigvals)+1)],<br>    np.cumsum(eigvals)/sum(eigvals),<br>    color=&#39;red&#39;<br>)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/1*i35cTmJU4tNLMy1cZEgwOQ.png" /><figcaption>Fig 7. Variance explained by each principal component. From my own authorship.</figcaption></figure><p>In this case, PC1 represents 80% of the variance of the original data, with the remaining 20% belonging to PC2. Furthermore, we can choose to use only the first principal component, in which case the data would look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/386/1*zeTFtPYvuFtQXp_7TeidkA.png" /><figcaption>Fig 8. Projection of the data in the direction of the first eigenvector. From my own authorship.</figcaption></figure><p>This is the projection of the data in the direction of the first eigenvector. It does not look very useful right now. <strong>What if we instead choose data that belong to three classes? How would PCA look?</strong></p><h3>PCA on multiclass data</h3><p>Let us create a dataset with three classes that can be linearly separable:</p><pre>from sklearn.datasets import make_blobs<br>X, y = make_blobs()<br><br><br>plt.scatter(X[:,0], X[:,1],c=y)<br>plt.legend()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/377/1*ka8_rKEwT9uXqAmXFF3nUA.png" /><figcaption>Fig 9. Multi-class data. From my own authorship.</figcaption></figure><p>If we apply PCA to the data above, this would be the plot of the principal components:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/370/1*lzhyFFCkiLwuJrWTV66Z9g.png" /><figcaption>Fig 10. Multi-class uncorrelated data. From my own authorship.</figcaption></figure><p>And this would be the plot of the first component (the projection of the data in the direction of the eigenvector corresponding to the largest eigenvalue):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/386/1*qqsdfMzZCNTKv5oEEwXplw.png" /><figcaption>Fig 11. Projection of the data in the direction of the first eigenvector. This is the dimensionality reduction. From my own authorship.</figcaption></figure><p>It works! The data still looks easily separable by a linear model.</p><h3>Advantages and disadvantages</h3><p>Like everything in science, there is no silver bullet. Here is a list of advantages and disadvantages that you should take into account before using PCA with real-world data.</p><h4>Advantages of PCA</h4><ol><li><strong>Dimensionality Reduction: </strong>PCA allows for the reduction of high-dimensional data into a lower-dimensional space while preserving most of the important information. This can be useful for data visualization, computational efficiency, and dealing with the curse of dimensionality.</li><li><strong>Decorrelation: </strong>PCA transforms the original variables into a new set of uncorrelated variables called principal components. This decorrelation simplifies the analysis and can improve the performance of downstream machine learning algorithms that assume independence of features.</li><li><strong>Noise Reduction:</strong> The lower-dimensional representation obtained through PCA tends to filter out noise and focus on the most significant variations in the data. This can enhance the signal-to-noise ratio and improve the robustness of subsequent analyses.</li></ol><h4>Disadvantages of PCA</h4><ol><li><strong>Linearity Assumption: </strong>PCA assumes that the underlying data relationships are linear. If the data has complex non-linear relationships, PCA may not capture the most meaningful variations and could provide suboptimal results.</li><li><strong>Interpretability: </strong>The principal components obtained from PCA are linear combinations of the original features. It can be difficult to relate the principal components back to the original variables and understand their exact meaning.</li><li><strong>Sensitivity to Scaling:</strong> PCA is sensitive to the scaling of the input variables. If the variables have different scales, those with larger variances can dominate the analysis, potentially leading to biased results. Proper feature scaling is crucial for obtaining reliable results with PCA.</li><li><strong>Outliers:</strong> PCA is sensitive to outliers since it focuses on capturing the variance in the data. Outliers can significantly influence the principal components and distort the results.</li></ol><h3>Home-brewed PCA implementation</h3><p>Now that we have covered the details of Principal Component Analysis, all that remains is to create a class that encapsulates the aforementioned behavior and that could be reused in future problems.</p><p>For this implementation, the scikit-learn interface will be used, which has the following methods:</p><ul><li>fit()</li><li>transform()</li><li>fit_transform()</li></ul><h4>Constructor</h4><p>No complex logic is needed. The constructor will just define the number of components (features) that the transformed data will have.</p><pre>import numpy as np<br>from scipy.linalg import eigh<br><br>class PCA:<br>    &quot;&quot;&quot;Principal Component Analysis.<br>    &quot;&quot;&quot;<br>    def __init__(self, n_components):<br>        &quot;&quot;&quot;Constructor of the PCA class.<br><br>        Parameters:<br>        ===========<br>        n_components: int<br>            The number of dimensions for the transformed data.<br>            Must be less than or equal to n_features.<br>        &quot;&quot;&quot;<br>        self.n_components = n_components<br>        self._fit_instance = False</pre><h4>The fit method</h4><p>The fit method will apply steps 1–4 from the previous section.</p><ul><li>Centering the data</li><li>Computing the covariance matrix</li><li>Computing eigenvalues, eigenvectors and sorting them</li><li>Enforcing determinism by flipping the signs of the eigenvectors</li></ul><p>It will also store the eigenvalues and vectors, as well as the sample mean, as object attributes to transform new data later.</p><pre>    def fit(self, X):<br>        &quot;&quot;&quot;Compute eigenvectors to transform data later<br><br>        Parameters:<br>        ===========<br>        X: np.array of shape [n_examples, n_features]<br>            The data matrix<br>        <br>        Returns:<br>        ===========<br>        None<br>        &quot;&quot;&quot;<br>        # Fit the mean of the data and center it<br>        self.mean = np.mean(X, axis=0)<br>        X_centered = X - self.mean<br><br>        # Compute covariance matrix<br>        cov_mat = np.matmul(X_centered.T, X_centered)/(len(X_centered) - 1)<br><br>        # Compute eigenvalues, eigenvectors and sort them<br>        eigenvalues, eigenvectors = eigh(cov_mat)<br>        self.eigenvalues, self.eigenvectors = self._sort_eigen(eigenvalues, eigenvectors)<br><br>        # Get the explained variance rations<br>        self.explained_variance_ratio = self.eigenvalues/np.sum(self.eigenvalues)<br>        <br>        # Enforce determinism by flipping the eigenvectors<br>        self.eigenvectors = self._flip_eigenvectors(self.eigenvectors)[:, :self.n_components]<br><br>        self._fit_instance = True</pre><h4>The transform method</h4><p>It will apply steps 1, 5, and 6:</p><ul><li>Centering new data using the stored sample mean</li><li>Extracting the new PC features</li><li>Reducing the dimensionality by picking n_components dimensions.</li></ul><pre>    def transform(self, X):<br>        &quot;&quot;&quot;Project the data in the directions of the eigenvectors.<br><br>        Parameters:<br>        ===========<br>        X: np.array of shape [n_examples, n_features]<br>            The data matrix<br>        <br>        Returns:<br>        ===========<br>        pcs: np.array[n_examples, n_components]<br>            The new, uncorrelated features from PCA.<br>        &quot;&quot;&quot;<br>        if not self._fit_instance:<br>            raise Exception(&quot;PCA must be fitted to the data first! Call fit()&quot;)<br>       <br>        X_centered = X - self.mean<br>        return np.dot(X_centered, self.eigenvectors)</pre><h4>The fit_transform method</h4><p>For simplicity of implementation. This method will apply the fit() function first and transform() later. I am sure you can figure out a more clever definition.</p><pre>def fit_transform(self, X):<br>       &quot;&quot;&quot;Fits PCA and transforms the data.<br>       &quot;&quot;&quot;<br>       self.fit(X)<br>       return self.transform(X)</pre><h4>Helper functions</h4><p>These methods were defined as separate components, instead of applying all the steps in the fit()function to make the code more readable and maintainable.</p><pre>    def _flip_eigenvectors(self, eigenvectors):<br>        &quot;&quot;&quot;Enforce determinism by changing the signs of the eigenvectors.<br>        &quot;&quot;&quot;<br>        max_abs_cols = np.argmax(np.abs(eigenvectors), axis=0)<br>        signs = np.sign(eigenvectors[max_abs_cols, range(eigenvectors.shape[1])])<br>        return eigenvectors*signs<br>    <br>    def _sort_eigen(self, eigenvalues, eigenvectors):<br>        &quot;&quot;&quot;Sort eigenvalues in descending order and their corresponding eigenvectors.<br>        &quot;&quot;&quot;<br>        indices = eigenvalues.argsort()[::-1]<br>        return eigenvalues[indices], eigenvectors[:, indices]</pre><h4>Testing the class</h4><p>Let’s use the previous example with our PCA class:</p><pre>from pca import PCA<br><br># Using our PCA implementation<br>pca = PCA(n_components=1)<br>X_transformed = pca.fit_transform(X)<br><br># Plotting the first PC<br>plt.scatter(X_transformed[:,0], [0]*len(X_transformed),c=y)<br>plt.legend()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/386/1*qqsdfMzZCNTKv5oEEwXplw.png" /><figcaption>Fig 12. Results using the PCA class. From my own authorship.</figcaption></figure><h3>Conclusion</h3><p>Having many features with small data can be harmful and will, most likely, result in overfitting. Principal Component Analysis is a tool that can help alleviate this problem. It is a dimensionality reduction technique that works by finding projection directions for the data in a way that the original variability is preserved as much as possible, and the resulting features are uncorrelated. Moreover, the variance explained by each new feature, or principal component, can be measured. Then, the user can choose how many principal components and how much variance is enough for the task. Finally, be sure to know your data first, as PCA works with samples that can be linearly separated and can be sensitive to outliers.</p><h3>References</h3><p>[1] Fernández, A. Dimensionality Reduction. Universidad Autónoma de Madrid. Madrid, Spain. 2022.</p><p>[2] Berrendero, J. R. Regresión lineal con datos de alta dimensión. Universidad Autónoma de Madrid. Madrid, Spain. 2022.</p><p>Connect with me on <a href="https://www.linkedin.com/in/hectorandresmv/">LinkedIn</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=62504b791ae9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/too-many-features-lets-look-at-principal-component-analysis-62504b791ae9">Too Many Features? Let’s Look at Principal Component Analysis</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Warehouse or Data lakehouse?]]></title>
            <link>https://hectormrejia.medium.com/data-warehouse-or-data-lakehouse-951d0a7e8ce8?source=rss-4f706205679a------2</link>
            <guid isPermaLink="false">https://medium.com/p/951d0a7e8ce8</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[business]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <dc:creator><![CDATA[Hector Andres Mejia Vallejo]]></dc:creator>
            <pubDate>Thu, 22 Sep 2022 08:53:15 GMT</pubDate>
            <atom:updated>2022-09-22T08:53:15.585Z</atom:updated>
            <content:encoded><![CDATA[<h4>The platform you need for your analytics apps</h4><blockquote>See the official article published in Factored’s (where I work) tech series <a href="https://factored.ai/portfolio/data-warehouse-data-lakes/">here</a>!</blockquote><p>Information is without a doubt the most valuable asset for modern businesses. It allows them to make good decisions and drive strategies. For decades, there was this concept of a data warehouse, where a company could store all its data and leverage it to deliver insights. Then, a much newer concept arrived, the data lakehouse. Which architecture is right for you? Is there a reference for how to use them?</p><p>In this article, we define and explore data analytics patterns with these two options. The data lake is not introduced here as it lacks a data governance layer, making it difficult to deliver efficient data analytics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AHDdVxsPy0ke9I3e" /><figcaption>A warehouse. Photo by <a href="https://unsplash.com/@chuttersnap?utm_source=medium&amp;utm_medium=referral">CHUTTERSNAP</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Analytics with the Data Warehouse</h3><p>A data warehouse is a central repository of structured data that powers business intelligence applications. <strong>Why is it important? </strong>Intuition and instinct are not enough to be successful. That is why we need data. A data warehouse enforces standardization and accuracy. According to a Gartner study, organizations believe that poor data quality is responsible for $15 million a year on average in losses [1]. Ensuring data is accurate and in a central location also boosts efficiency as it eliminates the need to ask for validations and assistance from support staff or other analysts and enables a myriad of ways to automate new processes.</p><p>This architecture has a governance layer that makes data secure, manageable, and scalable. It includes [2]:</p><ul><li>Metadata: To locate, tag, and discover the structure of the data</li><li>Data model: How is the data abstracted and organized?</li><li>Data lineage: The history of the data including its origin and all its transformations</li><li>KPIs: Key Performance Indicators that drive the design of the warehouse</li><li>ETL: The technology that makes the data usable.</li></ul><p>Furthermore, this concept was created to work with structured data. Meaning databases, containing relational tables. There is a common design pattern to create databases called the star schema. It consists of creating fact tables containing measurements from the common operation of the business, and dimensional tables that give context to the facts and contain data that does not change often.</p><p>As the data is structured, it is meant for businesses that have a concrete use case, where the shape of the data does not change over the short term. The most common example is a bank. First, you start asking questions like:</p><ul><li>Which location has the highest monthly transactions?</li><li>Which bank loaned the highest amount of money this year?</li></ul><p>The facts and dimensions would be joined, then aggregated using functions like summation, average, or median, over the field where the data will be grouped. In this case, the location of the banks. Finally, This aggregated data is transformed into visualizations that answer business questions quickly and intuitively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6aPVg3PwV1DQI-ONemrpwA.png" /></figure><h3>Analytics with the Data lakehouse</h3><p>As with the latter, a Data Lakehouse is also a repository of data. The difference is that lakehouses can also ingest semi-structured and unstructured data as well, to develop both batch and streaming applications. <strong>Why is it important?</strong> Nowadays, data comes from a large variety of sources and shapes. According to IBM, unstructured data accounts for over 80% of enterprise data [3].</p><p>Moreover, this architecture is built to work with open file formats like parquet and ORC, much like a data lake, allowing us to easily train machine learning models. These advancements enable advanced analytics applications that can be tailor-made for modern businesses.</p><p>Consider the example of an IoT company, which has thousands of sensors placed on a farm. It needs to monitor the properties of the soil in near real-time to optimize the use of water and other products. With the lakehouse, we can develop a multi-layer data model.</p><ul><li>The first layer ingests the data in its raw format.</li><li>The second layer transforms and validates the data. Reference data can also be joined at this stage.</li><li>The third layer aggregates the data for visualization</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I5x28qv9DogEuTZJzb_iFg.png" /></figure><p>The raw layer is possible because of the cheap, open storage system of the lakehouse. Further batch analytics can be performed from this layer to gain deeper insights that might have been missed in the real-time application and can follow the same multi-layer pattern explained above.</p><p>On top of all these capabilities, there is the same governance layer that warehouses provide on top of the data, making it usable, easily discoverable, traceable, and secure.</p><h3>Should we all follow a data lakehouse architecture?</h3><p>There is no silver bullet when it comes to data. One has to analyze the requirements for the use case and pick the tool that best suits their needs. If most of your data comes from relational tables, and the nature of the business does not rapidly change, then a data warehouse would be a great option. There are very mature platforms like Snowflake, Amazon Redshift, and Azure Synapse.</p><p>On the other hand, If you want to extract value from various types of data sources and your data models are likely to evolve, then a data lakehouse might be a good choice. There is also a two-tier architecture that uses a data lake, followed by a warehouse, but it is more costly and involves replicating data from one repository to the other. In addition, if you need to perform machine learning modeling, or other advanced analytics in a single platform, then consider a lakehouse.</p><h3>References</h3><p>[1] Moore, S. (2018). <em>How to Create a Business Case for Data Quality Improvement</em>. <a href="http://https//Www.Gartner.Com/Smarterwithgartner/How-to-Create-a-Business-Case-for-Data-Quality-Improvement.">Https://Www.Gartner.Com/Smarterwithgartner/How-to-Create-a-Business-Case-for-Data-Quality-Improvement.</a></p><p>[2] Inmon Bill, Levins, M., &amp; Srivastava, R. (2021). <em>Building the Data Lakehouse</em> (J. Hoberman, Ed.). Technics Publications.</p><p>[3] IBM Cloud Education. (2021). <em>Structured vs. Unstructured Data: What’s the Difference?</em> <a href="http://https//Www.Ibm.Com/Cloud/Blog/Structured-vs-Unstructured-Data.">Https://Www.Ibm.Com/Cloud/Blog/Structured-vs-Unstructured-Data.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=951d0a7e8ce8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Process On-Demand Data without Idle Databricks Clusters]]></title>
            <link>https://medium.com/data-science/process-on-demand-data-without-idle-databricks-clusters-cbf0bd99d8d6?source=rss-4f706205679a------2</link>
            <guid isPermaLink="false">https://medium.com/p/cbf0bd99d8d6</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[azure]]></category>
            <dc:creator><![CDATA[Hector Andres Mejia Vallejo]]></dc:creator>
            <pubDate>Thu, 18 Aug 2022 09:48:44 GMT</pubDate>
            <atom:updated>2022-08-18T09:48:44.365Z</atom:updated>
            <content:encoded><![CDATA[<h4>Adding cloud functions before your workflow for a cost-effective solution</h4><p>Databricks is a Platform-as-a-Service offering that lets Data Engineers perform ingestion, exploration, and transformation of data. Recently, they included the ability to create data pipelines into their platform with multi-task workflows. In addition, they provide a configurable and scalable data ingestion feature called the Autoloader. With it, you can ingest data both on-demand and in a scheduled fashion. However, if you want to ingest your data as soon as it comes, you must have your job active while it waits for the files to arrive. This is incompatible with the workflow feature and costs you money for having a compute cluster indefinitely turned on. Such is the case that a former company where I worked spent thousands of dollars on a single project having idle clusters listening for data.</p><p><strong>But do you have to do it that way though? </strong>Using cloud functions as your data sensor, you can cost-effectively trigger your Databricks pipelines. You will only use your cluster for actual processing. Here’s how!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Aw8HQZAex5Cf0xLj" /><figcaption>The cloud is everything now. Photo by <a href="https://unsplash.com/@z734923105?utm_source=medium&amp;utm_medium=referral">Jerry Zhang</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a>.</figcaption></figure><h3>0. Preliminaries</h3><p>You can find my companion repository <a href="https://github.com/hector6298/azFunctionsDatabricks/tree/master">here</a>. Microsoft Azure will be the Cloud provider for this particular exercise, but the concepts can be translated to other providers. The dataset will be the “<em>New York Yellow Taxi Trip Data” </em>and can be found <a href="https://www.kaggle.com/datasets/microize/newyork-yellow-taxi-trip-data-2020-2019">here</a>. However, there are many partitions of the dataset. Download one CSV file and create a new file with a subset of that data to minimize waiting when uploading it to the Azure Container in the testing step.</p><h3>1. File Staging Area</h3><p>You will need a storage service where your data is going to land and will be later picked by Databricks for consumption. Such service could be an Amazon S3, Azure Blob Storage Container, etc. This entry will use Azure services and VS Code.</p><p>In the Azure plugin, there is a tab called <strong><em>“Resources”</em></strong>. Click on the <strong><em>“+”</em></strong> button to the right and select <strong><em>“Create Storage Account”:</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2Fm3DQTCaOzmcYw0kyFmpw.png" /><figcaption>Creating a storage account. Screenshot was taken by the author.</figcaption></figure><p>Give it a unique name, and an Azure region. Then, check the Azure Activity log in the terminal area until the resource is created. Look for the resource in the left pane, right-click and create a new container:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qoI8nwx__oGgoeNi4wOExg.png" /><figcaption>Creating a blob container. Screenshot was taken by the author.</figcaption></figure><p>After the container is created, you will need to mount it on the Databricks filesystem. To do that, run this code cell inside your Databricks workspace:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c9323aa5a6d1a7c50c022d36a445ae23/href">https://medium.com/media/c9323aa5a6d1a7c50c022d36a445ae23/href</a></iframe><h3>2. Database and Tables</h3><p>Now, create the appropriate tables based on your data model. In this simple example, a single table is used:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/76c53f5c0b0fcf333c4d0f09aa0ea0a9/href">https://medium.com/media/76c53f5c0b0fcf333c4d0f09aa0ea0a9/href</a></iframe><p>This table will receive taxi trip data coming from the files in the staging area.</p><h3>3. Autoloader query</h3><p>This is the core of the workload! But first, let us define the required parameters in Databricks using dbutils:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/779b098a73232fe10a7b12465c00b722/href">https://medium.com/media/779b098a73232fe10a7b12465c00b722/href</a></iframe><p>The Autoloader query will define the ETL from the staging area to the table. Let us review the code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fba91d5d05405d92bb496fbe4641cbe4/href">https://medium.com/media/fba91d5d05405d92bb496fbe4641cbe4/href</a></iframe><ul><li>cloudFiles: It indicates to the structured streaming query, that you are going to use the Autoloader.</li><li>cloudFiles.format: The format of the data coming in. In this case, CSV.</li><li>cloudFiles.schemaHints: By default, all the discovered fields are going to be treated as strings. Here you can specify the datatypes of specific fields.</li><li>load(): To specify the location of the data that is going to be loaded. In this case, the location of the mounted container.</li><li>select(): To pick a subset of all the columns.</li><li>trigger(availableNow=True): Very important. It specifies to the query that you want to process all the data that has not been consumed yet and is available right now. Otherwise, this query will run every half a second by default.</li><li>table(): To store the data in a specified table.</li></ul><h3>4. The Databricks Job</h3><p>Now that the query is defined, the actual data pipeline is going to be created. The beauty of the redefined Databricks workflows is that you can define a set of tasks and dependencies. Something that could not be done on this platform before. For this easy example, a single task is going to be defined. You can read about how to create more complex workflows <a href="https://docs.databricks.com/workflows/jobs/jobs-quickstart.html">here</a>. Also, check out my <a href="https://hectormrejia.medium.com/use-these-methods-to-orchestrate-your-pipelines-2b8b614285ba">article</a> about pipeline orchestration in Databricks.</p><p>Inside <strong><em>“Path”</em></strong>, look for the location of the notebook where the Autoloader script is. Furthermore, issue the required arguments as depicted in the screenshot:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/623/1*brKAxnGGV5-QYykbaJh6hQ.png" /><figcaption>Creating the Databricks Job. Screenshot of own authorship.</figcaption></figure><p>a couple of things to note:</p><ul><li>dbfs:/mnt/test is the directory of the mounted Azure storage container, as defined earlier in the mounting snippet.</li><li>default is the name of the database that gets created by default.</li><li><strong><em>checkpoint_directory</em></strong> is set to /tmp/which is a directory that gets wiped out every time the cluster restarts. <strong>Never</strong> set it as a checkpoint directory on a real scenario, this is just for practice.</li></ul><h3>5. The Function App</h3><p>This is the key component to enabling the intended on-demand behavior. The function will be triggered as soon as the data arrives at the staging area and, in turn, it will run the Databricks Workflow using the Databricks Jobs API.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zKGe6M8LTlpCyN9dwlJ-ng.png" /><figcaption>On-demand processing. Image created by me using Azure icons :)</figcaption></figure><p>To create a function in VS Code, look for the Azure plugin. On the workspace tab select <strong><em>“+”</em></strong> and click <strong><em>“Create Function”</em></strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/473/1*NmkVakY-bNo58aC65mBjcg.png" /><figcaption>Creating a local function. Screenshot was taken by the author.</figcaption></figure><p>Now, one great thing is that we get a variety of options for the function trigger. This means that our <strong>Databricks workflow can be activated by any of these methods</strong>. For this exercise, follow these steps:</p><ul><li>Click on the <strong><em>“Azure Blob Storage”</em></strong> trigger.</li><li>Give the function a unique name and hit enter.</li><li>Click on <strong><em>“Create new local app setting”.</em></strong></li></ul><p>Your VS Code folder should now be populated with files from the created function. When the project is available in the editor, define the required environment variables inside local.settings.json:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b65457247f2cb155b03a3fbff96c5284/href">https://medium.com/media/b65457247f2cb155b03a3fbff96c5284/href</a></iframe><ul><li>&lt;your-storage-account-name&gt;_STORAGE: Will hold the connection string from your Azure storage account. To get the value, go to the Azure portal and select your storage account. Click on<strong><em>“Access Keys” </em></strong>in the left pane and copy the connection string.</li><li>DB_JOB_ID: Is the id of your Databricks job. After your job is created copy the id associated with it and paste it here.</li><li>DATABRICKS_HOST: Is the URL of your Databricks workspace. It should be similar to <em>“https://adb-xxxxxxxxx.xx.azuredatabricks.net/”</em></li><li>DATABRICKS_TOKEN: You need to create a Databricks Personal Access Token and insert the value here. This will allow the function to interact with your Databricks workspace.</li><li>Ignore the other parameters.</li></ul><p>After the parameters are set, we are going to implement the behavior of the function inside the __init__.py script:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ebd54170963de34eaacc553d2c8238c3/href">https://medium.com/media/ebd54170963de34eaacc553d2c8238c3/href</a></iframe><p>Things to note from this script:</p><ul><li>Databricks CLI is a tool made for the shell. However, this script is leveraging the fact that it is written in python and we can use those objects for the logic of this function. Include databricks-cli in the requirements.txt file.</li><li>The EnvironmentVariableConfigProvider is a class from the Databricks CLI and uses the DATABRICKS_HOST and DATABRICKS_TOKENenvironment variables to initialize the environment.</li><li>JobsService is also a class from the Datbaricks CLI, and allows you to create, run, list, and delete jobs. It is a very powerful tool.</li><li>The control flow of the code is designed to avoid multiple function executions to trigger many concurrent runs of the same job. There should be only one run at a time.</li></ul><p>After the code is complete we have to create the actual cloud resource and upload the configuration we just implemented. For that, under the Azure plugin, in the <strong><em>“Resources”</em></strong> pane click on <strong><em>“Create a Function App in Azure”</em></strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/593/1*TBmM1UhWCnSPWJVOND3TiQ.png" /><figcaption>Creating a function on the cloud. Screenshot by own authorship.</figcaption></figure><p>Moreover:</p><ul><li>Give the function a unique name and hit enter.</li><li>Click on the newest python runtime stack.</li><li>For this example, select East US as the region. In more realistic scenarios you should pick the region that best fits your business case.</li></ul><p>Upon the creation of the function. Right-click on the latter and select <strong><em>“Deploy to Function App”</em></strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/495/1*sGlLd2M9ALcP72fU_ruklg.png" /><figcaption>Deploying the code to the cloud function. Screenshot was taken by me.</figcaption></figure><p>This will upload the code to the cloud resource. To finalize, right-click under the <strong><em>“Application Settings”</em></strong> sub-menu inside your function and select <strong><em>“Upload Local Settings”</em></strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/488/1*tkvsDTVL89vvhzhpJdes1Q.png" /><figcaption>Uploading the variables to the cloud function. Screenshot was taken by the author.</figcaption></figure><p>It will set the local.settings.json contents as the environment variables of our function app. That is it! The Databricks workspace should be able to react to file uploads in your storage container and run the appropriate workflow.</p><h3>6. Testing the project</h3><p>Now that the infrastructure is ready, it is time to drop the sample dataset. The initial yellow trip data is too heavy to upload quickly. Furthermore, pick a small sample of records and upload it to the storage container:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xtj18AJCvwGS12TTbQLNCw.png" /><figcaption>Uploading a file to the storage container. Screenshot was taken by me :)</figcaption></figure><p>As soon as the file is loaded, the function will be triggered, and in turn, it will signal the Databricks job to run. This is a sample of the logs produced by the Azure function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zWXXfNYUTeo-CkYaz29crA.png" /><figcaption>Azure function run logs. Screenshot was taken by the author.</figcaption></figure><p>In addition, if you go to Databricks after a couple of minutes, the job will be finished:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AOsFSDawBmQEXWE6HQ3Wcw.png" /><figcaption>Databricks job runs. Screenshot of own authorship.</figcaption></figure><p>And the table should be populated with the yellow trip data records:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/721/1*ipURL91XXuwhjTHYFRH73w.png" /><figcaption>Querying the loaded data in the table. Screenshot was taken by me.</figcaption></figure><p>It works like magic!</p><h3>Additional Considerations</h3><p>The implementation of this project was done in Visual Studio Code and the Databricks workspace. However, if you need your infrastructure to be CI-CD or IaC friendly, you can always use the Azure resource manager templates and the Azure CLI to create the environment. Just extract the core concept of this article, and adapt it to your use case.</p><p>Additionally, test and tweak the behavior of the function and the Databricks job accordingly. This logic is not battle-tested for loading a large number of files at very small intervals. However, it should provide a starting point so that you can ingest data on-demand without the need for an indefinitely active cluster consuming resources and money while just listening for data.</p><p>Finally, the job is set to create a cluster in a cold start setting. This will add some minutes to the execution time. Consider creating compute pools so that clusters can use pre-warmed instances if you need your jobs to run as fast as possible.</p><p>Please clap and subscribe if you liked this content, and connect with me on my <a href="https://www.linkedin.com/in/hectorandresmv/">LinkedIn</a>. Cheers!👾</p><p><a href="https://github.com/hector6298">hector6298 - Overview</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cbf0bd99d8d6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/process-on-demand-data-without-idle-databricks-clusters-cbf0bd99d8d6">Process On-Demand Data without Idle Databricks Clusters</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a real-time dashboard for your IoT data]]></title>
            <link>https://hectormrejia.medium.com/building-a-real-time-dashboard-for-your-iot-data-d72edcfe4aee?source=rss-4f706205679a------2</link>
            <guid isPermaLink="false">https://medium.com/p/d72edcfe4aee</guid>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[dashboard]]></category>
            <category><![CDATA[iot]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Hector Andres Mejia Vallejo]]></dc:creator>
            <pubDate>Wed, 15 Jun 2022 14:39:55 GMT</pubDate>
            <atom:updated>2022-06-15T14:39:55.229Z</atom:updated>
            <content:encoded><![CDATA[<h4>Using Azure IoT Hub, IoT Edge, Machine Learning, and Bokeh</h4><p><em>Companion repo available. Read the intro below for the link!</em></p><p>You may face a use case where your data needs to be updated and visualized at a very high frequency. In addition, you would like to create a machine learning model to decide what to do on such high-frequency data, something a human cannot do. Well, with the help of the cloud and open-source tools we can build a framework for that. In this post, I will introduce a workflow and the cloud resources needed to build a pipeline that collects, enriches, and routes telemetry to both a storage sink and a streaming endpoint. This endpoint is used by a service that updates a real-time dashboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_QdcNq9_bu76dwll" /><figcaption>IoT is in the air. Photo by <a href="https://unsplash.com/@jorgedevs?utm_source=medium&amp;utm_medium=referral">Jorge Ramirez</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a>.</figcaption></figure><p>This post is based on a personal project for predictive maintenance using IoT telemetry. It has links on the text that will lead you to the corresponding code in each section. <strong><em>Check out this </em></strong><a href="https://github.com/hector6298/predictive_maintenance.git"><strong><em>link</em></strong></a><strong><em> for the root of the repository</em>.</strong> Note that it is only a proof of concept and is in no way production-ready.</p><h3>Index</h3><ul><li><a href="#6e87">Overview of the process in action</a></li><li><a href="#440e">Setting up the infrastructure</a></li><li><a href="#0099">Training the model in the cloud</a></li><li><a href="#2e5b">Creating and submitting the IoT Edge modules</a></li><li><a href="#949e">Important considerations</a></li></ul><h3><strong>Overview of the process in action</strong></h3><p>The flow of the pipeline in execution works as depicted:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*l6HPr5ni-O5UmtUi.png" /><figcaption>IoT to dashboard flow. Made with ❤ by me.</figcaption></figure><h4>IoT Edge</h4><p>It starts with a local server, where an <a href="https://docs.microsoft.com/en-us/azure/iot-edge/iot-edge-runtime?view=iotedge-2020-11">IoT Edge runtime</a> is installed, turning the server into an Edge device. IoT Edge allows us to easily deploy, manage, monitor docker containers or “modules”, and define communication routes among them.</p><p>These devices will have two modules deployed <strong>(click the links to see the code)</strong>:</p><ul><li><a href="https://github.com/hector6298/predictive_maintenance/blob/main/iot_edge/humid_telemetry/modules/send_telemetry/main.py">Telemetry reader</a></li><li><a href="https://github.com/hector6298/predictive_maintenance/blob/main/iot_edge/humid_telemetry/modules/ml_inference/main.py">Machine learning inference</a></li></ul><p>The sensor data is captured, sent to the ML module for inference, and streamed to the cloud using Azure IoT Hub. Once the code is defined, it has to be packaged into a Docker container and deployed to the devices. The deployment will be covered in the setup stages.</p><h4>IoT Hub</h4><p>The <a href="https://docs.microsoft.com/en-us/azure/iot-hub/iot-concepts-and-iot-hub">IoT Hub</a> resource enriches the data with the timestamp and device ID. Then, it sends the stream to a file storage-based data lake and to the built-in endpoint that is <a href="https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-about">Event Hubs</a> compatible. This endpoint will be used for the real-time dashboard.</p><h4>Real-time dashboard</h4><p>From the dashboarding service, the stream is captured using a consumer client and updates a data structure that feeds <a href="https://docs.bokeh.org/en/latest/">Bokeh</a> visualizations. In addition, there is a dog that closes the client when the application is being shut down, for a graceful exit. <strong>Check out the code for the visualization server </strong><a href="https://github.com/hector6298/predictive_maintenance/blob/main/dashboard/bokeh_dashboard.py"><strong>here</strong></a><strong>.</strong></p><h4>Setup stages</h4><p>Before our pipeline is operational, we need to set up all the required infrastructure. That is: the ML services, compute, container registries, message routings, message enrichments, and other resource interactions. This is done by three support DAGs powered by Apache Airflow:</p><ul><li>Infrastructure</li><li>Machine Learning training</li><li>IoT Edge Modules deployment to edge devices</li></ul><p>The setup is explained in the following sections. Here is a recording of the live dashboard with a single sensor:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/526/1*8G6XTYI2SS4enrN4V2q6CA.gif" /><figcaption>Screen recording of the dashboard in execution. Made with ❤ by me.</figcaption></figure><h3>Setting up the infrastructure</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*inrS4LOEGL3bDdljI8EQRg.png" /><figcaption>Infrastructure provisioning. Made with ❤ by me.</figcaption></figure><p>In this section, we deploy all the necessary cloud resources and compute to support our framework. <strong>Check </strong><a href="https://github.com/hector6298/predictive_maintenance/blob/main/airflow/dags/infrastructure_dag.py"><strong>here</strong></a><strong> to see the code for the infrastructure DAG</strong>. There are two main groups of infrastructure:</p><h4>Machine Learning</h4><p>Once this flow is triggered, we create an <a href="https://docs.microsoft.com/en-us/azure/machine-learning/overview-what-is-azure-machine-learning">Azure Machine Learning</a> workspace, a compute cluster to train our future models, and a container registry to build the inference micro-service. If these resources already exist, we just get them for the next steps.</p><h4>IoT resources</h4><p>This process executes in parallel to the creation of the ML resources. First, an IoT Hub is created. It will receive, enrich, and route our sensor telemetry. We create an Azure storage account, and a file storage service that will be used as a sink for our data. The <a href="https://docs.microsoft.com/en-us/azure/iot-hub/iot-hub-devguide-messages-d2c">sink</a> and <a href="https://docs.microsoft.com/en-us/azure/iot-hub/iot-hub-message-enrichments-overview">message enrichments</a> must be defined in IoT Hub, and our setup pipeline takes care of that in the final step of this stage.</p><h3>Training the model in the cloud</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NgEByz1rMlqz6f3YBBkTmw.png" /><figcaption>Machine Learning services flow. Made with ❤ by me.</figcaption></figure><p>At this stage, assuming a dataset is already prepared (see the important considerations section), it gets uploaded to a datastore. Then, we define the parameters for the training on the cloud. These parameters are mainly the directory of the training script, the compute resource object, and a conda environment object. All of these are implemented with the help of the <a href="https://docs.microsoft.com/en-us/python/api/overview/azure/ml/?view=azure-ml-py">Azure ML SDK for python</a>. Furthermore, we submit the ML experiment run and register a new model version.</p><p>Finally, the orchestrator downloads the latest model version on the local server. <strong>Check </strong><a href="https://github.com/hector6298/predictive_maintenance/blob/main/airflow/dags/training_dag.py"><strong>here</strong></a><strong> to see the training DAG</strong>. This model will be used to create the inference service in the next stage.</p><h3>Creating and submitting the IoT Edge modules</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MffTmyaTlFyQRjFn2yyj-g.png" /><figcaption>IoT Edge modules flow. Made with ❤ by me.</figcaption></figure><p>This stage focuses on deploying two IoT Edge modules: telemetry generation using the IoT sensor and machine learning inference with the downloaded model from the previous stage. The behavior for the two modules is already defined in python scripts. <strong>Check </strong><a href="https://github.com/hector6298/predictive_maintenance/blob/main/airflow/dags/iotedge_dag.py"><strong>here</strong></a><strong> to see the edge DAG.</strong></p><p>First, the orchestrator builds, tags, and pushes the images to the registry in parallel. Furthermore, it picks an IoT Edge deployment template and replaces the placeholders with specific values set as arguments in the orchestrator. These values are registry credentials and image locations. Finally, they are deployed to IoT devices.</p><h3>Important considerations</h3><h4>Training dataset compilation</h4><p>This step is not contemplated in the initial project. Ideally, you should use the telemetry held on the data lake to continuously build training datasets. This operation is backed by proper machine learning model observability.</p><h4>ML observability</h4><p>Model deployment is not the end of the hustle. You should check for shifts in the distribution of the input features, predictions, and also the labels produced for each training iteration. This way, you get ahead of model failures, because observability enables root cause analysis, and helps minimize model degradation.</p><h4>If my telemetry is massive, will the dashboard service break?</h4><p>This part was not properly treated since this framework is currently suited to support a handful of devices. However, to handle streaming data in very large quantities, we could set a <strong>key-value NoSQL database</strong> that is not <a href="https://en.wikipedia.org/wiki/ACID">ACID</a>, but <a href="https://www.dataversity.net/what-is-base/">BASE</a> compliant and with both input and output streaming capabilities. These databases make the data available as soon as it comes, meaning that replicas are eventually consistent, but enable the handling of data with very little latency.</p><p>The data could use a serverless resource to capture the data from the IoT Hub endpoint and send it to the database at scale. Moreover, we can filter and stream just the data that we need to the dashboarding service. In this way, we prevent the latter from being overwhelmed by too much data.</p><p>Thank you for taking the time to read my post. Please, leave a comment or clap the article if it was useful to you so I can improve on my writing.</p><p>I will see you next time!</p><p><a href="https://github.com/hector6298">hector6298 - Overview</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d72edcfe4aee" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Use these methods to orchestrate your pipelines]]></title>
            <link>https://hectormrejia.medium.com/use-these-methods-to-orchestrate-your-pipelines-2b8b614285ba?source=rss-4f706205679a------2</link>
            <guid isPermaLink="false">https://medium.com/p/2b8b614285ba</guid>
            <category><![CDATA[workflow]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-governance]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <dc:creator><![CDATA[Hector Andres Mejia Vallejo]]></dc:creator>
            <pubDate>Tue, 31 May 2022 22:46:26 GMT</pubDate>
            <atom:updated>2022-05-31T22:46:26.491Z</atom:updated>
            <content:encoded><![CDATA[<h4>Build better workflows and ensure data quality in Databricks</h4><p>This post builds on top of my <a href="https://hectormrejia.medium.com/save-time-by-building-a-continuous-training-pipeline-for-ml-part-1-799e95d7c0ec">previous article</a> on continuous training pipelines, but can also be read on its own. Given the components of a training pipeline and presented the stack for its implementation, for instance: MLflow, Delta Lake, hyperopt, and scikit-learn, we are going to integrate and fully automate them. For that, I will introduce some new and exciting Databricks features. Furthermore, at the end of the article, I discuss airflow as an alternative for more general workflows, not restricted to Databricks.</p><p>Let’s illustrate the components of our training pipeline:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lgyNOE0nfFeTx6fS.png" /><figcaption>Continuous training pipeline. Made with ❤ by me.</figcaption></figure><p>For the automation of the data ETL, we are going to introduce Delta Live Tables (DLT) and for the machine learning components, we are using Databricks workflows.</p><h3>Data ETL with Delta Live Tables</h3><p>DLT is a very useful feature in the Databricks ecosystem because it allows us to integrate all of our data ETL components in a Directed Acyclic Graph (DAG) and execute them periodically or continuously. In addition, it lets us:</p><ul><li>Track data lineage</li><li>Monitor data quality after implementing checks</li><li>Define behavior if data does not pass the checks</li></ul><p>It is a great tool to promote good data governance practices very easily. You can read more about DLT <a href="https://docs.microsoft.com/en-us/azure/databricks/data-engineering/delta-live-tables/delta-live-tables-quickstart">here</a>. Let’s see a short snippet on how to declare DLT tables using python:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6f959d6a05a05a1aa42ec3dde65d74e4/href">https://medium.com/media/6f959d6a05a05a1aa42ec3dde65d74e4/href</a></iframe><p>For a table to be declared, we must define a function with the decorator @dlt.table and its name will be also the name of the table. The same logic applies to views. Note that the @expect decorator is used to define a data quality check called expectation. We can create as many expectations as we need. After the pipeline is established, Databricks will render the DAG on the UI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UPqhok5PAJMfAuoirMEG7A.png" /><figcaption>A render of a sample data pipeline DAG in DLT.</figcaption></figure><p>You can see that Databricks presents metrics for the data that does not pass the checks, and shows the state of each stage in the DAG. This is how we build automated data pipelines and ensure data quality.</p><h3>Machine Learning Components with Databricks Workflows</h3><p>Recently, Databricks introduced workflows. They replace the old Databricks jobs, allowing developers to create multiple tasks and declaring dependencies among the tasks. In addition, these workflows can be executed periodically or on-demand.</p><p>In the context of our training pipeline, each component: model training, registry, and publishing has its own notebook. Each notebook will represent a single task in the workflow.</p><p>Moreover, a workflow can be created using the Databricks Jobs UI. However, to make it versionable, we can use JSON configuration files to declare the tasks, dependencies, and task arguments:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5aa13fa05670488179afe661c34e7062/href">https://medium.com/media/5aa13fa05670488179afe661c34e7062/href</a></iframe><p>and then use Databricks <a href="https://docs.databricks.com/dev-tools/cli/jobs-cli.html#create-a-job">Jobs CLI</a> 2.1 to create the workflow:</p><pre>databricks jobs create --json-file db_workflow.json</pre><p>It will return something like:</p><pre>{ &quot;job_id&quot;: 246 } // 246 is just an example</pre><p>Then you can execute it by entering:</p><pre>databricks jobs run-now --job-id 246</pre><p>Notice in the snippet that the first component, data_etl is the Delta Live Tables pipeline that we previously defined. You can<strong> execute a DLT pipeline as a task inside a workflow</strong>, which is very convenient. You can read more about Databricks workflows <a href="https://docs.databricks.com/data-engineering/jobs/jobs.html">here</a> and how to structure a JSON configuration <a href="https://docs.databricks.com/data-engineering/jobs/jobs-api-updates.html">here</a>.</p><p>Now if we go to the UI, you will see your training pipeline rendered and you can track the state of each task. <strong>You may also start to wonder</strong> if we can build pipelines using JSON and update them using the Jobs CLI, then we can use some CI/CD tool to test the pipeline and update it automatically. Yes, we can implement all sorts of automation, and use it to fit our data needs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/675/1*p5J-igMMk3HNp5ZavPM83A.png" /><figcaption>A sample workflow on Databricks.</figcaption></figure><h3>Alternative: Apache Airflow as the orchestrator</h3><p>Apache Airflow is an open-source technology that lets you build data pipelines using the same DAG concept easily using a python interface. An immediate benefit of using python code for your pipelines is that it is versionable. You can read more about apache airflow and a quick example <a href="https://airflow.apache.org/docs/apache-airflow/stable/tutorial.html">here</a>.</p><p>This option applies to more general implementation of data pipelines that use other technologies. However, it also has a special DatabricksOperator to include databricks tasks inside Airflow. See <a href="https://databricks.com/blog/2017/07/19/integrating-apache-airflow-with-databricks.html">here</a> for more information.</p><p>In addition, airflow implements extra <a href="https://www.astronomer.io/guides/airflow-sql-data-quality-tutorial/">SQL operators</a> to ensure data quality, if you have other relational databases in which you want to maximize observability and minimize data downtime.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AgM5huuOIA5SpEHHp0oVzg.png" /><figcaption>An example render of a DAG in airflow.</figcaption></figure><p>I hope this article can be useful to you. Thank you for taking the time to read it. Please, give me some support by clapping the article or leaving a constructive comment. What did I miss? What can I do better?</p><p>See you next time!</p><p><a href="https://github.com/hector6298">hector6298 - Overview</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2b8b614285ba" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learnings from my first two years in the tech industry]]></title>
            <link>https://hectormrejia.medium.com/learnings-from-my-first-two-years-in-the-tech-industry-dc62cd1d0031?source=rss-4f706205679a------2</link>
            <guid isPermaLink="false">https://medium.com/p/dc62cd1d0031</guid>
            <category><![CDATA[software]]></category>
            <category><![CDATA[careers]]></category>
            <category><![CDATA[learning]]></category>
            <category><![CDATA[experience]]></category>
            <dc:creator><![CDATA[Hector Andres Mejia Vallejo]]></dc:creator>
            <pubDate>Sun, 01 May 2022 20:54:56 GMT</pubDate>
            <atom:updated>2022-05-01T20:54:56.825Z</atom:updated>
            <content:encoded><![CDATA[<h4>Experiences from a Latin America data engineer</h4><p>After writing some blog posts about the cloud, machine learning, and engineering, I decided that it was a good time to express my thoughts and feelings, now that I am leaving the very first company that allowed me to work in the software and data industry. All of these experiences are condensed into eight topics, some of them are related, some are not. But, remember that this is only my opinion and this article is not technical at all.</p><p>let’s begin!</p><h4>1. Keep your LinkedIn sharp and interact often</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TiP0ox8cNgJUPuQg" /><figcaption>Photo by <a href="https://unsplash.com/@rswebsols?utm_source=medium&amp;utm_medium=referral">Souvik Banerjee</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The job market in Latin America is tough. The traditional ways of sending a physical resume and in-person interviews limit your capacity of applying to jobs. On top of that, as of 2022 in Ecuador, only 32% of the people have formal jobs, and 5% of job seekers won’t get them [1]. Knowing this, I have learned that LinkedIn is truly a powerful tool. It is where I got my first and my new current job without ever leaving my bedroom. You should aim to update your profile with the courses you completed, your projects, your achievements, and post content regularly. Part of the reason I have this blog is so that I can share on LinkedIn every new story that I have to tell and in turn, I get more visibility. Around the time I post new content on LinkedIn is when recruiters reach out to me the most, and that leads me to the second topic.</p><h4>2. Never Skip a recruiter’s message</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_j7rIoU73MrHJ4P8" /><figcaption>Photo by <a href="https://unsplash.com/@pradasg?utm_source=medium&amp;utm_medium=referral">Pradamas Gifarry</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>You never know where your next job offer is going to be, so reply to every recruiter that reaches out to you. I didn’t know this until recently. I was in the final steps of an application process for a data scientist position in a financial institution when I received a message from a recruiter. I remember thinking that I wouldn’t even bother because replying to those messages did not work out in the past and I was close to getting the other job anyway. Fast forward, the bank wanted a data scientist that could stay long term and work in the office. I didn’t go through because that would delay my desire to go to grad school abroad, and in a turn of events, I replied to the LinkedIn recruiter a week after, because at this point why not? right?</p><p>And I got the job! It was a remote position and the salary was considerably higher than the bank wanted to offer me. So, there you have it. Even if you are not currently open to new positions, politely saying no to them tells a lot about you. You can even expand your network if you connect with them, leaving the door for new opportunities in the future. Again, you never know.</p><h4>3. You can contribute to the culture of your team, but you won’t change it</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ptQ-V7smFiqMPOWX" /><figcaption>Photo by <a href="https://unsplash.com/@cwmonty?utm_source=medium&amp;utm_medium=referral">Chris Montgomery</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>I remember my first day at work in my first company, I was so motivated. Over a year and a half, I worked as much as I could and took courses to contribute to the tech stack of the team, which was not mature at all I must say. I even hosted several team learnings in the hopes of changing the culture of ever-increasing technical debt and the practice of non-scalable implementations of data pipelines. My teammates and I failed on that, and it was always the same phrase:</p><blockquote>Tenemos que entregar el proyecto ahora. Vamos con lo que tengamos y despues mejoramos.</blockquote><p>That means let’s go with what we have and we will improve later. The thing is that the “later” never came because it was a constant race for deadlines without the required resources for a good quality product. Time and time again, decision-makers would not listen to our suggestions, and I know we have deadlines! but we need to aim over the long term. Code maintainability and pipeline scalability keep your developers happy, I guess…</p><p>In the end, I learned that is best to analyze and know a company as much as you can, before working there.</p><h4>4. Work towards your goals and your learning process, not for the company</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*swdNu407Xs_b5pXn" /><figcaption>Photo by <a href="https://unsplash.com/@craftedbygc?utm_source=medium&amp;utm_medium=referral">Green Chameleon</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>I think this is a very important topic. I do feel proud of being part of a company and knowing that I am contributing to their success in a substantial way. That being with everyday coding, team learnings, or my initiatives. However, what I care about the most is my personal growth. I think about all my activities in terms of what I can learn from them. Take advantage of education budgets for continuous learning, get involved in challenging projects if you can. I have never understood why would anyone work till very late when they are not asked for it when extra hours are often not paid and our leads only reward work with more work, not recognition, not more money, not more courses or opportunities.</p><p>Then again, work as smart as you can, and be the guy that brings solutions, but think always about yourself.</p><h4>5. Do not miss out on life. Remote work is a blessing</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8Yn7iKwF3X0zN1s4" /><figcaption>Photo by <a href="https://unsplash.com/@mickeyoneil?utm_source=medium&amp;utm_medium=referral">Mickey O&#39;neil</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In my first year working I was always waiting for the day that the company says: “let’s get to the office again”. In my home, I felt very bored, alone, and I wanted to talk to people, maybe hang out with them after the office. I didn’t know at the time, but they would never go back to the office as they saw remote work as an opportunity to hire people from other companies and relieve themselves from the costs of having an office. Why didn’t I see it in the same way at the time? I am not constrained to a physical place anymore. I can go wherever I want, as long as I can secure an internet connection to do my job. For months I have been sitting in my bedroom waiting for my life to just happen by itself. Now, I have a very different perspective, I visit my friends from other cities, and fly to the US to see my family and girlfriend. I am even planning on visiting other nearby countries in Latam, with the promise of meeting new people and different cultures. So, if you are young, do not think of remote work in terms of isolation, because it is quite the opposite. It will open the door to the world!</p><h4>6. You are not constrained to the wage market of your country anymore</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aiM_eMpOaq_xNC0n" /><figcaption>Photo by <a href="https://unsplash.com/@andretaissin?utm_source=medium&amp;utm_medium=referral">Andre Taissin</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Let me start by stating the following: COVID-19 was a tragedy. Families were devastated, people were left with sequels from the disease, lockdowns, worldwide crisis, and shortages. But, it posed a great opportunity for software professionals in Latin America. Many tech companies are starting to realize our great potential and recruiting fresh new minds for remote work. The great thing about this is that these foreign companies offer way more attractive compensation packages than what the regional market offers and they value our work and opinion. However, for us to have that opportunity we have to be competitive, and that is why I work so hard on myself.</p><p>So yes, if you are a developer, take advantage of the situation, because now more than ever, it is easier to expand your horizons and work for foreign companies.</p><h4>7. Never get comfortable</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8vZkLBmuURVJUIBn" /><figcaption>Photo by <a href="https://unsplash.com/@isaacmsmith?utm_source=medium&amp;utm_medium=referral">Isaac Smith</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>I believe that once you feel comfortable with your job, with the value that you produce, or work because you have to, you are on track to stagnation. As a young developer, I think we should embrace this sort of uncomfortable feeling of walking into the unknown.</p><p>Get a new skill, apply to jobs just to see where you stand and what your value is, do a side project, side hustle, whatever. I like to leverage the budget for education that the company provides me, and in my free time after the gym, work on my side project which I would love to turn into a business. You see, Latin America is behind in terms of technology. But where others see problems, you could see that as an opportunity. The market is not as saturated as in more developed countries, so I am trying to find my opportunity, working little by little in my free time. I am convinced it will be all worth it.</p><h4>8. Sometimes, not being productive can be productive</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*P-T-pCUux2hikzgj" /><figcaption>Yeah, chill bro every now and then. Photo by <a href="https://unsplash.com/@somechrisguy?utm_source=medium&amp;utm_medium=referral">Chris Thompson</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Yes, we need to work smart, be constant and all, but let’s face it, we are human beings. As such, too much work can lead to burnout, decreasing your focus, your ability to have great ideas, and your overall productivity.</p><p>I find that playing video games for very short periods of time, or going out with my family and friends on a weekend afternoon is enough to recharge my batteries. I am not going to lie to you, this is also a reminder for myself since my brain is often thinking about returning to my desk and continuing writing code, which makes me feel anxious. So I am constantly working on feeling ok with not being productive all the time, but I will never let that determination of working toward building my career go away.</p><p>I feel that after writing this, I can start my new career chapter with renewed energies, but I am grateful for all my experiences so far. Thank you for taking the time to read my articles. See you at the next one!</p><p><a href="https://github.com/hector6298">hector6298 - Overview</a></p><h4>References</h4><p>[1] National Survey on employment, sub-employment, and unemployment. National Institute for statistics, 2022. Retrieved from: <a href="https://www.ecuadorencifras.gob.ec/estadisticas-laborales-marzo-2022/">https://www.ecuadorencifras.gob.ec/estadisticas-laborales-marzo-2022/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dc62cd1d0031" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scalable self-hosted runner system for GitHub actions]]></title>
            <link>https://hectormrejia.medium.com/scalable-self-hosted-runner-system-for-github-actions-509052905817?source=rss-4f706205679a------2</link>
            <guid isPermaLink="false">https://medium.com/p/509052905817</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[github-actions]]></category>
            <category><![CDATA[self-hosted-runner]]></category>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <dc:creator><![CDATA[Hector Andres Mejia Vallejo]]></dc:creator>
            <pubDate>Wed, 30 Mar 2022 05:27:18 GMT</pubDate>
            <atom:updated>2022-03-30T05:27:18.242Z</atom:updated>
            <content:encoded><![CDATA[<h4>An updated step-by-step guide on how to deploy a system capable of creating self-hosted runners on demand!</h4><p>Hello everyone! This article is intended for organizations that develop on private repositories and the minutes available from GitHub are not enough for their CI/CD needs. It is just a compilation of the steps performed on the GitHub office hours workshop video. You can see the whole playlist <a href="https://www.youtube.com/playlist?list=PL0lo9MOBetEEk9gIFox8EbCf1Co_4ppIO">here</a> and the companion repository <a href="https://github.com/Link-/actions-at-scale-ghoh">here</a>. However, I wrote this article for those who do not want to watch the entire three videos, which are a bit lengthy. Just note that the platform being used is Microsoft Azure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wIHw9I7F6NRZ7vLQ" /><figcaption>Just a beautiful city sunset before starting with the tutorial. Photo by <a href="https://unsplash.com/@nasimgs?utm_source=medium&amp;utm_medium=referral">Mohammed Nasim</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a>.</figcaption></figure><p>let’s begin!</p><h3>Organize deployments</h3><p>We are going to create a resource group called GitHubActionsRunners. This resource group will contain all the required Azure services to deploy our autoscaling solution for GitHub runners.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*TR9ln_m2NHaXW8WPo93WOQ.png" /><figcaption>The Azure resource group.</figcaption></figure><p>You should have Azure CLI installed, otherwise check here for how to do that. First, log into your azure account:</p><pre>az login</pre><p>Now, on your shell run the following commands:</p><pre>az account set --subscription &lt;subscription&gt;</pre><p>Create the resource group:</p><pre>az group create --name GithubActionsRunners --location &lt;region&gt;</pre><h3>Create Azure kubernetes Cluster</h3><p>As my GitHub actions workloads are not very resource intensive, I will stick to the predefined Standard_DS2node type for my worker pool. With this command we can create an Azure Kubernetes Services (AKS) cluster:</p><pre>az aks create \<br>--resource-group GithubActionsRunners\<br>--name GithubActionsRunnersAKS \<br>--enable-addons monitoring \<br>--node-count 3 \<br>--node-vm-size Standard_DS2 \<br>--generate-ssh-keys</pre><h3>Connecting to AKS cluster</h3><p>Enter the following command:</p><pre>az aks get-credentials --resource-group GithubActionsRunners \<br>                       --name GithubActionsRunnersAKS</pre><p>This will enable you to use kubectl commands on your AKS cluster. Make sure you have it installed. If not, check <a href="https://kubernetes.io/docs/tasks/tools/">this</a> out.</p><h3>Provisioning the networking infrastructure</h3><p>We need to create an Application Gateway, assign a public Ip and connect it with our AKS cluster. This way, we can create a webhook server to allow the controller to spin-up our future runners on-demand.</p><p>First, create a public IP service:</p><pre>az network public-ip create \<br>               --resource-group GithubActionsRunners \<br>               --name APGWPublicIp \<br>               --allocation-method Static \<br>               --sku Standard</pre><p>Then, create a vnet for the application gateway. This vnet will be different from that of the AKS cluster to avoid any overlap on the IP assignment.</p><pre>az network vnet create --name appgwVNet \<br>                       --resource-group GithubActionsRunners \<br>                       --address-prefix 11.0.0.0/8 \<br>                       --subnet-name appgwSubnet \<br>                       --subnet-prefix 11.1.0.0/16</pre><p>Now, create the application gateway itself:</p><pre>az network application-gateway create \<br>--resource-group GithubActionsRunners \<br>--name GithubActionsRunnersAPGW \<br>--location &lt;region&gt; \<br>--sku Standard_v2 \<br>--public-ip-address APGWPublicIp \<br>--vnet-name appgwVNet \<br>subnet appgwSubnet</pre><p>fetch its ID by running:</p><pre>APPGW_ID=$(az network application-gateway show — resource-group GithubActionsRunners — name GithubActionsRunnersAPGW — query “id” — output tsv)</pre><p>and attach it to the AKS cluster to allow ingress:</p><pre>az aks enable-addons --resource-group GithubActionsRunners \<br>                     --name GithubActionsRunnersAKS \<br>                     --addons ingress-appgw \<br>                     --appgw-id $APPGW_ID</pre><p>Furthermore, type the following commands to get the required information to create the peering in both directions of the gateway and the AKS.</p><pre>NODERESOURCEGROUP=$(az aks show — name GithubActionsRunnersAKS — resource-group GithubActionsRunners — query “nodeResourceGroup” — output tsv)</pre><pre>AKSVNETNAME=$(az network vnet list — resource-group $NODERESOURCEGROUP — query “[0].name” — output tsv)</pre><pre>AKSVNETID=$(az network vnet show — name $AKSVNETNAME — resource-group $NODERESOURCEGROUP — query “id” — output tsv)</pre><pre>APPGWVNETID=$(az network vnet show — name appgwVNet — resource-group GithubActionsRunners — query “id” — output tsv)</pre><ul><li>Application gateway to AKS peering:</li></ul><pre>az network vnet peering create --name AppGWtoAKSVnetPeering \<br>                    --resource-group GithubActionsRunners \<br>                    --vnet-name appgwVNet \<br>                    --remote-vnet $AKSVNETID \<br>                    --allow-vnet-access</pre><ul><li>AKS to application gateway peering:</li></ul><pre>az network vnet peering create --name AKStoAppGWVnetPeering \<br>                       --resource-group $NODERESOURCEGROUP \<br>                       --vnet-name $AKSVNETNAME \<br>                       --remote-vnet $APPGWVNETID \<br>                       --allow-vnet-access</pre><p>Finally, specify a DNS name for the public IP address. Go to your public IP service in your Azure portal and inside configuration, give a DNS name like so:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/1*U2oULMlGvgaXUkmBMFi-Rw.png" /><figcaption>Capture of the Public Ip settings.</figcaption></figure><h3>Installing cert-manager on AKS cluster</h3><p>According to the controller’s documentation, the system uses cert-manager for the certificate management of admission webhook. This means that we have to install it before configuring the controller on our cluster. Run the following:</p><pre>kubectl apply -f <a href="https://github.com/cert-manager/cert-manager/releases/download/v1.7.1/cert-manager.yaml">https://github.com/cert-manager/cert-manager/releases/download/v1.7.1/cert-manager.yaml</a> -n cert-manager</pre><p>This will install cert-manager under the cert-manager namespace. We should verify that the installation was successful by checking the deployments:</p><pre>kubectl get pods --namespace cert-manager</pre><p>Wait for a few seconds for the deployments to finish. We should have an output similar to this:</p><pre>NAME                                       READY   STATUS    RESTARTS   AGE <br>cert-manager-5c6866597-zw7kh               1/1     Running   0          2m <br>cert-manager-cainjector-577f6d9fd7-tr77l   1/1     Running   0          2m <br>cert-manager-webhook-787858fcdb-nlzsq      1/1     Running   0          2m</pre><p>Furthermore, create an Issuer to test that the webhook works. Create a YAML called test-resources.yaml and paste this:</p><pre>apiVersion: v1 <br>kind: Namespace <br>metadata: <br>  name: cert-manager-test <br>--- <br>apiVersion: cert-manager.io/v1 <br>kind: Issuer <br>metadata: <br>  name: test-selfsigned <br>  namespace: cert-manager-test <br>spec: <br>  selfSigned: {} <br>--- <br>apiVersion: cert-manager.io/v1 <br>kind: Certificate <br>metadata: <br>  name: selfsigned-cert <br>  namespace: cert-manager-test <br>spec: <br>  dnsNames: <br>    - example.com <br>  secretName: selfsigned-cert-tls <br>  issuerRef: <br>    name: test-selfsigned </pre><p>Create the test resources:</p><pre>kubectl apply -f test-resources.yaml</pre><p>Now check the status of the newly created certificate:</p><pre>kubectl describe certificate -n cert-manager-test</pre><p>We should have an output similar to this:</p><pre>... <br>Spec: <br>  Common Name:  example.com <br>  Issuer Ref: <br>    Name:       test-selfsigned <br>  Secret Name:  selfsigned-cert-tls <br>Status: <br>  Conditions: <br>    Last Transition Time:  2019-01-29T17:34:30Z <br>    Message:               Certificate is up to date and has not expired <br>    Reason:                Ready <br>    Status:                True <br>    Type:                  Ready <br>  Not After:               2019-04-29T17:34:29Z <br>Events: <br>  Type    Reason      Age   From          Message <br>  ----    ------      ----  ----          ------- <br>  Normal  CertIssued  4s    cert-manager  Certificate issued successfully</pre><p>Moreover, clean up the test resources:</p><pre>kubectl delete -f test-resources.yaml</pre><p>We have verified that cert-manager works!</p><h3>Create ingress controller and production issuer</h3><p>Now that the certificate manager works, we need to create a real Issuer and ingress controller. Create a YAML manifest called ingress-tls.yaml with the following content (Replace hosts with your hostname):</p><pre>apiVersion: networking.k8s.io/v1 <br>kind: Ingress <br>metadata: <br>  name: ingress-main <br>  annotations: <br>    kubernetes.io/ingress.class: azure/application-gateway <br>    cert-manager.io/cluster-issuer: letsencrypt-prod <br>    appgw.ingress.kubernetes.io/ssl-redirect: &quot;true&quot; <br>spec: <br>  tls: <br>    - hosts: <br>      - &lt;your-hostname-defined-in-APGWPublicIp&gt;<br>      secretName: tlsingress <br>  rules: <br>  - host: &lt;your-hostname-defined-in-APGWPublicIp&gt;<br>    http: <br>      paths: <br>      - path: / <br>        pathType: Prefix <br>        backend: <br>          service: <br>            name: test-app <br>            port: <br>              number: 80</pre><p>Now, let’s define a real Issuer by creating the following YAML, call it cluster-issuer-prod.yaml:</p><pre>apiVersion: cert-manager.io/v1 <br>kind: ClusterIssuer <br>metadata: <br>  name: letsencrypt-prod <br>spec: <br>  acme: <br>    server: <a href="https://acme-v02.api.letsencrypt.org/directory">https://acme-v02.api.letsencrypt.org/directory</a> <br>    email: &lt;your-email&gt; <br>    privateKeySecretRef: <br>      name: tlssprod <br>    solvers: <br>    - http01: <br>        ingress: <br>          name: ingress-main</pre><p><strong>Do not apply these resources just yet!</strong> Can you see that there is a test-app assigned to the route &quot;\&quot;? We will deploy a test app to ensure that the ingress is working correctly before moving on to installing the controller. Create a YAML called test-app.yaml and paste this into the file:</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: test-app<br>  namespace: default<br>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: test-app<br>  template:<br>    metadata:<br>      labels:<br>        app: test-app<br>    spec:<br>      containers:<br>      - name: test-app<br>        image: mcr.microsoft.com/azuredocs/aks-helloworld:v1<br>        ports:<br>        - containerPort: 80<br>          protocol: TCP<br>        env:<br>        - name: TITLE<br>          value: &quot;AKS Ingress test is successful!&quot;<br>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: test-app<br>  namespace: default<br>spec:<br>  ports:<br>  - protocol: TCP<br>    port: 80<br>    targetPort: 80<br>  selector:<br>    app: test-app</pre><p>deploy the test app on your Kubernetes:</p><pre>kubectl apply -f test-app.yaml -n default</pre><p>Then, apply the ingress controller and the issuer to AKS using:</p><pre>kubectl apply -f ingress-tls.yaml<br>kubectl apply -f cluster-issuer-prod.yaml</pre><p>Now, check the ingress:</p><pre>kubectl get ingress</pre><p>You should be able to see your hostname and public IP address. If you go to your browser and navigate to your hostname URL, you should get this screen:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/1*Iu7Ylx6ocsxZJ1buF-1FmA.png" /><figcaption>Successful test!</figcaption></figure><p>This means that we are on the right track and we can move on to configure our webhook and controller!</p><h3>Configuring GitHub App</h3><p>This step needs to be done prior to installing the actions runner controller on your kubernetes, because there are some secrets needed prior to the installation. First, we need to install GitHub CLI using <a href="https://www.anaconda.com/">conda</a>:</p><pre>conda install gh --channel conda-forge</pre><p>Now, let’s install a <a href="https://cli.github.com/">GitHub CLI</a> extension for token management called GH Token. We need a dependency jq to be installed first:</p><pre>wget <a href="https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64">https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64</a><br>sudo chmod +x ./jq-linux64<br>mv ./jq-linux64 jq<br>sudo mv ./jq /usr/bin</pre><p>With that set up, install the GH Token extension:</p><pre>gh auth login<br>gh extensions install Link-/gh-token</pre><p>We also need to install JWT-CLI, which is a command line tool that will help us work with JSON Web Tokens:</p><pre>wget <a href="https://github.com/mike-engel/jwt-cli/releases/download/4.0.0/jwt-linux.tar.gz">https://github.com/mike-engel/jwt-cli/releases/download/4.0.0/jwt-linux.tar.gz</a><br>tar xvf jwt-linux.tar.gz<br>sudo mv jwt /usr/bin/<br>rm -rf ./jwt-linux.tar.gz</pre><p>Next, on GitHub, go to your <strong>Organization’s settings</strong>, select <strong>developer settings</strong>, then go to<strong> GitHub Apps</strong>, and click create a new GitHub App. Give it the following information:</p><ul><li>Name for the App.</li><li>Homepage URL of your organization.</li><li>Uncheck Webhook <strong>just for now</strong>. We are going to enable that later.</li><li>Permissions <br> - Actions: Read-only<br> - Contents: Read-only<br> - Metadata: Read-only<br> - Self-hosted runners: Read and Write</li></ul><p>Then select “Create GitHub App”. You will then be redirected to the App Settings and you should see a “Install App” tab on the left side. Select it, and press the button “Install”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Re4yI8L7UybTk1fQYKYt9Q.png" /><figcaption>“Install the app” capture.</figcaption></figure><p>After the app is installed, in the “About” tab, copy the App Id:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PhcgG8plD6FfOnpQ7HWShw.png" /><figcaption>Capture of the App’s Settings.</figcaption></figure><p>Scroll down and generate a new private key:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/806/1*6-lEpIbHILf0vKBD5Ul0fw.png" /><figcaption>Private key UI section.</figcaption></figure><p>Save it on a secure location in your computer, then use both the contents of the <strong>private key</strong> and the <strong>installation ID</strong> with the gh token command to get the <strong>installation ID</strong>:</p><pre>gh token installations -i &lt;App-ID&gt; -k &lt;path-to-private-key&gt;</pre><p>If you encounter a message like this:</p><pre>{<br>  &quot;message&quot;: &quot;&#39;Expiration time&#39; claim (&#39;exp&#39;) is too far in the future&quot;,<br>  &quot;documentation_url&quot;: &quot;<a href="https://docs.github.com/rest">https://docs.github.com/rest</a>&quot;<br>}</pre><p>It is probably because of the difference between the GitHub server time and your local time. In that case, you might want to try this command:</p><pre>sudo date +%T -s $(curl -sI <a href="https://api.github.com/">https://api.github.com</a> | grep -Fi &quot;date&quot; | awk &#39;{ print $6 }&#39;) -u</pre><p>and re-run gh tokenagain. Get the value from the &quot;id&quot;key in the JSON output. We are finally ready to install the actions-runner-controller!</p><h3>Installing actions-runner-controller using HELM</h3><p>Refer to this <a href="https://helm.sh/docs/intro/install/">documentation</a> on how to install HELM package manager. Use the following commands to get the actions-runner-controller repository.</p><pre>helm repo add actions-runner-controller <a href="https://actions-runner-controller.github.io/actions-runner-controller">https://actions-runner-controller.github.io/actions-runner-controller</a></pre><pre>helm repo update</pre><p>We will need to createvalues.yaml manifest in which we are going to insert the three secrets obtained (App ID, Installation ID, and private key) and an additional random token as a webhook secret. It is a pretty long file so instead of pasting it in this article, you can copy the example file from the original repository <a href="https://github.com/Link-/actions-at-scale-ghoh/blob/main/actions-runner-controller/values.yaml.example">here</a>. I suggest you do CTRL+F after the file is copied and search for the following keys:</p><ul><li>github_app_id</li><li>github_app_installation_id</li><li>github_app_private_key</li><li>github_webhook_secret_token</li></ul><p>Then, insert the corresponding values next to those keys. For the additional github_webhook_secret_token key, I used a python library called secretsto generate a random hash value for it:</p><pre>import secrets<br># webhook_token variable contains the random token<br>webhook_token = secrets.token_urlsafe(16)<br>print(webhook_token) # copy the output value</pre><p>Save the manifest. With all the parameters set-up, type the following to deploy the controller:</p><pre>helm upgrade --install -f &quot;path/to/values.yaml&quot;\<br>             --namespace default\<br>             --create-namespace --wait \<br>             actions-runner-controller \<br>             &quot;actions-runner-controller/actions-runner-controller&quot;</pre><p>Check that the deployments were successful:</p><pre>kubectl get pods -n default</pre><p>We should have the following output:</p><pre>NAME                                                              READY   STATUS    RESTARTS   AGE <br>actions-runner-controller-546dfcc8b8-wc7mt                        2/2     Running   0          20s <br>actions-runner-controller-github-webhook-server-6bb9654dc8ckjgk   2/2     Running   0          20s <br>test-app-65d7765fdd-6rff6                                         1/1     Running   0          16h</pre><h3>Updating the Ingress for our actions runner controller</h3><p>Recall our ingress manifest called ingress-tls.yaml. Search for the httpkey. We want to insert the following path /runners-scaler right bellow the path /where our test app is:</p><pre>- path: /runners-scaler <br>  pathType: Prefix <br>  backend: <br>    service: <br>      name: actions-runner-controller-github-webhook-server <br>      port:  <br>        number: 80</pre><p>Then, just update the ingress:</p><pre>kubectl apply -f ingress-tls.yaml</pre><p>Verify that everything is working correctly:</p><pre>kubectl describe ingress ingress-main</pre><p>On the output, search for Events and you should see something similar to this:</p><pre>Events: <br>  Type    Reason             Age   From          Message <br>  ----    ------             ----  ----          ------- <br>  Normal  CreateCertificate  14s   cert-manager  Successfully created Certificate &quot;tlsingress&quot;</pre><p>Furthermore, use your browser or curl -G and insert your URL with the additional route that was just created, i.e: &lt;your-url&gt;/runners-scaler. It should return a message saying Webhook server is running. That means everything is ok! Eureka!</p><h3>Adding the Webhook URL to the GitHub APP</h3><p>Go back to the App Settings of your GitHub App. Under the “Webhook URL” textbox, insert the URL with the runner-scaler route. Right after that, there is a textbox called “Webhook secret”. Paste the hash that you have on your values.yamlfile that was generated using the previous python snippet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/739/1*_9J6wpKwW4IYxbytZpoCxw.png" /><figcaption>Enabling the Webhook on the GitHub App.</figcaption></figure><p>Save the changes. Finally, on the “Permissions &amp; Events” tab, scroll down to “Subscribe to events” and check:</p><ul><li>Workflow job</li><li>Workflow dispatch</li><li>Workflow Run</li></ul><p>Save your changes again. On the “Advanced” tab, you should see something like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/782/1*symU4w-KdMBadHnCkDV3XQ.png" /><figcaption>Recent Deliveries capture.</figcaption></figure><p>Which means that everything is good!</p><h3>Spinning the runners</h3><p>There are two ways the runners can be created. One is using deployments of kind Runnerand the other is using RunnerDeployment. On this tutorial, we are using RunnerDeployment as it will allow us to manage sets of runners. The Runner class only manages them individually.</p><h3>RunnerDeployment</h3><p>We need to create a YAML manifest for your deployment. Let’s call it basic-runners.yaml. Start by including the RunnerDeploymentkind:</p><pre>apiVersion: actions.summerwind.dev/v1alpha1<br>kind: RunnerDeployment<br>metadata:<br>  name: azure-runners<br>  namespace: default<br>spec:<br>  replicas: 0<br>  template:<br>    spec:<br>      organization: &lt;your-organization&gt;<br>      labels:<br>        - label1<br>        - label2<br>      env:<br>      - name: RUNNER_FEATURE_FLAG_EPHEMERAL<br>        value: &quot;true&quot;</pre><p>Without autoscaling, we can manually define the number of runners to create by including the replicas key, inside the spec key.</p><h3>HorizontalRunnerAutoscaler</h3><p>Autoscaling is implemented by backing a RunnerDeployment resource kind with a HorizontalRunnerAutoscalerkind. With this, every time there is a webhook event, the runners can be created automatically and then destroyed when finished. We need to append the specifications for the autoscaler to the basic-runners.yamlmanifest:</p><pre>---<br>apiVersion: actions.summerwind.dev/v1alpha1<br>kind: HorizontalRunnerAutoscaler<br>metadata:<br>  name: azure-runners-deployment-autoscaler<br>  namespace: default<br>spec:<br>  scaleTargetRef:<br>    name: azure-runners<br>  minReplicas: 0<br>  maxReplicas: 6<br>  scaleUpTriggers:<br>  - githubEvent: {}<br>    duration: &quot;1m&quot;</pre><p>and apply the resources:</p><pre>kubectl apply -f basic-runners.yaml</pre><p>With the above example, the webhook server scales example-runners by 1 replica, and destroys that runner 1 minute after it has finished its job.</p><p>By now you show be able to automatically create runners whenever there is a GitHub workflow queued on your actions tab. Only the jobs that match the labels that we set on basic-runners.yaml will be executed by our runners (you can see that in the YAML file, there are two labels called label1 and label2).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/893/1*q1H1IX5WNeO8pLVg4ZRWwg.png" /><figcaption>Testing the controller with a GitHub Actions workflow.</figcaption></figure><p><strong>Note:</strong> You can also create customized runners if you need certain tools or dependencies installed before doing any job. But, that is a topic for another article.</p><p>If you made it through here, thank you for reading my post. I know there are lots of steps, but it is really worth considering if you need a great a scalable self-hosted runners solution for GitHub actions. Check out my other articles if you like.</p><p>See you on the next one!</p><p><a href="https://github.com/hector6298">hector6298 - Overview</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=509052905817" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Save time by building a continuous training pipeline for ML. Part 1]]></title>
            <link>https://medium.com/analytics-vidhya/save-time-by-building-a-continuous-training-pipeline-for-ml-part-1-799e95d7c0ec?source=rss-4f706205679a------2</link>
            <guid isPermaLink="false">https://medium.com/p/799e95d7c0ec</guid>
            <category><![CDATA[ml-engineering]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[github-actions]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Hector Andres Mejia Vallejo]]></dc:creator>
            <pubDate>Fri, 28 Jan 2022 21:24:54 GMT</pubDate>
            <atom:updated>2022-05-22T23:47:09.265Z</atom:updated>
            <content:encoded><![CDATA[<h3>Save time by building a continuous training pipeline for ML</h3><h4>Design of the pipeline components</h4><p>From early projects, my team and I used to spend large amounts of time training, testing, writing the results, and deploying the model manually for the business case. It takes valuable time that could be used on research and innovation and puts it on tasks that would otherwise be saved using the right tools and some planning.</p><p>Let’s discuss here how to create a continuous training pipeline for your ML models using <strong>MLflow, scikit-learn</strong>, and <strong>Delta Lake</strong>. In addition, I will apply these tools for <strong>Databricks</strong>, which I use daily, but the concepts can be still applied if you use other platforms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VWjHSUgy27gx5Ygo" /><figcaption>Totally unrelated, but very peaceful image. Photo by <a href="https://unsplash.com/@billow926?utm_source=medium&amp;utm_medium=referral">billow926</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>0. Overview of the process</h3><p>Let us look at the following diagram to see which components should we build to create an automated training pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WPn4jmtt0JJWZqmO_6Euxg.png" /><figcaption>Continuous Training Pipeline. Made with ❤ by me.</figcaption></figure><p>We have <strong>data ETL</strong>, <strong>training</strong>, <strong>registry, </strong>and <strong>serving </strong>as essential components. In addition, we have isolated Databricks workspaces for development and production so we have to figure out a way to send the specifications for the model that will be registered in the Databricks production environment.</p><h3>1. Data ETL component</h3><p>This component depends on your use case. We can capture data from files, APIs, data streams, etc. Whatever the case may be, we can capture and process data for machine learning modeling in the following fashion:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/127/1*5sRRoeL2ey8L5Ol1sdKRuQ.png" /><figcaption>ETL diagram. Made by me.</figcaption></figure><ul><li>The captured data goes to a Data Lake or staging area and is stored as-is.</li><li>From this repository, the raw data is cleaned, validated for data types, and feature extraction is performed.</li><li>This pre-processed data is stored as a database table in a data warehouse. In my case, I use delta tables.</li><li>Train/validation/test splits are created and saved on the data warehouse.</li></ul><p>One great thing about using delta as a data lake and warehouse is that it implements versioning and time travel, so we never lose track of the evolution of our data. In addition, it is meant to be scalable.</p><h3>2. Model training component</h3><p>For a continuous training pipeline, we need to record the model performance for a given iteration, retrieve that metadata with the model, compare with previous versions, and set the best production model. Everything has to be implemented programmatically. Luckily, some tools will help us achieve this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/129/1*NhvYU72H5P7_gvM2G55ocg.png" /><figcaption>Model training diagram. Made by me</figcaption></figure><p>This component will use the <a href="http://hyperopt.github.io/hyperopt/"><strong>Hyperopt</strong></a><strong> </strong>package for hyperparameter tuning. It receives a function that trains a model and returns the best hyperparameters. Furthermore, that trained model is going to be tracked by <strong>MLflow. </strong>It will<strong> </strong>log its metrics, parameters, performance plots like ROC, and save the model binary. I will not show the code in this article since it is a little large. You can check it out in this <a href="https://gist.github.com/1c9a279564bae782800cd29c03732df8.git">gist</a>. For now, assume a function with the following signature:</p><pre>def run_experiment(experiment_id, run_name, project_path,<br>                   search_space, classifier_type, <br>                   X_train, X_test, X_valid,<br>                   y_train, y_test, y_valid,<br>                   model_binary=None, roc=False, max_evals = 150):</pre><ul><li>The argument experiment_id is an integer value that represents a MLflow experiment. An experiment is where the training runs will be stored, with all the information that we log as you saw on the mlflow_training snippet. You can create an experiment using the GUI or programmatically:</li></ul><pre>experiment_id = mlflow.create_experiment(experiment_name)</pre><ul><li>The run_name argument is just a string to name the MLflow run.</li><li>project_path is where all the artifacts will be stored.</li><li>search_space is the grid that will be explored by Hyperopt. For instance, a valid search space for aSGDClassifiercould be:</li></ul><pre>search_space = {<br>      &#39;alpha&#39;: hp.uniform(&#39;alpha&#39;,0,0.0001),<br>      &#39;max_iter&#39;:  scope.int(hp.quniform(&#39;max_iter&#39;, 50, 1000, 50)),<br>      &#39;l1_ratio&#39; : hp.uniform(&#39;l1_ratio&#39;,0.1,0.3),<br>      &#39;penalty&#39;: &quot;elasticnet&quot;,<br>      &#39;loss&#39;: &quot;modified_huber&quot;<br>    }</pre><ul><li>classifier_type is the scikit learn classifier class, e.g SGDClassifier.</li><li>x_train, x_test, y_train, y_test, x_valid, y_valid are the data splits.</li><li>max_evals is the maximum number of evaluations that hyperopt will perform for the tuning.</li></ul><p>Then we call the function like this:</p><pre>best_params, run id = run_experiment(experiment_id, <br>                                     model_name, <br>                                     project_path, <br>                                     search_space, <br>                                     SGDClassifier, <br>                                     X_train, X_test, X_valid, <br>                                     y_train, y_test, y_valid, <br>                                     model_binary=None, <br>                                     roc=True)</pre><h3>3. Model registry component</h3><p>With the training component, all our runs will be logged. However, we need a mechanism to register the best model out of all those runs, set it for production, and retrieve it whenever necessary. All that programmatically. This is where MLflow model registry comes in.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/110/1*v7ogGBSm7QabfUcs-_wvEw.png" /><figcaption>Model registry diagram. Made with ❤ by me.</figcaption></figure><p>From the training component, you can see that each run is set as a candidate run with mlflow.set_tag(&quot;candidate&quot;, “True&quot;). On this registry component we will retrieve all candidate runs, and all the versions of registered models, if applicable. Then, we will get the best run and the best-registered model in terms of AUC. After that, the model of the best run and the best-registered model are compared to set a final model as production.</p><p>You can check out the complete gist <a href="https://gist.github.com/c25d116bc317040b4d7f1035419d2bc3.git">here</a>. For now let’s check the signature for the public functions of the class ModelEvaluationPipeline:</p><pre>class ModelEvaluationPipeline():<br>    def  __init__(self, spark, <br>                  experimentID, <br>                  model_name,  <br>                  bootstrap_ratio):</pre><pre>    def run(self, X_test, Y_test):</pre><ul><li>experimentID is the same integer number used in the training component.</li><li>model_name is a string to name the model on the registry. If it already exists, then the versions of the model under that name will be also compared with the candidate runs.</li><li>The evaluation is performed by sampling multiple times the targets and predictions, then calculating AUC for each of them. This gives an array of AUCs. bootstrap_ratio is the minimum percentage that the current AUC array has to surpass by comparing it to the best AUC array so far.</li></ul><p>We call this component as follows:</p><pre>eval_pipeline = ModelEvaluationPipeline(spark, experiment_id,           <br>                                        model_name, bootstrap_ratio)<br>eval_pipeline.run(docs, y_test)</pre><p>If we only have one Databricks environment for every stage, then our CT pipeline stops here since we end up with a model set as Production ready by MLflow model registry. However, if for some reason, there are isolated environments, then we need to be able to send the artifacts from the development environment to the production environment.</p><h3>4. Model ublishing Component</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/108/1*r98_P3LEQEy-FfZvvocv0w.png" /><figcaption>Model Publishing component. Made by me.</figcaption></figure><p>This component works only if you have two isolated Databricks workspaces for development and production and you need to send the ML model to production when it is ready. My approach was just to have a shared filesystem where the model artifacts would go from the development environment. Then, the production environment would pick up those files and register the model on MLflow.</p><ol><li>First, we mount the shared storage on both Databricks environments. I use Azure Blob Storage. Check out how to do it <a href="https://docs.microsoft.com/en-us/azure/databricks/data/data-sources/azure/azure-storage">here</a>.</li><li>Send the artifacts after the model was registered in the development environment.</li></ol><p>Check out the gist <a href="https://gist.github.com/1fe8cb6be8c7485d63f2207a6cc14f68.git">here</a> for steps 1 and 2.</p><p>3. Track the model on the production environment, reusing the run_experiment function that was created in the model training component. The difference is that the parameter model_binary should have the artifact.</p><pre>run_experiment(experiment_id_prod, model_name, <br>               X_test=X_test, y_test=y_test,         <br>               model_binary=model_binary,<br>               roc=True)</pre><p>4. Now you should run the register component in the same way as we did on the development environment.</p><pre>eval_pipeline = ModelEvaluationPipeline(spark, prod_experiment_id,           <br>                                        model_name, bootstrap_ratio)<br>eval_pipeline.run(docs, y_test)</pre><p>That is it for the pipeline design. These are only the components needed to train, tune and register an ML model. On a later post, I will show the some orchestration tools to automate this continuous training process. Thank you for your support. If you liked this article, please clap.</p><p>See ya!</p><p><a href="https://github.com/hector6298">hector6298 - Overview</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=799e95d7c0ec" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/save-time-by-building-a-continuous-training-pipeline-for-ml-part-1-799e95d7c0ec">Save time by building a continuous training pipeline for ML. Part 1</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>